We cannot recognize all visual features (e.g., colors, shapes, etc.), at all locations at a single moment. Because of capacity and energy constraints, the human visual system must limit detailed processing to small subsets of the available information. One selective mechanism in the visual processing stream is referred to as visual attention. Here, we investigated bottom-up and top-down control processes of visual attention, using psychophysical and electrophysiological measures. In a first study, we found that rare distractors attracted attention more strongly than frequent ones. In two further studies, we demonstrated that the degree to which distractors capture attention is strongly influenced by the distractor-context. These findings suggest that processes analyzing stimulus statistics and stimulus context are critical determinants in the control of visual attention. We discuss how current theories of visual attention can account for these findings, or how they could be modified to accommodate the presented results.