Recent theories of selective attention assume that the more attention is required by a task, the earlier are irrelevant stimuli filtered during perceptual processing. Previous functional MRI studies have demonstrated that primary visual cortex (V1) activation by peripheral distractors is reduced by higher task difficulty at fixation, but it remains unknown whether such changes affect initial processing in V1 or subsequent feedback. Here we manipulated attentional load at fixation while recording peripheral visual responses with high-density EEG in 28 healthy volunteers, which allowed us to track the exact time course of attention-related effects on V1. Our results show a modulation of the earliest component of the visual evoked potential (C1) as a function of attentional load. Additional topographic and source localization analyses corroborated this finding, with significant load-related differences observed throughout the first 100 ms post-stimulus. However, this effect was observed only when stimuli were presented in the upper visual field (VF), but not for symmetrical positions in the lower VF. Our findings demonstrate early filtering of irrelevant information under increased attentional demands, thus supporting models that assume a flexible mechanism of attentional selection, but reveal important functional asymmetries across the VF.