Humans are able to categorize face properties with impressively short latencies. Nevertheless, the latency at which gaze recognition occurs is still a matter of debate. Through spatio-temporal analysis of high-density event-related potentials (ERP), we investigated the brain activity underlying the ability to spontaneously and quickly process gaze. We presented neutral faces with direct and averted gaze in a matching picture paradigm, where subjects had to detect repetition of identical faces and gaze was implicitly manipulated. The results indicate that faces with averted gaze were better discriminated than faces with direct gaze, and evoked stronger P100 amplitudes localized to the right fusiform gyrus. In contrast, direct gaze induced stronger activation in the orbital frontal gyrus at this latency. Later in time, at the beginning of the N170 component, direct gaze induced changes in scalp topography with a stronger activation in the right medial temporal gyrus. The location of these differential activations of direct vs. averted gaze further support the view that faces with averted gaze are perceived as less rewarding than faces with direct gaze. We additionally found differential ERP responses between repeated and novel faces as early as 50ms, thereby replicating earlier studies of very fast detection of mnestic aspects of stimuli. Together, these results suggest an early dissociation between implicit gaze detection and explicit identity processing.