Direction of another person’s eye gaze provides crucial information about their attention and intentions, which is essential for an effective social interaction. Event-related potential (ERP) measures offer precise temporal tracking of neural processes related to gaze perception. While the sensitivity of the ERP component N170 to face processing is principally agreed, the research on gaze direction effect on this component is thus far inconsistent. Here, we systematically reviewed literature on the sensitivity of N170 to gaze direction. We analysed if four factors, known to affect the face N170 (i.e., emotion, face orientation, task demand, and stimuli motion), were modulated by gaze direction. N170 sensitivity to gaze was reported the most in the studies that involved deviated faces, dynamic stimuli, and that used explicit tasks directly related to gaze or face processing. The present review provides a much-needed summary of the literature to date, highlighting the complexity of the effect of gaze direction on the N170 component, and the need of systematic studies investigating the combination of these factors.