Affective and social processes play a major role in everyday life, but appropriate methods to assess disturbances in these processes after brain lesions are still lacking. Past studies have shown that amygdala damage can impair recognition of facial expressions, particularly fear, as well as processing of gaze direction; but the mechanisms responsible for these deficits remain debated. Recent accounts of human amygdala function suggest that it is a critical structure involved in self-relevance appraisal. According to such accounts, responses to a given facial expression may vary depending on concomitant gaze direction and perceived social meaning. Here we investigated facial emotion recognition and its interaction with gaze in patients with unilateral amygdala damage (n = 19), compared to healthy controls (n = 10), using computer-generated dynamic face stimuli expressing variable intensities of fear, anger or joy, with different gaze directions (direct versus averted). If emotion perception is influenced by the self-relevance of expression based on gaze direction, a fearful face with averted gaze should be more relevant than the same expression with direct gaze because it signals danger near the observer; whereas anger with direct gaze should be more relevant than with averted gaze because it directly threatens the observer. Our results confirm a critical role for the amygdala in self-relevance appraisal, showing an interaction between gaze and emotion in healthy controls, a trend for such interaction in left-damaged patients but not in right-damaged patients. Impaired expression recognition was generally more severe for fear, but with a greater deficit for right versus left damage. These findings do not only provide new insights on human amygdala function, but may also help design novel neuropsychological tests sensitive to amygdala dysfunction in various patient populations.