Every day we encounter new people, interact with them, and form person impressions based on quick and automatic inferences from minimal contextual information. Previous studies have identified an extensive network of brain areas involved in familiar face recognition, but there is little evidence to date concerning the neural bases of negative vs. positive person impressions. In the present study, participants were repeatedly exposed to 16 unfamiliar face identities within a pseudo-interactive game context to generate a perception of either "friends" or "foes". Functional magnetic resonance imaging (fMRI) was then performed during an old/new memory task to assess any difference in brain responses to these now familiar face identities, relative to unfamiliar faces. Importantly, whereas facial expressions were always emotional (either smiling or angry) during the encoding phase, they were always neutral during the memory task. Our results reveal that several brain regions involved in familiar face recognition, including fusiform cortex, posterior cingulate gyrus, and amygdala, plus additional areas involved in motivational control such as caudate and anterior cingulate cortex, were differentially modulated as a function of a previous encounter, and generally more activated when faces were perceived as "foes" rather than "friends". These findings underscore that a key dimension of social judgments, based on past impressions of who may be supportive or hostile, may lead to long-lasting effects on memory for faces and thus influence affective reactions to people during a subsequent encounter even in a different (neutral) context.