Though we can almost pre-attentively categorize the valence of facial expressions, we experience emotional ambiguity when confronted with facial expressions in a context with incongruent emotional information. We simultaneously presented interfering background colors during forced-choice categorizations of negative (fear), neutral and positive (happy) expressions. Conflicting information induced strong and differential interference effects on a behavioral level which was mirrored in comparable activations on a neuronal level. Besides a common fronto-parietal attention network which was activated during interference resolution, we found differential interference effects for facial expressions. Incongruent trials with neutral expressions induced a distinct activation pattern in ventral visual regions particularly involved in deeper analysis for both the task-relevant facial expressions (fusiform (FFA) and occipital face area (OFA)) and the task-irrelevant color (V4). Compared to neutral expressions, incongruent trials including either negative or positive expressions elicited attenuated interference effects. Unlike incongruent trials with positive facial expressions which showed only sparse activation in frontal cortex, interference resolution during processing of negative facial expressions resulted in specific activations in regions (V3a, MT+, STS) which might be involved in processing of implicit dynamics of negative expressions. Thus, functional activations in visual processing regions might specifically be related to processing demands of different expressions.