We perceive emotions in the voices we hear, but do we handle positive and negative emotions the same way? Even more so if we expect, or not, a specific emotion? We tried to address this question using EEG to analyze activity generated by the categorization or the discrimination of auditory affect bursts. We found a greater activation for the categorization task at P200, suggesting a greater cognitive load in that type of task. We found a more negative activation for happiness than for fear at N100, suggesting that emotion is perceived very early through its physical characteristics. Some interaction effects showed that discrimination of happiness is lateralized at N100 according to whether it is a target emotion or not. Emotions receive a lateralized different processing depending on whether they are categorized or discriminated at P300. Consequently, the type of perceptual decision making has to be considered when studying emotion perception.