Research on emotion recognition ability (ERA) can inform the measurement of the emotion perception component in emotional intelligence. However, to date the question of whether ERA is a single unitary ability or whether independent skills are involved in the recognition of different modalities and/or emotions has been neglected. We studied this issue with the help of two ERA tests drawn from two emotion portrayal corpora. In Study 1, we investigated the dimensional structure of ERA in a set of 10 emotions presented in four modalities (audio, video, still picture, audio–video). In Study 2, we investigated a set of 14 emotions in the audio–video modality. Our results suggest that ERA might be conceptualized as a broad ability consisting of related skills involved in the recognition of positive and negative emotions. In addition, correlated residuals between pairs of similar emotions (e.g., irritation and anger) suggest the existence of specific ability facets within the valence-based skill dimensions.