Most computer models for the automatic recognition of emotion from nonverbal signals (e.g., facial or vocal expression) have adopted a discrete emotion perspective, i.e., they output a categorical emotion from a limited pool of candidate labels. The discrete perspective suffers from practical and theoretical drawbacks that limit the generalizability of such systems. The authors of this chapter propose instead to adopt an appraisal perspective in modeling emotion recognition, i.e., to infer the subjective cognitive evaluations that underlie both the nonverbal cues and the overall emotion states. In a first step, expressive features would be used to infer appraisals; in a second step, the inferred appraisals would be used to predict an emotion label. The first step is practically unexplored in emotion literature. Such a system would allow to (a) link models of emotion recognition and production, (b) add contextual information to the inference algorithm, and (c) allow detection of subtle emotion states.