Humans are highly social beings that interact with each other on a daily basis. In these complex interactions, we get along by being able to identify others' actions and infer their intentions, thoughts and feelings. One of the major theories accounting for this critical ability assumes that the understanding of social signals is based on a primordial tendency to simulate observed actions by activating a mirror neuron system. If mirror neuron regions are important for action and emotion recognition, damage to regions in this network should lead to deficits in these domains. In the current behavioural and EEG study, we focused on the lateral prefrontal cortex including dorsal and ventral prefrontal cortex and utilized a series of task paradigms, each measuring a different aspect of recognizing others' actions or emotions from body cues. We examined 17 patients with lesions including (n = 8) or not including (n = 9) the inferior frontal gyrus, a core mirror neuron system region, and compared their performance to matched healthy control subjects (n = 18), in behavioural tasks and in an EEG observation-execution task measuring mu suppression. Our results provide support for the role of the lateral prefrontal cortex in understanding others' emotions, by showing that even unilateral lesions result in deficits in both accuracy and reaction time in tasks involving the recognition of others' emotions. In tasks involving the recognition of actions, patients showed a general increase in reaction time, but not a reduction in accuracy. Deficits in emotion recognition can be seen by either direct damage to the inferior frontal gyrus, or via damage to dorsal lateral prefrontal cortex regions, resulting in deteriorated performance and less EEG mu suppression over sensorimotor cortex.