Can our emotions alter our ability to infer emotions and feel pain in others? Answering this question may shed light on human social brain functions and help avoid misunderstandings in real life. In experiment 1, participants watched joyful or fearful movie-clips, and then judged fear or joy in others based on reading about their situation. Using fMRI, we found reduced activity in brain areas implicated in mentalizing (superior temporal gyrus, precuneus, frontoparietal operculum) when others' emotions differed from those experienced by the participant. This accord with embodied accounts of affective mentalizing. In experiment 2, brain response to others' pain was reduced after watching fearful movie-clips in regions involved in empathy for pain (anterior insula). This supports a Broaden-and-build account where positive or negative affect respectively amplifies or attenuates reactivity to others. This work reveals that our emotions may dampen brain response to emotions and pain in others through different mechanisms.