What makes people experience varying degrees of empathy? Common accounts emphasize interpersonal attributes, including victims' group membership or social proximity to observers. Here we elucidate a distinct process: imagining the scenes surrounding victims. Although imagination has been shown to moderate empathy, the relative importance of its representational components is unknown. Using fMRI (N = 48), we identified activation maps preferentially associated with imagining places and persons, respectively. When participants imagined misfortunes happening to individuals in specific places, the place and person maps jointly predicted affective empathy, and, less consistently, prosocial behavior. Crucially, place-preferential activation was at least as predictive as person-preferential activation. Results were robust to several group-, participant-, trial-level, and covariate-adjusted analyses. Moreover, place-preferential activation itself was most strongly predicted by person liking, beyond place-related ratings. Our data are consistent with social affinity potentiating place imagination, which in turn increases empathy, above and beyond person imagination. These findings challenge person-centric views of empathy by suggesting that place-related representations are central to empathy and socially contingent.