The statement: “An agent harms a victim,” depicts a situation that triggers moral emotions. Depending on whether the agent and the victim are the self or someone else, it can lead to four different moral emotions: self-anger (“I harm myself”), guilt (“I harm someone”), other-anger (“someone harms me”), and compassion (“someone harms someone”). In order to investigate the neural correlates of these emotions, we examined brain activation patterns elicited by variations in the agent (self vs. other) and the victim (self vs. other) of a harmful action. Twenty-nine healthy participants underwent functional magnetic resonance imaging while imagining being in situations in which they or someone else harmed themselves or someone else. Results indicated that the three emotional conditions associated with the involvement of other, either as agent or victim (guilt, other-anger, and compassion conditions), all activated structures that have been previously associated with the Theory of Mind (ToM, the attribution of mental states to others), namely, the dorsal medial prefrontal cortex, the precuneus, and the bilateral temporo-parietal junction. Moreover, the two conditions in which both the self and other were concerned by the harmful action (guilt and other-anger conditions) recruited emotional structures (i.e., the bilateral amygdala, anterior cingulate, and basal ganglia). These results suggest that specific moral emotions induce different neural activity depending on the extent to which they involve the self and other.