Prior neuroimaging studies of music-evoked emotions have shown that music listening involves the activation of both cortical and subcortical regions. However, these regions could be differentially activated by music stimuli with varying affective valence and arousal. To better understand the neural correlates involved in the processing of pleasant and unpleasant emotions induced by music, while also considering the effect of arousal, we conducted a quantitative activation likelihood estimate (ALE) meta-analysis. We performed separate ALE analyses for the overall brain activation evoked by listening to emotional music (40 studies), for the brain activation during listening to unpleasant music (15 studies), for the brain activation while listening to pleasant music (17 studies), and for the brain activation while listening to emotional contrasted with neutral music (8 studies). Our results revealed the activation of a range of cortical and subcortical regions, including the amygdala, insula, striatum, thalamus, hippocampus, anterior cingulate gyrus, and superior temporal gyrus. Moreover, our findings indicated that certain regions were specifically activated based on the hedonic valence and arousal of the stimuli. Particularly, whereas the anterior cingulate cortex (ACC), dorsal striatum, and thalamus were dependent on arousal effects, amygdala activation was dependent on hedonic valence. The identification of brain networks preferentially activated during listening to pleasant and unpleasant music provides valuable clinical insights for the development of therapies targeting psychological disorders associated with emotion reactivity problems.