Some involvement of the human amygdala in the processing of facial expressions has been investigated in neuroimaging studies, although the neural mechanisms underlying motivated or emotional behavior in response to facial stimuli are not yet fully understood. We investigated, using functional magnetic resonance imaging (fMRI) and healthy volunteers, how the amygdala interacts with other cortical regions while subjects are judging the sex of faces with negative, positive, or neutral emotion. The data were analyzed by a subtractive method, then, to clarify possible interaction among regions within the brain, several kinds of analysis (i.e., a correlation analysis, a psychophysiological interaction analysis and a structural equation modeling) were performed. Overall, significant activation was observed in the bilateral fusiform gyrus, medial temporal lobe, prefrontal cortex, and the right parietal lobe during the task. The results of subtraction between the conditions showed that the left amygdala, right orbitofrontal cortex, and temporal cortices were predominantly involved in the processing of the negative expressions. The right angular gyrus was involved in the processing of the positive expressions when the negative condition was subtracted from the positive condition. The correlation analysis showed that activity in the left amygdala positively correlated with activity in the left prefrontal cortex under the negative minus neutral subtraction condition. The psychophysiological interaction revealed that the neural responses in the left amygdala and the right prefrontal cortex underwent the condition-specific changes between the negative and positive face conditions. The right amygdaloid activity also had an interactive effect with activity in the right hippocampus and middle temporal gyrus. These results may suggest that the left and right amygdalae play a differential role in effective processing of facial expressions in collaboration with other cortical or subcortical regions, with the left being related with the bilateral prefrontal cortex, and the right with the right temporal lobe.