Neuropsychological evidence has highlighted the role of the anterior temporal lobes in the processing of conceptual knowledge. That putative role is only beginning to be investigated with fMRI as methodological advances are able to compensate for well-known susceptibility artifacts that affect the quality of the BOLD signal. In this article, we described differential BOLD activation for pictures of animals and manipulable objects in the anterior temporal lobes, consistent with previous neuropsychological findings. Furthermore, we found that the pattern of BOLD signal in the anterior temporal lobes is qualitatively different from that in the fusiform gyri. The latter regions are activated to different extents but always above baseline by images of the preferred and of the nonpreferred categories, whereas the anterior temporal lobes tend to be activated by images of the preferred category and deactivated (BOLD below baseline) by images of the nonpreferred category. In our experimental design, we also manipulated the decision that participants made over stimuli from the different semantic categories. We found that in the right temporal pole, the BOLD signal shows some evidence of being modulated by the task that participants were asked to perform, whereas BOLD activity in more posterior regions (e.g., the fusiform gyri) is not modulated by the task. These results reconcile the fMRI literature with the neuropsychological findings of deficits for animals after damage to the right temporal pole and suggest that anterior and posterior regions within the temporal lobes involved in object processing perform qualitatively different computations.