The inferior frontal gyrus and inferior parietal lobe have been characterized as human homologues of the monkey “mirror neuron” system, critical for both action production (AP) and action recognition (AR). However, data from brain lesion patients with selective impairment on only one of these tasks provide evidence of neural and cognitive dissociations. We sought to clarify the relationship between AP and AR, and their critical neural substrates, by directly comparing performance of 131 chronic left-hemisphere stroke patients on both tasks—to our knowledge, the largest lesion-based experimental investigation of action cognition to date. Using voxel-based lesion-symptom mapping, we found that lesions to primary motor and somatosensory cortices and inferior parietal lobule were associated with disproportionately impaired performance on AP, whereas lesions to lateral temporo-occipital cortex were associated with a relatively rare pattern of disproportionately impaired performance on AR. In contrast, damage to posterior middle temporal gyrus was associated with impairment on both AP and AR. The distinction between lateral temporo-occipital cortex, critical for recognition, and posterior middle temporal gyrus, important for both tasks, suggests a rough gradient from modality-specific to abstract representations in posterior temporal cortex, the first lesion-based evidence for this phenomenon. Overall, the results of this large patient study help to bring closure to a long-standing debate by showing that tool-related AP and AR critically depend on both common and distinct left hemisphere neural substrates, most of which are external to putative human mirror regions.