Individuals with post-stroke aphasia have long been observed to show relatively preserved musical and rhythm abilities in the presence of varied, and often profound, language impairments. Accordingly, speech-language pathologists frequently use rhythm-based strategies (e.g., tapping) to facilitate speech output in people with aphasia. However, there is little empirical work to support the clinical practice of using rhythm techniques. In this study, we investigated the neural bases of rhythm in aphasia by combining thorough behavioral rhythm assessments with structural brain imaging. Individuals with chronic, post-stroke aphasia (n = 33) and a matched neurotypical control group (n = 29) completed a rigorous battery of rhythm production and perception tasks. We found marked individual variability within the aphasia group, with about one third of individuals showing impaired rhythm processing, while the remaining two-thirds performed within the control range. Using lesion-symptom mapping, we found that individual variability in tapping performance was associated with damage to a left temporoparietal area, extending into white matter specifically in the arcuate fasciculus. That is, individuals who struggled with tapping tended to have damage to this region. Tapping was also associated with language production scores, but not motor speech, in the aphasia group. These findings, which systematically link rhythm, language, and the brain, have the potential to be translated into clinical practice for understanding which patients may benefit the most from rhythm-based treatments. Our study in a population with focal brain injury complements evolutionary work highlighting the importance of the left temporoparietal region and underlying white matter for beat synchronization.