Speech perception can decline in middle age even when hearing thresholds remain normal, and the underlying neurobiological mechanisms are not well understood. In line with the age-related neural dedifferentiation hypothesis, we predicted that middle-aged adults show less distinct cortical representations of phonemes and acoustic-phonetic features relative to younger adults. In addition to an extensive audiological, auditory electrophysiological, and speech perceptual test battery, we measured electroencephalographic responses time-locked to phoneme instances (phoneme-related potential) in naturalistic, continuous speech and trained neural network classifiers to predict phonemes from these responses. Consistent with age-related neural dedifferentiation, phoneme predictions were less accurate, more uncertain, and involved a broader network for middle-aged adults compared with younger adults. Representational similarity analysis revealed that the featural relationship between phonemes was less robust in middle age. Electrophysiological and behavioral measures revealed signatures of putative cochlear neural degeneration (CND) and speech perceptual deficits in middle-aged adults relative to younger adults. In line with prior work in animal models, proxies of CND were associated with greater cortical dedifferentiation, explaining nearly a third of the variance in PRP prediction accuracy together with measures of acoustic neural processing. Notably, even after controlling for CND proxies and acoustic processing abilities, age-group differences in cortical PRP prediction accuracy remained. Overall, the results reveal “fuzzier” cortical phonemic representations in middle age, suggesting that age-related neural dedifferentiation may underlie speech perceptual challenges despite a normal audiogram.