To investigate speech in the developing brain, 94 children aged 4 to 7 years old were scanned using diffusion weighted imaging (DWI) magnetic resonance imaging. To increase sample size and performance variability, we included children with ADHD from a larger ongoing study (n = 47). Each child completed the Syllable Repetition Task (SRT), a validated measure of phoneme articulation. DWI data were modeled using restriction spectrum imaging to measure restricted and hindered diffusion properties in gray and white matter. We analyzed the diffusion data using whole brain analysis and automated fiber quantification (AFQ) analysis to establish tract profiles for the six fiber pathways thought to be important for supporting speech development. In the whole brain analysis, we found that SRT performance was associated with restricted diffusion in left and right inferior frontal gyrus, left and right pars opercularis, right pre-supplementary and supplementary motor area, and left and right cerebellar gray matter (p < 0.005). Age moderated these associations in left pars opercularis and the frontal aslant tract (FAT), but only the cerebellar findings survived a cluster correction. Analyses using AFQ highlighted differences in high and low performing children along specific tract profiles, most notably in left but not right FAT, in left and right superior longitudinal fasciculus III, and in the cerebellar peduncles. These findings suggest that individual differences in speech performance are reflected in structural gray and white matter differences as measured by restricted and hindered diffusion metrics, and offer important insights into developing brain networks supporting speech in very young children.