Accumulating literature has linked poverty to brain structure and function, particularly in affective neural regions; however, few studies have examined associations with structural connections or the importance of developmental timing of exposure. Moreover, prior neuroimaging studies have not used a proximal measure of poverty (i.e., material hardship, which assesses food, housing, and medical insecurity) to capture the lived experience of growing up in harsh economic conditions. The present investigation addressed these gaps collectively by examining the associations between material hardship (ages 1, 3, 5, 9, and 15 years) and white matter connectivity of frontolimbic structures (age 15 years) in a low-income sample. We applied probabilistic tractography to diffusion imaging data collected from 194 adolescents. Results showed that material hardship related to amygdala–prefrontal, but not hippocampus–prefrontal or hippocampus–amygdala, white matter connectivity. Specifically, hardship during middle childhood (ages 5 and 9 years) was associated with greater connectivity between the amygdala and dorsomedial pFC, whereas hardship during adolescence (age 15 years) was related to reduced amygdala–orbitofrontal (OFC) and greater amygdala–subgenual ACC connectivity. Growth curve analyses showed that greater increases of hardship across time were associated with both greater (amygdala–subgenual ACC) and reduced (amygdala–OFC) white matter connectivity. Furthermore, these effects remained above and beyond other types of adversity, and greater hardship and decreased amygdala–OFC connectivity were related to increased anxiety and depressive symptoms. Results demonstrate that the associations between material hardship and white matter connections differ across key prefrontal regions and developmental periods, providing support for potential windows of plasticity for structural circuits that support emotion processing.