Visuospatial working memory (VSWM) plays an important role in arithmetic problem solving, and the relationship between these two skills is thought to change over development. Even though neuroimaging studies have demonstrated that VSWM and arithmetic both recruit frontoparietal networks, inferences about common neural substrates have largely been made by comparisons across studies. Little work has examined how brain activation for VSWM and arithmetic converge within the same participants and whether there are age-related changes in the overlap of these neural networks. In this study, we examined how brain activity for VSWM and arithmetic overlap in 38 children and 26 adults. Although both children and adults recruited the intraparietal sulcus (IPS) for VSWM and arithmetic, children showed more focal activation within the right IPS, whereas adults recruited the bilateral IPS, superior frontal sulcus/middle frontal gyrus, and right insula. A comparison of the two groups revealed that adults recruited a more left-lateralized network of frontoparietal regions for VSWM and arithmetic compared with children. Together, these findings suggest possible neurocognitive mechanisms underlying the strong relationship between VSWM and arithmetic and provide evidence that the association between VSWM and arithmetic networks changes with age.