Understanding the precise relation between functional connectivity and structural (white matter) connectivity and how these relationships account for cognitive changes in older adults are major challenges for neuroscience. We investigate these issues using an approach in which structural equation modeling (SEM) is employed to integrate functional and structural connectivity data from younger and older adults ( n = 62), analyzed with a common framework based on regions connected by canonical tract groups (CTGs). CTGs (e.g., uncinate fasciculus) serve as a common currency between functional and structural connectivity matrices, and ensure equivalent sparsity in connectome information. We used this approach to investigate the neural mechanisms supporting memory for items and memory for associations, and how they are affected by healthy aging. We found that different structural and functional CTGs made independent contributions to source and item memory performance, suggesting that both forms of connectivity underlie age-related differences in specific forms of memory. Furthermore, the relationship between functional and structural connectivity was best explained by a general relationship between latent constructs—a relationship absent in any specific CTG group. These results provide insights into the relationship between structural and functional connectivity patterns, and elucidate their relative contribution to age-related differences in source memory performance. Author Summary Aging is associated with profound changes in how neural systems adapt to perform the same mental operations in youth. Memory functioning, in particular, demonstrates enormous neuroplastic changes in the pattern of distributed, connected networks that enable older adults to perform the same mnemonic operations. However, the relationship between the structural and functional connections supporting these operations is poorly understood. Here we develop a novel algorithm for comparing structural and functional connectivity, and use a comprehensive structural equation model (SEM) to show how these network characteristics contribute to behavioral performance in two forms of episodic memory retrieval. These results suggest that healthy aging is associated with specific ensembles of cooperative contributions from both functional and structural tract groups.