Testing older adults in the morning generally improves behavioral performance relative to afternoon testing. Morning testing is also associated with brain activity similar to that of young adults. Here, we used graph theory to explore how time of day (TOD) affects the organization of brain networks in older adults across rest and task states. We used nodes from the automated anatomical labeling atlas to construct participant-specific correlation matrices of fMRI data obtained during 1-back tasks with interference and rest. We computed pairwise group differences for key graph metrics, including small-worldness and modularity. We found that older adults tested in the morning and young adults did not differ on any graph metric. Both of these groups differed from older adults tested in the afternoon during the tasks—but not rest. Specifically, the latter group had lower modularity and small-worldness (indices of more efficient network organization). Across all groups, higher modularity and small-worldness strongly correlated with reduced distractibility on an implicit priming task. Increasingly, TOD is seen as important for interpreting and reproducing neuroimaging results. Our study emphasizes how TOD affects brain network organization and executive control in older adults.