Our everyday actions are often performed in the context of a social interaction. We previously showed that, in adults, selecting an action on the basis of either social or symbolic cues was associated with activations in the fronto-parietal cognitive control network, whereas the presence and use of social versus symbolic cues was in addition associated with activations in the temporal and medial prefrontal cortex (MPFC) social brain network. Here we investigated developmental changes in these two networks. Fourteen adults (21–30 years of age) and 14 adolescents (11–16 years) followed instructions to move objects in a set of shelves. Interpretation of the instructions was conditional on the point of view of a visible “director” or the meaning of a symbolic cue (Director Present vs. Director Absent) and the number of potential referent objects in the shelves (3-object vs. 1-object). 3-object trials elicited increased fronto-parietal and temporal activations, with greater left lateral prefrontal cortex and parietal activations in adults than adolescents. Social versus symbolic information led to activations in superior dorsal MPFC, precuneus, and along the superior/middle temporal sulci. Both dorsal MPFC and left temporal clusters exhibited a Director × Object interaction, with greater activation when participants needed to consider the directors' viewpoints. This effect differed with age in dorsal MPFC. Adolescents showed greater activation whenever social information was present, whereas adults showed greater activation only when the directors' viewpoints were relevant to task performance. This study thus shows developmental differences in domain-general and domain-specific PFC activations associated with action selection in a social interaction context.