Aphasic patients with multimodal semantic impairment following pFC or temporo-parietal (TP) cortex damage (semantic aphasia [SA]) have deficits characterized by poor control of semantic activation/retrieval, as opposed to loss of semantic knowledge per se. In line with this, SA patients show “refractory effects”; that is, declining accuracy in cyclical word–picture matching tasks when semantically related sets are presented rapidly and repeatedly. This is argued to follow a build-up of competition between targets and distractors. However, the link between poor semantic control and refractory effects is still controversial for two reasons. (1) Some theories propose that refractory effects are specific to verbal or auditory tasks, yet SA patients show poor control over semantic processing in both word and picture semantic tasks. (2) SA can result from lesions to either the left pFC or TP cortex, yet previous work suggests that refractory effects are specifically linked to the left inferior frontal cortex. For the first time, verbal, visual, and nonverbal auditory refractory effects were explored in nine SA patients who had pFC (pFC+) or TP cortex (TP-only) lesions. In all modalities, patient accuracy declined significantly over repetitions. This refractory effect at the group level was driven by pFC+ patients and was not shown by individuals with TP-only lesions. These findings support the theory that SA patients have reduced control over multimodal semantic retrieval and, additionally, suggest there may be functional specialization within the posterior versus pFC elements of the semantic control network.