Impairments of set shifting have been associated with damage to both the prefrontal cortex (PFC) and to the basal ganglia. The purpose of these experiments was to determine whether damage to the PFC was associated with shifting impairments per se or whether any switching deficits could be attributed to a reduction of working memory capacity. In contrast, shifting impairments were expected for Parkinson patients regardless of memory load, given that these patients seem to have no cognitive deficits other than when having to shift set. To vary working memory demands, a cue to the relevant dimension (letter or shape) in an odd-man-out task was presented or withheld. Pathology to prefrontal areas associated with normal aging was not linked to shifting deficits when working memory load was reduced in a comparison of older and younger adults (Experiment 1). In contrast, set-shifting abilities were still impaired for stroke patients with prefrontal damage regardless of working memory demands (Experiment 2). Parkinson patients were relatively unimpaired on this task (Experiment 2), but began to display shifting deficits when response competition was present in the display (Experiment 3).