The pFC is critical for cognitive flexibility (i.e., our ability to flexibly adjust behavior to changing environmental demands), but also for cognitive stability (i.e., our ability to follow behavioral plans in the face of distraction). Behavioral research suggests that individuals differ in their cognitive flexibility and stability, and neurocomputational theories of working memory relate this variability to the concept of attractor stability in recurrently connected neural networks. We introduce a novel task paradigm to simultaneously assess flexible switching between task rules (cognitive flexibility) and task performance in the presence of irrelevant distractors (cognitive stability) and to furthermore assess the individual “spontaneous switching rate” in response to ambiguous stimuli to quantify the individual dispositional cognitive flexibility in a theoretically motivated way (i.e., as a proxy for attractor stability). Using fMRI in healthy human participants, a common network consisting of parietal and frontal areas was found for task switching and distractor inhibition. More flexible persons showed reduced activation and reduced functional coupling in frontal areas, including the inferior frontal junction, during task switching. Most importantly, the individual spontaneous switching rate antagonistically affected the functional coupling between inferior frontal junction and the superior frontal gyrus during task switching and distractor inhibition, respectively, indicating that individual differences in cognitive flexibility and stability are indeed related to a common prefrontal neural mechanism. We suggest that the concept of attractor stability of prefrontal working memory networks is a meaningful model for individual differences in cognitive stability versus flexibility.