Abstract

Cognitive flexibility, the ability to appropriately adjust behavior in a changing environment, has been challenging to operationalize and validate in cognitive neuroscience studies. Here, we investigate neural activation and directed functional connectivity underlying cognitive flexibility using an fMRI-adapted version of the Flexible Item Selection Task (FIST) in adults (n = 32, ages 19–46 years). The fMRI-adapted FIST was reliable, showed comparable performance to the computer-based version of the task, and produced robust activation in frontoparietal, anterior cingulate, insular, and subcortical regions. During flexibility trials, participants directly engaged the left inferior frontal junction, which influenced activity in other cortical and subcortical regions. The strength of intrinsic functional connectivity between select brain regions was related to individual differences in performance on the FIST, but there was also significant individual variability in functional network topography supporting cognitive flexibility. Taken together, these results suggest that the FIST is a valid measure of cognitive flexibility, which relies on computations within a broad corticosubcortical network driven by inferior frontal junction engagement.

INTRODUCTION

Cognitive flexibility, or the readiness with which one can switch between mental processes to appropriately respond to environmental stimuli, supports the transition to adulthood and is critical for maintaining a job, navigating social relationships, and independent living (Burt & Paysnick, 2012; Kapp, Gantman, & Laugeson, 2011; Bailey, 2007). Greater cognitive flexibility is associated with resilience to negative life events and stress (Genet & Siemer, 2011) and higher levels of creativity (Chen et al., 2014). Despite the advantages of intact cognitive flexibility, rigorous examination of this construct has been elusive. Previous neuroimaging studies of cognitive flexibility have used a wide array of tasks (Armbruster, Ueltzhöffer, Basten, & Fiebach, 2012; Leber, Turk-Browne, & Chun, 2008; Badre & Wagner, 2006; Konishi et al., 1998), leading to somewhat inconsistent results.

Of paramount importance in accurately identifying the neural correlates of cognitive flexibility is using a task that has been carefully adapted for the fMRI environment. In the laboratory, cognitive flexibility is typically measured with set-shifting or task-switching behavioral paradigms (Dajani & Uddin, 2015). Prior task-based neuroimaging studies of cognitive flexibility have used both explicit (e.g., stimulus–response reversal paradigms [Cubillo et al., 2010], dimensional change card sort [Zelazo, 2006]) and inductive set-shifting tasks (e.g., modified Wisconsin Card Sort Tasks [WCST]; Konishi et al., 2002). Systematic reviews and meta-analyses that combine explicit and inductive tasks suggest that cognitive flexibility is instantiated in the brain via the interplay of many individual brain regions within two large-scale brain networks: the frontoparietal network (FPN), composed of the inferior frontal junction (IFJ), inferior frontal gyrus (IFG), dorsolateral pFC (dlPFC), and posterior parietal cortex (PPC), and the salience network (SN), composed of anterior insula (AI) and dorsal ACC (dACC; Dajani & Uddin, 2015; Kim, Cilles, Johnson, & Gold, 2012). A growing literature demonstrates that increased integration between the FPN and SN supports adaptive task control (Shine et al., 2016; Braun et al., 2015; Cohen, Gallen, Jacobs, Lee, & D'Esposito, 2014; Cole et al., 2013).

Neurocomputational mechanisms underlying cognitive flexibility are beginning to be explored. A physiologically plausible neural network model was able to reproduce participant behavior during cue-based task switching, as well as predict individual hemodynamic RT courses in the FPN (Ueltzhöffer, Armbruster-Genç, & Fiebach, 2015). In other empirical and computational work, reduced metastability in large-scale neural dynamics has been associated with reduced cognitive flexibility (Hellyer, Scott, Shanahan, Sharp, & Leech, 2015). Here, we aimed to build on this previous body of work by characterizing directed functional connectivity among neural systems during an inductive set-shifting task to distinguish circuitry supporting cognitive flexibility.

Past studies using inductive cognitive flexibility tasks introduce a potential confound: reasoning associated with identifying that the “rule” has switched. For tasks like the WCST or intradimensional/extradimensional shifting tasks, rule switches are signaled by experimental feedback; thus, the ability to identify a rule switch depends on how amenable participants are to the feedback. Because these tasks do not explicitly signal that the sorting principle or task has changed, errors might reflect difficulty in identifying a rule switch rather than cognitive flexibility per se. This is of particular concern when generalizing to clinical or developmental populations, who often exhibit difficulty learning from experimenter feedback (Prentice, Gold, & Buchanan, 2008).

Properties of tasks newly adapted for fMRI have rarely been evaluated, and most previous work has relied on explicit tasks. In this study, we overcome these limitations by adapting a psychometrically validated laboratory test of cognitive flexibility called the Flexible Item Selection Task (FIST) to the fMRI environment for the first time. The FIST is a set-shifting task requiring participants to repeatedly update the dimension by which they choose two sets of stimuli that “go together in one way” (Jacques & Zelazo, 2001). Adapted from the Visual–Verbal Test first developed for assessing adults (Feldman & Drasgow, 1951), this paradigm capitalizes on advantages of explicit tasks in instructing participants to consistently respond based on one stimulus type for a single selection and inductive tasks by allowing participants to choose which sorting principle with which to complete the task. The computer-based version of the FIST has undergone rigorous psychometric evaluation, demonstrating high internal consistency (RI = .91) and convergent validity with the WCST. The task has been used in both developmental and adult research settings, and participants of all ages find the task more challenging as more complex versions are introduced (Dick, 2014). Thus, our rationale was that, once we had created an fMRI-adapted version of the FIST and tested it in adults, the task could be further adapted and utilized for assessing cognitive flexibility in typically and atypically developing populations. The first aim of this study was to create a version of the FIST that could be administered in the fMRI environment, with the goal of achieving a more sophisticated understanding of the brain regions important for cognitive flexibility in neurotypical adults.

We expected to observe higher activation in the IFJ, IFG, dlPFC, AI, dACC, frontal pole, PPC, striatum, and thalamus during trials requiring cognitive flexibility (Kim et al., 2012). The reporting of group-averaged activation maps is conventional in cognitive neuroscience, but this approach is limited in that group results may be driven by subsets of participants within a sample or may not represent any one individual's data (Miller & Van Horn, 2007). Therefore, our second aim was to use an individual-level directed connectivity modeling approach to characterize common and unique aspects of brain network topology important for cognitive flexibility. Lastly, meta-analytic evidence demonstrates that brain activity levels decrease with practice, likely reflecting a reduced reliance on cognitive control that scaffolds early task learning (Chein & Schneider, 2005). Separate analyses of earlier and later completed runs of the task were also conducted to further explore this phenomenon.

METHODS

Participants

Participants were 32 adults aged 19–46 years (Mage = 25.29 years, SD = 6.42 years, 17 men) recruited from the University of Miami in Coral Gables, Florida, and the wider Miami community. Eleven participants were Hispanic/Latino, and 15 were not Hispanic/Latino (missing data for n = 6); 16 participants reported their race as “White,” and 4 reported their race as “other” (missing data for n = 12). All participants were right-handed as determined by self-report, with no reported history of psychological disorders. Informed consent was obtained for all participants, and they received $50 in compensation for their participation. The University of Miami institutional review board approved the study.

Flexible Item Selection Task

The FIST used in this study is an adapted version of the 4-Match FIST developed by (Dick, 2014). For each trial, participants are presented with four cards oriented vertically on which various stimuli are presented. Each of the cards contained images that varied along four dimensions: color (blue, green, red), shape (boat, flower, rabbit), size (large, 1.82 in.; medium, 0.83 in.; small, 0.38 in.), and number of images (one, two, three). These stimuli were created in Microsoft PowerPoint 2010 and were presented on white cards with a light gray background. For a full description of the images used and selection of dimensions, see Dick (2014).

Participants were trained to choose a pair of cards that were “the same in one way” (i.e., a selection) and were asked to make three selections per trial. Participants used a four-button rectangular button box (Figure 1) to select cards based on their location on the monitor using their right hand. Although there were four possible matches, we only required that participants provide three matches to reduce task difficulty and to avoid an excessively long trial duration. Two versions of the 4-Match FIST were administered to participants: (1) a self-paced computer-based task modeled after Dick (2014) and (2) an fMRI-adapted task. The computer-based task was designed to allow for behavioral metrics to be computed in a way that is comparable to the original study. Having participants complete both the computer-based and fMRI-adapted tasks made it possible for us to examine behavioral responses outside the scanner environment to determine the extent to which adaptations necessary for moving the task into the fMRI environment affected performance. Tasks were implemented in E-Prime 2.0 for presentation and recording of behavioral responses.

Figure 1. 

Top: A schematic of the fMRI-adapted 4-Match FIST. Participants are asked to choose three successive pairs of cards that “go together in one way” (“Now you choose”). During Control trials, the correct card pairs were highlighted by a thick black border, eliminating the need to enable cognitive flexibility, while still controlling for lower level visual and motor processes (“Follow along”). Fixation trials had jittered presentation times optimized for a fast event-related design. Bottom: Examples of “Flexibility” and novel “Control” trials for fMRI-adapted FIST.

Figure 1. 

Top: A schematic of the fMRI-adapted 4-Match FIST. Participants are asked to choose three successive pairs of cards that “go together in one way” (“Now you choose”). During Control trials, the correct card pairs were highlighted by a thick black border, eliminating the need to enable cognitive flexibility, while still controlling for lower level visual and motor processes (“Follow along”). Fixation trials had jittered presentation times optimized for a fast event-related design. Bottom: Examples of “Flexibility” and novel “Control” trials for fMRI-adapted FIST.

Computer-based Task

We sought to conceptually replicate the administration of the 4-Match FIST developed by Dick (2014). We used identical trials to those used in that study but switched the order of two successive trials to maximize differences in the correct button presses between consecutive trials. Additionally, our computer-based task differed from the earlier 4-Match in that we only required three selections per trial instead of four. For each trial, participants were prompted to “choose two cards that are the same in one way” (Jacques & Zelazo, 2001). Following each of the three selections per trial, participants were asked to verbally respond to the question, “how are [the cards] the same?” and an experimenter recorded this response by pressing a corresponding letter on a keyboard. Participants were free to choose any three combinations of cards and therefore were free to choose the dimension by which the cards matched. These trials were self-paced, and accuracy depended both on correct card selections (e.g., Cards 1 and 2) and the respective dimension by which they match (a verbal response such as “they are both blue”). In general, selections could be incorrect because of (1) incorrect card pair selection (i.e., the cards do not match along any dimension), (2) choosing the same card pair selection more than once per trial, or (3) choosing a correct pair of cards but specifying an incorrect dimension. For a trial to be considered correct, all three selections must have been correct. E-Prime (Psychology Software Tools) data were processed using an in-house script that was developed in MATLAB (The MathWorks) to calculate selection-level and trial-level accuracy. Computer-based task trial-level accuracy was used for behavioral analyses.

fMRI-adapted Task

To adapt the computer-based 4-Match FIST to the fMRI environment, several changes were applied (Figure 1). First, for trials indexing cognitive flexibility (“Flexibility trials”), participants were instructed to make their three selections one after the other, without indicating the dimension by which they chose their selections. Instead, participants were asked to simply think about why the cards are the same. In addition, participants were instructed to choose each pair of cards as quickly as possible, as these trials were of fixed duration (8 sec, based on pilot data from four adults). These changes were implemented to keep task length to a minimum while still maintaining an event-related design. Flexibility trials were indicated by a heading at the top of the screen: “Now you choose.” Twenty unique Flexibility trials were created for the fMRI-adapted FIST; all 20 trials for the fMRI task and the six trials for the computer-based task contained unique combinations of color, shape, size, and number of stimuli. Participants used a four-button response device to provide responses, which was identical to the response device used for the training and computer-based tasks.

Novel control trials were created for the fMRI task. These visually resembled the Flexibility trials exactly, but participants were provided the correct responses, indicated by a thick black border surrounding the pair of correct cards. Participants were instructed to “follow along” and simply press the buttons corresponding to the cards with the thick black border. Each correct selection, indicated by two cards with a thick border, appeared for 2.6 sec. Each trial, consisting of three selections, totaled 8 sec to match the length of Flexibility trials. Because we set each selection duration to 2.6 sec, RT for Control trials was not used in any analyses of this study. For each Flexibility trial, there was a respective Control trial that contained the same stimuli. This allowed us to isolate the engagement of cognitive flexibility from the processing of visual aspects of the stimuli and motor responses necessary for completing the Flexibility trials. To create each Control trial, three selections were systematically chosen of the four possible answers, leaving one “excluded dimension” for each Control trial. Each dimension was excluded about the same number of times across Control trials used for task training and the fMRI task (six to eight exclusions per dimension). To avoid participants learning the correct answers for Flexibility trials by observing the associated Control trial, associated Control trials corresponding to a Flexibility trial (with the same four cards) were not used in the same run. For both Flexibility and Control trials, the combination of two button presses for incorrect answers (six total combinations) were distributed evenly (e.g., Buttons 1 and 2 indicated an incorrect response 15% of the time).

Each participant completed four runs of the fMRI task, with each run consisting of 10 Flexibility and 10 Control trials. Runs 1 and 3 and Runs 2 and 4 contained the same trials but in a randomized order. This design naturally allowed for inferences into changes in directed functional network connectivity across time by comparing results for Runs 1 and 2 with results from Runs 3 and 4. Optseq2 was used to determine the jitter for the length of fixation trials, which ranged from 0 to 12 sec (https://surfer.nmr.mgh.harvard.edu/optseq/). Each run was 4 min in length.

Accuracy was calculated for Flexibility and Control trials based on recorded button presses; RT was also recorded for Flexibility trials. For a Flexibility trial to be accurate, a participant had to make six correct button presses (corresponding to three consecutive selections of pairs of cards) within the 8-sec interval. The button presses were divided into three consecutive pairs, and each pair was assumed to correspond to one selection. If the participant did not make six button presses, the trial was automatically scored as incorrect. If a participant selected a pair of cards that did not match along any of the four dimensions, the trial was incorrect. Control trial accuracy was calculated at the selection level. For a Control trial to be accurate, all three selections must have been accurate.

For each run of the fMRI-adapted task, a combined accuracy and RT metric was calculated to provide an index of task efficiency for participants who maintained a high level of accuracy (>80%). This metric was computed similar to that of the Dimensional Change Card Sort task NIH Toolbox measure, a measure of cognitive flexibility (Zelazo et al., 2014). As adults and older children generally slow down (increase RT) to maintain a high level of accuracy, individual differences in cognitive flexibility manifest as degrees of slowing rather than as errors. In contrast, children below approximately 6 years of age do not tend to exhibit speed/accuracy tradeoffs but instead respond quickly at the expense of accuracy (Zelazo et al., 2013). The combined accuracy and RT metric thus permits comparisons across different age groups, which is a planned future application for the FIST presented here. First, mean accuracy for Flexibility trials in a single run was transformed into a 5-point scaled metric by dividing mean accuracy by 2 (i.e., a mean accuracy of 9 of 10 trials translated into a score of 4.5). Next, RTs for accurate trials were transformed to a 5-point scale. Across all runs and participants for correct trials only, median RTs ranged from 3275 to 7427.5 msec. Median RTs were normally distributed (skew: −0.03 to 0.54, kurtosis: −0.79 to 0.03); therefore, no transformation was applied. The minimum RT value (3275 msec) was subtracted from original RT values, such that new values ranged between 0 and 4152.5 msec. Then, median RTs were algebraically transformed to a 5-point scale and reverse-scored according to Equation (1), where 4152.5 represents the sample-specific range of median RT values in msec:
5medianRT×54152.5
(1)
Based on the above equation, higher scores indicate faster RTs. Finally, for each run where accuracy reached at least 80%, rescaled accuracy and RTs were summed to constitute a singular variable ranging from 0 to 10, with higher scores indicating greater efficiency in cognitive flexibility. For participants who did not maintain a high level of accuracy, task efficiency scores were equivalent to their accuracy scores. By incorporating RT information for only runs with a high level of accuracy, we ensured that higher task efficiency scores reflected performance that was both accurate and fast. This efficiency metric was used as the dependent variable of interest in this study because it incorporated two complementary, but unique, pieces of information about task performance and because it provided more variability across the sample than accuracy or RT alone.

Experimental Procedure

All participants were trained on the FIST by an experimenter at a desktop computer before completing the computer-based and fMRI tasks. All demonstration and practice trials were unique from trials of the computer-based and fMRI task. To introduce participants to the FIST, they were first shown two trials of the 2-Match FIST, an easier version of the 4-Match used in this study. Next, an experimenter demonstrated two trials of the 4-Match while providing a scripted verbal explanation of their selections to ensure the participant understood how to respond to the 4-Match version. During these demonstration trials, experimenters presented all four dimensions as possible answers. Following the demonstration, participants completed two practice 4-Match trials on their own and were given feedback if they answered incorrectly. Then, participants completed the computer-based task.

After the computer-based task administration, participants were shown examples of Control and Flexibility trials to prepare for the fMRI-adapted version. An experimenter first explained how to respond to a single self-paced Control trial, which was followed by two timed practice control trials. Participants were introduced to the Flexibility trials of the fMRI task by completing a shortened version of the fMRI task, with both Flexibility and Control trials, at a computer. Finally, participants completed the same practice fMRI task in the mock scanner. Before administering the first run of the fMRI task in the scanner, the majority of participants completed a “refresher” FIST task that consisted of three practice trials to remind participants of task instructions and ensure acceptable accuracy levels for all runs of the fMRI task (n = 24, 75% of sample received the refresher).

MRI Data Acquisition

Task fMRI data were acquired for participants on a 3T GE Discovery 750 series scanner using an EPI sequence and a 32-channel head coil (repetition time [TR] = 2 sec, echo time = 30 sec, flip angle = 75°, 3.4 mm slices, voxel size = 3.4 isotropic mm). The first five volumes were immediately discarded to account for magnet stabilization, resulting in 122 volumes per run (for each of four runs). High-resolution T1-weighted FSPGR BRAVO scans were also acquired to facilitate registration of the functional image to standard space (inversion time = 650 msec, flip angle = 12°, field of view = 25.6 cm, 1 mm isotropic voxels). Additional structural and functional images were acquired but were not analyzed for this study.

Preprocessing

Raw functional and structural images were visually inspected for quality before preprocessing using a standardized in-house coding scheme. Preprocessing was conducted in FSL 5.0.9. First, structural images were brain extracted using FSL's BET tool. Using FEAT, fMRI data underwent rigid body motion correction with MCFLIRT, slice time correction, smoothing with a 6-mm kernel, high-pass filtering (100 sec), coregistration to the structural image and normalization to the 2 mm MNI template. Data were quality checked following structural brain extraction and normalization steps to ensure fidelity of preprocessing steps. Participants exhibited minimal motion during fMRI task data acquisition: across all runs, the average motion in the six rigid directions did not exceed 0.94 mm or degrees.

ANALYTIC PLAN

Task Validation

All descriptive statistics were computed using R 3.4.2 (R Core Team, 2017). Task experiment files, accuracy calculation scripts, and code for all R-based analyses are publicly available (https://github.com/xDinaDajani/fMRI_FIST_adult.git).

Reliability

Internal consistency was computed for each run of the fMRI task based on trial-level accuracy using Kuder–Richardson 20 formula (KR-20), which is suitable for dichotomous data (Kuder & Richardson, 1937). KR-20 was estimated using the DescTools R package (Signorell, 2017). We used established guidelines to interpret acceptable levels of internal consistency (>.70; Cicchetti & Sparrow, 1990).

Test–retest reliability was computed across all runs of the fMRI-adapted FIST, measured by the intraclass coefficient (ICC), using the run-level efficiency metric as the dependent variable. To compute the ICC, a two-way random effects model was used, that is, ICC(2, 4), where 4 represents the number of time points the variable of interest will be averaged across (Shrout & Fleiss, 1979), which is best suited for test–retest applications (Sainani, 2017). We report both the ICC(2, 4) formula and the ICC(2, 1) formula, where the former represents performance averaged across all four runs and the latter considers reliability for a single run. The ICC was calculated using the R package psych (Revelle, 2017). Established guidelines were used to interpret ICC values: poor (<.40), fair (.40–.59), good (.60–.74), and excellent (.75–1.00; Cicchetti, 1994). To determine the minimum number of runs (m) needed to obtain a reliable mean estimate of task performance, the following formula was used based on Shrout and Fleiss (1979), where ρ* is the minimum acceptable reliability coefficient (here, .75) and ρL is the lower bound of the 95% confidence interval for the ICC(2, 1) reliability estimate equation (Equation 2):
m=ρ*1ρLρL1ρ*
(2)

Validity

Convergent validity can be confirmed in the measure of interest by its coherence with independent measurements of similar constructs. The computer-based task used here closely resembled the psychometrically validated 4-Match FIST (Dick, 2014). Therefore, to test the convergent validity of our newly adapted fMRI version of the 4-Match FIST with the computer-based task, we correlated trial-level accuracy averaged across Runs 1–4 (from Flexibility trials only) with the computer-based trial-level accuracy.

Task-based fMRI Data Analyses

General Linear Model

Flexibility and Control trials (and their temporal derivatives) were modeled at the run level for each participant using a gamma hemodynamic response function. The six rigid motion estimates were included as nuisance regressors at the run level. Runs were combined within participants using a fixed effects analysis. Finally, data were combined across participants with a mixed-effects design using FLAME 1 to identify brain regions activated for two contrasts of interest: Flexibility–Control and Control–Flexibility trials. Main effects for Flexibility and Control trials were also modeled relative to implicit rest (i.e., fixation trials). Significant voxels were identified using a voxel-level threshold at a family-wise error (FWE)-corrected p < .05, which adequately controls the false-positive rate (Eklund, Nichols, & Knutsson, 2016).

Task-modulated Network Connectivity

The extended unified structural equation modeling (euSEM) approach (Gates, Molenaar, Hillary, & Slobounov, 2011) was used to estimate ROI to ROI directed network connectivity that is modulated by the Flexibility and Control conditions by applying a model search procedure called group iterative multiple model estimation (GIMME; Gates, Lane, Varangis, Giovanello, & Guskiewicz, 2017). GIMME is an iterative search algorithm that identifies the most parsimonious, best-fitting SEM to describe an individual's connectome within a user-specified limited number of ROIs. euSEM estimates direct and modulating external experimental influences on BOLD activity derived from event-related fMRI data. To account for individual differences in the hemodynamic response function, the parameters governing the generation of the Flexibility and Control vectors within GIMME are arrived at using the smooth finite impulse response (Lindquist, Meng Loh, Atlas, & Wager, 2009). Direct task effects can be interpreted as increases or decreases in activity in a particular ROI due to experimental manipulation, much like in the traditional activation-based GLM analyses. The difference here is that the relation only emerges if it is significant after covarying for other potential influences on the ROI, such as autoregressive effects (i.e., the extent to which the next TR activity can be predicted by the previous TR) and connectivity with other ROIs. Modulating effects model the influence that experimental manipulation has on relationships between regions. This method is similar to dynamic causal modeling (DCM; Friston, Harrison, & Penny, 2003), but DCM is limited in terms of the number of brain regions that can be used and the validity of its results depends on having strong prior understandings of the relationships among brain regions expected to change because of task manipulations. Because of the limited knowledge we have about functional and directed connectivity important for cognitive flexibility, the exploratory approach offered by euSEM confers an advantage over confirmatory approaches such as DCM.

euSEM considers both contemporaneous and lagged (t − 1) temporal effects, in addition to the experimental manipulation and modulating effect of the task on connectivity at t. For an euSEM conducted with two task conditions (u1 and u2) and a lag order of 1, we have Equation (3), adapted from Gates et al. (2011):
ηt=Aηt+ϕηt1+γ1u1,t+γ2u2,t+τ1ηtu1,t+τ2ηtu2,t+ζt
(3)
Here, A is a p × p matrix of the contemporaneous relations among the p ROIs, ηt is a p × 1 vector of ROI activity values at t, ϕ is a p × p matrix containing estimates of the lagged relations, uq,t is the value of the smooth finite impulse response convolved task onset vector q at t, γq is the estimate of the experimental effects on brain region activity, τq is a p × p matrix containing the estimates of the modulating effects of the task on the relations between brain regions, and ζt is a vector containing the residuals from the prediction of each brain regions activity at t.

The benefit of employing euSEM within the GIMME framework is that this method capitalizes on the information shared across participants to arrive at a group model that describes information for the majority of individuals. Importantly, the method used to arrive at the group-level model does not assume homogeneity but does find aspects (should they exist) that are similar across the sample. Furthermore, GIMME searches for subgroups of participants within the sample who share network structure using the Walktrap hierarchical clustering algorithm. Finally, GIMME estimates individual-level models by adding any additional paths to best explain that individual's model, using the group- and subgroup-level paths as priors.

An automatic search procedure is used to identify the optimal path model among prespecified ROIs. The number of ROIs that can be specified using the GIMME framework is limited by the power afforded by the length of the time series and the application of a search procedure, which may lead to unreliable results given too many ROIs relative to the length of the time series. We used previous literature to choose a subset of activated regions derived from this study's GLM analyses. We ensured inclusion of the following regions, which were activated (or deactivated) during the FIST and have been shown in previous task-based connectivity studies to be involved in executive function-modulated connectivity: IFJ (Dixon et al., 2018; Harding, Yücel, Harrison, Pantelis, & Breakspear, 2015), dlPFC (Harding et al., 2015), pre-SMA/dACC (Harding et al., 2015; Cohen et al., 2014), AI (Zhang, Zhang, Yao, & Zhao, 2015; Cohen et al., 2014), inferior parietal lobule (IPL; Harding et al., 2015; Zhang et al., 2015; Cohen et al., 2014), angular gyrus (AG; Dixon et al., 2018; Zhang et al., 2015), and visual cortex (Harding et al., 2015; Waskom, Kumaran, Gordon, Rissman, & Wagner, 2014; van Schouwenburg, den Ouden, & Cools, 2010). ROI coordinates were identified at the cluster peak Z value derived from one of three contrasts from the GLM analyses: (1) [Control and Flexibility] > Fixation (visual cortex) (Harding et al., 2015), (2) Control–Flexibility (AG), and (3) Flexibility–Control (all other ROIs). We limited analyses to the left hemisphere to reduce the number of ROIs, similar to previous studies (Harding et al., 2015). We also sought to include the region with the highest Z value for the Flexibility–Control contrast, which was centered in the cerebellum. This led to the inclusion of eight ROIs: medial cerebellar crus II, preSMA/dACC, left IPL (lIPL), left primary/secondary visual cortices (−8, −90, −12), left AI (lAI), left dlPFC (ldlPFC), left IFJ (lIFJ), and left AG (lAG; −56, −58, 28; see Table 2 for all other coordinates). Before time series extraction, data were preprocessed identically to the data for GLM analyses, similar to previous studies using event-related GIMME (Gates et al., 2011, 2017). Spheres of 4-mm radius were created around each of the eight coordinates (Figure 4A), and individual-level time series averaged within each ROI were extracted. These time series were fed forward to the GIMME analyses.

GIMME has been shown to be reliable with as few as 60 time points per person for the number of variables used here (Lane, Gates, Pike, Beltz, & Wright, 2019), but the euSEM has only been evaluated on time series of length 200 observations per person (Gates et al., 2011). Here, each run contained 122 time points. We combined Runs 1 and 2 by temporally concatenating each ROI's time series at the individual level, leading to a 244 × 1 time series (Timepoint 1). We also concatenated Runs 3 and 4 (Timepoint 2). Data from Timepoints 1 and 2 were analyzed in GIMME separately. The theoretical motivation for examining the first two runs separately from the last two runs comes from meta-analytic evidence demonstrating that activity decreases with practice in lateral prefrontal, dACC, posterior parietal, and cerebellar regions, likely reflecting a reduced reliance on cognitive control that scaffolds early task learning (Chein & Schneider, 2005).

The euSEM analysis was implemented using the gimme package in R (gimme Version 0.4.2; R Version 3.3.1; Gates et al., 2017; Lane & Gates, 2017; R Core Team, 2017). Task regressors for Flexibility and Control trials were identified as exogenous variables in the GIMME analysis. Two task modulatory effects were specified (lIFJ × Flexibility and lIFJ × Control); these interaction terms were calculated by multiplying the standardized ROI's time course with the event regressor. Fixation trials were not modeled and therefore represented an implicit baseline, serving as the reference category. All variables were standardized before model fitting.

The individual-level model search procedure favors model parsimony and stops after meeting criteria for excellent model fit for two of four fit indices: comparative fit index (CFI ≥ .95), non-normed fit index (NNFI ≥ .95), root-mean-square error of approximation (RMSEA ≤ .05), and the standardized root-mean-square residual (SRMR ≤ .05; Brown, 2006). These criteria were also used to identify good individual-level model fit in the current study. Individuals' models who did not meet criteria for good fit were not evaluated further, because it is uncertain if the final models represent these individuals well. We report the summary statistics of beta estimates for significant paths (p < .05).

Cluster Validation

To determine the validity of the cluster solution from the GIMME subgrouping step, stability and validity of the cluster solution was evaluated using the R package perturbR (Gates et al., 2019; Gates, Fisher, & Arizmendi, 2018). This algorithm incrementally introduces noise to the graph being clustered while maintaining the original graph's overall properties and compares resulting cluster solutions with the solution for the original graph (i.e., using the full sample). A stable solution will not change drastically given small changes to the network. Quantitatively, a cluster solution is said to be stable if the graph had 20% or more of its edges perturbed before the cluster solution for the rewired graph is as different as when 20% of the nodes are randomly placed into different clusters. This is quantified by two distinct, but complementary, metrics that describe the degree to which two community solutions differ: Hubert–Arabie Adjusted Rand Index and Variation of Information. To ensure the subgrouping solution is not simply capitalizing on chance where no true subgroups exist, a relative measure of cluster solution quality (i.e., modularity) was used to compare the original cluster solution's quality with a solution obtained from a random graph that contains no clusters. The cluster solution was considered valid if modularity for the original solution is greater than or equal to the 95th percentile of modularity obtained from random graphs.

Brain–Behavior Relationships

Adaptive LASSO, a feature selection tool, was used to identify variables generated from GIMME that were related to task efficiency on the FIST. LASSO uses an L1-regularized regression to identify the most relevant predictors in a model. Adaptive LASSO builds on the original LASSO model by including a coefficient-specific vector of adaptive weights, which contain unique regularization penalties for each coefficient (Zou, 2006). The adaptive weights were determined by an initial ridge regression with 10-fold cross-validation and defined as the inverse of the absolute values of the best ridge coefficients (i.e., the coefficients for the mean square error-minimizing lambda). This ridge regression was conducted to create a vector of the adaptive weights to be used for the adaptive LASSO algorithm. Adaptive LASSO was fit with 10-fold cross-validation. Selected coefficients and their beta estimates are reported at the mean square error-minimizing lambda. Total variance explained (R2) is reported for the model including all participants (n = 28).

Separate adaptive LASSO models were fit for Timepoints 1 and 2. Dependent variables were FIST efficiency averaged across Runs 1 and 2 (for Timepoint 1) and Runs 3 and 4 (for Timepoint 2). Beta estimates for contemporaneous paths generated from the GIMME analysis were used as independent variables, including intrinsic paths, main effects of Flexibility and Control trials, and task-modulated connectivity paths. Past work shows that more parsimonious network models relate to better working memory performance (Nichols, Gates, Molenaar, & Wilson, 2014). Therefore, an index of network model complexity, total number of contemporaneous paths, was also included as an independent variable. The total paths variable was calculated as an individual's sum of the number of contemporaneous intrinsic paths, main effects of Flexibility and Control trials, and task-modulated connectivity paths. Many individual-level paths were excluded from this analysis because they were zero-inflated due to being estimated for a minority of participants, and linear models are not optimized for modeling zero-inflated variables. Zero-inflated variables were identified quantitatively as any variable with a nonnormal distribution (i.e., absolute value of skew or kurtosis greater than or equal to 3). In all, this led to 30 independent variables included for Timepoint 1, with n = 28 participants. For Timepoint 2, 26 independent variables were included, with n = 28 participants. Adaptive LASSO was implemented with the R package glmnet (Friedman, Hastie, & Tibshirani, 2010; Version 2.0.16). We report the unstandardized beta coefficients for variables selected by the adaptive LASSO model.

RESULTS

Task Validation

We used an adapted version of the 4-Match FIST developed by Dick (2014), which is designed to challenge abstraction and cognitive flexibility skills (Figure 1). Several changes were applied to the computer-based FIST to create the fMRI-adapted FIST. For the fMRI version, we required timed responses (8 sec per trial), removed the requirement to identify the dimension by which to choose pairs of cards that match, included novel Control trials, and introduced a randomized jitter to the presentation and length of fixation trials.

Reliability

Participants completed four runs of the FIST in the scanner. Internal consistency for Runs 1 through 4, respectively, were 0.68, 0.81, 0.66, and 0.64. When reliability was assessed for Runs 1 and 2 combined, reliability reached .85 (95% CI [.84, .87]); reliability for Runs 3 and 4 was .79 (95% CI [.76, .81]). These values indicate good reliability. An efficiency metric combining accuracy and RT was computed to index behavioral performance (see Methods section). Task efficiency averaged across runs exhibited excellent test–retest reliability as indexed by ICC (ICC = .84, 95% CI [.69, .92]). Individual runs still demonstrated fair test–retest reliability (ICC = .56, 95% CI [.36, .74]). But, to achieve excellent reliability for task efficiency (i.e., ICC ≥ .75), it is necessary to obtain a mean score across at least six runs (m = 5.33).

Validity

Convergent validity of the fMRI-adapted FIST was assessed by its correlation with a previously validated measure of cognitive flexibility, the computer-based FIST originally developed by Dick (2014). Accuracy on Flexibility trials in the fMRI-adapted FIST was positively correlated with the computer-based task accuracy, r(30) = .36, p = .046 (Figure 2), suggesting that the fMRI-adapted version displays convergent validity with the computer-based task despite the modifications made.

Figure 2. 

Top row: Frequency of dimension choice. Box plots based on data from the computer-based FIST are displayed showing: (left) the number of trials that included a selection of a particular dimension type and (right) number of trials where a particular dimension was chosen for each selection. Color was most frequently chosen on the first selection. Number was least frequently chosen for any given trial and, if chosen, was most frequently chosen on the third selection. Middle row: Computer-fMRI task convergent validity (left). The x-axis represents trial-level computer-based 4-Match FIST accuracy, and the y-axis represents the fMRI-adapted 4-Match FIST trial-level accuracy averaged across all four runs. There was a significant correlation between the computer-based and fMRI-adapted tasks, r(30) = .36, p = .046. Task efficiency scores for each run of the fMRI-adapted FIST (right). Scores changed across runs in a cubic fashion, F(1, 93) = 10.61, p = .002, demonstrating an improvement in performance over time. Red diamonds indicate mean scores for each run. Bottom row: Differences in accuracy between flexibility and control trials.

Figure 2. 

Top row: Frequency of dimension choice. Box plots based on data from the computer-based FIST are displayed showing: (left) the number of trials that included a selection of a particular dimension type and (right) number of trials where a particular dimension was chosen for each selection. Color was most frequently chosen on the first selection. Number was least frequently chosen for any given trial and, if chosen, was most frequently chosen on the third selection. Middle row: Computer-fMRI task convergent validity (left). The x-axis represents trial-level computer-based 4-Match FIST accuracy, and the y-axis represents the fMRI-adapted 4-Match FIST trial-level accuracy averaged across all four runs. There was a significant correlation between the computer-based and fMRI-adapted tasks, r(30) = .36, p = .046. Task efficiency scores for each run of the fMRI-adapted FIST (right). Scores changed across runs in a cubic fashion, F(1, 93) = 10.61, p = .002, demonstrating an improvement in performance over time. Red diamonds indicate mean scores for each run. Bottom row: Differences in accuracy between flexibility and control trials.

Behavioral Data

Computer-based Task

Participants exhibited high performance on the computer-based task (trial-level accuracy [proportion of correct trials]: M = 0.88, SD = 0.17). A repeated-measures ANOVA suggested that participants performed equally well on all selections within a trial, F(2, 62) = 1.67, p = .20, although there was nonsignificant trend showing a decrease in performance across selections, F(1, 62) = 3.27, p = .076 (Selection 1: M = 0.97, SD = 0.08; Selection 2: M = 0.95, SD = 0.15; Selection 3: M = 0.93, SD = 0.09). When participants committed errors, they were most likely due to misidentifying the dimension by which cards match (52% of errors). Participants also identified cards that did not match along any dimension (31% of errors) and repeated their card choice (17% of errors).

Participants identified card pairs by “color” on nearly every trial (proportion of trials: M = .97, SD = .06), with “shape” being identified often as well (M = .83, SD = .31). The “size” dimension was identified less frequently (M = .68, SD = .38), and the “number” dimension was identified on the fewest proportion of trials (M = .52, SD = .38). Across participants, “color” was most frequently chosen on the first selection, whereas “size” and “number” were more frequently chosen on the third selections. The majority of participants tended to vary the order in which they used a particular dimension to choose card pairs. Most participants (56%) repeated a particular pattern for two of the six trials (e.g., repeating the following pattern: “color” for Selection 1, “size” for Selection 2, and “shape” for Selection 3); fewer participants never repeated a particular selection pattern (9%); even fewer repeated a pattern for four of the six trials (6%; Figure 2).

fMRI-adapted Task

Across all runs, participants exhibited high performance on both Flexibility and Control trials (accuracy: Mflex = 0.82, SDflex = 0.20, Mcontrol = 0.90, SDcontrol = 0.16; Table 1). A two-way repeated-measures ANOVA demonstrated that for all runs, accuracy was higher for Control trials compared with Flexibility trials, F(1, 31) = 7.43, p = .01. Furthermore, there was a significant cubic trend in task efficiency across runs, demonstrating an improvement in performance over time, F(1, 93) = 10.61, p = .002 (Figure 2). Post hoc contrasts, adjusted using Tukey's honestly significant difference test, revealed that task efficiency was lowest for Run 1 compared with all other runs (ps < .001), similar between Runs 2 and 3 (p = .91), and highest for Run 4 compared with all other runs (ps < .03).

Table 1. 
Behavioral Data from fMRI-adapted FIST Task
 MinMaxMeanSD
Run 1 
 Flexibility accuracy 0.00 1.00 0.72 0.23 
 Control accuracy 0.10 1.00 0.76 0.23 
 Flexibility median RT (msec) 4295.00 7427.50 5743.56 837.04 
 Flexibility accuracy-RT 0.00 8.27 5.57 1.94 
  
Run 2 
 Flexibility accuracy 0.00 1.00 0.84 0.22 
 Control accuracy 0.80 1.00 0.95 0.07 
 Flexibility median RT (msec) 3884.00 7095.00 5317.27 729.07 
 Flexibility accuracy-RT 0.00 9.27 6.66 1.92 
  
Run 3 
 Flexibility accuracy 0.20 1.00 0.83 0.18 
 Control accuracy 0.70 1.00 0.94 0.09 
 Flexibility median RT (msec) 4003.00 7035.50 5440.64 718.09 
 Flexibility accuracy-RT 1.47 8.93 6.53 1.59 
  
Run 4 
 Flexibility accuracy 0.40 1.00 0.88 0.16 
 Control accuracy 0.60 1.00 0.95 0.09 
 Flexibility median RT (msec) 3275.00 6556.50 5067.88 788.65 
 Flexibility accuracy-RT 4.19 9.50 7.22 1.36 
  
All runs 
 Flexibility accuracy 0.00 1.00 0.82 0.20 
 Control accuracy 0.10 1.00 0.90 0.16 
 Flexibility median RT (msec) 3275.00 7427.50 5390.15 798.37 
 Flexibility accuracy-RT 0.00 9.50 6.50 1.80 
 MinMaxMeanSD
Run 1 
 Flexibility accuracy 0.00 1.00 0.72 0.23 
 Control accuracy 0.10 1.00 0.76 0.23 
 Flexibility median RT (msec) 4295.00 7427.50 5743.56 837.04 
 Flexibility accuracy-RT 0.00 8.27 5.57 1.94 
  
Run 2 
 Flexibility accuracy 0.00 1.00 0.84 0.22 
 Control accuracy 0.80 1.00 0.95 0.07 
 Flexibility median RT (msec) 3884.00 7095.00 5317.27 729.07 
 Flexibility accuracy-RT 0.00 9.27 6.66 1.92 
  
Run 3 
 Flexibility accuracy 0.20 1.00 0.83 0.18 
 Control accuracy 0.70 1.00 0.94 0.09 
 Flexibility median RT (msec) 4003.00 7035.50 5440.64 718.09 
 Flexibility accuracy-RT 1.47 8.93 6.53 1.59 
  
Run 4 
 Flexibility accuracy 0.40 1.00 0.88 0.16 
 Control accuracy 0.60 1.00 0.95 0.09 
 Flexibility median RT (msec) 3275.00 6556.50 5067.88 788.65 
 Flexibility accuracy-RT 4.19 9.50 7.22 1.36 
  
All runs 
 Flexibility accuracy 0.00 1.00 0.82 0.20 
 Control accuracy 0.10 1.00 0.90 0.16 
 Flexibility median RT (msec) 3275.00 7427.50 5390.15 798.37 
 Flexibility accuracy-RT 0.00 9.50 6.50 1.80 

RT for Control trials are not reported because the presentation of each of the three selections were of fixed duration, limiting interpretability of Control trial-level RT.

fMRI Data

GLM

Task contrasts combined across all four runs were investigated at a voxel-level threshold of FWE-corrected p < .05. To explore whether activity levels change with practice over the trials (Chein & Schneider, 2005), separate analyses of earlier and later runs were conducted. As there was little evidence for change in activation level across runs, the results described below include data analyzed for all four runs of the task combined.

As hypothesized, during both Flexibility and Control trials, activation was observed in the lateral prefrontal cortices, PPC, AI, and dACC. Deactivation was observed in task-negative regions comprising the default mode network (DMN; e.g., AG, anterior temporal lobe, posterior cingulate (PCC), and ventromedial pFC. The Flexibility–Control contrast revealed stronger and more widespread activation for Flexibility trials relative to Control trials in canonical areas of the FPN and SN (Figure 3; Table 2). Specifically, there was stronger activation in the lIFJ extending into the left IFG, bilateral dlPFC, bilateral FEFs, bilateral AI, dACC/pre-SMA, and bilateral IPL during Flexibility versus Control trials. In addition, stronger activation during Flexibility trials was present bilaterally in the lower and higher order visual areas, posterior inferior temporal gyrus, precuneus, cerebellum, thalamus, globus pallidus, and a small portion of the head of the caudate. There were also small clusters that showed weaker deactivation for Flexibility trials compared with Control trials in the precuneus, cuneus, and lingual gyrus.

Figure 3. 

Top: Group activation maps for the Flexibility–Control and Control–Flexibility contrasts. Z maps are voxel-level thresholded at an FWE-corrected p < .05. Bottom: Group activation maps for the Flexibility–Control contrasts separated by early versus late runs. Red = Run 1 + Run 2 combined; yellow = Run 3 + Run 4 combined; orange = overlap. Z maps are voxel-level thresholded at an FWE-corrected p < .05. When directly contrasting the first and second half of the runs (Run 1 + 2 > Run 3 + 4), there were no significant differences observed for the Flexibility–Control contrast. In the opposite comparison (Run 3 + 4 > Run 1 + 2), a few isolated voxels in the right hemisphere were observed for the Flexibility–Control contrast.

Figure 3. 

Top: Group activation maps for the Flexibility–Control and Control–Flexibility contrasts. Z maps are voxel-level thresholded at an FWE-corrected p < .05. Bottom: Group activation maps for the Flexibility–Control contrasts separated by early versus late runs. Red = Run 1 + Run 2 combined; yellow = Run 3 + Run 4 combined; orange = overlap. Z maps are voxel-level thresholded at an FWE-corrected p < .05. When directly contrasting the first and second half of the runs (Run 1 + 2 > Run 3 + 4), there were no significant differences observed for the Flexibility–Control contrast. In the opposite comparison (Run 3 + 4 > Run 1 + 2), a few isolated voxels in the right hemisphere were observed for the Flexibility–Control contrast.

Table 2. 
Brain Regions with Significant Activation for the Flexibility–Control Contrast
Brain RegionCluster SizePeak ZPeak MNI
xyz
R medial cerebellar crus II 1174 8.07 −70 −30 
preSMA/dACC 601 7.85 −2 24 38 
L IPL and precuneus 2314 7.67 −36 −48 38 
L temporal occipital fusiform 1293 7.62 −34 −46 −22 
R cerebellar crus I 1341 7.49 28 −60 −36 
L AI 334 7.40 −32 18 −4 
L dlPFC 553 7.31 −50 32 26 
L FEF 249 7.29 −26 10 56 
Medial occipital cortex 912 6.97 −72 
R AI 295 6.88 32 26 −6 
R IPL 332 6.81 34 −66 40 
L thalamus, caudate and globus pallidus 44 6.71 −12 −12 
L IFJ 10 6.51 −46 10 30 
R thalamus 201 6.41 10 −8 
R globus pallidus 6.39 16 −4 −2 
R thalamus 6.34 10 −16 12 
R dlPFC 18 6.06 44 36 16 
R FEF 19 6.03 26 14 48 
R hippocampus 5.89 24 −24 −8 
L frontal pole 5.81 −48 44 
R cerebellar crus I 5.80 14 −78 −20 
R dlPFC 5.79 50 38 18 
Brain RegionCluster SizePeak ZPeak MNI
xyz
R medial cerebellar crus II 1174 8.07 −70 −30 
preSMA/dACC 601 7.85 −2 24 38 
L IPL and precuneus 2314 7.67 −36 −48 38 
L temporal occipital fusiform 1293 7.62 −34 −46 −22 
R cerebellar crus I 1341 7.49 28 −60 −36 
L AI 334 7.40 −32 18 −4 
L dlPFC 553 7.31 −50 32 26 
L FEF 249 7.29 −26 10 56 
Medial occipital cortex 912 6.97 −72 
R AI 295 6.88 32 26 −6 
R IPL 332 6.81 34 −66 40 
L thalamus, caudate and globus pallidus 44 6.71 −12 −12 
L IFJ 10 6.51 −46 10 30 
R thalamus 201 6.41 10 −8 
R globus pallidus 6.39 16 −4 −2 
R thalamus 6.34 10 −16 12 
R dlPFC 18 6.06 44 36 16 
R FEF 19 6.03 26 14 48 
R hippocampus 5.89 24 −24 −8 
L frontal pole 5.81 −48 44 
R cerebellar crus I 5.80 14 −78 −20 
R dlPFC 5.79 50 38 18 

Results are voxel-level thresholded at FWE-corrected p < .05. Clusters defined using FSL's cluster command using a Z > 5.75 threshold (except for two clusters which required a higher threshold, Z > 6.3, to break up into anatomically distinct regions: IFJ/dlPFC and R thalamus/L thalamus/L basal ganglia). Coordinates in white matter, brainstem, or outside the brain are not included in this table. Results are organized by peak Z value in descending order. R = right; L = left.

The Control–Flexibility contrast revealed primarily weaker deactivation (i.e., stronger deactivation for Flexibility trials) in regions comprising the DMN including medial prefrontal, PCC, AG, and anterior temporal regions (Figure 3; Table 3). The Control–Flexibility contrast also identified small clusters that had higher activation for Control relative to Flexibility trials in the right lateral occipital cortex, left parietal/central opercular cortex, and the right supramarginal gyrus.

Table 3. 
Brain Regions with Significant Activation for the Control–Flexibility Contrast
Brain RegionCluster SizePeak ZPeak MNI
xyz
R AG, MTG, and lateral occipital 507 8.07 66 −54 28 
L cerebellar crus II 64 6.99 −22 −78 −36 
L STG 264 6.86 −52 −14 
R MTG 678 6.84 50 −34 
L temporal pole 240 6.8 −52 12 −32 
mPFC/frontal pole 318 6.73 56 14 
L AG 219 6.58 −56 −58 28 
L parietal/central operculum 72 6.58 −42 −18 14 
ventral mPFC 195 6.35 56 −8 
L PCC 65 6.32 −8 −46 26 
R cerebellar crus II 6.18 26 −78 −36 
L posterior insula 12 6.07 −40 −4 −12 
R supramarginal gyrus 49 6.05 64 −26 22 
R hippocampus 22 6.01 24 −8 −24 
R supramarginal gyrus 17 6.01 64 −40 10 
R frontal pole 5.89 12 46 40 
R posterior insula 5.81 42 −8 
R PCC 5.79 −46 28 
Ventral ACC 5.78 −2 34 −2 
Brain RegionCluster SizePeak ZPeak MNI
xyz
R AG, MTG, and lateral occipital 507 8.07 66 −54 28 
L cerebellar crus II 64 6.99 −22 −78 −36 
L STG 264 6.86 −52 −14 
R MTG 678 6.84 50 −34 
L temporal pole 240 6.8 −52 12 −32 
mPFC/frontal pole 318 6.73 56 14 
L AG 219 6.58 −56 −58 28 
L parietal/central operculum 72 6.58 −42 −18 14 
ventral mPFC 195 6.35 56 −8 
L PCC 65 6.32 −8 −46 26 
R cerebellar crus II 6.18 26 −78 −36 
L posterior insula 12 6.07 −40 −4 −12 
R supramarginal gyrus 49 6.05 64 −26 22 
R hippocampus 22 6.01 24 −8 −24 
R supramarginal gyrus 17 6.01 64 −40 10 
R frontal pole 5.89 12 46 40 
R posterior insula 5.81 42 −8 
R PCC 5.79 −46 28 
Ventral ACC 5.78 −2 34 −2 

Results are voxel-level thresholded at FWE-corrected p < .05. Clusters defined using FSL's cluster command using a Z > 5.75 threshold. Coordinates in white matter, brainstem, or outside the brain are not included in this table. Peak coordinates from the L AG cluster were used in the GIMME analyses. Results are organized by peak Z value in descending order. R = right; L = left; MTG = middle temporal gyrus; STG = superior temporal gyrus.

Task-modulated Network Connectivity

To determine directed functional connectivity between activated regions, we applied a validated modeling approach to the task data called GIMME (Gates et al., 2011, 2017). We focused analyses on eight ROIs reported in the GLM analyses: medial cerebellar crus II, preSMA/dACC, lIPL, left primary/secondary visual cortices, lAI, ldlPFC, lIFJ, and lAG (Figure 4A). Model fit suffered when more than two interaction effects were included; thus, to limit the number of interaction effects analyzed, we focused on evidence of directed and undirected connectivity from studies of working memory and inhibition tasks (Harding et al., 2015; Zhang et al., 2015) and resting-state studies (Uddin, Supekar, Ryali, & Menon, 2011). We tested whether directed connectivity strength was modulated by Flexibility and/or Control trials between the lIFJ and all other ROIs by specifying task-modulatory effects (lIFJ × Flexibility and lIFJ × Control). Models were fit for two datasets that allowed inferences into changes in network topology across time: Timepoint 1, combined across Runs 1 and 2, and Timepoint 2, combined across Runs 3 and 4. For each timepoint, we report paths that were present for the majority of the sample (group-level paths), the majority of a subset of the sample (subgroup-level paths), and a minority of the sample (individual-level paths).

Figure 4. 

Network connectivity. Contemporaneous group- and subgroup-level paths (not individual-level paths) are displayed. ROIs for the GIMME analysis are displayed (A) and the results are shown for Timepoint 1 (B) and Timepoint 2 (C). Brain regions are colored according to path count matrices for B and C include all paths estimated, including group-, subgroup-, and individual-level paths. Matrices illustrate the number of participants in the sample who had a particular path estimated (of 26 participants). Matrices are asymmetric, representing directed connectivity. Columns are the independent variables, and rows are dependent variables. ROIs are as follows: medial cerebellar crus II, preSMA/dACC, lIPL, lAI, ldlPFC, lIFJ, lAG, and left primary/secondary visual cortices. Note that for Runs 1 and 2 (B), the visual ROI had a group-level main effect of control trials in addition to the subgroup-level main effect of flexibility trials.

Figure 4. 

Network connectivity. Contemporaneous group- and subgroup-level paths (not individual-level paths) are displayed. ROIs for the GIMME analysis are displayed (A) and the results are shown for Timepoint 1 (B) and Timepoint 2 (C). Brain regions are colored according to path count matrices for B and C include all paths estimated, including group-, subgroup-, and individual-level paths. Matrices illustrate the number of participants in the sample who had a particular path estimated (of 26 participants). Matrices are asymmetric, representing directed connectivity. Columns are the independent variables, and rows are dependent variables. ROIs are as follows: medial cerebellar crus II, preSMA/dACC, lIPL, lAI, ldlPFC, lIFJ, lAG, and left primary/secondary visual cortices. Note that for Runs 1 and 2 (B), the visual ROI had a group-level main effect of control trials in addition to the subgroup-level main effect of flexibility trials.

Model fit for the first timepoint was good for 28 of 32 participants' models (n = 28): CFI: M (SD) = 0.97 (0.010), NNFI: M (SD) = 0.94 (0.017), RMSEA: M (SD) = 0.07 (0.009), SRMR: M (SD) = 0.06 (0.016). This was true for the second timepoint as well, but included a different set of 28 participants (n = 28): CFI: M (SD) = 0.97 (0.008), NNFI: M (SD) = 0.94 (0.014), RMSEA: M (SD) = 0.07 (0.009), SRMR: M (SD) = 0.06 (0.015). Clustering solution validity analyses indicated that clusters were not modular, suggesting individuals from different subgroups were more similar than dissimilar in their directed connectivity profiles than would be expected if a separable categorical structure existed (Figure 5). Although the subgroup structure was not modular, previous simulation studies demonstrate the advantages of the subgrouping step in increasing the accuracy in the detection of paths using GIMME (Gates et al., 2017); therefore, we retained the results of this analysis for interpretation of paths but do not further explore subgroup comparisons in network topology or task performance.

Figure 5. 

Cluster validation results. The top row shows results for Timepoint 1, and the bottom row shows results for Timepoint 2. For Timepoint 1, the clustering solution was not stable nor modular. For Timepoint 2, the clustering solution was stable but not modular. For A and B, the black horizontal line represents the point at which 20% of participants were placed into different clusters than the original solution (20% of nodes perturbed). The dashed vertical line identifies the point at which the perturbed graph reached 20% of nodes perturbed (Timepoint 1: 9%, Timepoint 2: 21%). Black dots represent the perturbed graph based on the original clustering solution, whereas the red dots represent a perturbed random graph. (C) Modularity for the original clustering solution (Timepoint 1: 0.02, Timepoint 2: .04) was not better than expected by chance (Timepoint 1: >.11, Timepoint 2: >.12), suggesting that these clustering solutions did not yield modular subgroups.

Figure 5. 

Cluster validation results. The top row shows results for Timepoint 1, and the bottom row shows results for Timepoint 2. For Timepoint 1, the clustering solution was not stable nor modular. For Timepoint 2, the clustering solution was stable but not modular. For A and B, the black horizontal line represents the point at which 20% of participants were placed into different clusters than the original solution (20% of nodes perturbed). The dashed vertical line identifies the point at which the perturbed graph reached 20% of nodes perturbed (Timepoint 1: 9%, Timepoint 2: 21%). Black dots represent the perturbed graph based on the original clustering solution, whereas the red dots represent a perturbed random graph. (C) Modularity for the original clustering solution (Timepoint 1: 0.02, Timepoint 2: .04) was not better than expected by chance (Timepoint 1: >.11, Timepoint 2: >.12), suggesting that these clustering solutions did not yield modular subgroups.

For Timepoint 1, there were 10 group-level, 13 subgroup-level, and 55 individual-level paths estimated. The majority of participants demonstrated a significant positive main effect of Flexibility trials on the lIFJ (n = 24), M (SD) = 0.35 (0.17), range = −0.19 to 0.58, indicating that lIFJ activity increased in response to Flexibility trials, consistent with the GLM results (Figure 4B). There were also subgroup-level main effects of Flexibility trials in the lAG (n = 14), M (SD) = −0.27 (0.07), range = −0.38 to −0.13, and left visual cortex (n = 15), M (SD) = 0.21 (0.10), range = 0.08–0.47, meaning that lAG activity decreased in response to Flexibility trials and left visual cortex activity increased in response to Flexibility trials. The majority of participants demonstrated a significant main effect of Control trials on the left visual cortex (n = 25), M (SD) = 0.27 (0.15), range = −0.23 to 0.46. Intrinsic connectivity was evident for all participants primarily emanating from the lIFJ to five other nodes: cerebellum, dACC, lIPL, lAI, and ldlPFC. Task-modulated connectivity was evident only at the individual level, meaning these paths were estimated for a minority of participants. The most commonly estimated interaction effect between the lIFJ and Flexibility trials was for the lIFJ → left visual cortex path, which was significant for five participants, M (SD) = −0.09 (0.02), range = −0.12 to −0.07. Because none of these individuals had an intrinsic connection from lIFJ → left visual cortex, the negative estimates for this interaction effect indicate that increased lIFJ activity relates to decreased visual cortex activity during Flexibility trials. No participants had the lIFJ × Control → left visual cortex path estimated, which indicates that the task-modulated connectivity between the lIFJ and visual cortex is specific to Flexibility trials.

For Timepoint 2, there were 10 group-level, 9 subgroup-level, and 57 individual-level paths estimated, and the directed network connectivity models were largely consistent with the results from Timepoint 1 (Figure 4C). But there was a trend toward fewer contemporaneous paths estimated for Timepoint 2 compared with Timepoint 1 across the sample (n = 26), Timepoint 1: M (SD) = 24.88 (4.60), range = 16–34; Timepoint 2: M (SD) = 23.19 (3.69), range = 15–30, t(25) = 1.85, p = .076. Group-level paths common across timepoints included the main effect of Control trials on the left visual cortex (n = 27), M (SD) = 0.21 (0.17), range = −0.15 to 0.45, and seven intrinsic paths, including the five paths originating from the lIFJ reported for Timepoint 1. In addition, four subgroup-level intrinsic paths were estimated for both timepoints (e.g., ldlPFC → lIPL). Of note, the main effect of Flexibility trials on the lIFJ was estimated at the subgroup level, but not group level, for Timepoint 2, meaning that only a subset of participants had this path estimated (n = 14), M (SD) = 0.33 (0.42), range = −0.25 to 1.64. The subgroup-level main effects of Flexibility trials on the lAG and left visual cortex present at the first timepoint were only evident at the individual-level at the second timepoint. Two participants demonstrated Flexibility-modulated connectivity for the lIFJ → left visual cortex path, M (SD) = −0.09 (0.03), range = −0.11 to −0.07, and three different participants demonstrated Control-modulated connectivity for the lIFJ → left visual cortex path, M (SD) = −0.09 (0.03), range = −0.11 to −0.07. Together with the results of the GLM, results suggest that Flexibility trials lead to initial activation of the lIFJ, which in turn relates to increased activity in a number of other regions, including the cerebellum, dACC, lIPL, lAI, and ldlPFC, and a decrease in activity to the visual cortex in the service of flexible cognition. The preponderance of subgroup- and individual-level paths estimated also provide evidence of substantial interindividual variability in network topology for the FIST.

Brain–Behavior Relationships

The GIMME analyses produce a large amount of output variables, such as intrinsic connectivity paths, main effects of task trials and task-modulated interaction paths important for individuals' directed connectivity maps. To determine which of these variables were related to performance on the FIST, we used a feature selection tool called adaptive LASSO. For Timepoint 1, four intrinsic paths, one main effect of Flexibility trials, and one task-modulated interaction path were related to efficiency on the FIST (R2 = .69; Figure 6, top). Stronger functional connectivity for some intrinsic paths, such as dACC → cerebellum (b = 1.28, path estimated for n = 20), lIFJ → cerebellum (b = 1.97, path estimated for n = 28), and dlPFC → lIPL (b = 3.14, path estimated for n = 19) paths, were related to better task efficiency. On the other hand, stronger connectivity for other intrinsic paths, such as lIPL → cerebellum (b = −0.55, path estimated for n = 19), were related to worse task performance. Weaker deactivation of the lAG in response to Flexibility trials (i.e., more positive beta estimates) was related to better FIST performance (b = 3.56, path estimated for n = 22). Negative beta estimates for Flexibility-modulated directed connectivity from the lIFJ to the visual cortex (b = −3.22, path estimated for n = 5) were related to better task performance. For Timepoint 2, eleven intrinsic paths and two task main effects of were related to efficiency on the FIST (R2 = .69; Figure 6, bottom). Consistent with Timepoint 1, stronger lIFJ → cerebellum (b = 1.43, path estimated for n = 28), dlPFC → lIPL (b = 0.68, path estimated for n = 20), lIFJ → AG (b = 0.59, path estimated for n = 17), and dACC → visual (b = 2.05, path estimated for n = 15) connectivity related to better task performance. In addition, higher beta estimates for other paths, such as lAG → ldlPFC (b = −0.17, path estimated for n = 20), lIFJ → visual (b = −3.47, path estimated for n = 6), Flexibility → IPL (b = −0.63, path estimated for n = 5), and Control → visual (b = −4.14, path estimated for n = 28) paths related to worse performance.

Figure 6. 

Top: Variables selected by adaptive LASSO in brain–behavior analysis for Timepoint 1. Each subplot represents a selected variable, labeled on the x-axis. Unstandardized beta estimated determined with adaptive LASSO are reported in the title of each subplot. Subplots with x- and y-axis labels in red represent paths that were selected for both Timepoints 1 and 2. The number of participants for which each path was estimated in the GIMME analyses (n) is noted for each subplot. Bottom: Variable selected by adaptive LASSO in brain–behavior analysis for Timepoint 2.

Figure 6. 

Top: Variables selected by adaptive LASSO in brain–behavior analysis for Timepoint 1. Each subplot represents a selected variable, labeled on the x-axis. Unstandardized beta estimated determined with adaptive LASSO are reported in the title of each subplot. Subplots with x- and y-axis labels in red represent paths that were selected for both Timepoints 1 and 2. The number of participants for which each path was estimated in the GIMME analyses (n) is noted for each subplot. Bottom: Variable selected by adaptive LASSO in brain–behavior analysis for Timepoint 2.

DISCUSSION

The development and validation of tasks that capture real-world cognitive phenomena is exceptionally difficult, and adaptation of tasks to the fMRI environment presents a significant challenge for the field. We present an fMRI-adapted version of the FIST that has previously been psychometrically validated. We demonstrate that this fMRI-adapted task is a reliable measure of cognitive flexibility, demonstrating good internal consistency and excellent reliability. In addition, our fMRI-adapted task showed convergent validity with the computer-based administration outside the scanner. As predicted, we identified regions within the canonical FPN and SN that were more active in response to trials engaging cognitive flexibility compared with trials controlling for visual and motor responses. Consistent across participants, the lIFJ emerged as a central node in a network of cortical and subcortical regions that is directly engaged in cognitive flexibility. Finally, we demonstrate significant variability in functional connections at the individual level.

We observed strong activation in the superordinate frontocinguloparietal network during Flexibility trials. Specifically, we observed high activation levels in the IFJ, IFG, dlPFC, AI, dACC, IPL/IPS, posterior temporal cortex, extrastriate cortex, striatum, thalamus, and cerebellum (maximum z-statistic = 8.07). We extend past activation-based findings by reporting specific node-to-node directed functional connectivity patterns estimated at the individual-participant level that support cognitive flexibility.

Although our activation-based analyses identified a large number of cortical and subcortical regions active during Flexibility trials, directed network analyses clarified that only the lIFJ was directly activated by Flexibility trials for the majority of participants and that other FPN (i.e., ldlPFC, lIPL), SN (i.e., lAI, dACC), and cerebellar nodes were indirectly activated via their functional connections with the lIFJ. In other words, even when controlling for other regions' influence, the lIFJ was activated in response to the Flexibility condition. When we say that brain areas are indirectly activated by the IFJ, this is not the same as being indirectly connected or the same as undirected connectivity. What we mean is that, whereas regions such as the ldlPFC showed activation in univariate analyses, this relation between ldlPFC and task was better explained by an indirect effect between the ldlPFC and IFJ. Specifically, GIMME analyses demonstrated that only the lIFJ was directly activated by the task after controlling for other possible connections with other brain regions. This means that, once connectivity between the lIFJ and ldlPFC was taken into account, the task did not explain a significant amount of variability in the ldlPFC activity. Therefore, to explain the univariate activations of areas like the ldlPFC, GIMME analyses clarified that activations to the ldlPFC are most likely due to their functional connections with the lIFJ.

These indirect effects can be interpreted as follows: Controlling for other regions' influence, the ldlPFC does not evidence a direct relation with the Flexibility condition but is instead activated in response to lIFJ activity. In summary, we hypothesize that the lIFJ is the initial area of activation in response to engaging cognitive flexibility, which leads to engagement of other prefrontal, parietal, and cerebellar regions. This finding is consistent with a study of inhibition and working memory tasks showing that, among left-lateralized dACC, IPS, and dlPFC nodes, the lIFJ was the most likely site of initial engagement in response to task onsets (Harding et al., 2015). Brain signal variability in the lIFJ has been shown to specifically relate to better cognitive flexibility (Armbruster-Genç, Ueltzhöffer, & Fiebach, 2016), and individuals who tend to spontaneously initiate task switches show lower lIFJ activation during cognitive flexibility tasks (Armbruster et al., 2012). In line with the broader cognitive flexibility literature, we suggest that lIFJ activity and its functional connectivity with other brain regions most likely modulates cognitive flexibility via its primary role in updating cognitive sets (Kim et al., 2012; Derrfuss, Brass, Neumann, & von Cramon, 2005).

Another node in the FPN, the dlPFC, was indirectly activated in response to Flexibility trials. The dlPFC exhibits sustained activation during working memory tasks, suggesting its primary role is in maintenance of information (Cohen et al., 1997). Behavioral studies demonstrate that there are unique contributions of cognitive flexibility and working memory to FIST performance, and working memory maintenance contributes to item difficulty for the FIST (Dick, 2014). Overall, these results suggest that the activity and functional connectivity of the ldlPFC during the FIST may support behavioral performance due to its role in working memory maintenance.

In addition to activation of nodes within the FPN, the FIST engaged canonical regions of the SN—the AI and dACC (Uddin, 2015). Functional integration between the FPN and SN supports executive functions such as working memory (Shine et al., 2016). We found that the lIFJ was functionally connected to the lAI and dACC. Harding et al. (2015) reported that the lIFJ → dACC path was modulated by both working memory and inhibition tasks, further implicating FPN–SN functional integration in executive function (Harding et al., 2015). Here, we extend past findings by demonstrating that FPN–SN integration supports cognitive flexibility. Additional activation during Flexibility trials was observed in extrastriate visual areas. One possibility is that these trials presented a greater visual attention load (Tomasi, Chang, Caparelli, & Ernst, 2007) than Control trials.

Directed functional connectivity analyses demonstrated that regions outside the FPN and SN also play an important role in cognitive flexibility. Functional connectivity studies demonstrate that stronger “anticorrelation” between lateral pFC regions, such as the dlPFC and IFJ, and DMN regions is associated with higher intelligence (Cole, Yarkoni, Repovs, Anticevic, & Braver, 2012) and greater consistency in behavioral performance (Kelly, Uddin, Biswal, Castellanos, & Milham, 2008). Accumulating evidence indicates that stronger FPN–DMN connectivity may reflect the intrusion of distracting, internally generated thought (e.g., mind wandering; Anticevic et al., 2012).

We found that stronger functional connectivity between a region important for externally oriented cognition, the dlPFC, and a region of the DMN, the lAG, was related to poorer performance on the FIST for the majority of participants (n = 20). In a subset of participants (n = 22), Flexibility trials directly disengaged the lAG, consistent with the region's role within the DMN (Fransson, 2006). Paradoxically, weaker deactivation of the lAG during Flexibility trials was related to more efficient performance on the FIST for Timepoint 1. This contradicts literature demonstrating that lower DMN activity is associated with better performance on working memory tasks (Anticevic et al., 2012), suggesting that some regions in the DMN may function differently for cognitive flexibility tasks.

Though not observable at the level of whole-brain GLM analyses, some differences were apparent in individuals' connectomes from the first timepoint to the second timepoint, which may reflect a transition from task acquisition to optimal task performance. In line with prior findings from the task learning literature, we find that improvements in performance across timepoints may be indicative of increased task proficiency (Nichols et al., 2014). We also note that unique paths related to task performance across timepoints, which provides further evidence that distinct processes with different neurobiological underpinnings may be occurring across time. We suggest that network topology for the first timepoint reflects task acquisition and learning processes, whereas the second timepoint represents increased proficiency and automaticity in completing the FIST (Kübler, Dixon, & Garavan, 2006).

Limitations

Some limitations of the FIST are worth noting. First, some degree of precision and experimental control is necessarily sacrificed by the open-ended nature of the task. Participants are free to use the entire 8-sec period to comply with task instructions, and it is consequently unclear what they might be doing with any remaining time at the end of each trial. It is possible that some participants engage in mind wandering upon early completion of a trail, resulting in additional intertrial and interindividual variability.

Here, we demonstrate high ICC across runs of the FIST. However, a more conventional measure of test–retest reliability would be examination of performance across separate measurements occasions. Unfortunately, no such measurements are available from these participants at this time. Future work with larger samples tested across multiple sessions will be necessary to assess psychometric properties of the FIST.

Additionally, the current paradigm does not rule out the possibility that individual differences in basic information processing speed might influence task performance. A related concern regards the calculation of accuracy in the current study, which counted a trail as correct if six consecutive manual responses were given correctly. For researchers seeking to optimize this task in future iterations, it would be helpful to ensure that two button-pairs can be accurately attributed at the selection level, instead of having to depend on the six button-pair trial level.

Finally, we cannot completely rule out the possibility that the neuroimaging results we observe might be due to complexity differences or differences in task difficulty between the flexibility and control conditions, rather than cognitive flexibility per se. Still, our Flexibility–Control contrast revealed activations largely consistent with prior neuroimaging meta-analyses of cognitive flexibility (Kim et al., 2012), consistent with our interpretation that cognitive flexibility and not general task difficulty drive the current results.

Conclusions

We adapted a laboratory-based cognitive flexibility task, the FIST, to the fMRI environment. We provide evidence that the fMRI-adapted task is reliable and displays convergent validity with the computer-based version of the task. We also found strong evidence for validity of the task based on robust activation of canonical regions of the FPN, SN, and subcortical networks in response to Flexibility trials. Directed network connectivity analyses demonstrated that during Flexibility trials, participants initially engaged the lIFJ, which influenced activity in other frontoparietal, visual, and cerebellar regions. These results provide support for use of the FIST in future cognitive neuroscience investigations. Given that the FIST was initially developed for administration with developmental populations (Jacques & Zelazo, 2001) and reports of deficits in cognitive flexibility in clinical populations (Gioia, Isquith, Retzlaff, & Espy, 2002), we expect that adaptation of this task in developmental and clinical neuroimaging research will yield significant insights into the typical and atypical neural correlates of cognitive flexibility.

Acknowledgments

This work was supported by funding from the Canadian Institute for Advanced Research and the National Institute of Mental Health (R01MH107549) to L. Q. U. The authors gratefully acknowledge Kelly Duffy and Cara Arizmendi for their assistance with the GIMME analyses and Jason Nomi, Taylor Bolt, Catherine Burrows, and Shruti Vij for their assistance with data collection.

Reprint requests should be sent to Dina R. Dajani or Lucina Q. Uddin, Department of Psychology, University of Miami, P.O. Box 248185-0751, Coral Gables, FL 33124, or via-email: dina.r.dajani@gmail.com or l.uddin@miami.edu.

REFERENCES

REFERENCES
Anticevic
,
A.
,
Cole
,
M. W.
,
Murray
,
J. D.
,
Corlett
,
P. R.
,
Wang
,
X. J.
, &
Krystal
,
J. H.
(
2012
).
The role of default network deactivation in cognition and disease
.
Trends in Cognitive Sciences
,
16
,
584
592
.
Armbruster
,
D. J.
,
Ueltzhöffer
,
K.
,
Basten
,
U.
, &
Fiebach
,
C. J.
(
2012
).
Prefrontal cortical mechanisms underlying individual differences in cognitive flexibility and stability
.
Journal of Cognitive Neuroscience
,
24
,
2385
2399
.
Armbruster-Genç
,
D. J.
,
Ueltzhöffer
,
K.
, &
Fiebach
,
C. J.
(
2016
).
Brain signal variability differentially affects cognitive flexibility and cognitive stability
.
Journal of Neuroscience
,
36
,
3978
3987
.
Badre
,
D.
, &
Wagner
,
A. D.
(
2006
).
Computational and neurobiological mechanisms underlying cognitive flexibility
.
Proceedings of the National Academy of Sciences, U.S.A.
,
103
,
7186
7191
.
Bailey
,
C. E.
(
2007
).
Cognitive accuracy and intelligent executive function in the brain and in business
.
Annals of the New York Academy of Sciences
,
1118
,
122
141
.
Braun
,
U.
,
Schäfer
,
A.
,
Walter
,
H.
,
Erk
,
S.
,
Romanczuk-Seiferth
,
N.
,
Haddad
,
L.
, et al
(
2015
).
Dynamic reconfiguration of frontal brain networks during executive cognition in humans
.
Proceedings of the National Academy of Sciences, U.S.A.
,
112
,
11678
11683
.
Brown
,
T. A.
(
2006
).
Confirmatory factor analysis for applied research
.
The Guilford Press
.
Burt
,
K. B.
, &
Paysnick
,
A. A.
(
2012
).
Resilience in the transition to adulthood
.
Development and Psychopathology
,
24
,
493
505
.
Chein
,
J. M.
, &
Schneider
,
W.
(
2005
).
Neuroimaging studies of practice-related change: fMRI and meta-analytic evidence of a domain-general control network for learning
.
Cognitive Brain Research
,
25
,
607
623
.
Chen
,
Q.
,
Yang
,
W.
,
Li
,
W.
,
Wei
,
D.
,
Li
,
H.
,
Lei
,
Q.
, et al
(
2014
).
Association of creative achievement with cognitive flexibility by a combined voxel-based morphometry and resting-state functional connectivity study
.
Neuroimage
,
102
,
474
483
.
Cicchetti
,
D. V.
(
1994
).
Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology
.
Psychological Assessment
,
6
,
284
290
.
Cicchetti
,
D. V.
, &
Sparrow
,
S. S.
(
1990
). .
Cohen
,
J. D.
,
Perlstein
,
W. M.
,
Braver
,
T. S.
,
Nystrom
,
L. E.
,
Noll
,
D. C.
,
Jonides
,
J.
, et al
(
1997
).
Temporal dynamics of brain activation during a working memory task
.
Nature
,
386
,
604
608
.
Cohen
,
J. R.
,
Gallen
,
C. L.
,
Jacobs
,
E. G.
,
Lee
,
T. G.
, &
D'Esposito
,
M.
(
2014
).
Quantifying the reconfiguration of intrinsic networks during working memory
.
PLoS One
,
9
,
e106636
.
Cole
,
M. W.
,
Reynolds
,
J. R.
,
Power
,
J. D.
,
Repovs
,
G.
,
Anticevic
,
A.
, &
Braver
,
T. S.
(
2013
).
Multi-task connectivity reveals flexible hubs for adaptive task control
.
Nature Neuroscience
,
16
,
1348
1355
.
Cole
,
M. W.
,
Yarkoni
,
T.
,
Repovs
,
G.
,
Anticevic
,
A.
, &
Braver
,
T. S.
(
2012
).
Global connectivity of prefrontal cortex predicts cognitive control and intelligence
.
Journal of Neuroscience
,
32
,
8988
8999
.
Cubillo
,
A.
,
Halari
,
R.
,
Ecker
,
C.
,
Giampietro
,
V.
,
Taylor
,
E.
, &
Rubia
,
K.
(
2010
).
Reduced activation and inter-regional functional connectivity of fronto-striatal networks in adults with childhood attention-deficit hyperactivity disorder (ADHD) and persisting symptoms during tasks of motor inhibition and cognitive switching
.
Journal of Psychiatric Research
,
44
,
629
639
.
Dajani
,
D. R.
, &
Uddin
,
L. Q.
(
2015
).
Demystifying cognitive flexibility: Implications for clinical and developmental neuroscience
.
Trends in Neurosciences
,
38
,
571
578
.
Derrfuss
,
J.
,
Brass
,
M.
,
Neumann
,
J.
, &
von Cramon
,
D. Y.
(
2005
).
Involvement of the inferior frontal junction in cognitive control: Meta-analyses of switching and stroop studies
.
Human Brain Mapping
,
25
,
22
34
.
Dick
,
A. S.
(
2014
).
The development of cognitive flexibility beyond the preschool period: An investigation using a modified Flexible Item Selection Task
.
Journal of Experimental Child Psychology
,
125
,
13
34
.
Dixon
,
M. L.
,
De La Vega
,
A.
,
Mills
,
C.
,
Andrews-Hanna
,
J.
,
Spreng
,
R. N.
,
Cole
,
M. W.
, et al
(
2018
).
Heterogeneity within the frontoparietal control network and its relationship to the default and dorsal attention networks
.
Proceedings of the National Academy of Sciences, U.S.A.
,
115
,
E1598
E1607
.
Eklund
,
A.
,
Nichols
,
T. E.
, &
Knutsson
,
H.
(
2016
).
Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates
.
Proceedings of the National Academy of Sciences, U.S.A.
,
113
,
7900
7905
.
Feldman
,
M. J.
, &
Drasgow
,
J. A.
(
1951
).
A visual–verbal test for schizophrenia
.
Psychiatric Quarterly
,
25
,
55
64
.
Fransson
,
P.
(
2006
).
How default is the default mode of brain function?: Further evidence from intrinsic BOLD signal fluctuations
.
Neuropsychologia
,
44
,
2836
2845
.
Friedman
,
J.
,
Hastie
,
T.
, &
Tibshirani
,
R.
(
2010
).
Regularization paths for generalized linear models via coordinate descent
.
Journal of Statistical Software
,
33
,
1
22
.
Friston
,
K. J.
,
Harrison
,
L.
, &
Penny
,
W.
(
2003
).
Dynamic causal modelling
.
Neuroimage
,
19
,
1273
1302
.
Gates
,
K. M.
,
Fisher
,
Z.
, &
Arizmendi
,
C.
(
2018
).
Package ‘perturbR.’ (Version 0.1.2) [Computer software]
.
Gates
,
K. M.
,
Fisher
,
Z. F.
,
Arizmendi
,
C.
,
Henry
,
T. R.
,
Duffy
,
K. A.
, &
Mucha
,
P. J.
(
2019
).
Assessing the robustness of cluster solutions obtained from sparse count matrices
.
Psychological Methods
,
24
,
675
689
.
Gates
,
K. M.
,
Lane
,
S. T.
,
Varangis
,
E.
,
Giovanello
,
K.
, &
Guskiewicz
,
K.
(
2017
).
Unsupervised classification during time-series model building
.
Multivariate Behavioral Research
,
52
,
129
148
.
Gates
,
K. M.
,
Molenaar
,
P. C.
,
Hillary
,
F. G.
, &
Slobounov
,
S.
(
2011
).
Extended unified SEM approach for modeling event-related fMRI data
.
Neuroimage
,
54
,
1151
1158
.
Genet
,
J. J.
, &
Siemer
,
M.
(
2011
).
Flexible control in processing affective and non-affective material predicts individual differences in trait resilience
.
Cognition & Emotion
,
25
,
380
388
.
Gioia
,
G. A.
,
Isquith
,
P. K.
,
Retzlaff
,
P. D.
, &
Espy
,
K. A.
(
2002
).
Confirmatory factor analysis of the behavior rating inventory of executive function (BRIEF) in a clinical sample
.
Child Neuropsychology
,
8
,
249
257
.
Harding
,
I. H.
,
Yücel
,
M.
,
Harrison
,
B. J.
,
Pantelis
,
C.
, &
Breakspear
,
M.
(
2015
).
Effective connectivity within the frontoparietal control network differentiates cognitive control and working memory
.
Neuroimage
,
106
,
144
153
.
Hellyer
,
P. J.
,
Scott
,
G.
,
Shanahan
,
M.
,
Sharp
,
D. J.
, &
Leech
,
R.
(
2015
).
Cognitive flexibility through metastable neural dynamics is disrupted by damage to the structural connectome
.
Journal of Neuroscience
,
35
,
9050
9063
.
Jacques
,
S.
, &
Zelazo
,
P. D.
(
2001
).
The Flexible Item Selection Task (FIST): A measure of executive function in preschoolers
.
Developmental Neuropsychology
,
20
,
573
591
.
Kapp
,
S. K.
,
Gantman
,
A.
, &
Laugeson
,
E. A.
(
2011
).
Transition to adulthood for high-functioning individuals with autism spectrum disorders
. In
M.
Mohammad-Reza
(Ed.),
A comprehensive book on autism spectrum disorders
(pp.
451
478
).
London
:
InTech Open
.
Kelly
,
A. M.
,
Uddin
,
L. Q.
,
Biswal
,
B. B.
,
Castellanos
,
F. X.
, &
Milham
,
M. P.
(
2008
).
Competition between functional brain networks mediates behavioral variability
.
Neuroimage
,
39
,
527
537
.
Kim
,
C.
,
Cilles
,
S. E.
,
Johnson
,
N. F.
, &
Gold
,
B. T.
(
2012
).
Domain general and domain preferential brain regions associated with different types of task switching: A meta-analysis
.
Human Brain Mapping
,
33
,
130
142
.
Konishi
,
S.
,
Hayashi
,
T.
,
Uchida
,
I.
,
Kikyo
,
H.
,
Takahashi
,
E.
, &
Miyashita
,
Y.
(
2002
).
Hemispheric asymmetry in human lateral prefrontal cortex during cognitive set shifting
.
Proceedings of the National Academy of Sciences, U.S.A.
,
99
,
7803
7808
.
Konishi
,
S.
,
Nakajima
,
K.
,
Uchida
,
I.
,
Kameyama
,
M.
,
Nakahara
,
K.
,
Sekihara
,
K.
, et al
(
1998
).
Transient activation of inferior prefrontal cortex during cognitive set shifting
.
Nature Neuroscience
,
1
,
80
84
.
Kübler
,
A.
,
Dixon
,
V.
, &
Garavan
,
H.
(
2006
).
Automaticity and reestablishment of executive control—An fMRI study
.
Journal of Cognitive Neuroscience
,
18
,
1331
1342
.
Kuder
,
G. F.
, &
Richardson
,
M. W.
(
1937
).
The theory of the estimation of test reliability
.
Psychometrika
,
2
,
151
160
.
Lane
,
S. T.
, &
Gates
,
K. M.
(
2017
).
Automated selection of robust individual-level structural equation models for time series data
.
Structural Equation Modeling
,
24
,
768
782
.
Lane
,
S. T.
,
Gates
,
K. M.
,
Pike
,
H. K.
,
Beltz
,
A. M.
, &
Wright
,
A. G. C.
(
2019
).
Uncovering general, shared, and unique temporal patterns in ambulatory assessment data
.
Psychological Methods
,
24
,
54
69
.
Leber
,
A. B.
,
Turk-Browne
,
N. B.
, &
Chun
,
M. M.
(
2008
).
Neural predictors of moment-to-moment fluctuations in cognitive flexibility
.
Proceedings of the National Academy of Sciences, U.S.A.
,
105
,
13592
13597
.
Lindquist
,
M. A.
,
Meng Loh
,
J.
,
Atlas
,
L. Y.
, &
Wager
,
T. D.
(
2009
).
Modeling the hemodynamic response function in fMRI: Efficiency, bias and mis-modeling
.
Neuroimage
,
45(Suppl. 1)
,
S187
S198
.
Miller
,
M. B.
, &
Van Horn
,
J. D.
(
2007
).
Individual variability in brain activations associated with episodic retrieval: A role for large-scale databases
.
International Journal of Psychophysiology
,
63
,
205
213
.
Nichols
,
T. T.
,
Gates
,
K. M.
,
Molenaar
,
P. C.
, &
Wilson
,
S. J.
(
2014
).
Greater BOLD activity but more efficient connectivity is associated with better cognitive performance within a sample of nicotine-deprived smokers
.
Addiction Biology
,
19
,
931
940
.
Prentice
,
K. J.
,
Gold
,
J. M.
, &
Buchanan
,
R. W.
(
2008
).
The Wisconsin card sorting impairment in schizophrenia is evident in the first four trials
.
Schizophrenia Research
,
106
,
81
87
.
R Core Team
. (
2017
).
R: A language and environment for statistical com-puting
.
Vienna
:
R Foundation for Statistical Computing
.
Revelle
,
W. R.
(
2017
).
Psych: Procedures for personality and psychological research
. https://www.scholars.northwestern.edu/en/publications/psych-procedures-for-personality-and-psychological-research.
Sainani
,
K. L.
(
2017
).
Reliability Statistics
.
PM & R: The Journal of Injury, Function, and Rehabilitaion
,
9
,
622
628
.
Shine
,
J. M.
,
Bissett
,
P. G.
,
Bell
,
P. T.
,
Koyejo
,
O.
,
Balsters
,
J. H.
,
Gorgolewski
,
K. J.
, et al
(
2016
).
The dynamics of functional brain networks: Integrated network states during cognitive task performance
.
Neuron
,
92
,
544
554
.
Shrout
,
P. E.
, &
Fleiss
,
J. L.
(
1979
).
Intraclass correlations: Uses in assessing rater reliability
.
Psychological Bulletin
,
86
,
420
428
.
Signorell
,
A.
(
2017
).
DescTools: Tools for descriptive statistics (R Package Version 0.99.20)
.
Tomasi
,
D.
,
Chang
,
L.
,
Caparelli
,
E. C.
, &
Ernst
,
T.
(
2007
).
Different activation patterns for working memory load and visual attention load
.
Brain Research
,
1132
,
158
165
.
Uddin
,
L. Q.
(
2015
).
Salience processing and insular cortical function and dysfunction
.
Nature Reviews Neuroscience
,
16
,
55
61
.
Uddin
,
L. Q.
,
Supekar
,
K. S.
,
Ryali
,
S.
, &
Menon
,
V.
(
2011
).
Dynamic reconfiguration of structural and functional connectivity across core neurocognitive brain networks with development
.
Journal of Neuroscience
,
31
,
18578
18589
.
Ueltzhöffer
,
K.
,
Armbruster-Genç
,
D. J.
, &
Fiebach
,
C. J.
(
2015
).
Stochastic dynamics underlying cognitive stability and flexibility
.
PLoS Computational Biology
,
11
,
e1004331
.
van Schouwenburg
,
M. R.
,
den Ouden
,
H. E.
, &
Cools
,
R.
(
2010
).
The human basal ganglia modulate frontal-posterior connectivity during attention shifting
.
Journal of Neuroscience
,
30
,
9910
9918
.
Waskom
,
M. L.
,
Kumaran
,
D.
,
Gordon
,
A. M.
,
Rissman
,
J.
, &
Wagner
,
A. D.
(
2014
).
Frontoparietal representations of task context support the flexible control of goal-directed cognition
.
Journal of Neuroscience
,
34
,
10743
10755
.
Zelazo
,
P. D.
(
2006
).
The dimensional change card sort (DCCS): A method of assessing executive function in children
.
Nature Protocols
,
1
,
297
301
.
Zelazo
,
P. D.
,
Anderson
,
J. E.
,
Richler
,
J.
,
Wallner-Allen
,
K.
,
Beaumont
,
J. L.
,
Conway
,
K. P.
, et al
(
2014
).
NIH toolbox cognition battery (CB): Validation of executive function measures in adults
.
Journal of the International Neuropsychological Society
,
20
,
620
629
.
Zelazo
,
P. D.
,
Anderson
,
J. E.
,
Richler
,
J.
,
Wallner-Allen
,
K.
,
Beaumont
,
J. L.
, &
Weintraub
,
S.
(
2013
).
II. NIH toolbox cognition battery (CB): Measuring executive function and attention
.
Monographs of the Society for Research in Child Development
,
78
,
16
33
.
Zhang
,
Q.
,
Zhang
,
G.
,
Yao
,
L.
, &
Zhao
,
X.
(
2015
).
Impact of real-time fMRI working memory feedback training on the interactions between three core brain networks
.
Frontiers in Behavioral Neuroscience
,
9
,
244
.
Zou
,
H.
(
2006
).
The adaptive lasso and its oracle properties
.
Journal of the American Statistical Association
,
101
,
1418
1429
.