## Abstract

Visual working memory (VWM) is a capacity-limited cognitive resource that plays an important role in complex cognitive behaviors. Recent studies indicate that regions subserving VWM may play a role in the perception and recognition of visual objects, suggesting that conscious object perception may depend on the same cognitive and neural architecture that supports the maintenance of visual object information. In the present study, we examined this question by testing object processing under a concurrent VWM load. Under a high VWM load, recognition was impaired for objects presented in the left visual field, in particular when two objects were presented simultaneously. Multivariate fMRI revealed that two independent but partially overlapping networks of brain regions contribute to object recognition. The first network consisted of regions involved in VWM encoding and maintenance. Importantly, these regions were also sensitive to object load. The second network comprised regions of the ventral temporal lobes traditionally associated with object recognition. Importantly, activation in both networks predicted object recognition performance. These results indicate that information processing in regions that mediate VWM may be critical to conscious visual perception. Moreover, the observation of a hemifield asymmetry in object recognition performance has important theoretical and clinical significance for the study of visual neglect.

## INTRODUCTION

Although our subjective experience of the visual world seems to be rich and detailed, in actuality our internal representations are quite sparse. A striking demonstration of our visual system's processing limitations comes from the change detection task, in which observers first see a multi-item array and are asked to maintain as much information as possible. The display is removed from view, and after a brief delay, a probe array appears to which observers report whether a change has occurred. Accordingly, observers have to hold information “on-line” after it has been removed from view, a cognitive faculty called visual working memory (VWM). Observers typically perform well for arrays containing four items or less, but performance declines significantly beyond that limit (Cowan, 2001; Luck & Vogel, 1997).

Given the overlap in cognitive and neural mechanisms underlying VWM and object recognition, in addition to the presumed role of WM in providing a workspace for consciousness, it seems plausible that the processing limitations underlying our conscious experience of objects in our environment may be determined by activation levels in dorsal and ventral brain areas mediating object-based VWM, namely, the sIPS and the LOC. Consequently, the goal of the present study was to test the neural and cognitive processing limits when the representational capacity limit of VWM has been exceeded. In other words, if conscious object processing shares resources with capacity-limited VWM, are we blind to objects once neural and cognitive resources related to VWM capacity have been depleted? It has been shown that a high VWM load is associated with impaired detection of unexpectedly appearing images of objects (Todd, Fougnie, & Marois, 2005). This inattentional blindness is possibly related to the suppression of activation in the right TPJ that is observed under a high VWM load (Todd et al., 2005). The TPJ, which includes regions of the inferior parietal lobule and superior temporal gyrus, subserves stimulus-driven attentional processes to detect salient and behaviorally relevant stimuli in the environment (Serences et al., 2005; Downar, Crawley, Mikulis, & Davis, 2002; Corbetta, Kincade, Ollinger, McAvoy, & Shulman, 2000). Thus, because only a limited number of higher-order perceptual representations can be formed and maintained, it may be necessary to suppress the TPJ system to shield the contents of conscious awareness from task-irrelevant distraction (Anticevic, Repovs, Shulman, & Barch, 2010; Shulman, Astafiev, McAvoy, d'Avossa, & Corbetta, 2007).

Interestingly, unilateral damage to the (typically right) TPJ leaves neurological patients with profound impairments in conscious awareness for visual stimuli in the contralateral visual field (Danckert & Ferber, 2006; Mort et al., 2003; Karnath, Ferber, & Himmelbach, 2001), a deficit referred to as visual neglect. Thus, although the visual pathways and object-sensitive areas remain intact, damage to regions including and surrounding the right TPJ severely diminishes the patients' abilities to report, respond to, or orient toward stimuli located toward the contralesional field (Heilman, Watson, & Valenstein, 1993). Even if the symptoms of neglect improve as indexed by increased detection of stimuli presented on the left side, most of these patients continue having impaired explicit awareness for a contralesional stimulus when two stimuli compete for processing resources (Ferber, Danckert, Joanisse, Goltz, & Goodale, 2003; Karnath, 1988).

In the current study, we sought to test how perceptual information is processed under a high VWM load. Specifically, we predicted that a high VWM load would deplete resources in regions mediating VWM (e.g., sIPS) as well as functionally deactivate the TPJ and, as such, would lead to impairments in conscious awareness for subsequently presented objects. In Experiment 1, we tested whether a high VWM load would affect the conscious report of visual objects, and if so, whether these processing deficits would demonstrate a hemifield asymmetry, mimicking those observed in visual neglect following lesions to the right TPJ. In Experiment 2, we used fMRI to examine the neural substrate of the cognitive interplay between VWM load and conscious object recognition. Importantly, we used a multivariate analysis to identify networks of brain regions that are jointly affected by VWM load and object processing, in addition to an ROI approach to isolate localized effects of load on independently defined areas involved in VWM.

## EXPERIMENT 1

### Methods

#### Stimuli and Procedure

Stimuli were back-projected onto a 27 × 36 cm projection screen, viewed at a distance of 117 cm through a mirror located in the scanner. Stimuli were delivered at a refresh rate of 50 Hz. Participants used a four-button response box placed on their abdomen to make all responses. Participants were familiarized with the tasks and responses several days before the scan, as well as immediately before scanning. In addition to the experimental task, participants performed a VWM localizer task.

Stimuli and procedure were similar to those of Experiment 1 (Figure 1); however, no object detection probe was presented. The timing of the stimuli was modified for the scanner: Each trial began with a fixation point for 503 msec, followed by the memory sample presented for 151 msec. After a 603-msec delay, one or two objects were presented for 67 msec 10° to the left or right of the fixation. After another 536-msec delay, the memory probe was presented for 1558 msec, followed by the object recognition probe, lasting 3202 msec. Each trial was followed by a 1380–7380 msec ITI.

Factors of load (low, high) and presentation condition (unilateral left, unilateral right, bilateral) were fully crossed for six different trial types. Participants performed 42 trials of each of the six conditions, divided into seven runs of 36 trials (six of each condition). Null (fixation) periods were also presented between trials as part of the ITI. Trial stimuli and procedures were otherwise identical to that of Experiment 1. Trial and null-period sequence was assigned using optseq2 (surfer.nmr.mgh.harvard.edu/optseq/), and participants were assigned a random sequence of runs.

##### VWM localizer

To compare PLS activations to known VWM regions, we also performed a change detection localizer task immediately following the experimental runs. The localizer task consisted of a change detection task, similar to the one used in the experimental task, with no intervening object presentation. The delay period lasted 1200 msec, and the memory probe was presented for 1709 msec. Set sizes of 1, 3, and 5 were used. Participants performed 24 trials of each set size, in two runs of 36 trials. The VWM localizer was analyzed using K weighted regressors (Todd & Marois, 2004).

#### Image Acquisition and Data Preprocessing

Images were acquired on a 3-T Siemens Magnetom Trio whole-body scanner with a matrix 12-channel head coil. Anatomical images were acquired before the functional images using a MP-RAGE sequence (repetition time [TR] = 2 sec, echo time [TE] = 2.63 sec, 160 oblique axial slices, with a 1 mm3 voxel size, field of view [FOV] = 25.6 cm, acquisition matrix = 256 × 256). Functional volumes were obtained using a whole head T2*-weighted EPI sequence (TR = 2 sec, TE = 30 msec, flip angle = 70°, 30 oblique axial slices with interleaved acquisition, 3.125 × 3.125 × 5 mm voxel resolution, field of view [FOV] = 20 cm, acquisition matrix = 64 × 64). The first four volumes of each run were discarded to allow the magnetization to reach steady state.

For PLS analysis, image preprocessing was performed with SPM5 software. Slice timing was corrected to the first slice, and motion correction using a 3-D Fourier transform interpolation using a functional volume minimizing motion to less than 2 mm was performed. Participants' images were then spatially normalized to Montreal Neurological Institute (MNI) space and smoothed using a 6-mm FWHM Gaussian filter using SPM5, resulting in a voxel size of 4 × 4 × 4 mm.

Multivariate group analyses were carried out using the PLS software package (McIntosh et al., 1996). Analyses of localizer scans for ROI analyses were performed using BrainVoyager QX (Goebel, Esposito, & Formisano, 2006). Slice scan time correction, 3-D motion correction, and temporal filtering of images were performed. Functional runs were aligned to the anatomical scan using automatic coregistration and corrected by visual inspection. An ROI for the sIPS was obtained for nine of the participants by applying K weighted regressors to the average of two change detection localizer scans and identifying the peak region of activation that was within IPS. These coordinates were selected to match most closely to the Talairach coordinates reported by Xu and Chun (2006). The mean Talairach coordinates for these ROIs were (right/left) 26/−25, −63/−64, and 44/44.

#### PLS

PLS analysis (McIntosh et al., 1996, 2004) is a multivariate statistical tool that is used to identify regional activity change as a function of task demands. In other words, PLS determines those brain regions whose activity covaries with the experimental conditions. This approach is akin to a PCA (Friston, Frith, & Frackowiak, 1993) operating under the assumption that brain function relates to the activity of a number of brain regions rather than a single region. This approach is data driven, and all task conditions can be entered into the analysis simultaneously, yielding whole-brain patterns of activity that covary with the experimental design. The output of PLS analysis is a set of latent variables (LVs), which are components that reflect patterns of brain activity related to task conditions.

Analysis was performed on all trials from all six conditions of load (low and high) crossed with object presentation side (left, right, and bilateral). Activity was analyzed for an epoch of six TRs normalized to the first TR in the trial, defined here as the fixation period preceding the memory sample. For each TR, a “brain score” was calculated for each participant to index how strongly the participant contributes to the pattern of brain activity identified for that TR.

Significance of each LV was determined by permutation testing (McIntosh et al., 1996) using 500 permutations. The reliability of the saliences (or weights) for the brain voxels that characterize each pattern identified by the LVs was determined by an independent bootstrap estimation of the standard errors (Efron & Tibshirani, 1986). Peak voxels with a bootstrap ratio (BSR) greater than 3.5 were considered reliable, which is approximately equivalent to p < .005. Clusters containing at least 10 reliable voxels were extracted, and a local maximum for each cluster was defined as the voxel with a BSR higher than any other voxel in a 20-mm cube centered on that voxel. Although most regions showed reliable activation across multiple time points, for the purposes of clarity, results are reported from BSRs for the second (LV1) and fourth (LV2) TRs. Locations of the maxima are reported in the stereotaxic coordinates of MNI space.

### Results and Discussion

The behavioral task from Experiment 1 (Figure 1) was adopted for use with fMRI (see Methods). Importantly, because object detection performance in Experiment 1 was close to ceiling, the detection task was eliminated, and the object recognition probe immediately followed the change detection probe.

#### Behavioral Results

Participants correctly identified the object or objects presented (without incorrectly identifying one of the lure objects) on 84% of trials (Table 1). When compared with Experiment 1, overall recognition performance was substantially higher, indicating that either the detection probe or the presence of catch trials in Experiment 1 may have potentially affected recognition performance to some extent. As in Experiment 1, however, a repeated measures ANOVA revealed that participants identified significantly fewer items under a high VWM load, F(1, 9) = 19.7, p = .002. Furthermore, participants also correctly identified significantly fewer items under bilateral presentation, relative to unilateral presentation, F(1, 9) = 26.2, p = .001. Unlike in Experiment 1, however, there was no significant effect of Presentation Side, F(1, 9) = 0.12, p = .74. Importantly, under bilateral presentation, recognition performance tended to be worse for objects presented on the left as revealed by a trend between presentation condition and object location, F(1, 9) = 3, p = .12, indicating that recognition performance demonstrated a similar pattern to Experiment 1. None of the other interactions were significant, Fs < .7, ps > .4.

Table 1.

Recognition Accuracy in Experiment 2

Unilateral
Bilateral
Unilateral
Bilateral
Left
Right
Left
Right
Left
Right
Left
Right
Recognition accuracy (SEs) 0.96 (0.02) 0.92 (0.02) 0.82 (0.02) 0.86 (0.03) 0.86 (0.02) 0.85 (0.02) 0.70 (0.05) 0.74 (0.03)

Unilateral
Bilateral
Unilateral
Bilateral
Left
Right
Left
Right
Left
Right
Left
Right
Recognition accuracy (SEs) 0.96 (0.02) 0.92 (0.02) 0.82 (0.02) 0.86 (0.03) 0.86 (0.02) 0.85 (0.02) 0.70 (0.05) 0.74 (0.03)

#### fMRI Analysis

Task-PLS identified two significant LVs that together explained more than 50% of the covariance between the fMRI images and the six task conditions. The first LV accounted for the largest amount of variance in the data (32%, p < .001). The averaged brain scores are displayed in Figure 3A and B, and the clusters of activity are listed in Table 2 and displayed in Figure 3C. This LV revealed greater activation in the high-load conditions relative to the low-load conditions in regions largely corresponding to a load-dependent VWM network that has been identified in previous experiments (Xu & Chun, 2006; Todd & Marois, 2004, 2005). These regions include the bilateral IPS, bilateral lateral occipital, and anterior cingulate. In addition, activation in these regions negatively correlated with bilateral TPJ, which is consistent with previous experiments indicating TPJ suppression in response to increasing VWM load (Todd et al., 2005).

Figure 3.

Brain scores and peak clusters for LV1. (A) Mean brain scores as a function of VWM load and object presentation condition. Brain scores reflect the activation summed across the entire brain and averaged across participants. Higher brain scores are observed in the high-load condition relative to the low-load condition, and scores are greater when two objects are presented relative to only a single object. Error bars denote 95% confidence intervals. (B) Mean brain scores observed in LV1 as a function of time (TRs). The diverging effects of VWM load and number of objects can be observed. (C) Peak cluster activations observed in LV1 extracted from TR2. BSRs are displayed. The z axis of the MNI coordinates of each slice is indicated. Regions corresponding to areas previously demonstrated as being sensitive to VWM load are identified.

Figure 3.

Brain scores and peak clusters for LV1. (A) Mean brain scores as a function of VWM load and object presentation condition. Brain scores reflect the activation summed across the entire brain and averaged across participants. Higher brain scores are observed in the high-load condition relative to the low-load condition, and scores are greater when two objects are presented relative to only a single object. Error bars denote 95% confidence intervals. (B) Mean brain scores observed in LV1 as a function of time (TRs). The diverging effects of VWM load and number of objects can be observed. (C) Peak cluster activations observed in LV1 extracted from TR2. BSRs are displayed. The z axis of the MNI coordinates of each slice is indicated. Regions corresponding to areas previously demonstrated as being sensitive to VWM load are identified.

Table 2.

Peak Regions Associated with LV1

Location
Hemisphere
X
Y
Z
BSR
CG 21 42 16.3033
IPL/sIPS 36 −48 57 10.6997
IFG 27 21 −9 9.2704
Cerebellum −45 −63 −21 8.8244
MFG −30 57 8.512
MFG −48 27 27 8.4214
Cuneus −27 −78 30 8.0535
MFG 30 51 7.8974
Cerebellum 39 −63 −30 7.534
IPL/sIPS −36 −45 42 6.8633
MOG −36 −90 6.8292
IFG −27 21 −3 6.8265
Cerebellum −6 −84 −21 6.1512
MOG 54 −63 −12 6.0603
SFG −12 12 63 5.8333
MFG 36 48 12 5.7963
IFG −45 27 5.7951
MFG 42 18 21 5.6433
MTG 45 −72 5.4094
Caudate 5.2558
SFG −12 −21 27 4.7761
SFG 21 21 60 4.7157

Insula 39 −18 −3 −10.15
MeFG L? 51 24 −8.3541
SMG/TPJ −48 −54 30 −7.638
ITL 42 −3 −21 −7.1153
AC −9 48 −7.1078
SMG/TPJ 63 −48 24 −6.6135
SFG 33 60 −6 −6.4677
MeFG −12 36 −5.6494
STG/TPJ 48 −60 33 −5.333
MTG −54 −21 −12 −4.9569
Location
Hemisphere
X
Y
Z
BSR
CG 21 42 16.3033
IPL/sIPS 36 −48 57 10.6997
IFG 27 21 −9 9.2704
Cerebellum −45 −63 −21 8.8244
MFG −30 57 8.512
MFG −48 27 27 8.4214
Cuneus −27 −78 30 8.0535
MFG 30 51 7.8974
Cerebellum 39 −63 −30 7.534
IPL/sIPS −36 −45 42 6.8633
MOG −36 −90 6.8292
IFG −27 21 −3 6.8265
Cerebellum −6 −84 −21 6.1512
MOG 54 −63 −12 6.0603
SFG −12 12 63 5.8333
MFG 36 48 12 5.7963
IFG −45 27 5.7951
MFG 42 18 21 5.6433
MTG 45 −72 5.4094
Caudate 5.2558
SFG −12 −21 27 4.7761
SFG 21 21 60 4.7157

Insula 39 −18 −3 −10.15
MeFG L? 51 24 −8.3541
SMG/TPJ −48 −54 30 −7.638
ITL 42 −3 −21 −7.1153
AC −9 48 −7.1078
SMG/TPJ 63 −48 24 −6.6135
SFG 33 60 −6 −6.4677
MeFG −12 36 −5.6494
STG/TPJ 48 −60 33 −5.333
MTG −54 −21 −12 −4.9569

Clusters reported at TR2 with a BSR > 3.5 and a minimum of 10 voxels. CG = cingulate gyrus; IFG = inferior frontal gyrus; IPL = inferior parietal lobule; ITL = inferior temporal gyrus; MeFG = medial frontal gyrus; MFG = middle frontal gyrus; MOG = middle occipital gyrus; MTG = middle temporal gyrus; SFG = superior frontal gyrus; SMG = supramarginal gyrus; STG = superior temporal gyrus.

LV1 also yielded an activation pattern differentiating the bilateral conditions from the left and right unilateral conditions, regardless of the memory load (see Figure 3A). That is, under both low and high VWM loads, activity in VWM regions including sIPS and LOC was greater when two bilateral objects were presented than when only a single item was presented. This finding suggests that these regions, which have previously been shown to be involved in the encoding and maintenance of information in VWM, are also sensitive to the number of objects presented in the periphery during the delay period of a change detection task.

The second significant LV accounted for 23% of the variance in the data, p = .004. The mean brain scores for LV2 are presented in Figure 4A and B, and the corresponding clusters of activity are displayed in Figure 4C as well as in Table 3. Similar to the first LV, LV2 reflected a pattern of brain activity related to both the memory load and object presentation conditions. As can be seen in Figure 4C, the regions corresponding to the pattern of activation in LV2 are predominantly located in regions of the ventral temporal lobe that are typically characterized as the LOC, specifically the anterior lateral occipital region and the posterior fusiform sulcus. In addition, activity in these regions was greater in the bilateral than in the unilateral conditions. Given that these regions are typically more active in response to intact objects than to images of scrambled objects (Grill-Spector et al., 2000), it is likely that the pattern of activations identified by LV2 relates to visual object processing, although it may be possible these regions are also involved in the maintenance of information in VWM. In contrast to LV1, however, LV2 yielded greater activation in ventral temporal regions in the low as opposed to the high VWM load condition, suggesting that these regions are in fact unrelated to VWM. Interestingly, activity in the bilateral sIPS was also identified by this LV, indicating that the IPS is involved in both delineated networks (see General Discussion).

Figure 4.

Brain scores and peak clusters for LV2. (A) Mean brain scores as a function of VWM load and object presentation condition. Brain scores reflect the activation summed across the entire brain and averaged across participants. Higher brain scores are observed in the low-load condition relative to the high-load condition, and scores are greater when two objects are presented relative to only a single object. Error bars denote 95% confidence intervals. (B) Mean brain scores observed in LV2 as a function of time (TRs). Overall, the brain scores peak later in the trial relative to the scores in LV1 (see Figure 3). (C) Peak cluster activations observed in LV2 extracted from TR4. BSRs are displayed. The z axis of the MNI coordinates of each slice is indicated. Regions identified in LV2 largely correspond to object-sensitive LOC, containing the posterior fusiform sulcus (pFs) and lateral occipital (LO) regions. In addition, a similar pattern of activation was also observed in sIPS.

Figure 4.

Brain scores and peak clusters for LV2. (A) Mean brain scores as a function of VWM load and object presentation condition. Brain scores reflect the activation summed across the entire brain and averaged across participants. Higher brain scores are observed in the low-load condition relative to the high-load condition, and scores are greater when two objects are presented relative to only a single object. Error bars denote 95% confidence intervals. (B) Mean brain scores observed in LV2 as a function of time (TRs). Overall, the brain scores peak later in the trial relative to the scores in LV1 (see Figure 3). (C) Peak cluster activations observed in LV2 extracted from TR4. BSRs are displayed. The z axis of the MNI coordinates of each slice is indicated. Regions identified in LV2 largely correspond to object-sensitive LOC, containing the posterior fusiform sulcus (pFs) and lateral occipital (LO) regions. In addition, a similar pattern of activation was also observed in sIPS.

Table 3.

Peak Regions Associated with LV2

Location
Hemisphere
X
Y
Z
BSR
Caudate −15 15 12.1115
IOG/LOC 42 −72 −6 11.993
SPL/sIPS −24 −69 45 9.6444
FG/LOC −36 −51 −18 8.5085
PC −9 −69 12 8.3519
SPL/sIPS 30 −57 39 8.0772
IFG 54 12 24 7.982
Thalamus −9 −12 7.1964
IFG −51 15 21 7.0499
LG 18 −63 −9 6.869
Thalamus −3 6.5674
Putamen 18 18 −6 6.4142
STG −57 12 −12 6.3124
FG/LOC 33 −69 −27 6.2894
MeFG −6 −6 57 6.144
SOG −21 42 −18 6.0465
MOG/LOC −33 −87 5.9271
MFG −42 −3 54 5.7991
IPL −45 −36 48 5.5332
Cerebellum 12 −81 −33 5.1401
FG/LOC 33 −39 −27 5.0863
AC 21 24 4.9499
MeFG 18 51 4.6541
Location
Hemisphere
X
Y
Z
BSR
Caudate −15 15 12.1115
IOG/LOC 42 −72 −6 11.993
SPL/sIPS −24 −69 45 9.6444
FG/LOC −36 −51 −18 8.5085
PC −9 −69 12 8.3519
SPL/sIPS 30 −57 39 8.0772
IFG 54 12 24 7.982
Thalamus −9 −12 7.1964
IFG −51 15 21 7.0499
LG 18 −63 −9 6.869
Thalamus −3 6.5674
Putamen 18 18 −6 6.4142
STG −57 12 −12 6.3124
FG/LOC 33 −69 −27 6.2894
MeFG −6 −6 57 6.144
SOG −21 42 −18 6.0465
MOG/LOC −33 −87 5.9271
MFG −42 −3 54 5.7991
IPL −45 −36 48 5.5332
Cerebellum 12 −81 −33 5.1401
FG/LOC 33 −39 −27 5.0863
AC 21 24 4.9499
MeFG 18 51 4.6541

Clusters reported at TR4 with a BSR > 3.5 and a minimum of 10 voxels. AC = anterior cingulate; FG = fusiform gyrus; IFG = inferior frontal gyrus; IOG = inferior occipital gyrus; IPL = inferior parietal lobule; LG = lingual gyrus; LOC = lateral occipital complex; MeFG = medial frontal gyrus; MFG = middle frontal gyrus; MOG = middle occipital gyrus; MTG = middle temporal gyrus; PC = posterior cingulate; SFG = superior frontal gyrus; SMG = supramarginal gyrus; SOG = superior orbital gyrus; SPL = superior parietal lobule; STG = superior temporal gyrus.

##### Regression analysis

Given the anatomical regions and brain scores corresponding to the two significant LVs identified by the task PLS analysis, object processing under a concurrent VWM load appears to depend on the activation of two orthogonal but partially overlapping networks of brain regions: The first group of regions includes areas that are sensitive to the encoding and maintenance of information in VWM (e.g., IPS, LOC, and TPJ), whereas the second set of brain regions involves areas that are sensitive to images of objects (primarily ventral temporal regions). The question remains whether activity in either of these networks corresponds to object recognition performance. That is, because Task-PLS is data driven (independent of performance outcomes), the brain scores represent only the covariance between the task conditions and the corresponding fMRI activation. If the patterns of activation extracted by the LVs actually reflect the changes in object recognition performance, then activation in these LVs (i.e., brain scores) should be directly tied to recognition performance. To examine this relationship, we performed a regression analysis to determine whether brain scores of either LV were predictive of object recognition performance. By regressing brain scores onto recognition accuracy, we can establish to what extent the covariance extracted by these LVs conveys meaningful information with respect to object recognition performance. Because LVs are orthogonal to each other, brain scores from each LV were entered in separate regression analyses.

Across conditions, the average brain scores of LV1 were a significant factor in predicting averaged accuracy, b = .93, t(4) = −4.94, p = .008. Furthermore, this relationship is also evident when predicting individual participant accuracy scores, across all conditions, b = −.5, t(58) = −4.43, p < .001. That is, individual brain scores in LV1 predict an individual's accuracy across all six conditions: As brain scores in LV1 increase (reflecting increases in fMRI activation in response to both memory load and object presentation), recognition accuracy decreases (Figure 5A). It is important to note that this negative relationship is precisely the opposite of what would be predicted if performance on the recognition task depended on VWM alone. That is, during VWM tasks, activation in areas that mediate VWM increases from low load to high load, whereas change detection performance remains relatively constant; decreases in performance occur only after capacity has been exceeded and memory-related activity asymptotes (Todd & Marois, 2004). In contrast, recognition performance in the present study decreases as activation of LV1 increases. Thus, this relationship suggests that, although recognition performance may be related to processes subserved by regions that support VWM, the recognition task does not reflect VWM per se.

Figure 5.

Relationship between brain scores and recognition performance. (A) Regression between brain scores in LV1 and object recognition accuracy. Increases in activation in regions associated with LV1 (see Figure 3) are accompanied by decreases in overall recognition accuracy. (B) Regression between brain scores in LV2 and the recognition scores. Recognition scores reflect the summed accuracy for left and right objects in the bilateral condition. As can be seen, all low-load conditions fall above the regression line, whereas all high-load conditions fall below it. Thus, increases in activation in regions extracted in LV2 are associated with a greater number of correctly recognized objects, even when absolute accuracy decreases. Legend: black diamond = low load, left; gray diamond = low load, right; gray circle = low load, bilateral; black triangle = high load, left; gray triangle = high load, right; gray square = high load, bilateral.

Figure 5.

Relationship between brain scores and recognition performance. (A) Regression between brain scores in LV1 and object recognition accuracy. Increases in activation in regions associated with LV1 (see Figure 3) are accompanied by decreases in overall recognition accuracy. (B) Regression between brain scores in LV2 and the recognition scores. Recognition scores reflect the summed accuracy for left and right objects in the bilateral condition. As can be seen, all low-load conditions fall above the regression line, whereas all high-load conditions fall below it. Thus, increases in activation in regions extracted in LV2 are associated with a greater number of correctly recognized objects, even when absolute accuracy decreases. Legend: black diamond = low load, left; gray diamond = low load, right; gray circle = low load, bilateral; black triangle = high load, left; gray triangle = high load, right; gray square = high load, bilateral.

In contrast to LV1, LV2 was not a significant predictor of either averaged recognition accuracy, b = −.08, p = .89, or of individual accuracy scores, b = −.06, p = .67. However, given that LV2 appears to correspond to activation in object recognition regions, it is possible that this activity is not linearly related to changes in absolute accuracy but is instead related to the amount of object information that is processed by these regions. To test for this possibility, we examined the summed accuracy for left and right objects in the bilateral conditions. That is, because two objects are presented in the bilateral condition, more objects may be processed by these regions, although overall accuracy is lower relative to the unilateral condition. The resulting scores reflect the processing of two objects in the bilateral conditions and are plotted against brain scores for LV2 in Figure 5B. As can been seen in this figure, the averaged brain scores of LV2 are predictive of the number of items correctly identified, b = .80, t(4) = 2.66, p = .056. This relationship is also significant when examining individual subject brain scores and performance across all conditions, b = .63, t(58) = 6.17, p < .001. That is, between the six conditions, activation in ventral temporal cortex increases with the number of objects correctly identified, with greater activity observed for bilateral conditions, as overall, more object information is present, although raw accuracy performance is impaired.

## GENERAL DISCUSSION

### VWM Load and Conscious Object Recognition

In the current experiments, we sought to test how perceptual information is processed under VWM load. The behavioral findings presented here extend previous findings (Todd et al., 2005) by demonstrating that under a high VWM load, even the processing of expected stimuli is impaired, but this impairment emerges only in higher levels of perceptual processing (i.e., in conscious recognition rather than in detection). The task-PLS fMRI results revealed two orthogonal but partially overlapping patterns of activation involved in the recognition of objects. Although one network of brain regions activated areas of extrastriate cortex that are typically associated with object recognition, activation in a larger network that included regions involved in the encoding and maintenance of information in VWM was also closely linked to recognition performance.

Our results demonstrate that under a concurrent VWM load, regions associated with capacity-limited VWM showed increases in response-to-object stimuli. Similarly, a recent finding demonstrated that activity in posterior parietal cortex is sensitive to perceptual load in addition to VWM load (Mitchell & Cusack, 2008). These findings are consistent with the idea that VWM and object perception share overlapping neural and cognitive resources. Specifically, it may be necessary to individuate and perceptually enhance object stimuli before they can be recognized or encoded into VWM (Xu & Chun, 2009) or to create an “object file” for those items (Pylyshyn, 1989; Kahneman & Treisman, 1984). A concurrent VWM load may, therefore, reduce the available resources for object individuation and enhancement, thereby affecting the ability to perform subsequent object recognition. Interestingly, our results demonstrate that, although both VWM (Todd & Marois, 2004) and perceptual load (Mitchell & Cusack, 2008) may have capacity limits in the posterior parietal cortex, these limitations may be somewhat independent, as activation in IPS continued to increase in response to object stimuli even when VWM capacity had been exceeded.

Importantly, although the two networks of regions isolated distinct brain regions, both networks also included the IPS. The activation of IPS in both networks suggests this region may play a particularly important role in the conscious recognition of objects. This adds to a growing body of literature indicating some amount of object selectivity in regions of the dorsal visual stream, in particular, the IPS (Konen & Kastner, 2008; Grill-Spector et al., 2000; Sereno & Maunsell, 1987). Unlike previous studies (e.g., Grill-Spector et al., 2000), however, our results here demonstrate that activation in this region correlates with object recognition performance. Although further studies will be required to address this question more directly, our results indicate that the role IPS plays in object recognition may depend on the perceptual or WM load demands of the task.

Overall, the PLS results revealed two distinct patterns of activation that were both correlated with different aspects of performance on the object recognition task. These results suggest that object recognition may not depend on activation of singular regions or even on the activation of a single network of regions (i.e., those in the ventral stream) but may instead depend on the interactions between multiple networks involved in different aspects of object processing. Further work is needed to precisely elucidate the precise relationship between these networks; however, the different patterns of activation observed in the two LVs may provide some insight: Although recognition accuracy decreases with increasing activation in LV1, increases in activation in LV2 correspond to increases in the number of objects correctly identified. One possible explanation for this relationship is that LV2 reflects the endpoint of conscious perception (the number of objects correctly identified), whereas LV1 represents activation of limited-capacity VWM and attentional resources. Under a low VWM load, object stimuli compete for limited resources, but enough resources remain for both objects to be processed, resulting in the high activation of object recognition regions and the successful recognition of both objects. Under a high VWM load, however, the increased demand on capacity-limited VWM resources results in limited perceptual processing of multiple object stimuli. As a consequence, activation in ventral object recognition areas decreases, and fewer objects are successfully recognized. According to this account, information processing in the IPS may act as the computational bottleneck limiting the amount of information that can be sent to higher perceptual areas. Thus, VWM may be part of a capacity-limited resource that is integral to conscious visual perception (Dell'Acqua, Sessa, Jolicoeur, & Robitaille, 2006). Protecting those contents from interference, therefore, may be a critical priority of the brain (Anticevic et al., 2010; Shulman et al., 2007). Thus, under high perceptual or WM load, deficits in perception may perhaps not be viewed as a shortcoming but rather an essential component of visual processing in its service of conscious visual perception.

### VWM Load and Visual Neglect

Recognition performance in the first experiment also demonstrated a significant hemifield asymmetry such that conscious object recognition was most impaired under the high-load condition for objects presented on the left side. This finding is consistent with the deficits observed in visual neglect (i.e., a deficit in reporting stimuli in the left visual field) and may have two potentially important implications. First, it elucidates the neural and cognitive mechanisms that underlie the deficits observed in both visual neglect and visual extinction. That is, the finding that recognition performance is closely associated with the deactivation of TPJ and the activation of regions associated with VWM load support the data suggesting that neglect is the result of damage to the right TPJ (Danckert & Ferber, 2006; Karnath et al., 2001) and that the impairments in visual awareness arise in part from impairments in VWM functioning (Ferber & Danckert, 2006; Wojciulik, Husain, Clarke, & Driver, 2001). Further support for this model comes from a recent study that demonstrates a contralateral bias for VWM information in IPS (Sheremata, Bettencourt, & Somers, 2010). Second, the results demonstrate that the “functional lesioning” of the right TPJ under a high VWM load may provide a working model for visual neglect in healthy populations, thereby enabling further tests of the mechanisms of this disorder. Similarly, a high VWM load may be a useful tool in the assessment and treatment of the extent of visual impairment poststroke.

To summarize, in the present study we sought to examine conscious object processing under a concurrent VWM load. Behaviorally, participants demonstrated impaired recognition performance under a high relative to a low VWM load. This was particularly true when multiple objects were presented and when objects were presented on the left side. Our fMRI data revealed that recognition performance depends on the activation of two networks of brain regions. In addition to the expected involvement of object-sensitive areas, the other network included brain regions involved in VWM encoding and maintenance. Thus, activation in capacity-limited VWM areas may play an important role in our ability to consciously perceive and identify visual objects.

## Acknowledgments

The authors thank Paul Lu and Carson Pun for assistance with data collection, Douglas Garret for statistical expertise, and Morgan Barense for helpful feedback. This work was supported by Natural Sciences and Engineering Research Council (grant 261203-08) and Canadian Institutes of Health Research (grant 79256) awarded to S. F.

Reprint requests should be sent to Stephen M. Emrich, Department of Psychology, University of Toronto, 100 St. George Street, Toronto, Ontario, M5S 3G3, Canada, or via e-mail: steve.emrich@utoronto.ca.

## REFERENCES

Anticevic
,
A.
,
Repovs
,
G.
,
Shulman
,
G. L.
, &
Barch
,
D. M.
(
2010
).
When less is more: TPJ and default network deactivation during encoding predicts working memory performance.
Neuroimage
,
49
,
2638
2648
.
Baars
,
B. J.
, &
Franklin
,
S.
(
2003
).
How conscious experience and working memory interact.
Trends in Cognitive Sciences
,
7
,
166
172
.
Corbetta
,
M.
,
,
J. M.
,
Ollinger
,
J. M.
,
McAvoy
,
M. P.
, &
Shulman
,
G. L.
(
2000
).
Voluntary orienting is dissociated from target detection in human posterior parietal cortex.
Nature Neuroscience
,
3
,
292
297
.
Cowan
,
N.
(
2001
).
The magical number 4 in short-term memory: A reconsideration of mental storage capacity.
Behavioral and Brain Sciences
,
24
,
87
114; discussion 114-185
.
Danckert
,
J.
, &
Ferber
,
S.
(
2006
).
Revisiting unilateral neglect.
Neuropsychologia
,
44
,
987
1006
.
Dell'Acqua
,
R.
,
Sessa
,
P.
,
Jolicoeur
,
P.
, &
Robitaille
,
N.
(
2006
).
Spatial attention freezes during the attention blink.
Psychophysiology
,
43
,
394
400
.
Downar
,
J.
,
Crawley
,
A. P.
,
Mikulis
,
D. J.
, &
Davis
,
K. D.
(
2002
).
A cortical network sensitive to stimulus salience in a neutral behavioral context across multiple sensory modalities.
Journal of Neurophysiology
,
87
,
615
620
.
Efron
,
B.
, &
Tibshirani
,
R.
(
1986
).
Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy.
Statistical Science
,
1
,
54
75
.
Ferber
,
S.
, &
Danckert
,
J.
(
2006
).
Lost in space—The fate of memory representations for non-neglected stimuli.
Neuropsychologia
,
44
,
320
325
.
Ferber
,
S.
,
Danckert
,
J.
,
Joanisse
,
M.
,
Goltz
,
H. C.
, &
Goodale
,
M. A.
(
2003
).
Eye movements tell only half the story.
Neurology
,
60
,
1826
1829
.
Friston
,
K. J.
,
Frith
,
C. D.
, &
Frackowiak
,
R. S. J.
(
1993
).
Principal component analysis learning algorithms: A neurobiological analysis.
Proceedings: Biological Sciences
,
254
,
47
54
.
Goebel
,
R.
,
Esposito
,
F.
, &
Formisano
,
E.
(
2006
).
Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: From single-subject to cortically aligned group general linear model analysis and self-organizing group independent component analysis.
Human Brain Mapping
,
27
,
392
401
.
Grill-Spector
,
K.
,
Kushnir
,
T.
,
Hendler
,
T.
, &
Malach
,
R.
(
2000
).
The dynamics of object-selective activation correlate with recognition performance in humans.
Nature Neuroscience
,
3
,
837
843
.
Heilman
,
K. M.
,
Watson
,
R. T.
, &
Valenstein
,
E.
(
1993
).
Neglect and related disorders.
In K. M. Heilman & E. Valenstein (Eds.),
Clinical neuropsychology
(3rd ed., pp.
279
336
).
New York
:
Oxford University Press
.
Hollingworth
,
A.
, &
Rasmussen
,
I. P.
(
2010
).
Binding objects to locations: The relationship between object files and visual working memory.
Journal of Experimental Psychology: Human Perception and Performance
,
36
,
543
564
.
Kahneman
,
D.
, &
Treisman
,
A.
(
1984
).
Changing views of attention and automaticity.
In R. Parasuramam & D. R. Davies (Eds.),
Varieties of attention
(pp.
29
62
).
New York
:
.
Kahneman
,
D.
,
Treisman
,
A.
, &
Gibbs
,
B. J.
(
1992
).
The reviewing of object files: Object-specific integration of information.
Cognitive Psychology
,
24
,
175
219
.
Karnath
,
H. O.
(
1988
).
Deficits of attention in acute and recovered visual hemi-neglect.
Neuropsychologia
,
26
,
27
43
.
Karnath
,
H. O.
,
Ferber
,
S.
, &
Himmelbach
,
M.
(
2001
).
Spatial awareness is a function of the temporal not the posterior parietal lobe.
Nature
,
411
,
950
953
.
Konen
,
C. S.
, &
Kastner
,
S.
(
2008
).
Two hierarchically organized neural systems for object information in human visual cortex.
Nature Neuroscience
,
11
,
224
231
.
Luck
,
S. J.
, &
Vogel
,
E. K.
(
1997
).
The capacity of visual working memory for features and conjunctions.
Nature
,
390
,
279
281
.
McIntosh
,
A. R.
,
Bookstein
,
F. L.
,
Haxby
,
J. V.
, &
,
C. L.
(
1996
).
Spatial pattern analysis of functional brain images using partial least squares.
Neuroimage
,
3
,
143
157
.
McIntosh
,
A. R.
,
Chau
,
W. K.
, &
Protzner
,
A. B.
(
2004
).
Spatio-temporal analysis of event-related fMRI data using partial least squares.
Neuroimage
,
23
,
764
775
.
Mitchell
,
D. J.
, &
Cusack
,
R.
(
2008
).
Flexible, capacity-limited activity of posterior parietal cortex in perceptual as well as visual short-term memory tasks.
Cerebral Cortex
,
18
,
1788
1798
.
Mort
,
D. J.
,
Malhotra
,
P.
,
Mannan
,
S. K.
,
Rorden
,
C.
,
Pambakian
,
A.
,
Kennard
,
C.
,
et al
(
2003
).
The anatomy of visual neglect.
Brain
,
126
,
1986
1997
.
Pashler
,
H.
(
1988
).
Familiarity and visual change detection.
Perception & Psychophysics
,
44
,
369
378
.
Pylyshyn
,
Z.
(
1989
).
The role of location indexes in spatial perception: A sketch of the FINST spatial-index model.
Cognition
,
32
,
65
97
.
Robitaille
,
N.
,
Grimault
,
S.
, &
Jolicoeur
,
P.
(
2009
).
Bilateral parietal and contralateral responses during maintenance of unilaterally encoded objects in visual short-term memory: Evidence from magnetoencephalography.
Psychophysiology
,
46
,
1090
1099
.
Serences
,
J. T.
,
Shomstein
,
S.
,
Leber
,
A. B.
,
Golay
,
X.
,
Egeth
,
H. E.
, &
Yantis
,
S.
(
2005
).
Coordination of voluntary and stimulus-driven attentional control in human cortex.
Psychological Science
,
16
,
114
122
.
Sereno
,
A. B.
, &
Maunsell
,
J. H. R.
(
1987
).
Shape selectivity in primate lateral intraparietal cortex.
Journal of Experimental Psychology: Human Perception and Performance
,
12
,
388
391
.
Sheremata
,
S. L.
,
Bettencourt
,
K. C.
, &
Somers
,
D. C.
(
2010
).
Hemispheric asymmetry in visuotopic posterior parietal cortex emerges with visual short-term memory load.
Journal of Neuroscience
,
30
,
12581
12588
.
Shulman
,
G. L.
,
Astafiev
,
S. V.
,
McAvoy
,
M. P.
,
d'Avossa
,
G.
, &
Corbetta
,
M.
(
2007
).
Right TPJ deactivation during visual search: Functional significance and support for a filter hypothesis.
Cerebral Cortex
,
17
,
2625
2633
.
Todd
,
J. J.
,
Fougnie
,
D.
, &
Marois
,
R.
(
2005
).
Visual short-term memory load suppresses temporo-parietal junction activity and induces inattentional blindness.
Psychological Science
,
16
,
965
972
.
Todd
,
J. J.
, &
Marois
,
R.
(
2004
).
Capacity limit of visual short-term memory in human posterior parietal cortex.
Nature
,
428
,
751
754
.
Todd
,
J. J.
, &
Marois
,
R.
(
2005
).
Posterior parietal cortex activity predicts individual differences in visual short-term memory capacity.
Cognitive, Affective, and Behavioral Neuroscience
,
5
,
144
155
.
Wojciulik
,
E.
,
Husain
,
M.
,
Clarke
,
K.
, &
Driver
,
J.
(
2001
).
Spatial working memory deficit in unilateral neglect.
Neuropsychologia
,
39
,
390
396
.
Xu
,
Y.
, &
Chun
,
M. M.
(
2006
).
Dissociable neural mechanisms supporting visual short-term memory for objects.
Nature
,
440
,
91
95
.
Xu
,
Y.
, &
Chun
,
M. M.
(
2007
).
Visual grouping in human parietal cortex.
Proceedings of the National Academy of Sciences, U.S.A.
,
104
,
18766
18771
.
Xu
,
Y.
, &
Chun
,
M. M.
(
2009
).
Selecting and perceiving multiple visual objects.
Trends in Cognitive Sciences
,
13
,
167
174
.