Abstract

Visual working memory (VWM) is a capacity-limited cognitive resource that plays an important role in complex cognitive behaviors. Recent studies indicate that regions subserving VWM may play a role in the perception and recognition of visual objects, suggesting that conscious object perception may depend on the same cognitive and neural architecture that supports the maintenance of visual object information. In the present study, we examined this question by testing object processing under a concurrent VWM load. Under a high VWM load, recognition was impaired for objects presented in the left visual field, in particular when two objects were presented simultaneously. Multivariate fMRI revealed that two independent but partially overlapping networks of brain regions contribute to object recognition. The first network consisted of regions involved in VWM encoding and maintenance. Importantly, these regions were also sensitive to object load. The second network comprised regions of the ventral temporal lobes traditionally associated with object recognition. Importantly, activation in both networks predicted object recognition performance. These results indicate that information processing in regions that mediate VWM may be critical to conscious visual perception. Moreover, the observation of a hemifield asymmetry in object recognition performance has important theoretical and clinical significance for the study of visual neglect.

INTRODUCTION

Although our subjective experience of the visual world seems to be rich and detailed, in actuality our internal representations are quite sparse. A striking demonstration of our visual system's processing limitations comes from the change detection task, in which observers first see a multi-item array and are asked to maintain as much information as possible. The display is removed from view, and after a brief delay, a probe array appears to which observers report whether a change has occurred. Accordingly, observers have to hold information “on-line” after it has been removed from view, a cognitive faculty called visual working memory (VWM). Observers typically perform well for arrays containing four items or less, but performance declines significantly beyond that limit (Cowan, 2001; Luck & Vogel, 1997).

The neural substrate and electrophysiological signature of this processing bottleneck have been pinpointed to the posterior parietal cortex (in particular, the intraparietal sulcus, IPS) and lateral occipital areas, as activity in these regions increases concomitantly with an increase in working memory (WM) load and plateaus at a given subject's capacity (Robitaille, Grimault, & Jolicoeur, 2009; Xu & Chun, 2006; Todd & Marois, 2004, 2005). Although much has been discovered about the mechanisms of this capacity limit, it is unclear how VWM capacity relates to other aspects of conscious visual experience such as object recognition. It has been argued that for an object percept to reach conscious awareness, it must be maintained in a capacity-limited global cognitive workspace as an episodic representation called an object file (Kahneman, Treisman, & Gibbs, 1992; Kahneman & Treisman, 1984; see also Baars & Franklin, 2003). Object files contain basic information about an object's spatial and temporal coordinates and high-level information about its features such as shape, color, and texture. Recently, Hollingworth and Rasmussen (2010) demonstrated that VWM shares some properties with object files. Thus, interfering with VWM processes may affect the ability to create and sustain object files, thereby limiting their access to conscious awareness. Similarly, Xu and Chun (2009) have proposed a two-stage object file theory. In the first stage, objects are individuated, meaning that coarsely defined candidate objects are selected for representation based on their spatial locations (see also Pylyshyn, 1989). The neural substrate of this selection process has been pinpointed to the inferior IPS, in which activation levels are modulated by up to four separate spatial locations, but not by object complexity (Xu & Chun, 2006, 2007). In the second stage, these selected sparse representations are enhanced with more detailed information to allow for object identification. This process seems to be subserved by the superior IPS (sIPS) and the lateral occipital complex (LOC), an area associated with conscious object recognition (Grill-Spector, Kushnir, Hendler, & Malach, 2000). Furthermore, a recent study has demonstrated that capacity-limited areas subserving VWM are also sensitive to perceptual load (Mitchell & Cusack, 2008). Thus, VWM and object recognition are not only mediated by partially overlapping neural resources but the mechanisms defining VWM capacity limits may be related to limits in perceptual processes (i.e., individuation and representation).

Given the overlap in cognitive and neural mechanisms underlying VWM and object recognition, in addition to the presumed role of WM in providing a workspace for consciousness, it seems plausible that the processing limitations underlying our conscious experience of objects in our environment may be determined by activation levels in dorsal and ventral brain areas mediating object-based VWM, namely, the sIPS and the LOC. Consequently, the goal of the present study was to test the neural and cognitive processing limits when the representational capacity limit of VWM has been exceeded. In other words, if conscious object processing shares resources with capacity-limited VWM, are we blind to objects once neural and cognitive resources related to VWM capacity have been depleted? It has been shown that a high VWM load is associated with impaired detection of unexpectedly appearing images of objects (Todd, Fougnie, & Marois, 2005). This inattentional blindness is possibly related to the suppression of activation in the right TPJ that is observed under a high VWM load (Todd et al., 2005). The TPJ, which includes regions of the inferior parietal lobule and superior temporal gyrus, subserves stimulus-driven attentional processes to detect salient and behaviorally relevant stimuli in the environment (Serences et al., 2005; Downar, Crawley, Mikulis, & Davis, 2002; Corbetta, Kincade, Ollinger, McAvoy, & Shulman, 2000). Thus, because only a limited number of higher-order perceptual representations can be formed and maintained, it may be necessary to suppress the TPJ system to shield the contents of conscious awareness from task-irrelevant distraction (Anticevic, Repovs, Shulman, & Barch, 2010; Shulman, Astafiev, McAvoy, d'Avossa, & Corbetta, 2007).

Interestingly, unilateral damage to the (typically right) TPJ leaves neurological patients with profound impairments in conscious awareness for visual stimuli in the contralateral visual field (Danckert & Ferber, 2006; Mort et al., 2003; Karnath, Ferber, & Himmelbach, 2001), a deficit referred to as visual neglect. Thus, although the visual pathways and object-sensitive areas remain intact, damage to regions including and surrounding the right TPJ severely diminishes the patients' abilities to report, respond to, or orient toward stimuli located toward the contralesional field (Heilman, Watson, & Valenstein, 1993). Even if the symptoms of neglect improve as indexed by increased detection of stimuli presented on the left side, most of these patients continue having impaired explicit awareness for a contralesional stimulus when two stimuli compete for processing resources (Ferber, Danckert, Joanisse, Goltz, & Goodale, 2003; Karnath, 1988).

In the current study, we sought to test how perceptual information is processed under a high VWM load. Specifically, we predicted that a high VWM load would deplete resources in regions mediating VWM (e.g., sIPS) as well as functionally deactivate the TPJ and, as such, would lead to impairments in conscious awareness for subsequently presented objects. In Experiment 1, we tested whether a high VWM load would affect the conscious report of visual objects, and if so, whether these processing deficits would demonstrate a hemifield asymmetry, mimicking those observed in visual neglect following lesions to the right TPJ. In Experiment 2, we used fMRI to examine the neural substrate of the cognitive interplay between VWM load and conscious object recognition. Importantly, we used a multivariate analysis to identify networks of brain regions that are jointly affected by VWM load and object processing, in addition to an ROI approach to isolate localized effects of load on independently defined areas involved in VWM.

EXPERIMENT 1

Methods

Participants

A total of 32 right-handed, healthy young adults (24 women) ages 18–46 years (M = 22.4) from the University of Toronto participated in the experiment for partial course credit or monetary compensation ($10 CAD/hr). All procedures were approved by the University of Toronto Research Ethics Board.

Stimuli and Procedure

To manipulate VWM load, a change detection task similar to the one used by Todd and Marois (2004) was employed. In addition, participants had to perform a concurrent object detection/recognition task (Figure 1). Each trial began with a centrally presented black dot presented on a gray (Red, Green, Blue [RGB] = 128,128,128) background, and participants were told to maintain fixation throughout the task. Next, the memory sample, consisting of either one (low load) or three (high load) uniquely colored discs, was presented for 166 msec. Participants were asked to remember as many of the colored discs as possible over a brief delay. The disc colors were randomly selected from one of seven alternatives: red (RGB = 255,0,0), green (0,255,0), dark green (0,128,0), blue (0,0,255), white (255,255,255), black (0,0,0), cyan (0,255,255), magenta, (255,0,255), yellow (255,255,0), and light yellow (255,255,128). The discs subtended 0.38° × 0.38° of visual angle, separated by a minimum of 0.24°, and disc locations were randomly assigned from positions of a 9 × 9 grid centered around fixation. The memory array was followed by a central fixation point for 600 msec, at which point zero, one, or two simple objects were presented for 66 msec 10° to the left and/or right of the central fixation. Each object subtended a maximum of 1° × 1° of visual angle. Objects were uniquely and randomly selected from a pool of 11 items chosen from Wingdings and Webdings fonts (

graphic
,
graphic
,
graphic
,
graphic
,
graphic
,
graphic
,
graphic
,
graphic
,
graphic
,
graphic
,
graphic
). After an additional 550 msec delay (1217 msec total), participants were prompted to make three responses. First, participants were presented with a memory probe. A single-colored disc was presented for 1766 msec. Participants were told to indicate with a right-hand key press whether this disc matched the color presented in the same location in the initial memory sample. Half of the trials were match trials, and the other half were nonmatch trials. On half of the nonmatch trials, a new color (not in the original memory sample) was presented, whereas on the other half, one of the previously presented colors was presented in a new location. After the memory probe, participants were probed how many objects were presented in the periphery during the delay period of the memory task. This object detection probe lasted 2517 msec, during which participants could indicate whether zero, one, or two objects were presented using the index, middle, and fourth fingers, respectively. Following the object detection probe, participants were presented with an object recognition probe lasting 4516 msec. Two objects were presented, and participants could indicate whether one, both, or none of these objects had been presented during the delay period of the memory task. Participants responded with their index or middle fingers for the objects, in order, and could press both keys if both sample items had been presented. The fourth finger was used to indicate if neither of the probe objects had been presented. When only one object was presented during the delay period (unilateral left or unilateral right), half of the trials were match, and the other half were nonmatch. For bilateral trials (objects presented on the left and right), one quarter of the trials were nonmatch trials; match trials were equally divided between probing left-only, right-only, or both objects. For all conditions, nonmatch objects were assigned randomly from the remaining pool of objects, and the locations of the matching items in the probe display was counterbalanced, regardless of presentation side in the experimental trial. Each trial concluded with an intertrial interval (ITI) of 2400 msec. Participants were seated 57 cm away from a 21-in. CRT monitor, which presented the stimuli with a 60-Hz refresh rate.

Figure 1. 

Schematic of experimental trials. The trial begins with a fixation period, immediately followed by the memory sample of the change detection task. The memory load was manipulated by presenting either one (low load) or three (high load) memory items. Participants are told to remember as many of the colored items over the 1216-msec delay period as possible. Half-way through the delay period, participants were presented with one object (unilaterally on the left or right), two objects (bilaterally on the left and right), or no objects. Following the remaining delay, participants were given a memory probe, in which they indicated whether the probe item was a different color from the original memory sample (50% of trials). Immediately following the memory probe, object detection (Experiment 1) and object recognition (Experiments 1 and 2) probes were given.

Figure 1. 

Schematic of experimental trials. The trial begins with a fixation period, immediately followed by the memory sample of the change detection task. The memory load was manipulated by presenting either one (low load) or three (high load) memory items. Participants are told to remember as many of the colored items over the 1216-msec delay period as possible. Half-way through the delay period, participants were presented with one object (unilaterally on the left or right), two objects (bilaterally on the left and right), or no objects. Following the remaining delay, participants were given a memory probe, in which they indicated whether the probe item was a different color from the original memory sample (50% of trials). Immediately following the memory probe, object detection (Experiment 1) and object recognition (Experiments 1 and 2) probes were given.

Object presentation conditions were divided equally between low and high VWM loads. The factors of VWM load (low and high), object location (none, left, right, and both), and match condition (unilateral: nonmatch, match; bilateral: nonmatch, match-left, match-right, match-both) were fully crossed, with 12 trials in each cell, for a total of 216 trials. Trials were presented in a random sequence, organized in eight blocks of 27 trials each, with self-timed breaks between blocks. Before the beginning of the experiment, participants were given between 10 and 50 trials to familiarize themselves with the task and the responses. The total experiment lasted about 1 hr. Participants were instructed that their primary task was the change detection task and that they should perform as best as they could on this task. Data from six subjects were removed from analysis because of inattentiveness or for failing to understand the task instructions.

Analysis

Change detection performance was measured as a function of memory capacity (K) according to the formula developed by Pashler (1988) and modified by Cowan (2001), where accuracy is scaled by the memory load set size: K = set size × (hits + correct rejections − 1).

Results and Discussion

In the first experiment, we examined whether conscious object processing is impaired when objects are presented under a high VWM load. That is, we predicted that, because regions that mediate VWM are sensitive to perceptual load (Mitchell & Cusack, 2008) and overlap with those involved in establishing “object files” (Xu & Chun, 2009), object processing would be impaired under a high VWM load but that these deficits would be most pronounced at the level of conscious recognition, with sensory-driven processes remaining relatively intact. In other words, the detection of sudden onsets in the periphery can proceed regardless of VWM manipulations; the higher-order process of object recognition, however, is crucially affected by VWM load. Furthermore, these deficits would be greatest for stimuli in the left visual field, as the TPJ suppression that occurs under high VWM load would produce lateralized deficits in conscious awareness that parallel those of visual neglect. Finally, if object perception and VWM share common capacity-limited processes, then recognition should be worse when multiple objects are presented, as each of the objects will compete for limited resources.

Change Detection Task

Examining memory performance by calculating memory capacity for each condition (K), participants remembered significantly more change detection items in the high-load (2.1) relative to the low-load (0.9) condition, t(22) = −18.5, p < .001. Although performance in the high-load condition is relatively low compared with previous studies using a similar design (Todd & Marois, 2004), likely as a result of the dual-task performance, the results confirm that more items were encoded and stored in VWM in the high-load condition. Importantly, no differences in memory performance were observed when two objects were presented relative to when only a single object was presented, ts < 1.1, ps > .32, indicating that participants were able to prioritize the change detection task and that VWM performance was unaffected by the demands of the recognition task.

Object Detection Performance

Following the memory probe, participants were asked to indicate how many object stimuli were presented in the periphery during the delay period of the change detection task. Participants correctly reported the number of items on 96% of all trials (Figure 2A). A 2 (VWM Load: low vs. high) × 4 (Object Presentation: none, unilateral-left, unilateral-right, bilateral) repeated measures ANOVA revealed that accuracy (% correct − % incorrect) was greater in the low-load condition than in the high-load condition, F(1, 22) = 11.6, p = .003. Neither the main effect of object presentation condition nor the interaction were significant, F(3, 66) = .7, p = .56, and F(3, 66) = 0.8, p = .53, respectively. Although detection performance decreased under a high memory load, the decrease in performance likely reflected an increase in errors rather than a decrease in detection, as indicated by a trend in the increase of false alarms in the high-load condition when no objects were presented, t(25) = −1.8, p = .082. Thus, by and large, participants were able to detect the presence of unilateral and bilateral objects presented in the periphery during the delay period of a VWM task. Importantly, performance was unaffected by the objects' locations (left or right) or the number of objects (unilateral or bilateral).

Figure 2. 

Experiment 1: Detection and recognition performance. (A) The proportion of correct responses (hits–false alarms) to the object detection probe as a function of memory load and object presentation condition. Object detection performance is largely unaffected by memory load or object condition. (B) The proportion of correctly identified objects in the object detection probe as a function of memory load and object presentation condition. Under a high VWM load, recognition performance is particularly impaired for objects presented in the left visual field. Error bars denote SEMs.

Figure 2. 

Experiment 1: Detection and recognition performance. (A) The proportion of correct responses (hits–false alarms) to the object detection probe as a function of memory load and object presentation condition. Object detection performance is largely unaffected by memory load or object condition. (B) The proportion of correctly identified objects in the object detection probe as a function of memory load and object presentation condition. Under a high VWM load, recognition performance is particularly impaired for objects presented in the left visual field. Error bars denote SEMs.

Object Recognition Performance

Following the object detection probe, participants were provided with two object recognition probes and were told to indicate whether either or both had been presented during the delay period. Participants correctly identified the object or objects presented (without incorrectly identifying one of the lure objects) on 65% of trials (Figure 2B). Because object recognition performance could be assessed separately for left and right items, a full 2 (VWM Load: low vs. high) × 2 (Object Presentation: unilateral vs. bilateral) × 2 (Object Location: left vs. right) repeated measures ANOVA was performed to compare the effects of presentation location, and the number of objects, as a factor of memory load. As with the object detection performance, participants identified significantly fewer objects under a high VWM load, F(1, 22) = 28.9, p < .001. Furthermore, participants correctly identified significantly fewer objects under bilateral presentation, relative to unilateral, F(1, 22) = 105.4, p < .001. Importantly, the effect of Object Location was also significant, F(1, 22) = 4.4, p = .048, with significantly worse performance for objects on the left than on the right side. The interaction between Load and Location (left vs. right) was marginally significant, F(1, 22) = 3.2, p = .088. Collapsing across presentation conditions (unilateral/bilateral), paired t tests revealed that recognition performance was impaired in the left hemifield under a high VWM load, t(22) = −2.87, p = .009, but not under a low VWM load, t(22) = −0.19, p = .85. None of the other interactions were significant, Fs < 0.5, ps > .5.

Overall, our results show that, although participants were largely able to detect the presence of object stimuli presented in the periphery during the delay period of a change detection task, performance on the object recognition task was significantly worse under a high VWM load relative to a low VWM load. Furthermore, object recognition performance was also worse under bilateral conditions, relative to when only a single object was presented. These results are consistent with the findings that regions related to VWM and object processing share overlapping neural and cognitive resources (Xu & Chun, 2006, 2007, 2009; Mitchell & Cusack, 2008). That is, when VWM capacity has been exceeded, the amount of information that can be subsequently processed by these regions is limited, resulting in impairments in conscious object processing.

In addition, the results demonstrated a hemifield asymmetry in object recognition performance that bears a striking resemblance to the impairments observed in visual neglect: When two stimuli are presented bilaterally and compete for processing resources, patients will show impaired awareness for the contralesional stimulus (Karnath, 1988). Here we found in healthy participants that under a high VWM load, recognition performance was lowest for objects presented in the left visual field, particularly under bilateral presentation. Overall, the distinction between spared detection and impaired recognition, in particular on the left side of the display, fits nicely with findings in the neglect literature that, although these patients are unable to report consciously on information presented on the left side, information nevertheless affects their behavior (Ferber et al., 2003).

EXPERIMENT 2

The results of the behavioral experiment are consistent with the idea that overlapping neural resources between object recognition and VWM may result in object processing deficits under a high VWM load. Although the observed lateralized asymmetry is likely related to the functional inactivation of the right TPJ (Todd et al., 2005), it is unclear from these results whether the general processing deficits are related to overlapping cognitive and neural mechanisms between VWM and object recognition or whether these deficits are a result of general dual-task interference effects that are unrelated to VWM capacity limitations. If object recognition performance is related to VWM load-specific effects, a number of predictions can be made: First, regions subserving capacity-limited VWM (e.g., sIPS) should demonstrate increasing activation in response to the object stimuli, in addition to the VWM load. It is unclear, however, whether this activity should plateau when VWM capacity is reached or whether object processing can proceed independent of memory capacity limits. Second, if object recognition performance depends in part on activation in VWM regions, then conscious recognition performance should vary as a function of activation in VWM regions as well as ventral stream object recognition areas. Consequently, we adapted the behavioral task for use with fMRI to examine the cortical mechanisms underlying the observed load-dependent impairments in conscious report. Specifically, we used the multivariate technique partial least squares (PLS), which can identify whether certain brain areas (i.e., IPS and lateral occipital regions) form a common network in which activation covaries across VWM load and object presentation conditions (McIntosh, Chau, & Protzner, 2004; McIntosh, Bookstein, Haxby, & Grady, 1996). We then compared these patterns of activation with recognition performance to determine the extent to which activation in these networks contribute to recognition performance under VWM load. We further used independent ROI analyses to examine effects in specific VWM regions.

Methods

Participants

Twelve young adults from the University of Toronto community participated in the experiment. Data from two subjects were excluded because of excessive motion or inattentiveness during the scan resulting in a total of 10 participants (8 women) aged 18–25 years (M = 22.3). All participants were right-handed and had no history of neurological impairments. Participants received $50–60 remuneration for participation. All procedures were approved by the University of Toronto Research Ethics Board and the Ethics Research Board of Baycrest.

Stimuli and Procedure

Stimuli were back-projected onto a 27 × 36 cm projection screen, viewed at a distance of 117 cm through a mirror located in the scanner. Stimuli were delivered at a refresh rate of 50 Hz. Participants used a four-button response box placed on their abdomen to make all responses. Participants were familiarized with the tasks and responses several days before the scan, as well as immediately before scanning. In addition to the experimental task, participants performed a VWM localizer task.

Experimental task

Stimuli and procedure were similar to those of Experiment 1 (Figure 1); however, no object detection probe was presented. The timing of the stimuli was modified for the scanner: Each trial began with a fixation point for 503 msec, followed by the memory sample presented for 151 msec. After a 603-msec delay, one or two objects were presented for 67 msec 10° to the left or right of the fixation. After another 536-msec delay, the memory probe was presented for 1558 msec, followed by the object recognition probe, lasting 3202 msec. Each trial was followed by a 1380–7380 msec ITI.

Factors of load (low, high) and presentation condition (unilateral left, unilateral right, bilateral) were fully crossed for six different trial types. Participants performed 42 trials of each of the six conditions, divided into seven runs of 36 trials (six of each condition). Null (fixation) periods were also presented between trials as part of the ITI. Trial stimuli and procedures were otherwise identical to that of Experiment 1. Trial and null-period sequence was assigned using optseq2 (surfer.nmr.mgh.harvard.edu/optseq/), and participants were assigned a random sequence of runs.

VWM localizer

To compare PLS activations to known VWM regions, we also performed a change detection localizer task immediately following the experimental runs. The localizer task consisted of a change detection task, similar to the one used in the experimental task, with no intervening object presentation. The delay period lasted 1200 msec, and the memory probe was presented for 1709 msec. Set sizes of 1, 3, and 5 were used. Participants performed 24 trials of each set size, in two runs of 36 trials. The VWM localizer was analyzed using K weighted regressors (Todd & Marois, 2004).

Image Acquisition and Data Preprocessing

Images were acquired on a 3-T Siemens Magnetom Trio whole-body scanner with a matrix 12-channel head coil. Anatomical images were acquired before the functional images using a MP-RAGE sequence (repetition time [TR] = 2 sec, echo time [TE] = 2.63 sec, 160 oblique axial slices, with a 1 mm3 voxel size, field of view [FOV] = 25.6 cm, acquisition matrix = 256 × 256). Functional volumes were obtained using a whole head T2*-weighted EPI sequence (TR = 2 sec, TE = 30 msec, flip angle = 70°, 30 oblique axial slices with interleaved acquisition, 3.125 × 3.125 × 5 mm voxel resolution, field of view [FOV] = 20 cm, acquisition matrix = 64 × 64). The first four volumes of each run were discarded to allow the magnetization to reach steady state.

For PLS analysis, image preprocessing was performed with SPM5 software. Slice timing was corrected to the first slice, and motion correction using a 3-D Fourier transform interpolation using a functional volume minimizing motion to less than 2 mm was performed. Participants' images were then spatially normalized to Montreal Neurological Institute (MNI) space and smoothed using a 6-mm FWHM Gaussian filter using SPM5, resulting in a voxel size of 4 × 4 × 4 mm.

Multivariate group analyses were carried out using the PLS software package (McIntosh et al., 1996). Analyses of localizer scans for ROI analyses were performed using BrainVoyager QX (Goebel, Esposito, & Formisano, 2006). Slice scan time correction, 3-D motion correction, and temporal filtering of images were performed. Functional runs were aligned to the anatomical scan using automatic coregistration and corrected by visual inspection. An ROI for the sIPS was obtained for nine of the participants by applying K weighted regressors to the average of two change detection localizer scans and identifying the peak region of activation that was within IPS. These coordinates were selected to match most closely to the Talairach coordinates reported by Xu and Chun (2006). The mean Talairach coordinates for these ROIs were (right/left) 26/−25, −63/−64, and 44/44.

PLS

PLS analysis (McIntosh et al., 1996, 2004) is a multivariate statistical tool that is used to identify regional activity change as a function of task demands. In other words, PLS determines those brain regions whose activity covaries with the experimental conditions. This approach is akin to a PCA (Friston, Frith, & Frackowiak, 1993) operating under the assumption that brain function relates to the activity of a number of brain regions rather than a single region. This approach is data driven, and all task conditions can be entered into the analysis simultaneously, yielding whole-brain patterns of activity that covary with the experimental design. The output of PLS analysis is a set of latent variables (LVs), which are components that reflect patterns of brain activity related to task conditions.

Analysis was performed on all trials from all six conditions of load (low and high) crossed with object presentation side (left, right, and bilateral). Activity was analyzed for an epoch of six TRs normalized to the first TR in the trial, defined here as the fixation period preceding the memory sample. For each TR, a “brain score” was calculated for each participant to index how strongly the participant contributes to the pattern of brain activity identified for that TR.

Significance of each LV was determined by permutation testing (McIntosh et al., 1996) using 500 permutations. The reliability of the saliences (or weights) for the brain voxels that characterize each pattern identified by the LVs was determined by an independent bootstrap estimation of the standard errors (Efron & Tibshirani, 1986). Peak voxels with a bootstrap ratio (BSR) greater than 3.5 were considered reliable, which is approximately equivalent to p < .005. Clusters containing at least 10 reliable voxels were extracted, and a local maximum for each cluster was defined as the voxel with a BSR higher than any other voxel in a 20-mm cube centered on that voxel. Although most regions showed reliable activation across multiple time points, for the purposes of clarity, results are reported from BSRs for the second (LV1) and fourth (LV2) TRs. Locations of the maxima are reported in the stereotaxic coordinates of MNI space.

Results and Discussion

The behavioral task from Experiment 1 (Figure 1) was adopted for use with fMRI (see Methods). Importantly, because object detection performance in Experiment 1 was close to ceiling, the detection task was eliminated, and the object recognition probe immediately followed the change detection probe.

Behavioral Results

Change detection task

In the VWM task, participants correctly recalled significantly more items in the high-load (2.38) than in the low-load (0.96) condition, t(9) = −15.7, p < .001. The average maximum K estimate obtained in loads 3 and 5 during the localizer task was 3.0 items, significantly greater than the 2.38 items obtained in the high-load condition, t(9) = −4, p = .003, indicating that the high-load change detection task may not have completely depleted memory resources. Specifically, the secondary recognition task may have had a modest effect on change detection performance, as the average K estimate in the high-load condition was moderately reduced relative to the K estimate obtained from the same set size of the independent VWM localizer (2.67), t(9) = −2.1, p = .07.

Object recognition task

Participants correctly identified the object or objects presented (without incorrectly identifying one of the lure objects) on 84% of trials (Table 1). When compared with Experiment 1, overall recognition performance was substantially higher, indicating that either the detection probe or the presence of catch trials in Experiment 1 may have potentially affected recognition performance to some extent. As in Experiment 1, however, a repeated measures ANOVA revealed that participants identified significantly fewer items under a high VWM load, F(1, 9) = 19.7, p = .002. Furthermore, participants also correctly identified significantly fewer items under bilateral presentation, relative to unilateral presentation, F(1, 9) = 26.2, p = .001. Unlike in Experiment 1, however, there was no significant effect of Presentation Side, F(1, 9) = 0.12, p = .74. Importantly, under bilateral presentation, recognition performance tended to be worse for objects presented on the left as revealed by a trend between presentation condition and object location, F(1, 9) = 3, p = .12, indicating that recognition performance demonstrated a similar pattern to Experiment 1. None of the other interactions were significant, Fs < .7, ps > .4.

Table 1. 

Recognition Accuracy in Experiment 2


Low Load
High Load
Unilateral
Bilateral
Unilateral
Bilateral
Left
Right
Left
Right
Left
Right
Left
Right
Recognition accuracy (SEs) 0.96 (0.02) 0.92 (0.02) 0.82 (0.02) 0.86 (0.03) 0.86 (0.02) 0.85 (0.02) 0.70 (0.05) 0.74 (0.03) 

Low Load
High Load
Unilateral
Bilateral
Unilateral
Bilateral
Left
Right
Left
Right
Left
Right
Left
Right
Recognition accuracy (SEs) 0.96 (0.02) 0.92 (0.02) 0.82 (0.02) 0.86 (0.03) 0.86 (0.02) 0.85 (0.02) 0.70 (0.05) 0.74 (0.03) 

fMRI Analysis

Task-PLS

Task-PLS identified two significant LVs that together explained more than 50% of the covariance between the fMRI images and the six task conditions. The first LV accounted for the largest amount of variance in the data (32%, p < .001). The averaged brain scores are displayed in Figure 3A and B, and the clusters of activity are listed in Table 2 and displayed in Figure 3C. This LV revealed greater activation in the high-load conditions relative to the low-load conditions in regions largely corresponding to a load-dependent VWM network that has been identified in previous experiments (Xu & Chun, 2006; Todd & Marois, 2004, 2005). These regions include the bilateral IPS, bilateral lateral occipital, and anterior cingulate. In addition, activation in these regions negatively correlated with bilateral TPJ, which is consistent with previous experiments indicating TPJ suppression in response to increasing VWM load (Todd et al., 2005).

Figure 3. 

Brain scores and peak clusters for LV1. (A) Mean brain scores as a function of VWM load and object presentation condition. Brain scores reflect the activation summed across the entire brain and averaged across participants. Higher brain scores are observed in the high-load condition relative to the low-load condition, and scores are greater when two objects are presented relative to only a single object. Error bars denote 95% confidence intervals. (B) Mean brain scores observed in LV1 as a function of time (TRs). The diverging effects of VWM load and number of objects can be observed. (C) Peak cluster activations observed in LV1 extracted from TR2. BSRs are displayed. The z axis of the MNI coordinates of each slice is indicated. Regions corresponding to areas previously demonstrated as being sensitive to VWM load are identified.

Figure 3. 

Brain scores and peak clusters for LV1. (A) Mean brain scores as a function of VWM load and object presentation condition. Brain scores reflect the activation summed across the entire brain and averaged across participants. Higher brain scores are observed in the high-load condition relative to the low-load condition, and scores are greater when two objects are presented relative to only a single object. Error bars denote 95% confidence intervals. (B) Mean brain scores observed in LV1 as a function of time (TRs). The diverging effects of VWM load and number of objects can be observed. (C) Peak cluster activations observed in LV1 extracted from TR2. BSRs are displayed. The z axis of the MNI coordinates of each slice is indicated. Regions corresponding to areas previously demonstrated as being sensitive to VWM load are identified.

Table 2. 

Peak Regions Associated with LV1

Location
Hemisphere
X
Y
Z
BSR
High Load > Low Load, Bilateral > Unilateral 
CG 21 42 16.3033 
IPL/sIPS 36 −48 57 10.6997 
IFG 27 21 −9 9.2704 
Cerebellum −45 −63 −21 8.8244 
MFG −30 57 8.512 
MFG −48 27 27 8.4214 
Cuneus −27 −78 30 8.0535 
MFG 30 51 7.8974 
Cerebellum 39 −63 −30 7.534 
IPL/sIPS −36 −45 42 6.8633 
MOG −36 −90 6.8292 
IFG −27 21 −3 6.8265 
Cerebellum −6 −84 −21 6.1512 
MOG 54 −63 −12 6.0603 
SFG −12 12 63 5.8333 
MFG 36 48 12 5.7963 
IFG −45 27 5.7951 
MFG 42 18 21 5.6433 
MTG 45 −72 5.4094 
Caudate 5.2558 
SFG −12 −21 27 4.7761 
SFG 21 21 60 4.7157 
 
High Load < Low Load, Bilateral < Unilateral 
Insula 39 −18 −3 −10.15 
MeFG L? 51 24 −8.3541 
SMG/TPJ −48 −54 30 −7.638 
ITL 42 −3 −21 −7.1153 
AC −9 48 −7.1078 
SMG/TPJ 63 −48 24 −6.6135 
SFG 33 60 −6 −6.4677 
MeFG −12 36 −5.6494 
STG/TPJ 48 −60 33 −5.333 
MTG −54 −21 −12 −4.9569 
Location
Hemisphere
X
Y
Z
BSR
High Load > Low Load, Bilateral > Unilateral 
CG 21 42 16.3033 
IPL/sIPS 36 −48 57 10.6997 
IFG 27 21 −9 9.2704 
Cerebellum −45 −63 −21 8.8244 
MFG −30 57 8.512 
MFG −48 27 27 8.4214 
Cuneus −27 −78 30 8.0535 
MFG 30 51 7.8974 
Cerebellum 39 −63 −30 7.534 
IPL/sIPS −36 −45 42 6.8633 
MOG −36 −90 6.8292 
IFG −27 21 −3 6.8265 
Cerebellum −6 −84 −21 6.1512 
MOG 54 −63 −12 6.0603 
SFG −12 12 63 5.8333 
MFG 36 48 12 5.7963 
IFG −45 27 5.7951 
MFG 42 18 21 5.6433 
MTG 45 −72 5.4094 
Caudate 5.2558 
SFG −12 −21 27 4.7761 
SFG 21 21 60 4.7157 
 
High Load < Low Load, Bilateral < Unilateral 
Insula 39 −18 −3 −10.15 
MeFG L? 51 24 −8.3541 
SMG/TPJ −48 −54 30 −7.638 
ITL 42 −3 −21 −7.1153 
AC −9 48 −7.1078 
SMG/TPJ 63 −48 24 −6.6135 
SFG 33 60 −6 −6.4677 
MeFG −12 36 −5.6494 
STG/TPJ 48 −60 33 −5.333 
MTG −54 −21 −12 −4.9569 

Clusters reported at TR2 with a BSR > 3.5 and a minimum of 10 voxels. CG = cingulate gyrus; IFG = inferior frontal gyrus; IPL = inferior parietal lobule; ITL = inferior temporal gyrus; MeFG = medial frontal gyrus; MFG = middle frontal gyrus; MOG = middle occipital gyrus; MTG = middle temporal gyrus; SFG = superior frontal gyrus; SMG = supramarginal gyrus; STG = superior temporal gyrus.

LV1 also yielded an activation pattern differentiating the bilateral conditions from the left and right unilateral conditions, regardless of the memory load (see Figure 3A). That is, under both low and high VWM loads, activity in VWM regions including sIPS and LOC was greater when two bilateral objects were presented than when only a single item was presented. This finding suggests that these regions, which have previously been shown to be involved in the encoding and maintenance of information in VWM, are also sensitive to the number of objects presented in the periphery during the delay period of a change detection task.

The second significant LV accounted for 23% of the variance in the data, p = .004. The mean brain scores for LV2 are presented in Figure 4A and B, and the corresponding clusters of activity are displayed in Figure 4C as well as in Table 3. Similar to the first LV, LV2 reflected a pattern of brain activity related to both the memory load and object presentation conditions. As can be seen in Figure 4C, the regions corresponding to the pattern of activation in LV2 are predominantly located in regions of the ventral temporal lobe that are typically characterized as the LOC, specifically the anterior lateral occipital region and the posterior fusiform sulcus. In addition, activity in these regions was greater in the bilateral than in the unilateral conditions. Given that these regions are typically more active in response to intact objects than to images of scrambled objects (Grill-Spector et al., 2000), it is likely that the pattern of activations identified by LV2 relates to visual object processing, although it may be possible these regions are also involved in the maintenance of information in VWM. In contrast to LV1, however, LV2 yielded greater activation in ventral temporal regions in the low as opposed to the high VWM load condition, suggesting that these regions are in fact unrelated to VWM. Interestingly, activity in the bilateral sIPS was also identified by this LV, indicating that the IPS is involved in both delineated networks (see General Discussion).

Figure 4. 

Brain scores and peak clusters for LV2. (A) Mean brain scores as a function of VWM load and object presentation condition. Brain scores reflect the activation summed across the entire brain and averaged across participants. Higher brain scores are observed in the low-load condition relative to the high-load condition, and scores are greater when two objects are presented relative to only a single object. Error bars denote 95% confidence intervals. (B) Mean brain scores observed in LV2 as a function of time (TRs). Overall, the brain scores peak later in the trial relative to the scores in LV1 (see Figure 3). (C) Peak cluster activations observed in LV2 extracted from TR4. BSRs are displayed. The z axis of the MNI coordinates of each slice is indicated. Regions identified in LV2 largely correspond to object-sensitive LOC, containing the posterior fusiform sulcus (pFs) and lateral occipital (LO) regions. In addition, a similar pattern of activation was also observed in sIPS.

Figure 4. 

Brain scores and peak clusters for LV2. (A) Mean brain scores as a function of VWM load and object presentation condition. Brain scores reflect the activation summed across the entire brain and averaged across participants. Higher brain scores are observed in the low-load condition relative to the high-load condition, and scores are greater when two objects are presented relative to only a single object. Error bars denote 95% confidence intervals. (B) Mean brain scores observed in LV2 as a function of time (TRs). Overall, the brain scores peak later in the trial relative to the scores in LV1 (see Figure 3). (C) Peak cluster activations observed in LV2 extracted from TR4. BSRs are displayed. The z axis of the MNI coordinates of each slice is indicated. Regions identified in LV2 largely correspond to object-sensitive LOC, containing the posterior fusiform sulcus (pFs) and lateral occipital (LO) regions. In addition, a similar pattern of activation was also observed in sIPS.

Table 3. 

Peak Regions Associated with LV2

Location
Hemisphere
X
Y
Z
BSR
Low Load > High Load, Bilateral > Unilateral 
Caudate −15 15 12.1115 
IOG/LOC 42 −72 −6 11.993 
SPL/sIPS −24 −69 45 9.6444 
FG/LOC −36 −51 −18 8.5085 
PC −9 −69 12 8.3519 
SPL/sIPS 30 −57 39 8.0772 
IFG 54 12 24 7.982 
Thalamus −9 −12 7.1964 
IFG −51 15 21 7.0499 
LG 18 −63 −9 6.869 
Thalamus −3 6.5674 
Putamen 18 18 −6 6.4142 
STG −57 12 −12 6.3124 
FG/LOC 33 −69 −27 6.2894 
MeFG −6 −6 57 6.144 
SOG −21 42 −18 6.0465 
MOG/LOC −33 −87 5.9271 
MFG −42 −3 54 5.7991 
IPL −45 −36 48 5.5332 
Cerebellum 12 −81 −33 5.1401 
FG/LOC 33 −39 −27 5.0863 
AC 21 24 4.9499 
MeFG 18 51 4.6541 
Location
Hemisphere
X
Y
Z
BSR
Low Load > High Load, Bilateral > Unilateral 
Caudate −15 15 12.1115 
IOG/LOC 42 −72 −6 11.993 
SPL/sIPS −24 −69 45 9.6444 
FG/LOC −36 −51 −18 8.5085 
PC −9 −69 12 8.3519 
SPL/sIPS 30 −57 39 8.0772 
IFG 54 12 24 7.982 
Thalamus −9 −12 7.1964 
IFG −51 15 21 7.0499 
LG 18 −63 −9 6.869 
Thalamus −3 6.5674 
Putamen 18 18 −6 6.4142 
STG −57 12 −12 6.3124 
FG/LOC 33 −69 −27 6.2894 
MeFG −6 −6 57 6.144 
SOG −21 42 −18 6.0465 
MOG/LOC −33 −87 5.9271 
MFG −42 −3 54 5.7991 
IPL −45 −36 48 5.5332 
Cerebellum 12 −81 −33 5.1401 
FG/LOC 33 −39 −27 5.0863 
AC 21 24 4.9499 
MeFG 18 51 4.6541 

Clusters reported at TR4 with a BSR > 3.5 and a minimum of 10 voxels. AC = anterior cingulate; FG = fusiform gyrus; IFG = inferior frontal gyrus; IOG = inferior occipital gyrus; IPL = inferior parietal lobule; LG = lingual gyrus; LOC = lateral occipital complex; MeFG = medial frontal gyrus; MFG = middle frontal gyrus; MOG = middle occipital gyrus; MTG = middle temporal gyrus; PC = posterior cingulate; SFG = superior frontal gyrus; SMG = supramarginal gyrus; SOG = superior orbital gyrus; SPL = superior parietal lobule; STG = superior temporal gyrus.

Regression analysis

Given the anatomical regions and brain scores corresponding to the two significant LVs identified by the task PLS analysis, object processing under a concurrent VWM load appears to depend on the activation of two orthogonal but partially overlapping networks of brain regions: The first group of regions includes areas that are sensitive to the encoding and maintenance of information in VWM (e.g., IPS, LOC, and TPJ), whereas the second set of brain regions involves areas that are sensitive to images of objects (primarily ventral temporal regions). The question remains whether activity in either of these networks corresponds to object recognition performance. That is, because Task-PLS is data driven (independent of performance outcomes), the brain scores represent only the covariance between the task conditions and the corresponding fMRI activation. If the patterns of activation extracted by the LVs actually reflect the changes in object recognition performance, then activation in these LVs (i.e., brain scores) should be directly tied to recognition performance. To examine this relationship, we performed a regression analysis to determine whether brain scores of either LV were predictive of object recognition performance. By regressing brain scores onto recognition accuracy, we can establish to what extent the covariance extracted by these LVs conveys meaningful information with respect to object recognition performance. Because LVs are orthogonal to each other, brain scores from each LV were entered in separate regression analyses.

Across conditions, the average brain scores of LV1 were a significant factor in predicting averaged accuracy, b = .93, t(4) = −4.94, p = .008. Furthermore, this relationship is also evident when predicting individual participant accuracy scores, across all conditions, b = −.5, t(58) = −4.43, p < .001. That is, individual brain scores in LV1 predict an individual's accuracy across all six conditions: As brain scores in LV1 increase (reflecting increases in fMRI activation in response to both memory load and object presentation), recognition accuracy decreases (Figure 5A). It is important to note that this negative relationship is precisely the opposite of what would be predicted if performance on the recognition task depended on VWM alone. That is, during VWM tasks, activation in areas that mediate VWM increases from low load to high load, whereas change detection performance remains relatively constant; decreases in performance occur only after capacity has been exceeded and memory-related activity asymptotes (Todd & Marois, 2004). In contrast, recognition performance in the present study decreases as activation of LV1 increases. Thus, this relationship suggests that, although recognition performance may be related to processes subserved by regions that support VWM, the recognition task does not reflect VWM per se.

Figure 5. 

Relationship between brain scores and recognition performance. (A) Regression between brain scores in LV1 and object recognition accuracy. Increases in activation in regions associated with LV1 (see Figure 3) are accompanied by decreases in overall recognition accuracy. (B) Regression between brain scores in LV2 and the recognition scores. Recognition scores reflect the summed accuracy for left and right objects in the bilateral condition. As can be seen, all low-load conditions fall above the regression line, whereas all high-load conditions fall below it. Thus, increases in activation in regions extracted in LV2 are associated with a greater number of correctly recognized objects, even when absolute accuracy decreases. Legend: black diamond = low load, left; gray diamond = low load, right; gray circle = low load, bilateral; black triangle = high load, left; gray triangle = high load, right; gray square = high load, bilateral.

Figure 5. 

Relationship between brain scores and recognition performance. (A) Regression between brain scores in LV1 and object recognition accuracy. Increases in activation in regions associated with LV1 (see Figure 3) are accompanied by decreases in overall recognition accuracy. (B) Regression between brain scores in LV2 and the recognition scores. Recognition scores reflect the summed accuracy for left and right objects in the bilateral condition. As can be seen, all low-load conditions fall above the regression line, whereas all high-load conditions fall below it. Thus, increases in activation in regions extracted in LV2 are associated with a greater number of correctly recognized objects, even when absolute accuracy decreases. Legend: black diamond = low load, left; gray diamond = low load, right; gray circle = low load, bilateral; black triangle = high load, left; gray triangle = high load, right; gray square = high load, bilateral.

In contrast to LV1, LV2 was not a significant predictor of either averaged recognition accuracy, b = −.08, p = .89, or of individual accuracy scores, b = −.06, p = .67. However, given that LV2 appears to correspond to activation in object recognition regions, it is possible that this activity is not linearly related to changes in absolute accuracy but is instead related to the amount of object information that is processed by these regions. To test for this possibility, we examined the summed accuracy for left and right objects in the bilateral conditions. That is, because two objects are presented in the bilateral condition, more objects may be processed by these regions, although overall accuracy is lower relative to the unilateral condition. The resulting scores reflect the processing of two objects in the bilateral conditions and are plotted against brain scores for LV2 in Figure 5B. As can been seen in this figure, the averaged brain scores of LV2 are predictive of the number of items correctly identified, b = .80, t(4) = 2.66, p = .056. This relationship is also significant when examining individual subject brain scores and performance across all conditions, b = .63, t(58) = 6.17, p < .001. That is, between the six conditions, activation in ventral temporal cortex increases with the number of objects correctly identified, with greater activity observed for bilateral conditions, as overall, more object information is present, although raw accuracy performance is impaired.

The finding that activation in both networks is closely linked with recognition performance suggests that conscious object recognition depends on the coactivations of these regions. It is possible, however, that there are alternative explanations for these observed effects. For example, one might argue that the relationship between activation in areas sensitive to VWM load and recognition performance indicates that the recognition task is in fact a VWM task; that is, that performance decreases as activation increases because participants are trying to store a greater number of items in capacity-limited VWM. Although our task clearly requires participants to maintain object information after it has been removed from view, we do not believe that the correlation between recognition performance and VWM load is purely driven by the maintenance aspect of our design. As noted above, although activation in LV1 is related to performance on the recognition task, this relationship is inconsistent with a memory-based account. Specifically, recognition performance decreases as activity in memory areas increases. In addition, the identification of two distinct LVs reveals that multiple neural and cognitive processes are at play during the task. Even if the activation of LV2 were related to memory for the object stimuli, we would expect these regions to exhibit a pattern of activity consistent with the memory demands of the task. That is, as memory demands increase, activation should increase proportionally. The distinct and orthogonal patterns of activation observed in these LVs suggest that these areas reflect different aspects of task performance. Furthermore, the total number of items combined between the tasks (three change detection items, two objects) exceeded the average maximum K obtained in the VWM localizer task (3) which included a condition with five to-be-remembered items. Thus, if performance were related to memory encoding and maintenance alone, then brain scores (representing activation levels) in VWM regions should have reached a plateau at the low-load bilateral condition (a total of three objects with participants' capacity estimated at three items) with no further increase in the high-load conditions (combined number of objects: 4 or 5). In contrast, our results revealed that brain scores in LV1 demonstrated significant increases in the high-load bilateral condition relative to all other conditions, indicating a continued increase in activation in response to object stimuli. To confirm this effect, we compared activation levels in the sIPS ROI (see Methods) as measured during the independent localizer runs and as measured during the experimental task.

VWM ROI analysis

The results of LV1 suggest that regions subserving VWM are sensitive to the number of object stimuli presented during the delay period of a VWM task, even when VWM resources have already been exhausted. To find converging evidence for this observation, we compared the activation in the sIPS ROI identified using the localizer task (see Methods) to that observed during the experimental task. During the change detection only localizer task, activation in the sIPS did not increase significantly from Load 3 to Load 5, d = .05, t(8) = −.75, p = .48, indicating that activation in VWM regions asymptotes with capacity (K = 3), consistent with previous studies (Xu & Chun, 2006; Todd & Marois, 2004). In the experimental task, however, activation in the high-load bilateral condition (combined number of items: 5) was significantly greater than that of the low-load bilateral condition (three items), d = .13, t(8) = 2.3, p = .05. Thus, activation in regions supporting VWM continued to increase in response to the visual load, although the total load (VWM change detection + object load) exceeded the capacity of VWM. The fact that brain scores continue to increase suggests that the objects presented in the periphery, while having an effect on activation in IPS and other VWM regions, were not encoded and maintained in VWM in the same way as those in the VWM task.

GENERAL DISCUSSION

VWM Load and Conscious Object Recognition

In the current experiments, we sought to test how perceptual information is processed under VWM load. The behavioral findings presented here extend previous findings (Todd et al., 2005) by demonstrating that under a high VWM load, even the processing of expected stimuli is impaired, but this impairment emerges only in higher levels of perceptual processing (i.e., in conscious recognition rather than in detection). The task-PLS fMRI results revealed two orthogonal but partially overlapping patterns of activation involved in the recognition of objects. Although one network of brain regions activated areas of extrastriate cortex that are typically associated with object recognition, activation in a larger network that included regions involved in the encoding and maintenance of information in VWM was also closely linked to recognition performance.

Our results demonstrate that under a concurrent VWM load, regions associated with capacity-limited VWM showed increases in response-to-object stimuli. Similarly, a recent finding demonstrated that activity in posterior parietal cortex is sensitive to perceptual load in addition to VWM load (Mitchell & Cusack, 2008). These findings are consistent with the idea that VWM and object perception share overlapping neural and cognitive resources. Specifically, it may be necessary to individuate and perceptually enhance object stimuli before they can be recognized or encoded into VWM (Xu & Chun, 2009) or to create an “object file” for those items (Pylyshyn, 1989; Kahneman & Treisman, 1984). A concurrent VWM load may, therefore, reduce the available resources for object individuation and enhancement, thereby affecting the ability to perform subsequent object recognition. Interestingly, our results demonstrate that, although both VWM (Todd & Marois, 2004) and perceptual load (Mitchell & Cusack, 2008) may have capacity limits in the posterior parietal cortex, these limitations may be somewhat independent, as activation in IPS continued to increase in response to object stimuli even when VWM capacity had been exceeded.

Importantly, although the two networks of regions isolated distinct brain regions, both networks also included the IPS. The activation of IPS in both networks suggests this region may play a particularly important role in the conscious recognition of objects. This adds to a growing body of literature indicating some amount of object selectivity in regions of the dorsal visual stream, in particular, the IPS (Konen & Kastner, 2008; Grill-Spector et al., 2000; Sereno & Maunsell, 1987). Unlike previous studies (e.g., Grill-Spector et al., 2000), however, our results here demonstrate that activation in this region correlates with object recognition performance. Although further studies will be required to address this question more directly, our results indicate that the role IPS plays in object recognition may depend on the perceptual or WM load demands of the task.

Overall, the PLS results revealed two distinct patterns of activation that were both correlated with different aspects of performance on the object recognition task. These results suggest that object recognition may not depend on activation of singular regions or even on the activation of a single network of regions (i.e., those in the ventral stream) but may instead depend on the interactions between multiple networks involved in different aspects of object processing. Further work is needed to precisely elucidate the precise relationship between these networks; however, the different patterns of activation observed in the two LVs may provide some insight: Although recognition accuracy decreases with increasing activation in LV1, increases in activation in LV2 correspond to increases in the number of objects correctly identified. One possible explanation for this relationship is that LV2 reflects the endpoint of conscious perception (the number of objects correctly identified), whereas LV1 represents activation of limited-capacity VWM and attentional resources. Under a low VWM load, object stimuli compete for limited resources, but enough resources remain for both objects to be processed, resulting in the high activation of object recognition regions and the successful recognition of both objects. Under a high VWM load, however, the increased demand on capacity-limited VWM resources results in limited perceptual processing of multiple object stimuli. As a consequence, activation in ventral object recognition areas decreases, and fewer objects are successfully recognized. According to this account, information processing in the IPS may act as the computational bottleneck limiting the amount of information that can be sent to higher perceptual areas. Thus, VWM may be part of a capacity-limited resource that is integral to conscious visual perception (Dell'Acqua, Sessa, Jolicoeur, & Robitaille, 2006). Protecting those contents from interference, therefore, may be a critical priority of the brain (Anticevic et al., 2010; Shulman et al., 2007). Thus, under high perceptual or WM load, deficits in perception may perhaps not be viewed as a shortcoming but rather an essential component of visual processing in its service of conscious visual perception.

VWM Load and Visual Neglect

Recognition performance in the first experiment also demonstrated a significant hemifield asymmetry such that conscious object recognition was most impaired under the high-load condition for objects presented on the left side. This finding is consistent with the deficits observed in visual neglect (i.e., a deficit in reporting stimuli in the left visual field) and may have two potentially important implications. First, it elucidates the neural and cognitive mechanisms that underlie the deficits observed in both visual neglect and visual extinction. That is, the finding that recognition performance is closely associated with the deactivation of TPJ and the activation of regions associated with VWM load support the data suggesting that neglect is the result of damage to the right TPJ (Danckert & Ferber, 2006; Karnath et al., 2001) and that the impairments in visual awareness arise in part from impairments in VWM functioning (Ferber & Danckert, 2006; Wojciulik, Husain, Clarke, & Driver, 2001). Further support for this model comes from a recent study that demonstrates a contralateral bias for VWM information in IPS (Sheremata, Bettencourt, & Somers, 2010). Second, the results demonstrate that the “functional lesioning” of the right TPJ under a high VWM load may provide a working model for visual neglect in healthy populations, thereby enabling further tests of the mechanisms of this disorder. Similarly, a high VWM load may be a useful tool in the assessment and treatment of the extent of visual impairment poststroke.

To summarize, in the present study we sought to examine conscious object processing under a concurrent VWM load. Behaviorally, participants demonstrated impaired recognition performance under a high relative to a low VWM load. This was particularly true when multiple objects were presented and when objects were presented on the left side. Our fMRI data revealed that recognition performance depends on the activation of two networks of brain regions. In addition to the expected involvement of object-sensitive areas, the other network included brain regions involved in VWM encoding and maintenance. Thus, activation in capacity-limited VWM areas may play an important role in our ability to consciously perceive and identify visual objects.

Acknowledgments

The authors thank Paul Lu and Carson Pun for assistance with data collection, Douglas Garret for statistical expertise, and Morgan Barense for helpful feedback. This work was supported by Natural Sciences and Engineering Research Council (grant 261203-08) and Canadian Institutes of Health Research (grant 79256) awarded to S. F.

Reprint requests should be sent to Stephen M. Emrich, Department of Psychology, University of Toronto, 100 St. George Street, Toronto, Ontario, M5S 3G3, Canada, or via e-mail: steve.emrich@utoronto.ca.

REFERENCES

Anticevic
,
A.
,
Repovs
,
G.
,
Shulman
,
G. L.
, &
Barch
,
D. M.
(
2010
).
When less is more: TPJ and default network deactivation during encoding predicts working memory performance.
Neuroimage
,
49
,
2638
2648
.
Baars
,
B. J.
, &
Franklin
,
S.
(
2003
).
How conscious experience and working memory interact.
Trends in Cognitive Sciences
,
7
,
166
172
.
Corbetta
,
M.
,
Kincade
,
J. M.
,
Ollinger
,
J. M.
,
McAvoy
,
M. P.
, &
Shulman
,
G. L.
(
2000
).
Voluntary orienting is dissociated from target detection in human posterior parietal cortex.
Nature Neuroscience
,
3
,
292
297
.
Cowan
,
N.
(
2001
).
The magical number 4 in short-term memory: A reconsideration of mental storage capacity.
Behavioral and Brain Sciences
,
24
,
87
114; discussion 114-185
.
Danckert
,
J.
, &
Ferber
,
S.
(
2006
).
Revisiting unilateral neglect.
Neuropsychologia
,
44
,
987
1006
.
Dell'Acqua
,
R.
,
Sessa
,
P.
,
Jolicoeur
,
P.
, &
Robitaille
,
N.
(
2006
).
Spatial attention freezes during the attention blink.
Psychophysiology
,
43
,
394
400
.
Downar
,
J.
,
Crawley
,
A. P.
,
Mikulis
,
D. J.
, &
Davis
,
K. D.
(
2002
).
A cortical network sensitive to stimulus salience in a neutral behavioral context across multiple sensory modalities.
Journal of Neurophysiology
,
87
,
615
620
.
Efron
,
B.
, &
Tibshirani
,
R.
(
1986
).
Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy.
Statistical Science
,
1
,
54
75
.
Ferber
,
S.
, &
Danckert
,
J.
(
2006
).
Lost in space—The fate of memory representations for non-neglected stimuli.
Neuropsychologia
,
44
,
320
325
.
Ferber
,
S.
,
Danckert
,
J.
,
Joanisse
,
M.
,
Goltz
,
H. C.
, &
Goodale
,
M. A.
(
2003
).
Eye movements tell only half the story.
Neurology
,
60
,
1826
1829
.
Friston
,
K. J.
,
Frith
,
C. D.
, &
Frackowiak
,
R. S. J.
(
1993
).
Principal component analysis learning algorithms: A neurobiological analysis.
Proceedings: Biological Sciences
,
254
,
47
54
.
Goebel
,
R.
,
Esposito
,
F.
, &
Formisano
,
E.
(
2006
).
Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: From single-subject to cortically aligned group general linear model analysis and self-organizing group independent component analysis.
Human Brain Mapping
,
27
,
392
401
.
Grill-Spector
,
K.
,
Kushnir
,
T.
,
Hendler
,
T.
, &
Malach
,
R.
(
2000
).
The dynamics of object-selective activation correlate with recognition performance in humans.
Nature Neuroscience
,
3
,
837
843
.
Heilman
,
K. M.
,
Watson
,
R. T.
, &
Valenstein
,
E.
(
1993
).
Neglect and related disorders.
In K. M. Heilman & E. Valenstein (Eds.),
Clinical neuropsychology
(3rd ed., pp.
279
336
).
New York
:
Oxford University Press
.
Hollingworth
,
A.
, &
Rasmussen
,
I. P.
(
2010
).
Binding objects to locations: The relationship between object files and visual working memory.
Journal of Experimental Psychology: Human Perception and Performance
,
36
,
543
564
.
Kahneman
,
D.
, &
Treisman
,
A.
(
1984
).
Changing views of attention and automaticity.
In R. Parasuramam & D. R. Davies (Eds.),
Varieties of attention
(pp.
29
62
).
New York
:
Academic Press
.
Kahneman
,
D.
,
Treisman
,
A.
, &
Gibbs
,
B. J.
(
1992
).
The reviewing of object files: Object-specific integration of information.
Cognitive Psychology
,
24
,
175
219
.
Karnath
,
H. O.
(
1988
).
Deficits of attention in acute and recovered visual hemi-neglect.
Neuropsychologia
,
26
,
27
43
.
Karnath
,
H. O.
,
Ferber
,
S.
, &
Himmelbach
,
M.
(
2001
).
Spatial awareness is a function of the temporal not the posterior parietal lobe.
Nature
,
411
,
950
953
.
Konen
,
C. S.
, &
Kastner
,
S.
(
2008
).
Two hierarchically organized neural systems for object information in human visual cortex.
Nature Neuroscience
,
11
,
224
231
.
Luck
,
S. J.
, &
Vogel
,
E. K.
(
1997
).
The capacity of visual working memory for features and conjunctions.
Nature
,
390
,
279
281
.
McIntosh
,
A. R.
,
Bookstein
,
F. L.
,
Haxby
,
J. V.
, &
Grady
,
C. L.
(
1996
).
Spatial pattern analysis of functional brain images using partial least squares.
Neuroimage
,
3
,
143
157
.
McIntosh
,
A. R.
,
Chau
,
W. K.
, &
Protzner
,
A. B.
(
2004
).
Spatio-temporal analysis of event-related fMRI data using partial least squares.
Neuroimage
,
23
,
764
775
.
Mitchell
,
D. J.
, &
Cusack
,
R.
(
2008
).
Flexible, capacity-limited activity of posterior parietal cortex in perceptual as well as visual short-term memory tasks.
Cerebral Cortex
,
18
,
1788
1798
.
Mort
,
D. J.
,
Malhotra
,
P.
,
Mannan
,
S. K.
,
Rorden
,
C.
,
Pambakian
,
A.
,
Kennard
,
C.
,
et al
(
2003
).
The anatomy of visual neglect.
Brain
,
126
,
1986
1997
.
Pashler
,
H.
(
1988
).
Familiarity and visual change detection.
Perception & Psychophysics
,
44
,
369
378
.
Pylyshyn
,
Z.
(
1989
).
The role of location indexes in spatial perception: A sketch of the FINST spatial-index model.
Cognition
,
32
,
65
97
.
Robitaille
,
N.
,
Grimault
,
S.
, &
Jolicoeur
,
P.
(
2009
).
Bilateral parietal and contralateral responses during maintenance of unilaterally encoded objects in visual short-term memory: Evidence from magnetoencephalography.
Psychophysiology
,
46
,
1090
1099
.
Serences
,
J. T.
,
Shomstein
,
S.
,
Leber
,
A. B.
,
Golay
,
X.
,
Egeth
,
H. E.
, &
Yantis
,
S.
(
2005
).
Coordination of voluntary and stimulus-driven attentional control in human cortex.
Psychological Science
,
16
,
114
122
.
Sereno
,
A. B.
, &
Maunsell
,
J. H. R.
(
1987
).
Shape selectivity in primate lateral intraparietal cortex.
Journal of Experimental Psychology: Human Perception and Performance
,
12
,
388
391
.
Sheremata
,
S. L.
,
Bettencourt
,
K. C.
, &
Somers
,
D. C.
(
2010
).
Hemispheric asymmetry in visuotopic posterior parietal cortex emerges with visual short-term memory load.
Journal of Neuroscience
,
30
,
12581
12588
.
Shulman
,
G. L.
,
Astafiev
,
S. V.
,
McAvoy
,
M. P.
,
d'Avossa
,
G.
, &
Corbetta
,
M.
(
2007
).
Right TPJ deactivation during visual search: Functional significance and support for a filter hypothesis.
Cerebral Cortex
,
17
,
2625
2633
.
Todd
,
J. J.
,
Fougnie
,
D.
, &
Marois
,
R.
(
2005
).
Visual short-term memory load suppresses temporo-parietal junction activity and induces inattentional blindness.
Psychological Science
,
16
,
965
972
.
Todd
,
J. J.
, &
Marois
,
R.
(
2004
).
Capacity limit of visual short-term memory in human posterior parietal cortex.
Nature
,
428
,
751
754
.
Todd
,
J. J.
, &
Marois
,
R.
(
2005
).
Posterior parietal cortex activity predicts individual differences in visual short-term memory capacity.
Cognitive, Affective, and Behavioral Neuroscience
,
5
,
144
155
.
Wojciulik
,
E.
,
Husain
,
M.
,
Clarke
,
K.
, &
Driver
,
J.
(
2001
).
Spatial working memory deficit in unilateral neglect.
Neuropsychologia
,
39
,
390
396
.
Xu
,
Y.
, &
Chun
,
M. M.
(
2006
).
Dissociable neural mechanisms supporting visual short-term memory for objects.
Nature
,
440
,
91
95
.
Xu
,
Y.
, &
Chun
,
M. M.
(
2007
).
Visual grouping in human parietal cortex.
Proceedings of the National Academy of Sciences, U.S.A.
,
104
,
18766
18771
.
Xu
,
Y.
, &
Chun
,
M. M.
(
2009
).
Selecting and perceiving multiple visual objects.
Trends in Cognitive Sciences
,
13
,
167
174
.