Abstract

Although humans can hold multiple items in mind simultaneously, the contents of working memory (WM) can be selectively prioritized to guide future behavior. We explored whether the “same-object” benefits in visual processing may also be observed in visual WM. fMRI data were collected while participants performed a multistep serial retrocuing task in which they first viewed two 2-D objects (coherently moving colored dots). During retention, an initial relevance cue then indicated whether only the first or only the second object (“object-relevant”), or only the color of both objects or only their direction of motion would be relevant for the remainder of the trial (“feature-relevant”). On “object-relevant” trials, the ensuing priority cues selected either one of the features (“color” or “direction”) bound to the relevance-cued object, whereas on “feature-relevant” trials, the priority cues selected one of the two relevance-cued features. Using multivariate inverted encoding models, we found a same-object benefit on object-relevant trials in occipitotemporal regions: On feature-relevant trials, the first priority cue triggered a strengthening of the neural representation of the cued feature and a concomitant weakening to baseline of the uncued feature, whereas on object-relevant trials, the cued item remained active but did not increase in strength and the uncued item weakened but remained significantly elevated throughout the delay period. Although the stimulus-specific representation in frontoparietal regions was weak and uneven, these regions closely tracked the higher order information of which stimulus category was relevant for behavior throughout the trial, suggesting an important role in controlling the prioritization of information in visual WM.

INTRODUCTION

Working memory (WM) is a cognitive function that enables the mental retention of information in the absence of sustained input from the physical world, its manipulation, and its use for guiding behavior. Current accounts of sensory WM hold that it relies on attentional mechanisms also involved in the prioritization of information perceived in the environment (D'Esposito & Postle, 2015; Kiyonaga & Egner, 2013; Gazzaley & Nobre, 2012; Oberauer & Hein, 2012). Consistent with this view is the fact that instructing an individual to prioritize a subset of information being held in WM with a retrodictive cue (hereafter, “retrocue”) improves its subsequent recall at the expense of uncued information, in a manner comparable to the effects on visual perception of prospectively cuing a location or a feature in an impending visual scene (Sahan, Verguts, Boehler, Pourtois, & Fias, 2016; Zokaei, Manohar, Husain, & Feredoes, 2014; Pertzov, Bays, Joseph, & Husain, 2013; Lepsien & Nobre, 2007; Griffin & Nobre, 2003). The retrocuing technique has also been used to test theories about the capacity and the temporal dynamics of the putative focus of attention (or “region of direct access”) in state-based models of WM (as reviewed in LaRocque, Lewis-Peacock, & Postle, 2014). In one series of studies, we have used a multistep delayed serial retrocuing (DSR) procedure in which participants are first presented with two sample items to hold in WM, then after a retention interval, an initial retrocue indicates which of the two will be tested by the impending memory probe. Because participants know, however, that this probe will be followed by a second retrocue and a second probe and because they know there is an equal probability that the second retrocue will prioritize either sample item, they cannot forget the item not cued by the initial retrocue. This creates a portion of the trial in which two items are being held in WM, but only one is a “prioritized memory item” (PMI). Initially, in studies with fMRI (LaRocque, Riggall, Emrich, & Postle, 2017; Rose et al., 2016; Lewis-Peacock & Postle, 2012) and with EEG (Rose et al., 2016; LaRocque, Lewis-Peacock, Drysdale, Oberauer, & Postle, 2013), multivariate pattern analyses (MVPAs) showed that decoding the neural representation of the PMI improved following the initial retrocue, whereas decoding the neural representation of the initially uncued (and, therefore, “unprioritized”) memory item (UMI) dropped to baseline levels. More recently, there have been reports of multivariate evidence for the UMI, with an item represented in a different region (e.g., Christophel, Iamshchinina, Yan, Allefeld, & Haynes, 2018) or in a different neural code (e.g., van Loon, Olmos-Solis, Fahrenfort, & Olivers, 2018) when a UMI relative to when a PMI. This study was designed to use a multivariate inverted encoding modeling (IEM) approach, which offers advantages over multivariate decoding approaches (Serences & Saproo, 2012), to address several questions that arise from these observations: (1) Is one effect of selection to increase the strength of the neural representation of the PMI? (2) Does the degradation of MVPA decodability of the UMI truly correspond to a weakening of its neural representation? (3) Regardless of the answer to (2), is the retrocuing effect on the neural representation of the UMI sensitive to its status as a discrete object versus as a feature in a multidimensional object? The answers to these questions will have implications for broader questions, such as whether principles of object- and feature-based attention also apply to WM, and whether a complex object is represented in WM as more than the set of features that define it.

Well-established principles of visual attention, such as biased competition (Desimone & Duncan, 1995) and divisive normalization (Carandini & Heeger, 2012), have the potential to account the finding that retrocuing leads to a weakening of the neural representation of the UMI. That is, prioritization of one among multiple mnemonic representations could be achieved via top–down signals from frontoparietal systems that are important for the endogenous control of attention (e.g., Nelissen, Stokes, Nobre, & Rushworth, 2013). Importantly, the dynamics of biased competition have been demonstrated, with MVPA, to influence the population-level representation of objects in a manner that would be predicted from single-neuron studies, both in analyses of extracellular recordings from neurons in monkey ventral temporal cortex (Zhang et al., 2011) and in unpublished fMRI data from humans performing a selective attention task (Sheldon, Saad, Sahan, Meyering, & Postle, Submitted). Indeed, it is the effects suggesting the operation of object-based attention, in a different previous fMRI study of WM, that motivate the present experiment.

In an fMRI study of DSR by Lewis-Peacock, Drysdale, and Postle (2015), participants first viewed an image of a real-world object (e.g., a baseball) and were then cued as to what dimension of that stimulus would be interrogated by a memory probe: its silhouette outline, the phonology of its name, or the semantic category to which it belonged. Although the performance of classifiers trained independently to discriminate visual from phonological from semantic processing strengthened and weakened in a manner congruent with the cues, unlike in studies that required WM for two discrete objects, decoding for the uncued stimulus features did not drop to baseline levels. One possible account of this observation was that we were observing a neural correlate of an analogue of the “same-object” benefit that is seen in visual selective detection (Driver, 2001; Vecera, Behrmann, & McGoldrick, 2000; Duncan, 1984). That is, perhaps the uncued stimulus dimensions retained some level of activity because they were an inherent part of the same object from which a different stimulus dimension had been selected, and attention therefore spread to all components of the selected object. Limitations of that study's design, however, precluded a strong test of this possibility.

The question of whether multidimensional objects are represented as bound objects in visual WM remains contentious in WM research. On one hand, there are several studies suggesting that objects defined by a conjunction of two or more features can be maintained just as well as can single-feature objects, suggesting that the elementary units of visual WM are integrated objects (e.g., Peters, Kaiser, Rahm, & Bledowski, 2015; Luria & Vogel, 2011; Woodman & Vogel, 2008; Luck & Vogel, 1997). Others have argued, however, that the elementary units of visual WM are the features that make up complex visual objects and that the various features of an object are simultaneously stored in dimension-specific channels (e.g., Burmester & Fougnie, 2016; Bays, Wu, & Husain, 2011; Wheeler & Treisman, 2002). Furthermore, according to these feature-based accounts, feature binding in WM only occurs when attention is exerted over the to-be-bound features. For instance, Wheeler and Treisman (2002) showed that same-object benefits were observed in visual WM only when participants were not holding competing multifeature objects in WM, presumably because these would disrupt sustained attentional control. The design of this study may also help to address this debate.

To address more directly whether principles of object-based attention can be observed during visual WM, we designed this study to compare the neural effects of selecting one feature from among two 1-D objects being held concurrently in WM versus those of selecting one feature from a single 2-D compound object being held in WM. Additionally, for this study, we adopted an analytic method that would allow us to quantify the effects of selection on the strength of WM representations. A limitation of the MVPA decoding approach, such as what we have used in many previous DSR studies, is that it does not index the strength of neural representations per se. For example, although one can observe systematic changes in MVPA performance with the manipulation of, say, the number of items being held in WM (e.g., Emrich, Riggall, LaRocque, & Postle, 2013), the interpretation of such a finding with regard to the neural representation of stimulus information is equivocal. Although it could be that a load-related decline in the strength of the neural representations of items leads to the decline in MVPA decoding, there are equally plausible alternative explanations: Perhaps increasing load changes the level of stimulus-nonspecific noise that nonetheless influences the performance of the decoder, or perhaps increasing load changes the nature of the neural code, but not of the amplitude per se, of stimulus representations. IEM, in contrast, entails the fitting of data to one or more a priori models that specify the mapping from multiple sensor-level signals into a hypothesized population-level representation. This affords the quantification of different parameters of the model fit, such that one can estimate, in our case, the extent to which selection might strengthen or weaken our model's estimated reconstruction of the neural representation of stimulus information.

METHODS

Participants

Ten neurologically healthy students from the University of Wisconsin–Madison (three women, 18–30 years, M = 22 years, all right-handed) participated in three 2-hr scanning sessions in return for payment. One participant was excluded from the analyses because of excessive head movement. Another participant was an author of this study (A. D. S.). All participants had normal or corrected-to-normal vision and reported having normal color vision. The research complied to the guidelines of the University of Wisconsin–Madison's Health Sciences institutional review board, and all participants gave written informed consent.

Design

The experiment comprised a one-item delayed recall (a.k.a. “delayed estimation”) task and a multiple serial retrocuing (MSR) task (Figure 1). The purpose of the delayed recall task was twofold: to serve as a localizer that was independent of the MSR task and to train feature dimension-specific IEMs for testing on the MSR data. The primary rationale for this approach is that reconstructions by an IEM that is trained on an independent data set can be used for quantitative comparisons between experimental conditions of interest (e.g., the effect of priority status on estimates of the strength of the delay period representation of motion in the MSR task). Operationally, because we were interested in the effects of prioritization on mnemonic representations, we trained IEMs trained on the final repetition time (TR) of the delay period from the one-item delayed recall task for testing on data from the MSR task. Our reasoning was that such “one-item delay” models would constitute relatively robust and “pure” models of the representation of stimulus motion and stimulus color in WM: robust because multivariate estimates of stimulus information are stronger when just one item is being held in WM (e.g., Sprague, Ester, & Serences, 2014; Emrich et al., 2013) and “pure” because additional processes such as context binding (Gosseries et al., 2018) and attention shifting (e.g., Thigpen, Petro, Oschwald, Oberauer, & Keil, 2019) were less likely to be engaged. We made this choice with the knowledge that a limitation of this approach would be that that all the results that it would generate could only be interpreted in terms of the codes engaged by the one-item delayed recall task. That is, if one or more neural codes are engaged uniquely by the MSR task, our approach would not be sensitive to this. Nonetheless, our experience from previous studies (e.g., Cai, Sheldon, Yu, & Postle, 2019; LaRocque et al., 2017) gave us confidence that this approach would be effective at testing our principal hypotheses of interest.

Figure 1. 

(A) Illustration of a trial from each condition of the one-item delayed recall task. (B) Sample-evoked feature-nonselective VOT ROI and delay-evoked feature-nonselective frontoparietal ROI from a representative participant. The multivariate analyses were performed at the single participant level and were restricted to the areas that showed increased BOLD activity either evoked by the sample (red area) or sustained throughout the delay period (blue area) of the one-item delayed recall task. Note that the frontoparietal regions are combined here for visualization purposes, whereas the frontal and parietal regions were treated separately in subsequent analyses and figures. (C) Illustration of a trial from the Bound condition and a trial from Unbound condition of the MSR task, together with group-, condition-, and trial-averaged BOLD percent signal change from the ROIs illustrated in B. The BOLD percent signal change time series were temporally smoothed using cubic spline interpolation. The width of the traces denotes the SEM. Note that the trial diagrams of the one-item delayed recall and the MSR tasks are cartoon representations of the actual stimuli used during experimenting.

Figure 1. 

(A) Illustration of a trial from each condition of the one-item delayed recall task. (B) Sample-evoked feature-nonselective VOT ROI and delay-evoked feature-nonselective frontoparietal ROI from a representative participant. The multivariate analyses were performed at the single participant level and were restricted to the areas that showed increased BOLD activity either evoked by the sample (red area) or sustained throughout the delay period (blue area) of the one-item delayed recall task. Note that the frontoparietal regions are combined here for visualization purposes, whereas the frontal and parietal regions were treated separately in subsequent analyses and figures. (C) Illustration of a trial from the Bound condition and a trial from Unbound condition of the MSR task, together with group-, condition-, and trial-averaged BOLD percent signal change from the ROIs illustrated in B. The BOLD percent signal change time series were temporally smoothed using cubic spline interpolation. The width of the traces denotes the SEM. Note that the trial diagrams of the one-item delayed recall and the MSR tasks are cartoon representations of the actual stimuli used during experimenting.

The delayed recall task began with the presentation of a sample stimulus, either a patch of uniformly colored static dots whose color varied from trial to trial or a patch of gray dots moving with 100% coherence in a direction that varied from trial to trial. Data from this task were used to train IEMs to learn the neural bases of perceiving and remembering “color” and “direction of motion.” We note that it is possible that, when moving dots are presented at high contrast within a circular aperture, as is the case here, it is possible that the signals that support direction-of-motion decoding may not (only) correspond to motion processing but perhaps (also) to other factors, such as the transients generated by appearance/disappearance of individual dots at the opaque boundary of the aperture (e.g., an aperture-inward bias; Wang, Merriam, Freeman, & Heeger, 2014). Importantly, the possible ambiguity about the precise computations underlying the signals generated by this condition are not problematic for interpreting our results, because our interest is not in the neural bases of motion perception, per se, but rather in the neural bases of attention to either of two visual features that had the subjective properties (for the participant) of being categorically different—one reproducible on a color bar and the other with a radial dial—and the objective properties (for the experimenters) of being varied along orthogonal dimensions—values in color space for static dots or direction of motion for color-invariant gray dots. For expository parsimony, from this point onward, we will refer to these feature dimensions as “color” and “direction.”

Each trial of the task of primary experimental interest (the MSR task) began with the serial presentation of two 2-D stimuli, each a patch of coherently moving colored dots. Next, a “relevance cue” designated what information from the two sample stimuli would be relevant for that trial: the color and direction of either the first or the second sample, or the color of both samples or the direction of both samples. Thus, a “relevance cue” indicating “〈First〉” or “〈Second〉” would designate a trial that would require selection of a feature bound to a 2-D object, whereas a “relevance cue” indicating “〈Color〉” or “〈Direction〉” would designate a trial that would require selection of the relevant feature of one from among two objects. Following the “relevance cue,” each trial proceeded in the same way as many previous D(ual)SR tasks: a first “prioritization cue” indicated which of the two relevant features would be tested by the first recall probe, then a second “prioritization cue” indicated, with a probability of .5, which of the same two relevant features would be tested by the second recall probe (Figure 1). For the remainder of this report, we will refer to trials when the “relevance cue” designated the “〈First〉” or “〈Second〉” sample stimulus as Bound trials (because they entailed prioritizing a feature that, when perceived in the sample display, was bound to another feature as part of a 2-D compound stimulus) and trials when the “relevance cue” designated the “〈Color〉” or “〈Direction〉” of the two sample stimuli as Unbound trials.

Experimental Procedure

Stimuli

All sample stimuli comprised 400 dots (0.08 in diameter) displayed within an invisible circular aperture (7.75° in diameter). Delayed recall tasks featured 1-D trials: On direction trials, gray dots (L = 38, a = 0, b = 0) moved with 100% coherence at a constant speed of 3.2°/sec in a direction that was randomly sampled (without replacement) from a list of 180 vectors that spanned the full space of 360° in increments of 2°. The dots were repeatedly redrawn on each frame of the monitor with a refresh rate of 60 Hz. The direction recall interface was a dial: a white circle (7.75° in diameter) presented centrally with a white radius line (.05° wide) extending from the center to the edge of the circle (like the “needle” of an analog speedometer). On each trial, the initial angle of the needle was determined randomly, and it could be made to rotate in a clockwise or counterclockwise direction by movement of a trackball.

On color trials, the dots were stationary and appeared in a color that was randomly sampled (without replacement) from of a list of 180 values that spanned the full color space of 360° in increments of 2°. This color list was generated from an evenly distributed circle on the CIE L*a*b color space, centered at (L = 80, a = 20, b = 20) with a radius of 60. The white point for the CIE XYZ space defining the LAB colors was set to (1, 1, 1). All colors had an equal luminance and brightness and only varied in hue. The color recall interface was a horizontal bar (12.14° × 1.55°) appearing at the center of the screen, its color transitioning smoothly across all possible colors in the color space, and a superimposed vertical white line (0.78° long, 0.05° wide; like the analog “tuning bar” on a radio). On each trial, the initial position of the tuning bar was determined randomly, and it could be made to translate horizontally along the color bar, to the left or to the right, by movement of the trackball.

Behavioral Tasks

Scanning was performed across three sessions, each on a different day: Days 1–16, 15-trial blocks of one-item delayed recall; Day 2–8, 15-trial blocks of one-item delayed recall (for a total of 360 trials of delayed recall) plus six 10-trial blocks of MSR; Day 3–10, 10-trial blocks of MSR (for a total of 160 trials of MSR). Before the start of each scanning session, participants were given instructions, and the tasks were practiced both outside and inside the scanner. The task procedures in each phase are described in detail below.

Delayed Recall

Each trial started with a central presentation of a white fixation cross (0.78° width and height) against a black background (2 sec), followed by the central presentation of the sample (1 sec). The sample was either a patch of coherently moving gray dots or a patch of static dots all presented in a uniform color. The white cross returned during the subsequent 9-sec delay period, after which recall was prompted during a 3-sec window. On direction trials, participants were instructed to click a response key when they had rotated the needle to an angle that matched their memory of the direction of motion of the sample dots. For both trial types, the needle/tuning bar became thicker (0.13°) when the response was registered and remained at the selected position for the remainder of the 3-sec response window, followed by feedback (1 sec; errors ≤15° elicited “great,” >15° and <30° elicited “good,” and ≥30° elicited “poor”), followed by a gray fixation cross displayed throughout the 8-sec intertrial interval (ITI). Trial type was pseudorandomly interleaved across the whole experiment, with each trial type being equiprobable, hence unpredictable to the participants.

Multiple Serial Retrocuing

In the MSR task, sample stimuli were 2-D compound objects that combined the features of the delayed recall stimuli: patches of moving, colored dots. After two samples were presented, a “relevance cue” indicated what would be the critical to-be-remembered information for that trial: cues indicating “〈First〉” or “〈Second〉” designated one of the two initially presented stimuli (i.e., the Bound condition); whereas cues indicating “〈Color〉” or “〈Direction〉” designated one of the two initially presented features (i.e., the Unbound condition). Then, the remainder of all trials unfolded with two serially occurring sequences of “prioritization cue”-delay-probe (Figure 1), with each prioritization cue indicating which feature would be tested by the ensuing recall probe. The logic was that memory load was equated across both conditions—four feature tokens were initially presented, then the “relevance cue” indicated which two of these four were relevant for the trial, presumably reducing the memory load to two feature tokens—and the factor of principal theoretical interest was whether the two trial-relevant feature tokens were bound together in the same object or were drawn from two discrete objects. Note that, in this design, the factor boundedness is confounded with category homogeneity, in that Bound trials always required memory for a color and a direction whereas Unbound trials always required memory for two colors or for two directions. However, because previous studies have shown a drop-to-baseline of the MVPA decodability of the UMI regardless of whether the two (unbound) memory items are drawn from the same or from different categories (LaRocque et al., 2013, 2017; Lewis-Peacock & Postle, 2012), this confound was deemed unlikely to complicate the interpretation of the results.

Whereas “relevance cues” presented a single word displayed in brackets—“〈First〉” or “〈Second〉” for Bound trials; “〈Direction〉” or “〈Color〉” for Unbound trials—“priority cues” used the same four words but without brackets. Note that the same word could never appear as both types of cues on the same trial (i.e., after a “relevance cue” of “〈First〉” or “〈Second〉,” the subsequent “priority cues” could only be “Direction” or “Color,” and vice versa). In both conditions, Priority Cue 2 was equally likely to cue the feature token that had or that had not been cued by Priority Cue 1, resulting in 20 “stay” trials—in which the same feature token was probed twice—and 20 “switch” trials, per cell in our design. The total duration of a trial was 52 sec, with participants performing 160 trials in randomized order across 16 blocks of 10 trials each. All stimulus parameters were the same as in the delayed recall task unless specified otherwise, and trial timing is illustrated in Figure 1. Both feature dimensions were randomly drawn (with replacement) from the full 360° of their respective feature spaces, in increments of 1°. The feature values of the second sample were constrained to a minimum angular separation of 40° relative to the first sample.

Behavioral Data Analysis

Performance was assessed using both a descriptive approach and a model-fitting approach. For each, a continuous measure of error for each response was obtained as the angular distance between the reported feature value and the true feature value. For the descriptive approach, a precision measure was then calculated as the reciprocal of the standard deviation of the error (calculated with Fischer's formula with a correction for systematic underestimation as outlined in Bays, Catalao, & Husain, 2009; paulbays.com/). The descriptive precision measures for each of the two probes were then submitted to a 2 × 2 repeated-measures ANOVA, with Category selected by the “relevance cue” (Bound or Unbound) and Feature Dimension (direction or color) as within-participant factors. Trials on which no responses were given were excluded from the analyses (3%). An alpha level of .05 was applied, and Bonferroni correction was used on multiple tests to control for false-positives in post hoc testing.

The model-fitting analysis used a mixture model that decomposes the sources of error into a mixture of Gaussian variability around the target color (PT), a Gaussian variability around the nontarget (PNT; also referred to as “misbinding” or “swap” errors), and a fixed probability of random guessing (PU). This model applied to the current data set can be described as follows:
pθˆ=1γβϕκθˆθ+γ12π+βϕκθˆθ,
where θ is the target feature (probed), θˆ is the value reported by the participants, and ϕκ is the von Mises distribution (circular analog of the Gaussian), with a mean of zero and concentration parameter κ. The probability of misremembering the target feature is β with the orientation value of the nontarget feature. The probability of random guessing is captured by γ. Maximum likelihood estimates of the parameters κ, β, and γ were separately obtained for each participant in each category and feature dimension conditions only for the first probe. The model components were then independently subjected to a univariate repeated-measures ANOVA with Relevance Cue (Bound vs. Unbound) and Feature Dimension (color vs. direction) as a within-participant factor. Mixture modeling was only applied to the data from Probe 1 because there was not a sufficient number of trials per condition to fit the model to the Probe 2 data (due to the additional factor of “stay” vs. “switch”).

Data Acquisition and Preprocessing

Whole-brain images were acquired with the 3-T MRI scanner (Discovery MR750; GE Healthcare) at the Lane Neuroimaging Laboratory at the University of Wisconsin–Madison. High-resolution T1-weighted images were acquired for all participants with a fast spoiled gradient echo sequence (TR = 8.132 msec, time echo = 3.18 msec, 12° flip angle, 156 axial slices, 256 × 256 in-plane, 1.0 mm isotropic). BOLD-sensitive data were acquired using a gradient-echo, echoplanar sequence (TR = 2 sec, time echo = 25 msec) within a 64 × 64 matrix (39 sagittal slices, 3.5 mm isotropic).

fMRI Data Analysis

The fMRI data were analyzed with Analysis of Functional NeuroImages (AFNI) software package (afni.nimh.nih.gov; Cox, 1996). The first three volumes of each EPI series were included to allow magnetic saturation and were then removed from the analysis. The preprocessing pipeline for each day of scanning included the following steps. All volumes were spatially realigned to the first volume of the first functional run using rigid body realignment. Slice time correction was then applied to these functional volumes. Skull-stripped anatomical images were generated to which the functional images were coregistered to. Linear, quadratic, and cubic trends were removed from each run to reduce the influence of scanner drift. Data were then converted to percent signal change. For univariate analyses, data were spatially smoothed with 4 mm FWHM Gaussian kernel and transformed into Talairach space. The (inverted) transformation parameters obtained in this step were then later used in the generation of the masks by transforming them from Talairach space to the native subject space. For mutlivariate analyses, data were z-scored separately within run for each voxel and left in their native space. As a common native space for all the volumes acquired across different days of scanning, we defined the volumes of the second day of scanning as a reference to which the volumes of Days 1 and 3 were realigned to.

Generation of ROIs

ROIs were generated as a conjunction of anatomically and functionally defined voxels.

First, anatomical ROIs were generated using the Talairach anatomical atlas (TTatlas; https://sscc.nimh.nih.gov/afni/doc/misc/afni_ttatlas/index_html). Coordinates for relevant gyri in the TTatlas were used to generate masks for each gyrus, which were then warped into an individual's native space and aggregated to create three regional masks. The frontal anatomical mask comprised the precentral, anterior cingulate, inferior frontal, middle frontal, superior frontal, and medial frontal gyri. The parietal anatomical mask was similarly generated and comprised the posterior cingulate gyrus, precuneus, inferior parietal, and superior parietal lobules. Importantly, this included the intraparietal sulcus. The ventral occipitotemporal (VOT) mask comprised the lingual, fusiform, inferior occipital, inferior temporal, middle occipital, superior occipital gyri, and the cuneus.

Next, we fit a general linear model, separately for each participant, to the data from the delayed recall task. Regressors of interest were delta functions placed at the beginning of stimulus onset, and a nine second boxcar modeling the delay period, all convolved with a canonical hemodynamic response function. Nuisance covariates modeled head motion and block effects. From the solution of the general linear model, we extracted, from each anatomical region, the top 400 voxels with the highest positive t statistic associated with each of several contrasts: [Samplecolor − baseline], [Sampledirection − baseline], [Delaycolor − baseline], and [Delaydirection − baseline] to construct “feature-selective” ROIs; [(Samplecolor+direction) − baseline] and [(Delaycolor+direction) − baseline] to construct “feature-nonselective” ROIs. Of the resultant functionally defined ROIs, different instantiations would be most suitable for different analyses.

Multivariate Inverted Encoding Modeling

Conceptually, an IEM affects a projection of the data from a large number of individual sensors into a single population-level, distributed representation (Brouwer & Heeger, 2009). To implement it, we followed the steps laid out in Ester, Sprague, and Serences (2015), first, for each feature dimension (color and direction), modeling the response of each voxel as a linear sum of nine hypothetical information channels, which, taken together, spanned the full stimulus space for each feature. This relationship can be expressed in the form of the following general linear equation:
B1=WC1
(1)
where B1 (v voxels × n trials) is the observed BOLD response in each voxel in each trial, C1 (k channels × n trials) reflects the expected responses for each information channel on each trial. For each feature, a basis set of nine modified von Mises functions (Equation 4 below) was generated, each centered (by varying the μ parameter) around one of nine orientations, each 40° apart (at 20°, 60°, 100° through 340°), so as to cover the full 360° feature space. Each basis function can be construed as a channel in stimulus representation space. We set the α, k, and β parameters to 1, 7, and 0, respectively, to best approximate tuning properties of middle temporal neurons (Duijnhouwer, Noest, Lankheet, van den Berg, & van Wezel, 2013). To model color, we extracted a circular portion of L*A*B color space using the procedure similar to that outlined in Brouwer and Heeger (2009).
The first step in implementing the model (the training phase) is to regress a portion of the voxel data B1 (v voxels × n trials) against the basis set, using ordinary least squares regression (Equation 2), to generate a weight matrix:
Wˆ=B1C1TC1C1T1
(2)
This weight matrix W (v voxels × k channels) constitutes a mapping from “channel space” to “voxel space.” Thus, W represents the relative strength (or weight) of the contribution each channel makes to the voxel's overall response. This set of weights is sometimes referred to as a population receptive field or a voxel tuning function.
In the second step (the testing phase), the weight matrix is inverted, such that it now constitutes a mapping from “voxel space” to “channel space,” and applied to the remaining voxel data, B2 (v voxels × n trials), to generate an estimated representation in channel space, C2 (k channels × m trials):
C2=WˆTWˆ1WˆTB2
(3)

As an initial validation step, we implemented a leave-one-run-out cross-validation procedure where, for each fold, one of the runs from the delayed recall task was set aside and each time point from the remaining runs was used to generate a weight matrix for each feature dimension (color and direction) within each ROI. We then inverted the weight matrix and applied it to data from the left-out run to generate reconstructions in channel space (also referred to as “channel tuning functions”). Reconstructions from each iteration of the leave-one-run-out procedure were then aligned and averaged together to generate reconstructions for the delayed recall task, which we then quantified using the procedure outlined below. These results are shown in Figure 4. Important to note is that the results from the analyses of the one-item delayed recall data indicated that the IEM reconstruction of the neural representation of direction was markedly superior than that for color and, furthermore, that the reconstruction of direction was markedly stronger in the VOT (Figure 4) than in the parietal and frontal ROIs (Figures 10 and 11). Therefore, to maximize the sensitivity for addressing our questions of principal interest, we focused on the representation of direction in the VOT ROI by training a “one-item delay” IEM from TR 6 of the delayed recall task and testing it on data from the MSR task.

Once the weight matrix was generated from delayed recall data, data from each time point in the MSR task (testing phase) was multiplied by the inverted weight matrix as described in Equation 3 to generate a reconstruction time course of direction. Each of these feature-specific reconstruction time courses were then circularly shifted to a common center (0°) and averaged with those from like trials. Thus, for example, to generate the “Prioritized” reconstructions for the Unbound condition (Figure 5A), channel outputs from trials for which “〈Direction〉” was “relevance”-cued and for the item that was cued by Priority Cue 1 were aligned along the “priority”-cued item's direction and averaged together. To generate the smooth, 360-point functions shown in the IEM figures (6 and 7), we repeated the IEM analysis a total of 39 times and shifted the centers of the direction or color channels by 1° on each iteration.

Reconstructions were then quantified using a bootstrapping procedure similar to Ester et al. (2015). In each ROI, for each feature dimension, each time point, and each condition, reconstructions from all nine participants were randomly sampled with replacement nine times to generate a 9 × 360 dimension resampled reconstruction matrix. This resampled matrix was averaged across the first dimension (participants), yielding an averaged reconstruction that was then fit with the following von Mises response function:
fx=αeκcosμx1+β
(4)
where x is a vector of 360 channel responses. μ, κ, and β correspond to the center (i.e., mean), concentration (i.e., inverse of width), and baseline (i.e., vertical offset) of the function, respectively, and α corresponds to the amplitude of the function (i.e., vertical stretching/scaling). Fitting was performed using a combination of a grid search procedure and ordinary least squares regression. The response function (Equation 4) was defined by setting the μ parameter to 0 (reflecting our alignment across trials to a 0° centered channel) and the β parameter to 1. A range of plausible κ values was then defined (from 1 to 30 in increments of 0.1). For each κ value, a design matrix was then generated containing the response function and a constant term (i.e., a vector of 1 sec). Fitting the reconstructed feature tuning curves yielded estimates of α and β, the regression coefficients for the design matrix and constant terms, respectively. The best fitting curve that minimized the sum of squared errors between response function and data was then selected.

This procedure was repeated 10,000 times in total, yielding 10,000 bootstrapped estimates of amplitude, baseline, and concentration. To test whether the amplitudes of the PMI and UMI representations were significantly above baseline levels, (one-tailed) p values for the robustness of the PMI and UMI feature reconstructions were separately calculated in the Unbound and Bound conditions by assessing the percentage of bootstrapped iterations whose amplitude estimates were negative. In other words, statistical significance at an alpha level of .05 implies that at least 95% of resampled reconstructions have a positive amplitude (ppos).

To test whether feature reconstructions of PMIs were stronger than those of UMIs, we computed the difference between the bootstrapped amplitudes of the PMI and UMI reconstructions, forming a distribution of difference scores. We assessed the percentage of bootstrapped iterations whose amplitudes were negative. In other words, statistically significant differences (at p < .05) between the PMI and UMI conditions would indicate that 95% of the differences in the resampled amplitude estimates of the PMI and UMI reconstructions were positive (ppos). The difference between the PMI and UMI reconstructions were separately calculated in the Unbound and Bound conditions. The same principle of statistical testing was applied to the baseline parameter. We were particularly interested in the attentional modulations of feature representations late in the delay period, namely, in time points 15 and 21 in response to Priority Cue 1 and Priority Cue 2, respectively. Therefore, the tests in the results sections are mainly focused on these time points (see Figure 6). However, reconstructions of the entire time courses of the delay periods of both priority cues are presented in Figures 14 and 15, along with the statistics on the amplitude and baseline differences between PMI and UMI feature reconstructions.

Delay 4, following the presentation of Priority Cue 2, differed from Delay 3 because only in the former could the uncued feature be dropped from WM, being no longer relevant for task performance. The results from Delay 4, therefore, could provide insight about whether the putative “dropping” operation (Lewis-Peacock et al., 2015) differs as a function of boundedness. Because analyses of data from Delay 4 could only use half of each participant's data, due to only switch trials affording a comparison of the transition of an item from PMI to UMI versus “irrelevant memory item” (IMI) following Priority Cue 1 versus Priority Cue 2, respectively, we restricted our analyses of Delay 4 of MSR trials to TR 21, the final TR expected to reflect delay period processing uncontaminated by the processing of Probe 2 (see, e.g., the BOLD time courses in Figure 1), and carried out the same analyses on TR 15, to allow for like-to-like comparison of stimulus feature representations between Delay 3 and Delay 4. Over the course of data analysis, for the reconstruction of some UMIs and IMIs, the output of the channels near 0° dropped below those of the flanking channels, generating “flipped” or “opposite” (cf. van Loon et al., 2018) reconstructions. Because these outcomes were not predicted, we evaluated whether these “negative reconstructions” differed significantly from 0 by calculating two-tailed p values: First, the probabilities of obtaining a positive (ppos) or a negative (pneg) amplitude among the 10,000 bootstrapped amplitudes were calculated; next, two-tailed p values were obtained using:
p=2×minppos,pneg
(5)

Multivariate Pattern Analyses

We carried out MVPAs to clarify and/or refine the interpretation of some of the findings from the IEM analyses, using L2-regularized logistic regression (with a lambda penalty term of 25) applied to z-scored BOLD, and implemented with the Princeton Multivoxel Pattern Analysis toolbox (www.pni.princeton.edu/mvpa/) and custom scripts in MATLAB (cf. Lewis-Peacock, Drysdale, Oberauer, & Postle, 2012). MVPA was carried out on two levels of stimulus information: (i) within feature (color and direction labeled as belonging to one of four quadrants in their respective 360° stimulus spaces, carried out in feature-selective ROIs) and (ii) between feature (color vs. direction, carried out in feature-nonselective ROIs).

To assess the representation of stimulus level of information, we trained two classifiers, one from color-sensitive voxels and one from motion direction-sensitive voxels as in the IEM analyses, separately for each ROI, to classify motion direction and color values categorized as belonging to one of four quadrants (centered at 45°, 135°, 225°, and 315°, each spanning a 90° wedge of positions within the full 360° range of possible colors and motion directions). Categorizing stimuli in this way enabled us to determine whether coarse stimulus information might be decodable in the frontal and parietal ROIs for which the IEMs failed to reconstruct exemplar-specific feature information. Classifiers were trained on late delay period data from the delayed recollection task TR 6, with k-fold cross-validation (train on 23 runs, test on the 24th), classification accuracy averaged across folds and compared against chance performance with two-tailed t tests (Bonferroni corrected). Note that, as we did for IEM analyses, we focused on the late delay period of the one-item delayed recall task for MVPA classifier training.

Additionally, to assess the representation of higher order information about stimulus category, we trained classifiers (in feature-nonselective ROIs) to accurately distinguish between the categories of color versus direction on data from the one-item delayed recall task. MVPA methods were the same as described above, with the exception that only two labels were used for training (“color,” “direction”), and so statistical significance was assessed with two-tailed t tests comparing accuracy to chance performance (50%).

Finally, to assess evidence for cognitive control-related activity, we also applied the category-level decoders trained on data from the one-item delayed recall task to data from the MSR task. Because a hallmark of control is that it should dynamically track changing contingencies within individual trials, we carried out these analyses by applying late-delay classifiers from the one-item delayed recall task to every TR of “switch” trials from the Bound condition of the MSR task. This would generate classification time courses for MSR trials that featured within-trial switches of priority between stimulus category. For this analysis, at each time point of the MSR task, and for each category, a measure of pattern similarity was computed between the voxel patterns for that TR and the late-delay patterns from the one-item delayed recall task. Using logistic regression, each category's pattern similarity score was then converted into “classifier evidence,” a value between 0 and 1 that can be interpreted as the extent to which the pattern at the tested TR matches the pattern learned by the classifier (i.e., conceptually similar to a correlation coefficient; cf. Lewis-Peacock & Postle, 2012; Polyn, Natu, Cohen, & Norman, 2005). Average classifier estimates were computed by sorting trials according to the feature dimension selected by Priority Cue 1 and the feature dimension selected by Priority Cue 2. Statistical significance of the evidence between the feature dimensions as function of the priority cues was computed by pairwise t tests at each time point (Bonferroni-corrected).

RESULTS

Behavioral

Model-free Measures

Analysis of the precision of responses revealed only main effects of Feature Dimension: Probe 1, F(1, 8) = 10.76, p < .05, ηp2 = .57, with an overall higher precision for direction (M = 2.99, SE = .61) than for color (M = 1.44, SE = .19) responses (other effects ns); Probe 2, F(1, 8) = 7.62, p < .05, ηp2 = .49, with an overall higher precision for direction (M = 2.46, SE = .49) than for color (M = 1.53, SE = .25) responses (other effects ns; Figure 2). Follow-up analyses comparing the probes revealed that the drop in precision for direction in response to Probe 2 relative to Probe 1 was statistically significant, t(8) = 4.25, p < .01, CI [.25, .86], whereas the precision for color did not statistically differ across probes, t(8) = −0.23, p = .82, CI [−.27, .22].

Figure 2. 

Model-free behavioral results. (A) Probe 1: Recall was less precise for color than for direction trials, but insensitive to binding status. (B) Probe 2: Recall was less precise for color than for direction trials, and insensitive to binding condition and to “stay”/“switch” status of Priority Cue 2. The y-axis represents the precision, which was calculated as the reciprocal of SD of errors across trials. As the parameter spaces for orientation and color are circular, we used Fisher's definition of SD for circular data (Fisher, 1993), subtracting the value expected for chance; therefore, a precision value of zero corresponds to responding at random. Note that we converted the angle degrees into radians defining the measure of the y-axis to be 1/rad.

Figure 2. 

Model-free behavioral results. (A) Probe 1: Recall was less precise for color than for direction trials, but insensitive to binding status. (B) Probe 2: Recall was less precise for color than for direction trials, and insensitive to binding condition and to “stay”/“switch” status of Priority Cue 2. The y-axis represents the precision, which was calculated as the reciprocal of SD of errors across trials. As the parameter spaces for orientation and color are circular, we used Fisher's definition of SD for circular data (Fisher, 1993), subtracting the value expected for chance; therefore, a precision value of zero corresponds to responding at random. Note that we converted the angle degrees into radians defining the measure of the y-axis to be 1/rad.

Mixture Modeling

Although inspection of results for κ (Figure 3A) suggests qualitatively similar patterns to those for precision, ANOVAs indicated, instead, a greater sensitivity to boundedness (main effect of Relevance Cue, F(1, 8) = 5.66, p < .05, ηp2 = .41, with κ higher in the Bound (M = 20.02, SE = 2.91) than the Unbound (M = 13.94, SE = 1.4) condition), and the difference between feature dimensions no longer meeting the threshold for significance, F(1, 8) = 4.84, p = .059, ηp2 = .38 (direction trials, M = 21.75, SE = 3.78; color trials, M = 12.22, SE = 1.50). The interaction between boundedness and feature dimension for κ did not reach significance (F < 1).

Figure 3. 

Mixture-model estimates for Probe 1 responses: (A) concentration parameter, (B) probability of response to target, (C) swap errors, and (D) guesses.

Figure 3. 

Mixture-model estimates for Probe 1 responses: (A) concentration parameter, (B) probability of response to target, (C) swap errors, and (D) guesses.

The greater difficulty of color than direction performance, as suggested by the descriptive statistics, was captured in the model's estimates of PT, F(1, 8) = 7.51, p < .05, ηp2 = .48 (with a higher PT for direction, M = .95, SE = .016, than for color, M = .89, SE = .036, responses; other Fs < 1), and of PNT, F(1, 8) = 8.17, p < .05, ηp2 = .51 (with a lower PNT for direction, M = .02, SE =.006, than for color, M = .059, SE =.016; other Fs < 1). There were no differences in PU (Fs < 4).

IEM Reconstructions

One-item Delayed Recall

Sample-evoked ventral occipitotemoral ROI.

In the sample-evoked VOT ROI, stimulus feature reconstructions were markedly superior for direction than for color. For direction, a model that was trained on TR 4 (i.e., the volume collected from 5 to 6 sec after sample onset, when the sample-evoked BOLD response was expected to be maximal; cf. Figure 1C) yielded a robust reconstruction when tested at that same TR (with leave-one-run-out k-fold cross-validation). Furthermore, sweeping this model across all 12 TRs of the trial yielded reliable reconstructions spanning from the first TR of the delay period through to the second TR after the response window, thereby demonstrating robust cross-temporal generalization and indicating that stimulus direction was represented, in part, with a perceptual neural code throughout the trial (Figure 4A). The same qualitative pattern of reconstruction was observed when this process was repeated with a model trained on data from TR 6, which was intended to capture signal primarily attributable to delay period processing (Figure 4B) Finally, direction reconstruction was achieved at all but the TR preceding sample presentation when a model was trained and tested at each TR (i.e., “along the diagonal” of a cross-temporal matrix; Figure 4C). For color, an IEM could be successfully trained at TR 4 but did not generalize to any other TRs (Figure 4D), and one trained at TR 6 generalized to TR 5 and TR 8 (Figure 4E). When models were trained and tested at each TR, the reconstruction of color information was successful for only a subset of TRs associated with the perception/encoding and retention of color information, as well as for TRs during the ITI that followed the probe (Figure 4F). Although one might expect poorer IEM reconstruction for the stimulus feature that was remembered less well (i.e., color recall was inferior to direction recall), it could also be the case that the reconstruction of neural representations of color would have been more robust had we attempted to optimize for IEM the plane of the slice through CIE space that we selected to generate our stimuli. Additionally, our method does not allow us to know the extent to which verbalization may have contributed to color WM performance, a possibility that would not be expected to produce robust IEM in VOT ROI.

Figure 4. 

Time courses of IEM reconstructions from the one-item delayed recall task, in feature-selective VOT ROIs. Rows illustrate results for direction (top: A, B, and C) and color (bottom: D, E, and F), and columns illustrate the results for three procedures: train on TR 4 (a “perception/encoding” model) and test at every TR (left: A and D); train on TR 6 (a “delay” model) and test at every TR (middle: B and E); and train and test at each TR (“along the diagonal”; right: C and F). In each panel, channel is arrayed along the horizontal axis, from −160° to 160°, time (in TRs) proceeds from top to bottom, and the dependent data are channel responses (averaged after aligning each trial to 0°). Significance of the reconstruction at each TR, determined by bootstrapping, is indicated by asterisks (*p < .05, **p < .01, ***p < .001; ****p < .0001).

Figure 4. 

Time courses of IEM reconstructions from the one-item delayed recall task, in feature-selective VOT ROIs. Rows illustrate results for direction (top: A, B, and C) and color (bottom: D, E, and F), and columns illustrate the results for three procedures: train on TR 4 (a “perception/encoding” model) and test at every TR (left: A and D); train on TR 6 (a “delay” model) and test at every TR (middle: B and E); and train and test at each TR (“along the diagonal”; right: C and F). In each panel, channel is arrayed along the horizontal axis, from −160° to 160°, time (in TRs) proceeds from top to bottom, and the dependent data are channel responses (averaged after aligning each trial to 0°). Significance of the reconstruction at each TR, determined by bootstrapping, is indicated by asterisks (*p < .05, **p < .01, ***p < .001; ****p < .0001).

Delay-evoked parietal and frontal ROIs.

In parietal cortex, successful reconstruction of sample direction was restricted to just a few TRs associated with encoding/early delay and with recall/response and no successful reconstructions of sample color (Figure 10). In frontal cortex, the pattern was similar, with the exception that there were a few successful reconstructions of sample color in ITI TRs (Figure 11).

Multiple Serial Retrocuing

The results from the analyses of the one-item delayed recall data indicated that the IEM reconstruction of the neural representation of direction was markedly superior than that for color and, furthermore, that the reconstruction of direction was markedly stronger in the VOT (Figure 4) than in the parietal and frontal ROIs (Figures 10 and 11). Therefore, to maximize the sensitivity for addressing our question of principal interest, we focused on the representation of direction in the VOT ROI by training a “one-item delay” IEM from TR 6 of the delayed recall task and testing it on data from the MSR task.

On trials of all types, the neural representation of direction in the VOT was robust during the delay period prior to the “relevance cue” (TR 6 in Figure 5).

Figure 5. 

Time course of the neural representation of direction during Unbound (A, B) and Bound (C, D), across the first half of the MSR task, as a function of priority status in feature-selective VOT ROIs. The top row of panels illustrates the Unbound trials, and the bottom row illustrates the Bound trials. The panels on the left side illustrate the reconstruction of the direction of motion when it was cued by Priority Cue 1, and the panels on the right side illustrate the reconstruction of the direction of motion when it was not cued by Priority Cue 1. All display conventions within each panel are the same as Figure 4.

Figure 5. 

Time course of the neural representation of direction during Unbound (A, B) and Bound (C, D), across the first half of the MSR task, as a function of priority status in feature-selective VOT ROIs. The top row of panels illustrates the Unbound trials, and the bottom row illustrates the Bound trials. The panels on the left side illustrate the reconstruction of the direction of motion when it was cued by Priority Cue 1, and the panels on the right side illustrate the reconstruction of the direction of motion when it was not cued by Priority Cue 1. All display conventions within each panel are the same as Figure 4.

Unbound condition, Direction relevance cue.

The reconstructions of the direction of the sample that would become the PMI (Figure 5A) and of the sample that would become the UMI (Figure 5B) were both robust during the TRs leading up to and immediately following Priority Cue 1 (TR 11), with statistically comparable reconstructions from the data collapsed across TR 9, TR 10, and TR 11 (two-tailed, p = .69).

Once Priority Cue 1 designated one item to be the PMI and the other to be the UMI, the amplitude of the reconstructions diverged markedly. IEM of the PMI produced robust delay period reconstructions from TR 12 through TR 15, with the reconstruction strengthening across the delay preceding Probe 1 (slope = 0.039, p < .05). For IEM of the UMI, in contrast, the amplitude of the 0° channel declined significantly from TR 12 to TR 15 (slope = −0.069, p < .001), and no reconstructions were reliable from TR 13 to TR 16. Indeed, the output of the channels near 0° channel dropped below that of flanking channels, although this trend toward a significantly “negative” channel tuning function did not achieve significance when the data were collapsed across TR 13, TR 14, and TR 15 (two-tailed, p = .27). Finally, statistical comparisons confirmed that reconstructions of the PMI were higher in amplitude than those of the UMI (signal from both collapsed across TR 13 through TR 15, one-tailed, p < .0001).

No reconstructions were reliable in the parietal and frontal ROIs. Unexpectedly, however, in all ROIs, the overall magnitude of IEM channel outputs increased markedly from Delay 1 (TR 4 through TR 6) to Delay 2 (TR 8 through TR 10), as reflected in significant increases in the values of the baseline parameter in the VOT (p < .001, two-tailed) and parietal (p < .01, two-tailed) ROIs; for the frontal ROI (p = .11). Importantly, these effects were not accompanied by changes in the overall BOLD signal intensity. (Possible interpretations of changes in baseline will be considered in the Multivariate Pattern Classification section.)

The IMI following Priority Cue 2.

At TR 21, on “stay” trials, neither the PMI nor the IMI could be reconstructed (Figure 6B). On “switch” trials, however, the IEM reconstruction of the newly cued PMI (which had been flat and nonsignificant during the previous delay; Figure 6A) was robust at the end of Delay 4 (TR 21) and that of the IMI trended in the opposite direction: that is, the reconstruction displayed minimal channel output along the aligned direction and maximal output for channels corresponding to the opposite direction. Because such a flipped reconstruction was not predicted, we assessed its significance with a two-tailed test, which indicated that it missed the threshold for significance (p = .06; Figure 6C). At TR 21, the PMI and IMI reconstructions differed in amplitude (p < .01) and in baseline (p < .05, two-tailed). Overall, the IEM reconstructions of the PMI were significantly stronger than the IMI in the entire interval following Prioritization Cue 2 (signal from both collapsed across TR 19 through TR 21, one-tailed, p < .001).

Figure 6. 

The reconstruction of the neural representation of direction as function of priority status from individual TRs of the Unbound (top) and Bound trials (bottom) in the feature-selective VOT ROI: at TR 15 (immediately prior to Probe 1; A, D), TR 21 (immediately prior to Probe 2; B, E) on “stay” trials, and TR 21 (C, F) on “switch” trials. Note that, at TR 15, prioritized memory information (PMI, red) and the unprioritized memory information (UMI, blue) are both still task relevant as both stimuli are equally likely to be prioritized by Priority Cue 2. At TR 21, however, only the PMI is now relevant for the impending Probe 2, and the uncued feature has become the “irrelevant memory information” (IMI, blue). In each plot, channel is arrayed along the horizontal axis, and the dependent data are channel responses (averaged after aligning each trial to 0°). The width of each reconstruction trace represents the SEM across participants, interpolated across the 360 discrete data points in the averaged data. Significance of the reconstructions of the PMI and UMI is indicated by red and blue asterisks, respectively (*p < .05, **p < .01, ***p < .001, ****p < .0001). Significance of the difference between the baseline parameter of the PMI versus the UMI reconstruction are indicated with black circles (• = p < .05; •• = p < .01).

Figure 6. 

The reconstruction of the neural representation of direction as function of priority status from individual TRs of the Unbound (top) and Bound trials (bottom) in the feature-selective VOT ROI: at TR 15 (immediately prior to Probe 1; A, D), TR 21 (immediately prior to Probe 2; B, E) on “stay” trials, and TR 21 (C, F) on “switch” trials. Note that, at TR 15, prioritized memory information (PMI, red) and the unprioritized memory information (UMI, blue) are both still task relevant as both stimuli are equally likely to be prioritized by Priority Cue 2. At TR 21, however, only the PMI is now relevant for the impending Probe 2, and the uncued feature has become the “irrelevant memory information” (IMI, blue). In each plot, channel is arrayed along the horizontal axis, and the dependent data are channel responses (averaged after aligning each trial to 0°). The width of each reconstruction trace represents the SEM across participants, interpolated across the 360 discrete data points in the averaged data. Significance of the reconstructions of the PMI and UMI is indicated by red and blue asterisks, respectively (*p < .05, **p < .01, ***p < .001, ****p < .0001). Significance of the difference between the baseline parameter of the PMI versus the UMI reconstruction are indicated with black circles (• = p < .05; •• = p < .01).

Interim summary of results from Unbound condition.

For trials on which the “relevance cue” indicated “〈Direction〉,” the subsequent Priority Cue 1 influenced the representation of both features held in WM: The representation of the PMI increased in strength across the subsequent Delay 3, whereas the representation of the UMI decreased in strength to the point that it could no longer be reconstructed by the end of Delay 3. This is the pattern of results that would be expected if the principles of biased competition apply to the selection of one from among two WM representations of stimuli in the same way that they do for the selection of one from among two objects in a visual scene (e.g., Sheldon et al., Submitted). On trials when Priority Cue 2 prompted a switch of prioritization status, the IEM reconstruction of the newly designated PMI increased over the course of Delay 4, whereas that of the newly designated IMI decreased over the course of Delay 4, from being significantly positive at the beginning (Figure 6A) to being nonsignificant and, indeed, by the end of the delay period, bordering on flipped relative to the trained model. Additionally, the baseline parameter of this flipped reconstruction of the UMI during Delay 4 was higher than that of the PMI.

Bound condition.

After the “relevance cue” indicated “〈First〉” or “〈Second〉,” the reconstructions from the VOT ROI of the direction of the sample that would become the PMI (Figure 5C) and of the sample that would become the UMI (Figure 5D) were robust during the TRs leading up to and immediately following Priority Cue 1 (TR 11), with statistically comparable reconstructions from signal collapsed across TR9, TR10, and TR11 (two-tailed, p = .77). Unlike in the Unbound condition, however, the designation by Priority Cue 1 of the PMI and the UMI had only a relatively minor effect on the IEM reconstructions. Statistical comparisons confirmed that reconstructions of the PMI and UMI did not differ over the course of this delay period (signal from both collapsed across TR 13 through TR 15, one-tailed, p = .14). Although the IEMs of the PMI reconstructions were sustained across the ensuing delay period, their strength did not increase (slope = 0.026, p = .25). Furthermore, although the amplitude of the reconstructions of the UMI decreased across this delay period (slope = −0.050, p < .05), they remained statistically significant across the delay period, and only beginning with TR 15 did the reconstruction of the UMI decline in amplitude to a point at which it was significantly lower than that of the PMI (p < .05).

Results from the Bound condition also differed markedly from the Unbound condition in the parietal and frontal ROIs, in that representations of the direction of the PMI became significant with the onset of Priority Cue 1 and for a few TRs into the ensuing Delay 3 (Figures 12 and 13), as well as in the frontal ROI of the UMI for a single TR.

The overall magnitude of IEM channel outputs increased from Delay 1 to Delay 2, as reflected in the values of the baseline parameters, although, as with the Unbound condition, this increase only reached significance in the VOT (two-tailed, p < .0001) and parietal (two-tailed, p < .05) ROIs.

The IMI following Priority Cue 2.

The patterns at late Delay 4 (TR 21) mirrored those from the Unbound condition: On “stay” trials, neither the PMI nor the IMI could be reconstructed (Figure 6E); on “switch” trials, reconstruction of the PMI was significantly positive (p < .0001) and that of the IMI was nonsignificant, but trending toward a flipped reconstruction (p = .09, two-tailed), and the two differed significantly from each other (p < .0001). Moreover, the overall IEM reconstructions of the PMI were significantly stronger than the UMI (signal from both collapsed across TR 19 through TR 21, one-tailed, p < .001).

Interim summary of results from Bound condition.

On trials when “direction” was cued by Prioritization Cue 1, the IEM reconstruction of the PMI remained robust across the ensuing delay period, but it did not increase in strength. On trials when “color” was cued by Prioritization Cue 1, although the amplitude of the neural representation of the UMI declined across the ensuing delay period, it nonetheless remained significantly elevated throughout the delay (Figure 5). In comparison to the Unbound condition, these results are consistent with the idea that a cardinal principle of object-based attention may apply to visual WM in a manner similar to visual perception: When one feature of an object is selected, the benefits of attention extend to all features of that object (e.g., Egly, Driver, & Rafal, 1994; Duncan, 1984). Although we cannot rule out the possibility that the UMI may have also benefited from the allocation of spatial attention to the stimulus of which it was a part, this possibility seems unlikely because, by this account, spatial attention would also be expected to boost the strength of the IMI, but this was not observed: Following Prioritization Cue 2, the representation of the PMI at TR 21 was positive, and the representation of the IMI was nonsignificant (and, indeed, trending in a direction opposite to what a spatial attention account would predict). Because this pattern mirrored what was observed in the Unbound condition, the implication is that the dynamics of “dropping” a no-longer-needed feature from WM may be similar across these two conditions.

Comparisons between conditions.

To quantify comparisons across conditions, we first established that the amplitudes of the reconstructions of direction that would become the PMI during Delay 3 (i.e., at TR 12–TR 15) were comparable during the time immediately preceding Priority Cue 1 (i.e., at TR 9–TR 11, p = .31; ns). Next, comparison of the change in PMI amplitudes across Delay 3 indicated that the strengthening of the reconstruction of the PMI that was observed in the Unbound condition did not differ significantly from the flat slope of the amplitude of the PMI in the Bound condition (slope difference = .01; two-tailed, p = .74). Finally, at TR 15, the amplitudes of the PMI did not differ between Bound and Unbound conditions (two-tailed, p = .52). Turning to the UMI, we first established that the amplitudes of the representations of direction that would become the UMI during Delay 3 were comparable during the time immediately preceding Priority Cue 1 (i.e., at TR 9–TR 11: two-tailed, p = .62; ns). Next, comparison of the slopes of the decline in the strength of the UMI across Delay 3 indicated that the weakening of the reconstruction of the UMI across Delay 3 did not differ significantly between the two conditions (slope difference = −0.019, p = .55). At TR 15, however, the amplitudes of the UMI differed between Bound and Unbound conditions (two-tailed, p < .05). Across the delay period (collapsing across the TR 13 through TR 15), the magnitude of the PMI–UMI difference was not different between Unbound and Bound conditions (two tailed, p = .31).

For Delay 4, on “switch” trials, the pattern was qualitatively similar for Bound and Unbound conditions: Reconstructions at TR 21 were significantly positive for the PMI and trending toward flipped for the IMI. The amplitudes of the PMI did not differ between Bound and Unbound conditions (two-tailed, p = .31), nor did the amplitudes of the IMI differ between Bound and Unbound conditions (two-tailed, p = .30). The two conditions differed quantitatively, however, in that, across the delay period (collapsing across TR 19 through TR 21), the magnitude of the PMI–IMI difference was greater in the Unbound than in the Bound condition (p < .05, two-tailed).

Interpretation of the differences in the Bound versus Unbound UMI at TR 15 and IMI at TR 21 must be qualified by the fact that the factor of binding (bound/unbound) was confounded with category congruity (different/same). For example, the lower amplitude of the Unbound UMI at TR 15 could be due in part to the fact that the PMI on these trials was also a direction of motion, which may have resulted in greater interitem interference relative to Bound trials, on which the PMI was a color. One argument against this alternative account is that, although such interitem interference might also be expected to impede the transition of the Unbound feature from UMI to PMI on switch trials, this transition was, to the contrary, greater than it was on Bound trials. In addition to this line of argumentation, however, it was also important to find evidence in these for a same-object effect for contrasts to which this confound did not apply. This was accomplished by examining patterns of IEM reconstructions of motion during epochs when color was the cued stimulus feature.

Color-cued trial epochs.
Unbound condition, Color relevance cue.

When the relevance cue indicated that the colors of the two stimuli were the critical to-be-remembered features, the strength of the neural representation of the direction of motion of the two stimuli could not be reconstructed at the end of Delay 2 (TR 10; Figure 7B). There remained, however, evidence of an effect of feature binding on these trials, in that Priority Cue 1 had the effect of dissociating the strength of the reconstructions of the two no-longer-relevant directions of motion: Although the reconstruction of the direction associated with the cued color was flat at the end of the ensuing delay period (TR 15), that of the direction associated with the uncued color became significant in the flipped direction (p < .0001), and the two differed significantly from each other (p < .0001; Figure 7D).

Figure 7. 

The effects of relevance-cued “dropping” of direction information from WM. (A) Reconstructions of direction at TR 7, the time of onset of the “relevance cue.” (B) The reconstruction of direction, at TR 10, on trials when the “relevance cue” had specified “〈Color〉,” and so the direction associated with both objects could be dropped. (C) The reconstruction of direction, at TR 10, on trials when the “relevance cue” had specified “〈First〉” or “〈Second〉,” object as being task-relevant (i.e., the red trace indicating the cued and the blue trace indicating the uncued object). (D) The reconstruction of the dropped directions in B, now at TR 15, as a function of Priority Cue 1 specifying the prioritized target object (red) and dropped object (blue). (E) The reconstruction of the directions in C, now at TR 15, as a function of Priority Cue 1 specifying the prioritized feature of the relevance cued object (red) and the dropped object (blue). All display conventions are the same as Figure 8.

Figure 7. 

The effects of relevance-cued “dropping” of direction information from WM. (A) Reconstructions of direction at TR 7, the time of onset of the “relevance cue.” (B) The reconstruction of direction, at TR 10, on trials when the “relevance cue” had specified “〈Color〉,” and so the direction associated with both objects could be dropped. (C) The reconstruction of direction, at TR 10, on trials when the “relevance cue” had specified “〈First〉” or “〈Second〉,” object as being task-relevant (i.e., the red trace indicating the cued and the blue trace indicating the uncued object). (D) The reconstruction of the dropped directions in B, now at TR 15, as a function of Priority Cue 1 specifying the prioritized target object (red) and dropped object (blue). (E) The reconstruction of the directions in C, now at TR 15, as a function of Priority Cue 1 specifying the prioritized feature of the relevance cued object (red) and the dropped object (blue). All display conventions are the same as Figure 8.

Bound condition, Color Priority Cue 1.

When the relevance cue indicated that one of the two stimuli would be relevant for the remainder of the trial, the reconstruction of the direction of the relevance-cued stimulus was highly significant at the end of Delay 2 (TR 10; Figure 7C, p < .001), whereas that of the uncued stimulus became flipped (TR 10; Figure 7C, p < .01), and the baseline parameter differed between the two (TR 10; Figure 7C, p < .0001). When Priority Cue 1 then cued color, the reconstruction of the direction of that stimulus was significantly above baseline at the end of the ensuing delay period (TR 15; p < .05), whereas the reconstruction of the direction of the irrelevant stimulus was flat, and the two differed significantly (p < .05).

Interim summary of results from color-cued trials.

These analyses provided evidence of a same-object effect in conditions that were not complicated by confounding factors. After a Color relevance cue, direction was irrelevant for the remainder of the trial. Nonetheless, IEM reconstructions of direction differed significantly after Priority Cue 1 prompted the prioritization of one of the two colors, with the reconstruction of the direction linked to the uncued color becoming significantly flipped. On bound trials, when Priority Cue 1 designated the relevant object's color, the reconstruction of that object's direction was significant at the end of the ensuing delay period and significantly greater than the flat reconstruction of the direction of the no-longer-relevant object.

Multivariate Pattern Classification

Delayed Recall Task

After binning stimuli into arbitrarily defined quadrants, direction of motion could be successfully decoded from the VOT ROI, t(8) = 3.45, p < .05, but not from the parietal or frontal ROIs (ts < 1.36; Figure 8A). Decoding of color was unsuccessful in all ROIs (ts < 1.03; Figure 12A). Decoding of trial type (color vs. direction) was successful in all ROIs: VOT, t(8) = 6.72, p < .001; parietal, t(8) = 10.61, p < .001; and frontal, t(8) = 8.4, p < .001 (Figure 8B).

Figure 8. 

MVPA decoding of stimulus feature information from the late delay period of the one-item delayed recall task. (A) Performance of classifiers trained on data labeled according to post hoc-defined quadrants in 360° stimulus space. (B) Performance of classifiers trained on data labeled according to feature category (i.e., color vs. direction). Statistically reliable classification is denoted by asterisks (*p < .05, **p < .01, ***p < .001, ****p < .0001).

Figure 8. 

MVPA decoding of stimulus feature information from the late delay period of the one-item delayed recall task. (A) Performance of classifiers trained on data labeled according to post hoc-defined quadrants in 360° stimulus space. (B) Performance of classifiers trained on data labeled according to feature category (i.e., color vs. direction). Statistically reliable classification is denoted by asterisks (*p < .05, **p < .01, ***p < .001, ****p < .0001).

MSR Task

MVPA of stimulus category.

Although parietal and frontal cortex are generally associated with the control of WM (e.g., Brincat, Siegel, von Nicolai, & Miller, 2018; Pribram, Ahumada, Hartog, & Ross, 1964), for the successful decoding of trial type (e.g., Figure 8B) to be interpreted as reflecting control-related activity, one would want to see that it dynamically tracks changing contingencies within individual trials. We assessed this possibility by applying late-delay classifiers from the one-item delayed recall task to every TR of “switch” trials from the Bound condition of the MSR task, so as to generate classification time courses. (Note that only the Bound condition included within-trial switches between stimulus categories.) These analyses, carried out in feature-nonselective ROIs in parietal and frontal cortex, revealed that information about color and direction were represented, to the same extent, from the beginning of the trial until TR 11 (the onset of Prioritization Cue 1). At this point in the trial, classifier evidence for the cued feature increased steeply and evidence for the uncued feature decreased steeply. At TR 17 (the onset of Prioritization Cue 2, a “switch” cue in these analyses), these patterns reversed, with evidence for the newly prioritized feature rising precipitously, and evidence for the newly unprioritized falling precipitously (Figure 9A).

Figure 9. 

The time course of model performance on “switch” trials from the Bound condition of the MSR task. (A) Trial-averaged time courses of MVPA evidence for the representation of direction of motion, on trials when motion was cued by Priority Cue 1 (red) versus trials when color was cued by Priority Cue 1 (blue). (B) Trial-averaged time courses of the value of the baseline parameter from IEM of the same trials from A. Asterisks indicate TRs for which values from the two trial types differ significantly. (C) Trial-averaged time courses of the BOLD percent signal changes of the same trials from A. These BOLD time courses between the color and motion direction priority cued trials that were statistically indistinguishable from each other suggest that the MVPA and IEM baseline differences from A and B, respectively, cannot be explained by an overall difference in the univariate signal intensity level. Note that the time courses were temporally smoothed using cubic spline interpolation. The width of the time courses denotes the SEM.

Figure 9. 

The time course of model performance on “switch” trials from the Bound condition of the MSR task. (A) Trial-averaged time courses of MVPA evidence for the representation of direction of motion, on trials when motion was cued by Priority Cue 1 (red) versus trials when color was cued by Priority Cue 1 (blue). (B) Trial-averaged time courses of the value of the baseline parameter from IEM of the same trials from A. Asterisks indicate TRs for which values from the two trial types differ significantly. (C) Trial-averaged time courses of the BOLD percent signal changes of the same trials from A. These BOLD time courses between the color and motion direction priority cued trials that were statistically indistinguishable from each other suggest that the MVPA and IEM baseline differences from A and B, respectively, cannot be explained by an overall difference in the univariate signal intensity level. Note that the time courses were temporally smoothed using cubic spline interpolation. The width of the time courses denotes the SEM.

Time course of the baseline parameter from IEM mirrors MVPA of stimulus category.

Inspection of the IEM time courses of the first half of the MSR trials (Figure 5), as well as the late-delay reconstructions from Delay 4 (Figure 6), reveals considerable variation in the baseline parameter of IEMs. By definition, this parameter does not relate directly to stimulus representation. To explore the possibility that these patterns of variation may track the priority of category-level information and thereby possibly index control-related activity, we plotted the values of the baseline parameter from tests of the late-delay IEM of one-item delayed recall on each TR of the trials of the MSR task that are featured in Figure 9A. As illustrated in Figure 9B, the fluctuations of the baseline parameter closely followed those of the MVPA of feature category. Importantly, univariate BOLD fluctuations for these different trial types were statistically indistinguishable, indicating that this evidence for task-related variation of multivariate category-level information in the parietal and frontal VOIs is not a mere byproduct of fluctuations in overall signal intensity (Figure 9C).

Interim summary of MVPA results.

The results of MVPA of delayed recall data binned post hoc into quadrants in stimulus-feature space were broadly consistent with the IEM results, in that evidence for stimulus-level representation of sample information was only reliably found for the dimension of direction of motion and only in the VOT ROI. This reinforces the idea that stimulus-specific direction information was most prominently represented in VOT cortex. Classification at the more abstract level of stimulus category (i.e., color vs. direction), however, was reliable in all three ROIs. Furthermore, when the late-delay delayed recall decoder was swept across data from the MSR task, it revealed that the representation of stimulus category in all ROIs was priority dependent and with a time course that was tightly coupled to the structure of the task: MVPA evidence for both categories was comparable at the beginning of the trial, prior to item prioritization, and closely tracked prioritization once prioritization cuing began. Coupled with the weak and uneven evidence for stimulus-level representation in frontal and parietal ROIs, these results are consistent with the idea that frontal and parietal networks were preferentially involved in controlling the maintenance of and changing of the priority of VOT-supported stimulus representations. Finally, the post hoc comparison of these MVPA time courses with the time courses of fluctuations in the baseline parameter of IEM suggest that the latter, too, may provide an index of the control of representations in visual WM.

DISCUSSION

The results from our study of a MSR task yielded two novel sets of empirical observations, each with important implications for our understanding of the mechanisms underlying visual WM. The first relates to the effects of Prioritization Cue 1 on stimulus representations. When two pieces of remembered information are drawn from separate objects, IEM indicates that the prioritization of one results in the strengthening of its neural representation and in the weakening, to baseline levels, of the active neural representation of the unprioritized item (even though this UMI must be retained in WM). When these two pieces of remembered information are drawn from the same object, in contrast, the effects of prioritization are markedly different: The active representation of the PMI is sustained but does not increase in strength, and the strength of the active representation of the UMI declines but nonetheless remains significantly above baseline. Although these two observations confound boundedness with category homogeneity of the two memory items, comparable same-object effects are also observed on color-cued trials, when the category homogeneity does not pertain. The second novel observation arises from contrasting the weak and uneven representation of stimulus identity in parietal and frontal cortex, whether assessed by IEM or MVPA, with the robust cue-locked dynamics of MVPA decoding of stimulus category in these two regions. These patterns are consistent with a role for these regions in the representation of priority.

Object-based Attention in Visual WM

The differential pattern of results in the Unbound versus the Bound condition suggests that key principles governing object-based attentional prioritization in visual perception also apply to visual WM. When the two remembered items belonged to separate objects (Unbound), the biasing of their competition for representation (Desimone & Duncan, 1995) resulted in the strengthening of the neural representation of the PMI at the expense of the strength of the UMI. When, however, the two remembered items belonged to the same object (Bound), the neural representation of the UMI remained elevated, consistent with an automatic spread of object-based attention to all elements of the remembered object. The theoretical implications of this finding are twofold. First, by illustrating object-based attention-like effects in visual WM, they extend the boundary conditions for which it can be said that visual WM appears to arise from “nothing more” than attention allocated to neural representations of objects not currently accessible to the eyes (cf. Myers, Stokes, & Nobre, 2017; Chun, 2011; Postle, 2006; Cowan, 1995). Second, they support the idea that multidimensional objects are represented in visual WM as bound objects, not as a collection of unbound features (cf. Park, Sy, Hong, & Tong, 2017; Bays et al., 2011; Luria & Vogel, 2011; Woodman & Vogel, 2008; Wheeler & Treisman, 2002; Luck & Vogel, 1997).

Controlling Priority in Visual WM

Stimulus representation in parietal and frontal cortex was markedly weaker than in the VOT ROI, whether assessed by IEM or by MVPA. There was, however, clear evidence that these two regions tracked the higher order information of which stimulus category was prioritized during each epoch of the trial, and they did so with a high degree of temporal precision. This is consistent with the idea that cue-driven changes in priority were implemented in WM via activity in frontoparietal circuits, whose representation of the prioritized category may have acted as a source of top–down bias on high-fidelity representations of stimulus features in VOT cortex (cf. Sheldon et al., Submitted; Nelissen et al., 2013).

Evidence for the Active Removal of Information from WM?

There are, in principle, two ways that information can exit WM: Its links to trial-specific context can decay after attention has been shifted away from it, or it can be actively removed, via suppression, recoding, or some other mechanism. Our results provide suggestive evidence for an active mechanism: In three instances when a retrocue indicated that the uncued information was no longer relevant for behavior (i.e., designated it a IMI), its neural representation transitioned from a robust reconstruction of the trained model to one that approached (after Priority Cue 2) or achieved (after the “relevance cue”) a state that was flipped relative to when it had been in the focus of attention. Although we cannot know from these data what mechanism(s) would have effected this change in representation (e.g., inhibition, recoding), we can postulate that they may only be engaged when task demands require the active removal of information from WM. In the one-item delayed recall task, in contrast, the neural representation of the sample item seems to just “fade away” at the end of the trial.

A mechanistically noncommittal interpretation of the retrocue-triggered transformation of IMIs is to suggest that once the identity of the information that will be relevant for the remainder of the MSR trial is known, the resultant IMI is processed in a manner that makes it least likely that it will interfere with performance on the remainder of the trial. A similar phenomenon has been observed in a different variant of the MSR task (with fMRI; Yu & Postle, 2018), in a 2-back task (with EEG; Wan, Cai, Samaha, & Postle, In Press), and in a dual serial visual search task (van Loon et al., 2018). It could be that recoding unprioritized information is a general mechanism for preventing that information from interfering with performance that needs to be guided by a PMI. By this account, removing an item from WM would be accomplished in a two-step process: First, recode it so that it is less likely to interfere with the PMI; second, let the recoded representation decay.

APPENDIX

Figure 10. 

Time courses of IEM reconstructions from the one-item delayed recall task, in feature-selective parietal ROIs. Rows illustrate results for direction (top: A, B, and C) and color (bottom: D, E, and F), and columns illustrate the results for three procedures: train on TR 4 (a “perception/encoding” model) and test at every TR (left: A and D); train on TR 6 (a “delay” model) and test at every TR (middle: B and E); and train and test at each TR (“along the diagonal”; right: C and F). In each panel, channel is arrayed along the horizontal axis, from −160° to 160°, time (in TRs) proceeds from top to bottom, and the dependent data are channel responses (averaged after aligning each trial to 0°). Significance of the reconstruction at each TR, determined by bootstrapping, is indicated by asterisks (*p < .05, **p < .01, ***p < .001, ****p < .0001).

Figure 10. 

Time courses of IEM reconstructions from the one-item delayed recall task, in feature-selective parietal ROIs. Rows illustrate results for direction (top: A, B, and C) and color (bottom: D, E, and F), and columns illustrate the results for three procedures: train on TR 4 (a “perception/encoding” model) and test at every TR (left: A and D); train on TR 6 (a “delay” model) and test at every TR (middle: B and E); and train and test at each TR (“along the diagonal”; right: C and F). In each panel, channel is arrayed along the horizontal axis, from −160° to 160°, time (in TRs) proceeds from top to bottom, and the dependent data are channel responses (averaged after aligning each trial to 0°). Significance of the reconstruction at each TR, determined by bootstrapping, is indicated by asterisks (*p < .05, **p < .01, ***p < .001, ****p < .0001).

Figure 11. 

Time courses of IEM reconstructions from the one-item delayed recall task, in feature-selective frontal ROIs. All display conventions the same as Figure 10.

Figure 11. 

Time courses of IEM reconstructions from the one-item delayed recall task, in feature-selective frontal ROIs. All display conventions the same as Figure 10.

Figure 12. 

Time course of the neural representation of direction during Unbound (A, B) and Bound (C, D) conditions, across the first half of the MSR task, as a function of priority status in feature-selective parietal ROIs. The top row illustrates the Unbound trials, and the bottom row illustrates the Bound trials. The panels on the left side illustrate the reconstruction of the direction of motion when it was cued by Priority Cue 1, and the panels on the right side illustrate the reconstruction of the direction of motion when it was not cued by Priority Cue 1. All display conventions within each panel are the same as Figure 10.

Figure 12. 

Time course of the neural representation of direction during Unbound (A, B) and Bound (C, D) conditions, across the first half of the MSR task, as a function of priority status in feature-selective parietal ROIs. The top row illustrates the Unbound trials, and the bottom row illustrates the Bound trials. The panels on the left side illustrate the reconstruction of the direction of motion when it was cued by Priority Cue 1, and the panels on the right side illustrate the reconstruction of the direction of motion when it was not cued by Priority Cue 1. All display conventions within each panel are the same as Figure 10.

Figure 13. 

Time course of the neural representation of direction during Unbound (A, B) and Bound (C, D) conditions, across the first half of the MSR task, as a function of priority status in feature-selective frontal ROIs. All display conventions within each panel are the same as Figure 10.

Figure 13. 

Time course of the neural representation of direction during Unbound (A, B) and Bound (C, D) conditions, across the first half of the MSR task, as a function of priority status in feature-selective frontal ROIs. All display conventions within each panel are the same as Figure 10.

Figure 14. 

(A) Individual time point plots for unbound trials in VOT of delay period of the Priority Cue 1. Channel response is plotted on the y-axis for individual time points, and the specific motion direction channel is plotted on the x-axis, centered around 0°. Each row is a set of reconstructions from a single time point starting with the Priority Cue 1 onset (TR 11) and ending with the probe onset (TR 15). Red traces represent reconstructions of directions on trials when direction was cued (prioritized reconstructions, PMI), and blue traces represent reconstructions of direction on trials when direction was uncued (unprioritized reconstructions, UMI). (B) Individual time point plots for unbound “stay” and “switch” trials in VOT of delay period of the Priority Cue 2. Each row is a set of reconstructions from a single time point starting with the Priority Cue 2 onset (TR 17) and ending with the probe onset (TR 21). Note that, in the delay period of Priority Cue 2, only the PMI is relevant for the impending Probe 2, and the uncued feature has become the irrelevant memory information (IMI, blue). Significant differences between the prioritized and unprioritized motion directions for the amplitude (one-tailed bootstrap tested) and baseline (two-tailed bootstrap tested) estimates are indicated by the asterisks and dots, respectively (*p < .05, **p < .01, ***p < .001; ****p < .0001).

Figure 14. 

(A) Individual time point plots for unbound trials in VOT of delay period of the Priority Cue 1. Channel response is plotted on the y-axis for individual time points, and the specific motion direction channel is plotted on the x-axis, centered around 0°. Each row is a set of reconstructions from a single time point starting with the Priority Cue 1 onset (TR 11) and ending with the probe onset (TR 15). Red traces represent reconstructions of directions on trials when direction was cued (prioritized reconstructions, PMI), and blue traces represent reconstructions of direction on trials when direction was uncued (unprioritized reconstructions, UMI). (B) Individual time point plots for unbound “stay” and “switch” trials in VOT of delay period of the Priority Cue 2. Each row is a set of reconstructions from a single time point starting with the Priority Cue 2 onset (TR 17) and ending with the probe onset (TR 21). Note that, in the delay period of Priority Cue 2, only the PMI is relevant for the impending Probe 2, and the uncued feature has become the irrelevant memory information (IMI, blue). Significant differences between the prioritized and unprioritized motion directions for the amplitude (one-tailed bootstrap tested) and baseline (two-tailed bootstrap tested) estimates are indicated by the asterisks and dots, respectively (*p < .05, **p < .01, ***p < .001; ****p < .0001).

Figure 15. 

Individual time point plots for bound trials in VOT of delay period of the Priority Cue 1 (A) and Priority Cue 2 (B). Conventions are the same as in Figure 14.

Figure 15. 

Individual time point plots for bound trials in VOT of delay period of the Priority Cue 1 (A) and Priority Cue 2 (B). Conventions are the same as in Figure 14.

Reprint requests should be sent to Muhammet I. Sahan, Department of Experimental Psychology, Ghent University, H Dunanlaan 2, 9000 Ghent, Belgium, or via e-mail: muhammet.sahan@ugent.be.

REFERENCES

Bays
,
P. M.
,
Catalao
,
R. F. G.
, &
Husain
,
M.
(
2009
).
The precision of visual working memory is set by allocation of a shared resource
.
Journal of Vision
,
9
,
7
.
Bays
,
P. M.
,
Wu
,
E. Y.
, &
Husain
,
M.
(
2011
).
Storage and binding of object features in visual working memory
.
Neuropsychologia
,
49
,
1622
1631
.
Brincat
,
S. L.
,
Siegel
,
M.
,
von Nicolai
,
C.
, &
Miller
,
E. K.
(
2018
).
Gradual progression from sensory to task-related processing in cerebral cortex
.
Proceedings of the National Academy of Sciences, U.S.A.
,
115
,
E7202
E7211
.
Brouwer
,
G. J.
, &
Heeger
,
D. J.
(
2009
).
Decoding and reconstructing color from responses in human visual cortex
.
Journal of Neuroscience
,
29
,
13992
14003
.
Burmester
,
A.
, &
Fougnie
,
D.
(
2016
).
The sum is no more than its parts: No evidence for bound features during multi-feature visual change detection
.
Journal of Vision
,
16
,
1068
.
Cai
,
Y.
,
Sheldon
,
A. D.
,
Yu
,
Q.
, &
Postle
,
B. R.
(
2019
).
Overlapping and distinct contributions of stimulus location and of spatial context to nonspatial visual short-term memory
.
Journal of Neurophysiology
,
121
,
1222
1231
.
Carandini
,
M.
, &
Heeger
,
D. J.
(
2012
).
Normalization as a canonical neural computation
.
Nature Reviews Neuroscience
,
13
,
51
62
.
Christophel
,
T. B.
,
Iamshchinina
,
P.
,
Yan
,
C.
,
Allefeld
,
C.
, &
Haynes
,
J.-D.
(
2018
).
Cortical specialization for attended versus unattended working memory
.
Nature Neuroscience
,
21
,
494
496
.
Chun
,
M. M.
(
2011
).
Visual working memory as visual attention sustained internally over time
.
Neuropsychologia
,
49
,
1407
1409
.
Cowan
,
N.
(
1995
).
Attention and memory: An integrated framework
.
New York
:
Oxford University Press
.
Cox
,
R. W.
(
1996
).
AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages
.
Computers and Biomedical Research
,
29
,
162
173
.
Desimone
,
R.
, &
Duncan
,
J.
(
1995
).
Neural mechanisms of selective visual attention
.
Annual Review of Neuroscience
,
18
,
193
222
.
D'Esposito
,
M.
, &
Postle
,
B. R.
(
2015
).
The cognitive neuroscience of working memory
.
Annual Review of Psychology
,
66
,
115
142
.
Driver
,
J.
(
2001
).
A selective review of selective attention research from the past century
.
British Journal of Psychology
,
92
,
53
78
.
Duijnhouwer
,
J.
,
Noest
,
A. J.
,
Lankheet
,
M. J. M.
,
van den Berg
,
A. V.
, &
van Wezel
,
R. J. A.
(
2013
).
Speed and direction response profiles of neurons in macaque MT and MST show modest constraint line tuning
.
Frontiers in Behavioral Neuroscience
,
7
,
22
.
Duncan
,
J.
(
1984
).
Selective attention and the organization of visual information
.
Journal of Experimental Psychology: General
,
113
,
501
517
.
Egly
,
R.
,
Driver
,
J.
, &
Rafal
,
R. D.
(
1994
).
Shifting visual attention between objects and locations: Evidence from normal and parietal lesion subjects
.
Journal of Experimental Psychology: General
,
123
,
161
177
.
Emrich
,
S. M.
,
Riggall
,
A. C.
,
LaRocque
,
J. J.
, &
Postle
,
B. R.
(
2013
).
Distributed patterns of activity in sensory cortex reflect the precision of multiple items maintained in visual short-term memory
.
Journal of Neuroscience
,
33
,
6516
6523
.
Ester
,
E. F.
,
Sprague
,
T. C.
, &
Serences
,
J. T.
(
2015
).
Parietal and frontal cortex encode stimulus-specific mnemonic representations during visual working memory
.
Neuron
,
87
,
893
905
.
Fisher
,
N. I.
(
1993
).
Statistical analysis of circular data
.
Cambridge
:
Cambridge University Press
.
Gazzaley
,
A.
, &
Nobre
,
A. C.
(
2012
).
Top–down modulation: Bridging selective attention and working memory
.
Trends in Cognitive Sciences
,
16
,
129
135
.
Gosseries
,
O.
,
Yu
,
Q.
,
LaRocque
,
J. J.
,
Starrett
,
M. J.
,
Rose
,
N. S.
,
Cowan
,
N.
, et al
(
2018
).
Parietal-occipital interactions underlying control- and representation-related processes in working memory for nonspatial visual features
.
Journal of Neuroscience
,
38
,
4357
4366
.
Griffin
,
I. C.
, &
Nobre
,
A. C.
(
2003
).
Orienting attention to locations in internal representations
.
Journal of Cognitive Neuroscience
,
15
,
1176
1194
.
Kiyonaga
,
A.
, &
Egner
,
T.
(
2013
).
Working memory as internal attention: Toward an integrative account of internal and external selection processes
.
Psychonomic Bulletin & Review
,
20
,
228
242
.
LaRocque
,
J. J.
,
Lewis-Peacock
,
J. A.
,
Drysdale
,
A. T.
,
Oberauer
,
K.
, &
Postle
,
B. R.
(
2013
).
Decoding attended information in short-term memory: An EEG study
.
Journal of Cognitive Neuroscience
,
25
,
127
142
.
LaRocque
,
J. J.
,
Lewis-Peacock
,
J. A.
, &
Postle
,
B. R.
(
2014
).
Multiple neural states of representation in short-term memory? It's a matter of attention
.
Frontiers in Human Neuroscience
,
8
,
5
.
LaRocque
,
J. J.
,
Riggall
,
A. C.
,
Emrich
,
S. M.
, &
Postle
,
B. R.
(
2017
).
Within-category decoding of information in different attentional states in short-term memory
.
Cerebral Cortex
,
27
,
4881
4890
.
Lepsien
,
J.
, &
Nobre
,
A. C.
(
2007
).
Attentional modulation of object representations in working memory
.
Cerebral Cortex
,
17
,
2072
2083
.
Lewis-Peacock
,
J. A.
,
Drysdale
,
A. T.
,
Oberauer
,
K.
, &
Postle
,
B. R.
(
2012
).
Neural evidence for a distinction between short-term memory and the focus of attention
.
Journal of Cognitive Neuroscience
,
24
,
61
79
.
Lewis-Peacock
,
J. A.
,
Drysdale
,
A. T.
, &
Postle
,
B. R.
(
2015
).
Neural evidence for the flexible control of mental representations
.
Cerebral Cortex
,
25
,
3303
3313
.
Lewis-Peacock
,
J. A.
, &
Postle
,
B. R.
(
2012
).
Decoding the internal focus of attention
.
Neuropsychologia
,
50
,
470
478
.
Luck
,
S. J.
, &
Vogel
,
E. K.
(
1997
).
The capacity of visual working memory for features and conjunctions
.
Nature
,
390
,
279
281
.
Luria
,
R.
, &
Vogel
,
E. K.
(
2011
).
Shape and color conjunction stimuli are represented as bound objects in visual working memory
.
Neuropsychologia
,
49
,
1632
1639
.
Myers
,
N. E.
,
Stokes
,
M. G.
, &
Nobre
,
A. C.
(
2017
).
Prioritizing information during working memory: Beyond sustained internal attention
.
Trends in Cognitive Sciences
,
21
,
449
461
.
Nelissen
,
N.
,
Stokes
,
M.
,
Nobre
,
A. C.
, &
Rushworth
,
M. F. S.
(
2013
).
Frontal and parietal cortical interactions with distributed visual representations during selective attention and action selection
.
Journal of Neuroscience
,
33
,
16443
16458
.
Oberauer
,
K.
, &
Hein
,
L.
(
2012
).
Attention to information in working memory
.
Current Directions in Psychological Science
,
21
,
164
169
.
Park
,
Y. E.
,
Sy
,
J. L.
,
Hong
,
S. W.
, &
Tong
,
F.
(
2017
).
Reprioritization of features of multidimensional objects stored in visual working memory
.
Psychological Science
,
28
,
1773
1785
.
Pertzov
,
Y.
,
Bays
,
P. M.
,
Joseph
,
S.
, &
Husain
,
M.
(
2013
).
Rapid forgetting prevented by retrospective attention cues
.
Journal of Experimental Psychology: Human Perception and Performance
,
39
,
1224
1231
.
Peters
,
B.
,
Kaiser
,
J.
,
Rahm
,
B.
, &
Bledowski
,
C.
(
2015
).
Activity in human visual and parietal cortex reveals object-based attention in working memory
.
Journal of Neuroscience
,
35
,
3360
3369
.
Polyn
,
S. M.
,
Natu
,
V. S.
,
Cohen
,
J. D.
, &
Norman
,
K. A.
(
2005
).
Category-specific cortical activity precedes retrieval during memory search
.
Science
,
310
,
1963
1966
.
Postle
,
B. R.
(
2006
).
Working memory as an emergent property of the mind and brain
.
Neuroscience
,
139
,
23
38
.
Pribram
,
K. H.
,
Ahumada
,
A.
,
Hartog
,
J.
, &
Ross
,
L.
(
1964
).
A progress report on the neurological processes disturbed by frontal lesions in primates
. In
J. M.
Warren
&
K.
Akert
(Eds.),
The frontal granular cortex and behavior
(pp.
28
55
).
New York
:
McGraw-Hill
.
Rose
,
N. S.
,
LaRocque
,
J. J.
,
Riggall
,
A. C.
,
Gosseries
,
O.
,
Starrett
,
M. J.
,
Meyering
,
E. E.
, et al
(
2016
).
Reactivation of latent working memories with transcranial magnetic stimulation
.
Science
,
354
,
1136
1139
.
Sahan
,
M. I.
,
Verguts
,
T.
,
Boehler
,
C. N.
,
Pourtois
,
G.
, &
Fias
,
W.
(
2016
).
Paying attention to working memory: Similarities in the spatial distribution of attention in mental and physical space
.
Psychonomic Bulletin & Review
,
23
,
1190
1197
.
Serences
,
J. T.
, &
Saproo
,
S.
(
2012
).
Computational advances towards linking BOLD and behavior
.
Neuropsychologia
,
50
,
435
446
.
Sheldon
,
A. D.
,
Saad
,
E.
,
Sahan
,
M. I.
,
Meyering
,
E.
, &
Postle
,
B. R.
(
Submitted
).
Attention biases competition for visual representation via enhancement of targets and inhibition of nontargets
.
Sprague
,
T. C.
,
Ester
,
E. F.
, &
Serences
,
J. T.
(
2014
).
Reconstructions of information in visual spatial working memory degrade with memory load
.
Current Biology
,
24
,
2174
2180
.
Thigpen
,
N.
,
Petro
,
N. M.
,
Oschwald
,
J.
,
Oberauer
,
K.
, &
Keil
,
A.
(
2019
).
Selection of visual objects in perception and working memory one at a time
.
Psychological Science
,
30
,
1259
1272
.
van Loon
,
A. M.
,
Olmos-Solis
,
K.
,
Fahrenfort
,
J. J.
, &
Olivers
,
C. N. L.
(
2018
).
Current and future goals are represented in opposite patterns in object-selective cortex
.
eLife
,
7
,
e38677
.
Vecera
,
S. P.
,
Behrmann
,
M.
, &
McGoldrick
,
J.
(
2000
).
Selective attention to the parts of an object
.
Psychonomic Bulletin & Review
,
7
,
301
308
.
Wan
,
Q.
,
Cai
,
Y.
,
Samaha
,
J.
, &
Postle
,
B. R.
(
In Press
).
Tracking stimulus representation across a 2-back visual working memory task
.
Royal Society Open Science
.
Wang
,
H. X.
,
Merriam
,
E. P.
,
Freeman
,
J.
, &
Heeger
,
D. J.
(
2014
).
Motion direction biases and decoding in human visual cortex
.
Journal of Neuroscience
,
34
,
12601
12615
.
Wheeler
,
M. E.
, &
Treisman
,
A. M.
(
2002
).
Binding in short-term visual memory
.
Journal of Experimental Psychology: General
,
131
,
48
64
.
Woodman
,
G. F.
, &
Vogel
,
E. K.
(
2008
).
Selective storage and maintenance of an object's features in visual working memory
.
Psychonomic Bulletin & Review
,
15
,
223
229
.
Yu
,
Q.
, &
Postle
,
B. R.
(
2018
).
Different states of priority recruit different neural codes in visual working memory
.
bioRxiv
. https://doi.org/10.1101/334920.
Zhang
,
Y.
,
Meyers
,
E. M.
,
Bichot
,
N. P.
,
Serre
,
T.
,
Poggio
,
T. A.
, &
Desimone
,
R.
(
2011
).
Object decoding with attention in inferior temporal cortex
.
Proceedings of the National Academy of Sciences, U.S.A.
,
108
,
8850
8855
.
Zokaei
,
N.
,
Manohar
,
S.
,
Husain
,
M.
, &
Feredoes
,
E.
(
2014
).
Causal evidence for a privileged working memory state in early visual cortex
.
Journal of Neuroscience
,
34
,
158
162
.

Author notes

*

These authors contributed equally to this work.