Evidence is accumulating that the classic two-stage model of visual STM (VSTM), comprising iconic memory (IM) and visual working memory (WM), is incomplete. A third memory stage, termed fragile VSTM (FM), seems to exist in between IM and WM [Vandenbroucke, A. R. E., Sligte, I. G., & Lamme, V. A. F. Manipulations of attention dissociate fragile visual STM from visual working memory. Neuropsychologia, 49, 1559–1568, 2011; Sligte, I. G., Scholte, H. S., & Lamme, V. A. F. Are there multiple visual STM stores? PLoS One, 3, e1699, 2008]. Although FM can be distinguished from IM using behavioral and fMRI methods, the question remains whether FM is a weak expression of WM or a separate form of memory with its own neural signature. Here, we tested whether FM and WM in humans are supported by dissociable time–frequency features of EEG recordings. Participants performed a partial-report change detection task, from which individual differences in FM and WM capacity were estimated. These individual FM and WM capacities were correlated with time–frequency characteristics of the EEG signal before and during encoding and maintenance of the memory display. FM capacity showed negative alpha correlations over peri-occipital electrodes, whereas WM capacity was positively related, suggesting increased visual processing (lower alpha) to be related to FM capacity. Furthermore, FM capacity correlated with an increase in theta power over central electrodes during preparation and processing of the memory display, whereas WM did not. In addition to a difference in visual processing characteristics, a positive relation between gamma power and FM capacity was observed during both preparation and maintenance periods of the task. On the other hand, we observed that theta–gamma coupling was negatively correlated with FM capacity, whereas it was slightly positively correlated with WM. These data show clear differences in the neural substrates of FM versus WM and suggest that FM depends more on visual processing mechanisms compared with WM. This study thus provides novel evidence for a dissociation between different stages in VSTM.

Every day, we experience a rich and complex visual world. Our brain constantly translates meaningless fragmented input into coherent objects and scenes. However, our attentional capabilities are limited, and we can only report the few items that we happen to attend to. So what happens to items that are not cognitively accessed? Do these remain fragmentary and meaningless? Or are they processed up to a level where perceptual inferences take place about image composition? To investigate this, we recorded brain activity using fMRI while participants viewed images containing a Kanizsa figure, an illusion in which an object is perceived by means of perceptual inference. Participants were presented with the Kanizsa figure and three matched nonillusory control figures while they were engaged in an attentionally demanding distractor task. After the task, one group of participants was unable to identify the Kanizsa figure in a forced-choice decision task; hence, they were “inattentionally blind.” A second group had no trouble identifying the Kanizsa figure. Interestingly, the neural signature that was unique to the processing of the Kanizsa figure was present in both groups. Moreover, within-subject multivoxel pattern analysis showed that the neural signature of unreported Kanizsa figures could be used to classify reported Kanizsa figures and that this cross-report classification worked better for the Kanizsa condition than for the control conditions. Together, these results suggest that stimuli that are not cognitively accessed are processed up to levels of perceptual interpretation.

The visual system has been commonly subdivided into two segregated visual processing streams: The dorsal pathway processes mainly spatial information, and the ventral pathway specializes in object perception. Recent findings, however, indicate that different forms of interaction (cross-talk) exist between the dorsal and the ventral stream. Here, we used TMS and concurrent EEG recordings to explore these interactions between the dorsal and ventral stream during figure-ground segregation. In two separate experiments, we used repetitive TMS and single-pulse TMS to disrupt processing in the dorsal (V5/HMT + ) and the ventral (lateral occipital area) stream during a motion-defined figure discrimination task. We presented stimuli that made it possible to differentiate between relatively low-level (figure boundary detection) from higher-level (surface segregation) processing steps during figure-ground segregation. Results show that disruption of V5/HMT + impaired performance related to surface segregation; this effect was mainly found when V5/HMT + was perturbed in an early time window (100 msec) after stimulus presentation. Surprisingly, disruption of the lateral occipital area resulted in increased performance scores and enhanced neural correlates of surface segregation. This facilitatory effect was also mainly found in an early time window (100 msec) after stimulus presentation. These results suggest a “push–pull” interaction in which dorsal and ventral extrastriate areas are being recruited or inhibited depending on stimulus category and task demands.

It has been proposed that visual attention and consciousness are separate [Koch, C., & Tsuchiya, N. Attention and consciousness: Two distinct brain processes. Trends in Cognitive Sciences, 11, 16–22, 2007] and possibly even orthogonal processes [Lamme, V. A. F. Why visual attention and awareness are different. Trends in Cognitive Sciences, 7, 12–18, 2003]. Attention and consciousness converge when conscious visual percepts are attended and hence become available for conscious report. In such a view, a lack of reportability can have two causes: the absence of attention or the absence of a conscious percept. This raises an important question in the field of perceptual learning. It is known that learning can occur in the absence of reportability [Gutnisky, D. A., Hansen, B. J., Iliescu, B. F., & Dragoi, V. Attention alters visual plasticity during exposure-based learning. Current Biology, 19, 555–560, 2009; Seitz, A. R., Kim, D., & Watanabe, T. Rewards evoke learning of unconsciously processed visual stimuli in adult humans. Neuron, 61, 700–707, 2009; Seitz, A. R., & Watanabe, T. Is subliminal learning really passive? Nature, 422, 36, 2003; Watanabe, T., Náñez, J. E., & Sasaki, Y. Perceptual learning without perception. Nature, 413, 844–848, 2001], but it is unclear which of the two ingredients—consciousness or attention—is not necessary for learning. We presented textured figure-ground stimuli and manipulated reportability either by masking (which only interferes with consciousness) or with an inattention paradigm (which only interferes with attention). During the second session (24 hr later), learning was assessed neurally and behaviorally, via differences in figure-ground ERPs and via a detection task. Behavioral and neural learning effects were found for stimuli presented in the inattention paradigm and not for masked stimuli. Interestingly, the behavioral learning effect only became apparent when performance feedback was given on the task to measure learning, suggesting that the memory trace that is formed during inattention is latent until accessed. The results suggest that learning requires consciousness, and not attention, and further strengthen the idea that consciousness is separate from attention.

Consciousness can be manipulated in many ways. Here, we seek to understand whether two such ways, visual masking and pharmacological intervention, share a common pathway in manipulating visual consciousness. We recorded EEG from human participants who performed a backward-masking task in which they had to detect a masked figure form its background (masking strength was varied across trials). In a within-subject design, participants received dextromethorphan (a N -methyl- d -aspartate receptor antagonist), lorazepam (LZP; a GABA A receptor agonist), scopolamine (a muscarine receptor antagonist), or placebo. The behavioral results show that detection rate decreased with increasing masking strength and that of all the drugs, only LZP induced a further decrease in detection rate. Figure-related ERP signals showed three neural events of interest: (1) an early posterior occipital and temporal generator (94–121 msec) that was not influenced by any pharmacological manipulation nor by masking, (2) a later bilateral perioccipital generator (156–211 msec) that was reduced by masking as well as LZP (but not by any other drugs), and (3) a late bilateral occipital temporal generator (293–387 msec) that was mainly affected by masking. Crucially, only the intermediate neural event correlated with detection performance. In combination with previous findings, these results suggest that LZP and masking both reduce visual awareness by means of modulating late activity in the visual cortex but leave early activation intact. These findings provide the first evidence for a common mechanism for these two distinct ways of manipulating consciousness.

Humans largely guide their behavior by their visual representation of the world. Recent studies have shown that visual information can trigger behavior within 150 msec, suggesting that visually guided responses to external events, in fact, precede conscious awareness of those events. However, is such a view correct? By using a texture discrimination task, we show that the brain relies on long-latency visual processing in order to guide perceptual decisions. Decreasing stimulus saliency leads to selective changes in long-latency visually evoked potential components reflecting scene segmentation. These latency changes are accompanied by almost equal changes in simple RTs and points of subjective simultaneity. Furthermore, we find a strong correlation between individual RTs and the latencies of scene segmentation related components in the visually evoked potentials, showing that the processes underlying these late brain potentials are critical in triggering a response. However, using the same texture stimuli in an antisaccade task, we found that reflexive, but erroneous, prosaccades, but not antisaccades, can be triggered by earlier visual processes. In other words: The brain can act quickly, but decides late. Differences between our study and earlier findings suggesting that action precedes conscious awareness can be explained by assuming that task demands determine whether a fast and unconscious, or a slower and conscious, representation is used to initiate a visually guided response.

The presupplementary motor area (pre-SMA) is considered key in contributing to voluntary action selection during response conflict. Here we test whether individual differences in the ability to select appropriate actions in the face of strong (conscious) and weak (virtually unconscious) distracting alternatives are related to individual variability in pre-SMA anatomy. To this end, we scanned 58 participants, who performed a masked priming task in which conflicting response tendencies were elicited either consciously (through primes that were weakly masked) or virtually unconsciously (strongly masked primes), with structural magnetic resonance imaging. Voxel-based morphometry revealed that individual differences in pre-SMA gray-matter density are related to subjects' ability to voluntary select the correct action in the face of conflict, irrespective of the awareness level of conflict-inducing stimuli. These results link structural anatomy to individual differences in cognitive control ability, and provide support for the role of the pre-SMA in the selection of appropriate actions in situations of response conflict. Furthermore, these results suggest that flexible and voluntary behavior requires efficiently dealing with competing response tendencies, even those that are activated automatically and unconsciously.

Cognitive control allows humans to overrule and inhibit habitual responses to optimize performance in challenging situations. Contradicting traditional views, recent studies suggest that cognitive control processes can be initiated unconsciously. To further capture the relation between consciousness and cognitive control, we studied the dynamics of inhibitory control processes when triggered consciously versus unconsciously in a modified version of the stop task. Attempts to inhibit an imminent response were often successful after unmasked (visible) stop signals. Masked (invisible) stop signals rarely succeeded in instigating overt inhibition but did trigger slowing down of response times. Masked stop signals elicited a sequence of distinct ERP components that were also observed on unmasked stop signals. The N2 component correlated with the efficiency of inhibitory control when elicited by unmasked stop signals and with the magnitude of slowdown when elicited by masked stop signals. Thus, the N2 likely reflects the initiation of inhibitory control, irrespective of conscious awareness. The P3 component was much reduced in amplitude and duration on masked versus unmasked stop trials. These patterns of differences and similarities between conscious and unconscious cognitive control processes are discussed in a framework that differentiates between feedforward and feedback connections in yielding conscious experience.

In texture segregation, an example of scene segmentation, we can discern two different processes: texture boundary detection and subsequent surface segregation [Lamme, V. A. F., Rodriguez-Rodriguez, V., & Spekreijse, H. Separate processing dynamics for texture elements, boundaries and surfaces in primary visual cortex of the macaque monkey. Cerebral Cortex, 9, 406–413, 1999]. Neural correlates of texture boundary detection have been found in monkey V1 [Sillito, A. M., Grieve, K. L., Jones, H. E., Cudeiro, J., & Davis, J. Visual cortical mechanisms detecting focal orientation discontinuities. Nature, 378, 492–496, 1995; Grosof, D. H., Shapley, R. M., & Hawken, M. J. Macaque-V1 neurons can signal illusory contours. Nature, 365, 550–552, 1993], but whether surface segregation occurs in monkey V1 [Rossi, A. F., Desimone, R., & Ungerleider, L. G. Contextual modulation in primary visual cortex of macaques. Journal of Neuroscience, 21, 1698–1709, 2001; Lamme, V. A. F. The neurophysiology of figure ground segregation in primary visual-cortex. Journal of Neuroscience, 15, 1605–1615, 1995], and whether boundary detection or surface segregation signals can also be measured in human V1, is more controversial [Kastner, S., De Weerd, P., & Ungerleider, L. G. Texture segregation in the human visual cortex: A functional MRI study. Journal of Neurophysiology, 83, 2453–2457, 2000]. Here we present electroencephalography (EEG) and functional magnetic resonance imaging data that have been recorded with a paradigm that makes it possible to differentiate between boundary detection and scene segmentation in humans. In this way, we were able to show with EEG that neural correlates of texture boundary detection are first present in the early visual cortex around 92 msec and then spread toward the parietal and temporal lobes. Correlates of surface segregation first appear in temporal areas (around 112 msec) and from there appear to spread to parietal, and back to occipital areas. After 208 msec, correlates of surface segregation and boundary detection also appear in more frontal areas. Blood oxygenation level-dependent magnetic resonance imaging results show correlates of boundary detection and surface segregation in all early visual areas including V1. We conclude that texture boundaries are detected in a feedforward fashion and are represented at increasing latencies in higher visual areas. Surface segregation, on the other hand, is represented in “reverse hierarchical” fashion and seems to arise from feedback signals toward early visual areas such as V1.

Attentive processing is often described as a competition for resources among stimuli by mutual suppression. This is supported by findings that activity in extrastriate cortex is suppressed when several stimuli are presented simultaneously, compared to a single stimulus. In this study, we randomly varied the number of simultaneously presented figures (set size) in an attention-demanding change detection task, while we recorded multiunit activity in striate cortex (V1) in monkeys. After figure–background segregation, activity was suppressed as set size increased. This effect was stronger and started earlier among cells stimulated by the background than those stimulated by the figures themselves. As a consequence, contextual modulation, a correlate of figure–background segregation, increased with set size, approximately 100 msec after its initial generation. The results indicate that suppression of responses under increasing attentional demands differentially affects figure and background responses in area V1.

In a backward masking paradigm, a target stimulus is rapidly (<100 msec) followed by a second stimulus. This typically results in a dramatic decrease in the visibility of the target stimulus. It has been shown that masking reduces responses in V1. It is not known, however, which process in V1 is affected by the mask. In the past, we have shown that in V1, modulations of neural activity that are specifically related to figure-ground segregation can be recorded. Here, we recorded from awake macaque monkeys, engaged in a task where they had to detect figures from background in a pattern backward masking paradigm. We show that the V1 figure-ground signals are selectively and fully suppressed at target-mask intervals that psychophysically result in the target being invisible. Initial response transients, signalling the features that make up the scene, are not affected. As figure-ground modulations depend on feedback from extrastriate areas, these results suggest that masking selectively interrupts the recurrent interactions between V1 and higher visual areas.

Here we propose a model of how the visual brain segregates textured scenes into figures and background. During texture segregation, locations where the properties of texture elements change abruptly are assigned to boundaries, whereas image regions that are relatively homogeneous are grouped together. Boundary detection and grouping of image regions require different connection schemes, which are accommodated in a single network architecture by implementing them in different layers. As a result, all units carry signals related to boundary detection as well as grouping of image regions, in accordance with cortical physiology. Boundaries yield an early enhancement of network responses, but at a later point, an entire figural region is grouped together, because units that respond to it are labeled with enhanced activity. The model predicts which image regions are preferentially perceived as figure or as background and reproduces the spatio-temporal profile of neuronal activity in the visual cortex during texture segregation in intact animals, as well as in animals with cortical lesions.