From which regions of the brain do conscious representations of visual stimuli emerge? This is an important but controversial issue in neuroscience because some studies have reported a major role of the higher visual regions of the ventral pathway in conscious perception, whereas others have found neural correlates of consciousness as early as in the primary visual areas and in the thalamus. One reason for this controversy has been the difficulty in focusing on neural activity at the moment when conscious percepts are generated in the brain, excluding any bottom–up responses (not directly related to consciousness) that are induced by stimuli. In this study, we address this issue with a new approach that can induce a rapid change in conscious perception with little influence from bottom–up responses. Our results reveal that the first consciousness-related activity emerges from the higher visual region of the ventral pathway. However, this activity is rapidly diffused to the entire brain, including the early visual cortex. These results thus integrate previous “higher” and “lower” views on the emergence of neural correlates of consciousness, providing a new perspective for the temporal dynamics of consciousness.
There is considerable evidence for the “unconscious” processing of sensory stimuli in the brain (Sterzer, Jalkanen, & Rees, 2009; Fang & He, 2005). How such unconscious neural responses are converted into conscious representations involving awareness is an important problem in neuroscience (Dehaene, Changeux, Naccache, Sackur, & Sergent, 2006; Block, 2005; Crick & Koch, 2003; Tononi & Edelman, 1998). One solution is to identify brain regions where this conversion from unconscious to conscious representations takes place. Previous studies have examined these regions using psychological paradigms that produce spontaneous alternations of perceptual states, such as binocular rivalry and multistable stimuli (Sterzer, Kleinschmidt, & Rees, 2009). In these methods, one can induce fluctuations in conscious percepts with input stimuli kept constant, thereby dissociating brain responses directly associated with changes in conscious perception (neural correlates of consciousness) from those that are simply reacting to sensory stimulation. Combining these methods with neurophysiological recordings (Maier et al., 2008; Sheinberg & Logothetis, 1997; Leopold & Logothetis, 1996) and neuroimaging techniques (Haynes, Deichmann, & Rees, 2005; Lee, Blake, & Heeger, 2005; Wunderlich, Schneider, & Kastner, 2005; Tong & Engel, 2001), many studies have successfully identified brain regions, especially those in the ventral visual pathway for object recognition (Haxby et al., 1991; Mishkin & Ungerleider, 1982), that show neural responses consistent with the concept of neural correlates of consciousness. However, the question of “from which stage of the visual pathway do the conscious representations first emerge” remains controversial.
The ventral visual pathway for object recognition can be roughly classified into the following two stages: the lower and higher regions. Neurons in the lower visual regions such as V1 and V2 have smaller receptive fields and encode simple features of stimuli (e.g., orientation of a luminance grating; Hubel & Wiesel, 1979), whereas those in the higher regions such as V4 and the interior temporal cortex have larger receptive fields and show selective responses to specific categories of shapes or objects (Fujita, Tanaka, Ito, & Cheng, 1992). Studies using unit cell recordings from monkeys have argued that neurons in the higher visual region rather than the lower visual region play a major role in creating conscious representations (Sheinberg & Logothetis, 1997; Leopold & Logothetis, 1996). These studies have found that, during binocular rivalry, activities of neurons increase when their preferred stimuli (e.g., a grating with a specific orientation or complex shapes such as faces) come into consciousness compared with when they do not, indicating that neural processing in these higher regions is directly associated with conscious percepts. In contrast, V1 and V2 neurons have shown few or no changes in response to perceptual alternations during binocular rivalry (Maier et al., 2008; Leopold & Logothetis, 1996). Thus, these results suggest that the conscious representations of stimuli emerge from the higher visual region rather than the lower visual region.
However, recent studies using fMRI have provided data that are somewhat inconsistent with this view. These studies have revealed that the lower region in the visual pathway, such as V1 and the LGN, shows responses that are coupled with perceptual changes in binocular rivalry (Haynes et al., 2005; Lee et al., 2005; Wunderlich et al., 2005; Tong & Engel, 2001). Some researchers have indicated that this activity in the early visual pathway may reflect top–down signals from the higher region (Tononi & Koch, 2008), although the properties of these top–down signals have not been completely clarified, partly because of the limited temporal resolution of fMRI. Consequently, the stages of the ventral pathway from which conscious representations emerge remain to be elucidated.
A direct approach to clarify this issue would be to isolate neural activities at the moment when conscious percepts are generated in the brain. However, it has been difficult to use these earlier psychophysical methods (e.g., binocular rivalry) to identify a precise moment at which a new conscious percept emerges because perceptual changes with these methods usually occur at a low speed (Kim & Blake, 2005), gradually switching from one percept to another (not transiently). To solve this problem, we have used a new approach that can induce a rapid change in conscious perception with little influence from bottom–up responses. In particular, we focused on neural activity in the disinhibition phase of continuous flash suppression (CFS; Tsuchiya & Koch, 2005). In CFS, a rapid sequence of colored flashes presented to one eye of the participant renders a target image (e.g., grating) invisible to the other eye (Figure 1). When the continuous flashes disappear, however, the grating is released from dichoptic suppression, changing into consciousness as if it had been presented after the continuous flashes disappeared (subjective-onset trials, Figure 1C). Taking advantage of the fine temporal resolution of magneto-encephalography (MEG), we measured the neural activity at the moment of this transition from unconscious to conscious states of perception, thereby characterizing the temporal dynamics when conscious percepts are generated in the brain.
As control trials for subjective-onset trials, we employed no-stimulus trials (Figure 1A) and physical-onset trials (Figure 1B). In physical-onset trials, the grating is presented at the offset (not onset) of continuous flashes. Participants should perceive the grating after the flashes disappear (as in subjective-onset trials), although this perception is caused by bottom–up signals from the retina. Comparison of neuromagnetic responses between physical- and subjective-onset trials helps in distinguishing differences in neural activity between conscious representations triggered by bottom–up signals (physical-onset trials) and those created through internal disinhibition of unconscious signals in the brain (subjective-onset trials). In particular, our purpose was to discern whether the first neural responses related to conscious perception emerge from the lower or higher visual region of the ventral pathway. If the consciousness-related activity emerges from the higher visual regions as suggested by previous neurophysiological studies using monkeys (Sheinberg & Logothetis, 1997; Leopold & Logothetis, 1996), we should observe the transmission of neural signals from the higher to lower visual regions in subjective-onset trials, whereas normal transmission in a feedforward (from lower to higher) direction should be observed in physical-onset trials.
This study comprised four experiments. We present our main results in the first experiment (Experiment 1a, Figures 1,2345–6), in which MEG signals in physical- and subjective-onset trials are compared. Supplementary results of a separate behavioral experiment (Experiment 1b, Figure 7) are provided to ensure that the subjective-onset grating in Experiment 1a remained invisible during the continuous flashes. Finally, we measured neuromagnetic responses again in Experiments 2a and 2b (Figures 8 and 9, respectively), with physical- and subjective-onset trials completely indistinguishable to observers.
We conducted four experiments (Experiments 1a, 1b, 2a, and 2b). In Experiments 1a and 2b, we measured neural responses of participants using MEG, whereas Experiments 1b and 2a were behavioral studies to supplement the MEG data. The number of participants were 12 (Experiment 1a), 10 (Experiment 1b), 5 (Experiment 2a), and 11 (Experiment 2b). Four participants participated in both MEG experiments (Experiments 1a and 2b). Informed consent was obtained from each participant after the study had been explained. Approval for these experiments was obtained from the ethics committee of the National Institute for Physiological Sciences, Okazaki, Japan.
Experiment 1a (MEG Study)
All visual stimuli were generated using the Matlab Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) and presented on a screen at a refresh rate of 60 Hz. For dichoptic viewing, we presented stimuli at two different locations on the screen. A square area (delineated by white lines, 6.1° × 6.1°) on the left half of the screen comprised stimuli for the left eye of the participant, whereas another area on the right half comprised stimuli for the right eye. Stimuli at these two locations were fused using a mirror stereoscope placed in front of the participant's eyes.
Every trial began with a fixation of 1 sec (pre-flash period), followed by a rapid sequence of colored Mondrian patterns (continuous flashes) to the dominant eye. We determined each participant's eye dominance using a variation of the Porta test (Roth, Lora, & Heilman, 2002). In this test, participants extended one arm and aligned a forefinger vertically with a target object in the room, with both eyes open. Next, they alternately closed each eye, and the dominant eye was defined as the eye that caused a larger alignment change between the forefinger and target object when closed. Eleven of the 12 participants had dominant right eyes. The duration of each Mondrian pattern in the continuous flashes was 50 msec, and 21 different patterns were presented in one trial (flash period, 1.05 sec). After the continuous flashes, the screen returned to a fixation lasting for 1.8 sec (post-flash period). In contrast, stimuli to the nondominant eye differed in the three types of trials (Figure 1A–C). In no-stimulus trials, only a fixation point was shown throughout the trial. In physical-onset trials, the white fixation was shown during the flash period but was followed by a grayscale grating (vertical or horizontal; size, 2.54° × 2.54°; spatial frequency, 2.54°/cycle) presented at the beginning of the post-flash period. The location of the grating was randomly selected from the four corners of the square field. In subjective-onset trials, the grating appeared synchronously with the onset of the continuous flashes, remaining on the screen until the end of the trial. The visibility of the grating during the flash period was examined later in a separate experiment (see Experiment 1b below).
Over the course of an experimental session, three types of trials above were randomly intermixed with two additional types of trials. The first type was “high-acuity subjective-onset” trials in which a grating with a higher luminance contrast and higher spatial frequency (0.38°/cycle) than that in subjective-onset trials was presented in the flash and post-flash periods. Because of a high visual acuity for the grating, the participants sometimes perceived it consciously even during the flash period, yielding trials with incomplete binocular suppression. This condition was necessary in a later experiment (see Experiment 1b) and was thus introduced here to equalize protocols and data quality across experiments. The second type was “catch” trials involving a brief change in the fixation point during the flash period. Although the fixation point over the continuous flashes was normally small (0.15°) and white, it turned into a black point of larger size (0.46°) in catch trials. This change in the fixation point lasted 300 msec, with its timing randomly selected within the flash period of 1.05 sec. No grating was presented in the flash or post-flash periods of catch trials.
The participants were asked to perform two tasks in every trial. During the flash period, their task was to detect a possible fixation change (in catch trials). After the flash disappeared, their second task was to judge the orientation of the grating. For 6 of the 12 participants, the vertical grating was presented as a target stimulus and the horizontal grating was shown as a nontarget stimulus. They pressed a button when they perceived the target stimulus at any location in the post-flash period. For the remaining participants, the horizontal grating was presented as the target stimulus, and the participants did not respond to the nontarget stimulus (vertical grating). The ratio of target/nontarget trials was 1:8 in both physical- and subjective-onset trials, and we only analyzed the MEG data from nontarget trials to avoid any influence of motor-related neural activity caused by the manual movement of the button presses. In brief, the participants detected the fixation change during the flash period and responded to the grating with the target orientation during the post-flash period (if neither event occurred, the button was not to be pressed). This dual-task procedure guided the attention of the participants to the central visual field in the flash period and to the peripheral visual fields in the post-flash period, which strongly suppressed conscious perception of the low-acuity grating during the continuous flashes (see Experiment 1b and Figure 7).
One experimental session consisted of 84 trials (12 no-stimulus trials, 24 physical-onset trials, 24 subjective-onset trials, 12 high-acuity subjective-onset trials, 6 target trials, and 6 catch trials). Physical- and subjective-onset trials were further subdivided into two types in which the nontarget grating was presented to the left and right visual fields. In the left visual field trials, the grating appeared at the upper or lower left corner of the square area, whereas in the right visual field trials, it appeared at the upper or lower right corner. A total of eight types of trials were randomly intermixed, and the participants performed six sessions per experiment.
MEG Recordings and Data Analysis
Neural activity during flash and post-flash periods was measured using a whole-head 306-channel MEG system (Vector-view, ELEKTA Neuromag, Helsinki, Finland), which recorded changes in neuromagnetic signals from 102 different positions over the scalp. A sensor element in each recording position consisted of one magnetometer and two orthogonal planar-type gradiometers (one for latitudinal and the other for longitudinal directions of changes in neuromagnetic signals). Thus, we analyzed MEG signals recorded from 204 gradiometers at 102 sensor positions. The signals from the planar-type gradiometers are strongest when the sensors are located just above local cerebral sources (Nishitani & Hari, 2002). The measurements were performed with a band-pass filter of 0.1–200 Hz at a sampling rate of 1000 Hz. Neural responses to the physical- and subjective-onsets of the nontarget gratings were investigated by recording visual-evoked fields (VEFs) time-locked to the beginning of the post-flash period. For each of the eight conditions above, neuromagnetic waveforms in 72 trials at maximum were averaged, with an averaging epoch of −1100 to 1000 msec relative to the offset of the continuous flashes. Trials in which signal variations were larger than 3000 fT/cm were excluded from analysis.
We selected 12 gradiometers (at six sensor positions) around the occipital pole as a group for the early visual cortex (Area I, Figure 2A), whereas another group representing neural signals from the higher visual region in the ventral pathway consisted of the same number of sensors over lateral (ventral) regions of the scalp (Area II). Locations of these two areas were based on our previous MEG studies (Noguchi, Inui, & Kakigi, 2004). Areas I and II were defined for each hemisphere, resulting in a total of four areas (Areas I and II × left and right hemispheres) over the entire brain.
After obtaining the across-sensor VEF for each condition of each area, we averaged the waveforms between the two hemispheres (Figure 3A and B). This between-hemisphere average was based on the stimulus–hemisphere correspondence (contralateral or ipsilateral). Thus, the across-sensor VEF of the right visual field trials in the left hemisphere (either physical- or subjective-onset) was averaged with that of the left visual field trials in the right hemisphere (contralateral condition). We then corrected for differences in baseline signals of VEFs across conditions. First, a baseline signal in each condition was calculated using VEF amplitudes from −10 to 0 msec, relative to the flash offset. On the basis of the differences in baseline signals across conditions, each VEF in the physical- and subjective-onset trials was corrected such that it shared the same baseline (−10 to 0 msec) as that in the no-stimulus trial. Grand-averaged data across the 12 participants were calculated after all procedures above were applied to the individual data.
Finally, we obtained neuromagnetic responses related to the perception of the physical- and subjective-onset gratings (a grating-related activity) by subtracting across-hemisphere VEFs among trials (Figure 4). For example, the activity related to the perception of the physical-onset grating was acquired as the differential VEF between physical-onset and no-stimulus trials. The significance of the grating-related activity was evaluated by a one-group t test applied for each time point of the differential VEF. The difference was considered significant only when p values from those one-group t tests were less than 0.05 at five consecutive time points.
Distributed Source Estimations of Neuromagnetic Signals
In addition to the areal mean and current dipole approaches (Nishitani & Hari, 2002), we estimated source locations of MEG signals using Statistical Parametric Mapping 5 (SPM5, available on-line at www.fil.ion.ucl.ac.uk/spm/). After VEFs of two gradiometers at each sensor position were combined as described above, the waveforms were low-pass-filtered at 30 Hz and down-sampled to 200 Hz. Three-dimensional source reconstructions of MEG signals were performed by the multiple sparse priors (MSP) approach (Friston et al., 2008) of SPM5. First, a tessellated cortical mesh was created for each participant using a template brain from the Montreal Neurological Institute. This mesh contained 7200 vertices and served as a brain model to estimate current source distributions. Positions of current dipoles were restricted to a cortical surface, and they were evenly placed at each node of the mesh. Second, this dipole mesh was spatially coregistered with a sensor space of MEG, based on the digitized locations of the nasion and bilateral peri-auricular points of each participant(Henson et al., 2007). We then constructed a single-shell, spherical head model for forward solutions, as implemented in BrainStorm (available on-line at neuroimage.usc.edu/brainstorm). Finally, the forward model was inverted with the MSP approach, mapping the contribution (source strength) of each dipole on the mesh to MEG sensor waveforms. We performed these MSP estimations for each time point, trial, and subject.
For second-level (group-level) analysis across the 12 participants, the MSP solutions for each trial and each participant were averaged over all time points belonging to a time window of interest (0–200 msec in Figure 6A and B). The resultant MSP averages were then analyzed by SPM random-effect analysis (Henson et al., 2007). Differences between trials (no-stimulus vs. physical-onset trials in Figure 6A) were evaluated by voxel-wise t tests of SPM5. The threshold for a significant difference was set at a p value of less than .005, false discovery rate-corrected for multiple comparisons.
Experiment 1b (Behavioral Experiment)
In Experiment 1b, we examined whether the low-acuity grating (in Experiment 1a) remained invisible during the flash period of subjective-onset trials. As shown in Figure 7A, every trial in Experiment 1b consisted of pre-flash and flash periods. The fixation point without a grating was presented to the nondominant eye of the participants in no-stimulus trials, whereas the low-acuity grating was presented in low-grating trials, as in Experiment 1a. The trials with high-acuity grating (high-grating trials, corresponding to high-acuity subjective-onset trials in Experiment 1a) were also employed to ascertain and maintain the participants' motivation for the tasks (Sterzer, Jalkanen, et al., 2009). After the flashes disappeared, the participants completed three tasks sequentially. In the first task, they answered whether the fixation in the flash period underwent changes (forced choice), as in a previous experiment. In the second task (detection task), they were asked to make yes/no judgments of the grating during the flash periods as an indicator for conscious perception of the grating. They answered “yes” when they perceived any type of grating at any location. In the last task (discrimination task), they described the orientation of the grating (vertical or horizontal) during the continuous flashes. This question was asked even if they had answered “no” in the second task to examine the possibility of unconscious (subliminal) processing of “invisible” stimuli. One experimental session included 96 trials (32 no-stimulus trials, 32 low-grating trials, 16 high-grating trials, and 16 catch trials). The participants performed one or two sessions per experiment.
Experiment 2a (Behavioral Experiment)
Although we measured VEFs to the physical- and subjective-onset gratings in Experiment 1, our preliminary experiment, in which behavioral detection times for those two types of gratings were compared (data not shown), indicated that the participants generally detected the physical-onset grating faster than the subjective-onset grating. This suggested that perceptual latency of the grating (intervals from the flash offset to the time when participants perceived the onset of the grating) were not comparable between physical- and subjective-onset trials in Experiment 1, which may affect interpretations of our MEG data. Therefore, we measured VEFs in physical- and subjective-onset trials again, with perceptual sequences between the two types of trials closely matched (Experiment 2). For this purpose, we first estimated the difference in the perceptual latency between the physical- and subjective-onset gratings (Experiment 2a) by the temporal order judgment task (McDonald, Teder-Salejarvi, Di Russo, & Hillyard, 2005; Stelmach & Herdman, 1991).
In every trial of Experiment 2a, we presented two (physical- and subjective-onset) gratings, one at the left and another at the right visual fields (Figure 8A). When the flashes disappeared, the subjective-onset grating remained on the screen for a variable duration (the lag period, variable from 100 to 400 msec in steps of 50 msec), followed by the physical-onset grating presented at the other visual field. These gratings appeared at the upper half of the square field for three participants and at the lower field for the remaining two participants. As in previous experiments, the participants monitored a possible fixation change (in catch trials) during the flash period. If no fixation change was detected, they moved to the second (temporal order judgment) task after the flash disappeared, reporting which of the two gratings appeared first (left or right, forced choice). For each lag period, we measured the percentage that participants reported the subjective-onset grating as the first one (% subjective-first). These changes in % subjective-first as a function of lag duration were then fitted with a psychometric sigmoid function. The difference in the perceptual latency between the physical- and subjective-onset gratings was estimated on the basis of a point of subjective equality, a crossing point of the psychometric curve with a 50% threshold. One experimental session consisted of 74 trials (70 temporal order judgment trials and 4 catch trials, randomly intermixed). In half of the temporal order judgment trials, the subjective-onset grating was presented at the left visual field, whereas in the other half, they were presented at the right visual field. Each participant performed three sessions.
Experiment 2b (MEG Experiment)
On the basis of the results of Experiment 2a, we again recorded VEFs in physical-onset, subjective-onset, and no-stimulus trials. All procedures were identical to those in Experiment 1a (MEG experiment), except that a blank period of 200 msec was introduced between the flash offset and presentation of the physical-onset grating, such that perceptual sequences of physical-onset trials were equalized with those of subjective-onset trials.
Experiment 1a (MEG Experiment)
Mean (±SE across subjects) detection rates of fixation change and the target grating were 89.1% (±3.3%) and 93.9% (±1.9%), respectively. Figure 2A showed VEFs in no-stimulus trials recorded by 204 gradiometers of a whole-head MEG, with time 0 corresponding to the flash offset. Clear MEG responses were observed primarily in the occipital and lateral (ventral) regions in both hemispheres, which was consistent with our previous study (Noguchi et al., 2004). Note that strong neural responses measured through the MEG gradiometers are represented either as positive or negative deflections of neuromagnetic waveforms. Figure 2B shows enlarged VEFs at three gradiometers (two in the posterior and one in the lateral regions) in which the waveforms in no-stimulus, physical-onset, and subjective-onset trials are superimposed. Physical- and subjective-onset trials were further subdivided into trials with the grating presented at the left and right visual fields. Because we averaged the waveforms from −1100 to 1000 msec after the flash offset, the MEG responses in the first half (−1000 to 0 msec) were caused by onset of the continuous flashes, whereas those in the second half (0–1000 msec) primarily indicated neural activity in response to the onset of the grating. At all three gradiometers in Figure 2B, the MEG responses related to the perception of the grating can be observed as neuromagnetic components selective for physical- and subjective-onset trials compared with no-stimulus (baseline) trials. These responses to the grating are clearly seen specifically when the grating was presented at the visual fields contralateral to the sensor positions (arrows in Figure 2B).
Collective Neuromagnetic Signals in Lower and Higher Visual Regions
To discern whether neural activity related to conscious perception emerged from the lower or higher visual region in the ventral pathway, we compared VEFs in these two regions. This comparison was made using the areal mean approach (Altmann et al., 2008; Tarkiainen et al., 2003) in which a collective (across-sensor) VEF over a subset (group) of gradiometers was calculated (see Methods). We selected 12 gradiometers around the occipital pole as a group for the early visual cortex (Area I, Figure 2A) and another group representing the signals from the higher visual region consisted of the same number of gradiometers over lateral (ventral) regions of the scalp (Area II). Figure 3A and B show these across-sensor VEFs in physical- and subjective-onset trials, with the data from the two hemispheres averaged. The red lines represent the waveforms in response to the contralateral condition, whereas blue lines represent the waveforms in response to the ipsilateral grating. The data in no-stimulus trials are also plotted (black) as baselines. Note that in these across-sensor VEFs, strong MEG responses are shown as upward deflections of the waveforms.
In all three (physical-onset, subjective-onset, and no-stimulus) trials, the onset of the continuous flashes (at −1050 msec) induced strong neuromagnetic responses approximately −1000 to 700 msec (the flash responses; Figure 3A and B, left). These responses primarily consisted of two peaks, one at approximately 80–100 msec and another at 150–200 msec after the flash onset, which was consistent with previous studies of MEG and electroencephalography (EEG; Liu, Harris, & Kanwisher, 2002; Noesselt et al., 2002; Hillyard & Anllo-Vento, 1998). Analysis using current dipoles indicated that the initial MEG component approximately 80–100 msec (M1) emerged from the early visual cortex (Figure 3C), whereas anatomical sources for the 150–200 msec component (M2) were located in the ventrolateral regions in both hemispheres, presumably corresponding to the higher visual regions in the ventral pathway. In fact, amplitudes of the M1 component were larger than those of the M2 component in Area I (Figure 3A and B, top left), indicating that the waveforms in this area primarily represented signals from the early visual cortex. In contrast, this relationship was reversed in Area II, in which the amplitudes of M2 exceeded those of M1 (Figure 3A and B, bottom left).
Grating-related Activity of Physical- and Subjective-onset Trials
Neuromagnetic responses to the grating are enlarged in Figure 3A and B (right). The waveforms in physical-onset trials (Figure 3A, right) showed basically the same patterns as the responses to the continuous flashes described above. The M1 and M2 components were observed to be dominant in Area I (Figure 3A, top right) and Area II (Figure 3A, bottom right), respectively. In contrast, the waveforms in subjective-onset trials (Figure 3B, right) were totally different from those in physical-onset trials. First, the subjective-onset waveforms showed no significant M1 component in Area I (Figure 3B, top right). In the contralateral and ipsilateral gratings, no distinct MEG responses to the grating were induced until after 200 msec, which presumably indicated the lack of a physical presentation of the grating after the flashes in subjective-onset trials. Second, despite the lack of an early MEG component in Area I, substantial neural activity in Area II was observed in the contralateral subjective-onset trial, which began as early as 110–130 msec after the flashes disappeared (Figure 3B, bottom right).
These characteristics of the subjective-onset trials are summarized in Figure 4. In this figure, we show differential waveforms in which the across-sensor VEFs in no-stimulus trials were subtracted from those in physical- or subjective-onset trials. In Area I, initial neuromagnetic signals from the early visual cortex (M1) were not observed in subjective-onset trials, although both the physical- and subjective-onset gratings produced delayed activity at a later period of 250–400 msec (Figure 4A). Figure 4B compares the differential waveforms of subjective-onset trials between Areas I and II. Point-by-point one-group t tests indicated that significant (p < .05 at five consecutive points, relative to the no-stimulus trials) activity related to perception of the subjective-onset grating was initially observed in Area II at 126 msec (green arrow), followed by delayed response in Area I at 225 msec after the flash offset (black arrow). The temporal dynamics of the neural activity in the subjective-onset trials are shown topographically in the contour maps depicted over layouts of all MEG sensors (Figure 4C). These maps indicated that the first neural response to the subjective-onset grating was detected by MEG sensors in anterior-lateral regions (140 msec). This activity was fed back to the posterior regions around the early visual cortex (230 msec) and then propagated to the entire brain, especially to lateral regions in both hemispheres (350 msec).
Comparisons of VEFs between Anterior and Posterior Halves of the Ventral Pathway
To investigate VEFs at a broader region in the ventral pathway, we defined six areas (Areas 1–6) along the ventrolateral regions of both hemispheres, obtaining the across-sensor VEF for each of these six areas (see Figure 5A for locations of these areas). The resultant waveforms in the subjective-onset trials are shown in Figure 5B. When the subjective-onset grating was perceived, the first MEG responses (100–150 msec) were observed in regions covering the anterior half (Areas 3–6) of the contralateral hemisphere. These responses were followed by delayed activity propagating into all six areas. The waveforms in the ipsilateral condition (Figure 5B, right) indicated that the initial MEG components in Areas 3–6 (100–150 msec) were selective for the contralateral condition, which eliminated the possibility that those components in the anterior regions indicated signals caused by eye blinking and microsaccades.
Source Estimation Analysis of Neuromagnetic Data
We finally performed distributed source estimation of MEG data using SPM5 (see Methods). The second-level (group-level) comparison of MSP estimations between physical-onset and no-stimulus trials (time window, 0–200 msec) revealed significant neuromagnetic responses in early and ventral visual cortices as well as several areas in the frontal lobe (Figure 6A), indicating neural activity elicited by presentation of the physical-onset grating. In contrast, no significant responses were observed in the early visual cortex (around the occipital pole) in comparison of the subjective-onset and no-stimulus trials, although this comparison showed significant activities in the ventral pathway and frontal areas (Figure 6B). These results indicated a major role of the higher visual region rather than the early visual region in creating conscious representations of the subjective-onset grating.
Another difference in the MSP results between the physical- and subjective-onset trials was the lack of activity in the lateral frontal regions in the subjective-onset trials. This may be related to downward deflections in VEF waveforms in the initial 100 msec after the flash offset (see differential VEFs at Areas 5 and 6 in Figure 5B, left). Although reasons for these negative responses in the anterior regions were unclear (and we confirmed that these downward deflections were not statistically significant, p > .05, one-group t test), such negative VEFs specific to subjective-onset trials may result in the lack of neural responses in the lateral frontal regions in Figure 6B.
Experiment 1b (Behavioral Experiment)
In Experiment 1b, we examined whether the low-acuity grating (in Experiment 1a) had remained invisible during the flash period of the subjective-onset trials. The results of the three tasks in Experiment 1b are shown in Figure 7. In Task 1, the detection rate of the fixation change was 98.4 ± 0.8%, with a false alarm rate of 0.2 ± 0.2% (mean ± SE across subjects). In contrast, detection (hit) rates of the grating in Task 2 were 1.3 ± 0.7% in low-grating and 38.8 ± 15.4% in high-grating trials, with a false alarm rate of 0.2 ± 0.2% in no-stimulus trials. No difference was observed between the false alarm rate and hit rate in the low-grating trials [t(9) = 1.48, p = .17], indicating that the low-acuity grating during the flashes was subjectively invisible. Finally, accuracies in Task 3 (discrimination of the direction of the grating) were 49.1 ± 1.7% in low-grating and 65.9 ± 8.4% in high-grating trials. The accuracy in the low-grating trials was not significantly different from the chance level [t(9) = 0.54, p = .60], which negated the possibility of unconscious processing of the grating in the flash period. Overall, the results in Experiment 1b showed that the low-acuity grating during the continuous flashes remained invisible both subjectively and objectively.
Experiment 2a (Temporal Order Judgment Task)
One of the purposes of Experiment 2 was to record VEFs again, with the perceptual sequences in the physical- and subjective-onset trials closely matched. For this purpose, we first estimated the difference in the perceptual latency of the physical- and subjective-onset gratings using the temporal order judgment task. As shown in Figure 8B, the point of subjective equality of the psychometric function was 208 ± 24 msec (mean ± SE across subjects), indicating that the perceptual latency of the subjective-onset grating was equal to that of the physical-onset grating presented at 208 msec after the flash offset. Thus, in the next experiment (Experiment 2b), we introduced a blank period of 200 msec between the flash offset and the presentation of the physical-onset grating, thereby equalizing the perceptual sequences between the two types of trials.
Experiment 2b (MEG Experiment)
In Experiment 2b, we measured VEFs again with perceptual latency of the physical- and subjective-onset gratings matched. Mean (± SE across subjects) detection rates of the fixation change and the target grating were 89.6% (±3.4%) and 97.0% (±0.9%), respectively. The across-sensor VEFs in Areas I and II are shown in Figure 9. In physical-onset trials (Figure 9A), neuromagnetic responses to the grating were clearly observed from 200 to 700 msec after the flash offset. These grating-related responses were more distinct than those in Experiment 1a (Figure 3A), probably because the introduction of the 200-msec blank after the flash period restored an excitability of visual neurons (that had been lowered by the continuous flashes of 1.05 sec). Although these augmented responses to the physical-onset grating in Experiment 2b made the distinction of M1 and M2 less clear, the temporal order of neural responses between Areas I and II was the same as that in Experiment 1a. As shown in differential waveforms of the across-sensor VEFs (Figure 9C, the physical-onset minus the no-stimulus trials), the first significant activity elicited by the grating was observed in Area I (at 307 msec, p < .05, at five consecutive time points, one-group t test), followed by that in Area II at 323 msec (note that the physical-onset grating was presented at 200 msec in this experiment). The across-sensor VEFs in the subjective-onset trials (Figure 9B) also resembled those in Experiment 1a (Figure 3B). Significant responses related to conscious perception of the grating were first seen in Area II (97 msec, green arrow in Figure 9D), not in Area I. These results thus confirmed a major role of the higher visual region rather than the lower visual region in conscious perception of the subjective-onset grating.
Similarity of Neural Waveforms between Physical- and Subjective-onset Trials
As shown in Figure 9, neuromagnetic responses in the physical-onset trials were substantially different from those in the subjective-onset trials. However, given that we equalized perceptual sequences for these two types of trials in Experiment 2b, one could expect some similarities in MEG waveforms between them (because the conscious percept was the same, the neural correlates should also be the same). Thus, we examined this similarity by performing correlation analysis of MEG waveforms between the physical- and subjective-onset trials. Because our preliminary experiment indicated that the detection time for the subjective-onset grating was about 680 msec, we performed correlation analysis using differential waveforms from 0 to 700 msec after the offset of the continuous flashes. Mean (±SE across 11 subjects) correlation coefficients between the waveforms in the physical- and subjective-onset trials (black and green lines in Figure 9C and D) were 0.33 (±0.09) in Area I and 0.32 (±0.09) in Area II. No significant difference in correlation coefficients was observed between the two areas (t = 0.16, p = .88). We then narrowed the time window of correlation analysis from 0–700 msec to 200–500 msec because our data indicated a close relationship between delayed activity observed in this time period and consciousness (Figures 4B and 9B). The resultant correlations of waveforms between the physical- and subjective-onset trials were higher in Area II (0.29 ± 0.12) than in Area I (0.16 ± 0.16), although this difference did not reach significance (t = 1.15, p = .28). Overall, our data showed that there were some correlations in MEG waveforms between the two types of trials and that these correlations were comparable or somewhat higher in the later visual regions (Area II) than the early visual regions (Area I).
Neural Responses to an Invisible Grating during the Flash Period
In Experiments 1a and 2b, we found two MEG components related to the perception of the subjective-onset grating: the initial response in the higher visual regions (around 100 msec after the flash offset) and delayed activity after 200 msec (Figures 5 and 9). An important and unanswered question was “which component of these MEG signals (initial or delayed one) was directly associated with consciousness?” One possibility was that a conscious percept of the grating was generated as a result of the initial activity in the higher visual areas. Another possibility, however, was that the conscious percept did not emerge until delayed activity was diffused into the entire brain. In the latter case, the first neural activity in the higher regions would be necessary but not sufficient for the emergence of consciousness.
To investigate this, we performed additional analysis of MEG signals during the flash period (rather than the post-flash period) using the data from 19 subjects who participated in either Experiment 1a or 2b. In the flash period of the subjective-onset trials, the grating was continuously presented to the nondominant eye of the participants, although no grating was given in the physical-onset and no-stimulus trials. Comparing the neuromagnetic signals in the subjective-onset trials with those in the other two types of trials, we should observe MEG responses related to the processing of “unseen grating” during the flash period. The results (differential VEFs in Areas I and II) are shown in Figure 10. We found significant (p < .05 for five consecutive time points) responses to the unseen grating in Area I (lower regions) and Area II (higher regions) in the first 200 msec. In contrast, no significant activity was observed after 200–400 msec, indicating that delayed activity in response to the grating was inhibited during the flash period. These results showed that the initial response in the higher visual regions (Area II) was evoked even when stimuli remained invisible to the participants. VEFs in Figure 10 thus suggested a close relationship between delayed activity, rather than the initial activity, and consciousness.
We investigated the temporal dynamics of neural activity when visual stimuli come into consciousness. Focus on the disinhibition phase of CFS enabled us to measure an activity at the moment of emergence of a conscious percept in the human brain. The areal mean VEFs (Figure 3A) in the physical-onset trials indicated that the first neuromagnetic responses (M1) were observed in the posterior regions (Area I), followed by the M2 response in the ventrolateral regions (Area II). Because our analysis using dipoles (Figure 3C) suggested that source locations for the M1 and M2 components were the lower and higher (ventral) visual regions, respectively, these results showed transmission of neural signals in a feedforward direction (from the lower to higher regions). In contrast, the areal mean VEFs in the subjective-onset trials (Figure 3B) revealed that the first neural responses to the grating were observed in Area II (not Area I), although this activity was diffused into both areas after 200 msec. These patterns of neural responses in the subjective-onset trials were further confirmed by contour map analysis (Figure 4C) and the expanded areal mean approach covering the six areas (Areas 1–6) along the ventral pathway (Figure 5). Overall, these results indicated transmission of neural signals from the higher to lower regions in the subjective-onset trials. Consistent with this view, source estimation analysis using the MSP approach (Figure 6) showed lack of activity in the lower visual areas at an initial 200 msec of the subjective-onset trials, suggesting a dominant role of the higher visual areas in conscious perception of these gratings. Finally, we replicated our results with the perceptual sequences in the physical-onset trials matched with those in the subjective-onset trials (Experiment 2, Figures 8 and 9).
Latency of the Consciousness-related Activity in the Ventral Pathway
Although several recent EEG and electrocorticography studies have investigated neural responses underlying visual awareness, there has been controversy about the latency of the first neural activity related to conscious perception. For example, a previous EEG study used a paradigm of attentional blink and compared EEG waveforms when participants reported the conscious perception of the target stimuli (“seen” trials) with those when participants did not report a conscious perception (“unseen” trials), concluding that neural responses more than 250 msec were critical for the target to access consciousness (Sergent, Baillet, & Dehaene, 2005). In contrast, an electrocorticography study used backward masking and found a significant difference in amplitudes of the N170 component between the “recognized” and “unrecognized” trials of target stimuli (Fisch et al., 2009). Finally, a study combining binocular rivalry with EEG measurements reported a neural correlate of visual awareness at a latency as early as 100 msec (P1 component; Roeber et al., 2008). One reason for this inconsistency may have been the use of nontarget visual stimuli to manipulate consciousness of the target stimuli. In attentional blink and backward masking, the mask stimuli are presented just before or after the target stimuli to render the target invisible (Fisch et al., 2009; Sergent et al., 2005). For measurements of the EEG waveforms time-locked to the perceptual alternations in the binocular rivalry, some changes in input stimuli (visual transient) were required to induce rapid alternations of conscious representations (Roeber et al., 2008). These stimulations to manipulate consciousness provoked strong bottom–up neural responses over broad regions of the visual cortex, resulting in a mixture of sensory responses with the neural components related to consciousness.
In contrast, in this study, emergence of conscious perception was triggered by the offset (not onset) of the continuous flashes. Compared with an onset of ordinary visual stimuli, the offset of these dynamic visual noises substantially inhibits the neuromagnetic responses in the visual cortex (Noguchi et al., 2004; Okusa, Kaneoke, Koyama, & Kakigi, 1998), thus providing a paradigm suitable for isolating consciousness-related activity. Our results consequently reveal that the first neural activity related to conscious perception of the subjective-onset grating emerges from the higher visual regions. Interestingly, the latency of this consciousness-related response was found to be short (approximately 100 msec after the offset of the continuous flashes; Figures 5 and 9), which may be consistent with a recent study reporting a temporal advantage of invisible stimuli for accessing consciousness (Wu, Busch, Fabre-Thorpe, & VanRullen, 2009).
Temporal Dynamics of the Neural Activity Underlying Conscious Perception
As described in the Introduction, there have been some inconsistencies between previous electrophysiological and fMRI data. Although studies using electrophysiological methods have indicated that conscious representations emerge from neural processing in the higher (not the lower) visual regions in the ventral pathway, fMRI studies have reported neural activity associated with changes in conscious percepts as early as in V1 and LGN. Our study provides data that are highly consistent with both groups of studies.
VEFs in the subjective-onset trials (Figures 4 and 5) show that the first activity related to conscious perception emerges from the anterior half of the lateral (ventral) regions in the contralateral hemisphere. These results support the view from previous electrophysiological data (Sheinberg & Logothetis, 1997; Leopold & Logothetis, 1996) indicating a major role for the ventral pathway (and the frontal cortex) in conscious perception (Crick & Koch, 2003). However, our results can also explain recent fMRI results reporting that activity in the early visual cortex correlated with changes in conscious percepts (Haynes et al., 2005; Lee et al., 2005; Wunderlich et al., 2005; Tong & Engel, 2001). Although the initial responses to the subjective-onset grating were observed in the anterior regions, these were followed by delayed activity diffusing into broad regions of the brain, including the lower visual regions (Figure 5B). Our results thus integrate previous “higher” (electrophysiological) and “lower” (fMRI) views on the emergence of consciousness, providing new perspectives on the temporal dynamics of conscious perception.
Neural Responses Directly Associated with Emergence of Consciousness
Although we identified two MEG components related to the perception of the subjective-onset grating (the initial response in the higher visual regions and delayed activity after 200 msec), our analysis in Figure 10 indicate a closer relationship of delayed rather than initial activity with consciousness. This association between consciousness and delayed activity is consistent with previous studies. A recent theoretical study argued that neural networks over broad regions in the brain play a critical role in emergence of conscious representations (Tononi & Koch, 2008). In line with this view, several neuroimaging studies reported that oscillatory brain rhythms synchronized across distant brain regions, such as those in theta and gamma bands, were strongly coupled with changes in conscious percepts in the binocular rivalry (Doesburg, Green, McDonald, & Ward, 2009) and an apparent motion across visual hemispheres (Rose & Buchel, 2005). Given that delayed activity in this study was observed in a broad region of the brain (Figures 4 and 5), our delayed activity may also indicate neural signals for communications across multiple cortical areas (as the oscillation signals). Although these neural communications were inhibited during the flash period (as seen in a lack of delayed activity to the subjective-onset grating; Figure 10A), the disappearance of continuous flashes enabled an exchange of visual information regarding the grating between lower and higher visual regions, which triggered the emergence of delayed activity and conscious percept of the grating in the post-flash period.
This work was supported by a grant from the Japan Society for the Promotion of Science for Young Scientists to Y. N. and a grant from Japan Society for the Promotion of Science for Young Researchers to T. Y. We thank Mr. Y. Takeshima (National Institute for Physiological Sciences, Japan) for his technical support. The authors would like to thank Enago (www.enago.jp) for the English language review.
Reprint requests should be sent to Yasuki Noguchi, Department of Psychology, Kobe University, 1-1 Rokkodai-cho, Nada, Kobe 657-8501, Japan, or via e-mail: firstname.lastname@example.org.