Abstract

The neural underpinnings of perceptual awareness have been extensively studied using unisensory (e.g., visual alone) stimuli. However, perception is generally multisensory, and it is unclear whether the neural architecture uncovered in these studies directly translates to the multisensory domain. Here, we use EEG to examine brain responses associated with the processing of visual, auditory, and audiovisual stimuli presented near threshold levels of detectability, with the aim of deciphering similarities and differences in the neural signals indexing the transition into perceptual awareness across vision, audition, and combined visual–auditory (multisensory) processing. More specifically, we examine (1) the presence of late evoked potentials (∼>300 msec), (2) the across-trial reproducibility, and (3) the evoked complexity associated with perceived versus nonperceived stimuli. Results reveal that, although perceived stimuli are associated with the presence of late evoked potentials across each of the examined sensory modalities, between-trial variability and EEG complexity differed for unisensory versus multisensory conditions. Whereas across-trial variability and complexity differed for perceived versus nonperceived stimuli in the visual and auditory conditions, this was not the case for the multisensory condition. Taken together, these results suggest that there are fundamental differences in the neural correlates of perceptual awareness for unisensory versus multisensory stimuli. Specifically, the work argues that the presence of late evoked potentials, as opposed to neural reproducibility or complexity, most closely tracks perceptual awareness regardless of the nature of the sensory stimulus. In addition, the current findings suggest a greater similarity between the neural correlates of perceptual awareness of unisensory (visual and auditory) stimuli when compared with multisensory stimuli.

INTRODUCTION

During waking hours, signals are continually impinging upon our different sensory organs (e.g., eyes, ears, skin), conveying information about the objects present and the events occurring within our environment. This flood of information challenges the limited processing capabilities of our central nervous system (James, 1890). As a consequence, much work within cognitive psychology and neuroscience has sought to understand how the human brain tackles this challenge by effectively filtering, segregating, and integrating the various pieces of sensory information to generate a coherent perceptual Gestalt (Murray & Wallace, 2012; Treisman & Gelade, 1980; Broadbent, 1958).

The bulk of the evidence to date with regard to the intersection between the bottleneck of information processing and perceptual awareness has been derived from studies focused on the visual system (Dehaene, Lau, & Kouider, 2017; Koch, 2004; Zeki, 2003). In fact, all major neurobiological theories regarding perceptual awareness, all emphasizing the importance of engaging widely distributed brain networks (Tallon-Baudry, 2012; van Gaal & Lamme, 2012; Naghavi & Nyberg, 2005), have been derived from observations within the visual neurosciences (Faivre, Arzi, Lunghi, & Salomon, 2017; Sanchez, Frey, Fuscà, & Weisz, 2017). In parallel, the neural markers associated with perceptual awareness have been derived from observations probing the visual system. Early fMRI (Dehaene et al., 2001), EEG (Del Cul, Baillet, & Dehaene, 2007; Sergent, Baillet, & Dehaene, 2005), and electrocorticographical (Gaillard et al., 2009) studies suggested that perceptual awareness was associated with the broadcasting of neural signals beyond primary (visual) cortex (Lamme, 2006) and more specifically engaging frontoparietal regions (Dehaene, Changeux, Naccache, Sackur, & Sergent, 2006). Arguably, the most consistent signature associated with this generalized neural recruitment is the P3b ERP. Namely, although early EEG components are similar regardless of whether stimuli enter perceptual awareness, stimuli that are perceived (vs. nonperceived) additionally yield components at later latencies. Subsequent studies converged on the observation that perceived stimuli broadcasted or triggered activity beyond that disseminated by nonperceived stimuli but emphasized that the neural ignition associated with awareness resulted in neural patterns that were both more reproducible (Schurger, Pereira, Treisman, & Cohen, 2010) and stable (Schurger, Sarigiannidis, & Dehaene, 2015) than patterns seen for nonperceived stimuli. In the latest itineration of the argument emphasizing the recruitment of global neural networks, researchers have highlighted the pivotal role of neural networks that are both integrated and differentiated (Cavanna, Vilas, Palmucci, & Tagliazucchi, 2017; Koch, Massimini, Boly, & Tononi, 2016a, 2016b; Tononi, Boly, Massimini, & Koch, 2016). Within this latter framework, the complexity of both resting state and evoked neural responses has emerged as a marker for perceptual awareness (Schartner, Carhart-Harris, Barrett, Seth, & Muthukumaraswamy, 2017; Andrillon, Poulsen, Hansen, Léger, & Kouider, 2016; Sarasso et al., 2015; Schartner et al., 2015; Casali et al., 2013).

It has been assumed that these theories of and neural markers for perceptual awareness gleaned from the visual system apply across sensory domains, an assumption that indeed comes with some supporting evidence. For example, there is a late sustained neural activity in perceived as opposed to nonperceived auditory stimulation conditions (Sadaghiani, Hesselmann, & Kleinschmidt, 2009). However, there are also important differences across sensory modalities, such as the association of auditory awareness with neural activity in fronto-temporal, as opposed to frontoparietal, networks (Joos, Gilles, Van de Heyning, De Ridder, & Vanneste, 2014). In an important recent contribution, Sanchez and colleagues (2017) demonstrated that, by applying machine learning techniques, it is possible to decode perceptual states (i.e., perceived vs. nonperceived) across the different sensory modalities (i.e., vision, audition, somatosensory). Although it is interesting that decoding of perceptual states across modalities is feasible, this observation does not tell us whether (and how) the brain performs this task. Last, Sanchez and colleagues (2017) have probed perceptual states across unisensory modalities, but to the best of our knowledge, no study has characterized differences between perceived and nonperceived stimuli across both unisensory and multisensory modalities. This knowledge gap is important, as in recent years, keen interest has emerged concerning the role played by multisensory integration in the construction of perceptual awareness (Faivre et al., 2017; O'Callaghan, 2017; Deroy, Chen, & Spence, 2014; Spence & Deroy, 2013). Indeed, as discussed above, theoretical models posit an inherent relationship between the integration of sensory information and perceptual awareness. For example, mathematical and neurocognitive formulations, such as integrated information theory (Tononi, 2012), global neuronal workspace theory (Dehaene & Changeux, 2011), and recurrent/reentrant networks (Lamme, 2006), postulate—explicitly or implicitly—that the integration of sensory information is a prerequisite for perceptual awareness. For example, integrated information theory posits that a particular spatio-temporal configuration of neural activity culminates in subjective experience when the amount of integrated information is high. In many of these views, subjective experience (i.e., perceptual awareness) relates to the degree to which information generated by a system as a whole exceeds that independently generated by its parts.

Motivated by this theoretical perspective emphasizing information integration in perceptual awareness and noting that our perceptual Gestalt is built upon a multisensory foundation, we argue that multisensory neuroscience is uniquely positioned to inform our understanding of perceptual awareness (Salomon et al., 2017; Blanke, Slater, & Serino, 2015; Noel, Wallace, & Blake, 2015; Faivre, Mudrik, Schwartz, & Koch, 2014; Mudrik, Faivre, & Koch, 2014; in addition, see Deroy et al., 2014, for a provocative argument implying that unisensory-derived theories of perceptual awareness cannot be applied to multisensory experiences). Consequently, in the current work, we aim to characterize electrophysiological indices of perceptual awareness across both unisensory (visual alone, auditory alone) and multisensory (combined visual–auditory) modalities. More specifically, we aim to establish whether previously reported neural markers of visual awareness generalize across sensory modalities (from vision to audition) onto the promotion of multisensory experiences. In the current study, we examine EEG responses to auditory, visual, and combined audiovisual stimuli presented close to the bounds of perceptual awareness. Analyses are centered around previously reported indices of visual awareness—the presence of late components in evoked potentials during perceived but not nonperceived trials (e.g., Dehaene et al., 2017; Dehaene & Changeux, 2011) as well as changes in neural reproducibility (Schurger et al., 2010) and complexity (Koch et al., 2016a, 2016b; Tononi et al., 2016).

METHODS

Participants

Twenty-one (mean age = 22.5 ± 1.9 years, median = 21.2 years, range = 19–25 years; nine women) right-handed graduate and undergraduate students from Vanderbilt University took part in this experiment. All participants reported normal hearing and had normal or corrected-to-normal eyesight. All participants gave written informed consent to take part in this study, the protocols for which were approved by Vanderbilt University Medical Center's Institutional Review Board. EEG data from two participants were not analyzed as we were unsuccessful in driving their target detection performance within a predefined range (see below; see Figure 1, dotted lines), and thus data from 19 participants formed the actual analyses presented here.

Figure 1. 

Experimental design and methods. (A) Experimental design: Participants fixated a small audiovisual device controlled via a microcontroller and reported via button press the detection of targets (audio, visual, or audiovisual). Targets were presented within visual and auditory noise whose levels were adaptively adjusted over trials. Catch trials (no targets) were also presented. (B) Auditory and visual noise levels were adjusted online for each participant to generate auditory and visual detection rates between 30% and 45%. Each line represents a single participant and plots their detection rate as a function of trial number. Participants converged on stable performance after approximately 200 trials per stimulus condition. Thus, these first 200 trials were not analyzed. Two participants exhibited high false detection rates (catch trial; dotted lines), and thus their EEG data were not analyzed. These same participants are depicted with dotted lines for audio (blue), visual (red), and audiovisual (green) conditions. Note that false alarms (catch trials) remain stable across the duration of the experiment, indicating that results are not likely to be affected by training or fatigue effects.

Figure 1. 

Experimental design and methods. (A) Experimental design: Participants fixated a small audiovisual device controlled via a microcontroller and reported via button press the detection of targets (audio, visual, or audiovisual). Targets were presented within visual and auditory noise whose levels were adaptively adjusted over trials. Catch trials (no targets) were also presented. (B) Auditory and visual noise levels were adjusted online for each participant to generate auditory and visual detection rates between 30% and 45%. Each line represents a single participant and plots their detection rate as a function of trial number. Participants converged on stable performance after approximately 200 trials per stimulus condition. Thus, these first 200 trials were not analyzed. Two participants exhibited high false detection rates (catch trial; dotted lines), and thus their EEG data were not analyzed. These same participants are depicted with dotted lines for audio (blue), visual (red), and audiovisual (green) conditions. Note that false alarms (catch trials) remain stable across the duration of the experiment, indicating that results are not likely to be affected by training or fatigue effects.

Materials and Apparatus

Visual and auditory target stimuli were controlled via a microcontroller (SparkFun Electronics, Redboard, Boulder, CO) under the control of purpose-written MATLAB (The MathWorks, Natick, MA) and Arduino scripts. The microcontroller drove the onset of a green LED (3-mm diameter, 596- to 572-nm wavelength, 150 mcd) and a Piezo Buzzer (12-mm diameter, 9.7 mm tall, 60 dB (SPL), 4 kHz, 3-V rectangular wave). Target stimuli were 10 msec in duration (square wave, onset and offset < 1 msec, as measured via oscilloscope). The LED was mounted on the Piezo Buzzer, thus forming a single audiovisual object that was placed at the center of a 24-in. computer monitor (Asus VG248QE, LED-backlit, 1920 × 1080 resolution, 60-Hz refresh rate). In addition to the targets, to adjust participants' detection rates, we online adjusted the luminance and amplitude of background visual and auditory white noise with the Psychophysics toolbox (Brainard, 1997; Pelli, 1997). The luminance (achromatic and uniform) of the screen upon which the audio and visual targets were mounted was adjusted between 0 and 350 cd/m2 in steps of 4 RGB units (RGB range = 0–255 units, initial = [140, 140, 140] RGB), and auditory noise comprised variable-intensity white noise broadcast from two speakers placed symmetrically to the right and left sides of the monitor (Atlas Sound EV8D 3.2 Stereo). The white noise track initialized at 49 dB and adjusted in 0.4-dB increments (44.1-kHz sampling rate). Visual and auditory noise were adjusted by a single increment every 7–13 trials (uniform distribution) to maintain unisensory detection performance between 30% and 45%. This low unisensory detection rate was chosen to ensure satisfactory bifurcation between “perceived” and “nonperceived” trials in both unisensory and multisensory trials (Murray & Wallace, 2012).

Procedure and Experimental Design

Participants were fitted with a 128-electrode EGI Netstation EEG and seated 60 cm away from the stimulus and noise generators. Participants completed 12–14 blocks containing 200 repetitions of target detection, in which no-stimulus (catch trials), auditory-only, visual-only, and audiovisual trials were distributed equally and interleaved pseudorandomly. We employed a subjective measure of awareness (similar to a yes/no detection judgment; Merkile, Smilek, & Eastwood, 2001; see Figure 1) in conjunction with an extensive set of EEG analyses (electrical neuroimaging framework; Brunet, Murray, & Michel, 2011; see below). Thus, albeit perceptual awareness may arguably occur without the capacity for explicit report (see Eriksen, 1960), here, we operationalize perceptual awareness as the detection and report of sensory stimuli (see below for signal detection analyses suggesting that criterion for detection was unchanged across experimental conditions and detection reports likely reflected perceptual awareness). Participants were asked to respond, via manual response (button press), as quickly as possible when they detected a stimulus. Interstimulus interval was composed of a fixed duration of 800 msec, plus a uniformly distributed random duration between 0 and 2000 msec. The total duration of the experiment was approximately 3 hr 30 min, with rest periods in between blocks of approximately 5 min.

EEG Data Acquisition and Rationale

We contrasted participants' EEG responses for perceived (i.e., detected) versus nonperceived (i.e., nondetected) unisensory (i.e., either visual or auditory) and multisensory (i.e., conjoint visual and auditory) stimuli to determine whether indices of visual awareness generalize across sensory domains. High-density continuous EEG was recorded from 128 electrodes with a sampling rate of 1000 Hz (Net Amps 200 amplifier, Hydrocel GSN 128 EEG cap; EGI Systems, Inc.) and referenced to the vertex. Electrode impedances were maintained below 50 kΩ throughout the recording procedure and were reassessed at the end of every other block. Data were acquired with Netstation 5.1.2 running on a Macintosh computer and online high-pass filtered at 0.1 Hz.

Analysis

Behavioral

Data were compiled for detection as a function of the sensory modality stimulated, where “detection” refers to a manual response immediately after presentation of a stimulus or a pair of stimuli. Two participants generated false alarm rates (reports of stimulus detection on catch trials when no stimulus was presented) that exceeded 2.5 SDs of the population average (false alarm rates = ∼20% compared with 8.2%; see Figure 1), leading to exclusion of their data from further analysis. Data were analyzed for RTs and in light of signal detection theory (Macmillan & Creelman, 2005; Tanner & Swets, 1954). To quantify sensitivity and response bias to the detection of near-threshold sensory stimuli across different sensory modalities, reports of detection during the presence of auditory, visual, or audiovisual stimuli were considered as hits. Analogously, reports of the presence of sensory stimulation during a catch trial were taken to index false alarms. Noise and signal distributions were assumed to have an equal variance, and sensitivity (i.e., d′) and response criteria (i.e., c) were calculated according to equations in Macmillan and Creelman (2005). Note that the assumption of equal variance does not affect quantification of the response criteria and simply scales sensitivity. Regarding RTs, data were trimmed for trials in which participants responded to stimuli within 100 msec of stimulus (0.9% total data trimmed) and were then aggregated.

EEG Preprocessing

As illustrated in Figure 1A, after 200 trials of each sensory condition (four blocks), relatively few adjustments of auditory and visual noise were needed to maintain participants within the predefined range of 30–45% unisensory detection performance (see also Control Analyses in Supplementary Materials online). That is, 65.42% for all audio noise adjustments were undertaken during the first 200 trials (thus, 34.58% were undertaken during the last 500 experimental trials), and 60.75% of visual noise adjustments happened during that same period (leaving 39.25% of visual noise changes occurring during the 500-trial experimental phase). Thus, EEG analysis (below) was restricted to the last 400–500 trials per sensory condition to reduce variability in the stimulus statistics. Data from these trials were exported to EEGLAB (Delorme & Makeig, 2004), and epochs were sorted according to sensory condition (i.e., A, V, AV, or none) and detection (perceived vs. nonperceived). Epochs from −100 to 500 msec after target onset were high-pass filtered (zero phase, eighth-order Butterworth filter) at 0.1 Hz, low-pass filtered at 40 Hz, and notch filtered at 60 Hz. EEG epochs containing skeletomuscular movement, eye blinks, or other noise transients and artifacts were removed by visual inspection. After epoch rejection, every condition (4 [sensory modalities: none, audio, visual, and audiovisual] × 2 [perceptual report: perceived and nonperceived]) was composed of an average of 179.16 ± 39 trials (average epoch rejection = 23.5%), with the exception of the catch perceived condition, which had 23.2 ± 3.9 trials, and catch nonperceived condition, which had 307.45 ± 31.5 trials. Excluding catch trials, there was no effect of sensory modality, perceptual report, or interaction between these with regard to total amount of trials (all ps > .19). Channels with poor signal quality (e.g., broken or excessively noisy electrodes) were then removed (6.2 electrodes on average, 4.8%). Data were rereferenced to the average and baseline corrected to the prestimulus period. Excluded channels were reconstructed using spherical spline interpolation (Perrin, Pernier, Bertrand, Giard, & Echalier, 1987). To account for the inherent multiple comparisons problem in EEG, we set alpha at <.01 for at least 10 consecutive time points (Guthrie & Buchwald, 1991), and most statistical reporting in the results states significant periods as “all ps < .01.”

Global Field Power

The global electric field strength was quantified using global field power (GFP; Lehmann & Skrandies, 1980). This measure is equivalent to the standard deviation of the trial-averaged voltage values across the entire electrode montage at a given time point and represents a reference- and topographic-independent measure of evoked potential magnitude. This measure is used here to index the presence (or absence) of late evoked potentials during perceived versus nonperceived visual, auditory, and audiovisual trials. On a first pass, we calculated average GFPs for each participant as well as for the sample as a whole (i.e., grand average) and for every condition. Then, the topographic consistency test (TCT; Koenig & Melie-Garcia, 2010) was applied across the entire epoch (−100 to 500 msec poststimulus) for each condition to determine whether there was statistical evidence for a consistent evoked potential. Subsequently, the TCT was applied at each time point for those conditions demonstrating a significant evoked potential to ascertain period during which evoked potentials were reliably evoked. For these analyses, alpha was a priori set to .05, false discovery rate (FDR) corrected (Genovese, Lazar, & Nichols, 2002)—the default alpha assumed by the test (Koenig & Melie-Garcia, 2010). After demonstrating the presence of evoked potentials relative to baseline (see above), we conducted a 3 (Sensory modality: audio, visual, audiovisual) × 2 (Perceptual state: perceived vs. nonperceived) repeated-measures ANOVA at each time point (−100 prestimuli onset to 500 msec poststimuli onset). Separate t tests across states of perception (perceived vs. nonperceived) for the different modalities (audio, visual, and audiovisual) were equally conducted. Last, to ascertain true multisensory interactions, we contrasted the GFP evoked by the audiovisual condition to the sum of the unisensory responses (e.g., Cappe, Thelen, Romei, Thut, & Murray, 2012). As a control, we equally index the GFP evoked by detected (i.e., false alarms) and nondetected (i.e., correct rejections) catch trials to ascertain whether either the noise features utilized to mask targets or the simple fact of reporting detection was sufficient to engender a GFP differentiation between conditions. The GFP analysis was solely conducted on participants with at least 20 false alarm trials (13 of 20 participants). For this analysis, a random subset of correct rejection trials was pulled for each individual to match the number of false alarm and correct rejection trials at an individual participant level. Complementing the GFP analyses, the topography exhibited by the different conditions was likewise examined. However, these are presented in the supplementary materials (see Figure S2) and not in the main text, as no strong theoretical prediction exists regarding a neural correlate of consciousness across unisensory and multisensory domains in topography (although see Britz, Díaz Hernàndez, Ro, & Michel, 2014).

Intertrial Variability Analyses

To probe the reproducibility of evoked potentials during different perceptual states and as elicited by stimuli of different modalities in a relatively simple manner, principal component analysis (PCA) was performed within each participant. More specifically, PCA identified the number of orthogonal dimensions, expressed as a proportion of the total possible (e.g., number of trials analyzed), needed to express a certain amount of the trial-to-trial variability (90% in the present case) for each channel. In a deterministic system with highly stereotyped responses, only a few dimensions are needed to capture most of the variability. To the extent that trial-to-trial recordings differ from one another, total variability increases, and PCA dimensionality increases. In the present case, each participant's data were divided into channel- and experimental-condition-specific matrices of single-trial data, with trials as rows and time points as columns. The dimensionality of each matrix was determined as a minimum number of principal components capturing 90% of the variance across trials. This number was further expressed as a percentage of the total number of dimensions and was taken as a measure of trial-to-trial variability for a given channel. For the audio, visual, and audiovisual conditions (for both perceived and nonperceived trials), the 120 trials whose mean most faithfully represented the average GFPs (determined via minimization of absolute value residuals), and thus the average response, were analyzed to maintain the number of potential dimensions equal across conditions. For the catch trials, all false-alarm catch trials were taken, and an equal number of correct-rejection catch trials were randomly selected on a participant-by-participant basis. This PCA was performed on a 101-msec wide sliding window (first originating at −100 msec and terminating at 0 msec poststimuli onset, 1-msec step size) to determine the temporal time course of the trial-to-trial variability (note that this time course analysis is thus smoothed). Results (Figure 5) are reported as the percentage of extra dimensions needed for each sensory modality to explain trial-to-trial variance in the perceived versus nonperceived conditions. As for the GFP analyses, catch trials were separately analyzed as a control procedure. A random subset of correct-rejection catch trials was sampled for each participant to match the number of correct rejections and false alarm trials. This last analysis was solely undertaken for participants with at least 20 false alarm trials (13 of 20 participants).

Lempel-Ziv Complexity

Last, Lempel-Ziv (LZ) complexity was quantified for each condition as a measure of complexity indirectly related to functional differentiation/integration (Sanchez-Vives, Massimini, & Mattia, 2017; Koch et al., 2016a, 2016b; Tononi et al., 2016; Casali et al., 2013). LZ is the most popular of the Kolmogorov classes (routinely used to generate TIFF images and ZIP files) and measures the approximate amount of nonredundant information contained within a string by estimating the minimal size of the “vocabulary” necessary to describe the entirety of the information contained within the string in a lossless manner. LZ can be used to quantify distinct patterns in symbolic sequences, especially binary signals. Before applying the LZ algorithm, as implemented in calc_lz_complexity.m, we first down-sampled our signal from 1000 to 500 Hz and converted it to a binary sequence. For every participant and every trial separately, we first full-wave rectified the signal and then assigned a value of “1” to a time point if the response was 2 SDs above the mean baseline value for that particular trial (−100 to 0 msec poststimuli onset). If the response was not 2 SDs above the mean baseline, a value of “0” was assigned. Next, binary strings were constructed for each trial by column-wise concatenating the values at each of the 128 electrodes (Casali et al., 2013) for the entire period poststimuli. Finally, the LZ complexity algorithm determined the size of the dictionary needed to account for the pattern of binary strings observed. The same procedure was repeated after shuffling the binary data after column-wise concatenation. This procedure was undertaken to calculate surrogate data with a priori maximal complexity given the entropy in the original data set. Finally, LZ was normalized by expressing it as the fraction of nonshuffled complexity divided by the shuffled version of the measure (see Andrillon et al., 2016, for a similar approach).

RESULTS

Behavioral RT

As expected from classical multisensory paradigms, a one-way repeated-measures ANOVA with four sensory conditions (none, audio, visual, and audiovisual) demonstrated a significant effect on RTs (F(3, 60) = 103.193, p < .0001). As illustrated in Figure 2A, this effect may have been driven by false alarms during catch trials, which were very slow (mean catch trials = 0.975 ± 0.10 sec [mean ± 1 SEM]) because neither stimulus was presented. In fact, the mean RT for catch trials (0.975 sec) was no different from the statistically expected value drawn from a fixed duration of 800 msec, plus a random duration between 0 and 2 sec described by a uniform distribution (one-sample t test to 0.9, p = .09). That is, on average, participants false alarmed halfway through the interstimulus interval. Thus, a one-way repeated-measures ANOVA with three conditions (audio, visual, and audiovisual) was performed and demonstrated a significant effect of Sensory modality (F(1, 20) = 720.19, p < .001, η2 = 0.97). The main effect was driven by the multisensory condition being fastest (M = 0.555 ± 0.09 sec), followed by the auditory (M = 0.588 ± 0.10 sec) and then the visual (M = 0.633 ± 0.11 sec) conditions (all comparisons are paired-samples t test with p < .046, Bonferroni corrected). Detection of audiovisual stimuli was faster than detection of the fastest unisensory stimulus defined on a participant-by-participant basis (audiovisual vs. fastest unisensory, p = .012; see Figure 2A and Methods for details).

Figure 2. 

Psychophysical results. (A) Mean RTs per sensory modality condition in response to perceived stimuli. Please note that the y axis does not commence at 0 msec but at 400 msec. (B) Sensitivity (i.e., d′) and (C) criterion for audio (blue), visual (red), and audiovisual (green) conditions. Error bars indicate +1 SEM across participants.

Figure 2. 

Psychophysical results. (A) Mean RTs per sensory modality condition in response to perceived stimuli. Please note that the y axis does not commence at 0 msec but at 400 msec. (B) Sensitivity (i.e., d′) and (C) criterion for audio (blue), visual (red), and audiovisual (green) conditions. Error bars indicate +1 SEM across participants.

Behavioral Sensitivity

On average, participants responded “yes” on 8.2% (mean) ± 1.1% (SEM) of the catch trials (i.e., false alarms), 45.1 ± 3.9% of the audio trials (d′ = 1.21, c = 0.92), 41.7 ± 4.6% of the visual trials (d′ = 1.27, c = 0.89), and 64.8 ± 4.7% (d′ = 1.83, c = 0.88) of the audiovisual trials (Figure 2B). Thus, we were successful in driving participants' performance to a detection rate that allowed the bifurcation of data with regard to perceptual report—perceived versus nonperceived. Note that, as illustrated in Figure 1, false alarm rates remained constant throughout the experiment, suggesting little fatigue or learning effects. A one-way ANOVA and subsequent paired-samples t tests on sensitivity (i.e., d′) values extracted from signal detection analyses (Macmillan & Creelman, 2005; Tanner & Swets, 1954) suggested that participants were most sensitive to the multisensory presentations (F(2, 40) = 19.84, p < .001; paired-samples t tests on audiovisual d′ vs. most detected unisensory d′, p = .007). Last, response criterion (i.e., c) was unchanged across the different sensory conditions (F(2, 40) = 0.05, p = .94; see Figure 2B). Thus, the behavioral data from this task illustrate multisensory facilitation in the form of the frequency, sensitivity, and speed of stimulus detection, while showing no change in response criterion. This last observation is particularly important as it suggests that participants' overt reports of stimulus detection reflect perceptual awareness as opposed to a change in what they consider “reportable.”

GFP

TCT (Koenig & Melie-Garcia, 2010) over the entire poststimuli interval demonstrated a reliable evoked potential when participants were presented with auditory, visual, or audiovisual stimuli, both when participants reported perceiving or not perceiving the stimuli (all ps < .01, FDR corrected). In contrast, no consistent evoked potential was apparent during catch trials, regardless of whether participants reported a stimulus or not (all ps > .08, FDR corrected).

For auditory stimuli, examination of the temporal time course of evoked potentials revealed deviations from baseline between 64 and 112 msec poststimulus and then from 134 msec poststimulus throughout the rest of the epoch for trials in which the stimulus was perceived and for the interval between 72 and 448 msec poststimulus and then again from 461 msec poststimulus throughout the rest of the epoch when the stimuli were not perceived. For visual stimuli, deviations from baseline were seen between 76 and 90 msec poststimulus and then from 138 msec poststimulus throughout the rest of the epoch when the stimuli were perceived and between 90 and 354 msec poststimulus and then from 387 msec poststimulus throughout the rest of the epoch when the stimuli were not perceived. Finally, for the audiovisual condition, evoked potentials were consistently seen beginning at 45 msec poststimulus and throughout the rest of the epoch when the stimuli were perceived and beginning at 92 msec poststimulus and throughout the rest of the epoch when the stimuli were not perceived.

Contrasts of the GFPs between conditions demonstrated a significant difference between perceived versus nonperceived stimuli for each of the three sensory conditions (see Figure 3). The statistically significant difference between perceptual states (i.e., main effect of Perceptual state in a 2 [Perceptual state] × 3 [Sensory modality (excluding catch trials)] repeated-measures ANOVA, n = 19, all ps < .01) was transient for the interval spanning 53–72 msec poststimulus onset and sustained after 102 msec, with an almost complete absence of late (i.e., +300 msec) response components for nonperceived stimuli (see Sanchez et al., 2017; Sperdin, Spierer, Becker, Michel, & Landis, 2015; Gaillard et al., 2009; Del Cul et al., 2007; Sergent et al., 2005; Dehaene et al., 2001, for similar results as well as Dehaene & Changeux, 2011, for a review). Stated simply, both perceived and nonperceived stimuli generated similar early sensory responses (<∼120 msec poststimuli onset). In contrast, the presence of relatively late (>∼120 msec poststimuli onset) response components was associated with perceived stimuli. Also statistically significant was the main effect of stimulus modality in the intervals between 110 and 131 msec poststimulus (n = 19, all ps < .01; this likely reflects auditory evoked potentials) and between 194 and 240 msec poststimulus (n = 19, all ps < .01; this likely reflects visual evoked potentials; Luck, 2005). Not surprisingly, given the lack of significant evoked potentials in these conditions (see above), paired-sampled t tests revealed no difference in the GFP evoked by “perceived” and “nonperceived” catch trials (all ts(12) < 1, all ps > .57), although this analysis relied on a considerably reduced number of trials (see Methods). Furthermore, results revealed a significant interaction between perceptual state and sensory condition 115 msec poststimuli onset and onward. Separate t tests across perceptual states (perceived vs. nonperceived) for the different sensory conditions (audio, visual, and audiovisual) revealed that, for auditory stimuli, the GFP diverged for perceived versus nonperceived stimuli at 121 msec poststimulus onset. For visual stimuli, this divergence occurred at 219 msec, whereas for multisensory stimuli, the divergence began 234 msec after stimulus onset.

Figure 3. 

Audio, visual, and audiovisual GFPs. Mean GFP across the entire montage of electrodes for each experimental condition: auditory (blue), visual (red), and audiovisual (green). Lighter shades are used for perceived stimuli, whereas darker colors are used for nonperceived stimuli. Shaded areas represent SEM over all participants, and black bars indicate intervals over which GFP was significantly different (p < .01) across perceptual states. On the x axis, 0 indicates stimulus onset.

Figure 3. 

Audio, visual, and audiovisual GFPs. Mean GFP across the entire montage of electrodes for each experimental condition: auditory (blue), visual (red), and audiovisual (green). Lighter shades are used for perceived stimuli, whereas darker colors are used for nonperceived stimuli. Shaded areas represent SEM over all participants, and black bars indicate intervals over which GFP was significantly different (p < .01) across perceptual states. On the x axis, 0 indicates stimulus onset.

Next, we determined whether the difference in GFP magnitude for perceived versus nonperceived multisensory stimuli could be explained by a simple combination of the unisensory responses. To do so, we compared the multisensory responses (perceived and nonperceived) to the sum of the unisensory responses (perceived and nonperceived; see Cappe et al., 2012; Cappe, Thut, Romei, & Murray, 2010, for a similar analysis). To do so, the evoked potentials for the unisensory conditions were first summed, and then the GFP was extracted (see Methods). This analysis showed a significant main effect of Sensory modality (A + V > AV; see Figure 4) beginning at 183 msec (n = 19, repeated-measures ANOVA, all ps < .01) and a main effect of Perceptual state (perceived > nonperceived; see Figure 4, bottom) between 97 and 188 msec poststimulus onset and from 222 msec onward (n = 19, repeated-measures ANOVA, all ps < .01). Most importantly, the results indicated a significant interaction such that multisensory responses to perceived stimuli were weaker than the sum of the two unisensory responses in a manner that differed significantly from the comparison of multisensory responses with nonperceived stimuli (n = 19, 2 [Perceptual state] × 2 [Sum unisensory vs. multisensory] repeated-measures ANOVA interaction, all ps < .01, 251 msec onward; see Figure 4, dark area and line indicating significance). Follow-up analyses using paired t tests showed no difference between the pair and the sum when stimuli were not perceived (all ps > .043) but showed a difference between these conditions beginning 194 msec poststimulus onset (p < .01) when the stimuli were perceived.

Figure 4. 

GFP as a function of sensory modality, perceived state, and comparisons between the sum of unisensory conditions and multisensory conditions. Top row: Same as Figure 2, with average GFP traces over all participants for the three sensory conditions superimposed for nonperceived (left) and perceived (right) trials. Bottom row: A linear model of GFP depicting the actual multisensory response (green) relative to an additive model (sum: A + V; purple). Black area below the 0 μV line represents the difference between predicted and actual multisensory responses. Note the strong deviations from summation for the perceived multisensory conditions and the lack of such differences for the nonperceived conditions. The black horizontal bar indicates a significant difference (p < .01) between the GFP of the summed unisensory evoked potential and the GFP of the multisensory condition when perceived. Shaded area represents ±1 SEM across participants.

Figure 4. 

GFP as a function of sensory modality, perceived state, and comparisons between the sum of unisensory conditions and multisensory conditions. Top row: Same as Figure 2, with average GFP traces over all participants for the three sensory conditions superimposed for nonperceived (left) and perceived (right) trials. Bottom row: A linear model of GFP depicting the actual multisensory response (green) relative to an additive model (sum: A + V; purple). Black area below the 0 μV line represents the difference between predicted and actual multisensory responses. Note the strong deviations from summation for the perceived multisensory conditions and the lack of such differences for the nonperceived conditions. The black horizontal bar indicates a significant difference (p < .01) between the GFP of the summed unisensory evoked potential and the GFP of the multisensory condition when perceived. Shaded area represents ±1 SEM across participants.

Collectively, these GFP results highlight that audiovisual stimuli that are perceived result in late evoked potentials that are not present when stimuli are not perceived, mirroring what has been well established within the visual neurosciences (e.g., see Dehaene & Changeux, 2011, for a review) and what seems to be emerging within the auditory neuroscience (e.g., see Sadaghiani et al., 2009). Interestingly, the presence of this late component exhibits subadditivity when contrasting the sum of unisensory and multisensory conditions (e.g., see Cappe et al., 2012, for similar results), an observation that is not true when stimuli are not perceived—due to the lack of late evoked potentials.

Intertrial Variability

To extend analyses beyond response strength, we further employed measures that capture the variability (i.e., reproducibility) and complexity (next section) of EEG responses. Specifically, there are several measures that have been leveraged successfully for the characterization and differentiation of states of consciousness (e.g., coma vs. awake vs. anesthetized vs. dreaming; Schurger et al., 2015; Ecker et al., 2014; Casali et al., 2013). In the current work, we implement a relatively straightforward version of this strategy. To evaluate response variability across sensory conditions and perceptual states, we performed PCA on the EEG signal for each trial and participant on an electrode-by-electrode basis and identified the minimum number of principal components needed to capture 90% of the trial-to-trial variability (McIntosh, Kovacevic, & Itier, 2008). As illustrated in Figure 5, more dimensions were needed to account for intertrial response variability of perceived (vs. nonperceived) conditions. However, this difference was more prominent for unisensory conditions compared with multisensory conditions (Figure 5). More specifically, a 2 (Perceived vs. nonperceived) × 3 (Sensory modality; A, V, AV) repeated-measures ANOVA demonstrated a significant main effect of Sensory modality beginning 107 msec poststimulus onset and persisting throughout the entire epoch (p < .01), a main effect of Perceptual state beginning at 95 msec poststimulus onset and persisting throughout the rest of the epoch (p < .01), and a significant interaction between these variables beginning 99 msec poststimulus onset and persisting throughout the rest of the epoch (p < .01). The interaction is explained by a difference in the time at which the PCA bifurcated between perceptual states (if at all) for the different sensory conditions. For the unisensory condition, beginning at 91 msec after the auditory stimulus and at 239 msec after the visual stimulus, there was a significant increase in response variability trials in which the stimulus was perceived (p < .01, n = 19, paired-samples t test for both contrasts; Figure 5). In contrast, this increased variability for perceived trials was not apparent for the audiovisual stimuli (p > .09, n = 19, paired samples t test). Intertrial variability as quantified by the PCA was similar across perceptual states for the catch trials (all ts(12) < 1, p > .74).

Figure 5. 

Trial-by-trial EEG variability as a function of sensory modality and perceived state. For this analysis, PCA was performed on all channels and for every participant, and the number of dimensions needed to explain 90% of the trial-by-trial variance was calculated. One hundred twenty trials were selected for each condition, giving a theoretical maximum dimensionality of 120. The figure illustrates the number of additional dimensions needed (in percentage) to explain trial-by-trial variability in the perceived as opposed to the nonperceived state as a function of time for the three sensory conditions. Results suggest that audio (blue) and visual (red) trials exhibit a marked increase in dimensions needed to explain trial-to-trial variance during the time course of an epoch, a feature not seen in the audiovisual (green) condition. Shaded areas around curves represent SEM over all participants.

Figure 5. 

Trial-by-trial EEG variability as a function of sensory modality and perceived state. For this analysis, PCA was performed on all channels and for every participant, and the number of dimensions needed to explain 90% of the trial-by-trial variance was calculated. One hundred twenty trials were selected for each condition, giving a theoretical maximum dimensionality of 120. The figure illustrates the number of additional dimensions needed (in percentage) to explain trial-by-trial variability in the perceived as opposed to the nonperceived state as a function of time for the three sensory conditions. Results suggest that audio (blue) and visual (red) trials exhibit a marked increase in dimensions needed to explain trial-to-trial variance during the time course of an epoch, a feature not seen in the audiovisual (green) condition. Shaded areas around curves represent SEM over all participants.

EEG Complexity

The final theory-driven measure of interest here is a measure of capacity for information reduction—LZ complexity. This measure is of interest because of recent observations indicating that perceptual awareness may not emanate simply for the recruitment of broadly distributed networks but rather for the differentiation and integrations of activity among these networks (see Cavanna et al., 2017, for a recent review). These networks are postulated to fulfill axiomatic observations related to awareness (Tononi & Koch, 2015) that embody complex neural signatures of that mental state. Thus, here, LZ complexity—a measure of information reducibility—was measured across the poststimuli period of audio, visual, and audiovisual stimuli that were either perceived or not, and we queried whether similar patterns of complexity would apply across modalities (i.e., from visual to auditory) and number of modalities (i.e., from unisensory to multisensory). As illustrated in Figure 6, a 4 (Sensory modality; none, audio, visual, audiovisual) × 2 (Perceived vs. nonperceived) repeated-measures ANOVA revealed a significant main effect of Sensory modality (F(3, 57) = 44.92, p < .001), a significant main effect of Perceptual state (F(1, 18) = 40.82, p < .001), and a significant interaction between these variables (F(3, 57) = 3.21, p = .029). The main effect of Perceptual state was due to higher complexity for nonperceived stimuli (M = 0.24, SEM = 0.01) than for perceived stimuli (M = 0.19, SEM = 0.005; paired t test, t(18) = 6.32, p < .001). Regarding the main effect of Sensory modality, post hoc paired t tests (Bonferroni corrected) revealed that catch trials exhibited the most informationally complex patterns of activity, on average (M = 0.27, SEM = 0.007, all ps < .001), followed by auditory evoked potentials (M = 0.21, SEM = 0.010; contrasts to catch and audiovisual conditions significant with all ps < .03, but not the contrast to visual trials, p = .659), visual evoked potentials (M = 0.19, SEM = 0.010; contrast to audiovisual trials being nonsignificant, p = .253), and, finally, multisensory evoked potentials (M = 0.18, SEM = 0.008). The complexity of these multisensory responses was not significantly different from those of visual responses. The significant interaction was driven by the fact that there was a significant difference in evoked complexity between perceptual states (perceived vs. nonperceived) for catch trials (perceived: M = 0.24, SEM = 0.03; nonperceived: M = 0.30, SEM = 0.06; t(19) = 3.40, p = .003), auditory trials (perceived: M = 0.19, SEM = 0.05; nonperceived: M = 0.24, SEM = 0.04; t(19) = 6.63, p < .001), and visual trials (perceived: M = 0.17, SEM = 0.04; nonperceived: M = 0.22, SEM = 0.05; t(19) = 4.45, p < .001) stimulation. In contrast, this difference was not seen for audiovisual trials (perceived: M = 0.17, SEM = 0.03; nonperceived: M = 0.19, SEM = 0.04; t(19) = 1.32, p = .203). In fact, for the multisensory condition, Bayesian statistics suggested that, not solely, there is no evidence against the null hypothesis (as inferred via frequentist analyses described above), but in fact, there was considerable evidence for it (BF10 = 0.298 < 0.03, typically suggested as a cutoff favoring the null hypothesis; Jeffreys, 1961). Taken together, these analyses suggest that, although EEG complexity is generally decreased when stimuli are perceived (vs. nonperceived and normalizing for overall entropy) for unisensory stimuli, this is not true for multisensory stimuli. Interestingly, the decrease in complexity is also observed during catch trials when participants report perceiving a stimulus that is not present. Thus, the decrease in EEG evoked complexity is associated not only with physical stimulation but seemingly also with perceptual state.

Figure 6. 

Neural complexity differs as a function of perceived state and sensory modality. LZ complexity as a function of experimental condition: catch (leftmost), auditory (second panel), visual (third panel), or audiovisual (rightmost). Results suggest a significant difference between detected and nondetected stimuli for catch (black), auditory (blue), and visual (red) conditions, but not for audiovisual (green) trials. y Axis is normalized LZ during the entire poststimuli epoch (LZ for unshuffled data divided by shuffled data). Error bars indicate +1 SEM across participants.

Figure 6. 

Neural complexity differs as a function of perceived state and sensory modality. LZ complexity as a function of experimental condition: catch (leftmost), auditory (second panel), visual (third panel), or audiovisual (rightmost). Results suggest a significant difference between detected and nondetected stimuli for catch (black), auditory (blue), and visual (red) conditions, but not for audiovisual (green) trials. y Axis is normalized LZ during the entire poststimuli epoch (LZ for unshuffled data divided by shuffled data). Error bars indicate +1 SEM across participants.

DISCUSSION

A number of different neural markers of perceptual awareness have been proposed—from “neural ignition” and the presence of late evoked potentials (P3, P300, P3b; Dehaene et al., 2017; Dehaene & Changeux, 2011), to increased neural reproducibility (Schurger et al., 2010), to a high degree of information integration that can be indexed through measures such as EEG complexity (Koch et al., 2016a, 2016b; Tononi et al., 2016; Casali et al., 2013). Here, we sought to extend the use of these various measures posited to represent credible neural signatures of perceptual awareness for visual stimuli to multisensory perceptual processes—as much of our perceptual gestalt is constructed on a multisensory foundation. Collectively, our results support and extend prior work implicating neural signatures of perceptual awareness revealed in measures of EEG response strength, reproducibility, and complexity. We show, as has earlier work, that reproducibility and complexity indices of perceptual awareness are similar for visual and auditory conditions, but we also show that there exist significant differences in the indices of awareness associated with multisensory stimulation, differences that likely have important implications for furthering our understanding of multisensory perceptual awareness.

Neural Response Strength as a Modality-free Indicator of Perceptual Awareness

More specifically, conditions in which visual, auditory, or both visual and auditory stimuli were presented resulted in reliable variations in EEG response strength (as indexed via GFP) that covaried with perceptual state (i.e., was the stimulus perceived or not). In each of these conditions, comparison of perceived versus nonperceived stimuli revealed the presence of late evoked potentials that were only present under perceived circumstances. Thus, the presence of late evoked potentials appears to be a strong index of perceptual awareness under both unisensory and multisensory conditions. The striking absence of late EEG components to nonperceived stimuli resembles “ignition-like” single-unit responses to perceived stimuli that have been found in the temporal lobe of epileptic patients (Dehaene, 2014). This response pattern fits the assumption that conscious percepts arise late in the evolution of sensory responses, possibly because they necessitate more global brain activity (Noy et al., 2015; Dehaene & Changeux, 2011; Gaillard et al., 2009). This “ignition-like” effect, which at times has been difficult to capture in previous work (e.g., Silverstein, Snodgrass, Shevrin, & Kushwaha, 2015), likely results from several aspects of the current experiment. First, it may be argued that the lack of observable late responses in EEG signals may be due to our adaptive, online method of adjusting stimulus intensity—and not reflective of the manner in which individuals become aware of stimuli. This account, however, does not fully explain the GFP effects, as EEG analyses were restricted to the last 400–500 trials and in which auditory and visual noise levels were relatively fixed in intensity and the minimal changes in stimuli intensity did not provoke a change in GFP (see Control Analyses; Figure S1 online). Second, the current experiment is different from most previous EEG studies presenting stimuli at threshold (and demonstrating the occurrence of late EEG components, e.g., see Koch, 2004) in that, here, we interleave stimuli from different modalities (see Sanchez et al., 2017, for a similar observation of abolished late evoked responses for undetected stimuli in a multisensory context). Finally, it is possible that the clear presence of late evoked potentials in perceived trials but not in nonperceived trials arises because participants were working below the 50% detection rate and not at threshold (most prior work presented stimuli at threshold).

EEG Subadditivity in Multisensory Integration Is Associated with Perceived Stimuli

A second interesting observation regarding the GFP results relates to the comparison between the sum of unisensory evoked potentials (“sum”) and the multisensory response (“pair”). When stimuli were not perceived, there was no significant difference between the multisensory GFP and the GFP predicted by the sum of unisensory responses (i.e., no difference between sum and pair). In contrast, when the stimuli were perceived, the GFP of the audiovisual condition was distinctly subadditive when compared with the sum of the unisensory responses. Hence, although neural response strength (i.e., GFP) differentiates between perceptual states under both unisensory and multisensory conditions, the perceived multisensory response does not reflect a simple addition of the two unisensory responses. Indeed, subadditivity in EEG responses is often seen as a hallmark of multisensory processing (see Cappe et al., 2010, 2012, for examples), and here, it was evident only under perceived multisensory conditions, suggesting links between multisensory integration and perceptual awareness (see Baars, 2002, for a philosophical consideration arguing that conscious processing is involved in the merging of sensory modalities). Although a number of studies suggest that multisensory interactions may occur when information from a single sense is below the threshold for perceptual awareness (Salomon, Kaliuzhna, Herbelin, & Blanke, 2016; Aller, Giani, Conrad, Watanabe, & Noppeney, 2015; Lunghi, Morrone, & Alais, 2014; Lunghi & Alais, 2013), when both are presented at subthreshold levels after a period of associative learning (Faivre et al., 2014), or even when participants are unconscious (Arzi et al., 2012; Ikeda & Morotomi, 1996; Beh & Barratt, 1965), evidence for multisensory integration in the complete absence of perceptual awareness (without prior training) is conspicuously lacking (Faivre et al., 2017; Noel et al., 2015). The current results provide additional support for the absence of multisensory integration outside perceptual awareness but, as null results, must be interpreted with caution.

Across-Trial EEG Reproducibility Differentiates between Perceived and Nonperceived Unisensory but Not Multisensory Stimuli

The next putative index of perceptual awareness used in the current study was that of neural reproducibility (Schurger et al., 2015). This measure is predicated on the view that spatio-temporal neural patterns giving rise to subjective experience manifest as relatively stable epochs of neural activity (Britz et al., 2014; Fingelkurts, Fingelkurts, Bagnato, Boccagni, & Galardi, 2013). To address the stability of responses, we measured intertrial variability via a relatively straightforward metric, that is, PCA. Those results disclosed similar levels of neural reproducibility for visual and auditory conditions (although with different time courses) and a categorically distinct pattern for multisensory presentations. Specifically, there was no difference in neural reproducibility across trials for perceived versus nonperceived trials for the multisensory conditions, but there were reliable differences associated with the unisensory conditions. The increased variability for perceived unisensory stimuli runs counter to the view that responses to perceived trials are more reproducible (Schurger et al., 2010; Xue et al., 2010). However, we did not observe late response components to nonperceived stimuli, which reduces the amount of principal components that are needed to explain the variance of this part of the response. Indeed, the increase in principal components that are needed to explain the trial-to-trial variability for the perceived stimuli occurs very close in time to the bifurcation between perceived and nonperceived GFPs (auditory: GFP at 121 msec vs. PCA dimensionality increase at 91 msec; visual: GFP at 219 msec vs. PCA dimensionality increase at 239 msec). Thus, the relevant observation here is that both the strength (as indexed via GFP analyses) and the between-trial variability (as indexed via PCAs) seen in response to perceived multisensory stimuli are reduced in comparison with the unisensory conditions, with both of these effects appearing around the same time in the neurophysiological responses. On the other hand, in contrast to the observation that late evoked potentials seemingly index perceptual awareness regardless of sensory modality, the increase in reproducibility associated with perceived stimuli (Schurger et al., 2015) is most readily evident for multisensory stimuli. That is, although the observation derived from visual neurosciences indicating increased reproducibility for perceived stimuli (Schurger et al., 2015) may be applied to auditory neurosciences— the same pattern of results between auditory and visual modalities, although at different latencies—the PCA seems categorically different when probing perceived and nonperceived multisensory stimuli. These results highlight that, at least in the case of neural reproducibility, conclusions drawn from unisensory studies may not generalize to multisensory studies for work attempting to better understand the neural correlates of perceptual awareness.

The finding that signals of neural variability under multisensory conditions changed little as a function of perceptual state is consistent with computational models based on Bayesian inference (e.g., Körding et al., 2007) and maximum likelihood estimates. These models have been applied to psychophysical (Ernst & Banks, 2002), neuroimaging (Rohe & Noppeney, 2015, 2016), and electrophysiological (Boyle, Kayser, & Kayser, 2017; Fetsch, Deangelis, & Angelaki, 2013) observations concerning suprathreshold multisensory performance and collectively illustrate that the combination of sensory information across different modalities tends to decrease variability (i.e., increases signal reliability). Although the current study was not designed or analyzed to specifically pinpoint neural concomitants of multisensory integration, our findings may inform the models mentioned above by showing that, at least for the task employed in the current study, variance in the evoked neural response is more comparable across perceptual states for multisensory conditions compared with unisensory conditions. Interestingly, stimulus-induced reduction in neural variability has been observed across a wide array of brain areas and has been posited to be a general property of cortex in response to stimulus onset (Churchland et al., 2010). In subsequent work, it will be informative to examine whether, at the level of single neurons, variability (as measured through indices such as Fano factor; Eden & Kramer, 2010) decreases equally across perceptual states (while maintaining stimulus intensity near detection threshold) and whether these changes differ for unisensory brain responses compared with multisensory responses.

EEG Complexity Differentiates between Perceived and Nonperceived Unisensory but Not Multisensory Stimuli

Finally, consider that aspect of our results dealing with measured neural complexity associated with evoked responses due to visual, auditory, or audiovisual stimuli and as a function of perceptual state. In previous work, a derivative of this measure has successfully categorized patients along the continuum ranging from awake to asleep to minimally conscious and, finally, to comatose (see Casali et al., 2013). This work has shown that, when neural responses are evoked via TMS, they are less amenable to information compression when patients are conscious relative to when they are unconscious. To our knowledge, however, the present report is the first to examine EEG data complexity (compressibility) as a function of perceptual state and not as a function of level of consciousness. Our results indicate that evoked responses are less complex when either visual or auditory stimuli are perceived (compared with nonperceived). Interestingly, this difference was not evident under multisensory conditions. Furthermore, this measure was able to differentiate between the catch trials that were correctly “rejected” (i.e., no stimulus reported when no stimulus was presented) and false alarms (i.e., reports of the presence of a stimulus when none was presented—a possible analog of a hallucination). The switch in effect direction between levels of consciousness (i.e., more complex when patients are conscious) and perceptual state (i.e., more complex when stimuli are not perceived) likely is due to the fact that, in the former case, neural responses are artificially evoked—thus recruiting neural networks in a nonnatural manner—whereas in the current case, neural responses are evoked by true stimulus presentations. As an example, in the case of visual stimulus presentations, the present results indicate that neural information in the visual neural network architecture is more stereotyped for perceived versus nonperceived trials.

Conclusions

Taken together, the overall pattern of results (1) questions whether multisensory integration is possible before perceptual awareness (see Spence & Bayne, 2014, and O'Callaghan, 2017, for distinct perspectives on whether perceptual awareness may be uniquely multisensory or simply a succession of unisensory processes) and (2) questions the implicit assumption that all indices of perceptual awareness apply across all sensory modalities and conditions. Indeed, if assumed that the search for the neural correlates of perceptual awareness must result in a set of features that are common across all sensory domains (e.g., visual awareness, auditory awareness, audiovisual awareness), then the current findings would argue that the presence of late evoked potentials, as opposed to neural reproducibility or complexity, most closely tracks perceptual awareness. On the other hand, if one instead assumes that visual awareness, auditory awareness, and audiovisual awareness are categorically distinct (or nonexistent in the case of multisensory awareness; Spence & Bayne, 2014), then the current findings suggest a greater similarity between the neural correlates of perceptual awareness across visual and auditory modalities and not between unisensory and multisensory perceptual processes.

Acknowledgments

The authors thank Dr. Nathan Faivre for insightful comments on an early draft. The work was supported by an NSF Graduate Research Fellowship to J. P. N., by NIH grant HD083211 to M. T. W., and by Centennial Research Funds from Vanderbilt University to R. B.

Reprint requests should be sent to Jean-Paul Noel, 7110 MRB III BioSci Bldg., 465, 21st Ave. South, Nashville, TN 3721, or via e-mail: jean-paul.noel@vanderbilt.edu.

REFERENCES

Aller
,
M.
,
Giani
,
A.
,
Conrad
,
V.
,
Watanabe
,
M.
, &
Noppeney
,
U.
(
2015
).
A spatially collocated sound thrusts a flash into awareness
.
Frontiers in Integrative Neuroscience
,
9
,
16
.
Andrillon
,
T.
,
Poulsen
,
A. T.
,
Hansen
,
L. K.
,
Léger
,
D.
, &
Kouider
,
S.
(
2016
).
Neural markers of responsiveness to the environment in human sleep
.
Journal of Neuroscience
,
36
,
6583
6596
.
Arzi
,
A.
,
Shedlesky
,
L.
,
Ben-Shaul
,
M.
,
Nasser
,
K.
,
Oksenberg
,
A.
,
Hairston
,
I. S.
, et al
(
2012
).
Humans can learn new information during sleep
.
Nature Neuroscience
,
15
,
1460
1465
.
Baars
,
B. J.
(
2002
).
The conscious access hypothesis: Origins and recent evidence
.
Trends in Cognitive Sciences
,
6
,
47
52
.
Beh
,
H. C.
, &
Barratt
,
P. E. H.
(
1965
).
Discrimination and conditioning during sleep as indicated by the electroencephalogram
.
Science
,
147
,
1470
1471
.
Blanke
,
O.
,
Slater
,
M.
, &
Serino
,
A.
(
2015
).
Behavioral, neural, and computational principles of bodily self-consciousness
.
Neuron
,
88
,
145
166
.
Boyle
,
S. C.
,
Kayser
,
S. J.
, &
Kayser
,
C.
(
2017
).
Neural correlates of multisensory reliability and perceptual weights emerge at early latencies during audio-visual integration
.
European Journal of Neuroscience
,
46
,
2565
2577
.
Brainard
,
D. H.
(
1997
).
The psychophysics toolbox
.
Spatial Vision
,
10
,
433
436
.
Britz
,
J.
,
Díaz Hernàndez
,
L.
,
Ro
,
T.
, &
Michel
,
C. M.
(
2014
).
EEG-microstate dependent emergence of perceptual awareness
.
Frontiers in Behavioral Neuroscience
,
8
,
163
.
Broadbent
,
D.
(
1958
).
Perception and communication
.
London
:
Pergamon Press
.
Brunet
,
D.
,
Murray
,
M. M.
, &
Michel
,
C. M.
(
2011
).
Spatiotemporal analysis of multichannel EEG: CARTOOL
.
Computational Intelligence and Neuroscience
,
2011
,
15
.
Cappe
,
C.
,
Thelen
,
T.
,
Romei
,
V.
,
Thut
,
G.
, &
Murray
,
M. M.
(
2012
).
Looming signals reveal synergistic principles of multisensory integration
.
Journal of Neuroscience
,
32
,
1171
1182
.
Cappe
,
C.
,
Thut
,
G.
,
Romei
,
V.
, &
Murray
,
M. M.
(
2010
).
Auditory–visual multisensory interactions in humans: Timing, topography, directionality, and sources
.
Journal of Neuroscience
,
30
,
12572
12580
.
Casali
,
A. G.
,
Gosseries
,
O.
,
Rosanova
,
M.
,
Boly
,
M.
,
Sarasso
,
S.
,
Casali
,
K. R.
, et al
(
2013
).
A theoretically based index of consciousness independent of sensory processing and behavior
.
Science Translational Medicine
,
5
,
198ra105
.
Cavanna
,
F.
,
Vilas
,
M. G.
,
Palmucci
,
M.
, &
Tagliazucchi
,
E.
(
2017
).
Dynamic functional connectivity and brain metastability during altered states of consciousness
. https://doi.org/10.1016/j.neuroimage.2017.09.065.
Churchland
,
M. M.
,
Yu
,
B. M.
,
Cunningham
,
J. P.
,
Sugrue
,
L. P.
,
Cohen
,
M. R.
,
Corrado
,
G. S.
, et al
(
2010
).
Stimulus onset quenches neural variability: A widespread cortical phenomenon
.
Nature Neuroscience
,
13
,
369
378
.
Dehaene
,
S.
(
2014
).
Consciousness and the brain: Deciphering how the brain codes our thoughts
.
New York
:
Penguin
.
Dehaene
,
S.
, &
Changeux
,
J.-P.
(
2011
).
Experimental and theoretical approaches to conscious processing
.
Neuron
,
70
,
200
227
.
Dehaene
,
S.
,
Changeux
,
J.-P.
,
Naccache
,
L.
,
Sackur
,
J.
, &
Sergent
,
C.
(
2006
).
Conscious, preconscious, and subliminal processing: A testable taxonomy
.
Trends in Cognitive Sciences
,
10
,
204
211
.
Dehaene
,
S.
,
Lau
,
H.
, &
Kouider
,
S.
(
2017
).
What is consciousness, and could machines have it?
Science
,
358
,
486
492
.
Dehaene
,
S.
,
Naccache
,
L.
,
Cohen
,
L.
,
Bihan
,
D. L.
,
Mangin
,
J. F.
,
Poline
,
J. B.
, et al
(
2001
).
Cerebral mechanisms of word masking and unconscious repetition priming
.
Nature Neuroscience
,
4
,
752
758
.
Del Cul
,
A.
,
Baillet
,
S.
, &
Dehaene
,
S.
(
2007
).
Brain dynamics underlying the nonlinear threshold for access to consciousness
.
PLoS Biology
,
5
,
e260
.
Delorme
,
A.
, &
Makeig
,
S.
(
2004
).
EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis
.
Journal of Neuroscience Methods
,
134
,
9
21
.
Deroy
,
O.
,
Chen
,
Y. C.
, &
Spence
,
C.
(
2014
).
Multisensory constraints on awareness
.
Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences
,
369
,
20130207
.
Ecker
,
A. S.
,
Berens
,
P.
,
Cotton
,
R. J.
,
Subramaniyan
,
M.
,
Denfield
,
G. H.
,
Cadwell
,
C. R.
, et al
(
2014
).
State dependence of noise correlations in macaque primary visual cortex
.
Neuron
,
82
,
235
248
.
Eden
,
U. T.
, &
Kramer
,
M. A.
(
2010
).
Drawing inferences from Fano factor calculations
.
Journal of Neuroscience Methods
,
190
,
149
152
.
Eriksen
,
C. W.
(
1960
).
Discrimination and learning without awareness: A methodological survey and evaluation
.
Psychological Review
,
67
,
279
300
.
Ernst
,
M. O.
, &
Banks
,
M. S.
(
2002
).
Humans integrate visual and haptic information in a statistically optimal fashion
.
Nature
,
415
,
429
433
.
Faivre
,
N.
,
Arzi
,
A.
,
Lunghi
,
C.
, &
Salomon
,
R.
(
2017
).
Consciousness is more than meets the eye: A call for a multisensory study of subjective experience
.
Neuroscience of Consciousness
,
3
,
nix003
.
Faivre
,
N.
,
Mudrik
,
L.
,
Schwartz
,
N.
, &
Koch
,
C.
(
2014
).
Multisensory integration in complete unawareness: Evidence from audiovisual congruency priming
.
Psychological Science
,
25
,
2006
2016
.
Fetsch
,
C. R.
,
Deangelis
,
G. C.
, &
Angelaki
,
D. E.
(
2013
).
Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons
.
Nature Reviews Neuroscience
,
14
,
429
442
.
Fingelkurts
,
A. A.
,
Fingelkurts
,
A. A.
,
Bagnato
,
S.
,
Boccagni
,
C.
, &
Galardi
,
G.
(
2013
).
Dissociation of vegetative and minimally conscious patients based on brain operational architectonics: Factor of etiology
.
Clinical EEG and Neuroscience
,
44
,
209
220
.
Gaillard
,
R.
,
Dehaene
,
S.
,
Adam
,
C.
,
Clemenceau
,
S.
,
Hasboun
,
D.
,
Baulac
,
M.
, et al
(
2009
).
Converging intracranial markers of conscious access
.
PLoS Biology
,
7
,
e61
.
Genovese
,
C. R.
,
Lazar
,
N. A.
, &
Nichols
,
T.
(
2002
).
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
.
Neuroimage
,
4
,
870
878
.
Guthrie
,
D.
, &
Buchwald
,
J. S.
(
1991
).
Significance testing of difference potentials
.
Psychophysiology
,
28
,
240
244
.
Ikeda
,
K.
, &
Morotomi
,
T.
(
1996
).
Classical conditioning during human NREM sleep and response transfer to wakefulness
.
Sleep
,
19
,
72
74
.
James
,
W.
(
1890
).
The principles of psychology, in two volumes
.
New York
:
Henry Holt and Company
.
Jeffreys
,
H.
(
1961
).
Theory of probability
(3rd ed.).
Oxford classic texts in the physical science
.
Oxford
:
Oxford University Press
.
Joos
,
K.
,
Gilles
,
A.
,
Van de Heyning
,
P.
,
De Ridder
,
D.
, &
Vanneste
,
S.
(
2014
).
From sensation to percept: The neural signature of auditory event-related potentials
.
Neuroscience & Biobehavioral Reviews
,
42
,
148
156
.
Koch
,
C.
(
2004
).
The quest for consciousness: A neurobiological approach
.
Englewood, CO
:
Roberts
.
Koch
,
C.
,
Massimini
,
M.
,
Boly
,
M.
, &
Tononi
,
G.
(
2016a
).
Neural correlates of consciousness: Progress and problems
.
Nature Reviews Neuroscience
,
17
,
307
321
.
Koch
,
C.
,
Massimini
,
M.
,
Boly
,
M.
, &
Tononi
,
G.
(
2016b
).
Posterior and anterior cortex—Where is the difference that makes the difference?
Nature Reviews Neuroscience
,
17
.
Koenig
,
T.
, &
Melie-Garcia
,
L.
(
2010
).
A method to determine the presence of averaged event-related fields using randomization tests
.
Brain Topography
,
23
,
233
242
.
Körding
,
K. P.
,
Beierholm
,
U.
,
Ma
,
W. J.
,
Tenenbaum
,
J. B.
,
Quartz
,
S.
, &
Shams
,
L.
(
2007
).
Causal inference in multisensory perception
.
PLoS One
,
2
,
e943
.
Lamme
,
V. A.
(
2006
).
Towards a true neural stance on consciousness
.
Trends in Cognitive Sciences
,
10
,
494
501
.
Lehmann
,
D.
, &
Skrandies
,
W.
(
1980
).
Spatial analysis of evoked potentials in man—A review
.
Progress in Neurobiology
,
23
,
227
250
.
Luck
,
S. J.
(
2005
).
An introduction to the event-related potential technique
.
Cambridge, MA
:
MIT Press
.
Lunghi
,
C.
, &
Alais
,
D.
(
2013
).
Touch interacts with vision during binocular rivalry with a tight orientation tuning
.
PLoS One
,
8
,
e58754
.
Lunghi
,
C.
,
Morrone
,
M. C.
, &
Alais
,
D.
(
2014
).
Auditory and tactile signals combine to influence vision during binocular rivalry
.
Journal of Neuroscience
,
34
,
784
792
.
Macmillan
,
N. A.
, &
Creelman
,
C. D.
(
2005
).
Detection theory: A user's guide
(2nd ed.).
Hillsdale, NJ
:
Lawrence Erlbaum Associates
.
McIntosh
,
A. R.
,
Kovacevic
,
N.
, &
Itier
,
R. J.
(
2008
).
Increased brain signal variability accompanies lower behavioral variability in development
.
PLoS Computational Biology
,
4
,
e1000106
.
Merkile
,
P. M.
,
Smilek
,
D.
, &
Eastwood
,
J. D.
(
2001
).
Perception without awareness: Perspectives from cognitive psychology
.
Cognition
,
79
,
115
134
.
Mudrik
,
L.
,
Faivre
,
N.
, &
Koch
,
C.
(
2014
).
Information integration without awareness
.
Trends in Cognitive Sciences
,
18
,
488
496
.
Murray
,
M. M.
, &
Wallace
,
M. T.
(
2012
).
The neural bases of multisensory processes
.
Boca Raton, FL
:
CRC Press
.
Naghavi
,
H. R.
, &
Nyberg
,
L.
(
2005
).
Common fronto-parietal activity in attention, memory, and consciousness: Shared demands on integration?
Consciousness and Cognition
,
14
,
390
425
.
Noel
,
J. P.
,
Wallace
,
M. T.
, &
Blake
,
R.
(
2015
).
Cognitive neuroscience: Integration of sight and sound outside of awareness?
Current Biology
,
25
,
R157
R159
.
Noy
,
N.
,
Bickel
,
S.
,
Zion-Golumbic
,
E.
,
Harel
,
M.
,
Golan
,
T.
,
Davidesco
,
I.
, et al
(
2015
).
Ignition's glow: Ultra-fast spread of global cortical activity accompanying local “ignitions” in visual cortex during conscious visual perception
.
Consciousness and Cognition
,
35
,
206
224
.
O'Callaghan
,
C.
(
2017
).
Grades of multisensory awareness
.
Mind & Language
,
32
,
155
181
.
Pelli
,
D. G.
(
1997
).
The VideoToolbox software for visual psychophysics: Transforming numbers into movies
.
Spatial Vision
,
10
,
437
442
.
Perrin
,
F.
,
Pernier
,
J.
,
Bertrand
,
O.
,
Giard
,
M. H.
, &
Echalier
,
J. F.
(
1987
).
Mapping of scalp potentials by surface spline interpolation
.
Electroencephalography and Clinical Neurophysiology
,
66
,
75
81
.
Rohe
,
T.
, &
Noppeney
,
U.
(
2015
).
Cortical hierarchies perform Bayesian causal inference in multisensory perception
.
PLoS Biology
,
13
,
e1002073
.
Rohe
,
T.
, &
Noppeney
,
U.
(
2016
).
Distinct computational principles govern multisensory integration in primary sensory and association cortices
.
Current Biology
,
26
,
509
514
.
Sadaghiani
,
S.
,
Hesselmann
,
G.
, &
Kleinschmidt
,
A.
(
2009
).
Distributed and antagonistic contributions of ongoing activity fluctuations to auditory stimulus detection
.
Journal of Neuroscience
,
29
,
13410
13417
.
Salomon
,
R.
,
Kaliuzhna
,
M.
,
Herbelin
,
B.
, &
Blanke
,
O.
(
2016
).
Balancing awareness: Vestibular signals modulate visual consciousness in the absence of awareness
.
Consciousness and Cognition
,
36
,
289
297
.
Salomon
,
R.
,
Noel
,
J. P.
,
Lukowska
,
M.
,
Faivre
,
N.
,
Metzinger
,
T.
,
Serino
,
A.
, et al
(
2017
).
Unconscious integration of multisensory bodily inputs in the peripersonal space shapes bodily self-consciousness
.
Cognition
,
166
,
174
183
.
Sanchez
,
G.
,
Frey
,
J. N.
,
Fuscà
,
M.
, &
Weisz
,
N.
(
2017
).
Decoding across sensory modalities reveals common supramodal signatures of conscious perception
.
bioRxiv
. https://doi.org/10.1101/115535.
Sanchez-Vives
,
M. V.
,
Massimini
,
M.
, &
Mattia
,
M.
(
2017
).
Shaping the default activity pattern of the cortical network
.
Neuron
,
94
,
993
1001
.
Sarasso
,
S.
,
Boly
,
M.
,
Napolitani
,
M.
,
Gosseries
,
O.
,
Charland-Verville
,
V.
,
Casarotto
,
S.
, et al
(
2015
).
Consciousness and complexity during unresponsiveness induced by propofol, xenon, and ketamine
.
Current Biology
,
25
,
3099
3105
.
Schartner
,
M. M.
,
Carhart-Harris
,
R. L.
,
Barrett
,
A. B.
,
Seth
,
A. K.
, &
Muthukumaraswamy
,
S. D.
(
2017
).
Increased spontaneous MEG signal diversity for psychoactive doses of ketamine, LSD and psilocybin
.
Scientific Reports
,
7
,
46421
.
Schartner
,
M. M.
,
Seth
,
A.
,
Noirhomme
,
Q.
,
Boly
,
M.
,
Bruno
,
M. A.
,
Laureys
,
S.
, et al
(
2015
).
Complexity of multi-dimensional spontaneous EEG decreases during propofol induced general anaesthesia
.
PLoS One
,
10
,
e0133532
.
Schurger
,
A.
,
Pereira
,
F.
,
Treisman
,
A.
, &
Cohen
,
J. D.
(
2010
).
Reproducibility distinguishes conscious from non-conscious neural representations
.
Science
,
327
,
97
99
.
Schurger
,
A.
,
Sarigiannidis
,
I.
, &
Dehaene
,
S.
(
2015
).
Cortical activity is more stable when sensory stimuli are consciously perceived
.
Proceedings of the National Academy of Sciences, U.S.A.
,
112
,
E2083
E2092
.
Sergent
,
C.
,
Baillet
,
S.
, &
Dehaene
,
S.
(
2005
).
Timing of the brain events underlying access to consciousness during the attentional blink
.
Nature Neuroscience
,
8
,
1391
1400
.
Silverstein
,
B. H.
,
Snodgrass
,
M.
,
Shevrin
,
H.
, &
Kushwaha
,
R.
(
2015
).
P3b, consciousness, and complex unconscious processing
.
Cortex
,
73
,
216
227
.
Spence
,
C.
, &
Bayne
,
T.
(
2014
).
Is consciousness multisensory?
In
D.
Stokes
,
S.
Biggs
, &
M.
Matthen
(Eds.),
Perception and its modalities
(pp.
95
132
).
New York
:
Oxford University Press
.
Spence
,
C.
, &
Deroy
,
O.
(
2013
).
Crossmodal mental imagery
. In
S.
Lacey
&
R.
Lawson
(Eds.),
Multisensory imagery
(pp.
157
183
).
New York
:
Springer
.
Sperdin
,
H. F.
,
Spierer
,
L.
,
Becker
,
R.
,
Michel
,
C. M.
, &
Landis
,
T.
(
2015
).
Submillisecond unmasked subliminal visual stimuli evoke electrical brain responses
.
Human Brain Mapping
,
36
,
1470
1483
.
Tallon-Baudry
,
C.
(
2012
).
On the neural mechanisms subserving consciousness and attention
.
Frontiers in Psychology
,
2
,
397
.
Tanner
,
W. P.
, Jr.
, &
Swets
,
J. A.
(
1954
).
A decision-making theory of visual detection
.
Psychological Review
,
61
,
401
409
.
Tononi
,
G.
(
2012
).
Integrated information theory of consciousness: An updated account
.
Archives Italiennes de Biologie
,
150
,
56
90
.
Tononi
,
G.
,
Boly
,
M.
,
Massimini
,
M.
, &
Koch
,
C.
(
2016
).
Integrated information theory: From consciousness to its physical substrate
.
Nature Reviews Neuroscience
,
17
,
450
461
.
Tononi
,
G.
, &
Koch
,
C.
(
2015
).
Consciousness: Here, there and everywhere?
Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences
,
370
,
20140167
.
Treisman
,
A.
, &
Gelade
,
G.
(
1980
).
A feature integration theory of attention
.
Cognitive Psychology
,
12
,
97
136
.
van Gaal
,
S.
, &
Lamme
,
V. A. F.
(
2012
).
Unconscious high-level information processing: Implication for neurobiological theories of consciousness
.
Neuroscientist
,
18
,
287
301
.
Xue
,
G.
,
Dong
,
Q.
,
Chen
,
C.
,
Lu
,
Z.
,
Mumford
,
J. A.
, &
Poldrack
,
R. A.
(
2010
).
Greater neural pattern similarity across repetitions is associated with better memory
.
Science
,
330
,
97
101
.
Zeki
,
S.
(
2003
).
The disunity of consciousness
.
Trends in Cognitive Sciences
,
7
,
214
218
.