Change blindness—the failure to detect changes in visual scenes—has often been interpreted as a result of impoverished visual information encoding or as a failure to compare the prechange and postchange scene. In the present electroencephalography study, we investigated whether semantic features of prechange and postchange information are processed unconsciously, even when observers are unaware that a change has occurred. We presented scenes composed of natural objects in which one object changed from one presentation to the next. Object changes were either semantically related (e.g., rail car changed to rail) or unrelated (e.g., rail car changed to sausage). Observers were first asked to detect whether any change had occurred and then to judge the semantic relation of the two objects involved in the change. We found a semantic mismatch ERP effect, that is, a more negative-going ERP for semantically unrelated compared to related changes, originating from a cortical network including the left middle temporal gyrus and occipital cortex and resembling the N400 effect, albeit at longer latencies. Importantly, this semantic mismatch effect persisted even when observers were unaware of the change and the semantic relationship of prechange and postchange object. This finding implies that change blindness does not preclude the encoding of the prechange and postchange objects' identities and possibly even the comparison of their semantic content. Thus, change blindness cannot be interpreted as resulting from impoverished or volatile visual representations or as a failure to process the prechange and postchange object. Instead, change detection appears to be limited at a later, postperceptual stage.
We often do not notice limitations in our ability to perceive, process, and maintain information about our visual environment, but such limitations are evident when objective performance is assessed in experimental tasks. In the so-called change blindness paradigm, observers frequently fail to perceive when an object in a scene changes from one moment to the next if the change occurs simultaneously with a brief visual disruption (such as a saccade, a flicker, or a distracting stimulus; Ball, Elzemann, & Busch, 2014; Henderson & Hollingworth, 1999; O'Regan, Rensink, & Clark, 1999; Levin & Simons, 1997; Rensink, O'Regan, & Clark, 1997; Grimes, 1996). The disruption serves to mask the motion and contrast transients, so that the change cannot be perceived directly. Instead, change detection in these situations depends on preserving and comparing object representations of the prechange and postchange objects. These results obviously point to a limit in our ability to perceive visual scenes, and numerous studies have sought to characterize the processing stages at which this limitation occurs during change blindness (see Jensen, Yao, Street, & Simons, 2011; Rensink, 2002, for reviews).
Change detection can be considered a four-stage process (Rensink, 2002). First, information of the prechange display has to be encoded. This representation must then be maintained in short-term and/or long-term memory. Second, the postchange display must be encoded and maintained. Third, prechange and postchange representations must be compared. The change can either be detected immediately at change onset if the comparison is based on the representations in short-term memory. This kind of detection would be more akin to seeing the change happening. Alternatively, if the change is not detected immediately, detection can occur later based on mid-term or long-term memory representations, that is, if the observer recalls that a different object used to be at this location. Finally, the observer has to make a decision as to whether a change has or has not occurred.
Several authors have concluded that change blindness occurs because of a limitation at the first two stages, either because information encoding is strongly limited to the current focus of attention (Rensink et al., 1997; Blackmore, Brelstaff, Nelson, & Troscianko, 1995; O'Regan, 1992) or because visual representations are volatile and easily overwritten when a new object is displayed (Beck & Levin, 2003; Landman, Spekreijse, & Lamme, 2003; Becker & Pashler, 2002). Other findings point to a failure at a later stage. Indeed, several studies showed that recognition of the changing objects is better than change detection performance (Yeh & Yang, 2008; Beck, Peterson, & Angelone, 2007; Varakin & Levin, 2006; Hollingworth, 2005; Hollingworth & Henderson, 2002). Furthermore, Busch (2013) demonstrated that object representations are formed and stored, but not retrieved under change blindness, which precludes them from being used in the decision-making process. Collectively, these results indicate that change blindness does not rule out the encoding, maintenance, and storage of object representations. Interestingly, Mitroff, Simons, and Levin (2004) found that observers could sometimes remember both prechange and postchange object, even when they did not notice that any change had occurred, indicating that change blindness can be because of a failure to compare prechange and postchange objects. In this study, we tested the possibility that semantic features of prechange and postchange objects may be processed and possibly compared even under change blindness, which would indicate that the cause for change blindness arises at a relatively late stage in the processing chain.
Testing for this implicit object processing requires an index of successful processing when explicit change detection fails. ERPs are highly useful to this end. With behavioral measures, implicit or unconscious stimulus processing can be revealed only to the extent that it has an effect on overt behavior, but the processing leading up to this behavior remains unobservable. By contrast, ERPs provide a continuous electrophysiological measure of the stimulus processing that occurs between stimulus presentation and the behavioral response. Therefore, ERPs can be used to reveal the processing of the changing objects even if the behavioral response indicates that the change was not consciously detected.
We leveraged a paradigm previously used to study semantic integration and the N400 ERP component, which correlates with the semantic relation between two sentences or words (Kotz & Friederici, 2003; Rösler, Streb, & Haan, 2001; Kutas & Hillyard, 1980, 1984) or visual objects (Kovalenko, Chaumon, & Busch, 2012; McPherson & Holcomb, 1999; Barrett & Rugg, 1990). Specifically, when the first item (the prime) is semantically unrelated to the second item (the target), the ERP is more negative at approximately 400 msec time-locked to the target item (see Kutas & Federmeier, 2011, for a review). Because the meaning of prime and target object must be accessed before their semantic relation can be processed, the presence of a semantic mismatch effect indicates that (1) both objects were processed up to the level of object identification (as opposed to a mere detection of, e.g., a luminance change), (2) their semantic features were retrieved, and (3) matched with one another. Previous studies have demonstrated N400-like effects for word stimuli that were rendered invisible by masking (Kiefer, 2002; Stenberg, Lindgren, Johansson, Olsson, & Rosén, 2000) or the attentional blink paradigm (Vogel, Luck, & Shapiro, 1998), and such effects have even been observed during sleep (Ibáñez, San Martín, Hurtado, & López, 2008, 2009). Note that, although the N400 effect is not usually presented in the context of change detection, the transition from the first to the second word or object essentially constitutes a change. Thus, it is conceivable that N400-like effects show as well if this change remains invisible because of change blindness. Importantly, finding an N400 effect for undetected changes would provide strong evidence that semantic features of prechange and postchange objects are indeed encoded, integrated across time, and possibly even compared under change blindness, so that the cause of change blindness must occur after the integration stage. Note that we use the term “comparison” in a broader sense to indicate that the effect of semantic object features outlasts the presentation of these objects and is integrated across time, such that the semantic difference between two objects has a measurable effect on their processing.
To test if semantic features of prechange and postchange objects are integrated even when observers are unaware of the change, a paradigm must fulfill two apparently conflicting conditions. (1) Stimulus presentation must allow for sufficient perceptual encoding of the objects. If, for example, objects were presented too briefly, semantic mismatch ERP effects might be absent not because object representations were not semantically compared but because such representations could not be established in the first place. Thus, previous studies of visual N400 effects have typically used stimulus durations of several hundred milliseconds (e.g., Kovalenko et al., 2012; McPherson & Holcomb, 1999). Although N400 effects have been demonstrated even for briefly presented masked words (Kiefer, 2002), these priming effects seem to require that the stimuli be attended (Kiefer & Brendel, 2006) and have been demonstrated only for displays consisting of a single object. (2) Attentional stimulus processing must be limited to induce any change blindness. Thus, previous studies of change blindness have used displays composed of a large number of objects and/or used brief stimulus duration (e.g., Busch, 2013; Schankin & Wascher, 2007; Beck & Levin, 2003) to prevent the deployment of selective attention to the relevant objects. It is difficult to set up a task that fulfills both requirements, because conditions that facilitate perceptual processing of the objects tend to reduce change blindness, and vice versa. In this study, we solved this quandary in the following way. We presented scenes composed of eight objects. Participants were given an unlimited preview of each display, giving them ample time to identify and encode each object. After the preview, one of the objects changed, and this change could be semantically congruent or incongruent. In Experiment 1, we established that a semantic mismatch ERP effect is observed under these conditions when the change is clearly visible. In Experiment 2, we masked the changes with mudsplashes (O'Regan et al., 1999). In addition to the change detection task, participants also solved an unrelated attention-demanding primary task. Thus, change detection was not limited by reducing the perceptual information about the changing objects (e.g., by reducing stimulus duration or increasing the number of objects) but by limiting the attentional resources available for the detection task (see Beck, Rees, Frith, & Lavie, 2001, for a similar procedure). We demonstrate that, under these conditions, a semantic mismatch ERP effect is observed even under change blindness when participants were unable to report the semantic relationship of prechange and postchange objects.
EXPERIMENT 1: SEMANTIC PROCESSING OF CHANGES IN MULTIOBJECT DISPLAYS
In previous research, ERP correlates of semantic processing (i.e., the N400 effect) of visual objects have been studied using displays composed of only a single object at a time. Thus, prime and target objects were usually fully attended. However, studying change detection and change blindness requires more complex displays consisting of multiple distractor objects in addition to the changing object. At present, it is unclear if semantic mismatch effects are observed with multiobject displays. It is possible that the presence of distractors prevents semantic processing of the critical prime and target objects or that the effect of such processing is less prominent in the ERP signal. Thus, we first tested if multiobject displays in which the change from prime to target object is clearly visible (i.e., without inducing change blindness) produce a semantic mismatch effect, comparable to previous reports in the N400 literature.
Seven participants were tested (mean age = 24.7 years, SD = 4.9 years; five women, one left-handed) after signing informed consent. All participants reported being free of neurological or psychiatric disorders and had normal or corrected-to-normal vision. The experimental protocol was approved by the ethics committee of the German Psychological Society (DGPS).
Apparatus and Stimuli
The experiment was written in Matlab (MathWorks, Inc., Natick, MA) using the Psychophysics Toolbox (Brainard, 1997). Participants were seated in a dark, sound-attenuated chamber. Stimuli were presented on a calibrated 19-in. CRT monitor with a resolution of 1280 × 1024 and refresh rate of 100 Hz, placed at a distance of 56 cm from the participants' eyes. Head position was stabilized using a chin rest.
Stimuli were natural object images (cars, animals, tools, etc.), taken from a validated set of semantically related and unrelated object images that have been rated for semantic congruency by a large subject sample (Kovalenko et al., 2012). The set contains 400 prime object images, 400 unrelated target object images, and 400 related target object images. Importantly, related and unrelated images of this set are matched for physical properties.1 However, to verify that the physical properties of the semantically related target objects did not differ from those of the unrelated objects, we analyzed the luminance, visual complexity, and CIE color coordinates of all target images and compared these properties in related and unrelated objects with two-sample t tests.
Luminance was calculated with Matlab's rgb2gray function as the images' gray value averaged across pixels. Luminance did not differ between related and unrelated objects (t(798) = −0.553, p = .581, SD = 37.368).
Visual complexity was quantified using three measures of objective visual complexity suggested by Palumbo, Ogden, Makin, and Bertamini (2014), Bates et al. (2003), and Székely and Bates (2000): the size of JPG files, the GIF ratio, and the JPG75 ratio. The two latter ratios represent the percentage of file compression when transforming a BMP to GIF file. These measures have been shown to correlate with subjective measures of visual complexity (Palumbo et al., 2014; Székely & Bates, 2000). Complexity did not differ between related and unrelated objects (JPG size: t(798) = 0.45, p = .653, SD = 754; GIF ratio: t(798) = 0.851, p = .395, SD = 13.726; JPG75 ratio: t(798) = −0.099, p = .9215, SD = 20.541).
CIE lab color coordinates were calculated using the colorspace function for Matlab2 and subsequently averaging across pixels. CIE color coordinates did not differ between related and unrelated objects (CIE L: t(798) = −0.631, p = .528, SD = 14.968; CIE a: t(798) = 0.65, p = .516, SD = 14.258; CIE b: t(798) = −1.055, p = .292, SD = 19.315).
This analysis corroborates the analysis by Kovalenko et al. (2012) and confirms that the comparison of semantically related and unrelated objects is not confounded by the images' physical properties.
Every display comprised eight different objects. Each object subtended 4.3° visual angle (4.2 × 4 cm) placed on imaginary circles at 3.2° (2 objects), 8.9° (4 objects), and 9.5° (2 objects) from fixation. Additionally, each display contained five letters (four Ls; one T; size: 0.2° visual angle, 0.2 × 0.2 cm each) in the center of the screen. The five letters were arranged to form a cross shape (0.7 × 0.7 cm). Letters were oriented upwards or tilted by 90°, 180°, or 240°.
Experiment 1 consisted of a single session with 30 blocks of 16 trials each. Each block used a “base-scene” with a fixed set of eight objects. Thus, with the exception of the changed target object, the displays were identical in all 16 trials of a block. Objects never occurred in more than one block. At the beginning of each block, participants were given a preview of the base scene for as long as they required to memorize the objects in the scene (minimum duration was 10 sec). We used this procedure to make sure that semantic processing of the changing objects would not be restricted because of limited stimulus encoding.
Following the preview, each block contained 16 trials, half with semantically related and half with unrelated object changes. Each trial started with a blank screen with a central fixation cross lasting 500–1500 msec followed by the prechange display, which was always identical to the base scene. With the onset of the prechange display, the fixation cross changed into a cross-shaped array of five letters. The orientations and the placement of the letters were randomized across trials. After a presentation of 1000 msec, the prechange display was immediately replaced by the postchange display, in which one of the objects was changed. The to-be-changed object was selected at random such that, over the course of a block, each of the eight objects in a base-scene was exchanged once with its related (e.g., rail car changing to rail) and unrelated target object (e.g., rail car changing to sausage). Furthermore, the positions of the letters were again randomized. Although the Ls kept their orientation, the T changed its orientation in half of the trials (counterbalanced across change conditions). The postchange display lasted for 1300 msec. Following the offset of the postchange display, participants had to report the relatedness of the prime and target objects. They were instructed to guess if necessary and to prioritize accuracy over response speed. The paradigm is illustrated in Figure 1. Note that the letter array was irrelevant in Experiment 1 and was included only to be consistent with Experiment 2, where the letters were used in a dual task paradigm. Before the experiment started, participants completed 24–48 training trials with objects not used in the test phase.
EEG was recorded with a BioSemi Active-Two amplifier system (Amsterdam, The Netherlands) from 64 Ag/AgCl electrodes arranged according to the international 10–10 system and the two mastoids. To monitor for eye movements and blinks, the horizontal and vertical EOG was recorded from four additional channels. Two additional electrodes were used as reference and ground. Signals were sampled at 512 Hz with 24-bit conversion resolution and a 0.16-Hz high-pass and a 100-Hz low-pass filter. After recording, data were down-sampled to 256 Hz, low-pass filtered at 30 Hz, converted to an averaged mastoid reference, and epoched from −1300 to 2000 msec time-locked to change onset, that is, onset of the postchange display. After removing large artifacts such as electrode drifts and muscle activity, independent component analysis (ICA) was applied to the unaveraged raw data to correct for eye blinks and eye movements using the MATLAB toolbox EEGLAB and the extended infomax ICA algorithm (Delorme & Makeig, 2004). Raw data were baseline-corrected according to the interval preceding the prechange scene (i.e., −1300 to −1000 msec relative to onset of the postchange scene).
Behavioral Data Analysis
Because each object of a “base scene” changed exactly twice over the course of a block of trials, it is possible that participants kept track of changing objects to predict which object is likely to change on a given trial. Accordingly, accuracy should improve over the course of a block. However, in a debriefing interview, none of the participants reported having used such a strategy intentionally. Moreover, we compared the average d′relatedness of the first and the second half of each block to test if accuracy had improved.
EEG Data Analysis
We tested for effects of semantic processing by comparing semantically related with unrelated changes using only trials in which the relatedness question was answered correctly.
Nonparametric cluster permutation test
To our knowledge, our study is the first to investigate effects of semantic object processing using multiobject displays. Thus, our analysis was exploratory in the sense that we had no a priori assumptions about the latency and spatial distributions of such effects. Therefore, the analysis was conducted time point by time point at each of the 64 channels for the entire epoch (−1300 to 2000 msec). Although this analysis brings up the problem of multiple comparisons, any analysis that focuses on only the relevant channels and time points (when determined a posteriori) would bring up the problem of circular analysis (Kriegeskorte, Simmons, Bellgowan, & Baker, 2009). To address this problem, we used a conservative, non-parametric cluster permutation test introduced by Maris and Oostenveld (2007), which is implemented in the FieldTrip toolbox (Oostenveld, Fries, Maris, & Schoffelen, 2011). The advantage of this test is that differences between two conditions can be identified with minimal assumptions about the spatial and temporal distribution of effects.
The nonparametric cluster permutation test was conducted as follows: (1) ERPs (averaged across trials) for related and unrelated changes were compared across participants with a paired t test at each time point and channel. (2) Clusters were defined as significant effects (p < .05) at contiguous time points and/or adjacent electrodes, with the requirement that at least two adjacent electrodes showed a significant effect at the same time. (3) For each of these clusters, a cluster-level statistic was calculated as the maximum sum (maxsum) of the t values within the cluster. These clusters reflect potential effects in the measured data. (4) A null distribution of such cluster-level t statistics was computed by randomly permuting the data between the two conditions within every participant and applying Steps 2 and 3 to the clusters in this random partition. This procedure was repeated 10,000 times, yielding a distribution of cluster-level statistics under the null hypothesis that any differences between conditions are due to chance. (5) For each cluster in the measured data, a cluster-level p value was estimated as the proportion of the clusters in the null distribution exceeding the observed cluster-level test statistic of that cluster. A significant difference between related and unrelated changes was inferred if the p value of a given cluster was smaller than p = .025.
Note that this statistical test addresses the problem of multiple comparisons across time points and channels because effects observed in the measured data are tested against effects observed under the null hypothesis with the identical number of comparisons. We were specifically interested in significant “negative” clusters, indicating a more negative-going ERP for semantically unrelated compared to related objects. Thus, negative clusters would reflect an experimental effect similar to the N400 effect.
We analyzed the intracranial sources of the semantic mismatch effects to compare them to the sources of the N400 effect as reported in previous studies. To this end, we localized the sources of the ERP in the time range of ±3 time points around the time at which we found the largest t value within a cluster. We used a distributed linear inverse solution and applied the local autoregressive average regularization approach (LAURA), which uses biophysical laws as constraints (Grave de Peralta Menendez, Murray, Michel, Martuzzi, & Gonzalez Andino, 2004; Michel et al., 2004; Grave de Peralta Menendez, Gonzalez Andino, Lantz, Michel, & Landis, 2001). For the lead field matrix calculation, we applied the spherical model with anatomical constraints (SMAC method; Spinelli, Andino, Lantz, Seeck, & Michel, 2000), which transforms the MRI to the best-fitting sphere using homogeneous transformation operators. It then determines a regular grid of 3005 solution points in the gray matter of this spherical MRI and computes the lead field matrix using the known analytical solution for a spherical head model with three shells of different conductivities (Ary, Klein, & Fender, 1981). We first averaged the ERP data across the period of interest and estimated the inverse solution separately for each participant and condition. Nonparametric t tests, based on Monte Carlo bootstrapping methods,3 were calculated at each solution point. Only nodes with p < .05 (two-tailed) and clusters of at least six contiguous nodes were considered significant. This spatial criterion was determined using the AlphaSim program.4 The results of source estimations were rendered on the Montreal Neurologic Institute's average brain with Talairach and Tournoux (1998) coordinates.
Participants correctly judged the relatedness of prechange and postchange objects on average on 85% (SD = 6%) of all trials. Participants were significantly better at identifying unrelated changes than related changes (d′relatedness of 1.99 and 1.19, respectively; t(6) = 4.271, p = .005, SD = 0.5). Performance did not increase throughout the experimental block as indicated by a nonsignificant difference between d′relatedness in the first and second half of the block (d′relatedness of 1.94 and 1.98, respectively; t(6) = −0.942, p = .942, SD = 1.211).
The nonparametric cluster permutation test revealed that ERPs were more negative for semantically unrelated than for related changes in the interval of 613–1059 msec after change onset (mean t = −3.496, SD t = 0.915, p < .001). No other significant clusters were found (Figures 2 and 7A). This effect was widespread across the scalp with a maximum at frontocentral electrodes.
Source estimations were calculated for the maximum of this effect in time interval of 790–813 msec after change onset. Activation was significantly stronger (p < .05; kE = 6 contiguous solution points) within the left middle occipital gyrus for the unrelated versus related condition (see Figure 3A).
In Experiment 1, we found a semantic mismatch effect using visual multiobject displays, indicating that the brain is capable of processing the semantic nature of visual changes in the presence of distracting stimuli.
The mismatch effect occurred only after 600 msec postchange, much later than the conventional N400 effect as seen with single words or object images (for a review, see Kutas & Federmeier, 2011). It is possible that the longer latency of the mismatch effect is due to the presence of distractor stimuli in the display that are absent in classical N400 paradigms and that might delay the identification of the relevant target object. Nonetheless, the frontocentral topography of the semantic mismatch effect is well in line with the results of previous N400 studies (Kovalenko et al., 2012; McPherson & Holcomb, 1999; Holcomb & McPherson, 1994; Barrett & Rugg, 1990).
EXPERIMENT 2: SEMANTIC PROCESSING UNDER CHANGE DETECTION AND CHANGE BLINDNESS
In Experiment 1, we did not mask the transition from prime to target objects. As a result, participants correctly reported the semantic relatedness of prime and target objects on most of the trials, indicating that the change was easy to detect and change blindness rarely (if ever) occurred. As explained in the Introduction, finding a semantic mismatch effect in the comparison of semantically related and unrelated object changes implies that both prechange and postchange objects had been encoded and identified and that their semantic features had been integrated across time. Thus, in Experiment 2, we turn to the question whether the semantic mismatch effect persists when participants are not aware of the change. We used the same paradigm as in Experiment 1 but additionally induced change blindness by masking the change transient with mudsplashes (O'Regan et al., 1999). Furthermore, we introduced a demanding primary task to reduce the attentional resources available for the change detection task.
Methods were identical to Experiment 1 except for the following modifications.
Eighteen participants were tested (mean age = 24.1 years, SD = 4.6 years, nine women, two left-handed) after signing informed consent.
The experiment was divided into two sessions (approximately 3.5 hr each) held on two consecutive days. Each session comprised 25 blocks, and each block consisted of eight trials with semantically related changes, eight trials with unrelated changes, and eight trials without any change.
Up to the end of the first display, the stimulation sequence was identical to Experiment 1. During the first 300 msec of the second display, we presented mudsplashes: multicolored random patterns in the background of all objects and letters (visual angle of 4.9°). The second display was either identical to the first display (no-change trials, one third of all trials) or one of the objects in the first display was replaced by a new object (change trial).
After the end of the display sequence, participants first reported whether the letter “T” had changed its orientation in the second display. They were instructed to prioritize the letter orientation task over the other tasks. Next, participants reported whether they had detected a change among the objects (yes/no). Participants were encouraged to report even a weak glimpse of a change to minimize the number of trials labeled incorrectly as “change blind” that were in fact low-confidence detected trials. Finally, participants reported the relatedness of the changing objects, guessing if necessary. The relatedness task was presented as well when, in fact, no change had occurred in order not to reveal the proportions of change and no-change trials. The paradigm is illustrated in Figure 1.
The first analysis compared d′relatedness between “change detected” and “change blind” trials and analyzed whether d′relatedness in either condition was above chance level. The second analysis tested whether performance in the change detection and relatedness tasks depended on the accuracy of the primary letter orientation tasks. Because the proportion of correct responses in the primary task was close to 1 in all conditions (see Results), we quantified primary task performance on the basis of RTs (measured relative to the onset of the response prompt). Single trial RTs were classified as fast or slow according to each participant's median RT, and d′detection and d′relatedness were compared between fast and slow trials. As in Experiment 1, we tested whether performance improved within a block to exclude the possibility that participants kept track of changing objects. All analyses used two two-sided t tests; p values were Bonferroni-corrected (pBF) for multiple comparisons (two consecutive tests for each analysis).
All analyses were conducted after collapsing data from the two sessions. The first EEG analysis tested for semantic mismatch effects under change detection and, more importantly, under change blindness. To this end, we computed average ERP waveforms for unrelated and related changes separately for (1) detected changes with correct responses in the relatedness task and for (2) undetected changes irrespective of the relatedness judgment, since performance in the relatedness task was at chance under change blindness (see Results).
The second analysis was conducted to substantiate the finding of implicit semantic processing of object changes under change blindness. A principal concern in the study of implicit or subliminal perception is that trials in which the participant reports no awareness of a stimulus in a direct task (here: change detection and relatedness judgments) may represent low confidence rather than the complete absence of awareness. Thus, effects of stimulus processing in an indirect measure (here: semantic mismatch ERP effect) under change blindness can be due to conscious stimulus processing on low-confidence trials rather than unconscious processing. Ideally, the indirect measure should be interpreted as indicating unconscious processing only for participants with no sensitivity in the direct task, that is, for whom d′relatedness on change blind trials is zero. Although d′relatedness was, in fact, not significantly different from chance (see Results), it is unlikely to find participants who fulfill this criterion precisely. To address this problem, Greenwald and Draine (1998) have proposed a regression analysis, in which the indirect measure is regressed onto performance in the direct task (see also Hannula, Simons, & Cohen, 2005; Stenberg et al., 2000; Greenwald, Klinger, & Schuh, 1995). Here, the y-intercept of this regression gives an estimate of the indirect measure (i.e., semantic mismatch ERP effect) when d′relatedness = 0. To this end, we quantified each participant's mean amplitude of the semantic mismatch effect on change blind trials within the significant time × electrode cluster (see Experiment 1, Methods). We then regressed this measure onto participants' d′relatedness on change blind trials and tested the measures for slope and intercept for significance.
The third EEG analysis tested whether semantic mismatch ERP effects are influenced by performance in the primary letter orientation task. The dual-task literature suggests that attending to a primary task can delay the processing of semantic information (Vachon & Jolicoeur, 2011; Sommer & Hohlfeld, 2008; Hohlfeld, Mierke, & Sommer, 2004; Hohlfeld, Sangals, & Sommer, 2004). Thus, the latency of semantic mismatch ERP effects was expected to covary with response speed in the primary task. To test this, we quantified the semantic mismatch ERP effect irrespective of accuracy in the detection and relatedness tasks (on which response speed in the primary task had no effect) and compared semantic mismatch effects between trials with fast and slow responses in the primary task (see Behavioral Analysis above). On the basis of the results of the cluster test conducted in the first analysis, this comparison was restricted to the time range of 1500–2000 msec.
Performance in the primary letter orientation task was almost perfect (accuracy: 96%, SD = 2%), indicating that participants followed the instruction to prioritize this task. The median RT was 1322 msec after the onset of the response prompt (=2622 msec after change onset), with a 25% quantile of 949 msec and a 75% quantile of 2016 msec.
As in Experiment 1, relatedness judgments were more accurate for unrelated than for related changes when the change was detected (d′relatedness of 1.597 and 0.748, respectively; t(17) = −7.172, pBF < 0.001, SD = 0.503) but did not differ when the participants were change blind (d′relatedness of 0.011 and −0.108, respectively; t(17) = −0.389, pBF = 1, SD = 1.305). Importantly, d′relatedness was above chance level only for detected changes (t(17) = 16.793, pBF < 0.001, SD = 0.296) but was not different from chance for undetected changes (t(17) = −1.525, pBF = 0.29, SD = 0.135), indicating that change blindness prevented conscious semantic processing.
Response speed in the primary task had no effect on change detection (d′detection of 2.569 and 2.325, for fast and slow responses respectively; t(17) = 2.234, pBF = 0.078, SD = 0.463) or relatedness judgments (d′relatedness of 0.872 and 0.94, for fast and slow responses, respectively; t(17) = −1.268, pBF = 0.44, SD = 0.228).
As in Experiment 1, performance did not improve within a block (d′relatedness of 0.61 and 0.54, respectively; t(6) = 0.367, p = .718, SD = 0.861), making it unlikely that participants kept track of changing objects.
For detected changes, we found two clusters with frontocentral topography in which the ERP was significantly more negative for unrelated than for related changes (Figures 4 and 7B): at 1551–1707 msec (mean t = −3.021, SD t = 0.692, p = .018) and at 1723–1957 msec (mean t = −3.021, SD t = 0.661, p = .01). In source space, we found significantly stronger activation (p < .05; kE = 6 contiguous solution points) within the left middle superior frontal gyrus and the left superior temporal gyrus (Cluster 1; see Figure 3B) and within the left middle inferior temporal gyrus and the right insula (Cluster 2; see Figure 3C) for unrelated compared to related changes.
Importantly, we found a cluster of semantic mismatch effects with highly similar topography even for undetected changes at 1539–1738 msec (mean t = −3.002, SD t = 0.564, p = .005; Figures 4 and 7C). Source localization of this cluster indicated significantly stronger activation for unrelated versus related changes within the left inferior parietal area, the left middle occipital area, and the middle inferior temporal gyrus (see Figure 3D).
The regression analysis (Figure 5) confirmed that, for undetected changes, the magnitude of the semantic mismatch ERP effect did not depend on performance in the relatedness task, as indicated by the nonsignificant slope (slope = −1.326, t(17) = −0.718, p = .483). More importantly, the y-intercept was significantly different from zero, indicating that a semantic mismatch ERP effect was present even for a d′relatedness of zero (intercept = −1.062; t(17) = −4.091, p < .001).
Finally, we tested whether the semantic mismatch ERP effect occurred later on trials with slower RTs in the primary letter orientation task (see Figure 6). On fast trials, a semantic mismatch ERP effect emerged from the beginning of the cluster's time window and an early cluster (Cluster 1: 1500–1738 msec; mean cluster amplitude = −0.552 μV, mean t = −3.352, SD t = 0.985, p < .001) was followed by a later cluster (Cluster 2: 1758 to 1922 msec; mean cluster amplitude = −0.556 μV, mean t = −2.945, SD t = 0.655, p = .005). On slow trials, the semantic mismatch ERP effect was delayed and only present in the second half of the tested time range (1770–1906 msec; mean cluster amplitude = −0.58 μV, Mean t = −2.675, SD t = 0.425, p = .019).
In Experiment 2, we investigated semantic mismatch ERP effects under change detection and change blindness. To this end, we modified the paradigm from Experiment 1 by masking the changes with mudsplashes (O'Regan et al., 1999) and including no-change trials. We also introduced an attention-demanding primary task and tested participants' ability to detect changes and to judge their semantic relationship.
Note that semantic mismatch ERP effects in Experiment 2 were independent of the processing of the primary task. First, letter orientation (primary task) was fully counterbalanced with the semantic relationship of prime and target object (secondary task). Second, the semantic mismatch ERP effect was also found in Experiment 1, in which the letters were completely task irrelevant.
The semantic mismatch effect in Experiment 2 resembled the N400 in waveform and topography but occurred at a much longer latency (approximately 600 msec in Experiment 1 and 1650–1750 msec in Experiment 2), begging the question whether this effect represents a latency-shifted N400 or an entirely different component. In the following, we discuss factors that might have resulted in a latency shift of the cognitive processes underlying the N400. We also compare the cerebral sources of the semantic mismatch effect reported here and in previous studies of the N400. In summary, the relationship between the N400 and the semantic mismatch effect found here remains uncertain. However, because the main research question addressed in this study was whether any semantic mismatch effect is found under change blindness, whether or not the semantic mismatch effect found here is a genuine, latency-shifted N400 effect is only of secondary interest for this study.
Previous studies of the N400 have typically presented only a single word or object at a time, making it relatively easy to process the semantic content of these stimuli. By contrast, we used displays composed of eight different objects. Because the identity and location of the changing objects were unpredictable, participants were required to memorize and attend to all objects at once, whereas only a single object was changed to induce a semantic match or mismatch. Thus, the attentional and memorization load was significantly larger in this study as compared to previous studies, and this higher load may have contributed to the long latency of the semantic mismatch effect. Indeed, D'Arcy, Service, Connolly, and Hawco (2005) demonstrated that increasing working memory load from one to two potential prime sentences delayed the N400 by 50 msec. One could speculate that if one additional object delays the N400 by the same amount as one additional prime sentence, and if the latency of semantic mismatch effects is linearly related to the number of prime objects, a display comprising eight objects could delay the mismatch effect by hundreds of milliseconds. This might explain the delay of approximately 400 msec of the semantic mismatch effect relative to a typical N400 in Experiment 1.
Moreover, our Experiment 2 used a dual-task paradigm, in which participants processed an attention-demanding primary task before processing the semantic relation of the changing objects. Several studies have demonstrated that the N400 peak latency is delayed in task-switching paradigms and dual-task paradigms (Vachon & Jolicoeur, 2011; Sommer & Hohlfeld, 2008; Hohlfeld, Mierke, et al., 2004; Hohlfeld, Sangals, et al., 2004). Specifically, the studies by Hohlfeld and colleagues showed delays of N400 peak latency of up to 400 msec when two tasks closely overlapped in time. Hohlfeld, Mierke, et al. (2004) suggested that linguistic processes underlying the N400 are part of a central processing stage that can only be occupied by a single task at a time. Hence, the N400 is delayed for as long as participants are occupied with the primary task. In line with this proposal, we found that the semantic mismatch ERP effect occurred later on trials with slower RTs in the primary task (Figure 6), indicating that the semantic relationship was processed only once resources were unoccupied by the primary task. In summary, one could speculate that if dual-task constraints are additive with other task effects, the strong delay of the semantic mismatch effects in Experiment 2 might be explained by the combined effects of working memory load and the dual task.
The topographies of the semantic mismatch ERP effects in Experiments 1 and 2 resembled the topographies of the N400 effect (Kovalenko et al., 2012; McPherson & Holcomb, 1999; Holcomb & McPherson, 1994; Barrett & Rugg, 1990). EEG and MEG studies have localized the sources of this effect in a large network of unilateral and bilateral brain regions, comprising the inferior temporal lobe, the anterior-medial temporal lobe, the inferior/middle/superior frontal gyrus, the inferior/superior temporal sulcus, the middle/superior temporal gyrus, the angular gyrus, occipital areas (e.g., occipito-temporal junction, middle/inferior occipital cortex), the insula, and the parahippocampal gyrus (Geukes et al., 2013; Khateb, Pegna, Landis, Mouthon, & Annoni, 2010; Lau, Phillips, & Poeppel, 2008; Service, Helenius, Maury, & Salmelin, 2007; Van Petten & Luka, 2006; Silva-Pereyra et al., 2003; Halgren et al., 2002; Helenius, Salmelin, Service, & Connolly, 1998; Simos, Basile, & Papanicolaou, 1997). Thus, the sources found in this study, in particular in the middle and superior temporal gyrus, overlapped with the sources of the N400. However, we did not find sources in the anterior temporal lobe, unlike studies of the N400 effect in intracranial EEG recordings in patients (McCarthy, Nobre, Bentin, & Spencer, 1995; Nobre & McCarthy, 1995; Nobre, Allison, & McCarthy, 1994). Further research is required to clarify whether this discrepancy is due to the limitations of scalp-recorded EEG, differences in stimulus material (objects vs. words), or because the semantic mismatch ERP effect reported here is in fact not related to the N400.
Another ERP component that is often observed in studies requiring some form of change detection is the P3 or late positive complex (LPC). The amplitude of this component increases with response confidence, that is, it is more positive for high confidence (Eimer & Mazza, 2005; Finnigan, Humphreys, Dennis, & Geffen, 2002). Is it possible that the semantic mismatch effect found here represents an LPC, rather than an N400? In fact, the LPC typically follows the N400 (Finnigan et al., 2002) and it is maximally distributed at central and parietal channels. Thus, the LPC's latency and topography match the semantic mismatch effect found in this study. However, participants were more accurate at detecting semantically unrelated changes. Assuming that participants were also more confident about those unrelated changes, this would imply that the effect observed here reflects more negativity for high confidence. Thus, it appears unlikely that the semantic mismatch effect is related to the LPC component.
A strong claim that the semantic mismatch ERP effect under change blindness truly reflects unconscious semantic processing can be made only if this ERP effect is found even when performance in the direct behavioral task (i.e., semantic relatedness judgments) is at chance under change blindness. Hannula et al. (2005) and Stenberg et al. (2000) have argued that the most rigorous test for unconscious stimulus processing is to restrict the analysis of the indirect measure (here: semantic mismatch ERP effect) to trials on which participants report no awareness, and to demonstrate that observers' sensitivity in the direct behavioral task is at zero. Indeed, we found a robust semantic mismatch ERP effect under change blindness, while participants performed at chance level in the relatedness judgment task. Furthermore, we modeled the amplitude of the semantic mismatch effect under change blindness for an observer with zero sensitivity in the relatedness judgment task (i.e., d′relatedness = 0). To this end, we regressed the semantic mismatch ERP effect onto performance in the relatedness judgment task (see Stenberg et al., 2000; Greenwald et al., 1995, for a similar approach). Again, the amplitude of the semantic mismatch ERP effect under change blindness was significant (i.e., regression intercept was different from zero) even for a model-observer with zero sensitivity in the direct behavioral task. Our data provide evidence for unconscious semantic processing under change blindness because the semantic mismatch ERP in this study occurred irrespective of participants' state of awareness and their ability to identify the nature of change.
Change blindness—the failure to see changes in visual scenes—is a frequently used tool to study scene perception, attention and visual memory (for review, see Jensen et al., 2011; Rensink, 2002). Change blindness indicates that conscious visual processing is strongly limited outside the focus of attention, but researchers have been at odds as to the exact processing stage at which this limit occurs. Change blindness might occur (1) because stimulus encoding is limited outside the focus of attention (Rensink et al., 1997; Blackmore et al., 1995; O'Regan, 1992), (2) because the maintenance of prechange and/or postchange objects is not stable and representations are too easily overwritten (Beck & Levin, 2003; Landman et al., 2003; Becker & Pashler, 2002; Rensink, 2000; Rensink et al., 1997), and/or (3) because prechange and postchange object are just not compared (Mitroff et al., 2004). We investigated whether semantic features of changing objects give rise to semantic mismatch effects (also referred to as semantic priming) even when the change goes undetected. We reasoned that any ERP effect because of the semantic mismatch between prechange and postchange object requires (1) that both objects were encoded and processed up to the level of semantic analysis, (2) that information about the prechange object is maintained until presentation of the postchange object, and (3) that information about both objects is somehow compared or integrated.
Several studies have investigated semantic priming for invisible stimuli. Some of these studies have rendered prime or target stimuli invisible by bottom–up visual degradation using brief stimulus presentations and masking (Kiefer, 2002; Stenberg et al., 2000). Other studies employed the attentional blink paradigm, where two targets are embedded in a rapid serial presentation stream (Rolke, Heil, Streb, & Hennighausen, 2001; Vogel et al., 1998). If the stimulus onset asynchrony between the two targets is approximately 200 msec, the second target is likely to be missed because processing resources are still occupied with the first target. Both techniques yield reliable N400 effects even when participants are unaware of the relevant stimuli (Kiefer, 2002; Rolke et al., 2001; Stenberg et al., 2000; Vogel et al., 1998).
To our knowledge, this study is the first investigating neural markers of semantic object processing under change detection and change blindness. In contrast to masking and attentional blink, change blindness paradigms render object changes invisible for the observer by means of multiobject displays and concurrent tasks that overload short-term memory and attentional resources. Thus, change blindness occurs because of a lack of top–down attention and not due to visual degradation (Rensink et al., 1997). Therefore, it has been an open question whether semantic object processing persists even under change blindness. This question is important because most explanations of change blindness assume that object representations are either lost or not compared when change blindness occurs (see above; Mitroff et al., 2004; Beck & Levin, 2003; Landman et al., 2003; Becker & Pashler, 2002; Rensink et al., 1997; Blackmore et al., 1995; O'Regan, 1992). Thus, according to these theories, semantic features should not “survive” change blindness altogether or at least should not produce semantic mismatch effects. In this study, a semantic mismatch effect resembling the classical N400 (albeit at longer latencies) was found in the ERP when changes were detected (Experiments 1 and 2). Importantly, the semantic mismatch ERP effect persisted even under change blindness, that is, when participants did not detect the change (Experiment 2) and were not able to judge the semantic relatedness above chance level. This finding implies that change blindness does not preclude the encoding and maintenance of the changing objects and their semantic features. The semantic mismatch ERP effect also implies that even some form of semantic comparison or integration persists under change blindness. Future research needs to determine how exactly this semantic integration is related to the kind of comparison that is necessary for change detection (Mitroff et al., 2004).
Previous research has identified several mechanisms leading to semantic priming effects. Neely (1991) has argued that three separate mechanisms are related to different sub-components of the N400 effect evoked by unmasked words. According to the spreading activation account, presentation of a prime stimulus (e.g., car) activates a node in a semantic network representing conceptual information about this stimulus. Activation automatically spreads from this node to semantically related nodes. If the target stimulus is semantically unrelated to the prime stimulus, its processing is neither inhibited nor facilitated. However, if the target stimulus is semantically related to the prime (e.g., truck), its processing is facilitated because the relevant nodes are already pre-activated so that less activation is needed to pass a recognition threshold. The N400 effect evoked by invisible masked words is thought to reflect only automatic spreading activation (Kiefer, 2002). The spreading activation account holds that the onset of the semantic mismatch ERP effect occurs for prime-target SOAs smaller than 500 msec (Neely, 1991). Here we used a prime target SOA of 1300 msec, a time range untypical for automatic spreading activation effects. Moreover, automatic spreading activation should not interfere with other cognitive processes. However, the dual-task design in Experiment 2 clearly delayed the semantic mismatch ERP effect compared to the single-task design in Experiment 1. Thus, it seems unlikely that the semantic mismatch ERP effect found under change blindness is due to automatic spreading activation. Alternatively, in some situations the context provided by the prime stimulus may make it possible to predict the approximate set of nodes in the network that will be activated by the target stimulus. Unlike automatic spreading activation, expectation is thought to be under the observer's conscious strategic control by actively inhibiting unrelated targets. This mechanism cannot account for the present findings either because participants could not predict which one out of eight objects was about to change on a given trial. While both automatic spreading activation and expectation are anticipatory mechanisms, semantic matching is thought to occur only after lexical access has been completed for both prime and target. According to Neely (1991), semantic matching is under conscious control. Thus, conscious semantic matching is also unlikely to explain the semantic mismatch ERP effect under change blindness.
More recently, Rabovsky and McRae (2014) presented a modeling study suggesting that the N400 reflects an implicit semantic prediction error: a mismatch between the unconsciously extracted and expected content and an incoming target. This model is different from previous proposals in that it states that the matching of the prime and target is not under attentional or conscious control. If the semantic mismatch ERP effect under change blindness followed the same principle, this effect could reflect an implicit prediction error during the comparison of prechange and postchange objects. If so, our finding would indicate that the comparison between prechange and postchange objects can occur implicitly in spite of change blindness. Following this comparison, changes may still go undetected because of a failure at the subsequent decision stage or because the change transient (e.g., the luminance difference between prechange and postchange displays) did not exceed a certain threshold (Hollingworth, 2006) that is independent of the semantic comparison process. Either way, our results conflict with previous assumptions of a sparse and incomplete representation of our outside world and demonstrate instead that change blindness does not necessarily imply impoverished and unstable visual representations.
Here, we demonstrate a semantic mismatch ERP effect indicating the semantic processing of visual object changes. Importantly, this effect was found even when participants were not consciously aware of the change and were unable to report the semantic relationship of the changing objects. This finding implies that change blindness does not preclude the encoding and maintenance of the prechange and postchange objects' identities and possibly even the comparison or integration of their semantic content. Thus, change blindness does not necessarily result from impoverished or volatile visual representations. Thus, the encoding and processing of a visual scene outside the focus of attention is far less impoverished than previously believed.
This work was supported by the German Research Foundation (grant no. BU2400/1-1). Fosco Bernasconi was supported by the Swiss National Science Foundation (Grant No. PBLAP3-142766). We would like to thank Anne Elzemann and Svea C. Schröder for help with stimulus collection and data acquisition.
Reprint requests should be sent to Niko A. Busch, Institute of Medical Psychology, Charité University Medicine, Luisenstr. 57, Berlin, Germany, 10117, or via e-mail: email@example.com.
The images are available for download at www.oszillab.net/downloads.php.
Implemented in Cartool Sofware: brainmapping.unige.ch/cartool.
Available at: afni.nimh.nih.gov/afni/doc/manual/AlphaSim.