Less Is Not More: Neural Responses to Missing and Superfluous Accents in Context

In spoken communication, speakers use prosody—the melody and rhythm of speech—in ways that help the listener understand the message. The function of prosody is very prominent in West Germanic languages such as Dutch, German, and English (Vallduvi, 2002; Ladd, 1996) where speakers assign pitch accents to the most important information in the utterance (the focus element) and leave less important parts unaccented (the background elements). Languages differ in the exact instantiation of accent on elements in focus, so we will use the generic term “focus accent” to refer to the phenomenon in this article.

As an answer to the question, What did the club give to the player?, the sentence They gave (background) a BONUS (focus) to the player (background) has an appropriate focus accent, which emphasizes the segment that answers the question, while a sentence would be inappropriate in which background information receives a focus accent instead, as in They gave (background) a bonus (focus) to the PLAYER (background). In this sense, accents “focus” the listener's attention to the most important information (Wilson & Wharton, 2006), facilitating utterance interpretation (reviewed in Cutler, Dahan, & Van Donselaar, 1997). The function of focus accent in guiding attention to what is important is also clear from the fact that implausible information that is unaccented tends not to be noticed (Wang, Bastiaansen, Yang, & Hagoort, 2011).

The distinction between focus and background information within an utterance, also called “information structure,” derives from the discourse context, which determines which information is familiar and therefore backgrounded. In context, listeners may expect the important information in a certain position within the sentence to be marked prosodically as focus. Nooteboom and Kruyt (1987) have shown that listeners are capable of recognizing inappropriate accentuation in context: in their off-line rating study, listeners rejected sentences containing unaccented elements which were expected to be in focus (“missing” focus accents). Oddly, they tolerated accents on background elements (“superfluous” background accents), despite the fact that focus accent is hypothesized to have the effect of focusing attention on important information. This pattern brings to mind the famous minimalist principle of design that “less is more.” Here, less marking of information structure than necessary, as by a missing accent, leads to more processing difficulty. Intuitively, however, one would have expected that more marking of information structure, as by a superfluous accent, would be more noticeable and hence increase processing costs.

This sort of behavioral data reported by Nooteboom and Kruyt is important but relies on a conscious judgment. ERPs have been used in a number of recent studies to investigate the neural substrates of the processing of linguistic prosody. ERPs are useful because they directly measure brain activity; changes in neural processing related to various conditions can reveal the time point at which a difference is recognized without the need for an explicit task, as well as giving an indication about the nature of the brain responses involved.

Two linguistic functions of prosody have received the most attention in the ERP literature to date: the processing of prosodic boundaries and the use of pitch accents for focus marking (see Table 1 for an overview). Prosodic boundaries, typically consisting of slowed speech tempo and a pitch change, serve to divide speech into segments, usually at syntactic boundaries. The processing of prosodic boundaries per se, as compared with a sequence without a break, consistently evokes an early positivity resembling the P2 component for acoustic differences (Li, Wang, & Lu, 2010), and a late positive component, sometimes called the closure positive shift (CPS; Steinhauer & Friederici, 2001; Steinhauer, Alter, & Friederici, 1999).

In contrast, it is less clear which electrophysiological components underlie the interaction of focus and accentuation. Because focus accent is a potential guide to important information during language comprehension, a number of recent studies have used ERPs to investigate how this information is processed, in particular the response to missing focus accents and superfluous background accents. An extensive list of neural components (responses with different latencies, polarities, and scalp distributions) has been reported in the literature, interpreted as reflecting various neural processes (see Table 1). We believe this variation is present because of the large differences in materials, methods, and experimental designs used in previous studies, rather than a large variability in the way focus accent is processed. Most important in our eyes, in all but two previous ERP studies, participants had to explicitly judge the prosodic well-formedness of the stimuli.

First let us examine the variability among studies. On the one hand, some studies (Bögels, Schriefers, Vonk, & Chwilla, 2011; Heim & Alter, 2006; Magne et al., 2005; Hruska & Alter, 2004; Toepel & Alter, 2004) found effects with a negative polarity. These have frequently been interpreted as evidence for difficulty in semantic processing because of a mismatch with the context, producing an effect similar to the N400, an increased negativity seen over central and parietal electrodes in response to words that do not fit semantically (Kutas & Hillyard, 1980). The second part of Table 1 lists studies that specifically address the relationship between semantic processing and focus, indicated by either syntactic structure (e.g., clefts) or by intonation.

Alternatively, as suggested by Magne et al. (2005), the negativity could be interpreted as a task-related effect such as the “contingent negative variation” (CNV), a negativity that reflects the cognitive preparation for an upcoming stimulus to which the participant must react (Rugg & Coles, 1996; Walter, Cooper, Aldridge, McCallum, & Winter, 1964). The fact that negativities have been found in studies in which an explicit judgment has been used makes this a plausible alternative to the N400 and leads to a completely different view of why the negativity occurs. Unfortunately, it is difficult to tell the two effects apart. The CNV has approximately the same scalp distribution as the N400; it can be more prolonged in duration and has an onset latency varying between 260 and 470 msec (Folmer, Billings, Diedesch-Rouse, Gallun, & Lew, 2011). If the negativity disappears when no explicit judgment task is carried out, that would suggest that the explicit task contributes to the effect and that the reported negativity should be considered a CNV rather than an N400.

A number of positivities have also been reported either instead of or in addition to negativities. Their interpretation has also varied widely, but most often involving reference to the CPS or P600 components. Several studies (Toepel, Pannekamp, & Alter, 2007; Hruska & Alter, 2004; Toepel & Alter, 2004) have attributed positivities elicited by focus elements to the CPS component, a positivity found in response to prosodic parsing (Pannekamp, Toepel, Alter, Hahne, & Friederici, 2005; Steinhauer & Friederici, 2001; Steinhauer et al., 1999), which they then reinterpreted as a marker of information segmentation at focus positions. Because focus elements in these studies often occurred at phrase boundaries that give rise to prosodic parsing, the exact underlying source of the CPS remains ambiguous.

If the positivity reflects information segmentation rather than prosodic parsing, it ought not to vary across sentence position. The prosodic parsing account suggests that the CPS will only occur at clear prosodic boundaries. Existing attempts to disentangle whether the positivity depends on the sentence position of focus accents (Magne et al., 2005) have not supported either of the two views of the CPS. Although Magne et al. report distinct effects for prosodic mismatches in medial sentence position (which they interpreted as a P300) and in final sentence position (N400), no evidence for a CPS-like positivity was found. It is thus important to further investigate how these positivities are correlated with focus and boundary processing as well as to carry out an experiment in which focus accent does not occur at a prosodic boundary.

Unlike the CPS positivity that occurs irrespective of prosodic congruity, other positivities have been found in response to the incongruity of focus accents in context (Schumacher & Baumann, 2010). Such positivities may have a distinct neural source related to the processing of prosodic incongruity and can be interpreted as belonging to the P600 family, positivities that are found when language processing becomes effortful or reanalysis or repair is necessary (Brouwer, Fitz, & Hoeks, 2012; Burkhardt, 2007; Hoeks, Stowe, & Doedens, 2004; Kaan, Harris, Gibson, & Holcomb, 2000; Hagoort, Brown, & Groothusen, 1993; Osterhout & Holcomb, 1992). The functional interpretation of these positivities must take into account whether they reflect the processing of focus accents per se or are rather elicited by prosodic incongruity.

Finally, the positivity seen in a number of studies may actually be related to the P300 component, as suggested by Magne et al. (2005). The P3 is a broadly distributed positive deflection that is seen in response to novel or unexpected stimuli primarily when participants are instructed to pay attention to the stimuli. The P3 has been divided into two parts: the P3a, which is thought to be evoked by the identification of task-related novel events, and the P3b, which is generally linked to task-related decision processes (Picton, 1992; Donchin & Coles, 1988). Examining prosody processing when no secondary task is included should shed light on the extent to which novelty and decision-related processes can account for the positivity reported in these studies.

Biphasic responses have also been reported, in the form of a negativity followed by a positivity. These have been generally interpreted as an N400, followed by either a CPS or a P600. Interpreting biphasic responses is difficult because of the same issues already discussed above for negativities and positivities taken alone. For instance, the CNV is often followed by a positive component called the CNV-Resolution, which is claimed to reflect executive functions that re-establish a cognitive equilibrium such as set-shifting or resetting motor programs (Jackson, Jackson, & Roberts, 1999). Thus, the negativity may reflect expectation violation, and the positivity the resolution of the decision process. All in all, it is difficult to establish whether the findings of previous studies reflect the natural processing of prosody in context.

We have already mentioned that the choice of task may play a role in the variety of responses reported in the literature. The task-related nature of the CNV and P300 emphasize this possibility. In fact, there is evidence that changing the metalinguistic task modifies the neural response to linguistic prosody. Toepel and Alter (2004) showed that neutral accents in a contrastive context (some sort of an underspecified, e.g., missing accent) did not affect processing relative to contrastive accents when participants performed a comprehension task focusing on content but led to a significant biphasic (negative–positive) ERP pattern when listeners performed a prosodic judgment task. For contrastive accents in a neutral context (some sort of an overspecified, e.g., superfluous accent), a negativity was seen for the comprehension task as opposed to a late positivity for the prosodic judgment task. This pattern of no negative effect for a superfluous accent accompanied by a clear negativity for a missing accent has been reported a number of times in the literature in studies using a prosodic judgment task; one goal of the current experiment is to see whether less prosodic marking of focus (missing accent) indeed corresponds to more processing effort when no explicit prosodic judgment task is employed.

The goal of this study is to investigate whether listeners are sensitive to the appropriateness of prosody in the discourse and whether they process missing and superfluous accents in the same way when no prosodic judgment task is employed. Using a strictly controlled naturalistic paradigm, the study focuses on the interaction of prosody and the information structure provided by the linguistic context in short dialogues in Dutch (for materials, see Table 2). In one version, the context question sets up a contrast set on the direct object; the resolution of this choice is given in the answer where the direct object is in focus, whereas in the second version, the question context includes a contrast on the prepositional object, and the direct object in the answer is background information instead. The intonation pattern of the answer is either congruent or incongruent with the context-dependent foregrounding.

To avoid interference from task-related effects that may arise because of the judgment of prosodic congruity, participants performed a comprehension task on a limited number of trials that aimed to guarantee overall attention to the semantic coherence of the dialogues. Special care was taken to control for the following factors known to affect ERP responses: sentences were matched for length (in words), syntactic structure, target lemma frequency, target plausibility, and target expectedness. The last two factors were of special interest: plausibility and expectedness (see Section 2 of Table 1), because it has been shown that they interact with focus accent (Wang et al., 2011) and affect the amplitude of the N400 component more generally. All target nouns had lexically stressed initial syllables with long vowels, which reduced variance in word and accent identification points and allowed us to measure accent processing exclusively without any lexical stress variation (Ladd, Mennen, & Schepman, 2000). ERPs were time-locked to the acoustic onset of each target word rather than to the sentence onset, which would lead to jitter that could mask effects which are relatively short-lasting. Because congruous and incongruous sentences were identical (see Table 2), the baseline should not be an issue. Most importantly and in contrast to previous studies, special care was taken to place targets away from intonational phrase boundaries by placing the finite verb at the end of the sentence and making sure that no prosodic breaks were present at or close to the onset and offset of targets, as these may elicit a CPS (Steinhauer et al., 1999). We believe that by having taken these measures, our study provides an uncluttered view on the neural substrate underlying prosody processing in context.

In line with earlier findings in the literature showing a shallow processing of unaccented and backgrounded information (Li & Ren, 2012; Wang et al., 2011), it is hypothesized that in normal processing, missing accents will not be more noticeable than superfluous accents. On the basis of the known function of focus accent, which is to draw attention to important information, we expect the semantic content of the accented lexical item to be attended and the presence of an incongruous accent on background information to be noted. It is possible that missing accents will be responded to in the same way, but if there is a difference in processing, superfluous accents should require more processing resources than missing accents.

Twenty-nine right-handed Dutch native speakers (13 men, age = 18–29 years, mean = 21 years) with normal or corrected-to-normal vision and without any reported neurological, psychiatric, hearing, or language impairments were paid for participating. Participants signed a written informed consent in accordance with the Declaration of Helsinki. An additional six participants (two men) were not included because they did not meet predefined inclusion criteria (a minimum of 60% artifact free trials for any electrode used in the analysis in any condition). On average, the analysis was performed on 85% valid data over all conditions.

Stimulus construction started with 120 dialogue items (a question followed by an answer) in Dutch, the language used in this study, as illustrated in Table 2. Each question contained a contrastive set with a target noun (selected in the answer; bonus) and a nontarget noun (not selected in the answer; fine). To avoid variability in word identification points, both words had a syllable-initial lexical stress and equivalent average lemma frequencies (taken from the CELEX corpus; Baayen, Piepenbrock, & Van Rijn, 1993). Across conditions, contrast sets in the questions referred either to the direct object (“bonus or fine”) or to the prepositional object (“to the player or to the trainer”); the resolution of the choice represented the focus in the answer. Two further versions of each question were created, in which the linear order of the two contrasted items (e.g., “… bonus or fine …”) was reversed (e.g., “… fine or bonus…”); these versions were counterbalanced to avoid effects of linear presentation. Questions were followed by answers that were either prosodically congruous with a focus accent on one of the contrasted elements (answers (a) in Table 2) or prosodically incongruous with a focus accent on a backgrounded element from the question (answers (b) in Table 2). None of the answers contained semantically inappropriate information. Of interest in these answers were the direct object (i.e., bonus) and the prepositional object (i.e., player).

The plausibility of all target words was tested in Off-line Study 1 (“plausibility study”) with 96 non-Linguistics students who did not participate in the ERP experiment. Participants rated how plausible each target (bonus) and nontarget (fine) was to serve as an answer to the question (on a scale of 1 = very poor fit to 7 = very good fit). We also measured target expectedness by having participants indicate which word of the contrast pair (bonus or fine) they would select as the best answer to the question. On the basis of the results presented in Table 3, the 120 dialogue items were assigned to four item-groups with 30 dialogues each. Target-related factors did not differ significantly between conditions or across lists (all ps > .24).

To investigate prosody processing in naturally elicited speech, experimental stimuli were recorded as dialogues between two phonetically naive speakers: a male speaker produced the questions and a female speaker produced the answers. The speakers recorded clearly accented dialogues as a unit, speaking at a natural speech rate (6.4 syllables/sec) without any excessive emphasis. None of the stimuli contained any disfluencies or phrase boundaries; in fact, all utterances were produced as a single intonational phrase (Gussenhoven, 2005). To prevent unintended intonational differences between conditions, only congruous dialogues were recorded. Incongruous dialogues were generated on the basis of these congruous conditions by recombining questions and answers.

A total of 960 dialogues (120 dialogues × 2 linear orders × 2 question types × 2 answer types) were assigned to eight lists. Each participant was presented with one list of 120 dialogues (30 items × 4 conditions) using the Latin square format. None of the participants listened to more than one version of each sentence, and every participant listened to the experimental stimuli in a pseudorandomized order excluding more than two consecutive presentations of the same condition. In each list, half of the dialogues had focus on the direct object (n = 60) and the other half had focus on the prepositional object (n = 60). In each focus condition, half of the answers were prosodically congruous (focus was accented, n = 30), whereas the other half were prosodically incongruous (background was accented, n = 30). ERP processing differences cannot be attributed to differences in the acoustic characteristics of the stimuli, because the congruous and incongruous conditions were physically identical sentences (1a = 2b, 1b = 2a). All stimuli were normalized in loudness and analyzed acoustically.

An additional Off-line Study 2 (“prosodic congruity study”) was created with a subset of the stimuli to test whether mismatch conditions can be discriminated correctly. Seventeen Linguistics students that did not participate in the ERP study or Off-line Study 1 listened to a subset of dialogues taken from all conditions and indicated whether the question and the answer of a dialogue matched (scale of 1 = very poor fit to 7 = very good fit). No instruction was given with respect to prosodic well-formedness. A repeated-measures ANOVA with Accented Element (direct object vs. prepositional object) and Congruity (congruous vs. incongruous accent) as within-participants factors showed a highly significant main effect of Congruity, F(1, 16) = 245.6; p < .001, indicating that listeners were able to discriminate between congruous and incongruous prosody (average scores are given in Figure 1). No other effects were significant (all ps > .18).

Acoustic measures were performed using the software package Praat (Boersma & Weenink, 2010) and are displayed in Figure 2A and B. Accented direct objects and prepositional objects had a longer acoustic duration and a higher fundamental frequency (f0) relative to unaccented ones. Segmental lengthening under accentuation was larger for direct objects (86 msec) than for prepositional objects (36 msec). Accentuation also affected pitch excursion (difference between f0 max and f0 min), which was higher for accented elements (80 Hz) than for unaccented elements (28 Hz; Figure 2B).

The f0 contours of experimental stimuli were transcribed according to the transcription of Dutch intonation convention (Gussenhoven, 2005). Focus accents on direct objects (Figure 3A) and on prepositional objects (Figure 3B) showed the typical falling pitch contour for Dutch focus accents. In the transcription of Dutch intonation convention, the contour is transcribed as an H*L accent where the letters indicate the direction of pitch movement in the accented syllable, here a falling movement from H (high) to L (low) pitch, whereas the star denotes the pitch of the tone target in the accented syllable, here H (high). Figure 3 shows that the signal did not contain any disruptions of the f0 such as silent pauses or phrase tones in the vicinity of targets that would indicate a phrase boundary.

After electrode application, participants were seated in front of a computer screen in an electrically shielded room and completed a practice session before the actual experiment. Stimuli were presented auditorily via loudspeakers and were divided in two blocks of 60 dialogues (approximate block duration was 12 min). To minimize eye movement artifacts, participants fixated a black cross against a gray background, which appeared 100 msec before stimulus presentation and remained there until the end of the dialogue. In each trial, a question was presented (average duration = 2000 msec), followed by silence (500 msec), an answer (average duration = 2000 msec), and silence again (1200 msec). To encourage attentive processing, participants performed a comprehension task on 25% of all trials and indicated whether a probe word presented on the screen was semantically related to the preceding dialogue. Correct and incorrect responses were counterbalanced. After the response (or after the last silence period in trials without the comprehension task), four stars appeared on the screen (duration = 2000 msec) to indicate that participants had the opportunity to blink.

The EEG was recorded at 250 Hz using a 64-channel cap with Ag/AgCl electrodes, placed according to the international extended 10–20 system (Electro Cap International, Eaton, OH). All channels were amplified against the average of all connected inputs of the amplifier (TMS International, Enschede, The Netherlands). The amplifier measured DC without a high-pass filter but with a digital finite impulse response filter (cutoff frequency of 67.5 Hz) to avoid aliasing effects. After recording, electrodes were re-referenced to the algebraic average of left and right mastoid electrodes. Vertical eye movements and blinks were monitored via electrodes below and above the left eye, and horizontal movements from electrodes at the left and right canthus of each eye. Impedances were kept below 5Ω. All data were filtered off-line with a band-pass filter of 0.01–30 Hz.

Trials containing movement artifacts, ocular artifacts, or electrode drifts (determined by a ±75 μV voltage maximum) were rejected. Only participants with at least 60% valid data in all conditions for any electrode used in the analyses were included (n = 29). On average, EEG analysis was performed on 85% data per condition (SD = 24%). Number of rejected trials did not differ between conditions. ERPs were time-locked to the acoustic onset of each target word that was identical to the onset of its accented syllable.

ERP differences were identified in three time windows post target onset: Early time window (100–220 msec), N400 time window (300–500 msec), and late P600 time window (700–1000 msec). Average ERPs for a number of ROIs were computed as the average over several electrodes. Lateral ROIs included left anterior (FP1, AF3, AF7, F3, F5, F7), right anterior (FP2, AF4, AF8, F4, F6, F8), left central (FC3, FC5, C3, C5, CP3, CP5), right central (FC4, FC6, C4, C6, CP4, CP6), left posterior (P3, P5, P7, PO3, PO7, O1), and right posterior (P4, P6, P8, PO4, PO8, O2). Midline ROIs included anterior (FPz, AFz, Fz), central (FCz, Cz, CPz), and posterior (Pz, POz, Pz).

Repeated-measures ANOVAs were conducted separately for lateral and midline ROIs. ANOVAs for lateral electrodes were calculated with four within-subject factors: Accent (accented element vs. unaccented element), Congruity (contextually congruous accent vs. contextually incongruous accent), Anteriority (anterior vs. central vs. posterior regions), and Hemisphere (left hemisphere vs. right hemisphere). ANOVAs for midline ROIs included all factors except Hemisphere. ANOVAs were performed on mean voltage values and adjusted with the Huynh–Feldt correction for nonsphericity where appropriate. For direct objects, a 200-msec prestimulus baseline correction was calculated for segments with a duration of 1300 msec. For prepositional objects, a 100-msec within-stimulus baseline was chosen because processing differences were expected to have arisen after the perception of mismatches on the direct object (for similar reasoning and procedure, see, e.g., Mueller, 2009; Philips, Kazanina, & Abada, 2005).

Participants judged the semantic relatedness of a probe word to the preceding dialogue in 25% of all trials. Participants were attentive and comprehended dialogues successfully (average accuracy of 87% correct). Task performance was not affected by prosodic congruity.

ERP analyses concentrate on the direct object, whereas data for prepositional objects are regarded as exploratory: half of the time, prepositional objects were preceded by a direct object in a mismatch condition, which will have contaminated their processing (cf. Figure 2A for the average position of both elements in the sentence). Effects involving scalp distribution will be reported only if modified by the cognitive factors.

Statistical results are presented in Table 4, and ERP waveforms for all conditions are displayed in Figure 4. Marginally significant main effects or interactions (.05 ≤ p ≤ .10) will be reported in footnotes for future reference but will not be followed up or interpreted.

A four-way interaction of Accent × Congruity × Anteriority × Hemisphere, F(2, 56) = 3.223, p < .05, was found. Follow-up analyses with Anteriority as the split variable revealed an Accent × Congruity × Hemisphere interaction that was significant on posterior, F(1, 28) = 6.935, p < .05, marginal on central, F(1, 28) = 3.817, p = .06, and not significant on anterior regions, F(1, 28) = .075, p = .79. Following up on the posterior interactions with Accent as the split variable, the interaction of Congruity × Hemisphere was not significant for accented direct objects, F(1, 28) = 2.54, p = .12, or for unaccented direct objects, F(1, 28) = 2.659, p = .11. The mean voltage data suggest that the Congruity × Hemisphere interaction is triggered by left-sided posterior positivities for incongruously accented direct objects relative to congruous ones, probably coupled with a greater positivity for incongruously accented elements on the right. When looking at the Congruity effect on posterior regions separately for accented and unaccented direct objects, we found that it is present only for accented direct objects, F(1, 28) = 3.492, p = .07, because of a positivity for incongruously accented elements (superfluous accents) relative to congruously accented elements. No Congruity effect was found for incongruously unaccented direct objects (missing accents; F(1, 28) = .725, p = .4).

There was a main effect of Accent, F(1, 28) = 6.207, p < .05, showing that accented direct objects elicited positive waveforms relative to unaccented ones. There was a three-way interaction of Accent × Congruity × Hemisphere, F(1, 28) = 4.416, p < .05, showing a significant Congruity × Hemisphere interaction for accented elements, F(1, 28) = 11.807, p < .01, but not for unaccented elements, F(1, 28) = .319, p = .58. Post hoc tests on accented direct objects revealed a significant Congruity effect at right sites, F(1, 28) = 4.8, p < .05, but not at left sites, F(1, 28) = .190, p = .67. The mean voltage values in Figure 5 show that the Congruity effect was a right-lateralized negativity for incongruously accented elements (superfluous accents on background elements) as compared with congruously accented ones (focus accents). No such negative effect was elicited by incongruously unaccented elements (missing accents on focus elements; cf. Figure 6). For midline electrodes, there was a main effect of Accent, F(1, 28) = 9.726, p < .01, indicating that ERPs to accented direct objects were more positive than ERPs to unaccented ones. No other main effects or interactions were significant.

For lateral electrodes, the Congruity × Anteriority interaction was significant, F(2, 56) = 9.292, p < .01. Follow-up comparisons suggest that this is because of a marginal effect of Congruity (incongruous more positive than congruous) on posterior regions, F(1, 28) = 3.842, p = .06, and the absence of such an effect on anterior and central regions (all p values > .1).¹ On midline electrodes, there was also a significant interaction of Congruity × Anteriority, F(2, 56) = 3.378, p < .05; together these suggest that irrespective of whether the accent is missing or superfluous, direct objects with incongruous accentuation were more positive than congruous direct objects, but only at posterior sites. No other effects were significant.

As mentioned above, the analysis of prepositional objects has an exploratory character because the ERPs to the prepositional object will be affected by the processing of the preceding violation on the direct object. Statistical results are presented in Table 5, and ERP waveforms for all four conditions are displayed in Figure 7.

On lateral electrodes, there was a Congruity × Hemisphere interaction, F(1, 28) = 5.420, p < .05, but follow-ups did not reveal a statistically reliable Congruity effect (all ps > .42). According to the means, the interaction must have been due to a positivity for incongruous elements over the right hemisphere and a negativity over the left hemisphere. A Congruity × Anteriority interaction, F(2, 56) = 4.158, p < .05, did not show significant differences on any region (all ps > .18). The means suggest that the interaction came about by an anterior negativity and a posterior positivity for incongruous elements. On midline electrodes, there was a main effect of Accent, F(1, 28) = 6.314, p < .05, showing a positivity for accented prepositional objects relative to unaccented ones. No other effects reached significance.²

The ERPs for prepositional objects did not show the negativity that we found in this time window for direct objects but predominantly rather large positive-going waves starting around 300 msec that were elicited for both accented and unaccented incongruous elements [Accent × Congruity × Anteriority × Hemisphere, F(2, 56) = 3.361, p < .05]. Follow-up analyses suggested that this interaction resulted from centro-posterior positivities associated with incongruity, for both accented and unaccented words, with the effects for accented words larger at right hemisphere sites. These positivities were accompanied by anterior negativities that were larger for unaccented words, most clearly so at left frontal sites. For midline electrodes, accented prepositional objects elicited a positivity relative to unaccented ones that was reflected in a main effect of Accent, F(1, 28) = 14.924, p = .001. In addition, both accented [Accent × Anteriority, F(2, 56) = 5.214, p < .05] and incongruous prepositional objects [Congruity × Anteriority, F(2, 56) = 6.789, p < .01] triggered centro-posterior positivities.

Accented prepositional objects were more positive than unaccented ones because of a main effect of Accent, F(1, 28) = 4.391, p < .05. The positivity for accented elements had a centro-posterior distribution [Accent × Anteriority interaction, F(2, 56) = 19.591, p < .001; Accent effect at central sites, F(1, 28) = 4.237, p < .05, and at posterior sites, F(1, 28) = 35.282, p < .001]. Incongruous prepositional objects elicited posterior positivities relative to congruous ones [Congruity × Anteriority, F(2, 56) = 6.014, p < .05] that was due to a Congruity effect at posterior sites, F(1, 28) = 5.091, p < .05. On midline electrodes, accented prepositional objects were more positive than unaccented ones because of a main effect of Accent, F(1, 28) = 6.714, p < .05. The positivity was distributed at centro-posterior sites as revealed by an Accent × Anteriority interaction, F(2, 56) = 13.308, p = .001, with an Accent effect over central, F(1, 28) = 7.081, p < .05, and posterior regions, F(1, 28) = 30.403, p < .001.

Accent (present vs. absent) and Congruity (match vs. mismatch) interacted with each other at the direct object in both the early and the N400 time windows, but not in the later time window. In the early time window (100–220 msec), Congruity had an effect primarily on accented words: direct objects with superfluous accents elicited early positivities on left posterior sites, relative to direct objects with congruous accentuation. In the N400 time window (300–500 msec), incongruent accents elicited a right-lateralized centro-posterior negativity. No such effect was obtained for incongruous unaccented words (missing accents). In the late P600 time window (700–1000 msec), there was no interaction and both types of incongruous prosody, that is, missing and superfluous accents, were more positive than congruous prosody.

The exploratory analyses for the prepositional objects showed posterior positivities for incongruent relative to congruent prosody in the N400 and in the late time window, similar to the positivity for incongruence elicited by the direct object. In general, the processing of accentuation was evident in a broadly distributed main effect of Accent (300–500 msec), showing a positivity for accented relative to unaccented direct objects. A positive Congruity effect was also apparent for accented prepositional objects but it started somewhat earlier (100–220 msec, 700–1000 msec). Because of the fact that the processing of the prepositional object is affected by the congruity of the direct object earlier in the sentence, the following discussion addresses only ERP patterns associated with the direct objects.

The current study investigated the processing of linguistic prosody in context, particularly whether superfluous accents on background information and missing accents on focus information evoke distinct neural mechanisms in a natural paradigm without a prosodic task. Earlier studies have shown effects that we conjectured might be because of the specific tasks used in those studies. Additional variability in the results (see Table 1) may have resulted from issues involving the presence of prosodic boundaries and the time-locking and matching of stimuli, which we controlled in the current study.

The neural correlate of accentuation was evident as a broadly distributed positivity for accented elements relative to unaccented elements which started around 300 msec post onset of the direct object and 100 msec post-onset of the prepositional object. The positivity is independent of whether accent is congruent with the context or not and replicates earlier reports (Wang et al., 2011; Heim & Alter, 2006) of a positivity associated with the occurrence of a pitch accent. The effect can be interpreted as belonging to the P200 component for the processing of physical characteristics of accented elements or to the P300 for the attentive processing of accented elements (Heim & Alter, 2006). In our view, this effect is best described as an “accent positivity,” which consists of a sensory aspect related to the processing of acoustic features as well as of a cognitive aspect, which implies the attentive processing of prosodic prominence. The accent positivity is independent of information structure and contextual congruity and suggests that focus accent is processed in a bottom–up manner.

Possibly because of our strict time-locking and stimulus selection procedures, we were also able to discover very early effects (around 100 msec after target onset) for incongruous prosody, even in the absence of explicit instructions to attend to prosodic aspects of the stimuli. This early congruity effect (incongruous more positive than congruous) is present for accented words, but absent for unaccented words and likely reflects top–down processing of focus accent based on contextual information. Further evidence for a more elaborate processing of superfluous accents than missing ones was the negativity in the N400 time window that was triggered by superfluous accents but was absent for missing accents. This is not to say that missing accents on focused information went unrecognized. Both missing and superfluous accents triggered a late posterior positivity, resembling the P600 component. Thus, the neural response to prosodic congruity is qualitatively different between superfluous and missing accents early on, but very similar in the later P600 time window.

Our results are strikingly different from those reported in most previous studies, in which a prosodic judgment task was employed. These studies found that a missing accent leads to more processing difficulty than a superfluous one. However, our results are consistent with Toepel and Alter (2004), who also found evidence for a clear difference between the response to focus accent, depending on whether a prosodic judgment task was used or not. When no prosodic task was used, they found a broadly distributed negativity for superfluous accents (see also Wang et al., 2011). In our experiment, the superfluous accents also gave rise to a negativity. However, there were also important differences. Toepel and Alter did not find any other effect of incongruity for either superfluous or missing accents. In contrast, we found evidence for an increased processing cost in both mismatch conditions: For superfluous accents, there was an additional early positive effect (100–220 msec); for both types of incongruity there was a late posterior positivity (700–1000 msec). The use of stimuli which were time-locked to the onset of the accented syllable of targets might have been responsible for our divergent results, as well as the avoidance of phrase boundaries in the vicinity of targets, which allowed us to provide a clearer view on the neural correlates of processing focus accent per se.

The present results make it clear that specific, task-related neurocognitive mechanisms are active when a prosodic task is added (Table 1; current results; Toepel & Alter, 2004). In the Introduction, it was suggested that previous findings in studies employing a prosodic judgment task can be accounted for by what we know about task-related ERP components. Under a prosodic judgment task, a missing accent has most generally elicited a biphasic N400-P600 pattern (Hruska & Alter, 2004; Toepel & Alter, 2004; Hruska, Alter, Steinhauer, & Steube, 2001; but see Magne et al., 2005), whereas a superfluous accent gives rise to (i) no effect (Hruska et al., 2001), (ii) a negativity (however, only with a comprehension task, Toepel & Alter, 2004), or (iii) a late positivity (Hruska & Alter, 2004).

Missing and superfluous accents both represent violations of the normal alignment of prosody and information structure, namely the use of information in focus with an accent and background information without an accent. Despite this, there is an apparent asymmetry in neural processing between missing and superfluous accents, which becomes clear if one considers that their detection proceeds qualitatively differently, depending on the task at hand. With a prosodic judgment task, the listener may very well exploit the linguistic context, that is, the focus elements from the question, to predict the position of focus; thus, the detection of a missing accent where one is expected provides sufficient evidence for a prosodic mismatch decision. This may give rise to an effect like the CNV, which reflects the cognitive preparation for an upcoming stimulus to which the participant must react (cf. Magne et al., 2005). This negativity is often followed by a positive component called the CNV-Resolution, which is claimed to reflect executive functions that re-establish a cognitive equilibrium such as set-shifting or resetting motor programs (Jackson et al., 1999). Thus, the negativity often found for missing accents may reflect expectation violation, and the positivity could then index the resolution of the decision process: The participant becomes aware that the expected accent is indeed missing and that the stimulus is prosodically not well formed.

Processing a superfluous accent in the prosodic judgment task condition is different. There is no “warning” signal in the context that a critical stimulus is imminent and that a choice must be made at this particular point in the sentence. The superfluous accent is unexpected and most likely creates a surprise effect that might evoke a P300-like positivity for unexpected events (Picton, 1992; Donchin & Coles, 1988) rather than a CNV-like negativity for task-related expectation mismatch. In some cases, though, the superfluous accent may, for unknown reasons, escape detection altogether (Hruska et al., 2001). In summary, the findings in earlier studies seem to us to be artifacts of the added prosodic judgment task, obscuring the processes that are operational during “normal” speech processing.

Without a prosodic task that can modify the effects of incongruous prosody, we still find asymmetries in the processing of missing and superfluous accents. However, these seem to go in the opposite direction, with superfluous accents noticed earlier than missing ones. Superfluous accents give rise to an early positivity and an N400-like negativity, whereas no such effects are obtained for missing accents. This does not mean that the missing accent was “missed”: We did find a later positivity in response to both missing and superfluous accents. In addition, sentences with missing accents were clearly recognized as infelicitous in our Off-line Study 2. The exact nature of the late positivity is not completely clear. It resembles a P600, which has frequently been reported in cases where it is difficult to create a coherent representation for various reasons (Brouwer et al., 2012; Burkhardt, 2007; Hoeks et al., 2004; Kaan et al., 2000; Hagoort, et al., 1993; Osterhout & Holcomb, 1992). In line with previous accounts, we interpret the positivity for incongruous prosody as indicating effortful processing initiated to arrive at a coherent interpretation with respect to the preceding context; we will discuss this effect more extensively below.

Superfluous accents also gave rise to prominent early effects, triggering an early positive effect (100–220 msec post-onset). This early congruity effect has not been reported before, and we believe that it was because of our straightforward time-locking procedure and the extensive matching of experimental stimuli that we were able to detect it. The exact nature of this early positivity, however, is still a puzzle. It could be related to the P200 component evoked by changes in pitch direction (Friedrich, Kotz, Friederici, & Alter, 2004), but this seems unlikely as our positivity is triggered by accented elements (congruous focus accents vs. superfluous focus accents) in physically identical sentences, which differed only with respect to the preceding context. The positivity must thus be related to the incongruity with respect to the preceding discourse context. Exploration of the functional meaning of this early positive congruity effect must await further research.

Around 300 msec after the onset of a direct object with superfluous accent, a right-lateralized centro-posterior negativity was found, which resembles a standard N400 effect superimposed on the positive main effect of accentuation. Under the standard view of the N400 (Kutas & Hillyard, 1980), this negativity might reflect semantic integration demands caused by the interpretation of the prosodic mismatch and straightforwardly be interpreted as an N400 effect. That is, the superfluous accent may hinder the interpretation of background information as “given” and require its reinterpretation as “new” and in focus. Alternatively, Dutch focus accents can be used to indicate contrast when they occur in unexpected positions (Swerts, Krahmer, & Avesani, 2002). An unexpected accent might spur listeners to construct a contrastive interpretation for the element with a superfluous accent, and because contrast is not supported by the context, additional effort may be necessary. These effects are also very similar to effects seen in response to information structure mismatches such as in the repeated name penalty (LeDoux, Camblin, Swaab, & Gordon, 2006; Gordon, Grosz, & Gilliom, 1993). Using a reference form that is more prominent and elaborate than strictly required gives rise to an increase in N400. For instance, in a sentence such as “Pam washed the dishes while Pam talked about politics,” the second occurrence of Pam (underlined), in a position where a reduced form (e.g., she) is more appropriate, engenders a significantly larger N400 than in a control sentence. In a similar vein, in the current experiment the superfluous accent signals that the word contains important new information (e.g., Wilson & Wharton, 2006; Gussenhoven, 2005), which turns out not to be the case. Exactly how the N400 in response to superfluous accents should be interpreted is not completely clear: it is definitely a sign of additional difficulty as discussed above, but it may merely be a signal of an information structure mismatch or it may also reflect semantic activation or reprocessing. This should be looked into in future experiments.

As we have shown, late positivities were elicited by superfluous and missing accents in this study, which likely reflect the effortful processing of incongruous prosody aimed at salvaging an ill-formed utterance, as listeners try to make sense of what the speaker just communicated. Previous studies on prosody processing have attributed similar late positivities to the CPS (see Table 1) that is implicated in the processing of prosodic boundaries (Steinhauer & Friederici, 2001) or in the information segmentation at focus positions in context (Hruska & Alter, 2004; Toepel & Alter, 2004).

Unlike these CPS positivities, the late positivity in this study is clearly related to the processing of a mismatch between prosody and context and is therefore analyzed as a P600 effect. Late positivities in the current study cannot be straightforwardly interpreted as being effects of closure positivity as our stimuli were strictly controlled to avoid confounds with boundary-induced effects. As shown in Figure 3A and B, none of the experimental conditions contained any silent pauses, breaks, or pitch changes in the signal in the vicinity of targets which could have been confounded with a CPS response for prosodic parsing. Moreover, the positivities do not exclusively occur at focus positions but are elicited by both focus and background elements with incongruous prosody; thus, they cannot be exclusively attributed to focus segmentation.

Importantly, the comparison of congruous and incongruous conditions only used physically identical stimuli, and hence contextual congruity represents the only source of the positivity. The strongest evidence that late positivities in the present experiment do not reflect boundary processing is the fact that they occur not only after incongruously accented targets (Ladd, 1986) but also after incongruously unaccented targets. One might assume that accented words might generate the impression of a boundary because of their acoustic lengthening. However, no such segmental lengthening was measured for unaccented words, and these also gave rise to late positivities in the incongruous condition. The distribution of the positive congruency effect over posterior lateral and midline electrodes is identical in both conditions, which represents further evidence for its similar neural source.

We argue that the late positivities in our data are part of the P600 family and reflect general processes of making sense that are activated by prosodic mismatches (similar to Schumacher & Baumann, 2010). These positivities mark the workings of a general mechanism for the extended analysis of complex information, in this case prosodically misrealized information, and its integration in the discourse (e.g., Brouwer et al., 2012; Hoeks, Hendriks, Redeker, & Stowe, 2010; Burkhardt, 2007). A number of the studies reported in the literature have found late positivities for prosodic mismatches (see Table 1), regardless of whether a prosodic task was carried out, which suggests that the presence of a prosodic task is not the main source of the late positivity, though future research using strictly controlled materials will be needed to determine whether this is the case.

The current study has demonstrated that when listeners are not engaged in a conscious prosodic judgment task, they respond more strongly to accented background information than to unaccented focus information, and that this response is quite early (100 msec). This is not to say that listeners are unaware of missing accents as they clearly react to the incongruity of both sorts of contextual mismatch in a later stage of processing, underlining the importance of prosodic information to normal processing and integration of incoming information into the discourse context. Unlike previous studies in which a prosodic judgment task was used, however, our participants did not find that “less is more.”

This work was supported by an Ubbo Emmius Grant awarded to Diana V. Dimitrova. We would like to thank the three anonymous reviewers for their insightful suggestions, Ryan Taylor for technical assistance, and Myrte Gosen and Albert Everaarts for lending their voices to create the stimuli.

Reprint requests should be sent to Diana V. Dimitrova, Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, Kapittelweg 29, 6525 EN Nijmegen, The Netherlands, or via e-mail: [email protected].