When people talk, their speech is often accompanied by gestures. Although it is known that co-speech gestures can influence face-to-face communication, it is currently unclear to what extent they are actively used and under which premises they are prioritized to facilitate communication. We investigated these open questions in two experiments that varied how pointing gestures disambiguate the utterances of an interlocutor. Participants, whose event-related brain responses were measured, watched a video, where an actress was interviewed about, for instance, classical literature (e.g., Goethe and Shakespeare). While responding, the actress pointed systematically to the left side to refer to, for example, Goethe, or to the right to refer to Shakespeare. Her final statement was ambiguous and combined with a pointing gesture. The P600 pattern found in Experiment 1 revealed that, when pointing was unreliable, gestures were only monitored for their cue validity and not used for reference tracking related to the ambiguity. However, when pointing was a valid cue (Experiment 2), it was used for reference tracking, as indicated by a reduced N400 for pointing. In summary, these findings suggest that a general prioritization mechanism is in use that constantly monitors and evaluates the use of communicative cues against communicative priors on the basis of accumulated error information.

This experiment investigates the integration of gesture and speech from a multisensory perspective. In a disambiguation paradigm, participants were presented with short videos of an actress uttering sentences like “She was impressed by the BALL, because the GAME/DANCE….” The ambiguous noun (BALL) was accompanied by an iconic gesture fragment containing information to disambiguate the noun toward its dominant or subordinate meaning. We used four different temporal alignments between noun and gesture fragment: the identification point (IP) of the noun was either prior to (+120 msec), synchronous with (0 msec), or lagging behind the end of the gesture fragment (−200 and −600 msec). ERPs triggered to the IP of the noun showed significant differences for the integration of dominant and subordinate gesture fragments in the −200, 0, and +120 msec conditions. The outcome of this integration was revealed at the target words. These data suggest a time window for direct semantic gesture–speech integration ranging from at least −200 up to +120 msec. Although the −600 msec condition did not show any signs of direct integration at the homonym, significant disambiguation was found at the target word. An explorative analysis suggested that gesture information was directly integrated at the verb, indicating that there are multiple positions in a sentence where direct gesture–speech integration takes place. Ultimately, this would implicate that in natural communication, where a gesture lasts for some time, several aspects of that gesture will have their specific and possibly distinct impact on different positions in an utterance.

The present series of experiments explores several issues related to gesture–speech integration and synchrony during sentence processing. To be able to more precisely manipulate gesture–speech synchrony, we used gesture fragments instead of complete gestures, thereby avoiding the usual long temporal overlap of gestures with their coexpressive speech. In a pretest, the minimal duration of an iconic gesture fragment needed to disambiguate a homonym (i.e., disambiguation point) was therefore identified. In three subsequent ERP experiments, we then investigated whether the gesture information available at the disambiguation point has immediate as well as delayed consequences on the processing of a temporarily ambiguous spoken sentence, and whether these gesture–speech integration processes are susceptible to temporal synchrony. Experiment 1, which used asynchronous stimuli as well as an explicit task, showed clear N400 effects at the homonym as well as at the target word presented further downstream, suggesting that asynchrony does not prevent integration under explicit task conditions. No such effects were found when asynchronous stimuli were presented using a more shallow task (Experiment 2). Finally, when gesture fragment and homonym were synchronous, similar results as in Experiment 1 were found, even under shallow task conditions (Experiment 3). We conclude that when iconic gesture fragments and speech are in synchrony, their interaction is more or less automatic. When they are not, more controlled, active memory processes are necessary to be able to combine the gesture fragment and speech context in such a way that the homonym is disambiguated correctly.

Although the neurocognitive processes underlying the comprehension of figurative language, especially metaphors and idioms, have been studied extensively, less is known about the processing of irony. In two experiments using event-related brain potentials (ERPs), we examined the types of cognitive processes involved in the comprehension of ironic and literal sentences and their relative time course. The experiments varied in modality (auditory, visual), task demands (comprehension task vs. passive reading), and probability of stimulus occurrence. ERPs consistently revealed a large late positivity (i.e., P600 component) in the absence of an N400 component for irony compared to equivalent literal sentences independent of modality. This P600 was shown to be unaffected by the factors task demands and probability of occurrence. Taken together, the findings suggest that the observed P600 is related to irony processing, and might be a reflection of pragmatic interpretation processes. During the comprehension of irony, no semantic integration difficulty arises (absence of N400), but late inferential processes appear to be necessary for understanding ironic meanings (presence of P600). This finding calls for a revision of current models of figurative language processing.

We investigate how L1 phonology and semantics affect processing of interlingual homographs by manipulating language context before, and auditory input during, a visual experiment in the L2. Three experiments contained German–English homograph primes ( gift = German “poison”) in English sentences and was performed by German (L1) learners of English (L2). Both reaction times and event-related brain potentials were measured on targets reflecting the German meaning of the interlingual homograph. In Experiment 1, participants viewed a pre-experiment English film, then half of the participants ( n = 16) heard noise and the other half ( n = 16) heard German pseudowords during the experiment; in Experiment 2, participants ( n = 16) viewed a pre-experiment German film then heard noise; and in Experiment 3, participants ( n = 16) viewed the pre-experiment English film then heard real German words. Those who had viewed the English film then heard noise during Experiment 1 showed no L1 influence. Those who saw the English film but heard German pseudowords during Experiment 1, or viewed the German film before and heard noise during Experiment 2, showed L1 influence as indicated by N400 priming of L1-related targets in the first half of the experiment. This suggests that a pre-experiment film in the L1 or the presence of L1 phonology during the experiment slowed down adjustment to the L2 task. In Experiment 3 with real L1 words in the background, N400 priming of L1 meanings was observed throughout the entire experiment for lower-proficiency participants. We discuss our findings in terms of context types that affect L1-to-L2 adjustment.

The present series of experiments explored the extent to which iconic gestures convey information not found in speech. Electroencephalogram (EEG) was recorded as participants watched videos of a person gesturing and speaking simultaneously. The experimental sentences contained an unbalanced homonym in the initial part of the sentence (e.g., She controlled the ball …) and were disambiguated at a target word in the subsequent clause ( which during the game … vs. which during the dance …). Coincident with the initial part of the sentence, the speaker produced an iconic gesture which supported either the dominant or the subordinate meaning. Event-related potentials were time-locked to the onset of the target word. In Experiment 1, participants were explicitly asked to judge the congruency between the initial homonym-gesture combination and the subsequent target word. The N400 at target words was found to be smaller after a congruent gesture and larger after an incongruent gesture, suggesting that listeners can use gestural information to disambiguate speech. Experiment 2 replicated the results using a less explicit task, indicating that the disambiguating effect of gesture is somewhat task-independent. Unrelated grooming movements were added to the paradigm in Experiment 3. The N400 at subordinate targets was found to be smaller after subordinate gestures and larger after dominant gestures as well as grooming, indicating that an iconic gesture can facilitate the processing of a lesser frequent word meaning. The N400 at dominant targets no longer varied as a function of the preceding gesture in Experiment 3, suggesting that the addition of meaningless movements weakened the impact of gesture. Thus, the integration of gesture and speech in comprehension does not appear to be an obligatory process but is modulated by situational factors such as the amount of observed meaningful hand movements.

The present study investigated the processing of two types of artificial grammars by means of event-related brain potentials. Two categories of meaningless CV syllables were applied in each grammar type. The two grammars differed with regard to the type of the underlying rule. The finite-state grammar (FSG) followed the rule (AB) n , thereby generating local transitions between As and Bs (e.g., n = 2, ABAB). The phrase structure grammar (PSG) followed the rule A n B n , thereby generating center-embedded structures in which the first A and the last B embed the middle elements (e.g., n = 2, [A[AB]B]). Two sequence lengths ( n = 2, n = 4) were used. Violations of the structures were introduced at different positions of the syllable sequences. Early violations were situated at the beginning of a sequence, and late violations were placed at the end of a sequence. A posteriorly distributed early negativity elicited by violations was present only in FSG. This effect was interpreted as the possible reflection of a violated local expectancy. Moreover, both grammar-type violations elicited a late positivity. This positivity varied as a function of the violation position in PSG, but not in FSG. These findings suggest that the late positivity could reflect difficulty of integration in PSG sequences.

The present study investigated simultaneous processing of language and music using visually presented sentences and auditorily presented chord sequences. Music-syntactically regular and irregular chord functions were presented synchronously with syntactically correct or incorrect words, or with words that had either a high or a low semantic cloze probability. Music-syntactically irregular chords elicited an early right anterior negativity (ERAN). Syntactically incorrect words elicited a left anterior negativity (LAN). The LAN was clearly reduced when words were presented simultaneously with music-syntactically irregular chord functions. Processing of high and low cloze-probability words as indexed by the N400 was not affected by the presentation of irregular chord functions. In a control experiment, the LAN was not affected by physically deviant tones that elicited a mismatch negativity (MMN). Results demonstrate that processing of musical syntax (as reflected in the ERAN) interacts with the processing of linguistic syntax (as reflected in the LAN), and that this interaction is not due to a general effect of deviance-related negativities that precede an LAN. Findings thus indicate a strong overlap of neural resources involved in the processing of syntax in language and music.

The present event-related potential (ERP) study examined the role of tone and segmental information in Cantonese word processing. To this end, participants listened to sentences that were either semantically correct or contained a semantically incorrect word. Semantically incorrect words differed from the most expected sentence completion at the tone level, at the segmental level, or at both levels. All semantically incorrect words elicited an increased frontal negativity that was maximal 300 msec following word onset and an increased centroparietal positivity that was maximal 650 msec following word onset. There were differences between completely incongruous words and the other two violation conditions with respect to the latency and amplitude of the ERP effects. These differences may be due to differences in the onset of acoustic deviation of the presented from the expected word and different mechanisms involved in the processing of complete as compared to partial acoustic deviations. Most importantly, however, tonally and segmentally induced semantic violations were comparable. This suggests that listeners access tone and segmental information at a similar point in time and that both types of information play comparable roles during word processing in Cantonese.

Behavioral evidence suggests that spoken word recognition involves the temporary activation of multiple entries in a listener's mental lexicon. This phenomenon can be demonstrated in cross-modal word fragment priming (CMWP). In CMWP, an auditory word fragment (prime) is immediately followed by a visual word or pseudoword (target). Experiment 1 investigated ERPs for targets presented in this paradigm. Half of the targets were congruent with the prime (e.g., in the prime-target pair: AM-AMBOSS [anvil]), half were not (e.g., AM-PENSUM [pensum]). Lexical entries of the congruent targets should receive activation from the prime. Thus, lexical identification of these targets should be facilitated. An ERP effect named P350, two frontal negative ERP deflections, and the N400 were sensitive to prime-target congruency. In Experiment 2, the relation of the formerly observed ERP effects to processes in a modality-independent mental lexicon was investigated by presenting primes visually. Only the P350 effect could be replicated across different fragment lengths. Therefore, the P350 is discussed as a correlate of lexical identification in a modality-independent mental lexicon.

Numerous studies investigated physiological correlates of the processing of musical information in adults. How these correlates develop during childhood is poorly understood. In the present study, we measured event-related electric brain potentials elicited in 5and 9-year-old children while they listened to (major–minor tonal) music. Stimuli were chord sequences, infrequently containing harmonically inappropriate chords. Our results demonstrate that the degree of (in) appropriateness of the chords modified the brain responses in both groups according to music-theoretical principles. This suggests that already 5-year-old children process music according to a well-established cognitive representation of the major–minor tonal system and according to music-syntactic regularities. Moreover, we show that, in contrast to adults, an early negative brain response was left predominant in boys, whereas it was bilateral in girls, indicating a gender difference in children processing music, and revealing that children process music with a hemispheric weighting different from that of adults. Because children process, in contrast to adults, music in the same hemispheres as they process language, results indicate that children process music and language more similarly than adults. This finding might support the notion of a common origin of music and language in the human brain, and concurs with findings that demonstrate the importance of musical features of speech for the acquisition of language.

This series of three event-related potential experiments explored the issue of whether the underlying mechanism of working memory (WM) supporting language processing is inhibitory or activational in nature. These different cognitive mechanisms have been proposed to explain the more efficient processing of subjects with a high WM span compared to those with a low WM span. Participants with high and low WM span were presented with sentences containing a homonym followed three words later by a nominal disambiguation cue and a final disambiguation using a verb. At the position of the disambiguation cue, inhibitory or activational WM mechanisms predict contrasting results. When activation is the underlying mechanism for efficient processing, the prediction is that high memory span persons activate both meanings of the homonym equally in WM, whereas low memory span persons only have one meaning present. When inhibition is the underlying mechanism, the predictions are the reverse. The ERP data, in particular, the variations of the meaning related N400 component, showed clear evidence for inhibition as the underlying cognitive mechanism in high-span readers. For low-span participants the cueing towards the dominant or the subordinate meaning elicited an equivalently large N400 component suggesting that both meanings are active in WM. In highspan subjects, the dominant disambiguation cue elicited a smaller N400 than the subordinate one, indicating that for these subjects particularly the dominant meaning is active. The experiments showed that inhibitory processes are probably underlying WM used during language comprehension in high-span subjects. Moreover, they demonstrate that these subjects can use their inhibition in a more flexible manner than low-span subjects. The effects that these processing differences have on the efficiency of language parsing are discussed.

This experiment explored the effect of semantic expectancy on the processing of grammatical gender, and vice versa, in German using event-related-potentials (ERPs). Subjects were presented with correct sentences and sentences containing an article-noun gender agreement violation. The cloze probability of the nouns was either high or low. ERPs were measured on the nouns. The low-cloze nouns evoked a larger N400 than the high-cloze nouns. Gender violations elicited a left-anterior negativity (LAN, 300-600 msec) for all nouns. An additional P600 component was found only in high-cloze nouns. The N400 was independent of the gender mismatch variable; the LAN was independent of the semantic variable, whereas an interaction of the two variables was found in the P600. This finding indicates that syntactic and semantic processes are autonomous during an early processing stage, whereas these information types interact during a later processing phase.