In order to understand how emotional state influences the listener's physiological response to speech, subjects looked at emotion-evoking pictures while 32-channel EEG evoked responses (ERPs) to an unchanging auditory stimulus (“danny”) were collected. The pictures were selected from the International Affective Picture System database. They were rated by participants and differed in valence (positive, negative, neutral), but not in dominance and arousal. Effects of viewing negative emotion pictures were seen as early as 20 msec ( p = .006). An analysis of the global field power highlighted a time period of interest (30.4–129.0 msec) where the effects of emotion are likely to be the most robust. At the cortical level, the responses differed significantly depending on the valence ratings the subjects provided for the visual stimuli, which divided them into the high valence intensity group and the low valence intensity group. The high valence intensity group exhibited a clear divergent bivalent effect of emotion (ERPs at Cz during viewing neutral pictures subtracted from ERPs during viewing positive or negative pictures) in the time period of interest ( r Φ = .534, p < .01). Moreover, group differences emerged in the pattern of global activation during this time period. Although both groups demonstrated a significant effect of emotion (ANOVA, p = .004 and .006, low valence intensity and high valence intensity, respectively), the high valence intensity group exhibited a much larger effect. Whereas the low valence intensity group exhibited its smaller effect predominantly in frontal areas, the larger effect in the high valence intensity group was found globally, especially in the left temporal areas, with the largest divergent bivalent effects (ANOVA, p < .00001) in high valence intensity subjects around the midline. Thus, divergent bivalent effects were observed between 30 and 130 msec, and were dependent on the subject's subjective state, whereas the effects at 20 msec were evident only for negative emotion, independent of the subject's behavioral responses. Taken together, it appears that emotion can affect auditory function early in the sensory processing stream.

The interaction between auditory and visual speech streams is a seamless and surprisingly effective process. An intriguing example is the “McGurk effect”: The acoustic syllable /ba/ presented simultaneously with a mouth articulating /ga/ is typically heard as /da/ [McGurk, H., & MacDonald, J. Hearing lips and seeing voices. Nature, 264, 746–748, 1976]. Previous studies have demonstrated the interaction of auditory and visual streams at the auditory cortex level, but the importance of these interactions for the qualitative perception change remained unclear because the change could result from interactions at higher processing levels as well. In our electroencephalogram experiment, we combined the McGurk effect with mismatch negativity (MMN), a response that is elicited in the auditory cortex at a latency of 100–250 msec by any above-threshold change in a sequence of repetitive sounds. An “odd-ball” sequence of acoustic stimuli consisting of frequent /va/ syllables (standards) and infrequent /ba/ syllables (deviants) was presented to 11 participants. Deviant stimuli in the unisensory acoustic stimulus sequence elicited a typical MMN, reflecting discrimination of acoustic features in the auditory cortex. When the acoustic stimuli were dubbed onto a video of a mouth constantly articulating /va/, the deviant acoustic /ba/ was heard as /va/ due to the McGurk effect and was indistinguishable from the standards. Importantly, such deviants did not elicit MMN, indicating that the auditory cortex failed to discriminate between the acoustic stimuli. Our findings show that visual stream can qualitatively change the auditory percept at the auditory cortex level, profoundly influencing the auditory cortex mechanisms underlying early sound discrimination.

Neuromagnetic responses were recorded to frequent "standard tones of l000 Hz and to infrequent 1100-Hz "deviant" tones with a 24-channel planar SQUID gradiometer. Stimuli were presented at constant interstimulus intervals (ISIs) ranging from 0.75 to 12 sec. The standards evoked a prominent 100-msec response, N100m, which increased in amplitude with increasing ISI. N100m could be dissociated into two subcomponents with different source areas. The posterior component, N100m 2 , increased when the ISI grew up to 6 sec, whereas the more anterior component, N100m 2 , probably continued its growth beyond the 12-sec ISI. At ISIs from 0.75 to 9 sec, the deviants elicited additionally a mismatch field (MMF). The equivalent sources of both N100m and MMF were at the supra-temporal auditory cortex. We assume that auditory stimuli leave in the auditory system a trace that affects the processing of the subsequent stimuli. The decrement of the N100m amplitude as well as elicitation of MMF can be considered as indirect evidence of active traces. A behavioral estimate of the persistence of the sensory auditory memory was obtained in a separate experiment in which the subject compared, without attending to the stimuli, tones presented at the daerent ISIs. The subjects discriminated the stimuli better than merely by chance at ISIs of 0.75-9 sec. The ISI dependence of the behavioral estimate as well as of N100m 2 and MMF are similar enough to suggest a common underlying mechanism that retains information for a period of about 10 sec.

Event-related potentials (ERPs) to synthetic consonant–vowel syllables were recorded. Infrequent changes in such a syllable elicited a "mismatch negativity" as well as an enhanced N100 component of the ERP even when subjects did not pay attention to the stimuli. Both components are probably generated in the supratemporal auditory cortex suggesting that in these areas there are neural networks that are automatically activated by speech-specific auditory stimulus features such as formant transitions.