Abstract

We recorded intracranial local field potentials from structurally intact human visual cortex during several face processing tasks in a patient before brain surgery. Intracranial local field potentials were measured from subdural electrodes implanted in a right fusiform region with face-sensitive activity and a more medial location in posterior parahippocampal gyrus with house-selective activity. This electrode implantation allowed us to compare neural responses with different facial properties within two adjacent but functionally distinct cortical regions. Five experiments were conducted to determine the temporal dynamics of perceptual (Experiments 1 and 5), emotional (Experiments 2 and 3), and social (Experiment 4) effects on face-specific responses in the right fusiform. Our findings showed an early negative deflection (N200) that primarily reflected category-selective perceptual encoding of facial information, whereas higher order effects of face individuation, emotional expression, and gaze direction produced selective modulations in the same face-specific region during a later time period (from 200 to 1000 msec after onset). These results shed new lights on the time course of face recognition mechanisms in human visual cortex and reveal for the first time anatomically overlapping but temporally distinct influences of identity or emotional/social factors on face processing in right fusiform gyrus, which presumably reflect top–down feedback effects from distant brain areas.

INTRODUCTION

Visual object recognition is a fast, effortless, and efficient perceptual ability (Op de Beeck, Haushofer, & Kanwisher, 2008; Kourtzi & DiCarlo, 2006; Grill-Spector & Kanwisher, 2005; Palmer, 1999; Riesenhuber & Poggio, 1999). It is now well established that ventral occipito-temporal cortex is critically involved in object identification (Grill-Spector, 2003; Milner & Goodale, 1995; Ungerleider & Mishkin, 1982) and orderly organized according to discrete cortical regions with high processing selectivity (Op de Beeck et al., 2008; Grill-Spector, 2003; Lerner, Hendler, Ben-Bashat, Harel, & Malach, 2001; Levy, Hasson, Avidan, Hendler, & Malach, 2001; Grill-Spector et al., 1998). Converging results from human neuropsychology, monkey neurophysiology, and functional imaging (from ERPs or MEG to PET and fMRI) have shown that a region in extrastriate cortex is specifically responsive to face stimuli, corresponding to the so-called fusiform face area (FFA) in humans (see Kanwisher & Yovel, 2006; Tsao, Freiwald, Knutsen, Mandeville, & Tootell, 2003; Halgren et al., 1999; Kanwisher, McDermott, & Chun, 1997; Bentin, Allison, Puce, Perez, & McCarthy, 1996; Puce, Allison, Gore, & McCarthy, 1995; Allison, Ginter, et al., 1994; Gross, 1992), whereas another region is selectively responsive to houses and topographic scene information (corresponding to the parahippocampal place area, PPA; see Epstein, Graham, & Downing, 2003; Epstein, Harris, Stanley, & Kanwisher, 1999; Epstein & Kanwisher, 1998). During the last two decades, the functional mapping of these object-selective regions along the ventral occipito-temporal cortex has been intensively investigated and greatly refined, mainly because of the advancement of fMRI (see Op de Beeck et al., 2008; Grill-Spector, 2003).

By comparison, the exact temporal dynamic of neural activity in these highly selective regions has received less attention. Although the FFA is thought to primarily mediate the perceptual encoding of facial configurations (see Grill-Spector, Knouf, & Kanwisher, 2004; Tong, Nakayama, Moscovitch, Weinrib, & Kanwisher, 2000), many imaging studies have revealed that its activation is influenced by higher order face properties including their emotional value (such as fearful expressions, Vuilleumier, Armony, Driver, & Dolan, 2001; or fear-conditioned associations, Buchel & Dolan, 2000; Morris et al., 1998) as well as their social significance (such as gaze direction; George, Driver, & Dolan, 2001). Although such effects are usually attributed to modulatory top–down influences from higher order brain regions in limbic (Vuilleumier et al., 2001), fronto-parietal (Pourtois, Schwartz, Seghier, Lazeyras, & Vuilleumier, 2006; Beck, Rees, Frith, & Lavie, 2001), cingulate (Gobbini & Haxby, 2007; Druzgal & D'Esposito, 2001), or lateral temporal regions (Puce & Perrett, 2003; Winston, Strange, O'Doherty, & Dolan, 2002; Haxby, Hoffman, & Gobbini, 2000), the time course of cognitive and affective effects on perceptual processes is not fully known. For instance, it remains unclear whether different top–down influences on face processing in fusiform have overlapping or instead distinct time courses (see Vuilleumier, 2005; Sugase, Yamane, Ueno, & Kawano, 1999). It also remains unclear how selectively such influences act on fusiform cortex and neighboring areas. Moreover, to our knowledge, there is no comparable data on the time course of activation to house or scene stimuli in the parahippocampal cortex.

Nonetheless, a few pioneer studies have already provided important clues about the neurophysiological profile of face-specific responses in monkey (Gross, 1992; Rolls, Perrett, Caan, & Wilson, 1982) and human visual cortex (Puce & Perrett, 2003; Seeck et al., 2001; Allison, Ginter, et al., 1994; Halgren et al., 1994). In addition to scalp EEG or MEG recordings in healthy participants (e.g., Carmel & Bentin, 2002; Halgren, Raij, Marinkovic, Jousmaki, & Hari, 2000; Bentin et al., 1996; George, Evans, Fiori, Davidoff, & Renault, 1996), intracranial-evoked potentials have been obtained in patients with medically refractory epilepsy before surgery (see Seeck et al., 2001; Allison, Puce, Spencer, & McCarthy, 1999; McCarthy, Puce, Belger, & Allison, 1999; Halgren, Baudena, Heit, Clarke, Marinkovic, Chauvel, et al., 1994; Halgren, Baudena, Heit, Clarke, Marinkovic, & Clarke, 1994). These studies have clearly shown that pictures of faces evoke an early negative deflection peaking approximately 150–200 msec after stimulus onset (initially labeled “face-specific N200” based on intracranial recordings, see Allison, Puce, & McCarthy, 2002; Allison et al., 1999; Allison, Ginter, et al., 1994; or “N170” based on scalp recordings, see Bentin et al., 1996). The intracranial N200 (usually recorded from several sites in occipito-temporal cortex) and scalp N170 (recorded over lateral occipito-temporal electrodes) are thought to reflect a common neural process necessary for the structural/perceptual encoding of face configurations (see Itier & Taylor, 2004a, 2004b; Carmel & Bentin, 2002), corresponding to the main visual function of the FFA (Kanwisher & Yovel, 2006). However, it remains contentious whether the face-specific N170 ERP component reflects activity of the fusiform or instead more lateral temporal brain regions, including the superior temporal gyrus (see Thierry et al., 2006; Itier & Taylor, 2004a, 2004b; Henson et al., 2003). A series of elegant neurophysiological studies in patients implanted with depth electrodes in occipito-temporal cortex (Allison et al., 1999; McCarthy et al., 1999; Puce, Allison, & McCarthy, 1999) has previously characterized various electrophysiological properties of the face-specific N200 and demonstrated that these responses are larger in the right hemisphere, highly selective to faces relative to other visual object categories, altered by low-level perceptual changes in the image (such as “in-plane” or “in-depth” rotations), but insensitive to higher order influences such as emotional arousal, learning, familiarity, semantic priming, or explicit identification (see Puce et al., 1999; see also Halgren, Baudena, Heit, Clarke, Marinkovic, Chauvel, et al., 1994; Halgren, Baudena, Heit, Clarke, Marinkovic, & Clarke, 1994). In addition, other category-selective N200 components were described not only for whole faces but also for isolated face-parts, hands, objects, and high spatial frequency patterns (see Allison et al., 1999; McCarthy et al., 1999). Besides this N200 component, other category-selective ERP components (including the P150, P290, P350, and N700) have consistently been documented in intracranial studies performed in large groups of epileptic patients (see Allison et al., 1999; McCarthy et al., 1999; Puce et al., 1999; Halgren, Baudena, Heit, Clarke, Marinkovic, Chauvel, et al., 1994; Halgren, Baudena, Heit, Clarke, Marinkovic, & Clarke, 1994). A P350 component was found in isolation not only in the anterior fusiform but also in the mid-to-posterior fusiform sites. By contrast, P150, P290, and N700 components were recorded in mid-to-posterior fusiform gyrus (FG) sites, similarly to face N200s.

The main purpose of our study was to examine the neural dynamics of distinct sources of modulation on fusiform activity in a patient who underwent an unusual implantation of depth electrodes within the right ventral occipito-temporal cortex, covering both the fusiform and the parahippocampal gyri. This allowed us to compare the effects of affective facial cues on fusiform responses, relative to those previously described for the intracranial face-specific N200 (see Allison et al., 1999). In addition, the unique electrode implantation in this patient also provided us with the opportunity to contrast face-selective effects to those observed in a neighboring parahippocampal region with distinct, house-selective responses. We used a variety of experimental manipulations that ranged from low-level perceptual changes (including systematic modifications in geometry and spatial frequency content of faces) to higher order changes in emotional expression and social cues (such as gaze and head direction). We tested the hypothesis that the face-specific N200 is primarily concerned by an early perceptual analysis of facial information (Allison et al., 1999), whereas the modulations by emotion and social signals are exerted on the same face-specific responses but at later and partly overlapping periods (see Vuilleumier, 2005; Puce et al., 1999; Sugase et al., 1999).

METHODS

Case Description

The patient was a 17-year-old right-handed female who suffered from right parietal epilepsy since the age of 12 years (most likely caused by surgical removal of a cerebral abscess when 4 years old). Her seizures involved motor and sensitive disturbances (“Jacksonian” seizures) in the left (contralateral) limb. Her seizures started by a specific sensitive aura in the left hand, before sensory-motor symptoms spread to the left arm, and eventually bilaterally to the upper part of the body. Seizure frequency was variable over time (0–10 episodes per month). She never lost consciousness during seizures, although she encountered difficulties in breathing and in articulating because of motor symptoms. During postictal periods, she presented with a transient weakening of the left upper limb. All other sensory modalities (vision, audition, olfaction, and taste) were not affected by her disease, as confirmed by extensive clinical neurology exams. Although she was treated by several regular antiepiletic drugs, her seizures had become pharmacoresistant (refractory) over the years.

Detailed neuropsychological assessment showed no cognitive impairment (i.e., preserved vision, language, memory, and attention functions; normal IQ). Detailed MRI scans confirmed a lesion circumscribed to the right lateral and anterior parietal cortex (postcentral gyrus), whereas the occipital and the temporal lobes were structurally intact. Functional MRI and EEG-triggered MRI also confirmed right parietal seizures in the vicinity of the somatosensory cortex. Clinical EEG exams also disclosed slower spontaneous EEG activity over the right temporo-parietal cortex with characteristic interictal waveforms.

The patient was implanted with intracranial electrodes with the aim to better localize the epileptic focus before surgical treatment. A 64-channel grid was placed over the lateral surface of the right hemisphere, covering most of the right frontal and right parietal cortex. Eight additional subdural electrodes were also placed in the right anterior inferotemporal cortex because of clinical suspicion of a secondary epileptic focus in this region. In our study, we focused on the electrophysiological recordings from these subdural temporal electrodes (Figure 1), which encompassed the right FG and right posterior parahippocampal gyrus (PHG). These two inferotemporal regions were structurally intact in this patient, unlike the right lateral frontal and parietal cortex that showed structural anomalies responsible for the seizures (e.g., lesions close to the central sulcus). Hence, iLFPs from the right lateral frontal and parietal cortex were found to be contaminated by spiking activities, precluding the possibility to compute reliable ERP waveforms. No evidence of face-selective ERPs could be observed in these frontal and parietal sites. By contrast, spike-free iLFPs were recorded from the additional subdural electrodes placed in the right anterior inferotemporal cortex, with the exception of one electrode out of eight (i.e., the Electrode 3, located between the FG and PHG electrode, see Figure 1), where excessive EEG noise also prevented to compute reliable ERP components.

Figure 1. 

Electrode positions. (A) Coronal section through three-dimensional reconstruction from a T1-weighted MR, showing the location of the eight subdural electrodes implanted along the right occipito-temporal cortex. iLFPs recorded from the third electrode position (located between the FG and the PHG electrode) were contaminated by excessive EEG noise, and we therefore could not compute reliable ERP components for this electrode position. (B, C) Horizontal section showing the location of the electrode implanted in the right PHG (Talairach coordinates: +10x, −42y, −20z) and in the right FG (Talairach coordinates: +25x, −45y, −29z).

Figure 1. 

Electrode positions. (A) Coronal section through three-dimensional reconstruction from a T1-weighted MR, showing the location of the eight subdural electrodes implanted along the right occipito-temporal cortex. iLFPs recorded from the third electrode position (located between the FG and the PHG electrode) were contaminated by excessive EEG noise, and we therefore could not compute reliable ERP components for this electrode position. (B, C) Horizontal section showing the location of the electrode implanted in the right PHG (Talairach coordinates: +10x, −42y, −20z) and in the right FG (Talairach coordinates: +25x, −45y, −29z).

During clinical assessment, direct electrical stimulations of the subdural electrodes over temporal cortex were found to produce visual hallucinations, confirming an involvement of these areas in higher order visual perception (see also Puce et al., 1999). Stimulations were applied using a grass stimulator (300-μsec pulse at 50 Hz during 2 sec; intensity of 8 mA). Hallucinations were produced during direct stimulations for the five most medial electrode positions along the strip (Figure 1A). The patient reported a feeling of déjà vu, consisting of a (nonlateralized) colored still landscape superimposed on a transparent paper sheet. This hallucination persisted for a few seconds after the stimulations, and its content was not differentiated as a function of the five electrode positions along the strip where direct stimulations reliably evoked hallucinations. Because, these stimulations were performed for clinical purposes by the presurgery team, no systematic investigation nor further analysis of their experiential content was available.

At the time of our testing, the patient was free of any medication, according to a standard weaning protocol during presurgical evaluation. As recommended by the medical staff of the presurgical epilepsy evaluation unit (Geneva University Hospital), our neuropsychological testing could only take place if the patient had not experienced any clinical or subclinical seizure for several hours (typically half a day) before testing. No seizure was observed during our recordings or between recording sessions over the 4-day period of our investigations. As a result, any influence of ictal or postictal epilepsy activity (e.g., secondary generalization and cognitive impairments) was highly unlikely to have contaminated or biased our intracranial ERP results (see also Data analyses section).

General Procedure

We used similar settings for all five experiments described here. In our different tasks, the patient was asked either to detect predefined visual objects (Experiment 1) or to discriminate the gender or the identity of individual stimuli (Experiments 2–5) and made responses by pressing predefined keys on a dedicated button box using her right dominant hand. In all experiments, visual stimuli were shown on a computer screen at a viewing distance of about 1 meter, with its center roughly aligned at the eyes level. Pictures were presented against a homogenous dark background and had a similar size across all experiments [400 (width) × 540 (height) pixels on a 1024 × 768 resolution screen]. Pictures covered approximately 8° × 11° of visual angle. Average luminance of stimuli was approximately 25 cd/m2.

A common procedure of presentation was adopted across all experiments, with each stimulus shown centrally for a duration of 400 msec, followed by a constant interstimulus-interval (3950 msec in Experiments 2 and 3; 2950 msec in all others). Each stimulus was preceded by a fixation cross for 450 msec to signal the next trial. The different stimulus conditions were always presented in a fully random order. Stimulus presentation and response recording were controlled using E-Prime software (V1.1, http://www.pstnet.com/products/e-prime/). Testing took place during four consecutive days, each in a single session of approximately 30 min.

Experiment 1: Effects of Stimulus Type

The goal of Experiment 1 was to determine the presence of reliable electrophysiological responses to different types of visual stimuli (faces vs. houses) and to select electrodes with face-sensitive activation, using functional rather than just anatomical criteria. In addition, we also tested for differential effects related to changes in the configuration (upright vs. inverted stimuli) and identity content of faces (schematic vs. realistic stimuli).

Five different object categories (realistic houses, realistic upright faces, realistic inverted faces, schematic upright faces, and schematic inverted faces) were shown in a mixed, randomized order. Inverted faces were created by rotating the internal features of face stimuli (realistic and schematic) in such a way to break down the first-order relations that define face stimuli (Maurer, Grand, & Mondloch, 2002; Searcy & Bartlett, 1996; Rhodes, Brake, & Atkinson, 1993; see Figure 4). By doing so, we could determine selective modulations produced by the configural processing of facial geometry (i.e., face detection). Schematic faces were standard geometrical stimuli of three dots arranged in a facelike format (Figure 4) as previously used in developmental studies (Johnson, 2005), whereas realistic stimuli were digitalized pictures of real faces (and houses) used in previous behavioral studies (de Gelder & Rouw, 2001). By comparing schematic versus realistic faces, we could also determine any differential inversion effect for these two types of stimuli as well as any modulation by the absence versus presence of identity information (Maurer et al., 2002). On the basis of previous developmental and neuropsychological findings, we predicted earlier effects for mechanisms of face detection as compared with more elaborate mechanisms of face identification in the right FG (Johnson, 2005; Maurer et al., 2002).

Each of the five object categories (house and four face conditions) consisted of five different items, each repeated 10 times (distributed in two successive blocks separated by a brief pause). The patient was instructed to press a button to each occurrence of a house during the stimulus sequence (but note that similar visual cortical responses were obtained when houses were not targets in Experiment 2, indicating that this task-related factor did not confound the critical visual ERPs; see below). Performance was flawless, except for a single false alarm (mean RT for correct house detections = 548 msec).

Experiment 2: Effects of Visual Object Category versus Emotional Facial Expression

This experiment addressed a key question of our study, concerning the specificity and the time course of emotional modulations on face processing in visual cortex. On the basis of abundant evidence for increased activation to threat-related stimuli (Sabatinelli, Bradley, Fitzsimmons, & Lang, 2005; Vuilleumier et al., 2001; Morris et al., 1998), we compared iLFPs evoked by fearful and neutral faces in the electrode site showing face-sensitive responses.

Three different categories of visual stimuli were shown (photographs of houses, fearful, and neutral faces). All faces were full-frontal views, taken from the standard Karolinska face database (Lundqvist, Flykt, & Ohman, 1998); houses were those previously used in several brain-imaging studies (Vuilleumier, Richardson, Armony, Driver, & Dolan, 2004; Vuilleumier et al., 2001; Wojciulik, Kanwisher, & Driver, 1998). There were 130 trials, including 50 houses and 80 faces, half with a fearful expression and the other half with a neutral expression (10 different houses and 10 different faces, each repeated five and four times, respectively). The patient was instructed to report any immediate repetition of the same picture (one-back repetition task, with 10 targets for each stimulus category, total 20/130 trials). Therefore, all visual object categories were similarly task relevant (unlike Experiment 1 where houses were to-be-reported targets), allowing us to further compare category-selective responses to faces versus houses without any task confound (in addition to our critical comparison between fearful vs. neutral faces).

The patient correctly detected 14 of 20 immediate repetitions (accuracy = 70%; four misses for houses and two misses for faces), whereas she made a single false alarm (on a house stimulus). Mean RT for correct detections was 906 msec. Our analysis of iLFPs was performed only on trials corresponding to (correct) nonrepeated stimuli (uncontaminated by any motor-related activity).

Experiment 3: Effects of Fearful versus Happy Facial Expression

Because we used only fearful faces in Experiment 2, we could not determine whether any emotional effect was specific to threat and negative valence or more generally related to arousal and possibly elicited by positive stimuli as well (see Sabatinelli et al., 2005; Lang et al., 1998). We therefore run a new experiment in which we presented neutral, fearful, and happy faces. We presented face stimuli from the Ekman and Friesen (1976) set instead (including 10 different identities, shown with three possible expressions, repeated four times each). In total, 120 faces were presented in randomized order (40 for each emotion condition) during a single test session. The patient was instructed to perform a gender discrimination task on each and every face stimulus. We used a different task (gender discrimination instead of one-back repetition task), allowing us to further establish the generalization of face-sensitive ERPs recorded from FG and emotional modulations across different processing demands.

The patient performed correctly on 87% of trials, with roughly similar error rates across the three expression conditions (10%, 15%, and 12.5% for fearful, happy, and neutral faces, respectively), and similar RTs for correct gender judgments (566, 551, and 569, respectively).

Experiment 4: Effects Gaze and Head Direction

A second key question of our study, addressed in Experiment 4, concerned the modulation of face processing by nonemotional social signals, such as eye gaze and/or head orientation (Calder et al., 2007; Langton, Watt, & Bruce, 2000). Behavioral studies indicate that such factors provide important perceptual cues that can influence attention (Driver & Vuilleumier, 2001) and memory for faces (Vuilleumier, George, Lister, Armony, & Driver, 2005; Macrae, Hood, Milne, Rowe, & Mason, 2002) and interact with the perceived affective relevance of facial expressions (Adams, Gordon, Baird, Ambady, & Kleck, 2003). Similarly, brain-imaging results show greater activation of fusiform cortex and amygdala as a function of eye gaze direction in faces during fMRI (Pourtois, Schwartz, Seghier, Lazeyras, & Vuilleumier, 2005a; George et al., 2001; Kawashima et al., 1999) as well as increases of the N170 component in scalp EEG (Conty, N'Diaye, Tijus, & George, 2007; Puce & Perrett, 2003; Taylor, Itier, Allison, & Edmonds, 2001). Because such modulation of cortical responses to eye gaze might also be driven partly by top–down signals from amygdala or gaze-sensitive regions in STS (Puce & Perrett, 2003; George et al., 2001; Hoffman & Haxby, 2000), here we tested how perceived gaze direction might influence intracranial potentials elicited by faces in FG. In addition, because head orientation (front view vs. three quarter) may not only influence perceived gaze direction (Langton et al., 2000) but also modulate fusiform activation to neutral faces (Gilaie-Dotan & Malach, 2007; Kanwisher & Yovel, 2006; Leopold, Bondar, & Giese, 2006; Pourtois et al., 2005a; Pourtois, Schwartz, Seghier, Lazeyras, & Vuilleumier, 2005b), we compared responses to direct versus averted gaze in faces presented with either full-front or three-quarter orientations.

Face stimuli were selected from a set previously used in several brain-imaging studies (Pourtois et al., 2005a; Vuilleumier et al., 2005; George et al., 2001). There were four experimental conditions, with a 2 (head direction) × 2 (gaze direction) factorial design: Head orientation was either straight or rotated (∼30°) toward the left or right side (with an equal probability of both directions), whereas gaze direction was either straight or horizontally averted (∼30° toward the left or right side, with an equal probability). Faces included 65 different identities, each presented once in each of the four conditions in randomized order (260 trials in total, in two successive blocks with the same proportion of stimulus conditions).

The patient performed a gender discrimination task on each face. She was highly accurate in all four stimulus conditions (straight gaze–straight head, 94% correct; straight gaze–averted head, 89%; averted gaze–straight head, 88%; and averted gaze–averted head, 91%). Mean RTs on correct responses were also similar across conditions (direct gaze–straight head, 680 msec; direct gaze–deviated head, 633 msec; averted gaze–straight head, 672 msec; and averted gaze–deviated head, 676 msec).

Experiment 5: Effects of Spatial Frequency Content

Another fundamental physical attribute of visual stimuli is their spatial frequency content. Behavioral and brain-imaging results have demonstrated important differences in the processing of low versus high spatial frequencies (LSF vs. HSF) during face recognition, with an overall advantage of LSF over HSF for the extraction of configural face cues (Rotshtein, Vuilleumier, Winston, Driver, & Dolan, 2007; Goffaux, Gauthier, & Rossion, 2003; Nasanen, 1999; Schyns & Oliva, 1999) and for the recognition of some emotional expressions (Schyns, Petro, & Smith, 2007; Pourtois, Dan, Grandjean, Sander, & Vuilleumier, 2005; Vuilleumier, Armony, Driver, & Dolan, 2003). Moreover, ERPs and fMRI results in healthy subjects indicate that face-selective visual areas (including fusiform cortex) are modulated by spatial frequency information in both neutral faces (Rotshtein et al., 2007; Kanwisher & Yovel, 2006; Tanskanen, Nasanen, Montez, Paallysaho, & Hari, 2005; Eger, Schyns, & Kleinschmidt, 2004; Tong et al., 2000; George et al., 1999) and emotional faces (Pourtois et al., 2005; Vuilleumier et al., 2003; Winston, Vuilleumier, & Dolan, 2003; for evidence with visual scenes, see also Carretie, Hinojosa, Lopez-Martin, & Tapia, 2007).

In this experiment, the patients were presented with pictures of faces (Pourtois et al., 2005; Vuilleumier et al., 2003) that were either intact (broadband) or filtered to preserve the LSF (cutoff <6 cycles/image) or HSF content (cutoff >24 cycles/image; see Figure 5). In addition, each face stimulus could express either a neutral or a fearful emotional expression to test whether any effects of emotional expression on intracranial ERPs would be modulated by the spatial frequency content of faces (see Pourtois et al., 2005; Vuilleumier et al., 2003; Winston et al., 2003). For each condition (3 spatial frequency contents × 2 emotional expressions), 40 individual face stimuli were presented (in two successive blocks of 120 trials each, separated by a brief pause).

We used the same procedure as described in Experiments 3 and 4 (gender discrimination task). This task was chosen as it does not induce strong perceptual biases (or strategies) toward a specific spatial frequency range (e.g., low or high; see Winston et al., 2003; Schyns & Oliva, 1999). The patient's performance was generally accurate but slightly worse for HSF (62% correct, mean RT = 536 msec) as compared with LSF (76% correct, mean RT = 514 msec) or broadband faces (78% correct, 496 msec) but showed no systematic difference between emotion expressions across the three stimulus conditions (overall mean correct 73% for fearful faces and 71% for neutral faces, mean RT = 514 and 526 msec, respectively).

EEG Data Recording

iLFPs were continuously recorded (Ceegraph XL, Biologic System Corp, Mundelein, IL) with a sampling rate of 256 Hz (band-pass filter of 0.1–200 Hz) using eight depth electrodes (AD-Tech, electrode diameter = 6 mm, interelectrode spacing = 10 mm). The reference electrode was located at position Cz and the ground at position FCZ in the 10–20 international EEG system. Intracranial visual-evoked potentials were obtained by averaging local field potentials time locked to stimulus onset for each stimulus category separately. Individual epochs were low-pass filtered using a 30-Hz cutoff. A standard 20-Hz low-pass filter was further applied on the average ERP responses computed for each experimental condition in each experiment (depicted in Figures 2,345–6).

Figure 2. 

Focal category-selective effects. (A) Coronal section through three-dimensional reconstruction from a T1-weighted MR, showing the location of the eight subdural electrodes implanted along the right occipito-temporal cortex. The location of the FG electrode is encircled with black color, whereas two other adjacent electrodes are also encircled with dark and light gray. Note that iLFPs recorded from the third electrode position (located between the FG and PHG) were contaminated by excessive EEG noise and could not be used to compute reliable ERP components. (B) Intracranial ERPs recorded from FG and two adjacent electrodes during Experiment 1. The dark waveform shows the average response to realistic faces in the FG; the two gray waveforms show the average response to the same face stimuli for a more medial (dark gray) and a more lateral (light gray) electrode location. These results show highly circumscribed face-selective responses for the electrode implanted over the right FG. (C) Coronal section showing the location of the PHG electrode (encircled with black color) and two adjacent electrodes (dark and light gray). (D) Intracranial ERPs recorded from PHG and two adjacent electrodes during Experiment 1. The dark waveform shows the average response to houses in the PHG; the two gray waveforms show the average response to the same house stimuli in a more medial (dark gray) and a more lateral (light gray) electrode location. These results also indicate a clear peak of house-selective responses for the electrode over the right PHG.

Figure 2. 

Focal category-selective effects. (A) Coronal section through three-dimensional reconstruction from a T1-weighted MR, showing the location of the eight subdural electrodes implanted along the right occipito-temporal cortex. The location of the FG electrode is encircled with black color, whereas two other adjacent electrodes are also encircled with dark and light gray. Note that iLFPs recorded from the third electrode position (located between the FG and PHG) were contaminated by excessive EEG noise and could not be used to compute reliable ERP components. (B) Intracranial ERPs recorded from FG and two adjacent electrodes during Experiment 1. The dark waveform shows the average response to realistic faces in the FG; the two gray waveforms show the average response to the same face stimuli for a more medial (dark gray) and a more lateral (light gray) electrode location. These results show highly circumscribed face-selective responses for the electrode implanted over the right FG. (C) Coronal section showing the location of the PHG electrode (encircled with black color) and two adjacent electrodes (dark and light gray). (D) Intracranial ERPs recorded from PHG and two adjacent electrodes during Experiment 1. The dark waveform shows the average response to houses in the PHG; the two gray waveforms show the average response to the same house stimuli in a more medial (dark gray) and a more lateral (light gray) electrode location. These results also indicate a clear peak of house-selective responses for the electrode over the right PHG.

Figure 3. 

Category-selective responses. (A) Intracranial ERPs recorded from FG and (B) from PHG during Experiment 1. The black waveform shows the average response to faces; the gray-level waveform shows the average response to houses. Results show an early negativity (N200-like) larger to faces than houses in FG, but larger to houses than faces in PHG. (C) Intracranial ERPs from FG and (D) from PHG during Experiment 2, replicating the face- and house-selective results of Experiment 1, with a different stimulus set.

Figure 3. 

Category-selective responses. (A) Intracranial ERPs recorded from FG and (B) from PHG during Experiment 1. The black waveform shows the average response to faces; the gray-level waveform shows the average response to houses. Results show an early negativity (N200-like) larger to faces than houses in FG, but larger to houses than faces in PHG. (C) Intracranial ERPs from FG and (D) from PHG during Experiment 2, replicating the face- and house-selective results of Experiment 1, with a different stimulus set.

Figure 4. 

Face inversion effects (Experiment 1). (A) Intracranial ERPs recorded from FG, showing the average response to upright (black waveform) and inverted (red waveform) pictures of realistic faces. A significant inversion effect started after the face-specific N200 (at 190 msec). (B) ERP image (two-dimensional representation of Amplitude × Time data for all individual trials, see Delorme & Makeig, 2004) computed for the same conditions as in panel A. The amplitude of single-trial responses is plotted from −100 μV (blue) to +100 μV (red). Results show that the face inversion effect was highly reproducible across trials. (C) Intracranial ERPs from FG for upright (black waveform) and inverted (red waveform) pictures of schematic faces. A significant face inversion effect modulated the face-specific N200 (starting at 125 msec), with a larger amplitude to upright than inverted stimuli. (D) ERP image computed for the same conditions as in C.

Figure 4. 

Face inversion effects (Experiment 1). (A) Intracranial ERPs recorded from FG, showing the average response to upright (black waveform) and inverted (red waveform) pictures of realistic faces. A significant inversion effect started after the face-specific N200 (at 190 msec). (B) ERP image (two-dimensional representation of Amplitude × Time data for all individual trials, see Delorme & Makeig, 2004) computed for the same conditions as in panel A. The amplitude of single-trial responses is plotted from −100 μV (blue) to +100 μV (red). Results show that the face inversion effect was highly reproducible across trials. (C) Intracranial ERPs from FG for upright (black waveform) and inverted (red waveform) pictures of schematic faces. A significant face inversion effect modulated the face-specific N200 (starting at 125 msec), with a larger amplitude to upright than inverted stimuli. (D) ERP image computed for the same conditions as in C.

Figure 5. 

Emotional responses in Experiments 2 and 3. (A) Intracranial ERPs recorded from FG during Experiment 2, showing the average response to neutral faces (black waveform) and fearful faces (red waveform). A reliable increase to fearful faces arose after the face-specific N200, in the form of a sustained negative component. (B) Intracranial ERPs recorded from FG during Experiment 3, showing the average response to faces with neutral (black waveform), fearful (red waveform), and happy (blue waveform) expressions. Both fearful and happy faces elicited a larger negative and sustained response following the face-specific N200, relative to neutral faces. (C) ERP image computed for all single trials with neutral faces (Experiment 2), and (D) fearful faces (Experiment 2), showing a highly reliable enhancement of the late sustained negative response to fearful faces.

Figure 5. 

Emotional responses in Experiments 2 and 3. (A) Intracranial ERPs recorded from FG during Experiment 2, showing the average response to neutral faces (black waveform) and fearful faces (red waveform). A reliable increase to fearful faces arose after the face-specific N200, in the form of a sustained negative component. (B) Intracranial ERPs recorded from FG during Experiment 3, showing the average response to faces with neutral (black waveform), fearful (red waveform), and happy (blue waveform) expressions. Both fearful and happy faces elicited a larger negative and sustained response following the face-specific N200, relative to neutral faces. (C) ERP image computed for all single trials with neutral faces (Experiment 2), and (D) fearful faces (Experiment 2), showing a highly reliable enhancement of the late sustained negative response to fearful faces.

Figure 6. 

Effects of eye gaze (Experiment 4) and spatial frequency content (Experiment 5). (A) Intracranial ERPs recorded from FG showing the average response to all faces with direct gaze (black waveform) relative to all faces with averted gaze (red waveform), irrespective of head direction. Results showed a sustained negativity for faces with averted gaze, clearly starting after the face-specific N200 component. (B) Intracranial ERPs from FG showing the average response to broadband, HSF-filtered, and LSF-filtered faces (black, blue, and red waveforms, respectively) collapsed across facial expressions. Results show reliable modulations of both the amplitude and the latency of the face-specific N200 component as a function of spatial frequency content (see also Table 1). (C) ERP images were computed for all successive trials in the same three conditions with broadband faces (top panel), HSF-filtered (middle panel), and LSF-filtered faces (bottom panel), showing that the N200 component was systematically delayed and enhanced for HSF compared with broadband faces and enhanced with no onset delay for LSF compared with broadband faces (see Table 1).

Figure 6. 

Effects of eye gaze (Experiment 4) and spatial frequency content (Experiment 5). (A) Intracranial ERPs recorded from FG showing the average response to all faces with direct gaze (black waveform) relative to all faces with averted gaze (red waveform), irrespective of head direction. Results showed a sustained negativity for faces with averted gaze, clearly starting after the face-specific N200 component. (B) Intracranial ERPs from FG showing the average response to broadband, HSF-filtered, and LSF-filtered faces (black, blue, and red waveforms, respectively) collapsed across facial expressions. Results show reliable modulations of both the amplitude and the latency of the face-specific N200 component as a function of spatial frequency content (see also Table 1). (C) ERP images were computed for all successive trials in the same three conditions with broadband faces (top panel), HSF-filtered (middle panel), and LSF-filtered faces (bottom panel), showing that the N200 component was systematically delayed and enhanced for HSF compared with broadband faces and enhanced with no onset delay for LSF compared with broadband faces (see Table 1).

Electrode positions were determined by a brain CT scan performed after implantation and coregistered using SPM5 (http://www.fil.ion.ucl.ac.uk/spm/) to the patient's brain anatomy as obtained with MRI, which in turn was normalized to define Talairach coordinates of each of the recorded electrodes.

Data Analyses

A similar approach was systematically used across all experiments. Single-trial EEG epochs (−100/+1000 msec around the stimulus onset) were analyzed off-line, after removing all epochs where noise or possible epileptic spikes might have spread and contaminated the recorded sites (∼25%, on average, using stringent selection). The amplitude variance computed for each time point across spike-free trials was then used as dependent variable for statistical comparisons. To verify that visual responses were stable over time and across trials, we also computed an Amplitude × Time image (Delorme & Makeig, 2004) for all consecutive presentations of the stimuli (see Figures 4,5–6).

Because the error rate was very low and the task demands usually orthogonal to the main experimental dimensions of interest, all trials (correct and incorrect responses) were included in the EEG averages to increase the signal-to-noise ratio of stimulus-driven visual-evoked potentials (as well as to provide sufficient data points for nonparametric statistical analyses). In Experiment 2, only the (frequent) nonrepeated stimulus presentations were used to compute the averages. Noteworthy, during preprocessing, we did not find any systematic association of errors with epileptic spiking activity (in distant sites), indicating that errors were not triggered by or correlated with spiking events. Furthermore, the inclusion of residual errors did not alter the expression of our intracranial ERP data, as verified by reaveraging the data with correct trials only (these new reanalyses led to comparable results for both the N200 and later ERP components) and by carefully inspecting the single-trial decompositions (ERP images; see Delorme & Makeig, 2004). These single-trial decompositions (see Figures 4,5–6) confirmed that occasional errors did not alter the expression of the N200 or later ERP components.

The presence of significant differences between experimental conditions was determined by nonparametric statistical analyses on the basis of stringent randomization tests (see Pourtois, Peelen, Spinelli, Seeck, & Vuilleumier, 2007; Manly, 1991). Randomization provides a robust nonparametric statistical method without any assumption regarding data distribution, which compares the observed data set with random shuffling of the same values over many iterations (i.e., permutations). The method runs by repeating the shuffling many times (minimum of 5000 with the randomization tests used here) so as to be able to estimate the probability (here p < .05) that the data might be observed by chance. The significant alpha cutoff was set to p < .05, with an additional criterion of temporal stability for at least three consecutive time points (>11 msec at 256-Hz sampling rate). These statistical analyses were performed using Cartool software developed by Denis Brunet (http://brainmapping.unige.ch/Cartool.php).

RESULTS

Experiment 1: Effects of Stimulus Type

Experiment 1 allowed us to identify two electrodes with opposite category preferences for pictures of faces (schematic vs. real, upright vs. inverted) and pictures of houses (see Figure 3). Two important results were obtained.

First, we found a unique pattern of visual ERPs in electrodes implanted in ventral temporal cortex, with a striking dissociation between a lateral and a more medial electrode (see Figure 1). Realistic faces elicited a robust negative component following stimulus onset (relative to realistic houses) specifically at the lateral electrode, whereas the opposite was found for the more medial electrode, where houses (but not faces) elicited a striking negativity with a similar latency (Figure 3A and B). A direct statistical comparison (based on randomization tests, p < .05) confirmed a significantly larger and earlier peak amplitude of this negative activity to faces than to houses for the lateral electrode (−76.86 μV/144 msec vs. −60.48 μV/164 msec for peak amplitude and peak latency, respectively). Conversely, houses elicited a significantly larger negative deflection than faces for the more medial electrode (−60.28 μV/160 msec vs. −2.20 μV/128 msec for peak amplitude and peak latency, respectively). Similar electrophysiological characteristics were found when houses were not targets (see Results of Experiment 2).

Careful inspection of the anatomical location of electrodes indicated that the lateral electrode was implanted in a region unambiguously corresponding to the FG, whereas the more medial electrode site corresponded to the posterior PHG. Their Talairach coordinates (Talairach & Tournoux, 1988) after coregistration with the brain MRI (see Methods) were 25x, −45y, −29z, 10x for the lateral electrode (fusiform) and 10x, −42y, −20z for the more medial electrode (parahippocampal; Figure 1). As can be seen from Figure 2, the face-specific activity was circumscribed to the FG electrode, whereas conversely, the house-specific activity was clearly peaking at the PHG electrode. On the basis of the results of Experiment 1, we therefore used the face-sensitive electrode location (FG) to explore how other experimental manipulations with face stimuli (see Experiments 2–5) influenced intracranial ERP markers of face processing in this region.

These early category-selective responses shared clear electrophysiological similarities (early latency and negative polarity) with the intracranial N200 to faces previously recorded from human visual cortex (Allison et al., 1999; McCarthy et al., 1999). However, here we found slightly shorter latencies for face-sensitive responses (144 msec) as compared with these earlier studies (∼200 msec) as well as a clear category-related dissociation between two adjacent electrodes (with house-selective responses in the medial parahippocampal cortex that were not described in these former studies). The relative earlier latency of the face-sensitive N200 component in this patient, as compared with previous studies (Allison et al., 1999), might be due to the fact that our patient was a young adult (whereas previous studies typically tested older epileptic patients). In addition, we noted that Allison et al. (1999, 2002) and Allison, Ginter, et al. (1994) found no significant difference in either the latency or the amplitude of this category-selective response between ventral and more lateral occipito-temporal areas across several experiments carried out in various groups of epilepsy surgery patients. On the basis of this evidence, there was therefore no reason to suppose that the earlier category-selective effects observed in our study might be related to differences in the location of intracranial recordings relative to other studies (e.g., medial as opposed to more lateral sites).

Second, we compared visual ERPs recorded at the critical electrode for different types of faces: real versus schematic and upright versus inverted for each of these conditions (see Figure 4). Consistent with our predictions (and previous work, see Johnson, 2005; Maurer et al., 2002; Searcy & Bartlett, 1996; Rhodes et al., 1993), the face-sensitive electrode showed an earlier time course for the inversion effect with schematic faces relative to real faces (Figure 4 and Table 1). Statistical analyses showed that the peak of the N200 component had a larger amplitude for upright schematic faces (−154.39 μV/144 msec) relative to inverted schematic faces (−115.77 μV/144 msec; see Figure 4CD). But for realistic faces, the face inversion effect started later (184-msec poststimulus, see also McCarthy et al., 1999) and did not concern the peak of the early N200 itself (−76.86 μV/144 msec vs. −97.32 μV/144 msec for upright and inverted real faces, respectively). By contrast, following the peak of this early negative component (face-specific N200), inverted realistic faces elicited a sustained negative activity (−48.59 μV/190 msec) relative to normal/upright faces (−3.60 μV/190 msec; see Figure 4AB).

Table 1. 

Amplitude, Latency, and Statistical Comparisons for the Face-specific N200 and Later LFPs in Right FG (Experiments 1–5)

Experiment No.
Condition (Face Type)
Face-specific N200
Later LFPs
Peak Amplitude (μV)
Peak Latency (msec)
Statistical Comparison (for Amplitude)
p
Onset/Duration of Significant Difference (msec)
Experiment 1 Upright schematic −154.39 144 Upright vs. inverted (schematic) .03 ns 
Inverted schematic −115.77 144 
Upright realistic −76.86 144 Upright vs. Inverted (realistic) ns 190/204 
Inverted realistic −97.32 144 
Experiment 2 Fearful −77.28 140 Fearful vs. Neutral ns 316/684 
Neutral −88.07 140 
Experiment 3 Fearful −108.10 164 Fearful vs. Neutral ns 434/220 
Happy −79.86 164 Happy vs. Neutral ns 304/284 
Neutral −106.05 160 Fearful vs. Happy ns 588/28 
Experiment 4 Eye straight—head straight −9.60 148 Eye straight vs. Eye averted— ns 396/604 
Eye straight—head averted −0.60 148 Head straight vs. Head averted ns ns 
Eye averted—head straight −28.28 148 
Eye averted—head averted 3.35 148 
Experiment 5 Broadband −82.18 152 Broadband vs. LSF .001 152/560 
LSF −155.64 184 Broadband vs. HSF <.001 152/848 
HSF −128.83 190 LSF vs. HSF .001 212/140 
Experiment No.
Condition (Face Type)
Face-specific N200
Later LFPs
Peak Amplitude (μV)
Peak Latency (msec)
Statistical Comparison (for Amplitude)
p
Onset/Duration of Significant Difference (msec)
Experiment 1 Upright schematic −154.39 144 Upright vs. inverted (schematic) .03 ns 
Inverted schematic −115.77 144 
Upright realistic −76.86 144 Upright vs. Inverted (realistic) ns 190/204 
Inverted realistic −97.32 144 
Experiment 2 Fearful −77.28 140 Fearful vs. Neutral ns 316/684 
Neutral −88.07 140 
Experiment 3 Fearful −108.10 164 Fearful vs. Neutral ns 434/220 
Happy −79.86 164 Happy vs. Neutral ns 304/284 
Neutral −106.05 160 Fearful vs. Happy ns 588/28 
Experiment 4 Eye straight—head straight −9.60 148 Eye straight vs. Eye averted— ns 396/604 
Eye straight—head averted −0.60 148 Head straight vs. Head averted ns ns 
Eye averted—head straight −28.28 148 
Eye averted—head averted 3.35 148 
Experiment 5 Broadband −82.18 152 Broadband vs. LSF .001 152/560 
LSF −155.64 184 Broadband vs. HSF <.001 152/848 
HSF −128.83 190 LSF vs. HSF .001 212/140 

These results reveal face-sensitive responses in right FG, which are primarily sensitive to the face configuration (as seen with schematic stimuli, without any identity information) during the initial N200 activity (peaking around 144 msec), and become more sensitive to finer visual cues carrying identity information in a subsequent time range (from 190 msec onward), suggesting some temporal hierarchy in the perceptual encoding of facial information (starting with low-level geometry, followed by other higher level features).

Finally, although here we focused on the N200 component, we note the striking similarity in the general ERP morphology (i.e., typical sequence of P150/N200/P290 waveforms, followed by a prolonged N700) between our findings (see Figure 3) and earlier intracranial ERP results obtained in other epileptic patients (see Allison et al., 1999; McCarthy et al., 1999; Puce et al., 1999).

Experiment 2: Effects of Visual Object Category versus Emotional Facial Expression

We next turned to the main issues in our study, concerning the modulation of face processing by emotional and social signals. The first goal of this experiment was to test whether fearful faces elicited distinctive brain responses as compared with neutral faces, and to determine the time course of such effects. A second goal was to confirm the strong category-selective activation found in Experiment 1 (i.e., face-sensitive responses for the lateral electrode in right FG and house-selective responses for the more medial electrode in posterior PHG) using a different task and different stimulus set (see Methods). Because houses were the to-be-detected targets in Experiment 1, there was a potential confound in the latter experiment between the stimulus category and the target/nontarget dimension, which could be ruled out by this follow-up test.

Our first statistical comparison between houses and faces (collapsed across neutral and fearful expressions) clearly replicated the functional dissociation between fusiform and parahippocampal regions (see Figure 3C and D). In right FG, faces elicited a much larger early negative component than houses, with remarkably similar latencies as in Experiment 1 (−82.68 μV/140 msec vs. −26.59 μV/156 msec). Conversely, in posterior parahippocampal cortex, houses elicited two deflections in the same time range (an intracranial N200, −39.56 μV/152 msec, immediately followed by a large positive component, 80.33 μV/226 msec), whereas no such effects were seen for faces (N200, 34.84 μV/160 msec; positive component, 35.49 μV/226 msec; see Figure 3C and D). With respect to the prior results of Experiment 1, Experiment 2 yielded a new and important result showing that the house-specific N200 component generated in the right PHG was not influenced by higher order factors such as task requirement (for similar conclusions about the face-selective N200 component, see also Puce et al., 1999). In addition, whereas face stimuli were often task irrelevant in previous intracranial ERP studies examining visual object category differences (see Allison et al., 1999), in the present experiment, these category-selective responses were obtained when faces and houses were equally task relevant (i.e., one-back repetition detection, irrespective of category).

Hence, the results of Experiments 1 and 2 converge to demonstrate a functional dissociation between the PHG and the FG electrodes, the former showing “category-specific” effects and the latter showing “category-sensitive” effects according to definitional criteria proposed by previous intracranial ERP studies on the N200 (Allison et al., 1999). The N200 responses at the PHG electrode was deemed category specific because their amplitude to the preferred stimuli (houses) was at least twice the amplitude to the nonpreferred stimuli (faces) in both Experiments 1 and 2, whereas the FG electrode was deemed category sensitive as its amplitude was less amplified for the preferred (face) versus nonpreferred (house) stimulus category and appeared less selective when houses were task relevant (Experiment 1) as compared with when the two stimulus categories were equally task relevant (Experiment 2). Thus, these data also underscore that the category-sensitive N200 ERP component may potentially vary depending on the specific task requirements.

Our second statistical comparison concerned the differential effect of fearful relative to neutral faces (Figure 5A, C, and D). For fusiform, the intracranial N200 was found to have the same amplitude for fearful versus neutral faces (−77.28 μV/140 msec vs. −88.07 μV/140 msec; see Table 1), but fearful faces elicited a significantly larger and sustained negative activity as compared with neutral faces during a subsequent phase immediately following the N200 response (−44.12 μV/402 msec vs. −1.60 μV/402 msec; see Figure 5D). This emotional effect started 316 msec after stimulus onset and remained significant for the next 684 msec. Importantly, this emotional effect was circumscribed to the electrode in right FG (with clear face-sensitive responses) and not present (all p > .05) for the other electrode in posterior PHG.

Taken together, these results reveal that emotional signals produced a rather broad and sustained negative activity (∼316- to 1000-msec poststimulus), which clearly arose later than the category-selective activity (N200, peaking ∼140-msec poststimulus), and selectively concerned a face-sensitive region in FG (Figure 1C) but did not spread to an adjacent electrode with distinct category preferences (e.g., house-selective electrode). These findings suggest differential temporal windows for processing category-related and emotion-related information within the same cortical area, consistent with previous findings for face-selective neurons in the monkey (see Vuilleumier, 2005; Sugase et al., 1999).

Experiment 3: Effects of Fearful versus Happy Facial Expression

This experiment tested whether the emotional effect observed in Experiment 2 was specific to fearful faces or could also be evidenced for happy faces (which are also emotionally arousing but have an opposite positive valence as compared with fearful faces). Statistical analyses comparing the three conditions of Experiment 3 replicated our previous results with fearful faces (but with completely different face stimuli and task demands) but also revealed a similar emotional effect in the right FG for happy faces relative to neutral faces (see Figure 5B).

Thus, in the time range of N200, no significant difference was found between neutral, fearful, and happy face expression (see Table 1). In a subsequent phase, however, there was a significantly larger negative and sustained response to happy faces as compared with neutral faces (−69.76 μV/538 msec vs. −14.22 μV/538 msec; see Figure 5B) and likewise to fearful faces as compared with neutral faces (−91.05 μV/586 msec vs. −24.93 μV/586 msec). A direct statistical comparison between happy and fearful faces showed that this sustained negative component had a slightly earlier peak for happy relative to fearful faces (522- vs. 586-msec poststimulus), reflecting a relatively more prolonged effect for fearful relative to happy faces (Figure 5B), whereas we failed to find any reliable amplitude or latency difference between these two emotions during an earlier period (e.g., at the offset of the face-selective intracranial N200 and up to 588-msec poststimulus onset).

Importantly, none of these amplitude effects were observed for the more medial electrode with house-selective responses, suggesting a regionally specific modulation of face processing by emotional expressions.

These results therefore confirm that the arousal value of facial expressions (rather than valence) is primarily responsible for a differential neural responses in the right fusiform cortex, with a prolonged impact during a broad time window from ∼350- to ∼550-msec poststimulus onset and clearly subsequent to the early face-selective N200 activity (peak at ∼160 msec in this experiment).

Experiment 4: Effects Gaze and Head Direction

The next issue addressed in this experiment concerned whether fusiform responses were also influenced by changes in socially significant aspects of faces, related to the head or gaze direction. We systematically varied the orientation (either straight or averted) for the eyes and head of neutral faces, following previous studies of gaze processing that found that these two manipulations produced reliable changes in fusiform activation (Pourtois et al., 2005a; George et al., 2001) and N170 amplitude (Conty et al., 2007; Puce & Perrett, 2003; Taylor et al., 2001).

Compared with the other experiments, we note that the face-specific N200 had a relatively attenuated amplitude (see Table 1) and slightly different shape (see Figure 6A), although a typical and sharp response to faces was again clearly recorded from the lateral electrode in right FG (with no effect for the more medial electrode in posterior PHG). This variability between experiments may potentially be explained by several uncontrolled factors (including slight movements of the intracranial electrodes in the head across experimental sessions, different stimulus contexts or preparatory sets across tasks, as well as variations in internal states, vigilance, or medical condition of the patient, but note that no seizure was recorded over the 4 days of testing). But changes in experimental parameters or simple stimulus characteristics were unlikely to account for this variability. As in previous experiments, face stimuli in Experiment 4 were shown one at a time centrally (duration 400 msec), in randomized order (i.e., variations of gaze or head orientation occurred in a nonpredictive way, whereas the identity of the face stimulus always changed from one trial to the next) and with a constant ISI of 2950 msec (see Methods). Therefore, these experimental parameters made it unlikely that the preceding face stimulus could differentially prime or influence the presentation of the next face stimulus, ultimately resulting in some adaptation of the N200 component (see Conty et al., 2007). Thus, although the N200 component was somewhat different in Experiment 4 relative to the four other experiments (with lower in amplitude, see also Table 1), the exact reasons for this were not fully understood.

More importantly, we first tested for a main effect of head direction (by collapsing across faces with direct or averted eye gaze) but found no significant difference in fusiform activity during either the N200 time range or the later periods (see Table 1). By contrast, when we compared the two gaze directions (now collapsing across changes in head orientation), we found a reliable modulation of fusiform activity that selectively arose in the later periods (Figure 6A) following the N200 component (see also Table 1). Starting 396-msec poststimulus (and lasting up to 1000 msec), faces with averted gaze elicited a larger negative response as compared with faces with direct gaze (−26.72 μV/484 msec vs. 6.87 μV/484 msec; see Figure 6A). On the other hand, the N200 did not differ in these conditions (see Table 1).

Although the main effect of gaze direction identified above allowed us to determine significant differences regardless of head rotation and thus unrelated to a mere change in the visual appearance of faces (Conty, Tijus, Hugueville, Coelho, & George, 2006), we also observed a consistent modulation of the gaze effects by head orientation. Although our statistical analysis (based on permutations) could only provide pairwise comparisons between conditions (see Methods), but not directly test for any interaction effect between two experimental factors (i.e., eye gaze and head orientation), we performed two additional pairwise comparisons to confirm the presence of a significant difference between gaze effects as a function of head orientation. These statistical comparisons between eye gaze conditions revealed that the sustained negative deflection to averted versus direct gaze had an earlier and larger peak for faces with a straight head (−38.13 μV/418 msec vs. 18.48 μV/418 msec) as compared with the differential negativity evoked by averted versus straight gaze for faces with deviated head (−7.07 μV/784 msec vs. 12.82 μV/784 msec). However, the amplitude and the latency of the N200 peak were not statistically different across the four conditions (for details, see Table 1).

Results of Experiment 4 therefore indicate that face processing in right fusiform cortex was influenced by eye gaze direction more than by head orientation (see George et al., 2001), although a straight head position amplified the eye gaze effects (see Conty et al., 2007; Pourtois et al., 2005a), and such effects arose in later processing stages after the early category-selective N200 activity related to perceptual face encoding.

Experiment 5: Effects of Spatial Frequency Content

This final experiment tested if the face-selective N200 in right FG was modulated by the spatial frequency content (LSF, HSF, or broadband) of faces (by pooling the two emotional facial expressions, fearful vs. neutral). We first tested the hypothesis of an advantage of LSF over HSF information for the early perceptual stages of face processing, as reflected by the N200 activity in right fusiform. This hypothesis was confirmed by several pairwise statistical comparisons between the three SF conditions. Both the amplitude and the latency of the N200 was markedly altered as a function of spatial frequency content of faces (see Figure 6B and C and Table 1). Replicating results of Experiment 1, we found that, for the electrode in the right FG, broadband faces elicited an early and sharp negative component (intracranial N200) following stimulus onset (−82.18 μV/152 msec). For LSF faces, this component was significantly enhanced in amplitude (−155.64 μV/184 msec) as compared with broadband faces but had the exact same onset time (129 msec for significant difference relative to baseline in both conditions). By contrast, HSF faces elicited a delayed N200 (Figure 6BC), with a significantly later onset time (140 msec) than broadband or LSF faces (both p < .05), whereas peak amplitude (−128.83 μV/190 msec) was higher than for broadband faces and lower than for LSF faces. None of these amplitude or latency effects were observed for the more medial electrode showing house-selective responses in Experiments 1 and 2.

This pattern therefore broadly accords with previous findings (McCarthy et al., 1999) that the N200 component may exhibit both latency and amplitude modulations as a function of the spatial frequency content of face stimuli. In the study of McCarthy et al. (1999), the N200 showed progressively increasing latencies in the transition from broadband, LSF, and HSF faces. However, in their study, N200 amplitudes to broadband and LSF faces were not different, whereas HSF faces produced smaller N200 components than LSF faces. This latter difference between HSF and LSF was also observed in our case (see Figure 6BC). But here broadband stimuli produced smaller responses than LSF, unlike the results of McCarthy et al. (1999). In any case, these differences all converge to demonstrate that the N200 is sensitive to the spatial frequency content of faces.

Hence, more critically, these results corroborate the assumption that the intracranial N200 may not only be sensitive to configural information conveyed by LSF cues but also be closely related to the perceptual visibility of faces (see also McCarthy et al., 1999). On the one hand, there was an overall advantage of LSF over HSF faces to trigger N200 activity with the same onset latency as broadband faces, consistent with an important role of LSF for encoding face geometry (see Experiment 1 and McCarthy et al., 1999). By contrast, the delayed N200 to HSF faces was accompanied with poorer recognition performance (see Results). On the other hand, a degraded visibility of face stimuli by filtering (either LSF or HSF) led to a prolonged activation of the N200, with higher peak amplitude than for the intact/broadband faces, possibly reflecting longer accumulation time required for “perceptual evidence” (Ploran et al., 2007; Heekeren, Marrett, Bandettini, & Ungerleider, 2004).

Our second manipulation in this experiment concerned the emotional expression of faces to test the hypothesis that emotional responses to faces in fusiform cortex might show a distinctive pattern as a function of spatial frequency content of stimuli (Vuilleumier et al., 2003; Winston et al., 2003). For broadband and HSF faces, we failed to find any significant amplitude difference between fearful and neutral expressions for face-selective responses recorded in the right FG. By contrast, for LSF stimuli, the statistical analysis revealed that following the N200 component (from 308- to 570-msec poststimulus), fearful faces elicited a significantly larger sustained negative activity (−69.86 μV/504 msec) as compared with neutral faces (−31.85 μV/504 msec). However, the N200 component itself (see Table 1) was not modulated by emotional expression of LSF faces (−159.42 and −177.98 μV, for fearful LSF and neutral LSF faces, respectively), consistent with results of Experiments 2 and 3. In other words, we found an effect of emotion selectively for LSF faces in the right FG electrode, consistent with the assumption that the right hemisphere might be highly sensitive to lower spatial frequency information content, and that LSF information may be particularly critical for the perception of some emotional facial expressions (see Vuilleumier et al., 2003).

These results confirm that LSF information may provide important visual cues to face-selective regions in fusiform cortex to differentiate between neutral and fearful emotional expressions (see Carretie et al., 2007; Pourtois et al., 2005; Vuilleumier et al., 2003; Winston et al., 2003). Furthermore, these data further show that emotional effects took place during a later stage of face processing, relative to the early N200 activity related to initial encoding of facial geometry (Experiment 1). This pattern replicates the temporal sequence found in Experiments 2 and 3, suggesting an early emotion-independent category-selective processing followed by subsequent modulation of the same cortical area by emotional or social signals (or higher level identity-related information, see Experiment 1).

DISCUSSION

A number of important new results emerged from this study. (i) Not only did we extend earlier neurophysiological findings on the existence of face-specific N200 activity in the right FG (Allison et al., 1999) but also described for the first time another category-selective response to houses in an adjacent region within the right posterior PHG (see Epstein & Kanwisher, 1998; Kanwisher et al., 1997). (ii) We confirmed that the face-specific N200 responses in FG were primarily related to an early (<150-msec poststimulus onset), perceptual stage of face encoding (Allison et al., 1999), affected by low-level aspects such as geometry (Experiment 1) or spatial frequency (Experiment 5) cues. (iii) By contrast, higher order information such as emotional expression (both negative and positive) or gaze direction modulated the magnitude of neural activity in the same face-selective region but at later latencies, subsequent to the N200 component. The latter results provide the first evidence concerning the time course of affective and social signals during face processing in human visual cortex. Moreover, these data converge with similar results from single-cell recordings in monkey (Matsumoto, Okada, Sugase-Miyamoto, Yamane, & Kawano, 2005; Sugase et al., 1999), showing that neuronal responses may code for global category-related information before finer information about the expression or familiarity of individual face stimuli. This temporal sequence in the processing of different facial cues within the same cortical regions is consistent with the assumption that modulations of the FFA by emotion or eye gaze, previously observed in fMRI studies (George et al., 2001; Vuilleumier et al., 2001; Morris et al., 1998), may reflect feedback or reentrant influences from distant brain regions in mesiotemporal/limbic regions, possibly involving the amygdala (see Vuilleumier et al., 2001), lateral temporal regions such as the STS (see Puce & Perrett, 2003; Winston et al., 2002; George et al., 2001), and/or medial prefrontal and ACC (Gobbini & Haxby, 2007; Druzgal & D'Esposito, 2001). These new findings also converge with previous human intracranial studies that reported visual cortical responses with later latencies than the N200, similarly attributed to feedback effects from upstream brain regions (e.g., N700, see Puce et al., 1999). Below we discuss these new results and their implications in greater details.

Opposite Category-selective Effects in Right Fusiform and Parahippocampal Cortex

We found a striking double dissociation within the same patient at the level of the intracranial N200 component, between two nearby regions in the right FG and right posterior PHG (see Figure 1). Consistent with abundant imaging studies in humans (see Epstein & Kanwisher, 1998; Kanwisher et al., 1997), the right FG responded preferentially to pictures of face relative to house, whereas the opposite was true for the right PHG (see Experiments 1 and 2 and Figure 3), but with a remarkably similar pattern and latency for each of these two category-selective N200 components.

Although highly focal cortical iLFPs with a comparable latency have already been observed for pictures of faces, isolated face parts, hands, objects, or HSF patterns in fusiform cortex (Allison et al., 1999) and pictures of bodies in lateral occipital cortex (Pourtois et al., 2007), to our knowledge, house-selective responses have not previously been reported yet. These findings confirm the existence of specialized regions with category-selective preferences in the human ventral occipito-temporal cortex (Op de Beeck et al., 2008) and add support to the notion of specialized neural populations in PHG coding for houses or places (Epstein et al., 1999; Epstein & Kanwisher, 1998), with a similar early perceptual processing as observed for faces and other visual object categories in more posterior occipito-temporal areas. However, we also observe a reliable N200 component to houses in the FG when houses were task relevant (though this response was delayed and reduced compared with the N200-evoked faces, see results of Experiment 1 and Figure 3A). This appears consistent with studies showing that attention may modulate category-selective responses in fusiform cortex (Furey et al., 2006) and that neurons preferentially responding to faces may be embedded with other neurons responding preferentially to other visual object categories (Tsao et al., 2003).

This early house-selective response in PHG was characterized by a conspicuous and sharp phasic negative deflection, sharing both the morphology and the latency of the intracranial N200 component previously found for faces or other visual objects in fusiform cortex (Allison et al., 1999) and for bodies in lateral occipital cortex (Pourtois et al., 2007). Thus, in our patient, responses to houses in PHG were almost exactly as early (148 msec) as responses to faces in FG (142 msec) and were equally found in two different tasks where houses were targets (Experiment 1) or required the same response as faces (Experiment 2), demonstrating that this house-specific activity was reproducible and unrelated to any decision or motor-related process. Altogether, these results are consistent with the notion that neural activity associated with the N200 reflect a broad family of cortical processes underlying the structural encoding and perceptual categorization of visual objects (Allison, McCarthy, Nobre, Puce, & Belger, 1994).

Our data for the right FG also accord with previous neurophysiological results obtained in human epileptic patients implanted with depth electrodes in the ventral and lateral occipito-temporal cortex (see Allison et al., 1999; Allison, Ginter, et al., 1994). The electrophysiological properties of this visual-evoked response to faces (early latency, large amplitude, negative polarity, and monophasic activity) correspond well with the so-called “face-specific N200” (Allison et al., 1999), considered as the intracranial counterpart of the N170 in scalp EEG (Seeck et al., 2001; Bentin et al., 1996; Botzel & Grusser, 1989). We note however that, in our patient, the latency of this face-sensitive response (∼140- to 160-msec poststimulus onset) was slightly earlier than that of the N200 described by McCarthy et al. (1999), but the latter effects were surprisingly late, and our results seem more consistent with the typical N170 latency recorded in scalp EEG (Carmel & Bentin, 2002). A relatively delayed response in previous intracranial studies has been imputed to possible differences associated with the epileptic disease or drug treatment (Puce et al., 1999; Bentin et al., 1996). Another possibility for this faster latency in our patient might be that she was younger than epileptic patients investigated in these previous studies.

Importantly, the magnitude and the latency of face-specific responses in the right FG of our patient were modulated by elementary perceptual factors such as stimulus inversion or spatial frequency content (Experiments 1 and 5) but not affected by other high-level manipulations related to emotional expression (Experiments 2 and 3) or gaze direction (Experiment 4)—as we discussed in the next sections.

Face-specific N200 Reflects an Early Perceptual Stage of Face Encoding

In agreement with earlier findings (see McCarthy et al., 1999), we found that a disruption of face geometry by inversion affected neural responses recorded from the fusiform (see Experiment 1), but with different effects for schematic versus more realistic faces. An early effect of inversion on the face-selective N200 was obtained only with schematic faces (i.e., three dots arranged in a face gestalt, without finer visual information about internal features and identity information), starting 125-msec poststimulus, that is, already at the onset phase of the N200 component. By contrast, inversion of realistic faces (with more detailed features and specific identity information) did not significantly modulate the N200 but a subsequent phase instead, starting 195-msec poststimulus, during the offset phase of this component. This may suggest a temporally graded sensitivity of the face-specific N200 activity to different facial cues, with an early phase corresponding to the encoding of the face configuration per se (Johnson, 2005; Maurer et al., 2002), and a later phase of the component coding for more detailed visual cues including those that convey specific identity information (for single-cell data, see also Schyns et al., 2007; McCarthy et al., 1999; Sugase et al., 1999). Alternatively, it is also possible that our inversion of realistic faces (created by turning internal features upside down within an intact face contour) produced only a subtle disruption of the face configuration, therefore manifested by a later modulation of neural responses (for different face inversion manipulations, see McCarthy et al., 1999). Conversely, the specific impact on the N200 produced by inversion of schematic faces might be due to the fact that such stimuli had no facial property anymore, unlike the inverted realistic faces, resulting in distinct effects on the N200 in keeping with the view that this component could reflect an early perceptual categorization stage linked to face detection (Liu, Harris, & Kanwisher, 2002; Bentin et al., 1996).

McCarthy et al. (1999) reported that for central stimulus presentations, inverted (realistic) faces delayed the N200 relative to upright faces (while increasing its amplitude, although this effect was not significant in their study). These N200 effects were the same in the left and right hemisphere (McCarthy et al., 1999). In our study, we found that ERPs immediately consecutive to the face-sensitive N200 were enhanced in amplitude (but not delayed) for inverted, relative to upright realistic faces, whereas the amplitude and the latency of the N200 remained unaffected, a result that seems discrepant compared with McCarthy et al. (1999). However, we note that the faces used by McCarthy et al. (1999) and their in-plane face inversion were different from our stimuli and inversion manipulation (Experiment 1). Whereas McCarthy et al. (1999) simply presented their faces upside-down in the inverted condition (thereby disrupting early face detection mechanisms), here we swapped the in-plane orientation of internal facial features (presumably disrupting later stages of face recognition mechanisms) while the face contour was not changed. This led to a different face inversion manipulation in our study relative to McCarthy et al. (1999), which could account for the lack of latency (and amplitude) effect on the face-specific N200 with “realistic” faces in our case (see Experiment 1). However, note that we did obtain a significant effect of inversion for the N200 amplitude (similarly to McCarthy et al., 1999) but only with inverted schematic faces, as compared with upright. We surmise that our inversion manipulation with schematic faces had a different perceptual impact, as opposed to that with realistic faces (see Experiment 1 and Figure 4), because the former primarily affected configural face encoding whereas the latter presumably influenced later stages of face recognition on the basis of identity information. Our results are therefore generally consistent with the results of McCarthy et al. (1999), as both confirm that the face-specific N200 is primary sensitive to manipulations disrupting face detection processes (related to configural processing), whereas the immediately subsequent components are more concerned with face identity recognition (based on internal features), consistent with the predictions of cognitive models of face perception (see Maurer et al., 2002 and Figure 7).

Figure 7. 

Summary of main results in Experiments 1–5 for the right FG electrode. Combined together, our findings suggest a hierarchical and differential temporal encoding of face information in fusiform cortex following stimulus onset, including early perceptual effects at the level of the N200 (which is sensitive to configuration information, presumably conveyed by LSF cues), followed by the processing of more detailed identity information (presumably based on HSF cues), and finally by the processing of changeable or behaviorally relevant aspects of faces, including gaze direction and emotional expression. The recognition of individual face identity (face individuation) is hypothesized to primarily rely on the extraction of the first-order and second-order relations between the internal features within the facial configuration (see Maurer et al., 2002).

Figure 7. 

Summary of main results in Experiments 1–5 for the right FG electrode. Combined together, our findings suggest a hierarchical and differential temporal encoding of face information in fusiform cortex following stimulus onset, including early perceptual effects at the level of the N200 (which is sensitive to configuration information, presumably conveyed by LSF cues), followed by the processing of more detailed identity information (presumably based on HSF cues), and finally by the processing of changeable or behaviorally relevant aspects of faces, including gaze direction and emotional expression. The recognition of individual face identity (face individuation) is hypothesized to primarily rely on the extraction of the first-order and second-order relations between the internal features within the facial configuration (see Maurer et al., 2002).

In agreement with a sensitivity of the face-specific N200 to global information coding for the face configuration, our manipulation of spatial frequency content (Experiment 5) showed that the onset of the N200 was the same for LSF faces than for normal/broadband faces, whereas it was significantly delayed for HSF faces. This effect suggests that early stages of face processing in the right FG may depend on LSF inputs (global processing), more than on HSF (local processing), and thus converges with the assumption that LSF cues are not only processed faster than HSF cues (Bar, 2003; Bullier, 2001) but also crucial to convey information on facial geometry (Goffaux & Rossion, 2006; Sergent, 1995). On the other hand, both LSF and HSF faces evoked a fusiform N200 with a larger amplitude and prolonged duration, which might reflect a degraded visibility and poorer local information in these stimuli, leading to a prolonged processing time, in parallel to longer RTs in these two conditions. Recent findings from ERPs (Schyns et al., 2007) and fMRI (Ploran et al., 2007) in healthy subjects suggest that visual recognition involves a progressive accumulation of perceptual evidence before reaching threshold for accurate stimulus identification, which may require more time when stimuli are degraded (see also Thorpe, Fize, & Marlot, 1996).

Thus, at any rate, results of Experiments 1 and 5 clearly converge to show that the face-specific N200 is primarily associated with the early encoding/structural stages of face perception, most likely concerning the processing of facial geometry based on the extraction of global rather than local visual information (see Carmel & Bentin, 2002; Maurer et al., 2002; Sugase et al., 1999; Bruce & Young, 1986). Importantly, local facial details or higher order influences associated to emotional or social processing did not modulate the N200 but a subsequent period (see below). Evidently, although changes in facial expression and eye position also produce substantial visual changes in facial images, such factors do not alter the global configuration of faces.

Modulation of Face Processing by Emotional Expression and Gaze Direction

The major novel results of our study concerned the time course of neural responses to facial expression and gaze direction in the right fusiform cortex of our patient (Experiments 2–4). In two different experiments (Experiments 2 and 3), using different stimuli and different tasks, we found a consistent electrophysiological pattern of modulation by emotional face expression that specifically involved the face-sensitive electrode in fusiform. The house-selective electrode in PHG was not affected by face expressions, suggesting that these modulatory effects are category selective (see Peelen, Atkinson, Andersson, & Vuilleumier, 2007). Thus, relative to neutral faces, fearful and happy faces evoked a sustained negative activity subsequent to the face-specific N200, whereas the latter component remained unaffected by expression (see above and Figure 5).

This pattern is consistent with previous scalp ERP data that have also shown a modulation of late potentials during the perception of emotional faces (see Batty & Taylor, 2003; Krolak-Salmon, Fischer, Vighetto, & Mauguiere, 2001; for recent reviews, see also Vuilleumier & Pourtois, 2007; Schupp, Flaisch, Stockburger, & Junghofer, 2006). These new findings also converge with earlier human intracranial ERP studies that described later ERP components arising after the N200 and tentatively attributed to feedback from upstream brain regions (such as the N700 component, see Puce et al., 1999). Likewise, single-cell recordings in the monkey (Sugase et al., 1999) have obtained highly similar neurophysiological results for face-selective neurons, with a first response phase coding for global category-related information (e.g., face vs. nonface), and a second phase integrating modulatory signals from distant brain regions, most notably related to expression, familiarity, or identity. Here, we found that emotional expression produced a significant modulation of neural activity in fusiform cortex during a sustained period, extending roughly from ∼300- to 800-msec poststimulus onset, following the earlier modulations by category-selective and low-level perceptual information that arose approximately 150–200 msec.

This emotional enhancement of a late stage of face processing is highly consistent with the notion that such effect might result from regulatory feedback mechanisms from the amygdala (Vuilleumier, 2005; Vuilleumier, Richardson, et al., 2004), a brain region critically implicated in emotion recognition and known to have strong bidirectional connections with visual cortex (Amaral, Behniea, & Kelly, 2003). Several imaging studies have reported that activation of visual areas is not only increased to emotional faces as compared with neutral faces but also that such increase is correlated with the magnitude of a concomitant amygdala response (Peelen et al., 2007; Sabatinelli et al., 2005; Pessoa, Kastner, & Ungerleider, 2002; Morris et al., 1998). Moreover, an fMRI study in patients with medial-temporal lobe sclerosis showed that the typical enhancement of fusiform responses by emotion expression was lost in patients with damage to the amygdala (but not hippocampus), providing strong evidence for a direct causal role of the amygdala for modulating fusiform activity (Vuilleumier, Richardson, et al., 2004). To the best of our knowledge, our new data reveal for the first time the exact temporal course of such modulation in the human visual cortex and clearly demonstrate a delayed latency that is compatible with feedback or reentrant influences from distant sources in the amygdala (Vuilleumier, 2005).

Remarkably, we obtained a roughly similar pattern of emotional effects for happy and fearful facial expressions, relative to neutral faces (see Experiment 3; Figure 5), indicating that the amplitude modulation consecutive to the N200 is likely to reflect arousal value rather than valence or threat only. Although both fusiform and amygdala responses to facial expressions have most often been reported for negative stimuli (e.g., fear or anger; Surguladze et al., 2003), there is now clear evidence that arousing positive stimuli can also produce similar effects (Sabatinelli et al., 2005). Moreover, late sustained potentials to emotional stimuli have been found equally for happy and fearful faces in scalp EEG recordings (see Schupp et al., 2006; Krolak-Salmon et al., 2001).

We also note that this gain in amplitude for face-specific neural responses following the N200 was obtained in two different tasks where facial expression was not directly relevant to the task because the patient was asked either to detect immediate repetitions of different object identities (both faces and houses, Experiment 2) or to report the gender of individual faces (Experiment 3). These results therefore show that processing of emotional expression (either fearful vs. happy) and subsequent modulation of fusiform activity could take place in an “implicit” manner (e.g., Critchley et al., 2000), without an explicit intention to judge emotional expression. This is consistent with several previous fMRI (Vuilleumier, 2005) and EEG studies (Eimer & Holmes, 2007; Vuilleumier & Pourtois, 2007) reporting emotional responses in either the fusiform or the amygdala during “implicit” tasks (see Vuilleumier, Armony, & Dolan, 2004).

In addition, in our experiment manipulating spatial frequency content of faces (Experiment 5), we found a similar modulation of the neural activity following the N200 by emotional (fearful) expression, but only for LSF faces, not for HSF faces (in contrast to a main effect of spatial frequency affecting the early N200 activity, irrespective of expression, see above). This pattern converges with previous results from fMRI (Vuilleumier et al., 2003; Winston et al., 2003) and scalp ERPs (Carretie et al., 2007; Pourtois et al., 2005) showing greater emotional responses to LSF than HSF stimuli. Moreover, the latency of this emotional modulation by LSF stimuli (∼300–570 msec) was highly similar to that found for intact faces in our two other experiments (Experiments 2 and 3), in agreement with the idea that LSF inputs might provide important cues for coarse and rapid emotional recognition before eliciting feedback influences on visual areas (Bar, 2004; Vuilleumier et al., 2003). We note however that in this Experiment, we failed to find a significant modulation by emotional content for the broadband/unfiltered faces (which contained both LSF and HSF), unlike our findings for intact fearful faces in Experiment 2, and for both fearful and happy faces in Experiment 3. We have no explanation for this lack of effect in this particular condition, but it is most likely to reflect a type II error due to higher EEG noise, different stimulus set, or insufficient statistical power, given the strong and reproducible effect observed in our two other experiments for two different expression categories.

Finally, another novel result (Experiment 4) was that a similar late modulation of face-responsive cortex arose as a function of the direction of gaze, with a larger and sustained negativity starting approximately 390-msec poststimulus onset for faces with straight as compared with averted eye gaze. However, in our study, we failed to find a modulation of the face-sensitive N200 as a function of change in either head view or gaze direction. However, a previous study (McCarthy et al., 1999) reported that N200 was larger and earlier when the head faced the viewer (regardless of eye position). The N200 was also earlier when the eyes looked at the viewer rather than away (regardless of head position), suggesting that this early component can also be sensitive to social signals conveyed by face stimuli in some conditions. Scalp ERP data also suggest that eye aversion can elicit N170 amplitudes that are larger than those elicited to viewing eyes gazing directly at the viewer (see Puce, Smith, & Allison, 2000; but see Conty et al., 2007). These results suggest that some early stages of face encoding within the right FG might already be modulated by social cues (McCarthy et al., 1999); here we primarily observed a later feedback effect of gaze direction on face-sensitive activity within the right FG. In our study, this larger and sustained negativity is consistent with previous fMRI results in healthy subjects, showing enhanced activation in fusiform (George et al., 2001) and amygdala (Kawashima et al., 1999) in response to perceived direct gaze, presumably reflecting higher affective value and subjective personal relevance of direct gaze contact (Calder et al., 2007; Sander, Grafman, & Zalla, 2003). These data do not only confirm such modulation of face processing by socially significant gaze signals but in addition provide new insights on the temporal dynamics of these effects.

As social meaning or eye position cues are most probably processed in different and distant brain regions (such as the STS; see Puce & Perrett, 2003; Haxby et al., 2000), these late effects are also suggestive of some feedback or reentrant influences on fusiform activity. Although the latency of the gaze effects was roughly parallel to those of emotional expression and therefore compatible with similar modulatory signals from amygdala (for indirect evidence from fMRI connectivity analysis, see George et al., 2001), we note nevertheless that the time range of the former effects appeared slightly delayed relative to the latter (i.e., ∼400–1000 msec for gaze vs. ∼300–600 msec for expression). Although we cannot draw any firm conclusion from our single subject data, such differences could be consistent with more delayed influences from higher order cortical areas such as lateral temporal regions (e.g., STS) coding social or intentional dispositions from faces (Blakemore, Winston, & Frith, 2004; Haxby et al., 2000). Further research using time resolved techniques with EEG or MEG or additional patient studies will be needed to disentangle the different sources of influences modulating fusiform activity at distinct latency ranges.

Altogether, results of Experiments 2–4 validate several key predictions about the time course of visual facial information processing, as hypothesized by current distributed models of face recognition (Haxby et al., 2000; for a recent revised version, see also Gobbini & Haxby, 2007). A central aspect of these models is that some information is processed in a so-called extended system (including the amygdala and the posterior STS that mediate the recognition of emotion expression and the recognition of intention or mental states, respectively), which can bias, via top–down modulatory feedback projections, the processing of visual facial information (e.g., identity) taking place in a core system (centered on the inferotemporal cortex and FG). Our new intracranial results (Experiments 2–4) support this view, as two specific face attributes (emotional expression and gaze direction) were found to modulate late stages of processing within the FG (with no effect on the early face encoding stage associated with the face-specific N200 activity). In contrast, another prediction of these models is that manipulations altering the perceptual visibility of face stimuli (e.g., by inverting internal face features in Experiment 1 or by changing spatial frequency content in Experiment 5) should selectively lead to early modulations within the core system. Again, our results in Experiments 1 and 5 verified this second prediction. Thus, the time course of visual facial information processing uncovered by our intracranial recordings in FG does not only accord with the current neural models of face recognition (Gobbini & Haxby, 2007; Haxby et al., 2000) but also demonstrate dynamical interactions between distinct stages engaged by face processing (Vuilleumier, 2007; Vuilleumier & Pourtois, 2007), including top–down modulatory influences on FG from the amygdala during emotion processing and perhaps from STS during gaze direction processing. Note however that because our recordings focused on ventral temporal cortex and we found no reliable face-selective responses at other electrode sites in this patient (e.g., in frontal cortex), our study could not directly pinpoint the sources of top–down signals in regions of the extended system. Likewise, a limitation of this study stems from the testing of a single patient, which does not necessarily permit a straightforward generalization of the current results to the entire (healthy) population.

Conclusion

Our results provide a number of important new insights on the temporal dynamics of processing faces and affectively relevant facial information in human visual cortex (see schematic summary in Figure 7). In agreement with recordings of face-selective neurons in the monkey (Sugase et al., 1999), our intracranial data show that different features of faces are represented or integrated over time in fusiform cortex, concerning global category-selective information about facial configuration in the initial N200 time range, followed by more local information about face individuation, expression, or social cues (possibly in this temporal sequence, see Figure 7). Although the sustained enhancement produced by emotional and social signals is thought to mediate enhanced perception and attention to behaviorally relevant stimuli (Phelps, 2006; Vuilleumier, 2005), the prolonged duration of such effects as observed in our patient (i.e., up to 600- to 1000-msec poststimulus) might also suggest an important role in memory or plasticity processes influencing learning and subsequent recognition (Vuilleumier et al., 2005; Weinberger, 2004). More generally, our study highlights the importance of the temporal dynamics in addition to the anatomical segregation of cortical activity underlying face and object recognition.

Acknowledgments

This work was supported by grants from the Swiss National Science Foundation (NCCR grant no. 51A240-104897; Swiss National Centre for Competence in Research in Affective Sciences), the European Research Council (starting grant no. 200758 to G. P.), and the Ghent University (BOF grant no. 05Z01708 to G. P.). G. P. was also supported by grant BOF/GOA2006/001 of Ghent University. The authors thank five anonymous reviewers for their many helpful and insightful comments on our manuscript.

Reprint requests should be sent to Gilles Pourtois, Department of Experimental Clinical and Health Psychology, Ghent University, Henri Dunantlaan 2, 9000 Gent, Belgium, or via e-mail: gilles.pourtois@ugent.be.

REFERENCES

Adams
,
R. B.
, Jr.,
Gordon
,
H. L.
,
Baird
,
A. A.
,
Ambady
,
N.
, &
Kleck
,
R. E.
(
2003
).
Effects of gaze on amygdala sensitivity to anger and fear faces.
Science
,
300
,
1536
.
Allison
,
T.
,
Ginter
,
H.
,
McCarthy
,
G.
,
Nobre
,
A. C.
,
Puce
,
A.
,
Luby
,
M.
,
et al
(
1994
).
Face recognition in human extrastriate cortex.
Journal of Neurophysiology
,
71
,
821
825
.
Allison
,
T.
,
McCarthy
,
G.
,
Nobre
,
A.
,
Puce
,
A.
, &
Belger
,
A.
(
1994
).
Human extrastriate visual cortex and the perception of faces, words, numbers, and colors.
Cerebral Cortex
,
4
,
544
554
.
Allison
,
T.
,
Puce
,
A.
, &
McCarthy
,
G.
(
2002
).
Category-sensitive excitatory and inhibitory processes in human extrastriate cortex.
Journal of Neurophysiology
,
88
,
2864
2868
.
Allison
,
T.
,
Puce
,
A.
,
Spencer
,
D. D.
, &
McCarthy
,
G.
(
1999
).
Electrophysiological studies of human face perception: I. Potentials generated in occipitotemporal cortex by face and non-face stimuli.
Cerebral Cortex
,
9
,
415
430
.
Amaral
,
D. G.
,
Behniea
,
H.
, &
Kelly
,
J. L.
(
2003
).
Topographic organization of projections from the amygdala to the visual cortex in the macaque monkey.
Neuroscience
,
118
,
1099
1120
.
Bar
,
M.
(
2003
).
A cortical mechanism for triggering top–down facilitation in visual object recognition.
Journal of Cognitive Neuroscience
,
15
,
600
609
.
Bar
,
M.
(
2004
).
Visual objects in context.
Nature Reviews Neuroscience
,
5
,
617
629
.
Batty
,
M.
, &
Taylor
,
M. J.
(
2003
).
Early processing of the six basic facial emotional expressions.
Brain Research, Cognitive Brain Research
,
17
,
613
620
.
Beck
,
D. M.
,
Rees
,
G.
,
Frith
,
C. D.
, &
Lavie
,
N.
(
2001
).
Neural correlates of change detection and change blindness.
Nature Neuroscience
,
4
,
645
650
.
Bentin
,
S.
,
Allison
,
T.
,
Puce
,
A.
,
Perez
,
E.
, &
McCarthy
,
G.
(
1996
).
Electrophysiological studies of face perception in humans.
Journal of Cognitive Neuroscience
,
8
,
551
565
.
Blakemore
,
S. J.
,
Winston
,
J.
, &
Frith
,
U.
(
2004
).
Social cognitive neuroscience: Where are we heading?
Trends in Cognitive Sciences
,
8
,
216
222
.
Botzel
,
K.
, &
Grusser
,
O. J.
(
1989
).
Electric brain potentials evoked by pictures of faces and non-faces: A search for “face-specific” EEG-potentials.
Experimental Brain Research
,
77
,
349
360
.
Bruce
,
V.
, &
Young
,
A.
(
1986
).
Understanding face recognition.
British Journal of Psychology
,
77
,
305
327
.
Buchel
,
C.
, &
Dolan
,
R. J.
(
2000
).
Classical fear conditioning in functional neuroimaging.
Current Opinion in Neurobiology
,
10
,
219
223
.
Bullier
,
J.
(
2001
).
Integrated model of visual processing.
Brain Research, Brain Research Reviews
,
36
,
96
107
.
Calder
,
A. J.
,
Beaver
,
J. D.
,
Winston
,
J. S.
,
Dolan
,
R. J.
,
Jenkins
,
R.
,
Eger
,
E.
,
et al
(
2007
).
Separate coding of different gaze directions in the superior temporal sulcus and inferior parietal lobule.
Current Biology
,
17
,
20
25
.
Carmel
,
D.
, &
Bentin
,
S.
(
2002
).
Domain specificity versus expertise: Factors influencing distinct processing of faces.
Cognition
,
83
,
1
29
.
Carretie
,
L.
,
Hinojosa
,
J. A.
,
Lopez-Martin
,
S.
, &
Tapia
,
M.
(
2007
).
An electrophysiological study on the interaction between emotional content and spatial frequency of visual stimuli.
Neuropsychologia
,
45
,
1187
1195
.
Conty
,
L.
,
N'Diaye
,
K.
,
Tijus
,
C.
, &
George
,
N.
(
2007
).
When eye creates the contact! ERP evidence for early dissociation between direct and averted gaze motion processing.
Neuropsychologia
,
45
,
3024
3037
.
Conty
,
L.
,
Tijus
,
C.
,
Hugueville
,
L.
,
Coelho
,
E.
, &
George
,
N.
(
2006
).
Searching for asymmetries in the detection of gaze contact versus averted gaze under different head views: A behavioural study.
Spatial Vision
,
19
,
529
545
.
Critchley
,
H.
,
Daly
,
E.
,
Phillips
,
M.
,
Brammer
,
M.
,
Bullmore
,
E.
,
Williams
,
S.
,
et al
(
2000
).
Explicit and implicit neural mechanisms for processing of social information from facial expressions: A functional magnetic resonance imaging study.
Human Brain Mapping
,
9
,
93
105
.
de Gelder
,
B.
, &
Rouw
,
R.
(
2001
).
Beyond localisation: A dynamical dual route account of face recognition.
Acta Psychologica
,
107
,
183
207
.
Delorme
,
A.
, &
Makeig
,
S.
(
2004
).
EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.
Journal of Neuroscience Methods
,
134
,
9
21
.
Driver
,
J.
, &
Vuilleumier
,
P.
(
2001
).
Perceptual awareness and its loss in unilateral neglect and extinction.
Cognition
,
79
,
39
88
.
Druzgal
,
T. J.
, &
D'Esposito
,
M.
(
2001
).
A neural network reflecting decisions about human faces.
Neuron
,
32
,
947
955
.
Eger
,
E.
,
Schyns
,
P. G.
, &
Kleinschmidt
,
A.
(
2004
).
Scale invariant adaptation in fusiform face-responsive regions.
Neuroimage
,
22
,
232
242
.
Eimer
,
M.
, &
Holmes
,
A.
(
2007
).
Event-related brain potential correlates of emotional face processing.
Neuropsychologia
,
45
,
15
31
.
Ekman
,
P.
, &
Friesen
,
W. V.
(
1976
).
Pictures of facial affect.
Palo Alto, CA
:
Consulting Psychologists Press
.
Epstein
,
R.
,
Graham
,
K. S.
, &
Downing
,
P. E.
(
2003
).
Viewpoint-specific scene representations in human parahippocampal cortex.
Neuron
,
37
,
865
876
.
Epstein
,
R.
,
Harris
,
A.
,
Stanley
,
D.
, &
Kanwisher
,
N.
(
1999
).
The parahippocampal place area: Recognition, navigation, or encoding?
Neuron
,
23
,
115
125
.
Epstein
,
R.
, &
Kanwisher
,
N.
(
1998
).
A cortical representation of the local visual environment.
Nature
,
392
,
598
601
.
Furey
,
M. L.
,
Tanskanen
,
T.
,
Beauchamp
,
M. S.
,
Avikainen
,
S.
,
Uutela
,
K.
,
Hari
,
R.
,
et al
(
2006
).
Dissociation of face-selective cortical responses by attention.
Proceedings of the National Academy of Sciences, U.S.A.
,
103
,
1065
1070
.
George
,
N.
,
Dolan
,
R. J.
,
Fink
,
G. R.
,
Baylis
,
G. C.
,
Russell
,
C.
, &
Driver
,
J.
(
1999
).
Contrast polarity and face recognition in the human fusiform gyrus.
Nature Neuroscience
,
2
,
574
580
.
George
,
N.
,
Driver
,
J.
, &
Dolan
,
R. J.
(
2001
).
Seen gaze-direction modulates fusiform activity and its coupling with other brain areas during face processing.
Neuroimage
,
13
,
1102
1112
.
George
,
N.
,
Evans
,
J.
,
Fiori
,
N.
,
Davidoff
,
J.
, &
Renault
,
B.
(
1996
).
Brain events related to normal and moderately scrambled faces.
Brain Research, Cognitive Brain Research
,
4
,
65
76
.
Gilaie-Dotan
,
S.
, &
Malach
,
R.
(
2007
).
Sub-exemplar shape tuning in human face-related areas.
Cerebral Cortex
,
17
,
325
338
.
Gobbini
,
M. I.
, &
Haxby
,
J. V.
(
2007
).
Neural systems for recognition of familiar faces.
Neuropsychologia
,
45
,
32
41
.
Goffaux
,
V.
,
Gauthier
,
I.
, &
Rossion
,
B.
(
2003
).
Spatial scale contribution to early visual differences between face and object processing.
Brain Research, Cognitive Brain Research
,
16
,
416
424
.
Goffaux
,
V.
, &
Rossion
,
B.
(
2006
).
Faces are “spatial”-holistic face perception is supported by low spatial frequencies.
Journal of Experimental Psychology: Human Perception and Performance
,
32
,
1023
1039
.
Grill-Spector
,
K.
(
2003
).
The neural basis of object perception.
Current Opinion in Neurobiology
,
13
,
159
166
.
Grill-Spector
,
K.
, &
Kanwisher
,
N.
(
2005
).
Visual recognition: As soon as you know it is there, you know what it is.
Psychological Science
,
16
,
152
160
.
Grill-Spector
,
K.
,
Knouf
,
N.
, &
Kanwisher
,
N.
(
2004
).
The fusiform face area subserves face perception, not generic within-category identification.
Nature Neuroscience
,
7
,
555
562
.
Grill-Spector
,
K.
,
Kushnir
,
T.
,
Hendler
,
T.
,
Edelman
,
S.
,
Itzchak
,
Y.
, &
Malach
,
R.
(
1998
).
A sequence of object-processing stages revealed by fMRI in the human occipital lobe.
Human Brain Mapping
,
6
,
316
328
.
Gross
,
C. G.
(
1992
).
Representation of visual stimuli in inferior temporal cortex.
Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences
,
335
,
3
10
.
Halgren
,
E.
,
Baudena
,
P.
,
Heit
,
G.
,
Clarke
,
J. M.
,
Marinkovic
,
K.
,
Chauvel
,
P.
,
et al
(
1994
).
Spatio-temporal stages in face and word processing: II. Depth-recorded potentials in the human frontal and Rolandic cortices.
Journal of Physiology
,
88
,
51
80
.
Halgren
,
E.
,
Baudena
,
P.
,
Heit
,
G.
,
Clarke
,
J. M.
,
Marinkovic
,
K.
, &
Clarke
,
M.
(
1994
).
Spatio-temporal stages in face and word processing: I. Depth-recorded potentials in the human occipital, temporal and parietal lobes [corrected].
Journal of Physiology
,
88
,
1
50
.
Halgren
,
E.
,
Dale
,
A. M.
,
Sereno
,
M. I.
,
Tootell
,
R. B.
,
Marinkovic
,
K.
, &
Rosen
,
B. R.
(
1999
).
Location of human face-selective cortex with respect to retinotopic areas.
Human Brain Mapping
,
7
,
29
37
.
Halgren
,
E.
,
Raij
,
T.
,
Marinkovic
,
K.
,
Jousmaki
,
V.
, &
Hari
,
R.
(
2000
).
Cognitive response profile of the human fusiform face area as determined by MEG.
Cerebral Cortex
,
10
,
69
81
.
Haxby
,
J. V.
,
Hoffman
,
E. A.
, &
Gobbini
,
M. I.
(
2000
).
The distributed human neural system for face perception.
Trends in Cognitive Sciences
,
4
,
223
233
.
Heekeren
,
H. R.
,
Marrett
,
S.
,
Bandettini
,
P. A.
, &
Ungerleider
,
L. G.
(
2004
).
A general mechanism for perceptual decision-making in the human brain.
Nature
,
431
,
859
862
.
Henson
,
R. N.
,
Goshen-Gottstein
,
Y.
,
Ganel
,
T.
,
Otten
,
L. J.
,
Quayle
,
A.
, &
Rugg
,
M. D.
(
2003
).
Electrophysiological and haemodynamic correlates of face perception, recognition and priming.
Cerebral Cortex
,
13
,
793
805
.
Hoffman
,
E. A.
, &
Haxby
,
J. V.
(
2000
).
Distinct representations of eye gaze and identity in the distributed human neural system for face perception.
Nature Neuroscience
,
3
,
80
84
.
Itier
,
R. J.
, &
Taylor
,
M. J.
(
2004a
).
Source analysis of the N170 to faces and objects.
NeuroReport
,
15
,
1261
1265
.
Itier
,
R. J.
, &
Taylor
,
M. J.
(
2004b
).
N170 or N1? Spatiotemporal differences between object and face processing using ERPs.
Cerebral Cortex
,
14
,
132
142
.
Johnson
,
M. H.
(
2005
).
Subcortical face processing.
Nature Reviews Neuroscience
,
6
,
766
774
.
Kanwisher
,
N.
,
McDermott
,
J.
, &
Chun
,
M. M.
(
1997
).
The fusiform face area: A module in human extrastriate cortex specialized for face perception.
Journal of Neuroscience
,
17
,
4302
4311
.
Kanwisher
,
N.
, &
Yovel
,
G.
(
2006
).
The fusiform face area: A cortical region specialized for the perception of faces.
Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences
,
361
,
2109
2128
.
Kawashima
,
R.
,
Sugiura
,
M.
,
Kato
,
T.
,
Nakamura
,
A.
,
Hatano
,
K.
,
Ito
,
K.
,
et al
(
1999
).
The human amygdala plays an important role in gaze monitoring. A PET study.
Brain
,
122
,
779
783
.
Kourtzi
,
Z.
, &
DiCarlo
,
J. J.
(
2006
).
Learning and neural plasticity in visual object recognition.
Current Opinion in Neurobiology
,
16
,
152
158
.
Krolak-Salmon
,
P.
,
Fischer
,
C.
,
Vighetto
,
A.
, &
Mauguiere
,
F.
(
2001
).
Processing of facial emotional expression: Spatio-temporal data as assessed by scalp event-related potentials.
European Journal of Neuroscience
,
13
,
987
994
.
Lang
,
P. J.
,
Bradley
,
M. M.
,
Fitzsimmons
,
J. R.
,
Cuthbert
,
B. N.
,
Scott
,
J. D.
,
Moulder
,
B.
,
et al
(
1998
).
Emotional arousal and activation of the visual cortex: An fMRI analysis.
Psychophysiology
,
35
,
199
210
.
Langton
,
S. R.
,
Watt
,
R. J.
, &
Bruce
,
I. I.
(
2000
).
Do the eyes have it? Cues to the direction of social attention.
Trends in Cognitive Sciences
,
4
,
50
59
.
Leopold
,
D. A.
,
Bondar
,
I. V.
, &
Giese
,
M. A.
(
2006
).
Norm-based face encoding by single neurons in the monkey inferotemporal cortex.
Nature
,
442
,
572
575
.
Lerner
,
Y.
,
Hendler
,
T.
,
Ben-Bashat
,
D.
,
Harel
,
M.
, &
Malach
,
R.
(
2001
).
A hierarchical axis of object processing stages in the human visual cortex.
Cerebral Cortex
,
11
,
287
297
.
Levy
,
I.
,
Hasson
,
U.
,
Avidan
,
G.
,
Hendler
,
T.
, &
Malach
,
R.
(
2001
).
Center-periphery organization of human object areas.
Nature Neuroscience
,
4
,
533
539
.
Liu
,
J.
,
Harris
,
A.
, &
Kanwisher
,
N.
(
2002
).
Stages of processing in face perception: An MEG study.
Nature Neuroscience
,
5
,
910
916
.
Lundqvist
,
D.
,
Flykt
,
A.
, &
Ohman
,
A.
(
1998
).
The Karolinska Directed Emotional Faces (KDEF).
Stockholm
:
Department of Neurosciences, Karolinska Hospital
.
Macrae
,
C. N.
,
Hood
,
B. M.
,
Milne
,
A. B.
,
Rowe
,
A. C.
, &
Mason
,
M. F.
(
2002
).
Are you looking at me? Eye gaze and person perception.
Psychological Science
,
13
,
460
464
.
Manly
,
B. F.
(
1991
).
Randomization and Monte Carlo methods in biology.
London, UK
:
Chapman & Hall
.
Matsumoto
,
N.
,
Okada
,
M.
,
Sugase-Miyamoto
,
Y.
,
Yamane
,
S.
, &
Kawano
,
K.
(
2005
).
Population dynamics of face-responsive neurons in the inferior temporal cortex.
Cerebral Cortex
,
15
,
1103
1112
.
Maurer
,
D.
,
Grand
,
R. L.
, &
Mondloch
,
C. J.
(
2002
).
The many faces of configural processing.
Trends in Cognitive Sciences
,
6
,
255
260
.
McCarthy
,
G.
,
Puce
,
A.
,
Belger
,
A.
, &
Allison
,
T.
(
1999
).
Electrophysiological studies of human face perception. II: Response properties of face-specific potentials generated in occipitotemporal cortex.
Cerebral Cortex
,
9
,
431
444
.
Milner
,
D.
, &
Goodale
,
M.
(
1995
).
The visual brain in action.
Oxford
:
Oxford University Press
.
Morris
,
J. S.
,
Friston
,
K. J.
,
Buchel
,
C.
,
Frith
,
C. D.
,
Young
,
A. W.
,
Calder
,
A. J.
,
et al
(
1998
).
A neuromodulatory role for the human amygdala in processing emotional facial expressions.
Brain
,
121
,
47
57
.
Nasanen
,
R.
(
1999
).
Spatial frequency bandwidth used in the recognition of facial images.
Vision Research
,
39
,
3824
3833
.
Op de Beeck
,
H. P.
,
Haushofer
,
J.
, &
Kanwisher
,
N. G.
(
2008
).
Interpreting fMRI data: Maps, modules and dimensions.
Nature Reviews Neuroscience
,
9
,
123
135
.
Palmer
,
S. E.
(
1999
).
Vision science: Photons to phenomenology.
Boston
:
MIT Press
.
Peelen
,
M. V.
,
Atkinson
,
A. P.
,
Andersson
,
F.
, &
Vuilleumier
,
P.
(
2007
).
Emotional modulation of body-selective visual areas.
Social Cognitive and Affective Neuroscience
,
2
,
274
283
.
Pessoa
,
L.
,
Kastner
,
S.
, &
Ungerleider
,
L. G.
(
2002
).
Attentional control of the processing of neural and emotional stimuli.
Brain Research, Cognitive Brain Research
,
15
,
31
45
.
Phelps
,
E. A.
(
2006
).
Emotion and cognition: Insights from studies of the human amygdala.
Annual Review of Psychology
,
57
,
27
53
.
Ploran
,
E. J.
,
Nelson
,
S. M.
,
Velanova
,
K.
,
Donaldson
,
D. I.
,
Petersen
,
S. E.
, &
Wheeler
,
M. E.
(
2007
).
Evidence accumulation and the moment of recognition: Dissociating perceptual recognition processes using fMRI.
Journal of Neuroscience
,
27
,
11912
11924
.
Pourtois
,
G.
,
Dan
,
E. S.
,
Grandjean
,
D.
,
Sander
,
D.
, &
Vuilleumier
,
P.
(
2005
).
Enhanced extrastriate visual response to bandpass spatial frequency filtered fearful faces: Time course and topographic evoked-potentials mapping.
Human Brain Mapping
,
26
,
65
79
.
Pourtois
,
G.
,
Peelen
,
M. V.
,
Spinelli
,
L.
,
Seeck
,
M.
, &
Vuilleumier
,
P.
(
2007
).
Direct intracranial recording of body-selective responses in human extrastriate visual cortex.
Neuropsychologia
,
45
,
2621
2625
.
Pourtois
,
G.
,
Schwartz
,
S.
,
Seghier
,
M. L.
,
Lazeyras
,
F.
, &
Vuilleumier
,
P.
(
2005a
).
Portraits or people? Distinct representations of face identity in the human visual cortex.
Journal of Cognitive Neuroscience
,
17
,
1043
1057
.
Pourtois
,
G.
,
Schwartz
,
S.
,
Seghier
,
M. L.
,
Lazeyras
,
F.
, &
Vuilleumier
,
P.
(
2005b
).
View-independent coding of face identity in frontal and temporal cortices is modulated by familiarity: An event-related fMRI study.
Neuroimage
,
24
,
1214
1224
.
Pourtois
,
G.
,
Schwartz
,
S.
,
Seghier
,
M. L.
,
Lazeyras
,
F.
, &
Vuilleumier
,
P.
(
2006
).
Neural systems for orienting attention to the location of threat signals: An event-related fMRI study.
Neuroimage
,
31
,
920
933
.
Puce
,
A.
,
Allison
,
T.
,
Gore
,
J. C.
, &
McCarthy
,
G.
(
1995
).
Face-sensitive regions in human extrastriate cortex studied by functional MRI.
Journal of Neurophysiology
,
74
,
1192
1199
.
Puce
,
A.
,
Allison
,
T.
, &
McCarthy
,
G.
(
1999
).
Electrophysiological studies of human face perception: III. Effects of top–down processing on face-specific potentials.
Cerebral Cortex
,
9
,
445
458
.
Puce
,
A.
, &
Perrett
,
D.
(
2003
).
Electrophysiology and brain imaging of biological motion.
Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences
,
358
,
435
445
.
Puce
,
A.
,
Smith
,
A.
, &
Allison
,
T.
(
2000
).
ERPs evoked by viewing facial movements.
Cognitive Neuropsychology
,
17
,
221
239
.
Rhodes
,
G.
,
Brake
,
S.
, &
Atkinson
,
A. P.
(
1993
).
Whats lost in inverted faces.
Cognition
,
47
,
25
57
.
Riesenhuber
,
M.
, &
Poggio
,
T.
(
1999
).
Hierarchical models of object recognition in cortex.
Nature Neuroscience
,
2
,
1019
1025
.
Rolls
,
E. T.
,
Perrett
,
D. I.
,
Caan
,
A. W.
, &
Wilson
,
F. A.
(
1982
).
Neuronal responses related to visual recognition.
Brain
,
105
,
611
646
.
Rotshtein
,
P.
,
Vuilleumier
,
P.
,
Winston
,
J.
,
Driver
,
J.
, &
Dolan
,
R.
(
2007
).
Distinct and convergent visual processing of high and low spatial frequency information in faces.
Cerebral Cortex
.
Sabatinelli
,
D.
,
Bradley
,
M. M.
,
Fitzsimmons
,
J. R.
, &
Lang
,
P. J.
(
2005
).
Parallel amygdala and inferotemporal activation reflect emotional intensity and fear relevance.
Neuroimage
,
24
,
1265
1270
.
Sander
,
D.
,
Grafman
,
J.
, &
Zalla
,
T.
(
2003
).
The human amygdala: An evolved system for relevance detection.
Reviews in Neurosciences
,
14
,
303
316
.
Schupp
,
H. T.
,
Flaisch
,
T.
,
Stockburger
,
J.
, &
Junghofer
,
M.
(
2006
).
Emotion and attention: Event-related brain potential studies.
Progress in Brain Research
,
156
,
31
51
.
Schyns
,
P. G.
, &
Oliva
,
A.
(
1999
).
Dr. Angry and Mr. Smile: When categorization flexibly modifies the perception of faces in rapid visual presentations.
Cognition
,
69
,
243
265
.
Schyns
,
P. G.
,
Petro
,
L. S.
, &
Smith
,
M. L.
(
2007
).
Dynamics of visual information integration in the brain for categorizing facial expressions.
Current Biology
,
17
,
1580
1585
.
Searcy
,
J. H.
, &
Bartlett
,
J. C.
(
1996
).
Inversion and processing of component and spatial-relational information in faces.
Journal of Experimental Psychology: Human Perception and Performance
,
22
,
904
915
.
Seeck
,
M.
,
Michel
,
C. M.
,
Blanke
,
O.
,
Thut
,
G.
,
Landis
,
T.
, &
Schomer
,
D. L.
(
2001
).
Intracranial neurophysiological correlates related to the processing of faces.
Epilepsy and Behavior
,
2
,
545
557
.
Sergent
,
J.
(
1995
).
Hemispheric contribution to face processing: Patterns of convergence and divergence.
In R. J. Davidson & K. Hugdahl (Eds.),
Brain asymmetry
,
Cambridge, MA
:
MIT Press
.
Sugase
,
Y.
,
Yamane
,
S.
,
Ueno
,
S.
, &
Kawano
,
K.
(
1999
).
Global and fine information coded by single neurons in the temporal visual cortex.
Nature
,
400
,
869
873
.
Surguladze
,
S. A.
,
Brammer
,
M. J.
,
Young
,
A. W.
,
Andrew
,
C.
,
Travis
,
M. J.
,
Williams
,
S. C.
,
et al
(
2003
).
A preferential increase in the extrastriate response to signals of danger.
Neuroimage
,
19
,
1317
1328
.
Talairach
,
J.
, &
Tournoux
,
P.
(
1988
).
Co-planar stereotaxic atlas of the human brain.
New York
:
Thieme
.
Tanskanen
,
T.
,
Nasanen
,
R.
,
Montez
,
T.
,
Paallysaho
,
J.
, &
Hari
,
R.
(
2005
).
Face recognition and cortical responses show similar sensitivity to noise spatial frequency.
Cerebral Cortex
,
15
,
526
534
.
Taylor
,
M. J.
,
Itier
,
R. J.
,
Allison
,
T.
, &
Edmonds
,
G. E.
(
2001
).
Direction of gaze effects on early face processing: Eyes-only versus full faces.
Brain Research, Cognitive Brain Research
,
10
,
333
340
.
Thierry
,
G.
,
Pegna
,
A. J.
,
Dodds
,
C.
,
Roberts
,
M.
,
Basan
,
S.
, &
Downing
,
P.
(
2006
).
An event-related potential component sensitive to images of the human body.
Neuroimage
,
32
,
871
879
.
Thorpe
,
S.
,
Fize
,
D.
, &
Marlot
,
C.
(
1996
).
Speed of processing in the human visual system.
Nature
,
381
,
520
522
.
Tong
,
F.
,
Nakayama
,
K.
,
Moscovitch
,
M.
,
Weinrib
,
O.
, &
Kanwisher
,
N.
(
2000
).
Response properties of the human fusiform face area.
Cognitive Neuropsychology
,
17
,
257
279
.
Tsao
,
D. Y.
,
Freiwald
,
W. A.
,
Knutsen
,
T. A.
,
Mandeville
,
J. B.
, &
Tootell
,
R. B.
(
2003
).
Faces and objects in macaque cerebral cortex.
Nature Neuroscience
,
6
,
989
995
.
Ungerleider
,
L. G.
, &
Mishkin
,
M.
(
1982
).
Two cortical visual systems.
In D. J. Ingle, M. A. Goodale, & R. J. W. Mansfield (Eds.),
Analysis of visual behavior
(pp.
549
586
).
Cambridge, MA
:
MIT Press
.
Vuilleumier
,
P.
(
2005
).
How brains beware: Neural mechanisms of emotional attention.
Trends in Cognitive Sciences
,
9
,
585
594
.
Vuilleumier
,
P.
(
2007
).
Neural representation of faces in human visual cortex: The roles of attention, emotion, and viewpoint.
In N. Osaka, I. Rentschler, & I. Biederman (Eds.),
Object recognition, attention, and action
(pp.
109
128
).
New York
:
Springer
.
Vuilleumier
,
P.
,
Armony
,
J. L.
, &
Dolan
,
R. J.
(
2004
).
Reciprocal links between emotion and attention.
In R. J. S. Frackowiak & J. Mazziotta (Eds.),
Human brain function
(pp.
419
444
).
London
:
Elsevier
.
Vuilleumier
,
P.
,
Armony
,
J. L.
,
Driver
,
J.
, &
Dolan
,
R. J.
(
2001
).
Effects of attention and emotion on face processing in the human brain: An event-related fMRI study.
Neuron
,
30
,
829
841
.
Vuilleumier
,
P.
,
Armony
,
J. L.
,
Driver
,
J.
, &
Dolan
,
R. J.
(
2003
).
Distinct spatial frequency sensitivities for processing faces and emotional expressions.
Nature Neuroscience
,
6
,
624
631
.
Vuilleumier
,
P.
,
George
,
N.
,
Lister
,
V.
,
Armony
,
J. L.
, &
Driver
,
J.
(
2005
).
Effects of perceived mutual gaze and gender on face processing and recognition memory.
Visual Cognition
,
12
,
85
102
.
Vuilleumier
,
P.
, &
Pourtois
,
G.
(
2007
).
Distributed and interactive brain mechanisms during emotion face perception: Evidence from functional neuroimaging.
Neuropsychologia
,
45
,
174
194
.
Vuilleumier
,
P.
,
Richardson
,
M. P.
,
Armony
,
J. L.
,
Driver
,
J.
, &
Dolan
,
R. J.
(
2004
).
Distant influences of amygdala lesion on visual cortical activation during emotional face processing.
Nature Neuroscience
,
7
,
1271
1278
.
Weinberger
,
N. M.
(
2004
).
Specific long-term memory traces in primary auditory cortex.
Nature Reviews Neuroscience
,
5
,
279
290
.
Winston
,
J. S.
,
Strange
,
B. A.
,
O'Doherty
,
J.
, &
Dolan
,
R. J.
(
2002
).
Automatic and intentional brain responses during evaluation of trustworthiness of faces.
Nature Neuroscience
,
5
,
277
283
.
Winston
,
J. S.
,
Vuilleumier
,
P.
, &
Dolan
,
R. J.
(
2003
).
Effects of low-spatial frequency components of fearful faces on fusiform cortex activity.
Current Biology
,
13
,
1824
1829
.
Wojciulik
,
E.
,
Kanwisher
,
N.
, &
Driver
,
J.
(
1998
).
Covert visual attention modulates face-specific activity in the human fusiform gyrus: fMRI study.
Journal of Neurophysiology
,
79
,
1574
1578
.