Individual Differences in Face Identity Processing with Fast Periodic Visual Stimulation

It is generally understood that humans are face experts who can recognize a familiar face in a blink of an eye without conscious effort or forethought (Tanaka, 2001; Diamond & Carey, 1986). However, the past decade has seen a number of behavioral studies suggesting that, like most human traits and capacities, the ability to recognize face identities lies on a continuum (e.g., Rhodes, Jeffery, Taylor, Hayward, & Ewing, 2014; DeGutis, Wilmer, Mercado, & Cohan, 2013; Dennett, McKone, Edwards, & Susilo, 2012; Wilmer et al., 2012; Germine, Duchaine, & Nakayama, 2011; Wilhelm et al., 2010; Bowles et al., 2009; Russell, Duchaine, & Nakayama, 2009). On one end of the continuum are the “super-face recognizers” who can identify a vast catalog of faces after a single presentation under limited viewing conditions (Bobak, Bennetts, Parris, Jansari, & Bate, 2016; Russell et al., 2009). On the other end are people who have experienced no brain damage or trauma and yet have a lifelong impairment recognizing the faces of close friends and relatives (“developmental prosopagnosia” or “congenital prosopagnosia”; Duchaine & Nakayama, 2004, 2006a; Behrmann & Avidan, 2005). Within the extremes, face recognition performance as measured behaviorally is a normally distributed skill that is mediated by factors such as age (Germine et al., 2011; Hildebrandt, Sommer, Herzmann, & Wilhelm, 2010), sex (Sommer, Hildebrandt, Kunina-Habenicht, Schacht, & Wilhelm, 2013), or sociability (Halliday, MacDonald, Scherf, & Tanaka, 2014; Bate, Parris, Haslam, & Kay, 2010).

Individual differences in face identity processing are typically measured explicitly using behavioral methods. The most widely used behavior test is the Cambridge Face Memory Test (CFMT; Duchaine & Nakayama, 2006b), which requires participants to first memorize faces and then to recognize each of those faces among other foil faces under different viewing conditions (e.g., viewpoint changes, with added noise). In standard ERPs extracted from EEG recordings, the most prominent signature of neural activities related to face processing is the N170 component. Faces elicit a robust negative response in the brain wave approximately 170 msec after the onset of the face stimuli (Bentin, Allison, Puce, Perez, & McCarthy, 1996), and this N170 response is larger to faces than nonface objects with the largest difference observed at right occipital-temporal locations on the scalp (see Rossion & Jacques, 2011, for a review). Kaltwasser, Hildebrandt, Recio, Wilhelm, and Sommer (2014) and Herzmann, Kunina, Sommer, and Wilhelm (2010) used structural equation modeling techniques and found a modest relation across individuals between the N170 latency (but not amplitude) and behavioral performance of an explicit face recognition task using an old/new recognition paradigm.

However, whether individual differences in face identity processing can be captured and quantified without an explicit behavioral task remains an open question. This is an important issue because certain individuals may have difficulties understanding task instructions (young children, neurological or psychiatric populations, etc.), and the individual differences in performance at the CFMT or other explicit behavioral tests can be greatly influenced by factors such as memory, motivation, attention, decision processes, and so forth. Moreover, although it is often stated that faces can be recognized in a blink of an eye (i.e., a “single glance”; Hsiao & Cottrell, 2008; Willis & Todorov, 2006), faces are shown for a long (or an unlimited) period in explicit behavioral tests such as the CFMT or the widely used Benton Face Recognition Test (Benton & Van Allen, 1968). Hence, whether individual differences in face identity processing can be captured in a task where only one gaze fixation is allowed on a face remains unknown.

To shed light on these questions, the current study employed a fast periodic visual stimulation (FPVS) paradigm coupled with EEG recording to measure individual differences in face identity processing (Rossion & Boremanse, 2011). In the specific FPVS oddball paradigm used here (Liu-Shuang, Norcia, & Rossion, 2014), same identity face stimuli are presented at a fast fixed rate (6 Hz, or 6 images/sec, general stimulation frequency), allowing only one gaze fixation per face. Such periodic stimulation elicits a periodic response at the same driving frequency and its harmonics (often called “steady-state visual evoked potentials”), which can be captured using scalp EEG recordings (Regan, 1966, 1989; see Norcia, Appelbaum, Ales, Cottereau, & Rossion, 2015, for a review). Importantly, every fifth image in this stimulation sequence shows a different face identity, so that face identity changes occur at a rate of 1.2 Hz (Figure 1). In these conditions, the presence of a periodic response at 1.2 Hz and its harmonics (i.e., individuation response) directly reflects the detection of face identity changes (i.e., individuation response). In addition, the face individuation response captured in this way reflects high-level face-specific processing. For example, the individuation response resists large changes of stimulus size, but its magnitude is largely and significantly reduced with inversion and contrast reversal (Liu-Shuang et al., 2014), two image manipulations known to disproportionately affect perceptual processing of faces relative to other object categories (Yin, 1969; Galper, 1970, respectively). Moreover, a recent study showed that a specific impairment in face identity processing could be evidenced through this approach. More precisely, although all typically developed participants showed significant face individuation responses, a well-documented patient with acquired prosopagnosia (Rossion et al., 2003) showed no face individuation responses while her general visual response remained intact relative to these age-matched participants (Liu-Shuang, Torfs, & Rossion, 2016).

FPVS offers several additional advantages for measuring individual differences in face identity processing. First, FPVS measures face identity processing directly, without having to subtract or regress out any control condition responses. Most studies measuring face individuation in electrophysiology have used adaptation paradigms in which the response to a series of repeated face identities is compared with responses to different face identities (e.g., Jacques, d'Arripe, & Rossion, 2007; Heisz, Watter, & Shedden, 2006). In contrast, our FPVS paradigm contains an inherent subtraction such that responses at 1.2 Hz and harmonics are directly driven by the perceived change of face identity. Second, the data analysis is objective because only responses at the predefined frequency of stimulation and its harmonics need to be considered. This is one of the advantages of FPVS-EEG and frequency domain analysis over standard ERP measures because no arbitrary definition of the component time window is needed, which can be quite subjective and different across studies. Third, the periodic FPVS signal can easily be segregated from artifacts and spontaneous brain activity (Srinivasan, Bibi, & Nunez, 2006; Regan, 1989). More precisely, providing that a high-frequency resolution is obtained by recording and analyzing data from a sufficiently long time window, the signal in the EEG spectrum is found in a few specific and very narrow frequency bins corresponding to the stimulation frequencies (i.e., if stimulated at 6 Hz, responses will be found in frequency bins of 6 Hz and its harmonics only). In contrast, the background EEG noise and artifacts (alpha waves, blinks, muscle potentials, etc) are not periodic and will be distributed across many frequencies (broadband), thus having a reduced impact on the frequencies of interest (Rossion, 2014; Regan, 1989). As a result, FPVS signals are more resilient to noise as compared with ERP and require less strenuous preprocessing and artifact rejection steps, providing a measure of perceptual processing that is minimally transformed from the original data (Rossion, 2014). Therefore, only a small number of sequences are necessary to obtain a significant response, an advantage that is valuable in studies of individual differences in some specific populations (children, infants, people with cognitive disabilities, etc.). Fourth, the short presentation time of each face (e.g., 167 msec per face with a 6-Hz presentation rate) prevents extraneous saccades and cognitive processes from interfering with face identity processing, which only requires a very brief presentation time (Alonso-Prieto, Van Belle, Liu-Shuang, Norcia, & Rossion, 2013). Last but not least, the FPVS response can be obtained without contamination from decisional or motor processes, because the observers do not have to perform any explicit face identity processing tasks (e.g., Liu-Shuang et al., 2014). Therefore, based on these advantages, we hypothesize that FPVS will allow for sensitive and reliable measures of individual face identity processing abilities.

In summary, the purposes of the current study were to (1) investigate whether the FPVS approach is sensitive to individual differences in face identity processing and (2) study the relationship between the individual differences in this task-independent neurophysiological measure captured with FPVS and an explicit behavioral measure with CFMT.

We tested 49 participants (24 men, all right-handed) who were recruited from the University of Victoria with compensation of bonus course credit. All of the participants were white and reported normal or corrected-to-normal vision and no history of brain injury. The average age of the participants was 21.93 years with a standard deviation of 3.69 years. The experiment and consenting procedures were approved by the human research ethics committee of University of Victoria.

Participants first performed the CFMT before they went to the EEG laboratory. In the CFMT, participants first completed an introductory phase in which they memorized six target faces. A target face was first presented from three different views (front, right profile, left profile) for 3 sec per view. Participants then performed 3 three-alternative forced-choice trials, with one trial for each of the three views. The process was repeated for the remaining five faces, resulting in 18 trials. Next, participants studied frontal views of the same six target faces for 20 sec and then were presented with 30 forced-choice test displays in the no-noise phase. Each display contained one target face and two distractor faces. Participants were told to select the face that matched one of the original six target faces. The matching faces varied from their original presentation in terms of lighting condition, pose, or both. Next, participants completed the noise phase of the task where they were presented with the six target faces to study, followed by 24 test displays presented in Gaussian noise. The data from the trials in the no-noise phase and noise phase were used to calculate the behavioral face recognition performance scores. On average, participants took 10–15 min to perform the CFMT.

Stimuli used in the FPVS experiment consisted of full-front colored photographs of 25 male and 25 female faces with neutral facial expression (see Liu-Shuang et al., 2014). Each face picture was taken under the same conditions of lighting and background and with the same face-to-camera distance. External features such as hair and ears were cropped out using Adobe Photoshop, and the isolated faces were put against a neutral gray background. Final images were resized to a height of 250 pixels and width from 175 to 197 pixels. The average visual angle of the stimuli was 7.1° in height by 5.2° in width at a 58-cm viewing distance and 1280 × 1024 display resolution.

The experiment was composed of four 60-sec sequences, with two sequences presenting all female faces and two sequences presenting all male faces. Half of the participants were presented the sequences with the order of female-male-male-female, and the other half were presented with the order of male-female-female-male. Within each sequence (Figure 1A), face pictures were presented six times per second (frequency = 6 Hz) through sinusoidal contrast modulation using a custom script running with Psychtoolbox in MATLAB 2010 (The Mathworks, Inc., Natick, MA) (e.g., Liu-Shuang et al., 2014; Rossion & Boremanse, 2011). Each cycle lasted 166.7 msec and began with a uniform gray background from which a face appeared as its contrast increased. Full contrast was reached at 83.3 msec and then decreased at the same rate. The size of each face varied randomly between 80% and 120% in 2% steps at every cycle of presentation to minimize low-level face feature overlap. In every sequence, a fixation cross first appeared on the screen for 2–5 sec (randomly jittered across sequences), after which faces were shown at a rate of 6 Hz for 60 sec. Within the 60 sec of stimulation, one face was randomly selected as the “base” face (i.e., Face A) repeating throughout the sequence. Every fifth face was replaced by different identity “oddball” faces, randomly selected from the remaining 24 faces of the same sex (i.e., Faces B, C, D…; Figure 1B), forming a stimulation pattern of AAAABAAAACAAAADAAAAEAA…. As a result, participants were presented with a face six times every second (general base frequency rate, frequency = 6 Hz) and a face identity change five times every 6 sec (individuation frequency, frequency/5 = 1.2 Hz). Responses at the general base frequency rate and its harmonics reflect general visual processing, whereas responses at the individuation frequency harmonics reflect face identity processing. Participants performed an orthogonal task in which they were asked to detect the brief (200 msec) color change (from red to blue) of a fixation cross situated in the center of the face stimuli below the eyes, to ensure that they maintained a constant level of attention. Color changes occurred randomly in each sequence for eight times. No explicit instructions were given to the participants to pay attention to the face identities, and although all of the participants mentioned that they noticed the face identity change, none of them reported that the identity change was periodic. Participants performed at ceiling for this task, with an accuracy of 98.8% (95% CI [98.3%, 99.2%] and response time of 523.5 msec (95% CI [492.7, 554.4]).

The EEG was recorded using a montage of 36 electrode sites in accordance to the extended International 10–20 system (Jasper, 1958). Signals were acquired using Ag–AgCl ring electrodes mounted in a nylon electrode cap with an abrasive, conductive gel (EASYCAP GmbH, Herrsching-Breitbrunn, Germany). Signals were amplified by low-noise electrode differential amplifiers with a frequency response of DC 0.017–67.5 Hz (90-dB octave roll-off) and digitized at a rate of 250 Hz. Digitized signals were recorded to disk using Brain Vision Recorder software (Brain Products GmbH, Munich, Germany). The impedances were maintained below 10 kΩ. The EEG was recorded using the average reference.

EEG signal was processed using Letswave 5 (Mouraux & Iannetti, 2008) and MATLAB 2012. EEG data were 0.10- to 100-Hz band-pass filtered using a Butterworth filter with a slope of 24 dB/octet. All channels were rereferenced to an average reference. The continuous EEG recording of each participant was cropped into four 60-sec segments time-locked to the onset of the first cycle of presentation and the offset of the last cycle of presentation in each stimulation sequence. For the calculation of responses at the level of individual participants, data of the four sequences were first averaged in the time domain to reduce EEG noise (i.e., activity unrelated to the stimulation). A fast Fourier transform was then applied, and frequency amplitude was extracted at a resolution of 1/60 = 0.017 Hz. However, for the purposes of reliability calculation (see Reliability Analysis section), fast Fourier transform was also applied separately to each of the four sequences in the time domain. Baseline-corrected amplitudes were calculated following the method used in the previous studies (e.g., Liu-Shuang et al., 2016; Retter & Rossion, 2016; Dzhelyova & Rossion, 2014a). More precisely, at each frequency bin, the mean amplitude of the 20 surrounding frequency bins (10 on each side, excluding the immediately adjacent bin) was subtracted from the frequency bin of interest.

Group level z scores were used to determine the range of frequency harmonics to take into account and were calculated using grand-averaged frequency amplitudes pooled across all channels. The mean and standard deviation of the 20 frequency bins (10 on each side, excluding the immediately adjacent bin) surrounding the frequency of interest were used to calculate the z score. A conservative threshold of z > 3.1 (p < .001, one-tailed; i.e., signal > noise) was used, in line with previous research (Liu-Shuang et al., 2016). After the selection of the relevant harmonic frequencies, the face individuation response and the general visual response were quantified as the sum of these harmonics (Retter & Rossion, 2016).

The internal reliability of behavioral and EEG measures was calculated with Cronbach's alpha, using the individual items of the CFMT and individual sequences of the FPVS experiment.

General visual responses were found at 6 Hz and its harmonics (12 Hz, 18 Hz, …) in all participants. At the group level, responses were significant until the sixth harmonic (36 Hz; see Table 1A). We therefore considered the first six harmonics for analysis. As shown in Figure 2A, general visual responses had a medial occipital scalp distribution. For each participant, the aggregated general response was quantified as the sum of baseline-corrected amplitudes of the first six harmonics on channels Oz and POz. The internal consistency reliability (Cronbach's alpha) of this general visual response was .98 across the four individual sequences.

For the face individuation response, significant peaks were found at 1.2 Hz and its harmonics (2.4 Hz, 3.6 Hz, …) and remained significant at the group level until the sixth harmonic (7.2 Hz). On the basis of previous studies as well as the topography of individual harmonics in the current study, data were grouped into two occipitotemporal regions (right hemisphere: P8, PO8, TP10; left hemisphere: P7, PO7, TP9; Figure 2B). Despite the short recording time (i.e., 4 min), significant individuation responses were found at these regions in all participants but two. We quantified the face individuation response in each individual participant by summing the baseline-corrected amplitudes of the first six harmonics, excluding the fifth harmonic as it was confounded with the 6-Hz general frequency. At the group level, a t test showed that the aggregated individuation responses were significantly larger (t(48) = 2.43, p = .01, one-tailed) over the right occipital–temporal channels (M = 1.15, 95% CI [0.93, 1.37]) compared with the left occipital–temporal channels (M = 0.89, 95% CI [0.70, 1.09]). The internal consistency reliability (Cronbach's alpha) for the individuation response score across four sequences was .87 when the responses were aggregated from both the left and right occipital–temporal locations.

From visual inspection (Figure 3), individual differences are evident in the individuation response. The magnitude of the individuation response varied from 0.17 to 5.43 μV across participants, as can be seen from the histogram in Figure 4A. Regarding hemispheric lateralization, 18 of 49 participants (37%) showed left lateralization (below the diagonal line on Figure 4B). Upon closer inspection, the proportion of left-lateralized responses was larger for female (11/25 = 44%) compared with male (7/24 = 29%) participants, although this difference was not significant (χ² = 1.16, p = .28). To take this individual difference of hemispheric lateralization into account for further analyses, we aggregated the face individuation responses over the left and right occipitotemporal channels.

The averaged accuracy in the CFMT task was 74.1% (95% CI [69.7%, 78.5%]), with an accuracy of 77.6% (95% CI [73.2%, 82.0%]) in the no-noise phase and 69.6% (95% CI [64.2%, 75.0%]) in the noise phase. The internal reliability (Cronbach's alpha) for the CFMT based on the sample of the current study was .89 (54 items), with the reliabilities for the no-noise and noise phases of .86 (30 items) and .78 (24 items), respectively.

Pearson correlation coefficients between the general response, the individuation response, and the CFMT performance were calculated. The general response measured from the medial occipital channels was not correlated with the individuation response measured from occipitotemporal channels (r = .15, p = .29, 95% CI [−0.13, 0.41]). Moreover, there was no significant correlation between the general response and the CFMT performance (r = −.07, p = .65, 95% CI [−0.34, 0.25]). However, the CFMT performance significantly correlated with the aggregated individuation response from the left and right occipitotemporal channels (r = .30, p = .04, 95% CI [0.02, 0.54]; Figure 5).

The current study used an FPVS-EEG approach (Liu-Shuang et al., 2014; Rossion & Boremanse, 2011) to measure individual differences in face identity processing. We presented participants with face stimuli at a fast rate of 6 Hz (general base frequency rate), with face identity changes occurring every fifth face (1.2 Hz, individuation frequency). Within only 4 min of EEG recording, we found general responses at 6 Hz and harmonics and lrobust face individuation responses at 1.2 Hz and harmonics, thus replicating findings from previous studies (Liu-Shuang et al., 2014, 2016; Dzhelyova & Rossion, 2014a, 2014b). The magnitude of the face individuation response, but not of the general response, significantly correlated with behavioral performance in the CFMT.

Although the FPVS-EEG approach has been used in a number of studies to measure individualization of faces (see Rossion, 2014, for a review; Liu-Shuang et al., 2016, for the most recent study), it remained unknown whether it was sensitive enough to elicit reliable brain response from most individuals in a few minutes and whether it was sensitive to the individual differences in face identity processing in the normal population. Although most of the previous studies used a relatively small sample (e.g., 10–15 participants), the current study recruited a larger sample of individuals (n = 49) and showed that we were able to obtain significant face individuation response in all but two participants using a conservative threshold (i.e., the signal in the frequency bin of interest is significantly larger than the surrounding noise, at the level of z > 3.1). In addition, this response was consistently localized over the occipitotemporal channels in most of the participants. Yet, our measures showed significant individual differences in amplitude and lateralization pattern.

Interestingly, there was shared variance (9%) between individual EEG measures and behavioral performance on the CFMT, although a number of factors contribute to these measures. Nevertheless, the relatively weak correlation coefficient suggests that the two measures reflect different aspects of the face identity processing. It should be emphasized that the weak correlation cannot be attributed to low internal reliability of the measures. The fidelity of the measures is crucial because it constrains the overall magnitude of their correlation (Wilmer et al., 2012; Schmidt & Hunter, 1996). In previous studies, it has been shown that the CFMT has high internal consistency reliability (.86 < α < .90) in measuring face recognition ability (Wilmer et al., 2010), and this was replicated in the current study (α = .89). The current study also provided evidence, for the first time, that the FPVS-EEG measure has high reliability (α = .87 for the individuation response), despite the fact that it was obtained by only four 60-sec sequences of recording. Most impressively, different base faces were used in each stimulation sequence, and none of the sequences were identical. This high reliability may be attributed to the large number of variable individual face discrimination measures (i.e., 72 in a 60-sec stimulation sequence) contributing to the overall face individualization response in each sequence.

Because both the CFMT and the FPVS measures had high internal consistency reliability, the obtained correlation between the EEG marker of facial identity and the CMFT performance can be assumed to be robust and relatively noise free. Thus, the modest size of the correlation between these two measures can rather be explained by them targeting different aspects of face identity processing. On the one hand, the CFMT is an explicit and cognitively complex memory task that tests memory for face identity. Performance on the CFMT requires attention, memory (encoding, storage, and retrieval), and decision-making that go beyond the mere perceptual differentiation of face identity. Moreover, individual differences in the speed of face identity recognition are not considered in the CFMT measure, despite the fact that it is an important aspect of individual differences in this function (Wilhelm et al., 2010). On the other hand, the EEG individuation response captures face identity discrimination processes, without using an explicit behavioral task, involving posterior regions of the visual extrastriate cortex, and is thus closer to isolating perceptual processes. Moreover, in FPVS, the visual system is put under greater constraint such that face identity needs to be extracted at a single glance. Thus, we speculate that the shared variance between FPVS and CFMT indicates their common operation of processing face identity, whereas the unaccounted variance is likely due not only to the additional cognitive operations that are necessary for the CFMT task (decision-making strategies, memory, attention, etc.) but also to potential irrelevant factors contributing to the magnitude of the EEG responses obtained with FPVS (skull thickness, orientation of the sources due to cortical folding, etc.).

The current study is the first to study the correlation between the neural responses obtained using FPVS and behavioral measures in face processing. Future studies should correlate the FPVS responses and the performance of a behavior face perception task that requires participants to process faces in a briefer time duration. In addition, showing that the FPVS response correlates more with behavioral performance with faces than nonface object processing tasks can be another important evidence for the validity of the FPVS paradigm. Moreover, future studies could focus on isolating a “real” face identity processing response of the behavioral tasks, for instance, by using the shared variance of multiple face recognition tasks requiring different components of the face processing system (Bruce & Young, 1986) and considering both speed and accuracy (e.g., Sommer et al., 2013; Hildebrandt, Wilhelm, Schmiedek, Herzmann, & Sommer, 2011; Hildebrandt et al., 2010; Wilhelm et al., 2010). Aggregation of the responses from several measurements can average out the noise in each measurement and thus provide a clear measure of the cognitive processes of interest (Rushton, Brainerd, & Pressley, 1983). For example, the latent variable approach employed in the studies by Herzmann et al. (2010) and Kaltwasser et al. (2014) using structural equation modeling can be used to abstract from task-specific variance and measurement error. Those techniques are likely to provide a more valid estimation of brain–behavior relationships in face processing. Last but not least, in the EEG measure, future study can further improve the validity of the face individuation response by regressing out a weak individuation response to control stimuli such as inverted faces (Liu-Shuang et al., 2014) in the FPVS.

The current findings have critical practical implications. FPVS is a promising new technique that reveals individual differences in rapid individual face discrimination as indicated by a reliable, valid, objective, task-independent, and direct measure of brain activity (EEG). It provides opportunities to easily test large samples of participants from various backgrounds without confounding perceptual abilities with general cognitive abilities. Moreover, responses to a same experiment can be easily compared across development and between various neuro-typical and neuro-atypical populations.

This research was supported by the Chinese Scholarship Council fellowship awarded to Buyun Xu, a postdoctoral grant from the Belgian National Foundation for Scientific Research (no. 1230016F) awarded to Joan Liu-Shuang; PDR T.0207.16 FNRS and ERC facessvep 284025 grants awarded to Bruno Rossion; and the Temporal Dynamics of Learning Center (NSF grant no. SBE-0542013) and the National Sciences and Engineering Research Council of Canada grants awarded to James Tanaka. We thank Adriano Boremanse and Benvenuto Jacob for technique supports for setting up the toolboxes and computer for the FPVS testing and Bonnie Heptonstall and Alison Campbell for data collection. We also thank the action editor and two anonymous reviewers for their helpful comments.

Reprint requests should be sent to Buyun Xu, Department of Psychology, University of Victoria, Cornett A191, 3800 Finnerty Rd., Victoria, BC, Canada, or via e-mail: [email protected].