Abstract

In social interactions, it is often necessary to rapidly encode the association between visually presented faces and auditorily presented names. The present study used event-related potentials to examine the neural correlates of associative encoding for multimodal face–name pairs. We assessed study-phase processes leading to high-confidence recognition of correct pairs (and consistent rejection of recombined foils) as compared to lower-confidence recognition of correct pairs (with inconsistent rejection of recombined foils) and recognition failures (misses). Both high- and low-confidence retrieval of face–name pairs were associated with study-phase activity suggestive of item-specific processing of the face (posterior inferior temporal negativity) and name (fronto-central negativity). However, only those pairs later retrieved with high confidence recruited a sustained centro-parietal positivity that an ancillary localizer task suggested may index an association-unique process. Additionally, we examined how these processes were influenced by massed repetition, a mnemonic strategy commonly employed in everyday situations to improve face–name memory. Differences in subsequent memory effects across repetitions suggested that associative encoding was strongest at the initial presentation, and thus, that the initial presentation has the greatest impact on memory formation. Yet, exploratory analyses suggested that the third presentation may have benefited later memory by providing an opportunity for extended processing of the name. Thus, although encoding of the initial presentation was critical for establishing a strong association, the extent to which processing was sustained across subsequent immediate (massed) presentations may provide additional encoding support that serves to differentiate face–name pairs from similar (recombined) pairs by providing additional encoding opportunities for the less dominant stimulus dimension (i.e., name).

INTRODUCTION

The ability to rapidly encode an individual's face with their spoken name is an essential component of successful social interaction. Yet, despite the apparent automaticity with which we are able to determine the familiarity of a previously presented face (Palermo & Rhodes, 2007; Heisz, Watter, & Shedden, 2006), encoding and retrieving the name to which that face is associated often requires more effortful, strategic processing (Scanlan & Johnston, 1997). As a result, this task is known to prove particularly frustrating for individuals with even mild age-related memory impairment (James, 2004; Naveh-Benjamin, Guez, Kilb, & Reedy, 2004). Given their relative difficulty, relevance to everyday cognition, and susceptibility to cognitive impairment, face–name associations are an ideal stimulus for examining neurocognitive mechanisms underlying the binding of multimodal elements into episodic memories. Yet, although several recent fMRI studies have used visual face–name associations as to-be-learned stimuli to investigate hippocampal contributions to episodic encoding (Chua, Schacter, Rand-Giovannetti, & Sperling, 2007; Kirwan & Stark, 2004; Sperling et al., 2001, 2003; Small et al., 2001), to date, only one study has used auditory names and/or event-related potentials (ERPs) to characterize the time course of neocortical activity associated with this process (Guo, Voss, & Paller, 2005).

In addition, although clinical research has shown that face–name memory deficits are amenable to improvement from encoding strategies including imagery, elaboration, and repetition (Manasse, Hux, & Snell, 2005; Neuschatz, Preston, & Toglia, 2005; Hux, Manasse, Wright, & Snell, 2000), to our knowledge, studies specifically exploring the relationship between these strategies and neural correlates of face–name encoding have been limited to a single study of spaced repetition effects (Rand-Giovannetti et al., 2006). Thus, to extend our understanding of the neural substrates of face–name encoding, the present study aims to first characterize the study-phase ERP activity associated with high-quality discrimination of face–name associations from lower-quality discrimination and complete recognition failures (misses), then explore how these processes may vary across three successive repetitions (i.e., massed repetition).

Many studies suggest that successful encoding of face–name associations involves unique activity that cannot be explained simply as the sum of encoding the constituent elements. For example, Naveh-Benjamin et al. (2004), in comparing young and older adults' memory for faces, names, and face–name associations, found that recognition of faces and names in isolation was almost comparable across the two age groups, yet older participants were disproportionately poorer at recognizing the associations among them, suggesting that this particular binding process could be affected independently of memory for the items themselves. fMRI studies have confirmed a role for the anterior hippocampus in processes unique to the encoding of associative information, including face–name associations (Chua et al., 2007; Kirwan & Stark, 2004; Sperling et al., 2001, 2003; Small et al., 2001). However, association-specific activity is not limited to the hippocampus (or other limbic regions), as there is also evidence accruing that successful encoding of different stimulus features activate cortical areas responsible for the on-line processing of those features as well as unique areas (i.e., intraparietal sulcus) specifically involved in perceptual binding (Uncapher, Otten, & Rugg, 2006).

Results from ERP studies of multimodal face–name encoding are more ambiguous with regard to identification of activity unique to associative processes. Guo et al. (2005) recorded ERPs during encoding of visual face and auditory name pairs and sorted this activity as a function of subjects' subsequent memory for faces and names alone, and for their association. In their first experiment, all three old/new recognition memory tests were administered in succession, with the final face–name recognition task only testing “old” pairs, even though subjects were led to believe that half of the pairs were old and half were new. A centro-posterior positivity at 400–800 msec at encoding predicted subsequent memory for faces, whereas a fronto-central negativity at 200–600 msec predicted memory for names, but no significant subsequent memory effects for associative memory emerged. In a second experiment, the memory task was simplified with a blocked study–test design where shorter sets of face–name pairs were followed by both an immediate free recall test for the names and a matching test for face–name associations; a final face and name recognition test then occurred at the end of all blocks. In this case, both later remembered faces and face–name pairs elicited greater positive potentials with a similar centro-parietal topography and time course as in their first experiment. Although a centro-parietal positivity from 200 to 800 msec also appeared to differentiate successfully recalled names, this effect did not reach significance. Thus, activity associated with associative encoding of faces and names either failed to emerge or appeared as a quantitative enhancement of activity necessary for encoding of the face or name in isolation, rather than a qualitatively unique process.

In the present study, we approached the question of the neural correlates of successful face–name encoding by testing memory with intact and rearranged pairs of faces and names, a method of isolating associative memory that renders mere item familiarity a poor retrieval cue (Tsivilis, Otten, & Rugg, 2001; Donaldson & Rugg, 1998, 1999). As with most studies of successful encoding, we recorded neural activity during initial stimulus processing and then backsorted this activity as a function of retrieval success with the goal of differentiating activity elicited by later retrieved and forgotten face–name associations (e.g., difference due to memory [DM] effects; Paller, Kutas, & Mayes, 1987). We note that this approach to studying successful encoding has several caveats, namely, that any observed differences could simply reflect the greater memorability of certain stimuli rather than more stimulus-independent effects, or index general processes that are correlated with effective encoding but do not directly contribute to building a memory trace (i.e., attention). Finally, study-phase differences are heavily influenced by how successful retrieval is defined and may not be generalizable beyond a particular type of test. With these concerns in mind, we considered memory success to be a function of the quality of the retrieval performance. We attributed the highest level of face–name discrimination to cases in which subjects correctly recognized the “true” old pair with high confidence (HC) and rejected the two corresponding “false” new pairs (i.e., HC “perfect hits”). Participants also made a fair number of lower-confidence (LC) perfect and “imperfect” hits, the latter condition corresponding to situations where subjects recognized the true pair with LC, but failed to reject one or more of the false pairs. We look at this category as representing a lower quality of face–name discrimination than the “perfect hits.” Nonetheless, given that face–name recognition is often imperfect in everyday cognition (i.e., face is recognized, name is forgotten, misidentified, or retrieved with less confidence), we felt this was a valid condition to explore. Both of these categories were compared to “misses,” in which the subject failed to recognize the true pair, regardless of response to the false pair.

Our use of a single test of associative memory reduces confounds that might arise from multiple exposures to stimuli at test. However, in exchange for a purer measure of associative encoding, we lose the ability to directly isolate the contributions of individual item processing. Thus, we will address these contributions indirectly, first by examining our results for evidence of the earlier-onset frontal negativity and later-onset posterior positivity that Guo et al. (2005) argued were related to auditory name and visual face encoding, respectively, and secondly by conducting our own ancillary “localizer” experiment. In the localizer experiment, face and auditory name stimuli used in the primary experiment were presented individually, intermixed with modality-specific control stimuli, in the context of a 1-back working memory task. Localizers have been reliably used in fMRI studies of object recognition to define regions differentially associated with processing of particular object types (Saxe, Brett, & Kanwisher, 2006). Here we explore the possibility of extending this fundamental logic to ERP. We stress that the goal of the localizer was not to define separate underlying sources of auditory and visual processing per se, but to provide converging evidence together with results from Guo et al. for ERP scalp distributions that might be differentially related to item processing in one modality or another.

Although it is not uncommon for localizer and primary tasks to be somewhat different in fMRI studies, we acknowledge that direct comparability between topographies observed in the localizer and the primary associative memory task is limited by differences in task demands and memory systems engaged. Nonetheless, there also appear to be many commonalities in the basic sensory–perceptual processes involved in working memory and long-term memory (Summerfield & Mangels, 2006; Ranganath & Blumenfeld, 2005). Indeed, a recent study found stimulus-specific slow cortical potentials during working memory maintenance that were also predictive of long-term memory formation (Khader, Ranganath, Seemuller, & Rosler, 2007). Our localizer and primary task both involve identical stimuli, have a high degree of similarity in their timing and manner of presentation, and require deep, sustained processing of stimuli. The localizer task, however, places minimal demands on associative encoding across modalities. Thus, based on the view that successful encoding of information into either working memory or long-term memory will commonly elicit sustained activity in modality-specific sensory–perceptual regions, we suggest that it may be possible to use cross-task commonalities to help understand the general spatio-temporal distribution of ERPs associated with item-specific processing of our face and name stimuli.

Finally, we wished to explore the relationship between repetition and successful encoding by examining subsequent memory effects as a function of repetition across three sequential presentations (i.e., massed repetition). Although the effects of repetition priming on face and name items have been explored previously (Guillaume et al., 2009; Martin-Loeches, Sommer, & Hinojosa, 2005; Itier & Taylor, 2004; Schweinberger, Pickering, Burton, & Kaufmann, 2002), there has been little work thus far examining the specific effects of repetition on the episodic encoding of face–name associations. In one of the few studies on this topic, Rand-Giovannetti et al. (2006) used fMRI to investigate the neural bases of distributed face–name encoding in older adults. In that study, subjects memorized face–name pairs, each of which was presented three times at randomly varying lags. Whereas the first presentation of later remembered items induced activation of the hippocampus and multiple neocortical regions, including prefrontal, parietal, and fusiform cortices, in the second and third presentations, activations were limited to neocortical areas, suggesting a shift in encoding strategy. Due to a very low number of incorrect trials, however, no proper subsequent memory comparisons could be carried out. Thus, it was only possible to speculate that a beneficial effect of stimulus repetition at encoding might be related to the enhanced activity in the networks indexed by sustained neocortical engagement. Moreover, only older subjects were tested, and although memory performance was high and did not decline with delay, it is possible that these patterns represented compensatory activity rather than activity typical of young adults.

Spacing of the repetitions across intervening items (i.e., distributed repetition) generally improves episodic memory performance more than immediately sequenced repetition (i.e., massed repetition), particularly with regard to the recollective experience at retrieval (Mantyla & Cornoldi, 2002; Parkin, Gardiner, & Rosser, 1995). However, at least one study has shown that massed presentation enhances explicit memory (Challis & Sidhu, 1993). We also note that particularly for face–name associations, massed repetition holds somewhat greater ecological validity than distributed repetition, in that during personal introductions, mnemonic strategies often involve successive covert (or overt) repetition of the name while still in the presence of the individual's face. Indeed, it is often important for optimal social interaction to have successfully encoded the face and name during this initial introduction so that no further questioning of the name is needed when the individual is encountered later (i.e., distributed encoding is less socially desirable). To date, we know of no neuroimaging studies that have examined the relationship between massed repetition and successful episodic encoding of face–name associations, or even stimulus–stimulus associations more generally.

METHODS

Participants

We tested 24 Columbia University undergraduate students (age range: 18–23 years; 12 women). All were right-handed, native English-speakers, with normal or corrected-to-normal vision and no history of neurological disorders. All subjects gave informed consent to participate in the study, which was approved by Columbia University Institutional Review Board. To ensure 10 or more trials in each condition for ERP analyses, seven subjects were excluded because of low trial counts in one of the three categories of recognition performance (3 in HC hits, 2 in LC hits, and 2 in guesses–misses [GM]). Here, we will report behavioral and ERP results drawn on the final sample (n = 17; 9 women).

Stimuli

We used a set of 180 gray-scale photographs of unfamiliar faces derived from various sources including the AR database (Martinez & Benavente, 1998). Each photo was 300 by 300 pixels in size and was presented against a black background on a computer screen. Subjects were 60 cm from the screen, and the face stimuli subtended a visual angle of 7.6° by 7.6°. Faces were randomly matched with 180 moderate-frequency names spoken in a male voice and delivered through headphones with volume adjusted for subject comfort. Names were from the 1990 U.S. Census database and all were two syllables in length. Blocks of both female and male face–name pairs were included, but each block exclusively contained stimuli of the same gender.

Procedure

The experiment included twelve study–test blocks. In the study phase of each block, 15 unique face–name pairs were presented, randomly selected from the stimulus pool. During presentation, the face and name onset simultaneously, with face being shown for 1000 msec and the name taking an average of 1000 msec. Each face–name pair was repeated thrice consecutively, with each repetition separated by a 500-msec blank period. A crosshair was presented for 1500 msec between each face–name triplet sequence (Figure 1A). At the end of each study block, participants counted backward by 3 from an arbitrarily chosen three-digit number for about 20 sec. Following this retention interval, 15 old (true) and 15 new (false) face–name pairs were presented for a recognition memory test. The false pairs consisted of randomly rearranged face–name pairs from the same study block (for examples, refer to Figure 1B). Order of presentation of true and false pairs was also randomized across participants. The use of rearranged pairings ensured that later correct recognition responses could not be made simply by relying on familiarity for either the face or the name alone. Rather, accurate recognition required the episodic retrieval of the whole face–name pair. For each test item, participants provided an integrated recognition and confidence judgment using a 7-point scale (1 = sure old, 4 = guess, 7 = sure new). Subjects had an unlimited time to enter their responses.

Figure 1. 

(A) Example of item presentation sequence and timing during the study phase. (B) Example of true and false pairs included in the recognition memory test.

Figure 1. 

(A) Example of item presentation sequence and timing during the study phase. (B) Example of true and false pairs included in the recognition memory test.

EEG Recording

Continuous EEG was recorded throughout the experimental session with a sintered Ag/AgCl 64-electrode Quick-Cap and amplified using Neuroscan Synamps 2 with an A/D conversion rate of 500 Hz and bandpass of DC-100 Hz. Impedance was kept below 11 kΩ. EEG was initially referenced to Cz then converted to an average reference off-line. We compensated for blinks and other eye-movement artifacts using 2–6 PCA-derived ocular components (BESA 5.1.8). Data were also manually screened for excessive muscle artifacts or slow wave drifts. For averaging, the EEG was cut into epochs time-locked to each presentation of face–name pairs (−200 to 1000 msec). Following baseline correction to the 200-msec interval preceding the stimulus, a 30-Hz low-pass and 0.2-Hz high-pass filter (6 dB/octave roll-off, zero phase shift) were applied and remaining epochs containing excessive noise (±100 mV) were rejected before averaging.

Data Analysis

Behavioral Data

Recognition responses were classified taking into account the dimensions of confidence and accuracy. For each subject, recognition responses were classified as HC, LC, or guess. To account for individual bias in the use of the response scale, cutoffs for HC and LC were calculated individually for each subject. We first averaged the confidence ratings given by each subject for all items endorsed as old (i.e., Responses 1–3), regardless of whether they were correct or incorrect. For each subject, values greater than this average were labeled as HC. The same computation was carried out for the items endorsed as new (i.e., Responses 5–7). Across the group, the mean rating for old responses was 1.52 (SD = 0.24) and for new responses was 6.60 (SD = 0.22), suggesting that subjects exhibited a more or less equally high level of confidence in both their old and new responses overall.

Encoding success of each studied face–name pair was determined on the basis of how subjects responded to both the true pair and the two corresponding false pairs. A “perfect hit” was classified as an “old” response to the true pair at test (study item/true pair in Figure 1B), and correct rejection of both corresponding false pairs (e.g., False Pair 1 and False Pair 2 in Figure 1B). An “imperfect hit” was classified as a “hit” to the true pair, but failure to correctly reject one or more false pairs. In addition, we further classified perfect and imperfect hits according to whether the true pair was endorsed with HC or LC. Confidence in decisions to the false pairs was not included in this classification, but in general, the confidence of correctly rejected false pairs mirrored the confidence of the true pair for HC responses (i.e., an HC hit to the true pair was associated with HC correct rejections to the corresponding false pairs in 71% of the cases). Guess and miss categories referred to pairs in which the true pair received a response of 4 (guess) or 5–7 (new), respectively, regardless of what responses were made to the false pairs.

The proportions of responses for each confidence level (excluding guesses) were submitted to a 2 × 2 × 2 ANOVA with factors of item type (old, new), subject response (old, new), and confidence (high, low). We also compared the guessing rate for old and new items directly. For these and all of the following statistical analyses, we applied an alpha of .05, Greenhouse–Geisser correction for violations of the sphericity (corrected F values and degrees of freedom are reported), and Tukey's honestly significant difference (HSD) tests for post-hoc comparisons following significant interactions and main effects with factors greater than two levels.

ERP Data

For the ERP analysis, we sought to isolate categories of encoding success that were likely to be behaviorally meaningful, yet yield sufficient trials for ERP averaging. To this end, we created the following three memory categories: (a) perfect high-confidence hits (HC); (b) perfect and imperfect low-confidence hits (LC); and (c) misses and guesses (GM). These three categories can be viewed as representing associative memory of decreasing strength and discrimination. We opted to consolidate LC perfect and imperfect hits into a single category as we believe they are similarly characterized by weaker memory strength, although in the latter case this uncertainty was also coupled with poorer discrimination, as evidenced by endorsement of one or more of the false pairs as old. Misses, regardless of confidence, and guesses were grouped together, as they indicate the relative absence of associative memory. We excluded imperfect HC hits from the analysis, however, because they represent an ambiguous situation in which a strong feeling of remembering for the true pair was coupled with a failure to reject repairing of the face or name with another item. This situation occurred very rarely (see Behavioral Results below) and may have resulted from order effects in a given participant, wherein a false pair was presented earlier in the test sequence and endorsed as old, followed by even more certain recognition of the true pair later in the test.

At first presentation, trial counts for HC hits ranged from 48 to 122 (M = 77.5, SD = 19.7), for LC hits from 11 to 55 (M = 35.3, SD = 14.5), and for GM from 16 to 61 (M = 40.3, SD = 15.5). Although there was a mild increase in noise over the course of the three presentations, and therefore slightly greater trial attrition by the final presentation, trial counts at the third presentation were relatively similar to the first, with HC ranging from 50 to 105 (M = 76.5, SD = 19.2), LC hits ranging from 10 to 54 (M = 34.3, SD = 13.7), and GM ranging from 11 to 66 (M = 40.1, SD = 17.1).

On the basis of previous findings regarding ERPs associated with face perception (Rossion et al., 1999; Bentin, Allison, Puce, Perez, & McCarthy, 1996), we focused one aspect of our analyses on P120 and N170 components. As shown in Figures 2 and 3, the early segment of the poststimulus waveform appeared to be dominated by these potentials, as well as by an N1 and P2/VPP over fronto-central sites. It was more difficult to isolate activity likely to be associated with auditory name processing. Although auditory name stimuli have been shown to elicit an N1 and P2 over central sites (Muller & Kutas, 1996), the latencies of the potentials observed in the current study appeared to be more congruent with the inverse of the posterior visual potentials than potentials elicited uniquely by the name (see also Localizer Experiment). It is unlikely that this pattern was due to variation in the spectral characteristics of the names, given that other studies have shown insensitivity of the auditory ERP to whether the name starts with a fricative or plosive (Muller & Kutas, 1996).

Figure 2. 

Grand mean ERPs at encoding associated with subsequent memory performance, collapsed across the three consecutive stimulus presentations (weighted average). In this and all waveform figures, positive is plotted up and waveforms were smoothed with a 15-Hz filter for illustration purposes. Locations of the illustrated electrodes are shown as dark circles on the electrode montage. Horizontal axis indicates time in milliseconds.

Figure 2. 

Grand mean ERPs at encoding associated with subsequent memory performance, collapsed across the three consecutive stimulus presentations (weighted average). In this and all waveform figures, positive is plotted up and waveforms were smoothed with a 15-Hz filter for illustration purposes. Locations of the illustrated electrodes are shown as dark circles on the electrode montage. Horizontal axis indicates time in milliseconds.

Figure 3. 

Grand mean ERPs at encoding associated with each stimulus presentation, collapsed across subsequent memory performance (weighted average).

Figure 3. 

Grand mean ERPs at encoding associated with each stimulus presentation, collapsed across subsequent memory performance (weighted average).

P120 and the N170 components were largest over the right temporo-parietal area of the scalp, and thus, were quantified as the largest positive deflection from 50 to 170 msec and the largest negative deflection from 80 to 230 msec, respectively, at electrode P8. Peak amplitudes at this electrode were compared to amplitudes at the homologous left hemisphere electrode (P7).

In order to look for evidence of the sustained potentials described by Guo et al. (2005), we analyzed mean amplitudes along five midline electrode locations (Fz, FCz, Cz, CPz, and Pz), and along five pairs of inferior temporal electrodes (FT9/FT10, T7/T8, TP7/TP8, TP9/TP10, and CB1/CB2) at three contiguous intervals (300–500 msec, 500–700 msec, and 700–900 msec).

We initially conducted ANOVAs that included all relevant factors (memory, presentation, poststimulus interval, and electrode; plus hemisphere for inferior temporal electrodes). For the midline electrodes, although the four-way interaction was not significant, multiple three-way interactions were found, including Memory × Interval × Electrode [F(3.6, 58.9) = 3.02, p < .01, η2 = .19], Presentation × Interval × Electrode [F(2.5, 41.1) = 7.93, p = .001, η2 = .33], and Memory × Presentation × Interval [F(4.1, 66.4) = 2.15, p < .05, η2 = .14], all of which subsumed a number of significant two-way interactions and main effects. At inferior temporal electrodes, we found a three-way interaction of Memory × Presentation × Interval [F(3.1, 50.6) = 2.75, p < .05, η2 = .14], as well as a Presentation × Electrode × Hemisphere interaction [F(2.7, 44.1) = 3.24, p < .05, η2 = .16], and a series of two-way interactions including a Memory × Electrode interaction [F(2.9, 47.5) = 3.53, p < .05, η2 = .18].

Given the potential complexity of understanding the sources of these interactions, we opted to structure our data analysis in a hierarchical, hypothesis-driven manner. First, in order to explore the general time course and topography of the overall subsequent memory effects, we compared HC, LC, and GM trials collapsed across stimulus presentations (weighted-average). Second, to investigate the effect of massed repetition on study-phase activity, we compared ERPs elicited by the first, second, and third stimulus presentation collapsed over the subsequent memory categories (weighted-average). Third, to assess if and how each consecutive face–name presentation contributed to the subsequent memory effects observed in the general analysis, we repeated the subsequent memory analysis separately for each of the three presentations.

RESULTS

Behavioral Results

Recognition memory scores are shown for each confidence level in Table 1. Overall, subjects correctly recognized 75% of true face–name pairs. In most instances, subjects endorsed this “old” response to the true pair with HC while also correctly rejecting corresponding recombined (false) pairs, thereby resulting in an HC “perfect hit” rate of 47%. In the remaining cases, “old” responses to the true pair were either endorsed with LC and/or one or more of the false pairs also were given an “old” or “guess” response. Overall, however, false alarms (FAs) to false pairs occurred infrequently (8%). These results suggest that the task was fairly well calibrated, in that it resulted in neither ceiling nor floor effects.

Table 1. 

Mean Proportions (and Standard Deviations) of Recognition Scores as Function of Confidence


High-confidence
Low-confidence
Total
Hitsa 0.53 (0.12) 0.22 (0.09) 0.75 (0.09) 
 Perfect 0.47 (0.11) 0.14 (0.07) 0.61 (0.10) 
 Imperfect 0.06 (0.04) 0.08 (0.04) 0.14 (0.05) 
Misses 0.04 (0.03) 0.10 (0.07) 0.14 (0.09) 
Guesses to Old – – 0.11 (0.06) 
Correct Rejections 0.68 (0.14) 0.17 (0.10) 0.85 (0.06) 
False Alarms 0.02 (0.01) 0.06 (0.03) 0.08 (0.04) 
Guesses to New – – 0.07 (0.04) 

High-confidence
Low-confidence
Total
Hitsa 0.53 (0.12) 0.22 (0.09) 0.75 (0.09) 
 Perfect 0.47 (0.11) 0.14 (0.07) 0.61 (0.10) 
 Imperfect 0.06 (0.04) 0.08 (0.04) 0.14 (0.05) 
Misses 0.04 (0.03) 0.10 (0.07) 0.14 (0.09) 
Guesses to Old – – 0.11 (0.06) 
Correct Rejections 0.68 (0.14) 0.17 (0.10) 0.85 (0.06) 
False Alarms 0.02 (0.01) 0.06 (0.03) 0.08 (0.04) 
Guesses to New – – 0.07 (0.04) 

a“Perfect” hits denote proportions of correctly recognized true face–name pairs when both corresponding false pairs are rejected. If one or more false pairs were not rejected, the hit is described as an “imperfect” hit.

Comparison of confidence across response categories revealed a three-way interaction [F(1, 16) = 10.7, p < .01, η2 = .40]. Post-hoc tests showed that for both hits and correct rejections, HC responses occurred more frequently than LC responses, whereas for misses and FAs, HC and LC judgments occurred with equal frequency. The guessing rate was higher for old items [t(16) = 4.38, p < .001]. These results indicate that confidence judgments were used sensibly as they varied coherently with correct (hits and CR) and incorrect (misses and FA) responses.

ERP Results

Overall Memory Effects

As can be seen in Figure 2, subsequent memory effects were observed both at early visual components and at sustained potentials over midline and inferior temporal regions. These results are summarized in the first column of Table 2.

Table 2. 

Summary of ERP Results



Overall Subsequent Memory Effects
Overall Presentation Effects
Subsequent Memory Effects at Each Presentation
1st
2nd
3rd
Peaks 
P120  HC > GM 1 > 2 = 3 – – – 
N170  – 1 > 2 = 3 – – – 
 
Sustained Potentials 
300–500 msec 
 Midline positivity Fz – – – – HC > GM 
FCz HC > GM – – – HC = LC > GM 
Cz – 1 > 2 = 3 – – – 
CPz – 1 > 2 = 3 – – – 
Pz – 1 > 2 = 3 – – – 
 Inferior temporal negativity FT9/FT10 – – HC > GM – – 
T7/T8 HC = LC > GM – HC > GM – – 
TP7/TP8 HC = LC > GM TP8: 1 > 2 = 3 HC > GM – – 
TP9/TP10 HC > LC = GM TP10: 1 > 2 = 3 HC > GM – – 
CB1/CB2 HC > LC = GM CB2: 1 > 2 = 3 HC > GM – – 
500–700 msec 
 Midline positivity Fz – 1 < 2 = 3 – – – 
FCz – – – – LC > GM 
Cz – 1 > 2 = 3 HC > GM – – 
CPz HC > LC = GM 1 > 2 = 3 HC > LC = GM – – 
Pz HC > LC = GM 1 > 2 = 3 HC > GM – – 
 Inferior temporal negativity FT9/FT10 – FT10: 1 > 2 = 3 HC > LC > GM – – 
T7/T8 HC = LC > GM – HC > LC > GM – – 
TP7/TP8 HC = LC > GM HC > LC > GM – – 
TP9/TP10 HC > LC = GM TP10: 1 > 2 = 3 HC > LC > GM – – 
CB1/CB2 HC > LC = GM CB2: 1 > 2 = 3 HC > LC > GM – – 
700–900 msec 
 Midline positivity Fz – – – – LC > GM 
FCz – – – – – 
Cz HC > GM 1 > 2 = 3 HC > GM – – 
CPz HC > GM 1 > 2 = 3 HC > GM – – 
Pz HC > LC = GM 1 > 2 = 3 HC > LC = GM – – 
 Inferior temporal negativity FT9/FT10 – – HC = LC > GM – – 
T7/T8 HC = LC > GM T8: 1 > 2 = 3 HC = LC > GM – – 
TP7/TP8 HC = LC > GM TP8: 1 > 2 = 3 HC = LC > GM – – 
TP9/TP10 HC > LC = GM TP10: 1 > 2 = 3 HC = LC > GM – – 
CB1/CB2 HC > LC = GM CB2: 1 > 2 = 3 HC = LC > GM – – 


Overall Subsequent Memory Effects
Overall Presentation Effects
Subsequent Memory Effects at Each Presentation
1st
2nd
3rd
Peaks 
P120  HC > GM 1 > 2 = 3 – – – 
N170  – 1 > 2 = 3 – – – 
 
Sustained Potentials 
300–500 msec 
 Midline positivity Fz – – – – HC > GM 
FCz HC > GM – – – HC = LC > GM 
Cz – 1 > 2 = 3 – – – 
CPz – 1 > 2 = 3 – – – 
Pz – 1 > 2 = 3 – – – 
 Inferior temporal negativity FT9/FT10 – – HC > GM – – 
T7/T8 HC = LC > GM – HC > GM – – 
TP7/TP8 HC = LC > GM TP8: 1 > 2 = 3 HC > GM – – 
TP9/TP10 HC > LC = GM TP10: 1 > 2 = 3 HC > GM – – 
CB1/CB2 HC > LC = GM CB2: 1 > 2 = 3 HC > GM – – 
500–700 msec 
 Midline positivity Fz – 1 < 2 = 3 – – – 
FCz – – – – LC > GM 
Cz – 1 > 2 = 3 HC > GM – – 
CPz HC > LC = GM 1 > 2 = 3 HC > LC = GM – – 
Pz HC > LC = GM 1 > 2 = 3 HC > GM – – 
 Inferior temporal negativity FT9/FT10 – FT10: 1 > 2 = 3 HC > LC > GM – – 
T7/T8 HC = LC > GM – HC > LC > GM – – 
TP7/TP8 HC = LC > GM HC > LC > GM – – 
TP9/TP10 HC > LC = GM TP10: 1 > 2 = 3 HC > LC > GM – – 
CB1/CB2 HC > LC = GM CB2: 1 > 2 = 3 HC > LC > GM – – 
700–900 msec 
 Midline positivity Fz – – – – LC > GM 
FCz – – – – – 
Cz HC > GM 1 > 2 = 3 HC > GM – – 
CPz HC > GM 1 > 2 = 3 HC > GM – – 
Pz HC > LC = GM 1 > 2 = 3 HC > LC = GM – – 
 Inferior temporal negativity FT9/FT10 – – HC = LC > GM – – 
T7/T8 HC = LC > GM T8: 1 > 2 = 3 HC = LC > GM – – 
TP7/TP8 HC = LC > GM TP8: 1 > 2 = 3 HC = LC > GM – – 
TP9/TP10 HC > LC = GM TP10: 1 > 2 = 3 HC = LC > GM – – 
CB1/CB2 HC > LC = GM CB2: 1 > 2 = 3 HC = LC > GM – – 

Only results significant at the p < .05 or less are shown. See text for further explanation.

Early peaks

The P120 component, which was larger over the right hemisphere [F(1, 16) = 34.1, p < .001, η2 = .68], differed as a function of memory [F(2, 32) = 4.6, p < .05, η2 = .22], with later HC hits eliciting more positive amplitudes than later GM items. The N170 exhibited no significant effects of memory (F < 1), and only a marginal effect of hemisphere (p = .08).

Sustained potentials

Later memory performance was associated with modulation of the midline potentials [F(2, 32) = 6.32, p = .005, η2 = .28], however, this effect also interacted with interval and electrode in a three-way interaction [F(3.7, 60.2) = 3.3, p < .05, η2 = .17]. Given the potential complexity of this interaction, we opted to conduct a series of focused ANOVAs that dissected the memory effects at each poststimulus interval and electrode.

In the first interval (300–500 msec), subsequent memory effects were isolated at FCz [F(2, 32) = 3.31, p < .05, η2 = .17], where later HC items exhibited greater positivity than GM items. A similar differentiation between HC and GM items was also found in the middle interval (500–700 msec) over more posterior sites [CPz: F(2, 32) = 9.93, p < .001, η2 = .38; Pz: F(2, 32) = 13.21, p < .001, η2 = .45]. At these sites, HC items were also significantly more positive than LC hits. By the final interval (700–900 msec), subsequent memory effects were found extensively across the central and parietal regions [Cz: F(2, 32) = 3.62, p < .05, η2 = .18; CPz: F(2, 32) = 7.66, p < .01, η2 = .32; Pz: F(2, 32) = 9.64, p = .001, η2 = .37]. However, at Cz and CPz, these differences were limited to greater positivity for later HC hits compared to GM items, whereas at Pz, later HC hits were still more positive-going than either LC or GM items.

At inferior temporal locations, we observed a sustained negativity that was associated with successful encoding throughout the 300–900 msec poststimulus period [F(2, 32) = 9.6, p = .001, η2 = .37]. HC hits elicited more negative-going waveforms than either LC or GM pairs, which did not differ from each other overall. However, this effect interacted with electrode [F(2.8, 45.9) = 3.93, p < .001, η2 = .19], indicating that these effects varied somewhat across site. A series of 3 (memory) × 3 (interval) × 2 (hemisphere) ANOVAs at each electrode location revealed that this was the result of the relatively more posterior focus of the memory effects. Specifically, throughout this sustained period, no differences as a function of later memory were found at more fronto-temporal sites (FT9/FT10). However, at the most posterior temporal sites, items later associated with HC hits elicited more negative activity than those associated with either LC or GM responses [TP9/TP10: F(2, 32) = 7.27, p < .01, η2 = .31; CB1/CB2: F(2, 32) = 8.71, p = .001, η2 = .35]. At middle temporal sites, HC and LC hits did not differ, and were both associated with more negative amplitudes than GM items [T7/T8: F(2, 32) = 8.42, p = .001, η2 = .34; TP7/TP8: F(2, 32) = 6.48, p < .01, η2 = .28].

In summary, at most sites, robust differences where found between the most strongly encoded associations (HC hits) from forgotten associations (GM items). Weakly encoded associations (LC hits) exhibited activity more like forgotten items at superior (parietal) and inferior (temporal) posterior sites, but at middle temporal sites they acted similarly to the more strongly encoded associations. Midline subsequent memory effects also demonstrated a spatio-temporal gradient such that earlier effects (300–500 msec) were evident over frontal sites, but in the later portion of the epoch shifted to more posterior sites.

Effects of Presentation

ERP waveforms for each of the three stimulus presentations, collapsed across subsequent memory performance, are shown in Figure 3. As can be seen from the summary of results provided in the second column of Table 2, there was a large reduction in activity from the first to second presentation at many sites of interest, but little additional change between the second and third presentations.

Early peaks

Both the P120 and N170 demonstrated significant effects of presentation, [P120: F(1.4, 23.1) = 7.13, p < .01, η2 = .30; N170: F(1.1, 18.8) = 31.6, p < .001, η2 = 0.66], in that they diminished in amplitude between the first and second presentations, with no further change at the third presentation. Hemisphere effects found in the subsequent memory analysis were replicated here.

Sustained potentials

A 3 (presentation) × 3 (interval) × 5 (electrode) ANOVA on midline potentials resulted in a three-way interaction [F(2.4, 38.5) = 8.56, p < .001, η2 = .34], which investigation with more focused ANOVAs on specific electrodes and intervals revealed was due to differences in the pattern of effects at frontal and posterior sites. Across the centro-parietal electrodes (Cz, CPz, Pz), the first presentation was consistently associated with more positive-going waveforms than either the second or third presentations at all three poststimulus intervals (all ps < .01). At Fz, however, a slightly different pattern was found. Here, significant repetition effects were only found during 500–700 msec poststimulus period [F(1.3, 22.3) = 8.36, p < .01, η2 = .34], due to a more negative-going waveform in the first presentation than subsequent presentations. No significant repetition effects were found at FCz.

Over inferior temporal sites, a four-way interaction was found between interval, presentation, electrode, and hemisphere [F(4.4, 70.7) = 2.46, p < .05, η2 = .13]. Further 3 (presentation) × 2 (hemisphere) ANOVAs for each electrode pair at each poststimulus interval indicated significant interactions between hemisphere and presentation at the more posterior and inferior temporal electrodes (TP9/TP10, CB1/CB2) throughout all three intervals (all ps < .01). In all cases, this interaction was driven by more negative-going waveforms for the first presentation compared to the latter two presentations over the right hemisphere, with no differences emerging over the left hemisphere. A similar pattern was found at fronto-temporal sites from 500 to 700 msec [F(2, 32) = 4.19, p < .05, η2 = .20], and at middle temporal sites from 700 to 900 msec [T7/T8: F(2, 3) = 5.78, p < .01, η2 = 26; TP7/TP8: F(1.1, 18.7) = 6.78, p < .05, η2 = .29].

Differences Due to Memory at Each Presentation

In these analyses, subsequent memory effects were examined separately for each presentation. As summarized in the rightmost columns of Table 2, subsequent memory effects at posterior midline and inferior temporal sites that had appeared in the overall analysis were driven by effects occurring only at the first presentation. However, exploratory analyses revealed some differences over frontal sites at the third presentation. This apparent shift in topography between subsequent memory effects during the first and third presentations is depicted in Figure 4.

Figure 4. 

Scalp topography of the difference waves for the two successful encoding conditions (HC–GM and LC–GM) as a function of presentation (first vs. third) across the three intervals of interest. White depicts positive-going activity and gray depicts negative-going activity. The scaling factor is 0.20 μV.

Figure 4. 

Scalp topography of the difference waves for the two successful encoding conditions (HC–GM and LC–GM) as a function of presentation (first vs. third) across the three intervals of interest. White depicts positive-going activity and gray depicts negative-going activity. The scaling factor is 0.20 μV.

Early peaks

Neither the P120 nor the N170 components exhibited a subsequent memory effect when the three presentations were analyzed individually (ps > .1).

Sustained potentials

Figure 5 illustrates ERPs recorded at selected fronto-central and parietal electrodes, during the first and third stimulus presentations, averaged as function of subsequent memory. When the first presentation was analyzed in isolation, we found a significant three-way interaction of memory, interval, and electrode [F(2.8, 45.2) = 3.91, p < .05, η2 = .19]. Further investigation of this interaction revealed that the pattern of effects over central and parietal sites was highly similar to those found in the overall subsequent memory analysis. Specifically, significant subsequent memory effects did not emerge until the later two poststimulus intervals [500–700 msec: Cz, F(2, 32) = 3.51, p < .05, η2 = .18; CPz, F(2, 32) = 12.07, p < .001, η2 = .43; Pz, F(2, 32) = 12.89, p < .001, η2 = .44; 700–900 msec: Cz, F(2, 32) = 7.2, p < .01, η2 = .31; CPz, F(2, 32) = 9.95, p < .001, η2 = .38; Pz, F(2, 32) = 10.58, p < .001, η2 = .39], where later HC hits elicited greater positivity than later GM items. Later LC hits were significantly less positive than later HC hits from 500 to 700 msec at CPz, and from 700 to 900 msec at Pz.

Figure 5. 

Grand mean ERPs recorded during the first and third stimulus presentations, averaged as function of subsequent memory, shown at selected fronto-central and parietal midline electrodes.

Figure 5. 

Grand mean ERPs recorded during the first and third stimulus presentations, averaged as function of subsequent memory, shown at selected fronto-central and parietal midline electrodes.

Memory effects at frontal sites exhibited a different pattern however. The greater positivity of encoding activity associated with HC compared to GM responses that had appeared in the overall analysis (i.e., when collapsing over presentation) did not reach significance when examined at the first presentation in isolation. Thus, we sought to explore through focused analyses whether effects specific to the second or third presentation might have driven the overall subsequent memory effect at FCz. An initial ANOVA that included electrode as a factor (Memory × Interval × Electrode) found no significant main effects or interactions between electrode and memory or interval in either the second or third presentation. However, when we isolated our analysis of the early interval (300–500 msec) to FCz, a significant subsequent memory effect emerged in the third presentation [F(2, 32) = 5.2, p < .05, η2 = .24]. A similar exploratory analysis at Fz also revealed a significant effect of memory during this portion of the epoch [F(2, 32) = 4.2, p < .05, η2 = .20]. At both of these frontal sites, the memory effect was associated with later HC hits eliciting more positive-going activity than items associated with later GM responses. Additionally, at FCz, items associated with LC hits were also significantly more positive than GM items.

When we extended this exploratory analysis to the 500–700 msec period, we found a significant effect of memory at FCz only [F(2, 32) = 3.7, p < .05, η2 = .18], and when we examined the 700–900 msec period, the memory effect was significant at Fz only [F(2, 32) = 4.2, p < .05, η2 = .20]. Interestingly, in both of these later intervals, it was the later LC hits that were more positive-going than the later GM items. We note that no significant effects were found over central or parietal sites during the third presentation, suggesting that localized effects at frontal sites during the third presentation might indeed have been masked by extensive nonsignificant differences throughout the central and parietal electrode sites in ANOVAs that included all midline electrodes. Nonetheless, caution must be exercised in interpreting these exploratory analyses at focused frontal sites.

Similar to the pattern of effects found over centro-parietal sites, inferior temporal sites demonstrated a subsequent memory effect at the first stimulus presentation only [F(2, 32) = 12.4, p < .001, η2 = .43]. Furthermore, although this effect interacted with interval [F(4, 64) = 2.7, p < .05, η2 = .14], it did not interact with electrode, indicating that it was topographically widespread. A representative right-hemisphere electrode, CB2, is shown in Figure 6 (top) alongside with mean amplitudes averaged across all of inferior temporal electrodes (Figure 6, bottom). Overall, during the first interval (300–500 msec), HC hits were associated with more negative-going activity than GM items, but LC hits did not differ from either of these conditions. By the second interval (500–700 msec), HC hits were more negative than LC hits, which were, in turn, more negative than GM. In the last interval (700–900 msec), HC and LC hits did not differ and they both were associated with more negative amplitudes than GM items. Thus, whereas robust face–name encoding elicited greater activity than encoding failures across the encoding interval, weaker face–name encoding seemed to be associated with a relatively later onset of inferior temporal activity.

Figure 6. 

Subsequent memory effects at inferior temporal locations, during the first and third stimulus presentations. (A) Grand mean ERP waveform from a representative right inferior temporal electrode (CB2). (B) Mean amplitudes averaged across inferior temporal electrodes, plotted for each poststimulus interval. Error bars represent the standard error of the mean (SEM).

Figure 6. 

Subsequent memory effects at inferior temporal locations, during the first and third stimulus presentations. (A) Grand mean ERP waveform from a representative right inferior temporal electrode (CB2). (B) Mean amplitudes averaged across inferior temporal electrodes, plotted for each poststimulus interval. Error bars represent the standard error of the mean (SEM).

LOCALIZER EXPERIMENT

Methods

Participants

Seventeen new subjects were tested. The same selection criteria from the primary experiment were applied. EEG data from one subject were lost due to technical problems during acquisition, leaving 16 subjects for analysis.

Stimuli

Eight different categories of visual and auditory stimuli were used, with 50 exemplars in each category. These stimuli included the identical faces and auditory names used in the primary experiment, as well as three additional categories of control stimuli for each modality. For the visual modality, the control stimuli were scrambled faces, gray squares, and crosshairs. Scrambled faces were created by dividing each face image into a grid of 10 × 10 squares that were then randomly recombined. Scrambled faces and gray squares subtended the same visual angle as intact faces and also were matched for average luminance. For the auditory modality, the control stimuli were pseudonames (pronounceable rearranged versions of the target names), white noise created by randomly scrambling the auditory files of the intact names, and a pure tone of 250 Hz. The control stimuli primarily served to increase the difficulty of the task.

Procedure

Subjects were instructed to press the “Enter” key when two consecutive presentations from the same category occurred (e.g., an intact face followed by another intact face, regardless of specific identity). Presentation order was pseudorandomized to produce seven repetitions for each stimulus type. Visual stimuli were displayed for 1500 msec. The duration of the auditory stimuli was the same as in the main experiment. Interstimulus interval was 500 msec. During the task, continuous EEG was recorded using the same parameters as in the main experiment.

EEG Analysis

The primary goal of this ancillary study was to better understand whether the longer latency frontal, parietal, and inferior temporal activity observed during the primary experiment might be related differentially to item-specific perceptual processing of the faces or names. Therefore, rather than engaging in an in-depth analysis of the entire pattern of auditory and visual ERPs associated with proper name and face processing, we focused our analysis on midline and inferior temporal sites, using the same intervals as in the primary experiment (300–500 msec, 500–700 msec, and 700–900 msec).

Also, rather than analyzing all conditions, our analysis was restricted to intact faces and names, and scrambled faces and pseudonames. Scrambled faces and pseudonames disrupted the global integrity (and familiarity) of the critical stimuli, while retaining their local features. We reasoned that activity common across intact and scrambled versions of each stimulus type would indicate regions showing more general modality effects, whereas activity specific to intact versions would represent more holistic processing of each stimulus type. The other types of control stimuli were not included in order to simplify the analysis. We acknowledge, however, that by not including any nonface or nonname stimuli, this experiment provides little in the way of localization of face-specific or name-specific processes, but rather focuses on distinguishing between visual and auditory processing of stimuli identical to those used in the primary experiment.

We conducted a series of 2 (modality: visual vs. auditory) × 2 (stimulus: intact vs. scrambled) ANOVAs at each interval and electrode of interest (midline effects), with the addition of a hemisphere factor when analyzing inferior temporal electrodes. Criteria for significance and post-hoc analyses were the same as in the primary experiment. To avoid contaminating our analyses with target detection and response processes, trials corresponding to category repetitions were excluded.

Results

Midline Potentials

As shown in Figure 7, Fz showed an overall effect of modality during the initial interval (300–500 msec), with visual stimuli eliciting more negative-going waveforms than auditory stimuli [F(1, 15) = 8.3, p < .05, η2 = .35]. At the other midline electrodes, however, significant Modality × Stimulus interactions emerged (all Fs > 4.6, all ps < .05). At FCz and Cz, it appeared that these interactions were driven largely by the shorter peak latency of the P2 to intact faces compared to the other conditions. At CPz and Pz, however, this interaction was driven mainly by the scrambled faces exhibiting more sustained positivity compared either to all other stimuli (CPz), or just compared to auditory stimuli (Pz). Names and pseudonames did not differ significantly from each other at any of the midline electrodes.

Figure 7. 

Grand mean ERPs from the ancillary localizer task, illustrating the four critical conditions: intact faces, scrambled faces, intact names, and pseudonames (scrambled names).

Figure 7. 

Grand mean ERPs from the ancillary localizer task, illustrating the four critical conditions: intact faces, scrambled faces, intact names, and pseudonames (scrambled names).

From 500 msec onward, additional differences between auditory and visual waveforms were evident across the midline. From 500 to 700 msec, significant modality effects were found at fronto-central [F(1, 15) = 5.5, p < .05, η2 = .26] and central sites [F(1, 15) = 6.1, p < .05, η2 = .28], although the pattern was complex given that this interval appeared to capture a crossover point between the two conditions. During the last interval (700–900 msec), however, clearer modality effects emerged. Over frontal and central sites, auditory stimuli exhibited greater negative-going activity [Fz: F(1, 15) = 10.1, p < .01, η2 = .40; FCz: F(1, 15) = 6.9, p < .05, η2 = .31], whereas at the parietal site, this negativity reversed to a positivity for auditory stimuli and negativity for visual stimuli [Pz: F(1, 15) = 14.6, p < .01, η2 = .49]. There were no significant effects at Cz and CPz.

Taken together, these findings suggest that our visual stimuli were processed more rapidly and with a stronger posterior distribution than the auditory stimuli. In contrast, the auditory stimuli were associated with later sustained negativity over frontal sites. Indeed, a similar pattern for the auditory stimuli was found across the more anterior portion of the scalp, including more inferior and mid-temporal sites (also see below). The inverted polarity of the visual (negative) and auditory (positive) waveforms at parietal sites during the later portion of the epoch suggests that these responses might cancel each other out at the scalp if they were coactive (see also Guo et al., 2005).

Inferior Temporal Potentials

During the first (300–500 msec) and second (500–700 msec) intervals, visual stimuli were associated with more negative amplitudes than auditory stimuli across all inferior temporal sites (all ps < .0001). Modality × Hemisphere interactions at all but the most frontal sites (FT9/FT10) indicated that this difference occurred primarily over the left hemisphere during the first interval (all ps < .05), although this left hemisphere bias only continued into the 500–700 msec interval at the most posterior electrodes (TP9, CB1) [TP9/TP10: F(1, 15) = 6.2, p < .05, η2 = .29; CB1/CB2: F(1, 15) = 9.8, p < .01, η2 = .39]. Indeed, over right temporal electrodes (FT10, T8), a crossover between auditory and visual waveforms was already apparent [FT9/FT10: F(1, 15) = 17.2, p = .001, η2 = .53; T7/T8: F(1, 15) = 20.2, p < .0001, η2 = .57]. This crossover continued bilaterally through to the last interval (700–900 msec) at FT9/FT10 [F(1, 15) = 14.1, p < .01, η2 = .48] and T8 [F(1, 15) = 9.5, p < .01, η2 = .38]. In contrast, waveforms associated with the visual stimuli were significantly more negative-going at the more posterior inferior electrodes [TP9/TP10: F(1, 15) = 10.5, p = .005, η2 = .41; CB1/CB2: F(1, 15) = 21.9, p < .0001, η2 = .59]. These results confirm the presence of a sustained posterior inferior negativity that was more specific for visual (face) stimuli and a later-onset anterior inferior negativity that was more specific for auditory (name) stimuli. As with the parietal activity described in the previous section, we note that the opposing polarities of the auditory and visual sustained potentials at the more anterior sites (FT9/FT10; T7/T8) similarly create the possibility that they will cancel out if they were simultaneously active.

DISCUSSION

The goal of the present study was to contribute to the understanding of the ERPs underlying the successful encoding of face–name associations, in particular by describing how the strategy of sequential repetitions supports the encoding of these multimodal pairings. First, however, we considered the overall subsequent memory effects, which served to narrow down patterns of scalp-recorded activity related to different aspects of memory formation that we then investigated further as a function of repetition. In this overall analysis, we found that successful HC discrimination of true face–name associations from recombined (false) associations (i.e., “perfect hits”) was predicted by relatively early (300–500 msec) study-phase activity over fronto-central sites, followed by more sustained activity over central and parietal regions (500–900 msec). Thus, these results replicate certain aspects of the previous face–name encoding study by Guo et al. (2005). Additionally, we demonstrated that a sustained negativity at middle and posterior inferior temporal sites supported the encoding of these HC perfect hits throughout the 300–900 msec poststimulus period. At least in the overall analysis, the pattern of neural activity associated with subsequent LC perfect and imperfect hits was generally more similar to that of items for which encoding had failed (GM responses), the exception being at middle temporal sites, where they elicited comparable activity to HC perfect hits.

Our more detailed analysis of subsequent memory effects as a function of repetition indicated, however, that the centro-parietal and inferior temporal modulations that were associated with successful encoding of HC perfect hits in the overall analysis were driven by neural differences occurring only at the first presentation. In addition, the late (500–900 msec) inferior temporal activity associated with encoding of face–name pairs garnering lower-quality discrimination (LC perfect and imperfect hits) also appeared to be related primarily to first-presentation activity. In contrast, the frontal midline subsequent memory effect that emerged in the overall analysis was significant only when we explored the third presentation of the to-be-encoded associations. Here, items later attracting either HC or LC hits elicited a more positive-going waveform relatively early in the period of sustained activity (300–500 msec), and somewhat surprisingly, only associations later retrieved with LC continued to elicit more positive-going activity throughout the remainder of the epoch.

Before discussing the effects of repetition in more detail, we will first discuss the potential cognitive correlates of these observed subsequent memory effects. Converging evidence from past studies (Guo et al., 2005) and our localizer task suggests that the posterior inferior temporal and fronto-central subsequent memory effects may have been driven by item-specific face and name processing, respectively, rather than association-specific processes. In the localizer task, sustained visual (face) processing elicited prominent negative-going waveforms over the more posterior inferior temporal locations, consistent with sources in the ventral visual stream (Mnatsakanian & Tarkka, 2004, 2007). Name processing, in contrast, was more convincingly associated with a fronto-central negativity that inverted to a large positivity over posterior sites. Similarly, Guo et al. (2005) found evidence for a fronto-central negativity that predicted name memory in one of their experiment, although they did not analyze inferior temporal sites. Thus, the finding that subsequent memory effects for LC hits were limited to modulation at these inferior temporal and fronto-central sites suggests that item-specific, rather than associative, processes were primarily responsible for the lower-quality discrimination of these face–name pairs. The overlap with Guo et al. is not complete, however, given that in their experiment, subsequent memory was associated with relatively greater negativity, rather than the reduced negativity observed in the present experiment. The reason for this difference is unclear. It may be a function of our choice to use an average, rather than linked-ears, reference. However, we consider another alternative explanation, related to retrieval processing, later in the discussion.

The sustained centro-parietal subsequent memory effect appears qualitatively different from what would be predicted based on a simple summation of opposite polarity face- and name-related potentials (see Localizer Experiment), making it an excellent candidate for a specific relational process that supports the integration of different elements into a bound memory trace that is strong and robust against the interference from recombined pairs. Although greater positivity over centro-parietal sites has frequently been found during successful encoding of single items where associative processes are implied, such as when memory is measured through recall, recollection, or source memory (Guo, Duan, Li, & Paller, 2006; Duarte, Ranganath, Winward, Hayward, & Knight, 2004; Mangels, Picton, & Craik, 2001; Van Petten, Senkfor, & Newberg, 2000; Van Petten & Senkfor, 1996; Paller et al., 1987; Fabiani, Karis, & Donchin, 1986), this study confirms the relationship of this waveform to a binding process that is unlikely to be the result of processing of individual event elements. These findings are also in line with previous research proposing that associative encoding of face–name pairs is more than the sum of its parts, including fMRI studies finding that activation of the anterior hippocampus at encoding is a specific predictor of later memory for face–name pairs (Sperling et al., 2003), whereas activity in parahippocampal/perirhinal cortex is likely to support subsequent memory for faces only (Chua et al., 2007).

Sequential repetition of the face–name pair dramatically attenuated this centro-parietal waveform, and subsequent memory effects at these sites were significant only at the first presentation, suggesting that the foundation for successful binding of the face–name association was established primarily at the initial exposure, when these elements were highly novel. When considering the activity putatively associated more with item-specific processing, however, we observed what might be interpreted as shift in topography of the subsequent memory effects across repetitions. Specifically, at inferior temporal sites, subsequent memory effects were found for the first presentation only, whereas subsequent memory effects appeared to shift to frontal sites during the third presentation. One interpretation of this apparent shift is that attention and memory processing were initially drawn to processing the face and its integration with the name (i.e., centro-parietal effects), but by the third presentation, attention was focused primarily on the name. Focusing on the name during the final presentation may have been particularly useful strategy given that each of the faces was relatively different, but the names were all spoken in the same voice. This is not unlike a situation in which the same host is introducing you to a long series of individuals, or following such an introduction, one is repeating this series of names over and over again with one's own inner voice. This similarity may have reduced the distinctiveness of the name information somewhat, allowing name encoding to particularly benefit from additional processing.

Support for this hypothesis can be found in the significant habituation of the N170 and right-lateralized inferior temporal negativity with repetition compared to the relative lack of repetition effects at frontal and fronto-central sites. Resources may have become available to focus specifically on encoding the name only after the novelty of the face had sufficiently decreased. Thus, only at the third presentation could variability in the allocation of those resources to the name reliably differentiate successful from unsuccessful encoding. Extending the logic of this hypothesis further, it would appear that higher-quality discrimination of the face–name association (i.e., HC perfect hits) benefited from enhanced processing of the face and association during the initial presentation, whereas the lower-quality discrimination involved prolonged processing of the name during the final presentation. We also note that the emergence of significant subsequent memory effects for LC hits later in the epoch at both inferior temporal (first presentation) and fronto-central sites (third presentation) could suggest a more effortful and/or less efficient mode of processing compared to HC perfect hits.

Alternatively, subsequent memory effects observed at the third presentation may be related to retrieval processes that support encoding, rather than processes associated exclusively with encoding per se. Specifically, during the second and third repetitions of the pair at study, participants are likely to experience recognition of either or both attributes of the association, and this retrieval may serve to strengthen their memory traces. Based on research specifically investigating ERPs associated with recognition (e.g., Woodruff, Hayama, & Rugg, 2006; Curran, 2000, 2004; Duarte et al., 2004), we might expect explicit recollection of previous presentations to manifest as a positivity over centro-parietal regions from 400 to 800 msec—a pattern corresponding to the classic “old/new” effect in which recollected items elicit greater positivity than novel items, whereas increased familiarity of the items would be more likely to manifest as reduced negativity over fronto-central regions from 300 to 500 msec—a pattern corresponding to the FN400. There was no evidence of a centro-parietal subsequent memory effect in the second and third presentations. However, the morphology of the earlier (300–500 msec) frontal subsequent memory effect observed at the third presentation could be consistent with modulation of the familiarity-based FN400. Modulation of the FN400 by massed repetition is likewise consistent with findings that massed repetitions increase the familiarity of the memory trace (Mantyla & Cornoldi, 2002; Parkin et al., 1995), rather than improve recollection. Taken together, the finding that early (300–500 msec) frontal memory effect was equivalent for both HC and LC hits could be taken as support for the view that it represented increasing familiarity of the face–name stimuli.

Yet, the subsequent memory effects at these fronto-central sites also lasted beyond the time range of the typical FN400 effect. Although we cannot rule out the possibility that such sustained effects also represent familiarity-based retrieval, and indeed, they were beneficial for LC hits, in particular, the presence of a similar sustained waveform during unique presentation of the name in the localizer also supports the view that they were elicited by name-specific encoding.

Lastly, we consider the possibility that the associative memory effects at frontal and parietal sites represent sustained attention or other cognitive control processes that may be necessary, but not sufficient to encode the face–name associations. Such processes may create a cognitive “state” that is conducive to encoding without representing the specific processes involved in laying down the memory trace of the name and/or the face–name association (Otten, Henson, & Rugg, 2002). The success of these top–down control processes might have been particularly important for maintaining active encoding through to the third repetition, after the intrinsic novelty of the stimuli had decreased. Lateral prefrontal regions have demonstrated greater activity during associative encoding in many tasks, including those requiring the association of faces and names (Sperling et al., 2001, 2003). For example, Sperling et al. (2003) demonstrated increased functional connectivity between the anterior hippocampus and left inferior prefrontal cortex, as well as between left inferior prefrontal cortex and fusiform cortex, suggesting that left prefrontal cortex supported the successful binding of faces and names, through up-regulation of both face-specific and associative processing. Connectivity between left prefrontal cortex and posterior representations has also been recently shown to be integral in supporting the successful binding of face–house associations (Summerfield et al., 2006).

Conclusions

Studies examining the relationship between encoding activity and retrieval success take advantage of item-by-item variability in quality and quantity of attention, strategy, and neural efficiency that ultimately lead to more successful encoding for some items than others. The present study suggests that when the intent is to encode a multimodal association into episodic memory, the initial presentation is critical for establishing a strong association, but the extent to which processing is sustained across subsequent immediate (massed) presentations may provide additional encoding support that serves to differentiate highly similar pairs, particularly with regard to the less dominant or novel stimulus dimension (i.e., names). However, the functional role played by repetition cannot be univocally determined by this study alone. Our participants were not assigned a specific encoding task, and thus, we do not know if a particular strategy was adopted (e.g., imaging, rote rehearsal, elaboration) or how that strategy might have changed during the course of the three presentations. Future work is necessary to address how, at a neural level, massed and distributed repetitions differ in their benefit toward stimulus–stimulus associations, such as faces and names, and how different mnemonic strategies can be utilized to maximize successful encoding of face–name associations.

Acknowledgments

This research was supported by an NIH grant R21MH066129 to J.A. Mangels. Alberto Manzi was supported by funding from the Second University of Naples. We thank Sam Gershman, who provided assistance with development and execution of the study, as well as Matt Greene who provided assistance with programming and recording of all the name stimuli.

Reprint requests should be sent to Jennifer A. Mangels, Baruch College, City University of New York, 55 Lexington Ave., New York, NY 10010, or via e-mail: jenimangels@gmail.com.

REFERENCES

Bentin
,
S.
,
Allison
,
T.
,
Puce
,
A.
,
Perez
,
E.
, &
McCarthy
,
G.
(
1996
).
Electrophysiological studies of face perception in humans.
Journal of Cognitive Neuroscience
,
8
,
551
565
.
Challis
,
B. H.
, &
Sidhu
,
R.
(
1993
).
Dissociative effect of massed repetition on implicit and explicit measures of memory.
Journal of Experimental Psychology: Learning, Memory, and Cognition
,
19
,
115
127
.
Chua
,
E. F.
,
Schacter
,
D. L.
,
Rand-Giovannetti
,
E.
, &
Sperling
,
R. A.
(
2007
).
Evidence for a specific role of the anterior hippocampal region in successful associative encoding.
Hippocampus
,
17
,
1071
1080
.
Curran
,
T.
(
2000
).
Brain potentials of recollection and familiarity.
Memory & Cognition
,
28
,
923
938
.
Curran
,
T.
(
2004
).
Effects of attention and confidence on the hypothesized ERP correlates of recollection and familiarity.
Neuropsychologia
,
42
,
1088
1106
.
Donaldson
,
D. I.
, &
Rugg
,
M. D.
(
1998
).
Recognition memory for new associations: Electrophysiological evidence for the role of recollection.
Neuropsychologia
,
36
,
377
395
.
Donaldson
,
D. I.
, &
Rugg
,
M. D.
(
1999
).
Event-related potential studies of associative recognition and recall: Electrophysiological evidence for context dependent retrieval processes.
Brain Research, Cognitive Brain Research
,
8
,
1
16
.
Duarte
,
A.
,
Ranganath
,
C.
,
Winward
,
L.
,
Hayward
,
D.
, &
Knight
,
R. T.
(
2004
).
Dissociable neural correlates for familiarity and recollection during the encoding and retrieval of pictures.
Brain Research, Cognitive Brain Research
,
18
,
255
272
.
Fabiani
,
M.
,
Karis
,
D.
, &
Donchin
,
E.
(
1986
).
P300 and recall in an incidental memory paradigm.
Psychophysiology
,
23
,
298
308
.
Guillaume
,
C.
,
Guillery-Girard
,
B.
,
Chaby
,
L.
,
Lebreton
,
K.
,
Hugueville
,
L.
,
Eustache
,
F.
,
et al
(
2009
).
The time course of repetition effects for familiar faces and objects: An ERP study.
Brain Research
,
1248
,
149
161
.
Guo
,
C.
,
Duan
,
L.
,
Li
,
W.
, &
Paller
,
K. A.
(
2006
).
Distinguishing source memory and item memory: Brain potentials at encoding and retrieval.
Brain Research
,
1118
,
142
154
.
Guo
,
C.
,
Voss
,
J. L.
, &
Paller
,
K. A.
(
2005
).
Electrophysiological correlates of forming memories for faces, names, and face–name associations.
Brain Research, Cognitive Brain Research
,
22
,
153
164
.
Heisz
,
J. J.
,
Watter
,
S.
, &
Shedden
,
J. M.
(
2006
).
Automatic face identity encoding at the N170.
Vision Research
,
46
,
4604
4614
.
Hux
,
K.
,
Manasse
,
N.
,
Wright
,
S.
, &
Snell
,
J.
(
2000
).
Effect of training frequency on face–name recall by adults with traumatic brain injury.
Brain Injury
,
14
,
907
920
.
Itier
,
R. J.
, &
Taylor
,
M. J.
(
2004
).
Effects of repetition learning on upright, inverted and contrast-reversed face processing using ERPs.
Neuroimage
,
21
,
1518
1532
.
James
,
L. E.
(
2004
).
Meeting Mr. Farmer versus meeting a farmer: Specific effects of aging on learning proper names.
Psychology and Aging
,
19
,
515
522
.
Khader
,
P.
,
Ranganath
,
C.
,
Seemuller
,
A.
, &
Rosler
,
F.
(
2007
).
Working memory maintenance contributes to long-term memory formation: Evidence from slow event-related brain potentials.
Cognitive, Affective & Behavioral Neuroscience
,
7
,
212
224
.
Kirwan
,
C. B.
, &
Stark
,
C. E.
(
2004
).
Medial temporal lobe activation during encoding and retrieval of novel face–name pairs.
Hippocampus
,
14
,
919
930
.
Manasse
,
N. J.
,
Hux
,
K.
, &
Snell
,
J.
(
2005
).
Teaching face–name associations to survivors of traumatic brain injury: A sequential treatment approach.
Brain Injury
,
19
,
633
641
.
Mangels
,
J. A.
,
Picton
,
T. W.
, &
Craik
,
F. I.
(
2001
).
Attention and successful episodic encoding: An event-related potential study.
Brain Research, Cognitive Brain Research
,
11
,
77
95
.
Mantyla
,
T.
, &
Cornoldi
,
C.
(
2002
).
Remembering changes: Repetition effects in face recollection.
Acta Psychologica (Amsterdam)
,
109
,
95
105
.
Martin-Loeches
,
M.
,
Sommer
,
W.
, &
Hinojosa
,
J. A.
(
2005
).
ERP components reflecting stimulus identification: Contrasting the recognition potential and the early repetition effect (N250r).
International Journal of Psychophysiology
,
55
,
113
125
.
Martinez
,
A. M.
, &
Benavente
,
R.
(
1998
).
The AR Face Database.
Computer Vision Center Technical Report
,
24
.
Mnatsakanian
,
E. V.
, &
Tarkka
,
I. M.
(
2004
).
Familiar-face recognition and comparison: Source analysis of scalp-recorded event-related potentials.
Clinical Neurophysiology
,
115
,
880
886
.
Mnatsakanian
,
E. V.
, &
Tarkka
,
I. M.
(
2007
).
Spatio-temporal modeling of evoked brain activity during memory encoding and target comparison in visual tasks.
Brain Topography
,
20
,
21
30
.
Muller
,
H. M.
, &
Kutas
,
M.
(
1996
).
What's in a name? Electrophysiological differences between spoken nouns, proper names and one's own name.
NeuroReport
,
8
,
221
225
.
Naveh-Benjamin
,
M.
,
Guez
,
J.
,
Kilb
,
A.
, &
Reedy
,
S.
(
2004
).
The associative memory deficit of older adults: Further support using face–name associations.
Psychology and Aging
,
19
,
541
546
.
Neuschatz
,
J. S.
,
Preston
,
E. L.
, &
Toglia
,
M. P.
(
2005
).
Comparison of the efficacy of two name-learning techniques: Expanding rehearsal and name–face imagery.
American Journal of Psychology
,
118
,
79
101
.
Otten
,
L. J.
,
Henson
,
R. N.
, &
Rugg
,
M. D.
(
2002
).
State-related and item-related neural correlates of successful memory encoding.
Nature Neuroscience
,
5
,
1339
1344
.
Palermo
,
R.
, &
Rhodes
,
G.
(
2007
).
Are you always on my mind? A review of how face perception and attention interact.
Neuropsychologia
,
45
,
75
92
.
Paller
,
K. A.
,
Kutas
,
M.
, &
Mayes
,
A. R.
(
1987
).
Neural correlates of encoding in an incidental learning paradigm.
Electroencephalography and Clinical Neurophysiology
,
67
,
360
371
.
Parkin
,
A. J.
,
Gardiner
,
J. M.
, &
Rosser
,
R.
(
1995
).
Functional aspects of recollective experience in face recognition.
Consciousness and Cognition
,
4
,
387
398
.
Rand-Giovannetti
,
E.
,
Chua
,
E. F.
,
Driscoll
,
A. E.
,
Schacter
,
D. L.
,
Albert
,
M. S.
, &
Sperling
,
R. A.
(
2006
).
Hippocampal and neocortical activation during repetitive encoding in older persons.
Neurobiology of Aging
,
27
,
173
182
.
Ranganath
,
C.
, &
Blumenfeld
,
R. S.
(
2005
).
Doubts about double dissociations between short- and long-term memory.
Trends in Cognitive Sciences
,
9
,
374
380
.
Rossion
,
B.
,
Campanella
,
S.
,
Gomez
,
C. M.
,
Delinte
,
A.
,
Debatisse
,
D.
,
Liard
,
L.
,
et al
(
1999
).
Task modulation of brain activity related to familiar and unfamiliar face processing: An ERP study.
Clinical Neurophysiology
,
110
,
449
462
.
Saxe
,
R.
,
Brett
,
M.
, &
Kanwisher
,
N.
(
2006
).
Divide and conquer: A defense of functional localizers.
Neuroimage
,
30
,
1088
1096; discussion 1097–1089
.
Scanlan
,
L. C.
, &
Johnston
,
R. A.
(
1997
).
I recognize your face, but I can't remember your name: A grown-up explanation?
Quarterly Journal of Experimental Psychology A
,
50
,
183
198
.
Schweinberger
,
S. R.
,
Pickering
,
E. C.
,
Burton
,
A. M.
, &
Kaufmann
,
J. M.
(
2002
).
Human brain potential correlates of repetition priming in face and name recognition.
Neuropsychologia
,
40
,
2057
2073
.
Small
,
S. A.
,
Nava
,
A. S.
,
Perera
,
G. M.
,
DeLaPaz
,
R.
,
Mayeux
,
R.
, &
Stern
,
Y.
(
2001
).
Circuit mechanisms underlying memory encoding and retrieval in the long axis of the hippocampal formation.
Nature Neuroscience
,
4
,
442
449
.
Sperling
,
R. A.
,
Bates
,
J. F.
,
Cocchiarella
,
A. J.
,
Schacter
,
D. L.
,
Rosen
,
B. R.
, &
Albert
,
M. S.
(
2001
).
Encoding novel face–name associations: A functional MRI study.
Human Brain Mapping
,
14
,
129
139
.
Sperling
,
R. A.
,
Chua
,
E.
,
Cocchiarella
,
A.
,
Rand-Giovannetti
,
E.
,
Poldrack
,
R.
,
Schacter
,
D. L.
,
et al
(
2003
).
Putting names to faces: Successful encoding of associative memories activates the anterior hippocampal formation.
Neuroimage
,
20
,
1400
1410
.
Summerfield
,
C.
,
Greene
,
M.
,
Wager
,
T.
,
Egner
,
T.
,
Hirsch
,
J.
, &
Mangels
,
J.
(
2006
).
Neocortical connectivity during episodic memory formation.
PLoS Biology
,
4
,
e128
.
Summerfield
,
C.
, &
Mangels
,
J. A.
(
2006
).
Dissociable neural mechanisms for encoding predictable and unpredictable events.
Journal of Cognitive Neuroscience
,
18
,
1120
1132
.
Tsivilis
,
D.
,
Otten
,
L. J.
, &
Rugg
,
M. D.
(
2001
).
Context effects on the neural correlates of recognition memory: An electrophysiological study.
Neuron
,
31
,
497
505
.
Uncapher
,
M. R.
,
Otten
,
L. J.
, &
Rugg
,
M. D.
(
2006
).
Episodic encoding is more than the sum of its parts: An fMRI investigation of multifeatural contextual encoding.
Neuron
,
52
,
547
556
.
Van Petten
,
C.
, &
Senkfor
,
A. J.
(
1996
).
Memory for words and novel visual patterns: Repetition, recognition, and encoding effects in the event-related brain potential.
Psychophysiology
,
33
,
491
506
.
Van Petten
,
C.
,
Senkfor
,
A. J.
, &
Newberg
,
W. M.
(
2000
).
Memory for drawings in locations: Spatial source memory and event-related potentials.
Psychophysiology
,
37
,
551
564
.
Woodruff
,
C. C.
,
Hayama
,
H. R.
, &
Rugg
,
M. D.
(
2006
).
Electrophysiological dissociation of the neural correlates of recollection and familiarity.
Brain Research
,
1100
,
125
135
.