The neural correlates of the encoding of associations between pairs of words, pairs of pictures, and word–picture pairs were compared. The aims were to determine, first, whether the neural correlates of associative encoding vary according to study material and, second, whether encoding of across- versus within-material item pairs is associated with dissociable patterns of hippocampal and perirhinal activity, as predicted by the “domain dichotomy” hypothesis of medial temporal lobe function. While undergoing fMRI scanning, subjects (n = 24) were presented with the three classes of study pairs, judging which of the denoted objects fit into the other. Outside of the scanner, subjects then undertook an associative recognition task, discriminating between intact study pairs, rearranged pairs comprising items that had been presented on different study trials, and unstudied item pairs. The neural correlates of successful associative encoding—subsequent associative memory effects—were operationalized as the difference in activity between study pairs correctly judged intact versus pairs incorrectly judged rearranged on the subsequent memory test. Pair type–independent subsequent memory effects were evident in the left inferior frontal gyrus (IFG) and the hippocampus. Picture–picture pairs elicited material-selective effects in regions of fusiform cortex that were also activated to a greater extent on picture trials than on word trials, whereas word–word pairs elicited material-selective subsequent memory effects in left lateral temporal cortex. Contrary to the domain-dichotomy hypothesis, neither hippocampal nor perirhinal subsequent memory effects differed depending on whether they were elicited by within- versus across-material study pairs. It is proposed that the left IFG plays a domain-general role in associative encoding, that associative encoding can also be facilitated by enhanced processing in material-selective cortical regions, and that the hippocampus and perirhinal cortex contribute equally to the formation of inter-item associations, regardless of whether the items belong to the same or to different processing domains.
Episodic memory—memory for unique events—depends upon the ability to encode and store associations between the different components that constitute an event (Tulving, 1983). Associations can be formed between components that are the primary focus of attention (such as words or pictures requiring a discriminative response—henceforth, items) and background information that constitutes the items' context. In addition to these item–context associations, associations can also be formed between two or more items that belong to the same event—item–item associations. It is this second type of association that is the focus of the present study.
It has been reported that memory for item–item associations is impaired by damage to the hippocampus to a greater extent than is memory for the items themselves (e.g., Mayes et al., 2004; Turriziani, Fadda, Caltagirone, & Carlesimo, 2004; Giovanello, Verfaellie, & Keane, 2003; but see Gold, Hopkins, & Squire, 2006; Stark & Squire, 2003), suggesting that the hippocampus may play a particularly important role in associative memory. Findings from functional neuroimaging studies converge with these neuropsychological results. Notably, several studies have employed fMRI and the “subsequent memory procedure” (Brewer, Zhao, Desmond, Glover, & Gabrieli, 1998; Wagner et al., 1998) to identify regions where activity elicited by pairs (or triplets; Qin et al., 2007) of study items differed according to whether or not the items were correctly judged to have been studied together on a later memory test (Park & Rugg, 2008; Chua, Schacter, Rand-Giovannetti, & Sperling, 2007; Summerfield et al., 2006; Prince, Daselaar, & Cabeza, 2005; Jackson & Schacter, 2004; Kirwan & Stark, 2004; Sperling et al., 2003). In each of these studies, it was reported that successful associative encoding (encoding predictive of memory for inter-item associations rather than the items themselves) was accompanied by enhanced activity (“subsequent memory effects”) in the hippocampus, most consistently, in its anterior aspect, with some studies reporting additional effects in adjacent medial temporal lobe (MTL) cortex (e.g., Summerfield et al., 2006; Kirwan & Stark, 2004). Together, the neuropsychological and neuroimaging findings are consistent with the widely held view that a key role of the hippocampus is to capture and store associations between the different components of an event, hence, to support memory representations of the event as a whole (see Eichenbaum, Yonelinas, & Ranganath, 2007, and Davachi, 2006 for reviews).
In addition to implicating the hippocampus in successful encoding of item–item associations, fMRI studies have also reported that associative encoding is accompanied by subsequent memory effects in the left inferior frontal gyrus (IFG) (Park & Rugg, 2008; Chua et al., 2007; Summerfield et al., 2006; Prince et al., 2005; Jackson & Schacter, 2004; Sperling et al., 2003; see Qin et al., 2007 for an example of bilateral IFG effects). The consistency of this finding, and its seeming insensitivity to variations in study materials or study task, have led to the suggestion that, along with the hippocampus, the left IFG supports processes that play a key role in associative encoding (Park & Rugg, 2008; but see also Blumenfeld, Parks, Yonelinas, & Ranganath, 2011).
As was just alluded to, fMRI studies of the encoding of item–item associations have employed a variety of different stimulus materials, including face–name pairs (Chua et al., 2007; Sperling et al., 2003), face–name–location triplets (Qin et al., 2007), face–house pairs (Summerfield et al., 2006), and word pairs (Park & Rugg, 2008; Prince et al., 2005). To our knowledge, though, no studies have been reported in which subsequent associative memory effects elicited by different study materials were directly contrasted. The question whether these effects vary according to study material is of interest for two reasons. First, the question is relevant to the proposal that successful encoding of a stimulus event is supported by cortical regions engaged as the event is processed “on-line” (Rugg, Johnson, Park, & Uncapher, 2008; Rugg, Otten, & Henson, 2002). According to this proposal, there is no single cortical region or network dedicated to encoding. Rather, successful encoding is a “by-product” of on-line processing (Roediger, Weldon, & Challis, 1989; Morris, Bransford, & Franks, 1977), and occurs most readily for stimulus events and their attributes accorded relatively high levels of attentional resources (Uncapher & Rugg, 2009). By this argument, subsequent memory effects reflect the benefit to encoding that occurs when a stimulus event is allocated a relatively large share of available processing resources. Thus, subsequent memory effects should vary in their cortical loci depending on the processes engaged to support the on-line demands of the study task.
Evidence in favor of this proposal has come from two sources. First, several studies in which the study task was varied while holding stimulus materials constant have reported task-dependent subsequent memory effects in cortical regions selectively activated by the corresponding task (Park, Uncapher, & Rugg, 2008; Mitchell, Macrae, & Banaji, 2004; Fletcher, Stephenson, Carpenter, Donovan, & Bullmore, 2003; Otten, Henson, & Rugg, 2002; Otten & Rugg, 2001). None of these studies investigated associative encoding, however. In the only relevant study to date (Park & Rugg, 2008), we contrasted subsequent associative memory effects across study tasks requiring semantic or phonological similarity judgments. Cortical effects were localized exclusively to the left IFG in both tasks, raising the possibility that, unlike the cortical correlates of the encoding of single items (Mitchell et al., 2004; Otten et al., 2002; Otten & Rugg, 2001) or item–context associations (Park et al., 2008), the correlates of item–item encoding do not vary according to the nature of the study task.
A second line of evidence supporting the proposal that cortical subsequent memory effects differ according to the processes engaged during the on-line processing of a stimulus event comes from studies that contrasted effects elicited by different classes of event or stimulus attribute (Gottlieb, Uncapher, & Rugg, 2010; Uncapher & Rugg, 2009; Uncapher, Otten, & Rugg, 2006; Powell et al., 2005). For example, Uncapher et al. (2006) reported that successful encoding of the contextual features of color and location were associated with enhanced activity in regions implemented in the processing of the corresponding feature. Similarly, Gottlieb et al. (2010) reported that the encoding of visual and auditory words was associated with subsequent memory effects in corresponding modality-selective cortical regions.
The present experiment addressed the question whether, as in the case of item and item–context encoding, the cortical correlates of the encoding of item–item associations vary according to the nature of the stimulus materials, as would be expected if associative encoding too is supported by regions engaged during the processing of the study event. We addressed this question by comparing the subsequent memory effects elicited by word and picture pairs in a formally identical study task. Thus, we were able to search for regions demonstrating subsequent memory effects common to the two pair types (anticipating that these regions would include the left IFG), and for regions where the two classes of effect dissociated, focusing on the hypothesis that material-specific subsequent memory effects should overlap regions selectively activated by the corresponding class of study material (Rugg et al., 2002, 2008; Mitchell et al., 2004).
The second reason why material-selective subsequent associative memory effects are of interest stems from the proposal that memories for “within-“ and “across-domain” item–item associations differ in their dependence on perirhinal cortex and hippocampus, respectively. According to the “domain dichotomy” hypothesis (Mayes, Montaldi, & Migo, 2007), associations between items that belong to different processing domains—that is, items that are processed in functionally distinct cortical regions—depend crucially on the hippocampus. By contrast, associations between items that belong to the same processing domain can be supported by perirhinal cortex in the absence of a hippocampal contribution. To our knowledge, a prediction of the domain dichotomy hypothesis—that there should be a double-dissociation between perirhinal and hippocampal subsequent associative memory effects according to whether items belong to the same or different processing domains—has not been evaluated. By including a third category of item pairs—comprising a word and a picture rather than two words or two pictures—we were able to obtain data relevant to this prediction.
Twenty-four subjects participated in the experiment (12 women; 18–28 years old). All subjects were right-handed native English speakers with no self-reported history of neurological or psychiatric illness. They were recruited from the University of California, Irvine (UCI), community and remunerated for their participation. Informed consent was obtained prior to participation in accord with the requirements of the UCI Institutional Review Board.
Items were drawn from a pool of 752 color pictures of objects and their corresponding names. Experimental stimuli consisted of 376 unrelated item pairs, in which the denoted objects were presented either in word or picture form. There were three types of item pair: word–word, picture–picture, and word–picture pairs. For half of the word–picture pairs, the word was presented above the picture, whereas for the remaining pairs, the ordering was reversed. A study list comprised a pseudorandom ordering of 270 pairs, 90 of which were word–word, 90 picture–picture, and 90 word–picture pairs. Four additional pairs were used as buffers (a word–word pair, a picture–picture pair, and two differently ordered word–picture pairs). The study list also included 90 null trials, randomly interspersed among the experimental trials. A test list consisted of 360 pairs, made up of a random sequence of 270 studied pairs and 90 new pairs (30 word–word, 30 picture–picture, and 30 word–picture). All of the studied items were presented at test in the same formats as at study. One hundred eighty of the studied pairs were presented in the same pairing as at study (intact pairs), and 90 pairs were composed of studied items that had been repaired from study (rearranged pairs). Both study and test lists were constrained such that no pair type occurred more than three times consecutively. An additional 12 pairs were used to form practice lists. Study and test lists were constructed separately for each subject, such that the denoted objects were presented as pictures to half of the subjects and as words to the remaining half. The position (above or below fixation) of an item in the pair was also counterbalanced between subjects.
All study pairs were projected onto a screen viewed by the subject via a mirror mounted on the scanner head coil. Words were presented in white upper case Helvetica 30-point font at a maximum visual angle of 8° × 1.5°. Pictures were presented in color at a maximum visual angle of 9.5° × 9.5° including a frame.
Prior to the experiment proper, participants were given instructions and practice on the study task. The experiment consisted of a single study–test cycle. Each study trial began with a red fixation cross that appeared on the screen for 100 msec. A study pair was then displayed for 1300 msec, with one item presented just above and the other just below fixation. The pair was replaced by a white fixation cross for a further 2100 msec to give a stimulus onset asynchrony of 3500 msec. Participants were instructed to judge which of the denoted objects would “fit” inside the other, and to depress a corresponding button with the appropriate finger of the right hand. Assignment of fingers to responses was counterbalanced across subjects. The study list was presented across two scanning sessions that were separated by a break of approximately 2 min.
The associative recognition test was administered outside of the scanner approximately 30 min after the end of the study phase. Before the test proper, participants were instructed and practiced on the test using the items from the practice study list. The memory test required participants to indicate whether each test pair was (i) intact: two items studied in the same pairing as at study; (ii) rearranged: two studied items that had been paired with different items at study; (iii) new: two unstudied items. Participants were instructed to respond “new” if they were unsure whether both items in a pair had been studied. The test was self-paced.
A Philips Intera Achieva 3-T MR scanner (Philips Medical Systems, Andover, MA) equipped with an eight-channel head coil was used to acquire both T1-weighted high-resolution anatomical images (MP-RAGE pulse sequence, FOV 240 × 180 mm, 1 mm isotropic voxels, sagittal acquisition) and T2*-weighted echo-planar images (EPIs) (SENSE factor of 1.5, flip angle 70°, FOV 240 × 180 mm, TR 2000 msec, TE 30 msec) per volume. Each volume comprised 30 slices oriented parallel to the AC–PC line (3-mm-thick slice, 1 mm interslice gap, 3 mm isotropic voxels) acquired in an ascending sequence. Data were acquired during the study phase in two scanning sessions, each comprising 326 volumes. An additional five volumes were collected at the beginning of each session but discarded to allow for T1 stabilization. The 3.5-sec stimulus onset asynchrony gave an effective sampling rate of the hemodynamic response of 2 Hz.
fMRI Data Analysis
Data preprocessing and statistical analyses were performed with Statistical Parametric Mapping (SPM5, Wellcome Department of Cognitive Neurology, London, UK: www.fil.ion.ucl.ac.uk), implemented in Matlab R2007a (Mathworks, Natick, MA). For each subject, functional images were registered to the first image of each session and temporally realigned to the middle slice using sinc interpolation. The resulting data were normalized to a standard EPI template based on the Montreal Neurological Institute (MNI) reference brain (International Consortium for Brain Mapping: www.loni.ucla.edu/ICBM) and resampled into 3-mm isotropic voxels using nonlinear basis functions (Ashburner & Friston, 1999). Normalized images were smoothed with an isotropic 8-mm full-width half-maximum Gaussian kernel. The time series in each voxel were high-pass filtered to 1/128 Hz to remove low-frequency noise and scaled to a grand mean of 100 across voxels and volumes. T1-weighted anatomical images were coregistered to the mean EPI volume and normalized to a standard T1 template of the MNI brain.
Prior to model estimation, the functional time series were concatenated across sessions. For each subject, neural activity was modeled by a 1300-msec duration boxcar function that began with the onset of each study pair. The predicted blood oxygen level dependent response was modeled by convolving these neural functions with a canonical hemodynamic response function. In addition, six regressors were employed to model movement-related variance, and session-specific constant terms were employed to model mean image intensity in each session.
In the first stage of data analysis, parameter estimates for events of interest were estimated for each subject using a general linear model. Nonsphericity of the error covariance was accommodated by an AR(1) model, in which the temporal autocorrelation was estimated by pooling over suprathreshold voxels (Friston et al., 2002). The parameters for each covariate and the hyperparameters governing the error covariance were estimated using restricted maximum likelihood. In a second stage, linear contrasts of these subject-specific parameter estimates were computed, treating subjects as a random effect.
For analysis of subsequent memory effects, six events of interest were defined: “word–word intact” (intact word pairs that were correctly endorsed as intact on the later test); “word–word rearranged” (intact word pairs that were incorrectly judged as rearranged on the later test); “picture–picture intact” (intact picture pairs that were correctly endorsed as intact on the later test); “picture–picture rearranged” (intact picture pairs that were incorrectly judged as rearranged on the later test); “word–picture intact” (intact word–picture pairs that were correctly endorsed as intact on the later test); “word–picture rearranged” (intact word–picture pairs that were incorrectly judged as rearranged on the later test). All other study pairs, including the pairs that contributed the items to the rearranged test pairs, pairs later judged to be new, and buffers were modeled as events of no interest. Unless otherwise noted, reliable effects were identified by contrasts which combined a height threshold of p < .001 and a cluster extent threshold of 5 contiguous voxels. The peak voxels of clusters exhibiting reliable effect are reported in MNI coordinates. For illustrative purposes, cortical effects were visualized onto the PALS-B12 atlas (Van Essen, 2005) in SPM 5 space using Caret (Van Essen et al., 2001) through mean fiducial mapping.
RTs to study pairs that were later re-presented in their intact form at test are shown in Table 1, segregated by pair type and later memory judgment. An ANOVA with factors of memory judgment (intact, rearranged) and pair type (word–word, picture–picture, word–picture) was conducted on these data, paralleling the fMRI analyses described below. The ANOVA revealed a main effect of pair type [F(1.69, 38.96)1 = 11.57, p < .001], study RTs being longest for word–word pairs and shortest for picture–picture pairs (all pairwise ps < .05). There was neither a main effect of memory judgment nor an interaction between judgment and pair type (Fs < 1).
|Word–word||1957 (333)||1969 (307)|
|Picture–picture||1809 (334)||1863 (336)|
|Word–picture||1880 (340)||1875 (285)|
|Word–word||1957 (333)||1969 (307)|
|Picture–picture||1809 (334)||1863 (336)|
|Word–picture||1880 (340)||1875 (285)|
Mean associative hit rates (correct intact judgments) were 0.46 (SD = 0.20) for word–word pairs, 0.58 (SD = 0.17) for picture–picture pairs, and 0.52 (SD = 0.15) for word–picture pairs. Mean associative false alarm rates (incorrect intact judgments) were 0.15 (SD = 0.14), 0.19 (SD = 0.14), and 0.15 (SD = .14) for word–word, picture–picture, and word–picture pairs, respectively. ANOVA of the discriminability index, pHit − pFalse Alarm, gave rise to a significant effect of pair type [F(1.9, 44.3) = 4.80, p < .05]. Follow-up tests revealed that performance for word pairs (0.30) was significantly lower than that for either picture or word–picture pairs (0.39 and 0.38, respectively).
Subsequent memory analyses were based on contrasts between the activity elicited by study pairs that were later correctly endorsed as intact (associative hits) as opposed to pairs receiving an incorrect rearranged response (associative misses). Across subjects, there was a minimum of eight trials in each response category [picture–picture: range = 9–50 (mean = 26); word–word: 9–49 (25); word–picture: 8–51 (26)].
Material-independent Subsequent Associative Memory Effects
We identified subsequent associative memory effects common to the three types of study pairs by exclusively masking the main effect of subsequent memory (all correct intact > all incorrect rearranged, thresholded at p < .001) by the F contrasts of the three Study pair × Subsequent memory interactions (each thresholded at p < .1) that, between them, tested for all possible differences in the magnitudes of the subsequent memory effects as a function of pair type. Thus, voxels identified by this procedure demonstrate a significant (p < .001) subsequent associative memory effect that did not differ in magnitude (at p < .05, one-tailed) between the different pair types. The results of the procedure are illustrated in Figure 1 and reported in Table 2. Among the regions demonstrating material-independent effects were the left IFG, with effects extending from its dorsal to its ventral aspects, and the right posterior hippocampus. As is evident in Figure 2, the effects in this hippocampal region—which are of particular interest in view of the predictions of the domain dichotomy hypothesis—were independently reliable (one-tailed t tests) for each of the three pair types.
|Coordinates (x y z)|
|No. of Voxels|
|−48||33||−12||4.44||31||L Ventrolateral prefrontal cortex||47|
|−51||24||6||4.86||399||L Ventrolateral prefrontal cortex||45/9/8|
|33||−39||0||4.12||31||R Posterior hippocampus|
|−42||−69||21||3.49||19||L Middle temporal gyrus/angular gyrus||39|
|−48||−45||−18||3.41||20||L Fusiform cortex||37|
|Coordinates (x y z)|
|No. of Voxels|
|−48||33||−12||4.44||31||L Ventrolateral prefrontal cortex||47|
|−51||24||6||4.86||399||L Ventrolateral prefrontal cortex||45/9/8|
|33||−39||0||4.12||31||R Posterior hippocampus|
|−42||−69||21||3.49||19||L Middle temporal gyrus/angular gyrus||39|
|−48||−45||−18||3.41||20||L Fusiform cortex||37|
In view of prior findings of subsequent associative memory effects in the anterior hippocampus (see Introduction), we queried whether additional MTL effects were evident when the threshold for the main effect of subsequent memory was lowered to p < .005. This analysis revealed a cluster with a peak in the vicinity of the left anterior hippocampus (−21, −9, −15, Z = 3.07). The hippocampal peak is in close proximity to the hippocampal associative memory effect reported by Summerfield et al. (2006) for face–house associations (−18, −11, −18). The present effect survived small-volume correction (p < .05, corrected according to Gaussian random field theory; Worsley et al., 1996) based upon a 5-mm sphere centered on the locus of the previously reported effect. The mean clusterwise parameter estimates of the subsequent associative memory effects in the left anterior hippocampus are illustrated for each pair type in Figure 2. One-tailed t tests revealed that the effects were independently reliable for the picture–picture and word–word pairs only.
Material-dependent Associative Encoding Effects
We identified material-dependent subsequent associative memory effects using the data from the word–word and picture–picture study pairs, as no such effects would be predicted for mixed pairs. As in prior studies (e.g., Gottlieb et al., 2010), we used inclusive masking to search for regions where material-dependent differences in the magnitude of subsequent memory effects (indexed by directional Material × Subsequent memory interaction effects, thresholded at p < .01) overlapped with the corresponding directional effect of study material (pictures > words and vice-versa, each thresholded at p < .001). The conjoint significance level of these two orthogonal contrasts is p < .0001, as estimated by Fisher's procedure (Lazar, Luna, Sweeney, & Eddy, 2002). As is illustrated in Figure 3, subsequent memory effects selective for picture pairs were identified in bilateral fusiform cortex. The loci of these effects were, respectively, −36, −45, −21 (peak Z = 3.45); 39, −45, −27 (peak Z = 3.03); and 54, −51, −18 (peak Z = 2.77). Follow-up t tests revealed that, in each case, the mean clusterwise parameter estimates demonstrated significant subsequent memory effects for picture pairs (all p < .025, one-tailed) but not for word pairs. The same approach revealed no evidence for word-selective subsequent memory effects. However, when the threshold for the interaction contrast was lowered to p < .05 (giving a conjoint significance level of p < .0005), a cluster was identified in left lateral temporal cortex (middle temporal gyrus, BA 21; −66, −33, −3, peak Z = 2.15), as is illustrated in Figure 3. Follow-up t tests revealed that although the subsequent memory effect elicited by word pairs was significant (p < .025, one-tailed, for the mean clusterwise parameter estimates), the effect elicited by picture pairs was not.
No Dissociation between Hippocampal and Perirhinal Subsequent Memory Effects for Across- vs. Within-domain Study Pairs
Motivated by the domain dichotomy hypothesis (see Introduction), we searched specifically for hippocampal subsequent memory effects that were either uniquely associated with word–picture pairs, or that were enhanced for this pair type relative to the other two types. No such effects could be identified. Even at an uncorrected threshold of p < .05, there were no hippocampal voxels exhibiting a word–picture pair subsequent memory effect other than in the right posterior hippocampal region, where effects common to all three classes of study pair were evident (see Figure 1). Likewise, again at a threshold of p < .05, no voxels were revealed by the interaction contrasts testing for differences in the magnitude of word–picture and either word–word or picture–picture subsequent memory contrasts. In short, we could find no evidence to support the prediction that hippocampal subsequent memory effects would be enhanced for the encoding of across- rather than within-domain study pairs.
Analogously, we searched for perirhinal regions demonstrating either unique or enhanced subsequent memory effects for word–word or picture–picture pairs relative to word–picture pairs. Using the anatomical landmarks described by Insausti et al. (1998) to localize perirhinal cortex, we were unable to identify subsequent associative memory effects for any class of item pair. When we defined perirhinal cortex according to the anatomical criteria proposed by Ding and Van Hoesen (2010), however, picture–picture, though not word–word, associative subsequent memory effects were evident in the left hemisphere when the threshold was reduced to p < .01 (−39 −12 −33, Z = 3.01; Figure 4A). Word–picture pairs elicited slightly more robust effects that peaked at exactly the same locus (Z = 3.13; Figure 4B). In both cases, the effects survived small-volume correction (p < .05) within 5-mm spheres centered on the peak of either the associative subsequent memory effects localized to perirhinal cortex by Haskins, Yonelinas, and Ranganath (2008; −33, −12, −33) or the perirhinal subsequent memory effects predictive of memory for item–color associations reported by Staresina and Davachi (2010; −33, −9, −30).2
Successful encoding of all three classes of study pair was associated with enhanced activity in the left IFG, consistent with prior findings. Subsequent associative memory effects were also evident in the left anterior hippocampus, again consistent with previous findings. These hippocampal effects were accompanied by additional and more robust effects in the right posterior hippocampus. Word-selective subsequent memory effects were identified in left lateral temporal cortex, whereas picture-selective effects were evident in bilateral extrastriate visual cortex. We found no evidence that successful encoding of word–picture pairs placed more demands on the hippocampus than did the encoding of word or picture pairs, and nor was there any evidence for the opposite dissociation in perirhinal cortex. Below, we expand upon these results and their possible functional significance.
Here, we briefly discuss the behavioral results and also comment on the limitations that the retrieval test places on interpretation of the fMRI findings. The subjective nature of the study judgments (judging which of two entities would “fit into the other”) precludes an assessment of accuracy. RTs differed reliably across the three pair types, however, with picture pairs attracting the fastest responses and word pairs the slowest. This pattern is consistent with evidence that, other things being equal, pictures activate conceptual representations more quickly than words do (e.g., Stenberg, Radeborg, & Hedman, 1995; Smith & Magee, 1980; Potter & Faulconer, 1975). Importantly, the RT advantage for pictures did not interact with the factor of subsequent memory; indeed, subsequent memory effects on RT were nonsignificant, implying that the corresponding fMRI effects were not confounded with systematic differences in the difficulty of the study judgments. There were, however, significant differences between the pair types in associative memory performance, in that word pairs were remembered less well than picture or word–picture pairs. These differences are consistent with the well-known picture superiority effect in long-term memory (Mintzer & Snodgrass, 1999; Madigan, 1983; Nelson, Reed, & Walling, 1976; Paivio & Csapo, 1973).
Associative recognition tests of the kind employed in the present study are assumed to depend on memory for inter-item associations, on the grounds that recognition of each individual member of a test pair is insufficient to support the discrimination between intact and rearranged pairs. It is often also assumed that associative recognition depends primarily upon the same “recollective” retrieval processes that support the retrieval of qualitative information in tests of item recognition and recall, and receives little or no contribution from the processes that support the acontextual sense of familiarity that can also underlie recognition of single items (see Eichenbaum et al., 2007; Yonelinas, 1997). It has been suggested, however, that associative recognition can be supported by familiarity if item pairs become unitized during encoding so as to form a single “item-like” representation (Quamme, Yonelinas, & Norman, 2007; Yonelinas, Kroll, Dobbins, & Soltani, 1999; but see Mickes, Johnson, & Wixted, 2010 for an alternative view). From this perspective, it is important to note that the associative recognition test employed in the present study did not permit “intact” judgments to be segregated by whether they were based on recollection or familiarity. Thus, although the subsequent memory effects elicited by study pairs that went on to receive a correct “intact” endorsement likely reflect processes underlying the successful encoding of item–item associations,3 we cannot distinguish between processes supporting the later recollection of the study pairs and processes that might have supported their unitization and subsequent familiarity-based recognition.
We turn first to the subsequent associative memory effects that were common to the three types of study pair. Material-independent effects were prominent along much of the left IFG, extending into superior lateral frontal cortex (see Figure 1). Together with prior findings, the present results support the proposal that left inferior frontal cortex plays a key role in facilitating the encoding of item–item associations regardless of the nature of the study materials or the study task (Park & Rugg, 2008). In light of the strong evidence for functional heterogeneity within the IFG (e.g., Badre & Wagner, 2007; Poldrack et al., 1999), it seems unlikely that this role is limited to a single cognitive function.
Subsequent memory effects common to the three pair types were also evident in the hippocampus and adjacent MTL cortex. Notably, effects were identified in the vicinity of the left anterior hippocampal region implicated in associative encoding in several previous studies (Chua et al., 2007; Summerfield et al., 2006; Jackson & Schacter, 2004; Sperling et al., 2003). These anterior effects were not independently reliable for word–picture pairs, although the absence of a Pair type × Subsequent memory interaction cautions against the conclusion that the encoding of these pairs engaged this hippocampal region to a lesser extent than did encoding of the other pair types. Nonetheless, the absence of reliable word–picture effects in the anterior hippocampus, given the presence of significant effects for word and picture pairs, does not bode well for the domain dichotomy hypothesis of associative processing within the MTL (see below).
A material-independent subsequent memory effect was also evident in the right posterior hippocampus, a region that, to our knowledge, has not previously been implicated in associative encoding in studies employing the subsequent memory procedure (see Vogelaere, Santens, Achten, Boon, & Vingerhoets, 2010, for evidence of a posterior hippocampal contribution to associative encoding in a study employing a blocked experimental design). Why this region should have demonstrated subsequent associative memory effects in the present experiment but not in others is unclear. It will be of interest to determine whether the effects in this region reflected a contribution to the encoding of the study pairs that was independent of the contribution of the left anterior hippocampus—for example, by encoding the visual images formed in response to the study pairs rather than, say, associations between the semantic attributes of the two items—or whether instead these different hippocampal regions are parts of a more integrated network. The present finding that word–picture pairs elicited a posterior but not an anterior hippocampal subsequent memory effect hints at the possibility that the different effects are indeed functionally independent.
In addition to effects common to the three pair types, material-selective subsequent associative memory effects were also evident. Picture-selective subsequent memory effects were identified in bilateral fusiform cortex, where they overlapped with regions demonstrating enhanced activity for pictures relative to words. Significantly, these effects also overlapped regions previously reported to demonstrate subsequent memory effects for successful versus unsuccessful source memory of single pictures (Cansino, Maquet, Dolan, & Rugg, 2002). A complementary dissociation (albeit, evident only at a lower threshold) was evident for word pairs in a word-selective region of left lateral temporal cortex. These word-pair selective effects overlapped a region of the middle temporal gyrus previously implicated in the controlled retrieval of semantic information (Gold & Buckner, 2002). These findings bring the evidence for material selectivity of subsequent associative memory effects into line with findings for the successful encoding of individual items (Powell et al., 2005) and item–context associations (Gottlieb et al., 2010; Uncapher et al., 2006). Thus, the present findings suggest that—as was first proposed by Summerfield et al. (2006)—encoding of item–item associations is facilitated by enhanced processing within some of the same cortical regions that support the on-line processing of the individual study items. One possible explanation for this finding (cf., Cansino et al., 2002) is that enhanced processing of the individual study items provides more differentiated or specific input to the relational processes that support the formation of inter-item associations. Alternatively, the processes reflected by these material-selective subsequent memory effects may play a more direct role in the formation of inter-item associations, for example, by contributing to the formation of unitized representations of study pairs that support familiarity-driven associative recognition on the subsequent memory test (see above and Staresina & Davachi, 2010).
Although we were able to identify a cortical region demonstrating a word-selective subsequent memory effect, this effect was arguably modest given the extent of the cortical regions that were selectively activated by word relative to picture pairs. Indeed, as is evident in Figure 3, these word-selective regions encompassed much of the left IFG. This observation prompted us to ask whether any of the regions demonstrating material selectivity for words overlapped with the regions that demonstrate subsequent memory effects for both words and pictures. As can be seen in Figure 5, this is indeed the case: A substantial fraction of the middle and dorsal aspects of the left IFG where subsequent associative memory effects are found for both classes of material is also selectively activated by words relative to pictures. This finding is reminiscent of a result from a prior study in which we investigated subsequent associative memory effects elicited by word pairs that were subjected to either semantic or phonological similarity judgments (Park & Rugg, 2008). We identified a left IFG region, partially overlapping the region identified here, that demonstrated both a task effect (greater activation for the semantic task) and task-independent subsequent memory effects. This finding led us to propose that the left IFG supports cognitive operations that, while engaged more extensively in response to semantic than phonological processing demands, are critical for successful associative encoding regardless of the specific processing demands of the study task. The present findings suggest an analogous account, in which processes supported by the IFG are engaged more extensively during the relational processing of words than pictures, but support associative encoding of both classes of material. Together with prior reports of subsequent associative memory effects in the left IFG across a wide variety of stimulus materials and study tasks (Park & Rugg, 2008; Chua et al., 2007; Summerfield et al., 2006; Prince et al., 2005; Jackson & Schacter, 2004; Sperling et al., 2003), the findings from the present study suggest that the proposal that episodic encoding does not depend upon a “dedicated” cortical region (Rugg et al., 2002, 2008) may require qualification: Likely acting in concert with the MTL and material-specific cortical regions, the left IFG appears to play a pre-eminent role in the encoding of item–item associations.
In addition to the employment of word and picture pairs, which afforded the opportunity to investigate the effects of study material on subsequent associative memory effects, we also included “mixed” study pairs comprising a word and a picture. According to the domain dichotomy hypothesis of MTL function, memory for associations between items belonging to different processing domains is more dependent on hippocampally mediated binding operations than memory for associations between items belonging to the same processing domain. This is because the representations of within-domain items are more likely to converge and become bound together in extra-hippocampal MTL regions, especially perirhinal cortex. As already noted, we found no evidence to support this hypothesis; indeed, in as much as there were any trends for pair-type-selective subsequent memory effects in the hippocampus, these took the form of more extensive effects for the within- rather than the across-domain pairs. Additionally, perirhinal subsequent memory effects were equally evident for word–picture and picture–picture pairs, and were undetectable for word–word pairs. These findings are clearly inconsistent with the prediction, derived from the domain dichotomy hypothesis, that perirhinal effects should predominate for within- relative to across-domain pairs (it should be noted, however, that fMRI signal return in the perirhinal region, especially in its anterior extent, can be compromised by magnetic susceptibility artifact. Therefore, null findings for this region should be treated with circumspection).
Clearly, the findings with respect to the MTL offer no support for the domain dichotomy hypothesis. They converge with the results of a recent block-design fMRI study that also failed to find differential hippocampal activity for the encoding of across- and within-domain associations (face–face vs. face–laugh pairings; Holdstock, Crane, Bachorowski, & Milner, 2010), and with neuropsychological evidence that the hippocampus plays an equivalent role in memory for across- and within-domain item–item associations (Turriziani et al., 2004). Nonetheless, it could be argued that the nature of our study task means that the findings should not count against the domain dichotomy hypothesis. The task necessitated the generation and relational processing of visually based representations of study items, and this could have limited the impact of the within- versus across-domain manipulation by encouraging the processing of the words and pictures in a common representational domain.4
As already noted, reliable perirhinal subsequent memory effects were equally evident for both picture–picture and word–picture pairs, contrary to the prediction of the domain dichotomy hypothesis. According to one proposal (e.g., Haskins et al., 2008; Diana, Yonelinas, & Ranganath, 2007), perirhinal subsequent memory effects support associative recognition to the extent that the items can be unitized into a single item-level representation and subsequently recognized through a sense of familiarity for the representation. Our findings are consistent with this proposal if it is assumed that, on at least some proportion of study trials, pair members underwent unitization (for example, through the generation and encoding of a mental image that incorporated the referents of both items). Our findings can also be accommodated, however, under the proposal that the mnemonic functions of the hippocampus and surrounding MTL cortex are not sharply demarcated and that, like the hippocampus, perirhinal cortex contributes to relational memory for inter-item associations, albeit to a lesser extent (Squire, Wixted, & Clark, 2007). Resolution of this issue will require the employment of test procedures that segregate individual associative recognition judgments according to whether they are familiarity- or recollection-based.
This research was supported by the National Institute of Mental Health Grant MH074528.
Reprint requests should be sent to Heekyeong Park, Department of Psychology, University of Texas-Arlington, Arlington, TX 75069-19528, or via e-mail: firstname.lastname@example.org.
The degrees of freedom of F ratios are reported after Greenhouse–Geisser correction for nonsphericity.
We thank B. Staresina for providing us with these coordinates, which were not reported in Staresina and Davachi (2010).
The rationale for using the contrast between study pairs accorded correct “intact” versus incorrect “rearranged” judgments on the later memory test to identify the neural correlates of associative encoding rests on the assumption that memory for individual test items is independent of memory for their association. That is, it is assumed that item memory is equated in test pairs accorded correct versus incorrect associative recognition judgments. A reviewer noted that this assumption may be unwarranted, and thus, associative subsequent memory effects could be confounded with effects reflecting the encoding of item rather than associative memories. This point serves as a caveat to the interpretation not only of the present findings but also of those from prior studies that employed similar test procedures.
However, Mayes et al. (2004) have argued that elaborative study processing should not weaken the differential dependence of within- and across-domain item pairs on memory representations supported by perirhinal cortex (p. 780).