Abstract

One fifth of neurons in the medial-temporal lobe of human epilepsy patients respond selectively to categories of images, such as faces or cars. Here we show that responses of hippocampal neurons are rapidly modified as subjects alternate (over 60 sec) between two tasks (1) identifying images from a category, or (2) playing a simple video game superimposed on the same images. Category-selective responses, present when a subject identifies categories, are eliminated when the subject shifts to playing the game for 87% of category-selective hippocampal neurons. By contrast, responses in the amygdala are present during both tasks for 72% of category-selective amygdalar neurons. These results suggest that attention to images is required to evoke selective responses from single neurons in the hippocampus, but is not required by neurons in the amygdala.

INTRODUCTION

A key part of human conscious experience is the ability to flexibly deploy cognitive resources to analyze and react to different aspects of the environment, that is, to select different aspects of the world and attend to them. Such selection functions to filter the large amount of sensory information which is available at any instant in time, choosing that small subset which is important to guide behavior, and emphasizing it for further evaluation (Pashler, 1998, Chap. 1). Historically, the mechanisms underlying attentional selection have been envisioned as a specific processing bottleneck, either early (Treisman, 1969; Broadbent, 1958) or late in sensory processing (Norman, 1968; Deutsch & Deutsch, 1963), which limits the information reaching awareness. Recent theoretical accounts have advocated more distributed processes and limitations (Lavie, 2005; Yantis & Johnston, 1990), and these are consonant with neurophysiological recordings showing that attention affects both subcortical and cortical neural responses (Kastner & Pinsk, 2004; Shipp, 2004).

Our current knowledge of attentional mechanisms has been derived largely from noninvasive imaging (Haxby, Courtney, & Clark, 1998) and electrophysiological studies in human subjects (Luck & Girelli, 1998), as well as single-neuron recordings in nonhuman primates (Maunsell & Treue, 2006; Motter, 1998; Luck, Chelazzi, Hillyard, & Desimone, 1997; Moran & Desimone, 1985; Andersen & Mountcastle, 1983). Both noninvasive imaging and scalp recordings indicate those brain areas whose average inputs or activity change as attention is shifted. There are, of course, many patterns of change of single-neuron activity which produce the same average change in an area, or are produced by the same average change in inputs. This study was designed to extend knowledge of attentional mechanisms in human subjects derived from imaging studies by examining changes in single-neuron firing as subjects shifted between a task which focused attention on visual images and a task not involving the images.

Single-neuron activity can also be recorded in nonhuman primates, and indicates the single-neuron correlates of the types of attentional shifts for which these animals can be trained over several months. For human subjects, new tasks with different attentional demands can be learned in 1 min following verbal instruction. The experiments reported here were designed to extend primate recordings by measuring changes in single-neuron firing within the human brain during overt attentional shifts in a newly acquired task.

Previous experiments (Davis, Hutchison, Lozano, Tasker, & Dostrovsky, 2000) have shown baseline firing rate differences for neurons in the anterior cingulate during performance of different attentionally demanding tasks. The subjects of the present study were human epilepsy patients who shifted their attention between two tasks: identifying images of faces from among other categories and playing a simple video game superimposed on these images (Figure 1 
Figure 1

One screen frame captured during video game play. This frame shows an image from the famous people category and the superimposed video game. The triangular cannon is in the center; several shots are present on the screen and a rock which has been hit and is exploding is shown upper center.

Figure 1

One screen frame captured during video game play. This frame shows an image from the famous people category and the superimposed video game. The triangular cannon is in the center; several shots are present on the screen and a rock which has been hit and is exploding is shown upper center.

). The subjects shifted between these tasks over the course of 1-min intervals after receiving simple verbal instructions. The tasks were designed to evoke previously reported category-selective visual responses (Kreiman, Koch, & Fried, 2000), to permit the same visual display to be present during both tasks, and to be engaging for the patient.

During task performance, we recorded the firing activity of single neurons within the medial-temporal lobe (MTL). Previous studies reported category-selective visual responses within several MTL structures: the hippocampus, the parahippocampal gyrus, the amygdala, and the entorhinal cortex (Kreiman et al., 2000; Fried, MacDonald, & Wilson, 1997). Here we focus on the influence of selective attention on category-selective responses of single neurons in two of these structures, the hippocampus and the amygdala.

The hippocampus was of particular interest given its reciprocal connections to other brain areas involved in high-level visual processing (Felleman & Van Essen, 1991), as well as its involvement in declarative memory for items and events (Viskontas, Knowlton, Steinmetz, & Fried, 2007; Squire, 2004; Cohen et al., 1999; Squire & Zola, 1996). The amygdala was of interest given its role in the emotional evaluation of faces (Adolphs, Tranel, & Buchanan, 2005; Zald, 2003; Oya, Kawasaki, Howard, & Adolphs, 2002), its potential involvement in a rapid subcortical network for face processing (Pessoa, Japee, Sturman, & Ungerleider, 2006; Whalen et al., 2004; Compton, 2003), as well as its involvement in declarative and nondeclarative memory for emotional events (Adolphs et al., 2005; Phelps, 2004; Richter-Levin, 2004).

By examining changes in single-neuron firing as subjects shifted between two tasks over short time intervals, this design examines how the visual responses of single neurons change as human subjects shift attention in a relatively natural way. When subjects attended to images, we found category selectivity in one fifth of hippocampal and amygdalar neurons, in agreement with previous report (Kreiman et al., 2000). When attention was shifted between tasks, we found two distinct classes of cellular responses: The category selectivity of neurons in the hippocampus was strongly inhibited by shifting to game play, whereas the category selectivity of neurons in the amygdala was largely unaffected.

METHODS

Subjects were patients with pharmacologically resistant epilepsy. Extensive noninvasive evaluation did not yield concordant data corresponding to a single epileptogenic focus, and therefore, the patients were stereotactically implanted with up to 10 chronic intracranial depth electrodes for 1 to 2 weeks to determine the origin of their seizures for possible surgical resection. Through the lumen of the electrodes, up to eight platinum–iridium microwires (40 μm diameter) were inserted (Babb, Carr, & Crandall, 1973). All studies conformed with the guidelines of the Medical Institutional Review Board at UCLA. Recordings were obtained from 10 subjects (6 men; 9 right-handed; 17 to 44 years old). The sites of electrode implantation were based exclusively on clinical criteria, and the location was verified by structural magnetic resonance imaging. Individual microwires extended 4 mm from the tip, lying in a cone with an opening angle of less than 45°.

The activity of single neurons was isolated from the electric potential recorded between microwire tips and a nearby reference microwire. This difference was amplified (×10,000) and band-pass filtered (278–5000 Hz) and possible action potential events were isolated using a fixed threshold. The events were sorted to represent the activity of single neurons using the Spiker program (http://ramonycajal.mit.edu/kreiman/academia/spike_sorting.html). Only clusters of events with an initial depolarization followed by a hyperpolarization, minimal repetitions at power line harmonic frequencies, and a decrease in the fraction of interspike intervals near zero delay, were considered to arise from single neurons and were included in further analysis. Each cluster thus contains similar events over the entire recording session and across the changes between tasks.

The experimental task involves shifting attention between two superimposed video streams, similar to the technique used by Neisser and Becklen (1975) to induce unawareness of one of the streams. One stream consisted of a sequence of images selected randomly from several categories (4 to 7 based on clinical constraints): animals, cars, emotional faces, famous people, everyday objects, patterns, and spatial outdoor scenes. The images and categories were chosen from a set shown in previous studies to yield measurable neural responses (Kreiman et al., 2000). Each image subtended about 10° visual angle and was presented for 1 sec with 500 msec of black screen in between.

While these images were presented, an outline of a video game was presented over the images or the background. The video game (Figure 1) was a modified version of “Asteroids” (Atari, 1979), where the object is to rotate the central cannon and fire at moving rocks by depressing the space bar. During each game, the subjects were asked to either: (game play) play the video game while ignoring the images presented in the background, or (picture identification) press a key when images from one specific image category (faces, or in 4 experiments, outdoor scenes) were presented while ignoring the outline display of the last game, which was automatically replayed by the computer. Each game involved presentation of 30 images and lasted 45 sec.

The subjects alternated between game play and picture identification and would play 10 games per experimental session (with some variability due to clinical constraints). The visual display during each game of picture identification was identical to the display during the preceding game play (although this ordering introduces a potential confound, prior studies show it is unlikely to affect the results here; see Discussion). Each image was presented during multiple games and so the appearance of the game was random with respect to image presentations. On average, each subject participated in two sessions, normally on separate days.

This task was designed to present the same visual stimulus in two conditions and to shift the subject's attention between the images and the game. To verify a shift, normal subjects (n = 4) were asked to simultaneously play the game and identify pictures (patients were not asked to perform the tasks simultaneously as it would be relatively taxing during their postoperative recovery). During simultaneous performance, the error rate while identifying a specific image category was 2% (average number of errors = 15), 4.5 times higher than when they performed picture category identification alone (average number of errors = 3) during the initial 50 games played (approximately the total number played by the patients). Fisher's exact test rejects (p < .01) the hypothesis that the error rates were the same during simultaneous performance and picture identification.

Background firing rates were determined from the 500 msec prior to image presentation and were averaged over all images. Firing during the 1000 msec of image presentation was examined for category-selective visual responses, that is, whether a neuron responded specifically to one or more image categories during either game play or picture identification. Trials during game play and during picture category identification were analyzed in separate blocks. For picture identification, both correctly and incorrectly performed trials were analyzed together because this is most similar to the analysis during game play, and because the small number of error trials (<1%) would have little effect on the overall results.

A neuron was considered to have a category-selective visual response if there was a significant effect of image category on the average firing rates during 1000 msec of image presentation (one-way analysis of variance [ANOVA], p < .05). The p value for this test (Kleinbaum, 1988) measured the degree of category selectivity. For neurons with category-selective responses, Tukey's method of honest significant differences (Kleinbaum, 1988) was used to identify those pairwise comparisons between categories which were significant.

A neuron was considered to have a general response to presentation of any picture if the firing rate during 1000 msec of image presentation, averaged over all image categories, was significantly different from the average activity in the 500 msec preceding image presentation (unpaired t test, p < .05). A neuron was considered to have a change in baseline firing due to play condition if the average firing rate in the 500 msec preceding image presentation differed significantly between picture identification and game play (unpaired t test, p < .05).

RESULTS

In total, we recorded from 190 individual neurons in the amygdala (59 neurons), entorhinal cortex (38), hippocampus (36), parahippocampal gyrus (22), orbito-frontal cortex (13), supplementary motor area (13), and anterior cingulate gyrus (9) during 20 experiments in 10 patients. The limited number of neurons and category-selective neurons in areas other than the hippocampus and the amygdala prevents firm conclusions from being drawn about these other areas. Neurons in these other areas are discussed in the “Responses in Other Brain Areas” section, but are otherwise not analyzed further here. The average firing rate for neurons in the hippocampus and the amygdala during picture identification was 3.0 spikes/sec (sp/sec), in agreement with a previous report (Kreiman et al., 2000). There was no difference in background firing rate between brain areas (one-way ANOVA, p = .33) or between areas in the likelihood of the background firing rate changing between game play and picture identification (Fisher's exact test, p = .28).

Neurons Showing Category Selectivity

Consistent with a previous report (Kreiman et al., 2000), as the patient performed picture identification, 18 neurons (19%) showed a significant effect of picture category on the firing rate during 1 sec of image presentation (one-way ANOVA, p < .05; over three times the number expected by chance). Eleven of these selective neurons were in the amygdala (recorded from 5 patients) and seven were in the hippocampus (recorded from 6 patients). No systematic differences of category selectivity between sides of the brain containing or not containing subsequently resected areas were noted and so data from both sides were pooled by brain area.

An example of selective responses for a neuron in the amygdala during game play is shown in Figure 2,
Figure 2

Raster plots and peristimulus time histograms for a neuron in the amygdala during game play. The solid gray line shows the average firing rate during the 500 msec preceding picture presentation.

Figure 2

Raster plots and peristimulus time histograms for a neuron in the amygdala during game play. The solid gray line shows the average firing rate during the 500 msec preceding picture presentation.

; the average firing rates for this neuron during presentation of images from seven categories are shown in Figure 3A,
Figure 3

Average firing rates during image presentation for two neurons in the amygdala during picture identification (filled bars) and game play (unfilled bars). Background firing rates during each condition are shown by the horizontal lines (test for category selectivity compares to average bar height, not this background). (A) Same cell as Figure 2. Numbers below the bars show the relative response ratio for each category during both picture identification and game play. (B) Cell in the amygdala of a separate patient. Category selectivity is largely due to the excitatory response to famous people.

Figure 3

Average firing rates during image presentation for two neurons in the amygdala during picture identification (filled bars) and game play (unfilled bars). Background firing rates during each condition are shown by the horizontal lines (test for category selectivity compares to average bar height, not this background). (A) Same cell as Figure 2. Numbers below the bars show the relative response ratio for each category during both picture identification and game play. (B) Cell in the amygdala of a separate patient. Category selectivity is largely due to the excitatory response to famous people.

during both game play and picture identification. This neuron showed an excitatory response to images of objects and an inhibitory response to images of animals. Figure 3A also shows the background firing rates during these tasks. This neuron showed a clear category-selective visual response during both task types. It is important to note that category selectivity, as defined here, does not require a particular category to have a response significantly different from baseline; rather, there must be differences between the responses to the categories, which taken together, show a significant effect of the image category on the responses. Previous reports of category selectivity (Kreiman et al., 2000) additionally tested each category separately relative to baseline, performing up to nine such tests per neuron, which may account for the slightly lower percentage of selective cells by that method (14%).

Figure 3B shows the average responses of a separate amygdalar neuron (in a different patient) during presentation of images from the same seven categories. For this neuron, presentation of images of famous people evoked an excitatory response during both picture identification and game play.

Figure 4,
Figure 4

Average firing rates during image presentation for two hippocampal neurons with category-selective effects. Responses during picture identification (filled bars) and game play (unfilled bars) are shown. Background firing rates during each condition are shown by the horizontal lines. (A) Neuron in the hippocampus denoted by 3 in Figure 5, five categories of images shown. (B) Neuron in the hippocampus denoted by 4 in Figure 5, four categories of images shown.

Figure 4

Average firing rates during image presentation for two hippocampal neurons with category-selective effects. Responses during picture identification (filled bars) and game play (unfilled bars) are shown. Background firing rates during each condition are shown by the horizontal lines. (A) Neuron in the hippocampus denoted by 3 in Figure 5, five categories of images shown. (B) Neuron in the hippocampus denoted by 4 in Figure 5, four categories of images shown.

shows the average responses to image presentation for two hippocampal neurons with category-selective responses. The neuron in Figure 4A derives this selectivity primarily from an excitatory response to images of spatial scenes during picture identification. For this neuron, image categories with responses most different from the background firing rate during picture identification become more like the background rate during game play. The hippocampal neuron in Figure 4B derives selectivity primarily from an inhibitory response to images of cars during picture identification, and again, categories with responses most different from background during picture identification have responses closer to background during game play.

Changes in Selectivity between Tasks

These experiments were designed to study how category-selective MTL neurons are affected by shifting the task from one involving the images to one not involving the images. As illustrated in Figures 3 and 4, both increases or decreases of responses to particular categories can result in a change of category selectivity. The degree of response difference between categories, and the degree of response selectivity during either game play or picture identification, was measured by the p value of the one-way ANOVA test for an effect of category on the responses; smaller p values corresponding to a broader separation of responses to different categories. The negative log of the p value during picture identification and game play is plotted in Figure 5 
Figure 5

p Value of test for a category effect for each neuron during picture identification (negative base 10 logarithm on horizontal axis) and during game play (negative base 10 logarithm on vertical axis). All 95 neurons, with and without a significant category-selective effect, are shown. Symbols denote significant effects of image category on the response (one-way ANOVA, p < .05): filled dots = no effect; open circles = significant effect during picture identification or game play (not both); cross = significant effect during both picture identification and game play, although magnitude of effect may vary between tasks. Numbers near symbols indicate specific neurons: 1 = amygdala neuron shown in Figures 2 and 3A; 2 = amygdala neuron shown in Figure 3B; 3 = hippocampus neuron shown in Figure 4A; 4 = hippocampus neuron shown in Figure 4B. Selective cells in the hippocampus have a greater tendency to lie below the diagonal to the right, showing greater selectivity during picture identification. Although 7 of 11 amygdalar neurons lie below the diagonal, the effect is less pronounced overall, particularly considering the amygdalar neuron near the top, with a −log10(p) during game play of 16.

Figure 5

p Value of test for a category effect for each neuron during picture identification (negative base 10 logarithm on horizontal axis) and during game play (negative base 10 logarithm on vertical axis). All 95 neurons, with and without a significant category-selective effect, are shown. Symbols denote significant effects of image category on the response (one-way ANOVA, p < .05): filled dots = no effect; open circles = significant effect during picture identification or game play (not both); cross = significant effect during both picture identification and game play, although magnitude of effect may vary between tasks. Numbers near symbols indicate specific neurons: 1 = amygdala neuron shown in Figures 2 and 3A; 2 = amygdala neuron shown in Figure 3B; 3 = hippocampus neuron shown in Figure 4A; 4 = hippocampus neuron shown in Figure 4B. Selective cells in the hippocampus have a greater tendency to lie below the diagonal to the right, showing greater selectivity during picture identification. Although 7 of 11 amygdalar neurons lie below the diagonal, the effect is less pronounced overall, particularly considering the amygdalar neuron near the top, with a −log10(p) during game play of 16.

for each neuron in these two brain areas.

These p values can be compared to a chosen significance level (p < .05) to test whether a particular cell has a category-selective response during either picture identification or game play. In the hippocampus, seven neurons had a category-selective response (p < .05) during picture identification, whereas only one had a selective response during game play, and none had a selective response during both tasks.

In the amygdala, 11 neurons had a category-selective response (p < .05) during picture identification and 10 had a selective response during game play. Of these 10, 8 neurons also had a selective response during picture identification, thus 8 neurons had a category-selective response during both picture identification and game play. For these eight neurons, a pair of categories with a significant difference in response during picture identification (Tukey's method of honest significant differences, p < .05) also had a significant difference during game play 75% (18/24) of the time. Amygdalar neurons remain selective in both tasks and the specific categorical differences are largely the same for both tasks.

Also shown in the top panel of Figure 5 are two amygdalar neurons with very different responses among categories, creating much smaller p values than observed for other cells. For both of these neurons, the categories involving faces (emotional faces, famous people) had a very strong response compared to other categories. Although observed occasionally (2 of 95 neurons), the dataset reported here is too small to determine whether this is a separate phenomenon from category selectivity.

Attention devoted to picture identification thus has significantly different effects on category selectivity in the hippocampus and the amygdala. Fisher's exact test rejects (p < .025) the null hypothesis that the percentage of cells with a category-selective response in both picture identification and game play is unaffected by brain region (thus 8 of 59 for amygdala and 0 of 36 for hippocampus). This difference appears graphically in Figure 5. Neurons in the hippocampus have a greater tendency to lie below the diagonal, both for neurons with a significant category-selective response and those which fail to exceed the threshold F-ratio value for this test.

Background Firing Rate vs. Task Performed

When the firing rates during game play and picture identification were compared for each neuron, 37 of 95 (39%) neurons showed a significant change (t test, p < .05) in background firing rate between these conditions. Eighteen of these (49%) showed an increase in baseline firing during picture identification and 19 (51%) showed a decrease.

Relative Responses to Categories vs. Task Performed

As shown above, changes in category selectivity between picture identification and game play may be due to either increased or decreased firing for particular image categories. Because background firing may also change between task types, an increased firing rate during game play may represent an increased or decreased effective response, depending on whether that firing rate was an increase or decrease relative to the background rate; similarly for a change which is a decreased firing rate during game play.

To better understand these response changes, the relative response for a category was defined as the ratio of the average firing rate during presentation of images from the category to the baseline firing rate for all images during the same task. As noted in Figure 3A, this ratio is defined for each image category and task type, producing a ratio, ρpid during picture identification trials, and ρgp during game play trials, for each neuron and image category. The difference of the logs of these ratios, Δ = log(ρpid) − log(ρgp) then provides a measure of the magnitude and direction of the change in relative response.

The distribution of these differences for each neuron with a selective response during either task is shown in Figure 6 
Figure 6

Distribution of the difference of log response ratios for each brain area. Differences were computed as described in Methods, and the differences for all categories for all neurons in each brain area were grouped together. Boxes cover the 25% to 75% percentile of the data, with the horizontal bar at the median. Dashed lines with horizontal cap cover data not more than 1.5 times the interquartile range from the corresponding box edge. Circles show data points outside these ranges. For both the amygdala and the hippocampus, a change in relative response for a category was just as likely to increase as to decrease when the subject shifted from picture identification to game play.

Figure 6

Distribution of the difference of log response ratios for each brain area. Differences were computed as described in Methods, and the differences for all categories for all neurons in each brain area were grouped together. Boxes cover the 25% to 75% percentile of the data, with the horizontal bar at the median. Dashed lines with horizontal cap cover data not more than 1.5 times the interquartile range from the corresponding box edge. Circles show data points outside these ranges. For both the amygdala and the hippocampus, a change in relative response for a category was just as likely to increase as to decrease when the subject shifted from picture identification to game play.

for both brain regions. The median difference for either region is not significantly different from 0. Thus, when a neuron has a selective response during picture identification but not during game play, on average, the responses to half the categories have increased and half have decreased. The implications of this in terms of eye movements and receptive fields are discussed later (see Discussion).

In addition to category-selective responses, a minority of neurons showed a change in firing relative to baseline for presentation of an image from any category—a generalized response. Twenty percent of MTL neurons (19 of 95) showed a significant response during picture identification, 18% (17 of 95) showed a response during game play, 15 of these neurons also having a response during picture identification. Of the 21 neurons having a significant generalized response during either picture identification or game play, 5 had a category-selective visual response in either condition. Having a category-selective visual response and having a general response to image presentation appeared to be independent effects (Fisher's exact test, p = .76). Having a category-selective visual response also appeared independent of a shift in baseline firing rate (Fisher's exact test, p = .14).

Responses in Other Brain Areas

As noted earlier, limited numbers of neurons and category-selective neurons were recorded from brain areas other than the hippocampus and the amygdala. A primary distinction of interest here is whether a brain area has a fraction of neurons retaining category selectivity between tasks which is different from the amygdala, where the largest number of neurons was recorded. For the hippocampus, this fraction was significantly different from that found in the amygdala (Fisher's exact test, p < .025).

Table 1 

Table 1

Number of Neurons Recorded and Category Selective for Each Brain Area and Task

Area
No. Selective in Picture Identification
No. Selective in Game Play
No. Selective in Both Tasks
Total No. Recorded
Amygdala 11 10 59 
Anterior cingulate 
Entorhinal cortex 38 
Hippocampus 36 
Orbito-frontal cortex 13 
Parahippocampal gyrus 22 
Supplementary motor area 13 
Area
No. Selective in Picture Identification
No. Selective in Game Play
No. Selective in Both Tasks
Total No. Recorded
Amygdala 11 10 59 
Anterior cingulate 
Entorhinal cortex 38 
Hippocampus 36 
Orbito-frontal cortex 13 
Parahippocampal gyrus 22 
Supplementary motor area 13 
shows the numbers of neurons recorded, and the numbers which were category-selective during each task and during both tasks. Applying Fisher's exact test to determine whether the fraction of neurons retaining selectivity in both tasks is different from that observed in the amygdala yields a significant result (p < .05) for only the entorhinal cortex among these other brain areas. This result is obtained when only 3 neurons of 38 (7.9%) have category selectivity during picture identification.

One interpretation of this result is that the entorhinal cortex is acting similarly to the hippocampus, with few neurons that retain their category selectivity during both tasks. In addition, considering the entorhinal cortex to provide inputs to the hippocampus (Burwell & Witter, 2002), category selectivity is already modulated by task at the inputs to the hippocampus. A difficulty with this interpretation is the small number of neurons (3) which are category selective during picture identification.

An alternative explanation would be that there are few category-selective neurons in the input and that the hippocampus generates category selectivity, which is appropriate to the task being performed. Although the fraction of entorhinal cortical neurons which are category selective during picture identification is low compared to previous report (Kreiman et al., 2000 reported 25 of 153, 16.3%, of entorhinal cortical neurons to be category selective), that fraction is not significantly different (Fisher's exact test, p < .05) from the data reported here. Thus, it also appears reasonable to interpret these results for the entorhinal cortex as having recorded fewer category-selective neurons by chance, and then having found none with category selectivity during both tasks, as was the case in the amygdala in a larger number of neurons. Reliably distinguishing between these interpretations, and understanding the broader relationship of these brain areas to one another, will require collection of the responses of a larger number of neurons in these brain areas as tasks are varied.

Regarding the brain areas which failed to show a difference from the amygdala in the fraction of neurons with category selectivity during both tasks, the power to detect these differences is limited due to the limited number of cells recorded. For example, a post hoc power analysis of Fisher's exact test (Erdfelder, Paul, & Buchner, 1996), testing for a change in the fraction of neurons which retain selectivity across tasks from the 0.14 observed in the amygdala to 0.05 or less, with a significance level of .05, shows that the power achieved for a comparison to the brain area with the next largest number of cells, the entorhinal cortex with 38 neurons, is only 3.7%. Thus, it is not possible to conclude that failure to observe a difference in this fraction between brain areas indicates the effect is not present, rather it may not have shown itself due to chance.

No Response to Planning Key Presses

In agreement with previous recordings of single-neuron activity in human epilepsy patients, we have assumed that pressing a key does not, by itself, create responses in the human hippocampus and the amygdala (Reddy, Quiroga, Wilken, Koch, & Fried, 2006; Rutishauser, Mamelak, & Schuman, 2006; Kreiman et al., 2000; Fried et al., 1997). As an additional check, because the subject is pressing keys more rapidly during game play than during picture identification, we compared firing during 500 msec before and after each keypress for 12 selective neurons (randomly chosen from the 21 selective neurons). Only one neuron showed a significant change in firing rate before and after key depression greater than 0.05 sp/sec, and that neuron showed a change of 0.1 sp/sec during the response to the category of emotional expressions, which evoked responses greater than 3 sp/sec. This control shows that responses are not related simply to planning the keypress (which should result in a response prior to executing the keypress), although it does not exclude more complex forms of motor responses.

Magnitude of Eye Movements during the Two Tasks

Another difference between game play and picture identification is that the subjects may move their eyes more during game play, as there are targets in the periphery. Measurement of eye movements in normal subjects performing the same tasks (ISCAN II, Burlington, MA; frame rate = 60 Hz) shows that the standard deviation of eye position (in degrees from the line perpendicular to the display screen) is 5° during game play (thus from the center of the image to its edge) and 1° during picture identification. Possible implications of this difference are described in the Discussion.

DISCUSSION

A key finding in the present study is the different effect of changing tasks on single neurons in the human hippocampus and the amygdala. For neurons in the hippocampus, a sizable fraction (19%) has a category-selective visual response when the subject performs picture identification but this is virtually eliminated (15% of those remaining) when the subject shifts to game play. By contrast, for the majority (62%) of amygdala neurons with a category-selective response, the selectivity is present during both picture identification and game play.

This finding argues for the hippocampus processing only those visual stimuli to which the subject is attending. In terms of cognitive models, such as the hybrid model of Vogel, Luck, and Shapiro (1998), the hippocampus would reside in a stage beyond attentive filtering, and only those stimuli which the subject was attending would be available for forming durable and reportable memories (Squire, 2004; Cohen et al., 1999; Squire & Zola, 1996). As shown in separate experiments in normal subjects, subjects are largely unaware of unattended images in a pair of superimposed video streams (Jenkins, Lavie, & Driver, 2005; Wright, Katz, & Hughes, 1993; Rock & Gutman, 1981; Neisser & Becklen, 1975).

The preservation of responses in the amygdala, despite the demands of the task being performed, argues in favor of the amygdala providing preattentive evaluation of the images. This would be consistent with the amygdala being involved in a subcortical network for face recognition (Pessoa et al., 2006; Whalen et al., 2004; Compton, 2003), and with the amygdala providing an emotional assessment of stimuli (Adolphs et al., 2005; Zald, 2003; Oya et al., 2002). More generally, these neurons in the amygdala could serve to draw the attentional focus to important items in the environment.

Insofar as these recordings were performed in human epilepsy patients, there is always the question whether the presence of the disease might account for observed findings. In both these and prior studies (Kreiman et al., 2000), we have not observed a difference in visual response properties between subsequently resected brains areas and areas within the other hemisphere. Although this argues that these responses are representative of the operation of the normal human brain, these are recordings in epilepsy patients whose brains have a significant neurological illness and have been treated over decades with antiepileptic medications, which must qualify the conclusions drawn here. Nonetheless, this is presently the only method of recording single-neuron activity in human subjects which is employed in a sufficient number of cases to generate a reasonable dataset. Response differences between separate parts of the hippocampus (anterior vs. posterior) and the nuclei of the amygdala would be of particular interest, but the number of cells recorded in this study does not permit separation of these groups.

Recent functional magnetic resonance imaging studies suggest that processing in the amygdala is modulated as subjects shift their attention from emotional faces to an orientation discrimination task (Pessoa, Kastner, & Ungerleider, 2002). The exact relationship between the blood oxygenation level-dependent (BOLD) signal and single-unit activity is still poorly understood (Nair, 2005; Logothetis, 2002), although BOLD likely reflects the average inputs or firing activity in a brain area. Considered together with these single-neuron recordings, one possibility is that neurons in the amygdala provide evaluation and firing output which is independent of the visual items attended, even when their inputs vary, creating changes in the BOLD signal.

Because images were always presented in the background during game play prior to being presented as the objects of attention in picture identification, one must consider whether this ordering, given reported dependence of responses in the human MTL on the sequence of presentation (Rutishauser et al., 2006; Fried et al., 1997), could explain the results reported here. Rutishauser et al. (2006) described neurons with increased firing in response to either new or old stimuli, in roughly equal proportions. Such firing would not predict a differential response to categories because each image in each category was presented once as a new image. Rutishauser et al. also did not report any differences between neurons in the amygdala and the hippocampus, a central finding here.

Changing between picture identification and game play involves an overt shift of attention: Both the position of the eyes and the visual objects fixated change. This raises a specific explanation of the changes in hippocampal visual responses, namely, that they could be caused by changes of object position in retinotopic space, and consequently, by changes in position within the hippocampal receptive fields. Under this interpretation, greater eye movements during game play decrease category-selective responses because the images fall outside a receptive field with a central on response (Nowicka & Ringo, 2000; Sobotka, Nowicka, & Ringo, 1997; Sobotka & Ringo, 1996). The observed response changes between tasks would then be a result of an overt shift of attention. The response ratios shown in Figure 6 are not consistent with this interpretation, however, because roughly equal numbers of responses to individual categories increase and decrease when the video game is played. Additionally, the eye movements during game play (standard deviation of 5°) are smaller than the reported widths of primate hippocampal receptive fields (on the order of 20°) (Nowicka & Ringo, 2000).

The attentional shift between game play and picture identification is separate from spatially selective visual attention, such as demonstrated by covert shifts of attention, where the subject maintains eye fixation. It is a general shift between tasks, where each task may create different levels of arousal or vigilance. There is no benefit to the subject to direct any attention to the nonpertinent task, indeed, this would hinder performance in the task for which they are currently performing. We thus assume that a majority of attention is directed to the current task.

The recent report of failure of human MTL neurons to respond to images when changes in the images are not detected (Reddy et al., 2006) lends strong support to the view that the response changes described here are due to shifts of attention, if one assumes that a failure to detect a change occurs because it was not attended. That article did not describe differences between the hippocampus and the amygdala, the central finding here. This may be due to differences between change detection and the shifts of selective attention employed here, or due to the smaller number of neurons reported in that article.

The changes in responses in the hippocampus and the amygdala described here, considered both on a single-neuron and population basis, directly demonstrate changes of neuronal activity due to a rapidly and flexibly acquired task at the single-cell level in the human hippocampus, and a contrast between how single neurons in the human hippocampus and the amygdala respond to changes in visual objects which are attended.

Acknowledgments

I thank the patients for participating in this study and Tony Fields, Itzhak Fried, Eve Isham, and Leila Reddy for technical assistance. I also thank William Banks, Christof Koch, Gabriel Kreiman, Ernst Niebur, Patrick Wilken, and Charles Wilson for useful and informative discussions. This study was supported by the McDonnell-Pew Program in Cognitive Neuroscience (JSMF #20002058).

Reprint requests should be sent to Peter N. Steinmetz, Harrington Department of Bioengineering, Arizona State University, MC 9709, Tempe, AZ 85287, or via e-mail: Peter.Steinmetz@asu.edu.

REFERENCES

Adolphs
,
R.
,
Tranel
,
D.
, &
Buchanan
,
T. W.
(
2005
).
Amygdala damage impairs emotional memory for gist but not details of complex stimuli.
Nature Neuroscience
,
8
,
512
518
.
Andersen
,
R. A.
, &
Mountcastle
,
V. B.
(
1983
).
The influence of the angle of gaze upon the excitability of the light-sensitive neurons of the posterior parietal cortex.
Journal of Neuroscience
,
3
,
532
548
.
Babb
,
T. L.
,
Carr
,
E.
, &
Crandall
,
P. H.
(
1973
).
Analysis of extracellular firing patterns of deep temporal lobe structures in man.
Electroencephalography and Clinical Neurophysiology
,
34
,
247
257
.
Broadbent
,
D. E.
(
1958
).
Perception and communication
.
Oxford
:
Pergamon Press
.
Burwell
,
R. D.
, &
Witter
,
M. P.
(
2002
).
Basic anatomy of the parahippocampal region in monkeys and rats.
In M. Witter & F. Wouterlood (Eds.),
The parahippocampal region: Organization and role in cognitive function
(pp.
35
59
).
Oxford
:
Oxford University Press
.
Cohen
,
N. J.
,
Ryan
,
J.
,
Hunt
,
C.
,
Romine
,
L.
,
Wszalek
,
T.
, &
Nash
,
C.
(
1999
).
Hippocampal system and declarative (relational) memory: Summarizing the data from functional neuroimaging studies.
Hippocampus
,
9
,
83
98
.
Compton
,
R. J.
(
2003
).
The interface between emotion and attention: A review of evidence from psychology and neuroscience.
Behavioral and Cognitive Neuroscience Reviews
,
2
,
115
129
.
Davis
,
K. D.
,
Hutchison
,
W. D.
,
Lozano
,
A. M.
,
Tasker
,
R. R.
, &
Dostrovsky
,
J. O.
(
2000
).
Human anterior cingulate cortex neurons modulated by attention-demanding tasks.
Journal of Neurophysiology
,
83
,
3575
3577
.
Deutsch
,
J. A.
, &
Deutsch
,
D.
(
1963
).
Attention: Some theoretical considerations.
Psychological Review
,
70
,
80
90
.
Erdfelder
,
E.
,
Faul
,
F.
, &
Buchner
,
A.
(
1996
).
Gpower: A general power analysis program.
Behavioral Research Methods, Instruments and Computers
,
28
,
1
11
.
Felleman
,
D. J.
, &
Van Essen
,
D. C.
(
1991
).
Distributed hierarchical processing in the primate cerebral cortex.
Cerebral Cortex
,
1
,
1
47
.
Fried
,
I.
,
MacDonald
,
K. A.
, &
Wilson
,
C. L.
(
1997
).
Single neuron activity in human hippocampus and amygdala during recognition of faces and objects.
Neuron
,
18
,
753
765
.
Haxby
,
J. V.
,
Courtney
,
S. M.
, &
Clark
,
V. P.
(
1998
).
Functional magnetic resonance imaging and the study of attention.
In R. Parasuraman (Ed.),
The attentive brain
(pp.
123
142
).
Cambridge, MA
:
MIT Press
.
Jenkins
,
R.
,
Lavie
,
N.
, &
Driver
,
J.
(
2005
).
Recognition memory for distractor faces depends on attentional load at exposure.
Psychonomic Bulletin & Review
,
12
,
314
320
.
Kastner
,
S.
, &
Pinsk
,
M. A.
(
2004
).
Visual attention as a multilevel selection process.
Cognitive, Affective & Behavioral Neuroscience
,
4
,
483
500
.
Kleinbaum
,
D. G.
(
1988
).
Applied regression analysis and other multivariable methods.
Pacific Grove, CA
:
Duxbury Press
.
Kreiman
,
G.
,
Koch
,
C.
, &
Fried
,
I.
(
2000
).
Category-specific visual responses of single neurons in the human medial temporal lobe.
Nature Neuroscience
,
3
,
946
953
.
Lavie
,
N.
(
2005
).
Distracted and confused?: Selective attention under load.
Trends in Cognitive Sciences
,
9
,
75
82
.
Logothetis
,
N. K.
(
2002
).
The neural basis of the blood-oxygen-level-dependent functional magnetic resonance imaging signal.
Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences
,
357
,
1003
1037
.
Luck
,
S. J.
,
Chelazzi
,
L.
,
Hillyard
,
S. A.
, &
Desimone
,
R.
(
1997
).
Neural mechanisms of spatial selective attention in areas V1, V2, and V4 of macaque visual cortex.
Journal of Neurophysiology
,
77
,
24
42
.
Luck
,
S. J.
, &
Girelli
,
M.
(
1998
).
Electrophysiological approaches to the study of selective attention in the human brain.
In R. Parasuraman (Ed.),
The attentive brain
(pp.
71
94
).
Cambridge, MA
:
MIT Press
.
Maunsell
,
J. H.
, &
Treue
,
S.
(
2006
).
Feature-based attention in visual cortex.
Trends in Neurosciences
,
29
,
317
322
.
Moran
,
J.
, &
Desimone
,
R.
(
1985
).
Selective attention gates visual processing in the extrastriate cortex.
Science
,
229
,
782
784
.
Motter
,
B. C.
(
1998
).
Neurophysiology of visual attention.
In R. Parasuraman (Ed.),
The attentive brain
(pp.
51
69
).
Cambridge, MA
:
MIT Press
.
Nair
,
D. G.
(
2005
).
About being BOLD.
Brain Research, Brain Research Reviews
,
50
,
229
243
.
Neisser
,
U.
, &
Becklen
,
R.
(
1975
).
Selective looking: Attending to visually specified events.
Cognitive Psychology
,
7
,
480
494
.
Norman
,
D. A.
(
1968
).
Toward a theory of memory and attention.
Psychological Review
,
75
,
522
536
.
Nowicka
,
A.
, &
Ringo
,
J. L.
(
2000
).
Eye position-sensitive units in hippocampal formation and in inferotemporal cortex of the macaque monkey.
European Journal of Neuroscience
,
12
,
751
759
.
Oya
,
H.
,
Kawasaki
,
H.
,
Howard
,
M. A. R.
, &
Adolphs
,
R.
(
2002
).
Electrophysiological responses in the human amygdala discriminate emotion categories of complex visual stimuli.
Journal of Neuroscience
,
22
,
9502
9512
.
Pashler
,
H. E.
(
1998
).
The psychology of attention.
Cambridge, MA
:
MIT Press
.
Pessoa
,
L.
,
Japee
,
S.
,
Sturman
,
D.
, &
Ungerleider
,
L. G.
(
2005
).
Target visibility and visual awareness modulate amygdala responses to fearful faces.
Cerebral Cortex
,
16
,
366
375
.
Pessoa
,
L.
,
Kastner
,
S.
, &
Ungerleider
,
L. G.
(
2002
).
Attentional control of the processing of neural and emotional stimuli.
Brain Research, Cognitive Brain Research
,
15
,
31
45
.
Phelps
,
E. A.
(
2004
).
Human emotion and memory: Interactions of the amygdala and hippocampal complex.
Current Opinion in Neurobiology
,
14
,
198
202
.
Reddy
,
L.
,
Quiroga
,
R. Q.
,
Wilken
,
P.
,
Koch
,
C.
, &
Fried
,
I.
(
2006
).
A single-neuron correlate of change detection and change blindness in the human medial temporal lobe.
Current Biology
,
16
,
2066
2072
.
Richter-Levin
,
G.
(
2004
).
The amygdala, the hippocampus, and emotional modulation of memory.
Neuroscientist
,
10
,
31
39
.
Rock
,
I.
, &
Gutman
,
D.
(
1981
).
The effect of inattention on form perception.
Journal of Experimental Psychology: Human Perception and Performance
,
7
,
275
285
.
Rutishauser
,
U.
,
Mamelak
,
A. N.
, &
Schuman
,
E. M.
(
2006
).
Single-trial learning of novel stimuli by individual neurons of the human hippocampus–amygdala complex.
Neuron
,
49
,
805
813
.
Shipp
,
S.
(
2004
).
The brain circuitry of attention.
Trends in Cognitive Sciences
,
8
,
223
230
.
Sobotka
,
S.
,
Nowicka
,
A.
, &
Ringo
,
J. L.
(
1997
).
Activity linked to externally cued saccades in single units recorded from hippocampal, parahippocampal, and inferotemporal areas of macaques.
Journal of Neurophysiology
,
78
,
2156
2163
.
Sobotka
,
S.
, &
Ringo
,
J. L.
(
1996
).
Mnemonic responses of single units recorded from monkey inferotemporal cortex, accessed via transcommissural versus direct pathways: A dissociation between unit activity and behavior.
Journal of Neuroscience
,
16
,
4222
4230
.
Squire
,
L. R.
(
2004
).
Memory systems of the brain: A brief history and current perspective.
Neurobiology of Learning and Memory
,
82
,
171
177
.
Squire
,
L. R.
, &
Zola
,
S. M.
(
1996
).
Structure and function of declarative and nondeclarative memory systems.
Proceedings of the National Academy of Sciences, U.S.A.
,
93
,
13515
13522
.
Treisman
,
A. M.
(
1969
).
Strategies and models of selective attention.
Psychological Review
,
76
,
282
299
.
Viskontas
,
I. V.
,
Knowlton
,
B. J.
,
Steinmetz
,
P. N.
, &
Fried
,
I.
(
2007
).
Differences in mnemonic processing by neurons in the human hippocampus and parahippocampal region.
Journal of Cognitive Neuroscience
,
18
,
1654
1662
.
Vogel
,
E. K.
,
Luck
,
S. J.
, &
Shapiro
,
K. L.
(
1998
).
Electrophysiological evidence for a postperceptual locus of suppression during the attentional blink.
Journal of Experimental Psychology: Human Perception and Performance
,
24
,
1656
1674
.
Whalen
,
P. J.
,
Kagan
,
J.
,
Cook
,
R. G.
,
Davis
,
F. C.
,
Kim
,
H.
,
Polis
,
S.
,
et al
(
2004
).
Human amygdala responsivity to masked fearful eye whites.
Science
,
306
,
2061
.
Wright
,
R. D.
,
Katz
,
A. N.
, &
Hughes
,
E. A.
(
1993
).
Inattention and the perception of visual features.
Acta Psychologica (Amsterdam)
,
83
,
225
235
.
Yantis
,
S.
, &
Johnston
,
J. C.
(
1990
).
On the locus of visual selection: Evidence from focused attention tasks.
Journal of Experimental Psychology: Human Perception and Performance
,
16
,
135
149
.
Zald
,
D. H.
(
2003
).
The human amygdala and the emotional evaluation of sensory stimuli.
Brain Research, Brain Research Reviews
,
41
,
88
123
.