The lateral intraparietal area (LIP) is thought to play an important role in the guidance of where to look and pay attention. LIP can also respond selectively to differently shaped objects. We sought to understand to what extent short-term and long-term experience with visual orienting determines the responses of LIP to objects of different shapes. We taught monkeys to arbitrarily associate centrally presented objects of various shapes with orienting either toward or away from a preferred spatial location of a neuron. The training could last for less than a single day or for several months. We found that neural responses to objects are affected by such experience, but that the length of the learning period determines how this neural plasticity manifests. Short-term learning affects neural responses to objects, but these effects are only seen relatively late after visual onset; at this time, the responses to newly learned objects resemble those of familiar objects that share their meaning or arbitrary association. Long-term learning affects the earliest bottom–up responses to visual objects. These responses tend to be greater for objects that have been associated with looking toward, rather than away from, LIP neurons' preferred spatial locations. Responses to objects can nonetheless be distinct, although they have been similarly acted on in the past and will lead to the same orienting behavior in the future. Our results therefore indicate that a complete experience-driven override of LIP object responses may be difficult or impossible. We relate these results to behavioral work on visual attention.
The ability to use visual information to predict where important things will be in the near future has obvious value for an organism. Visual changes over space and time (such as those that usually accompany the appearance of a new object in the visual field) are likely to be important and accordingly can automatically capture attention (Franconeri, Hollingworth, & Simons, 2005; Abrams & Christ, 2003; Nakayama & Mackeben, 1989; Jonides & Yantis, 1988; Jonides, 1981: Posner, 1980, but see Müller, Geyer, Zehetleitner, & Krummenacher, 2009; Eimer & Kiss, 2008; Kiss, Jolicœur, Dell'Acqua, & Eimer, 2008; Folk & Remington, 1998; Folk, Remington, & Johnston, 1992). Visual attention can also be deliberately directed and maintained, and these two ways of visual orienting have been reported to follow different time courses, the former of which has a transient effect on performance with a rapid rise and fall whereas the latter takes more time to have its effects (Cheal & Lyon, 1991; Nakayama & Mackeben, 1989).
Attention has mainly been thought to be automatically captured by visual information in the periphery so that attention is oriented to the location of the visual objects or events that also initiate the attentional shift (Tipples, 2002; Cheal & Lyon, 1991; Jonides, 1981). Sometimes objects can nonetheless give important clues about where other things or events will be in the near future. Take a street sign with a leftward-pointing arrow and the words “look left” that prompts people to check for approaching cars in a particular direction. This, at least at a first glance, seems to be an indirect and symbolic way of representing space that would require the slow and deliberate visual orienting of sustained attention. However, it is now increasingly recognized that visual objects can both swiftly and automatically guide orienting away from themselves (Sigurdardottir, Michalak, & Sheinberg, 2014; Kuhn & Kingstone, 2009; Tipples, 2002, 2008; Ristic & Kingstone, 2006; Fischer, Castel, Dodd, & Pratt, 2003; Hommel, Pratt, Colzato, & Godijn, 2001; Driver et al., 1999; Friesen & Kingstone, 1998). For example, even centrally presented novel objects can guide people's eyes and attention in a particular direction because of the way that they are shaped, despite the fact that doing so is task-irrelevant or even detrimental to task performance (Sigurdardottir et al., 2014). These orienting effects are hard or impossible to fully overcome, and their time course resembles that of transient visual attention (Sigurdardottir et al., 2014).
Although objects that people have never seen before can guide attention, the orienting effects of some familiar objects such as arrows do appear to be particularly robust (Sigurdardottir et al., 2014; Kuhn & Kingstone, 2009; Tipples, 2002, 2008; Hommel et al., 2001). This might, at least in part, be due to the fact that arrows generally point to something important; a distant target has repeatedly been associated with this shape over the course of an entire lifetime. The deliberate training of an arbitrary association between a central visual stimulus and a target found in a particular peripheral location also leads to a seemingly obligatory attentional shift toward that location (Van der Stigchel, Mills, & Dodd, 2010; Dodd & Wilson, 2009). The attentional effects of such arbitrary associations appear to be quite weak and slow after a short training period but might increase in magnitude and speed with a longer training session (Van der Stigchel et al., 2010; Dodd & Wilson, 2009).
Visual attention and cueing effects have been less extensively studied in monkeys than in humans, but the general effects and the time course of spatial precueing appear to be comparable in the different species (Lee & McPeek, 2013). Humans and monkeys are also believed to have several homologous visually responsive brain regions, including a parietal region known as the lateral intraparietal area (LIP) in the monkey (Silver & Kastner, 2009; Konen & Kastner, 2008a, 2008b). The function of LIP is still a subject of debate, but the region has mainly been implicated in the visual guidance of spatial attention and saccadic eye movements (Balan & Gottlieb, 2006; Bisley & Goldberg, 2003, 2010; Andersen & Buneo, 2002; Colby & Goldberg, 1999; Gottlieb, Kusunoki, & Goldberg, 1998; Snyder, Batista, & Andersen, 1997), collectively known as visual orienting. LIP is structurally connected to multiple visual areas (Lewis & Van Essen, 2000; Felleman & Van Essen, 1991) and to several oculomotor structures (Prevosto, Graf, & Ugolini, 2010; Field, Johnston, Gati, Menon, & Everling, 2008; Ferraina, Pare, & Wurtz, 2002; Lewis & Van Essen, 2000; Stanton, Bruce, & Goldberg, 1995), making it perfectly situated to gather and combine various sources of visual information with the objective of guiding visual orienting.
LIP has been found to respond selectively to differently shaped visual objects (Janssen, Srivastava, Ombelet, & Orban, 2008; Konen & Kastner, 2008a, 2008b; Durand et al., 2007; Lehky & Sereno, 2007; Sereno & Amador, 2006; Sereno, Trinath, Augath, & Logothetis, 2002; Sereno & Maunsell, 1998). This is akin to many regions within the ventral visual stream (Palmeri & Gauthier, 2004; Logothetis & Sheinberg, 1996; Milner & Goodale, 1995; Goodale & Milner, 1992; Ungerleider & Mishkin, 1982), although the responses of LIP to visual objects are far less studied and understood. The fact that LIP is selective for the shape of objects and plays a role in visual orienting makes this parietal region a primary candidate for carrying out the necessary computations for extracting an orienting bias from an object, such as a leftward-pointing arrow or the words “look left,” that might have come about through the association of such objects with an important target in a distant location. In this context, it should be noted that particular effects of the shape of visual objects on spatial attention have been modeled based on LIP neural dynamics, and the shape responses of LIP neurons might better account for such shape-induced attentional effects than the shape responses of the ventral visual stream (Red, Patel, & Sereno, 2012; Patel, Peng, & Sereno, 2010).
The role of the parietal cortex in arbitrary associations is nonetheless somewhat controversial (for reviews on arbitrary visuomotor associations, see, e.g., Seger, 2009; Graybiel, 2005; Brasted & Wise, 2004; Hadj-Bouziane, Meunier, & Boussaoud, 2003; Murray, Bussey, & Wise, 2000; Passingham, Toni, & Rushworth, 2000; Wise & Murray, 2000). Functional neuroimaging studies have reported that neural activity in the parietal cortex can be affected by associative learning that happens over the course of a single session (Eliassen, Souza, & Sanes, 2003; Deiber et al., 1997) and that the parietal cortex can become progressively more involved as the arbitrary associations become increasingly automatic and overtrained (Grol, de Lange, Verstraten, Passingham, & Toni, 2006; Eliassen et al., 2003). Removing parts of the parietal cortex, however, does not seem to affect the learning of new associations or the retention of familiar ones (Pisella et al., 2000; Rushworth, Nixon, & Passingham, 1997). As an example, LIP neurons can become sensitive to colors if they have been arbitrarily associated with certain behaviors (Toth & Assad, 2002), but monkeys nonetheless do not have particular problems with relearning a similar task after parietal lesions, including a lesion of LIP (Rushworth et al., 1997).
We sought to understand to which extent and how LIP responses change through learning of arbitrary associations or, more specifically, how experience with visual orienting—which is most likely the primary task supported by LIP—can determine the nature of responses of LIP neurons to visual objects. We set out to answer three main questions: (1) How are the neural responses to visually presented objects affected by short-term learning of arbitrary associations between objects and orienting? (2) How are these responses affected by long-term experience with such arbitrary associations? (3) Can experience with arbitrary associations ever completely override the responses of LIP neurons to visual objects, or are these responses resistant to such experience?
Surgery, MRI, and Recordings
Two male macaca mulatta monkeys (monkey J: 10.5 kg, monkey R: 9.5 kg) were implanted with titanium head posts for restraining head movements during the training and recording sessions. The monkeys had two separate surgeries under isoflurane anesthesia. During the first surgery, we implanted a recording chamber of diameter 16 mm at approximately 5P and 12L over the right hemisphere. The chamber was made of MRI-compatible plastic material (PEEK, polyetheretherketone). The craniotomy was performed during a second surgery after structural MRI had verified that the chambers were located above the LIP.
The structural MRI was also used to properly position a metal guide tube so that an electrode would reach LIP. During each recording session, a tungsten microelectrode (1.5 MΩ, Alpha Omega, Alpharetta, GA) was lowered using a hydraulic microdrive (David Kopf Instruments, Tujunga, CA), and the neural signals were filtered and amplified (BAK Electronics, Sanford, FL). For these experiments, we did not use the “memory-guided saccade” task to specifically localize cells in area LIP. Although this task has been used for unit selection in LIP studies, it does not, alone, adequately distinguish neurons from within and outside LIP. For example, it has been reported that neurons from both LIP and the neighboring region 7a exhibit light-sensitive, memory (delay period), and saccade responses in a memory-guided saccade task (Andersen, Bracewell, Barash, Gnadt, & Fogassi, 1990). Also, many LIP neurons do not have significant delay period activity during a memory-guided saccade task (Ben Hamed, Duhamel, Bremmer, & Graf, 2001). As we were interested in characterizing all neurons in this area and not just those showing delay activity, we elected to map the receptive field properties as described below and to rely on the anatomical localization provided by the chamber aligned structural MRI. Time within a single session was also a key consideration, because before training the monkeys on new shape–saccade associations each day we mapped the neurons' spatial receptive fields and shape selectivity to choose familiar shapes that were appropriate for the location responses of the recorded neurons (see sections “Methods: Tasks and Stimuli: Tasks: Location Selectivity Mapping Task” and “Methods: Tasks and Stimuli: Tasks: Active Shape–Saccade Association Task”).
Regarding spatial receptive field properties found in area LIP, we note that Ben Hamed et al. (2001) mapped the location selectivity of LIP neurons with well-defined visual responses (regardless of the presence of delay period or eye movement-related activity) and reported that 11.7% of their 171 LIP neurons had the center of mass of their radio frequency in the ipsilateral hemifield. This proportion is not significantly different from that in our study (Fisher's exact test, p = .175; see section “Results: Location Selectivity Mapping Task”). The differences in the proportion of units with an ipsilateral preference might therefore be due to random differences in sampling. So although it is possible that some of the neurons in our study may have been located in adjoining cortical areas, our data are nonetheless compatible with these previous studies.
Eye movements were monitored and recorded using EyeLinkII (SR Research, Mississauga, Canada) with a 500-Hz sampling rate and streamed to a disk at 200 Hz. Experimental protocols were in accordance with animal care guidelines of the National Institutes of Health (National Research Council, 2011) and Brown University's Institutional Animal Care and Use Committee.
Tasks and Stimuli
Stimulus Generation and Presentation
All shapes, familiar and novel, were originally generated by randomly selecting and overlapping four black strokes or pieces out of a set of 64 and then scaling them so that all composite shapes had the same area. Example shapes can be seen in Figure 1. All stimuli were shown on a 1024 × 768 resolution screen with a refresh rate of 100 Hz, and the experiments were controlled using in-house software running on Windows XP (Microsoft, Redmond, WA) and QNX RTOS (QNX Software Systems, Ottawa, Canada).
The monkeys were trained on three tasks, run consecutively in each session (Figure 2). The first two tasks were mainly used for stimulus and unit selection. We will briefly report some results from these tasks, but our focus here will be on the main task, an active shape–saccade association task (see below).
Location selectivity mapping task
In the location selectivity mapping task, a yellow fixation square (side length 0.3°) appeared in the center of a light gray background. After the monkey acquired fixation, a dark gray target disk (radius 1°) was flashed for 110 msec at 7° eccentricity. In each trial, the disks were randomly chosen to appear in one of eight possible radial directions from the center (22.5°, 67.5°, 112.5°, 157.5°, 202.5°, 247.5°, 292.5°, or 337.5°; 0° corresponds to a location at the right of fixation; numbers increase in counterclockwise direction). After a brief delay of 60 msec, the fixation square jumped to that same peripheral location, and the monkey was given juice for making a saccade to it. We recorded from LIP neurons, which, based on online spike data, were shown to respond to one or more locations. The preferred (PREF) location was defined as the location that evoked the highest mean responses over a 250-msec window (from 40 to 290 msec after visual onset of the peripheral disk). The antipreferred (ANTI) location was a location of the same eccentricity but in the opposite radial direction, in other words 180° away from the PREF.
Passive shape-mapping task
Each trial of the passive shape-mapping task consisted of four rapid serial visual presentations where black shapes (diameter approximately 3°; for examples, see Figure 1) were flashed on a light gray background for 150 msec, with an ISI of 130 msec, in three possible locations: PREF (7° eccentricity), CENTER, and ANTI (7° eccentricity), as determined by online spike recordings from the location selectivity mapping task. The monkeys' only requirement was to maintain their gaze within 4° of the center of the screen, so the trials aborted if the monkeys looked to the PREF or ANTI locations. A yellow fixation square (side length 0.3°) was shown in the center of the screen. At 60 msec after the disappearance of the last shape, the fixation square jumped to one of four possible locations randomly picked to be up, down, right, or left of the center (6° eccentricity). These target locations never overlapped with the PREF or ANTI locations. The monkeys were rewarded for making a saccade to the new location of the square.
A shape and a location were chosen pseudorandomly for each presentation, so that all shapes were shown equally often in all three locations. A total of eight different shapes were shown during a recording session. In most recording sessions, each shape appeared 20 times in each location. Four of the shapes were highly familiar to each monkey because it had been trained over the course of months to associate them with particular locations during the active shape–saccade association task, as explained in more detail below. In each session, we also showed four novel shapes that the monkey had never seen before and therefore had no particular associations.
Active shape–saccade association task
The final and main task was an active shape–saccade association task where the eight shapes previously seen in the passive shape-mapping task now served as central precues, cueing the monkey to saccade either to the PREF location (PREF shapes) or the ANTI location (ANTI shapes) after a brief delay. Two of the novel shapes were randomly chosen to cue the PREF location and the other two cued the ANTI location. The same was true for the four familiar shapes, except that their associations were randomly chosen at the start of the monkey's training and this initial assignment to a location was maintained for the entire training and recording period. Each monkey trained on four sets of familiar shapes with four shapes in each, so they were highly familiar with 16 shapes, which were different for the two monkeys. In each recording session, a set of familiar shapes was chosen to match the PREF and ANTI locations of the neuron being recorded from.
The active shape–saccade association task was run in blocks of 96 trials each. The first block had equal numbers of novel and familiar shapes (two novel PREF shapes, two novel ANTI shapes, two familiar PREF shapes, and two familiar ANTI shapes, 12 trials per shape). Provided that the monkeys were willing to complete further trials, this first mixed block was in most cases followed by three blocks of trials where only the novel shapes were shown (two novel PREF shapes and two novel ANTI shapes, 24 trials per shape) to provide more experience with novel shapes. These novel blocks were followed by mixed blocks (two novel PREF shapes, two novel ANTI shapes, two familiar PREF shapes, and two familiar ANTI shapes, 12 trials per shape), again provided that the monkeys were willing to complete these blocks.
A trial started with the appearance of a central fixation square (side length 0.3°). After the monkey acquired fixation, one of the shapes from the previous passive shape-mapping task was randomly chosen to appear in the center of the screen and was visible for the remainder of the trial. The monkey was required to keep fixating within 1° of the screen center for 500 msec (shape period), after which the fixation square disappeared and two identical gray choice disks (radius 1°) appeared, one in the PREF location and the other in the ANTI location (choice period). The monkey was then free to saccade to one of the two choice disks. The shape served as a 100% valid central precue so it determined what was considered the correct choice. The monkey received visual and auditory feedback for his choice; a correct choice was followed by a low-pitched tone and the chosen disk was substituted by a black diamond, whereas an incorrect choice was followed by a high-pitched tone and the disappearance of the chosen disk. A fluid reward was given for choosing the correct location.
Cell Recording and Selection
Recorded action potentials were sorted offline using the WaveClus spike clustering algorithm (Quiroga, Nadasdy, & Ben-Shaul, 2004). From this, we identified a total of 117 units, 82 of which were suitable for further analysis based on the following criteria (monkey J: n = 44, monkey R: n = 38). We included cells for which (a) the assumed PREF and ANTI locations, as determined by online spikes from the location selectivity mapping task, corresponded to the actual PREF and ANTI locations, as determined by offline analysis; (b) firing rate for centrally presented shapes remained high enough for the maximum depth of selectivity index to be calculated (see Methods; Data Analysis); and (c) the monkeys completed all tasks, that is, the location selectivity mapping task, the passive shape-mapping task, and more than one block of the active shape–saccade association task.
Unless otherwise noted, all statistical tests were two-sided and had an alpha level of .05. For repeated-measures ANOVAs, results were Greenhouse–Geisser corrected if Mauchly's test of sphericity was significant.
When analyzing data from the passive shape-mapping and active shape–saccade association tasks, we aligned neural responses in every trial to the visual onset of each shape and counted the number of action potentials within a 50-msec window centered on the time of onset. We repeated this process for windows spaced 10 msec apart. Unless otherwise stated, any reference to timing in the following text indicates the center time of such a window.
Passive Shape-mapping Task: Selective Responses to Centrally Presented Shapes
Active Shape–Saccade Association Task: Congruency of Responses
To look for changes in neural responses related to shorter-term learning within a single day, we examined the responses to novel shapes in two blocks of the active shape–saccade association task: the first block in a session and the block with the best behavioral performance for novel shape cues in the same session, provided that the block included at least 30 novel shape trials and that the monkey showed any behavioral improvements after the first block (69 of 73 sessions, 77 of 82 units). Note that we refer to shape cues as novel as long as they have not been seen in previous sessions, and we use the terms “early” novel shape trials and “late” novel shape trials to refer to the “first” and “best” blocks, respectively. To examine long-term learning effects, we used the first block of trials to compare neural responses to familiar shapes that in the past had been associated multiple times with either orienting to the PREF or ANTI location of the recorded neuron.
We compared the distributions of each neuron's firing rates for shapes that cued the PREF location and shapes that cued the ANTI location. This was done separately for familiar shape trials, early novel shape trials, and late novel shape trials. In this analysis, trials were labeled as PREF shape trials or ANTI shape trials regardless of which particular PREF or ANTI shape was shown, that is, any possible response differences between same-meaning shapes were ignored.
Here, AUCPREF shapes over ANTI shapes is the area under the receiver operating characteristic curve that compares the distributions of responses to shapes that cued the PREF and ANTI locations, where higher AUC scores indicate greater responses to PREF shapes than ANTI shapes. A positive congruency score implies that the neural responses to central shapes that cued the PREF location were in general higher than to shapes that cued the ANTI location, that is, the responses were congruent with the spatial preference of a neuron. In other words, a neuron with a contralateral spatial preference (as assessed by the location selectivity mapping task) would in general respond more to central shapes that cued the contralateral spatial location and less to shapes that cued the ipsilateral location, and a neuron with an ipsilateral spatial preference would respond more to shapes that cued the ipsilateral location and less to those that cued the contralateral location. The reverse is true for a negative congruency score; it represents incongruent activation, so a neuron with a contralateral spatial preference would respond more to central shapes that cued the ipsilateral than the contralateral spatial location, and vice versa for neurons with an ipsilateral spatial preference. The greater the absolute value of a congruency score, the greater was the separation between the neural response distributions of PREF and ANTI shape cues.
Only correct trials of the active task were included, so in all cases the monkey eventually made an eye movement to the location cued by a central shape, regardless of whether it was novel or familiar.
Active Shape–Saccade Association Task: Distinctive Responses to Familiar Shapes
The distinction scores can theoretically range from 0 to 100, where 0 indicates that the neural responses to same-meaning familiar overlearned shapes are indistinguishable whereas 100 signifies that they are completely separable. In reality, the neural responses to two same-meaning shapes will almost always be somewhat different by chance alone. We wanted to know whether there were differences in the early shape responses to familiar shapes, although they had repeatedly been associated with the same orienting action.
We performed a one-sided permutation test to see whether distinction scores for same-meaning familiar shapes were significantly greater than expected by chance alone. For each shape pair, we shuffled the labels (shape A or B) of the responses in all familiar shape trials and calculated a vector of distinction scores based on the shuffled labels. We repeated the process 1000 times.
Location Selectivity Mapping Task
As previously described, data from the location selectivity mapping task were used to define a preferred location of each LIP unit. Most units had a preferred location in the contralateral hemifield (67 CONTRA units), whereas a minority preferred a location in the ipsilateral hemifield (15 IPSI units). The circular mean direction of the preferred location was 192°, and the circular standard deviation was 59.7°.
Passive Shape-mapping Task
In the passive shape-mapping task, more CONTRA units also showed greater responses for shapes presented contralaterally than ispilaterally, and vice versa for IPSI units. Although this does not guarantee that the neurons' spatial preference would not change dramatically under more active task conditions, this nonetheless gave us some reassurance that the spatial preference would hold across tasks and stimuli.
One-sample binomial tests at times 0 to 190 msec after visual onset of shapes in the passive task confirmed that more units kept their location preference (contralateral vs. ipsilateral) across the two tasks (location selectivity mapping task and passive shape-mapping task) than would be expected by chance (CONTRA units: significant at all times from 30 to 190 msec after visual shape onset, all significant ps < .001; IPSI units: significant from times 30 to 80 msec after visual shape onset, first significant p = .035, all other significant ps < .008). Figure 3 shows an example unit's responses to shapes shown in different locations.
In the main task (the active shape–saccade association task), the visual shapes were always presented in the center of the screen. We therefore briefly describe the neural responses to visually presented central shapes in the passive shape-mapping task. In this task, LIP neurons showed varying degrees of responses to visually presented central shapes. Some neurons did not seem to respond much to the shapes at all, whereas others responded to the shapes, sometimes selectively so.
The selectivity of responses to centrally presented shapes (see section Methods: Data Analysis; Passive Shape-Mapping Task: Selective Responses to Centrally Presented Shapes) varied greatly between neurons (Mfamiliar = 0.59, SDfamiliar = 0.191; Mnovel = 0.56, SDnovel = 0.179). The selectivity of responses to centrally presented familiar and novel shapes was significantly correlated (r(80) = 0.746, p < 1.0 × 10−6). The selectivity of responses to central familiar shapes was nonetheless significantly greater than to central novel shapes (see Figure 4; paired samples t test: t(81) = 2.183, p = .032). The selectivity of responses to the familiar, previously behaviorally relevant shapes therefore appears to be enhanced. The differences between the selectivity of responses to novel and familiar shapes were nonetheless slight and should be interpreted with some caution given the fact that eye position was not tightly controlled in this secondary task. We turn now to the main task, the active shape–saccade association task.
Active Shape–Saccade Association Task
Changes in Neural Responses after Short-term and Long-term Learning
We wanted to see if and how the responses to novel shapes in the active shape–saccade association task changed with short-term learning or learning over the course of a single session. We also wanted to contrast these shorter-term learning effects with the effects of long-term learning over the course of days, weeks or months.
As can be seen in Figure 5, the average congruency (see section Methods: Data Analysis: Active Shape–Saccade Association Task: Congruency of Responses) of the neural population changed across a trial of the active shape–saccade association task. We are mainly interested in the shape period or the 500 msec time period between the visual onset of shapes and choice disks, but we note that the responses of the neural population became congruent within 100 msec from the visual onset of the choice disks for all trial types (early novel shapes, late novel shapes, and familiar shapes).
We wanted to know whether and then how shape responses were affected by short- and long-term learning of associating shapes with orienting to either a neurons' preferred or antipreferred location. We started by looking at the response dynamics within the shape period of the active shape–saccade association task for early novel, late novel, and familiar shape trials. We first ran three separate repeated-measures ANOVAs with Time as the single factor. The time variable consisted of congruency scores for 10 nonoverlapping 50-msec time bins centered on 50–500 msec after stimulus visual onset. Congruency did not significantly vary over time for early (82 units, F(6.164, 468.435) = 0.702, p = .652) or late (77 units, F(3.645, 277.008) = 0.533, p = .851) novel shapes. Congruency did, however, vary over time for familiar shapes (82 units, F(3.019, 229.414) = 5.345, p = .001). Responses to familiar shapes were congruent right after visual onset but became increasingly incongruent as the start of the choice period drew nearer.
We followed up with single sample t tests at each time point and for each type of cue, that is, early novel (82 units), late novel (77 units), and familiar (82 units), where we looked at whether the congruency scores were significantly different from zero (30 tests in total). Using conventional significance levels, the early novel responses were significantly congruent 450 msec after visual shape onset (p = .032; all other ps > .074), late novel responses were never significantly congruent or incongruent (all ps > .055), and responses to familiar shapes were significantly congruent 50 msec (p = .0005) and 100 msec (p = .035) after visual shape onset and significantly incongruent 400 msec after visual shape onset (p = .036). Early neural responses (i.e., 50 msec after visual shape onset, M = 6.8, SD = 16.8) to familiar shapes were significantly congruent even when a stringent correction for multiple comparisons was applied (threshold of significance with Bonferroni correction: 0.0017). These early responses to familiar shapes were also significantly more congruent than those to both early (paired samples t test, t(81) = 2.518, p = .014) and late (paired samples t test, t(76) = 2.541, p = .013) novel shapes. Unlike for the novel shapes, the neurons showed characteristic response differences between familiar PREF and ANTI shape cues extremely early after the visual onset of the shapes. This difference was modest but robust; around twice as many neurons favored familiar shapes that cued their preferred rather than their antipreferred location.
These early responses to familiar shapes tended to be congruent regardless of whether a unit's preferred location was contralateral or ipsilateral (congruency at 50 msec after visual shape onset for units with contralateral preference, 67 units: M = 6.9, SD = 17.37; ipsilateral preference, 15 units: M = 5.9, SD = 14.49; independent samples t test for differences in means, t(80) = 0.214, p = .831). Units with contralateral preference would be exposed to the same visual stimuli as units with ipsilateral preference, but the shapes' meaning would differ; a shape associated with a contralateral unit's preferred location would be associated with an ipsilateral unit's antipreferred location and vice versa. Early congruent responses are therefore unlikely to stem from some accidental properties of the shapes themselves, but instead reflect the learned task-related relationship between a shape and a neuron's spatial selectivity.
At a first glance, LIP responses to centrally presented shapes might seem little affected by the learning that took place within a single session. However, a neural population whose mean responses are neither congruent nor incongruent might nonetheless have undergone experience-dependent changes that are not reflected in the average congruency scores. In addition to looking at population averages, we therefore looked at whether congruency scores for late novel shapes could be predicted based on congruency scores for familiar shapes over and above the prediction based on congruency scores for early novel shapes alone.
Specifically, we performed a hierarchical regression at each of the 10 nonoverlapping 50-msec time bins (centered on 50–500 msec after stimulus visual onset) in the shape period of the active shape–saccade association task. Congruency scores for late novel shapes (from 77 units) were treated as a dependent variable. Congruency scores for early novel shapes and familiar shapes (from the same 77 units) were entered as predictor variables in consecutive steps. We then looked at whether a model that included congruency scores of both early novel and familiar shapes predicted congruency scores for late novel shapes significantly better than a model where the congruency scores of early novel shapes were used as the sole predictor variable.
Congruency scores of early novel shapes alone significantly predicted congruency scores of late novel shapes at all time points in the shape period (minimum R2 = .081, maximum R2 = .307; all ps < .011). Adding congruency scores of the familiar shapes as a second independent variable significantly improved the predictive power of the statistical model at time 150 msec after visual shape onset (R2 change = .042, p = .032, βcongruency of familiar shapes = .205) and then again at time 300 msec after visual shape onset and at all times from thereon (minimum R2 change = .095, maximum R2 change = .225; all ps < .004; threshold for significance after Bonferroni correction: 0.005; all βcongruency of familiar shapes > .315). Although the congruency scores of the late novel shapes kept some similarity to the congruency scores of the early novel shapes throughout the shape period, they increasingly resembled the congruency scores of the familiar shapes as the shape period progressed.
Response Accuracy and Congruency
The monkeys' performance for familiar shapes in the first block of the active shape–saccade association task was in general very good (mean 92% correct across 73 sessions). Mean performance for early novel shapes over those same sessions was only 54%. Although their performance was significantly better than the 50% performance to be expected from mere guessing in a two-alternative forced-choice task (t(72) = 3.355, p = .001, one-sample t test), it was still low, indicating that the monkeys in general knew little about the meaning of the novel shapes in the first block of trials. Their performance, however, generally improved for the novel shapes over the course of the training session. Performance for late novel shapes was on average 82% (average over 69 sessions). Performance for novel shapes was on average 77% correct in the last block in each session (i.e., the last block with at least 30 completed novel shape trials; average performance of 73 sessions). The improvement in performance for novel shapes from the first to the last block was significant (t(72) = 13.011, p < 1.0 × 10−6 , paired samples t test).
The congruency scores for familiar shapes, but not early or late novel shapes, were significantly linearly related to behavioral performance. This was supported by three regression analyses: one for early novel shapes, one for late novel shapes, and one for familiar shapes. The dependent variables for the three regression models were performance levels (percent correct) for the early novel (82 units), late novel (77 units), and familiar shapes (82 units), respectively. For each regression model, we included 10 independent variables, that is, congruency scores for early novel/late novel/familiar shapes in each nonoverlapping time window in the shape period (50–500 msec after visual shape onset).
According to our regression analysis, the early novel congruency scores only explained 9% of the variability in performance for early novel shapes, and the regression model was not significant (R2 = .091, F(10, 71) = 0.714, p = .709). The late novel congruency scores explained 23% of the variability in performance, but this result did not reach significance (R2 = .226, F(10, 66) = 1.931, p = .056). The congruency scores for the familiar shapes, however, explained 25% of the variability in the performance for familiar shapes, and this effect was significant (R2 = .246, F(10, 71) = 2.317, p = .020). Looking at the individual predictor variables, congruency at time 50 msec after familiar shape visual onset significantly predicted performance on familiar shape trials, with greater congruency being associated with better performance (β = .256, p = .025). Congruency at time 500 msec after familiar shape visual onset also significantly predicted performance, now with lower congruency values being associated with greater accuracy (β = −.523, p = .017). All other predictor variables did not reach significance (all ps < .12).
Persistent Distinctive Shape-related Activity after Long-term Learning
We have shown that repeatedly associating visual shapes with orienting to particular locations can change how LIP neurons respond when those shapes are seen, so that even the earliest neural responses can reflect the orienting behavior to which the shapes have been linked. The question remains, then, to what extent the responses are overwritten by experience. Do responses to familiar shapes that cue the same location still retain some individual characteristics, although the monkeys have been extensively trained, over the course of several months, on reacting to them in the same way? We measured this by calculating distinction scores for familiar shapes (see section Methods: Data Analysis: Active Shape–Saccade Association Task: Distinctive Responses to Familiar Shapes).
The population means of the distinction scores at each time point in the active shape–saccade association task can be seen in Figure 6. Figure 6 also shows the distribution of shuffled distinction scores at each time point after visual shape onset in the active shape–saccade association task. The graph depicts how much LIP neurons tended to differentiate between same-meaning familiar shapes at any given time.
The average distinction score in the active shape–saccade association task first became significant 40 msec after visual onset (p = .026), marginally missed the significance level at 50 msec after visual onset (p = .056), and stayed significantly greater than expected by chance throughout the rest of the shape period (i.e., until 500 msec after visual onset, highest p = .044, lowest p < .001). The barely missed significance level at 50 msec in the active task is probably because of the fact that early responses to familiar shapes start to reflect the learned associations, as described above.
In this article, we have documented and compared the experience-dependent changes of LIP responses to visual objects after longer and shorter learning periods of arbitrary pairings between objects and orienting.
The Effects of Short-term Learning
Experience-dependent changes in LIP responses to visual objects start to unfold over a short period of learning, but these changes are seen relatively late after the visual onset of these objects (i.e., the congruency scores of familiar shapes do not become a significant predictor of the congruency scores of late novel shapes until 150 msec after visual shape onset, at the earliest). The effects of short-term learning do not manifest themselves as an overall increase or decrease in LIP responses to objects that have been paired with orienting toward or away from the neurons' preferred locations. Instead, the responses of LIP neurons to novel objects increasingly resemble activity seen for familiar objects that share their meaning (i.e., cue the same location).
This late information is therefore not related in any obvious way to the responses evoked by the presentation of visual stimuli in the preferred or antipreferred locations of LIP neurons; it might be independent of the neurons' spatial selectivity and could be considered akin to the categorical information that has been reported to exist in LIP (Fitzgerald, Freedman, & Assad, 2011; Freedman & Assad, 2006, 2009). Late categorical information, task or rule selective activity (Stoet & Snyder, 2004) could be relayed to LIP from prefrontal regions (Asaad, Rainer, & Miller, 2000) like the dorsolateral pFC with which it is structurally connected (Blatt, Andersen, & Stoner, 2004). Through top–down control, the pFC might be able to modulate information in more posterior regions according to task demands (Miller & D'Esposito, 2005; Chao & Knight, 1998; see Miller & D'Esposito, 2005, for a discussion on top–down control signals originating in the pFC; see Pan & Sakagami, 2012, for a review of categorical representations in the pFC; see Seger & Miller, 2010, for a review of the neuroscience of categorical learning).
The Effects of Long-term Learning
Our results indicate that the earliest, apparently visual responses of LIP neurons can carry information about well-established yet still arbitrary associations of objects with particular orienting actions (i.e., the responses to familiar shapes were significantly congruent 50–100 msec after visual shape onset). These visual responses tend to be greater for objects that have repeatedly been associated with looking toward, rather than away from, LIP neurons' preferred spatial locations. After a long training period, LIP neurons reflect an arbitrary association by responding to familiar central shape cues as if a weak visual stimulus was shown in the associated empty peripheral location or, alternatively, as if a weak motor plan to the empty peripheral location had been formed.
Orienting-related responses to visual shapes appear so early after their visual onset that it is highly unlikely that they are the result of motor commands fed back from other brain regions. It is also implausible that these early responses to visual shapes are inherited from ventral visual regions; although LIP is known to be interconnected with shape selective ventral areas (Ungerleider, Galkin, Desimone, & Gattass, 2008; Blatt et al., 2004; Webster, Bachevalier, & Ungerleider, 1994), the visual onset latencies of neurons in those regions tend to be long, with typical values around or over 100 msec (although this varies somewhat by subregion and can for a small minority of neurons be shorter; Kiani, Esteky, & Tanaka, 2005; Tamura & Tanaka, 2001; Schmolesky et al., 1998; Baylis, Rolls, & Leonard, 1987). It is therefore doubtful that object or shape information reaches LIP solely through a circuitous route through the ventral visual stream, although we do consider it likely that LIP eventually receives some information about visual objects from the temporal cortex and other ventral regions. Instead, these responses appear to be generated from the initial bottom–up wave of visual signals that reach LIP, presumably created de novo in the parietal cortex from yet unknown inputs.
Experience-resistant Responses to Visual Objects
LIP responses to visual objects can be modified by experience but are not completely overwritten by experience, at least not by the type of experience provided in this study. During the visual presentation of an object, before any overt behavioral response is allowed, LIP neurons do carry information about the orienting action associated with the object and which the monkey is going to perform (i.e., the congruency of responses to familiar shapes significantly varied over time within the shape period, see Figure 5, and these temporal fluctuations were significantly related to behavioral performance). During this same period, the responses to the visual objects can nonetheless be distinct although they were similarly acted on in the past and will lead to the same orienting behavior in the future (i.e., the distinction scores of familiar shapes were significantly greater than would be expected by chance throughout almost the entire shape period, see Figure 6). Neural responses to such objects can be separable and resistant to a complete experience-dependent overhaul despite the fact that the monkeys were trained over the course of many months to treat the objects as equivalent.
Relations to Behavior
Orienting guided by central cues is often described as endogenous, voluntary, or controlled, as opposed to the exogenous, reflexive, and automatic effects of peripheral cues (Müller & Rabbitt, 1989; Posner, 1980). Indeed, learning the meaning of novel central cues only has a measurable effect on neural responses in LIP relatively late after cue onset (150 msec after visual shape onset, at the earliest), presumably through top–down feedback, and these responses are nothing like the responses to peripheral visual stimuli in the locations cued by the central objects.
Still, our results are in alignment with the cumulating behavioral evidence that a former endogenous visual cue might be said to become exogenous with enough training (Van der Stigchel et al., 2010; Dodd & Wilson, 2009; Shaki & Fischer, 2008; Fischer, Warlop, Hill, & Fias, 2004; Fischer et al., 2003; Dehaene, Bossini, & Giraux, 1993). LIP neurons can respond to familiar central shape cues as if a weak stimulus is actually being presented in the empty peripheral location cued by the central object. The time course of these neural effects closely follows that of transient visual attention; the facilitatory behavioral effects of a peripheral cue are greatest for a target shown in that location around 50–150 msec after cue visual onset, and this facilitation can give way to inhibition with somewhat longer delays of around 250–500 msec (Castel, Chasteen, Scialfa, & Pratt, 2003; Nakayama & Mackeben 1989; Posner & Cohen, 1984); our familiar central shape cues also evoked responses that were maximally and significantly congruent at 50 msec after cue visual onset and these responses became incongruent around 300 msec after cue visual onset (significantly so at 400 msec after cue visual onset, see Figure 5). The neural effects are modest, but such small neural effects are in alignment with the small behavioral effects reported for stimuli like central arrows that can produce a difference in RTs for targets in cued and uncued locations as small as less than 10 msec and sometimes, depending on specific parameters, can be a few tens of milliseconds (see e.g., Ristic & Kingstone, 2006).
Furthermore, our results seem to support that the more pronounced these temporal fluctuations in neural responses to familiar central shape cues are, the more accurate the behavioral responses tend to be; we find that more congruent responses at time 50 msec after cue visual onset and more incongruent responses at time 500 msec after cue visual onset are significantly associated with greater behavioral accuracy for familiar shape cues. No such consistent relationship between congruency and behavioral performance was found for novel shapes, either before or after training within a single session. This is in line with studies that find that, whereas neural activity in the parietal cortex can be affected by associative learning within a single session (Eliassen et al., 2003; Deiber et al., 1997), the parietal cortex becomes increasingly involved as the arbitrary associations become more automatic and overtrained (Grol et al., 2006; Eliassen et al., 2003).
Our interpretation of these temporal fluctuations is that the familiar shape cues gained the ability to rapidly bias spatial attention to a particular peripheral location away from the central objects themselves. Such shifts of covert attention evoked by trained spatial stimuli are thought to be obligatory (Van der Stigchel et al., 2010). Because the target did not appear in this location until 500 msec after cue visual onset and because it was maladaptive for the monkeys to actually direct their gaze to that location until a target appeared, initial congruent responses gave way to incongruent responses until the target appeared and had to be acted on.
This interpretation can potentially be challenged on the grounds that a behavioral inhibition of return (IOR) effect typically only occurs in situations where a (peripheral) cue is nonpredictive of target location, so there is no particular reason for the subject to expect the target to appear at the cued location (for reviews on IOR, see Lupiáñez, Klein, & Bartolomeo, 2006; Klein, 2000). However, our interpretation might still hold if the main factor that determines whether or not an IOR effect occurs is not the predictability of the target, but instead whether or not it is task relevant and beneficial to maintain attention at the cued location—which it usually is when a cue is predictive and generally is not when a cue is not predictive. As Klein (2000) notes: “The initial response to a peripheral visual event is facilitation of the processing of nearby stimuli, presumably owing to a reflexive shift of attention towards the source of stimulation. However, when the event is not task-relevant and attention has had time to disengage from it, an inhibitory aftereffect can be measured….” Although our central cues were predictive of which peripheral stimulus the monkeys should eventually choose, saccading to the predicted location was not allowed during the shape period. Attending to and subsequently facilitating saccadic responses to the predicted location during the shape period might therefore have been detrimental to performance, and such responses that had been associated with the familiar objects might thus need to be inhibited. This is consistent with the evidence supporting the idea that “one effect of IOR is to inhibit responses that are normally associated with stimuli” (Klein, 2000).
The time course for the familiar shapes during the shape period appears to be different from that of the novel shapes (Figure 5). Only the responses of the familiar shapes and not that of early or late novel shapes show significant temporal fluctuations in congruency across the shape period, where congruent responses are followed by incongruent responses. This is consistent with previous work that suggests that associating central shape cues with particular peripheral target locations over a short training session might not suffice to affect exogenous shifts of attention (Sigurdardottir et al., 2014), that central cues might become more effective in inducing obligatory shifts of attention toward peripheral locations with longer training sessions (Van der Stigchel et al., 2010; Dodd & Wilson, 2009), and that IOR follows exogenous but not endogenous shifts of attention (Lupiáñez et al., 2006).
With enough training, an object of any shape might acquire the ability to bias orienting to a particular location. However, although our results show that experience can affect the responses of LIP neurons to visual objects (Figures 4 and 5), these neurons can nonetheless respond significantly differently to two objects that cue the same location despite a lengthy training period that encourages the monkeys to treat the two objects as equivalent (Figure 6). We speculate that persistent response differences to same-meaning objects reflect their inherent shape-derived orienting biases. Our own behavioral work (Sigurdardottir et al., 2014) shows that information derived from the shape of objects—even never-before-seen novel ones—can rapidly and automatically bias orienting to particular spatial locations. These links between shape and space, which can be thought of as initial hypotheses on where to look and pay attention, might be hard or impossible to fully overcome (Sigurdardottir et al., 2014).
The activity of LIP might best be understood as competing orienting biases or affordances (Cisek & Kalaska, 2010; Cisek, 2007; Gibson, 1979) or the relative merit of the possible sources of information worth exploring with the eyes and attention. We propose that the shape of objects, because of intrinsic properties and previous experience, systematically biases orienting (Sigurdardottir et al., 2014; Red et al., 2012; Patel et al., 2010; Theeuwes, Mathôt, & Kingstone, 2010; Ristic & Kingstone, 2006; Tipples, 2002; Vishwanath, Kowler, & Feldman, 2000; Melcher & Kowler, 1999; Egly, Driver, & Rafal, 1994; He & Kowler, 1991). We hypothesize that LIP plays a crucial role in extracting such a shape-induced orienting bias and that this bias contributes to the brain region's selective responses to visual objects of different shapes.
Thinking of LIP shape selectivity as serving the purpose of orienting might help to make sense of the puzzling finding that LIP and its putative human homologue can be relatively tolerant to image transformations like scaling and translation (Janssen et al., 2008; Konen & Kastner, 2008a, 2008b; Sereno & Maunsell, 1998). Such invariance has most often been thought to be a hallmark of the ventral visual pathway (Booth & Rolls, 1998; Logothetis & Sheinberg, 1996; Tanaka, 1996; Ito, Tamura, Fujita, & Tanaka, 1995; Desimone, Albright, Gross, & Bruce, 1984; Gross, 1973). Visual stimuli can however also show invariance of the orienting bias they evoke, such as when seeing a face tilted 90° evokes orienting shifts to the side to which the person's eyes would have been looking had the face been in its canonical upright position (Bayliss & Tipper, 2006; Bayliss, di Pellegrino, & Tipper, 2004). We expect LIP neurons to be tolerant to changes in a visual stimulus that preserve not its identity or form but its inherent or acquired orienting bias.
Several brain regions might play a role in the initial learning of a new mapping between a stimulus and a response, including parts of the premotor cortex, the pFC, the medial-temporal lobe, and the BG (Mattfeld & Stark, 2011; Seger, 2009; Williams & Eskandar, 2006; Pasupathy & Miller, 2005; Brasted & Wise, 2004; Brasted, Bussey, Murray, & Wise, 2003; Hadj-Bouziane & Boussaoud, 2003; Bussey, Wise, & Murray, 2001; Asaad, Rainer, & Miller, 1998; Tremblay, Hollerman, & Schultz, 1998; Murray & Wise, 1996; Chen & Wise, 1995a, 1995b). Brain areas other than the posterior parietal cortex might thus be responsible for establishing a new arbitrary link between seeing an object of a particular shape and orienting to a particular location. Learning-related feedback could nonetheless reach regions such as the LIP and be reflected in LIP's responses—albeit not its earliest visually evoked responses.
As learning progresses and behavior becomes more automatic, other parts of the brain might start to take part in or even take over the representation of overtrained associations (Seger, 2009; Nixon, McDonald, Gough, Alexander, & Passingham, 2004; Kurata & Hoffman, 1994; Passingham, 1988). It has been argued that subcortical regions gradually train cortical areas on the associations so that eventually the behavior can be supported by the cortex (Pasupathy & Miller, 2005).
Once the animals in our study had fully learned to arbitrarily associate shape cues with saccading to particular locations, they almost always looked to the preferred location of each recorded neuron following the presentation of particular shape cues and looked to the antipreferred location following the presentation of other shape cues. In the former case, over the course of a long training period, the shape cues were presumably repeatedly succeeded by increased activity of these LIP neurons because of the orienting behavior itself, and the corresponding visual inputs might consequently gradually strengthen over time through small changes of synaptic weights. In the latter case, shape cues were repeatedly followed by relatively lesser activity, and the corresponding visual inputs could have gotten relatively weaker with more experience.
When an association is highly overlearned, LIP might therefore become able to support extremely rapid arbitrary transformations between a visual stimulus and an orienting response independent of top–down feedback from regions such as the pFC (Swaminathan & Freedman, 2012). Without parietal cortex, the associations could still be remembered, but the associated behavior might be less automatic. LIP might therefore be involved in extracting an orienting bias from an object that comes about through the association of the object with a motor response to an important target in a distant location. One role of the dorsal stream might be to extract an orienting bias from an object as it relates to a key future spatial response, such as a future eye movement, reach, grasp, withdrawal, or spatial navigation.
This work was supported by National Science Foundation grant IIS-0827427 (David L. Sheinberg), National Science Foundation grant SBE-0542013 (Temporal Dynamics of Learning Center), National Institutes of Health grant R01EY14681 (David L. Sheinberg), and an International Fulbright Science and Technology award (Heida M. Sigurdardottir). The authors thank John Ghenne for his help with animal care and experiments.
Reprint requests should be sent to David L. Sheinberg, Department of Neuroscience, Brown University, Box G-LN, Providence, RI 02912, or via e-mail: firstname.lastname@example.org.