The division of cortical visual processing into distinct dorsal and ventral streams is a key concept in primate neuroscience [Goodale, M. A., & Milner, A. D. Separate visual pathways for perception and action. Trends in Neurosciences, 15, 20–25, 1992; Steele, G., Weller, R., & Cusick, C. Cortical connections of the caudal subdivision of the dorsolateral area (V4) in monkeys. Journal of Comparative Neurology, 306, 495–520, 1991]. The ventral stream is usually characterized as a “What” pathway, whereas the dorsal stream is implied in mediating spatial perception (“Where”) and visually guided actions (“How”). A subpathway emerging from the dorsal stream and projecting to the medial-temporal lobe has been identified [Kravitz, D. J., Saleem, K. S., Baker, C. I., & Mishkin, M. A new neural framework for visuospatial processing. Nature Reviews Neuroscience, 12, 217–230, 2011; Cavada, C., & Goldman-Raiuc, P. S. Posterior parietal cortex in rhesus monkey: I. Parcellation of areas based on distinctive limbic and sensory cortico-cortical connections. Journal of Comparative Neurology, 287, 393–421, 1989]. The current article studies the coordination of visual information typically associated with the dorsal stream (“Where”), with planned movements, focusing on the temporal lobe. We recorded extracellular activity from 565 cells in the human medial-temporal and frontal lobes while 13 patients performed cued hand movements with visual feedback (visuomotor task), without feedback (motor task), or observed visual feedback without motor movement (visual-only task). We discovered two different neural populations in the human medial-temporal lobe. One consists of motor-like neurons representing hand position, speed or acceleration during the motor task but not during the visuomotor or visual tasks. The other is specific to the parahippocampal gyrus (an area known to process visual motion [Gur, M., & Snodderly, D. M. Direction selectivity in V1 of alert monkeys: Evidence for parallel pathways for motion processing. Journal of Physiology, 585, 383–400, 2007; Sato, N., & Nakamura, K. Visual response properties of neurons in the parahippocampal cortex of monkeys. Journal of Neurophysiology, 90, 876–886, 2003]) and encodes speed, acceleration, or direction of hand movements, but only during the visuomotor task: neither during visual-only nor during motor tasks. These findings suggest a functional basis for the anatomical subpathway between the dorsal stream and the medial-temporal lobe. Similar to the recent expansion of the motor control process into the sensory cortex [Matyas, F., Sreenivasan, V., Marbach, F., Wacongne, C., Barsy, B., Mateo, C., et al. Motor control by sensory cortex. Science, 330, 1240–1243, 2010], our findings render the human medial-temporal lobe an important junction in the process of planning and execution of motor acts whether internally or externally (visually) driven. Thus, the medial-temporal lobe might serve as an integration node between the two processing streams. Our findings thus shed new light on the brain mechanisms underlying visuomotor coordination which is a crucial capacity for everyday survival, whether it is identifying and picking up food, sliding a key into a lock, driving a vehicle, or escaping a predator.
Visuomotor coordination is a skill employed in a wide range of activities, from the most fundamental, for example, identifying and picking up food, to the highly specialized, such as endoscopic operation or flying an airplane. These processes require the brain to process visual input information and transform it into an actual output motor plan, relying on high-level perception of the world being visualized. Visual perception has long been an interdisciplinary field integrating insights and data from neuroscience, psychology, and computation (Marr, 1982) and so is the field of motor perception and coordination (Klapp & Jagacinski, 2011; Carmeli, Bar-Yossef, Ariav, Levy, & Liebermann, 2008). In an early stage of cortical visual processing, in the primary visual cortex (V1), the information divides into two processing streams (Goodale & Milner, 1992; Steele, Weller, & Cusick, 1991). The dorsal stream courses through occipitoparietal cortex to the posterior half of the inferior parietal lobule (area PG) and from there to the dorsolateral PFC. The ventral stream, on the other hand, courses through the occipito-temporal cortex to area TE (rostral inferior temporal cortex) and continues into the ventrolateral PFC (Mishkin, Ungerleider, & Macko, 1983; Mishkin & Ungerleider, 1982; Ungerleider & Mishkin, 1982). Under this scheme, the medial-temporal lobe is considered an integral part of the ventral stream. Little evidence exists regarding the role of the medial-temporal lobe in visual guidance of movements and in visuomotor or motor tasks. The hippocampus has recently been described in the context of memory-guided automatic movements (Uemura, 2010) and is believed to participate in the control of the ventral-striatal loop (or limbic loop), which plays a role in the control of motor behavior (e.g., motor reactions to emotions; Fernández-Seara, Aznárez-Sanado, Mengual, Loayza, & Pastor, 2009; van Rooyen, Young, Larson, & Teskey, 2006; Duvernoy, Vannson, Cattin, & Naidich, 2005; Tesche & Karhu, 1999; Halgren, 1991; Miyashita, Rolls, Cahusac, Niki, & Feigenbaum, 1989). The entorhinal cortex represents self-position in the environment in the form of place cells (Howard & Natu, 2005) and grid cells (McNaughton, Battaglia, Jensen, Moser, & Moser, 2006) and velocity for path integration (i.e., inference of position by integration of velocity; Burak & Fiete, 2009; Blair, Gupta, & Zhang, 2008; Howard, Fotedar, Datey, & Hasselmo, 2005; Lengyel & Erdi, 2004). The parahippocampal gyrus is known to play a critical role in spatial navigation (Baumann, Chan, & Mattingley, 2010; Epstein, 2008; Ekstrom et al., 2003; Aguirre & D'Esposito, 1999; Maguire, Frackowiak, & Frith, 1997) and is also associated with certain types of motor output (Yan et al., 2006; Rocca et al., 2004), such as head movements (Robertson, Rolls, Georges-Francois, & Panzeri, 1999), oculomotor tasks (Nagel, Sprenger, Hohagen, Binkofski, & Lencer, 2008; Ploner et al., 2000), and imitation of meaningful actions (Rumiati et al., 2005). In addition to motor-related evidence, however, the parahippocampal gyrus is anatomically connected to several visual areas (e.g., V3A and dorsal V4; Blatt, Pandya, & Rosene, 2003; Felleman & Van Essen, 1991), is known to have receptive field characteristics (Wilson, Babb, Halgren, & Crandall, 1983), and is active during multiple aspects of visual information processing: response to views of familiar places (Ekstrom et al., 2003), encoding object location (Fischer, Spotswood, & Whitney, 2010; Sugiura, Shah, Zilles, & Fink, 2005; Maguire, Burgess, Donnett, O'Keefe, & Frith, 1998; Owen, Milner, Petrides, & Evans, 1996), spatial memory (Ploner et al., 2000), visual processing of moving objects (Gur & Snodderly, 2007; Sato & Nakamura, 2003), perception of biological motion (Rumiati et al., 2005; Ptito, Faubert, Gjedde, & Kupers, 2003), and even reading sentences describing motion (Wallentin, Ostergaarda, Lund, Ostergaard, & Roepstorff, 2005). Nevertheless, the transition from processing input information to generating an output motor plan has remained enigmatic. Our study has therefore focused on the representation of hand movements in the temporal lobe and their generation from visual information during visuomotor coordination.
Patients and Electrophysiology
Activity of single neurons was recorded in 13 patients (19–47 years old; mean = 32.2 years, SD = 10.1 years; five right-handed women, two left-handed women, five right-handed men, and one left-handed man) with pharmacologically intractable epilepsy undergoing invasive monitoring with intracranial depth electrodes to identify the seizure focus for potential surgical treatment. The patients participated in 22 recording sessions, each on a different day. Each patient had between 7 and 13 electrodes, implanted based on clinical criteria only (Fried et al., 1999). Their anatomical location was verified by MRI or computed tomography scans, coregistered to preoperative MRI. Each of these electrodes terminated with a set of nine 40-μm platinum–iridium microwires. Signals from these microwires were recorded at ∼28 kHz and bandpass filtered in the range 0.3–3 kHz using a 64-channel acquisition system (Neuralynx, Tucson, AZ) and spike sorted as described by Tankus, Yeshurun, and Fried (2009). The classification between single cells and multiunit cells was determined by the SUMU-LDA algorithm using the highest-accuracy linear discriminator obtained by Tankus, Yeshurun, and Fried (2009) and verified visually: 107 single units and 433 multiunits. Units recorded from in different sessions are treated as different in this study. All studies conformed with the guidelines of the medical institutional review board at the University of California, Los Angeles. All patients provided informed consent after the study was explained to them preoperatively. Participation in this study was completely voluntary.
Patients performed hand movements to control a computer cursor (with or without visual feedback of the consequences of their actions) or passively viewed the same visual feedback cues (Figure 1). All tasks were performed in the same session, and the visual objects involved, namely the cursor and target, were kept simple and did not incorporate components that would emphasize ventral stream processing to reveal dorsal-stream-like processing in the ventral stream.
In the visuomotor task (i.e., hand movements with visual feedback), the patients moved a joystick (Attack3, Logitech) to control a computer cursor. They first had to position the cursor inside a ring-shaped target at the center of the screen. Following a short delay period (random between 2 and 3 sec), the target would “jump” to one of eight randomly lit targets, equispaced on a virtual circle around the center. The patients would then move the cursor to follow the target (“Center-Out”; Moran & Schwartz, 1999). The target would return to the center, with the patients following it, and the task will repeat. Patients performed between 32 and 102 trials per visuomotor session (average = 79.5 trials, SD = 18.0). In the visual-only task, another individual performed the same Center-Out task, while the patient only watched the computer screen with no hand movements. The hand of this individual was outside the visual field of the patient. Patients were explicitly instructed to track the cursor with their eyes. As no eye-tracking was performed, this was monitored by an experimenter. The patients watched between 22 and 115 trials per visual-only session (average = 69.0 trials, SD = 25.2). In the motor task, the patients moved the joystick either to the left or right according to a cue to control for motor responses. Before each run of this task, we played/showed the patient 30 randomly selected cues and excluded from the analysis cue-responsive cells. The cue was either auditory (beep: left: 2500 Hz, 50 msec; right: 400 Hz, 50 msec; “Ready!”: 400 Hz, 200 msec; 17 sessions) or visual for hearing-impaired patients (filled square, 13.23 cm × 13.23 cm, displayed for 200 msec; left: blue; right: green; “Ready!”: red). Vision of the movement was prevented either by having the hand and joystick completely outside the field of view (16 sessions; monitored by an experimenter) or by closing the eyes (six sessions). The patients performed between 13 and 20 trials per session (average = 18.3 trials; SD = 1.8). We limit the movements to the left or right only because of the difficulty in memorizing the association of eight different cues with eight different directions.
To dissociate between hand movements and object–eye movements, seven of the patients controlled the Center-Out task by both the joystick and a stylus pen (Graphire3 6 × 8 tablet, Wacom Technology Co., Vancouver, WA) during the same session. Each device creates a different association between hand and object (and thus eye) movements. With the stylus pen, the speed profile of the object and hand is identical (up to a scaling that maps the working area of the tablet to screen area). With the joystick, a stationary position of the hand is translated into a constant speed of the object, whereas every movement of the hand generates acceleration or deceleration. We recorded from 224 units in eight sessions: 64 units in the parahippocampal gyrus, 52 in the hippocampus, 40 in the entorhinal cortex, 14 in the amygdala, and 54 in the anterior cingulate gyrus.
Only neurons with a firing rate of ≥0.5 spikes/sec were considered for any type of analysis. To relate the neural activity to the continuous kinematic parameters, speed and acceleration, we adopted the nondirectional correlation analysis suggested by Moran and Schwartz (1999), which divides movement time into 10 equal duration bins in each trial and adds 15 prebins of the same size before movement onset. We also added 15 postbins following the end of movement to test for feedback neural activity as well. We subtracted the tonic background component and added the firing rate bin-by-bin across all eight targets to reveal the nondirectional component of the discharge rate profile (“normalized firing rate”), as described by Moran and Schwartz (1999). We then computed the average lag for each unit based on the maximal absolute Pearson correlation between the binned firing rate and speed profiles and similarly for acceleration. Units with correlations significantly different from 0 (p < .05/31; Bonferroni correction for time lags) were deemed related to speed or acceleration, respectively.
Although the aforementioned analysis can relate the firing rate to speed or acceleration, it does not take into account the interdependencies of the two kinematic parameters. To address this issue, we employed the partial cross-correlation analysis (Stark, Drori, & Abeles, 2006), which estimates linear correlations between neural activity and a given parameter that are not affected by linear correlations with other features at multiple time delays. The method thus computes trial-by-trial correlations between neuronal activity and variations in kinematics. We applied the partial cross-correlation analysis to data aligned to movement onset and only to units that have already been declared significant by the nondirectional correlation method (that is, we required both criteria to hold). In the Results section, we describe only units that adhere to both criteria.
According to the partial cross-correlation method, a neuron related to speed is one that at a certain time lag is significantly responsive to speed independent of acceleration and also independent of position; similarly for relation to acceleration and position. The model for position as well as the computation of optimal time lags and significance criteria were described by Tankus, Yeshurun, Flash, and Fried (2009). A relation was declared significant if more than 50% of the entries in the optimal row/column were significant, and for each entry, the requirement was p < .05 following Bonferroni correction (factor of 71). We examined only time lags between −700 and 700 msec (in steps of 20 msec), despite the 2–3 sec hold period, to avoid mixture of trials. Negative time lags refer to firing before the related movement.
For directional responses, the mean firing rate of single- and multiunits during the response period of each trial (between the “Go!” signal and stop inside a target) was subjected to two different statistical tests, adopted from Crammond and Kalaska (1996). First, a two-way ANOVA (Directions vs. Repetitions) detected statistically different responses with the direction of movement (p < .01). Second, a nonparametric “bootstrapping” test (Cisek, Crammond, & Kalaska, 2003; Georgopoulos, Kettner, & Schwartz, 1988), determined the degree of directional bias in the task-related activity of each unit. Comparing the mean length of the distribution of its discharge across all eight movement directions with the chance of this distribution (by shuffling the directional “label” of each trial) decided whether the cell was directionally tuned (4000 shuffles; p < .01). Only when both criteria were met was a unit declared directionally tuned. We avoided the third (Rayleigh) test of Crammond and Kalaska (1996), because in many cases the observed tuning curves were not unimodal.
In the visual-only task, an experimenter monitored that the patients are attentive to the task. Nevertheless, the percentage of units related to the kinematic parameters was very low in all regions (in every region: <2%). To verify that the decreased neuronal modulations are not because of reduced attention to the passive observation task (Khan, Heinen, & McPeek, 2010; Garza, Eslinger, & Barrett, 2008; Chiba, Yamaguchi, & Eto, 2005; Boussaoud, 2001), we repeated the analysis using data limited to only the first 5 min of every session, during which higher attention of the subject was observed. This analysis found very low percentages of cells related to the kinematic parameters (speed: 0% [0/565]; acceleration: 0.4% [2/565]; position: 0.4% [2/565]).
We recorded from 565 units and analyzed 540 of them (25 were excluded due to response to the cue itself): 124 cells in the parahippocampal gyrus, 161 in the hippocampus, 87 in the entorhinal cortex, 59 in the amygdala, and 109 in the ACC. Using the three tasks (visuomotor, visual-only, and motor), we found two distinct populations of cells in the medial-temporal lobe. One population was related to the speed and/or direction of hand movement during visuomotor coordination, but not during visual-only or motor tasks. The other is a population of motor-like cells, which were related to the kinematic parameters speed, acceleration, and position during the motor task, but not during the visuomotor or visual-only tasks.
The first population of visuomotor units was found mainly in the parahippocampal gyrus (PHG), wherein 9% (11/124) of the units were related to speed during the visuomotor task, but not during the visual-only or motor tasks. In all other regions, the percentage of speed-related units during the visuomotor task was low (<6%), and its proportion differed significantly from that of the parahippocampal gyrus (p < 3E−4, χ2(1) = 13.6). Of the speed-related parahippocampal units, 36% (4/11) were also related to acceleration at different time lags during the visuomotor task, whereas the percentage of position-related visuomotor units was very low in all recorded regions (<1% in each area).
For example, the nondirectional component of the firing rate of a neuron in the right PHG was highly correlated with the average movement speed during the visuomotor task (r = .89) but showed much lower correlation during the visual-only or motor tasks (r = 0.62 and .69, respectively; Figure 2). Analyzing the relation of the firing rate of this cell with the speed profile on a trial-by-trial basis using partial cross correlation showed significant response to speed that is independent of acceleration only during the visuomotor task (Figure 2, graphs on the right).
But whose kinematic profile do these neurons plan: that of the hand or that of the object (the cursor), which is also shared with the eye tracking it? By dissociating hand trajectory from object–eye trajectory (see Experimental Paradigms under Experimental Methods), we were able to find examples of parahippocampal cells planning hand movement as well as cells planning object or eye movements. With the stylus pen, the activity of 9.4% (6/64) of the parahippocampal cells was related to speed (all other regions: 1.9% (3/160); significantly different proportion: p = .01, χ2(1) = 6.7). A third of these units encoded kinematic parameters in a manner consistent with encoding hand movements (speed with the stylus pen and acceleration with the joystick; position with the stylus pen and speed with the joystick), which cannot be explained by object or eye movements. Another third of the responsive units responded similarly for both control devices (i.e., independent of actual hand movements) suggesting encoding of kinematic parameters of the object or the tracking eye.
The parahippocampal gyrus is also the only region where we found directionally tuned units during the visuomotor task (PHG: 12% [15/124]; all other regions: 0% [0/416]; significantly different proportion: p = 6E−13, χ2(1) = 51.8). The vast majority of these units (13/15, 87%) responded neither during the visual-only task nor during the motor task. For example, a unit tuned to the up and down directions during the visuomotor task showed no significant directional tuning during the visual-only and motor tasks (Figure 3).
During the motor task, we found a nontrivial (>7%) percentage of neurons significantly related to the kinematic parameters (position, speed, and acceleration) in all recorded temporal lobe areas (to at least one parameter in each area; see Figure 4, left). The vast majority of units in this population were not significantly related to these parameters during the visuomotor or visual-only tasks (Figure 4, right). We thus found a second population of neurons in the medial-temporal lobe, motor-like neurons, which were related to kinematic parameters during the motor task, but not during the visuomotor or visual-only tasks.
An example is a neuron in the left PHG that was highly correlated (r = .97) with the average speed profile during the motor task, but exhibited a much lower correlation during the visuomotor or visual-only tasks (r = .71 and .59, respectively; Figure 5). The partial cross-correlation method of the firing rate of this cell with speed that is independent of acceleration was significant during the motor task, but not during the visuomotor or visual-only tasks (Figure 5, graphs on the right). The partial cross correlation of the firing rate with acceleration that is independent of speed was not significant. An acceleration-related example unit from the right hippocampus appears in Figure 6. Its nondirectional firing rate follows the acceleration profile closely (r = .96) during the motor task, but not during the visuomotor or visual-only tasks (r = .71 and .61, respectively), with significant partial cross correlation with acceleration that is independent of the correlation with speed.
Thus, we find consistent relations to movement, vision and their combination, because the two populations are almost completely limited to response to one task (visuomotor or motor) but not to the other two: All (100%, 11/11) PHG units that were speed-related during the visuomotor task did not respond significantly during the visual-only or motor task, and vice versa: the vast majority of units responsive during the motor task did not respond significantly during the visuomotor or visual-only tasks (Figure 4). This is also consistent with the fact that there were very few responses during the visual-only task (in every region: <2%).
Our study revealed and characterized two different populations of cells in the medial-temporal lobe: visuomotor cells, which are correlated with the kinematics of actual movements only in the presence of visual input, and motor-like neurons that participate in hand movements in the absence of visual input.
Our finding of a population of visuomotor speed- and direction-related neurons in the parahippocampal gyrus focuses on an early stage in the process of sensorimotor transformations: transformation between visual input and hand kinematics (output), which precedes known transformations between extrinsic and intrinsic coordinate frames of the acting limb (Kakei, Hoffmand, & Strick, 2003). The parahippocampal involvement in visuo-motor transformations suggests a physiological basis for the anatomical subpathway between the dorsal stream and the medial-temporal lobe (Kravitz, Saleem, Baker, & Mishkin, 2011; Cavada & Goldman-Raiuc, 1989). Thus, the subpathway, which connects the angular gyrus and superior occipital gyrus areas (Rushworth, Behrens, & Johansen-Berg, 2006) or the lateral intraparietal area (Blatt, Andersen, & Stoner, 1990) with the parahippocampal cortex, might be the source of action-guiding visual information to the visuomotor population we described in the parahippocampal gyrus. These findings suggest that the medial-temporal lobe—specifically the parahippocampal gyrus—plays a functional role in both “what” and “where”/“how” streams and render it an important junction in the process of planning and execution of motor acts. Hence, the medial-temporal lobe might serve as an integration node between the two processing streams.
The visuomotor population also expands the known role of the parahippocampal gyrus in spatial navigation, which includes path planning for locomotion based on visual information and, in particular, the relation to locomotive speed and running direction (Doeller, Barry, & Burgess, 2010). Thus, the parahippocampal gyrus is responsible, at the kinematic level, not only for locomotive navigation but also for spatial navigation of the hand or an object controlled by the hand (the cursor in our case). Further studies are needed to dissociate object movement from eye movements. This also provides a kinematic explanation for the recently found parahippocampal activation during motor execution (Mukamel, Ekstrom, Kaplan, Iacoboni, & Fried, 2010).
The population of motor-like neurons was related to kinematic parameters in all recorded temporal areas (H, PHG, EC and A) and according to two different computational methods. This evidence is in line with the recently described participation in motor control of areas classically considered as sensory cortex (Matyas et al., 2010). Our findings also extend movement-related evidence in the hippocampus (Halgren, 1991) and describe them at the kinematic level. As such, the hippocampus and parahippocampal gyrus not only represent places in a 3-D virtual environment (Ekstrom et al., 2003) but also the position of the hand. This implies that body shape and coordinates are represented in greater detail in these areas than reported so far: not only in large-scale coordinate systems in which the body location (“place”) is represented but also in proximity frames of reference, where different body parts (e.g., hands) can interact with the surrounding. Whereas evidence relating parahippocampal neural activation with motor control is very limited and lacks for kinematics (Yan et al., 2006), the current results demonstrate the encoding of hand speed and acceleration in this area.
In the EC, the only significantly encoded kinematic parameter in the motor-like population appears to be position, which expands its role from representation of 3-D position (location) of the subject, as in place (Howard & Natu, 2005) or grid (McNaughton et al., 2006) cells, to also that of the hand during movement. We suggest that the EC may thus participate in the formation of maps that allow one to orient in space, in terms of both self and hand locations. The amygdala was shown to be involved in movement-related perception (Bonda, Petrides, Ostry, & Evans, 1996) or feeling (Nakagawa et al., 2003; Uno et al., 2000) and in incentive processes controlling the performance of goal-directed instrumental actions (Shiflett & Balleine, 2010). Our findings enhance the understanding of these movement-related phenomena by relating cellular activity to kinematic parameters of an actual movement. These results can potentially reflect actual control of hand movements, recalls of a previously learnt trajectory from memory (Bar, 2009; Gelbard-Sagiv, Mukamel, Harel, Malach, & Fried, 2008) or arbitrary associations between antecedents (auditory or visual stimuli) and actions (movements; Murray, Bussey, & Wise, 2000; Wise & Murray, 1999, 2000).
To summarize, we found two populations of cells: motor-like, in multiple temporal areas (PHG, H, EC, A), and visuomotor, mainly in the parahippocampal gyrus, and demonstrated their participation in higher levels of motor activity, driven by either internal input or external visual input. Interestingly, even this high-level participation encodes the detailed, low-level kinematics. Our findings elaborate the known functional organization of the motor control system to also include the temporal lobe and has far reaching implications on our understanding of the generation of the motor plan from visual or internal inputs, on the visual processing pathways, and on the functional role of the temporal lobe.
We thank the patients for participating in our study. We also thank R. Mukamel, N. Suthana, and E. Behnke for their assistance; E. Stark and M. Abeles for the code of the partial cross-correlation method; and S. Shoham for insightful comments on this manuscript. This research was supported by Grant 2497 of the Paralyzed Veterans of America Research Foundation, the Dana Foundation, NINDS, Lady Davis Fellowship, and the L&L Richmond research fund.
Reprint requests should be sent to Ariel Tankus, Faculty of Biomedical Engineering Technion, Israel Institute of Technology, Haifa, 32000 Israel, or via e-mail: email@example.com.