How does the brain represent the passage of time at the subsecond scale? Although different conceptual models for time perception have been proposed, its neurophysiological basis remains unknown. We took advantage of a visual duration illusion produced by stimulus novelty to link changes in cortical activity in monkeys with distortions of duration perception in humans. We found that human subjects perceived the duration of a subsecond motion pulse with a novel direction longer than a motion pulse with a repeated direction. Recording from monkeys viewing identical motion stimuli but performing a different behavioral task, we found that both the duration and amplitude of the neural response in the middle temporal area of visual cortex were positively correlated with the degree of novelty of the motion direction. In contrast to previous accounts that attribute distortions in duration perception to changes in the speed of a putative internal clock, our results suggest that the known adaptive properties of neural activity in visual cortex contributes to subsecond temporal distortions.
Understanding how the brain represents time is necessary for our broader understanding of cognition. Most of us can attest that time perception is fluid and is easily modulated by external inputs. For example, an unexpected or novel visual stimulus is generally perceived to have a longer duration than a regularly occurring stimulus. Studying how stimulus properties affect our duration perception may lead to insights into the neural mechanisms of time perception. In this study, we examined the hypothesis that adaptation of neural responses in the middle temporal (MT) area of visual cortex contributes to changes in the perceived duration of a motion stimulus.
The perception of subsecond duration is easily manipulated in the laboratory using temporal illusions (Grondin, 2001). For example, it has been shown that visual onset (Kanai & Watanabe, 2006), intensity (Goldstone, Lhamon, & Sechzer, 1978), motion (Brown, 1995), and attention (Mattes & Ulrich, 1998) expand subjective time. When subjects are presented with a series of repeated stimuli, the first stimulus in the series is perceived to have a longer duration (Rose & Summers, 1995). Similarly, a novel or oddball stimulus embedded in a train of repeated stimuli is also perceived to have a longer duration (Pariyadath & Eagleman, 2007; Tse, Intrilligator, Rivest, & Cavanagh, 2004). This effect of stimulus novelty on subsecond duration perception was the focus of this study.
Although clock models have served as a dominant and useful theoretical concept in time perception (Gibbon, 1977; Treisman, 1963; Creelman, 1961), the neural mechanisms underlying subsecond time perception are not well understood (Mauk & Buonomano, 2004). For subsecond duration expansion due to an oddball stimulus, it has been proposed that enhanced arousal (Ulrich, Nitschke, & Rammsayer, 2006) or attention capture (Tse et al., 2004) elicited by the oddball stimulus may cause the brain's clock to run faster, which leads to an overestimation of time. This hypothesis, however, neglects the contributions that cortical adaptation may have on the temporal dynamics of neural responses.
Neural adaptation has been observed throughout sensory cortex. Adaptation causes a suppression of neural activity during stimulus repetition (also referred to as repetition suppression) and produces a rebound in neural response to a novel stimulus. For example, a larger neural response to the oddball stimulus due to neural adaptation caused by repeated stimuli (Kohn, 2007; Ranganath & Rainer, 2003) may lead to bottom–up capture of attention leading to a faster running clock. Such a mechanism would require that the “neural” duration of the sensory representation in the brain remains constant for the oddball and repeated stimuli. Alternatively, the modulation of sensory responses because of adaptation could be directly responsible for the changes in subjective duration (Eagleman & Pariyadath, 2009; Eagleman, 2008; Pariyadath & Eagleman, 2007). We wanted to know what aspects of adaptation, such as changes in the magnitude or duration of the neural responses, were correlated with changes in subsecond duration perception.
Previous studies have shown that neural adaptation affects cortical neurons as a function of their preference for the adapting stimulus. For example, extracellular in vivo recordings in area MT of visual cortex have shown that adaptation is directionally tuned (Priebe & Lisberger, 2002). Thus, neural responses are higher after adaptation to stimuli in the nonpreferred direction compared with adaptation to stimuli in the preferred direction. This causes area MT to respond more vigorously to a novel motion stimulus than a repeated motion stimulus (Kohn & Movshon, 2003; Priebe, Churchland, & Lisberger, 2002; Priebe & Lisberger, 2002; Van Wezel & Britten, 2002; Petersen, Baker, & Allman, 1985). In fact, novelty detection has been proposed as one functional benefit of neural adaptation (Kohn, 2007; Ranganath & Rainer, 2003). However, to date, the neural mechanisms underlying distortions of subsecond duration perception have not been addressed using electrophysiological recordings. This technique has the necessary temporal resolution to observe potentially small adaptive changes in neural amplitude or duration that are linked to changes in duration perception.
We examined the link between neural adaptation and duration perception by asking if different levels of motion direction novelty produced different distortions in subjective duration perception in humans, and if so, whether these distortions were correlated with adaptive changes in the amplitude and duration of neural responses in area MT. We used a random dot motion stimulus and changed motion direction to precisely create three levels of stimulus novelty. Motion direction is an easily quantifiable parameter and changes in direction can be produced without changing other basic stimulus characteristics such as contrast, size, or shape. This was an important aspect of our experimental design, because it enabled us to observe how duration perception was affected by novelty without other confounding stimulus characteristics. Importantly, our stimulus targeted area MT, a brain area that plays a pivotal perceptual role in motion perception (Born & Bradley, 2005).
Our results revealed that human subjects robustly experienced both expansion (high novelty) and contraction (no novelty) of subsecond duration perception relative to a control condition with random motion directions. We also found that the magnitude of both the expansion and contraction of duration perception was greater when subjects expected the test stimulus. Recording from area MT in monkeys viewing the same motion stimuli but performing a different behavioral task, we found that both the amplitude and duration of neural activity in MT was correlated with the human duration distortions. The average neural modulation in MT, however, was generally less than the average duration modulation experienced by the human subjects. Modeling the integration of MT signals by downstream cortical areas produced changes in the duration of the neural responses that were closer in magnitude to that observed in our human subjects. Thus, we conclude that simple integration of adapting sensory responses in visual cortex contributes to distortions in duration perception.
Our experimental design used humans performing a visual duration discrimination task and electrophysiological recordings from the visual area MT in monkeys. Both the human and animal component of our experiments used similar visual stimuli. The monkeys preformed a different behavioral task that required them to attend to the visual stimulus.
Subjects reported the duration of the last motion pulse (referred to as the test pulse) in a random dot stimulus (Figure 1A). Subjects participated in either a “fixed” task, where the number of motion pulses presented was constant, or a “variable” task, where the number of motion pulses varied.
To study if an oddball stimulus created by changing the direction of motion affected our subjective perception of duration, a standard versus test duration comparison using the method of constant stimuli was used (Figure 1B). Trials began and ended with 600 msec of random motion. In each trial, the test pulse appeared after a series of standard pulses, each 200 msec in duration. In the variable task, a random number of standard pulses (between 1 and 15) were presented in each trial. In the fixed task, five standard pulses were shown on every trial. The duration of the last or test pulse was varied at seven levels. The same values were used for all subjects for all conditions, and no attempt was made to position the values around a subject's point of subjective equality (PSE) for a condition, because it could have biased the subjects. After the trial ended, the subjects indicated using a key press whether they perceived the duration of the last pulse to be longer or shorter than the duration of the preceding pulses observed in that trial. No feedback was provided.
In the variable task, trials were divided between short (1–5 pulse), medium (5–10 pulse), and long (10–15 pulse) pulse numbers, and each level of test stimulus strength was repeated once for each pulse number group (therefore, each level was repeated three times in each block). Three conditions were randomly interleaved in each block. In the “repeat” condition, all motion pulses had the same direction, randomly selected from a set of 12 directions at the beginning of a trial. In the “oddball” condition, the test pulse moved opposite to standard pulses, which all had the same direction again randomly selected at the beginning of a trial. In the “random” condition, each pulse's direction was randomly selected in any given trial, taking care that no two directions occur sequentially. The subjects, therefore, responded to (3 pulse number groups) × (7 stimulus levels) × (3 conditions) = 63 trials in each block. Each block took approximately 15 min to complete. The subjects performed between 20 and 50 blocks. The fixed task was similar to the variable task, but only five standard pulses were presented every trial.
Two of the authors (NGS and SA) and four naive subjects participated in the variable task. In the fixed task, seven naive subjects participated. All subjects had normal vision and provided consent forms according to guidelines by McGill University and Baylor College of Medicine.
Using Matlab's psychophysics toolbox (Pelli, 1997), 75% coherent motion pulses of 200 msec separated by either 400 msec (variable task) or 350 msec (fixed task) interpulse intervals of 0% coherent random motion were displayed (Figure 1A). The random dot stimulus was of the limited-lifetime type that has been used extensively in psychophysical (Baker, Hess, & Zihl, 1991) and electrophysiological (Britten, Shadlen, Newsome, & Movshon, 1992) studies. In brief, for each frame, the probability that a certain dot will be plotted along the vector of the coherent motion was equal to the coherence value. If the coherence was zero, all the dots were replotted randomly. If a dot exited the circular aperture, it was wrapped and displayed with the appropriate offset on the opposite side. The circular dot patch, 7 degrees in diameter, was placed approximately 6 degrees to the left of the fixation point. Dot density was 5 dots/degree2, dot size = 0.09 degree, and the speed of motion pulses was 16 degrees/sec. The stimulus was shown in a dark room on a CRT monitor with a refresh rate of 100 Hz. The subjects sat approximately 57 cm away from the monitor and were instructed to fixate on a central point during the course of a trial. Eye position was not measured.
The fraction of “longer” responses was plotted against the test pulse's duration. Data were fit with a sigmoidal curve 1/(1 + exp (− b × (t − a))), wherein a is the midpoint of the curve and b is the slope at the duration equal to a. Parameter a is the duration that produced “longer” responses 50% of the time, also known as the PSE, which was used as a measure of the subjective duration of the test pulse. A duration distortion factor (DDF) was used to represent the effects in humans. DDF is simply the ratio of the true physical duration of the standard pulses (200 msec) to the PSE of each condition. A DDF > 1 represents subjects experienced a subjective duration expansion, whereas a DDF < 1 represents a subjective duration contraction.
We recorded from MT area of two monkeys (male and female; Macaca mulatta) using standard extracellular techniques. The animals first underwent a surgery to implant a custom-made titanium headpost. A few months later, a craniotomy was made over MT area based on an anatomical MRI scan done before surgery. A dorsal recording chamber was implanted over the craniotomy using dental cement. All animal procedures and care were approved by the McGill University Animal Care Committee and followed guidelines set forth by the Canadian Council on Animal Care.
Stimulus and Behavioral Task
The visual stimulus was essentially the same as that viewed by the human subjects. Random dot kinematograms were used to make a series of motion pulses, 200 msec in duration and 80% coherent, separated by 350 msec of random motion. The stimulus was of the limited dot lifetime, as described above. The trials started and ended with 350 msec of random motion. The size and speed of the stimulus was matched to the preferences of the recorded neurons. Dot size was 0.11 degree and dot density was 15 dots/degree2. The experiment (including stimulus presentation) was run by custom software. The stimulus was shown in a dark room on a CRT monitor with a vertical refresh rate of 160 Hz. Eye position was tracked with a video camera using an EyeLink 1000 system (SR Research, Kanata, Ontario, Canada). Any time the monkey's eye position exited a defined area around the fixation point (radius of 1.5–2.5 degrees), the trial immediately ended.
The monkey, positioned in a primate chair 57 cm from the monitor and head-fixed to the chair via the implanted headpost, began a trial by fixating a central fixation point and pressing a lever. At a random time after 3.8 sec up to a maximum of 8.5 sec (derived from an exponential distribution), a longer interpulse interval, referred to as the signal, was presented (typical signal intervals were in the range of 475–780 msec). The animal was trained to detect the signal by releasing a lever in a response window of 800 msec after signal appearance to receive a few drops of juice as reward. The amount of reward increased linearly with time to encourage the monkey to wait for the signal. If he released the lever prematurely during a trial, no reward was given.
Three conditions similar to the human psychophysics were randomly interleaved. The objective was to compare the response of MT neurons to their preferred and null direction in different contexts that made the same stimulus more or less novel. In the “repeat” condition, the direction of all pulses was matched to the preferred or null direction of the isolated unit. In the “oddball” condition, one pulse in Positions 2–7 had the preferred (or null) direction, whereas the others moved in the null (or preferred) direction (the null direction being 180 degrees opposite preferred). In the “random” condition, each pulse had a direction randomly selected from a set of eight directions, with only one pulse in Positions 2–7 moving in the preferred or null direction.
Two percent of trials consisted of catch trials, wherein no signal occurred. This made the hazard function after 4 sec approximately flat. Random direction pulses (random condition) were presented for the catch trials. The catch trials were not analyzed. Therefore, on average, the number of times a preferred and a null direction 200-msec pulse was observed by the monkey was balanced, and there was no adaptation to a specific direction during an experiment.
In each session, a tungsten microelectrode (0.5–1 MΩ) was slowly lowered down into the brain via a guide tube positioned inside a coordinate grid over the craniotomy area. For multiunit recordings, signals were amplified (Bak model A-1, Brockton, MA) and bandpass filtered (Krohn-Hite model 3362, Brockton, MA) between 350 Hz and 4 kHz, before being fed into an audio amplifier and on-line dual-window spike discriminator (Bak model DDIS-1). The amplifier headstage (Bak model A-1) was modified to allow a high-pass cutoff of 5 Hz so that local field potentials (LFPs) could be analyzed off-line. The full electrode signal used for the LFP analysis was low-pass filtered at 8 kHz and sampled at 25 kHz (Krohn-Hite model 3384).
In each session of the experiment, the multiunit receptive field (RF) was mapped manually by moving a bar over the visual field. Next, a patch of random dots was placed on the coordinates of the RF with the approximate estimated size to get the initial direction tuning, then speed tuning, and again direction tuning (this time with the preferred speed). A site was selected if its response was higher for coherent motion than the preceding 0% coherent random motion and if it had clear directional tuning, with excitation to preferred direction and inhibition to null direction (compared with its response to 0% coherent motion). The parameters of the interval detection stimulus were then matched to the recording site's preferences, and the animal performed the behavioral task for as long as possible. A typical experimental session consisted of about 1500 trials, divided equally among our conditions. This provided between 20 and 100 trials for each test pulse of each condition.
Spike Rate Analysis
A total of 49 multiunit sites were recorded. Spike times were convolved with a Gaussian filter with a standard deviation of 10 msec to estimate average firing rates. Spike times were also convolved by an exponential filter with a time constant of 100 msec to mimic temporal integration by higher cortical areas. This integrated rate signal was mainly used for comparing the conditions. As a measure of response variability, trial-by-trial coefficient of variation (CV) was calculated for each time point as mean/standard deviation of the spike rate signal. For each condition, the standard deviation used was the trial-by-trial standard deviation of the spike rate for each time point. Because motion is inferred by higher areas using a population readout of MT activity, we used the preferred–null response difference (P–N response) to approximate that population activity.
Response Duration Analysis
To estimate the P–N durations, we determined the onset and offset of the trial-averaged P–N response for each condition by finding when the neural response was significantly (i.e., more than three standard deviations) different from the prepulse response. The threshold was derived using the oddball's standard deviation signal. To estimate the integrated durations, we used a slightly different method because noise levels were much less. In this case, the onset and offset of the trial-averaged integrated P–N response for each condition was calculated using the time to half height. The threshold was derived from the integrated repeat and oddball average rate, the half of the mean of activity of which during motion presentation was used to determine the threshold. The same threshold was used for all three conditions.
To study LFP power in different frequency bands, power spectra were estimated using a multitaper method (Mitra & Pesaran, 1999; Thomson, 1982). The tapers used were Slepian functions derived by Matlab's “dpss” command using a time bandwidth product (TW) of 2. Only the first three tapers were used. Slepian functions have their frequency content maximally centered around a certain frequency band (−10 to 10 Hz in our case), with the first 2TW-1 tapers being orthogonal to each other. Increasing the TW reduces the variability (noise) of the estimate but reduces the frequency resolution. We used the lowest TW to have maximum frequency resolution and reduced noise by averaging across trials. It is believed that the multitaper method reduces the spectral leakage problem of ordinary fast Fourier transform (FFT) in an optimal manner.
For each trial, power spectral density (PSD) was derived for 200-msec windows, moved in steps of 10 msec. The LFP signal of each window was individually multiplied (point by point) by the three tapers in time domain; FFT was performed on each tapered signal (length of 200, padded with a factor of 2) and then averaged (with a weight of one for each) across the three to get an estimate of the multitapered PSD for that window of that trial. The PSD for a condition was then derived by averaging the resultant spectrogram across trials. For each condition, average power for low (1–20 Hz), medium (20–50 Hz), high (50–100 Hz), and very high (100–150 Hz) frequency bands was derived for each condition by averaging the PSD across the respective frequency bands and in time.
Analysis Measures and Statistical Tests
Responses were pooled from test motion pulses in different trials and averaged to give an average signal for that condition. The mean values for response amplitude, duration, CV, and LFP power for the duration of coherent motion presentation (0–200 msec) was used for analysis. Magnitude and duration values were normalized by dividing the respective value of the random condition. This normalization was done to ensure that the effects we observed, on average, were not dominated by either preferred or null adaptation during the course of an experiment. The magnitude of adaptation (for all measures of neural activity) was quantified by calculating the percent change of oddball relative to repeat condition (oddball minus repeat, divided by repeat).
In our first set of experiments, humans viewed a random dot kinematogram that contained a series of 200 msec coherent motion pulses separated by random motion for 350 msec (Figure 1A). Subjects indicated whether the last motion pulse (test) was longer or shorter than the previous five pulses (standard). We refer to this as the “fixed” task, because subjects knew when the test pulse would occur. A “variable” task variant was also conducted to examine the effects of expectation (see below).
Figure 1B illustrates the three conditions used in both the fixed and variable tasks: same direction for all pulses (repeat, blue), opposite direction in the last pulse (oddball, red), and random standard pulse directions (random, green). All three conditions were randomly interleaved, and the test pulse duration was varied to produce psychometric curves. By using motion, we were able to produce novelty without changing other basic stimulus characteristics and, thus, target only the motion processing areas of visual cortex.
Sample psychometric functions from one subject performing the fixed task (Figure 1C) shows that, relative to the random condition (green), the test pulse duration was subjectively perceived to be longer when it was an oddball direction (leftward shift of red curve) and shorter when it was the same direction as the standard pulses (rightward shift of blue curve). We used the point of subjective equality (PSE) (arrows in Figure 1C) to quantify duration distortion. PSE represents the test pulse duration that produced a behavioral performance of 0.5 and was estimated from the sigmoidal fit of the data. In the random condition, the average PSE (green arrow) was close to the physical standard pulse width of 200 msec. We quantified the effects of the oddball and repeat conditions by computing a duration distraction factor (DDF) (DDF = 200/PSE), which expresses the relative change in the perceived duration of the test pulse from the actual physical duration of 200 msec. As illustrated in Figure 1C, a DDF = 1.4 represents a 40% expansion of the subject's perceived duration of the oddball test pulse, whereas a DDF = 0.7 represents the opposite, a 30% contraction of the subject's perceived duration of the repeated test pulse. A DDF = 1 indicates no duration distortion.
Across seven subjects, the oddball motion direction in the fixed task was perceived as longer (p = .009) and the repeat direction was reliably perceived as shorter (p = .002, paired sample t tests) compared with the random condition (Figure 1D and E, filled triangles). A DDF near 1 suggests that there was no appreciable duration distortion for the random condition (p = .68, t test). In addition, for almost all subjects, the discriminability (slope of psychometric curve) in the oddball condition was higher than that in the repeat condition (Figure 1F, filled triangles).
To examine the effects of expectation on duration distortion, we also ran a variable version of the task in which a random number of standard pulses (1–15) appeared before the test pulse (Figure 1D–F, six subjects, open circles). Note that the interpulse interval in the variable task was 50 msec longer than that used in the fixed task. Although the duration distortion was larger in the fixed versus variable task, the direction of the effect was the same (p = .026 for oddball and 0.039 for repeat, paired sample t tests). Importantly, neither task appreciably affected the veridical 200 msec duration estimate of the test pulse (DDF ∼ 1) during the random condition (Figure 1D, p = .17, t test, variable task). Discriminability, as measured by the slope of the psychometric function, also was better for the oddball condition in the variable task (Figure 1F, open circles).
What can be inferred about the internal representation of the 200-msec standard pulse the subjects used when estimating the duration of the test pulse? For example, did subjects use a single fixed representation of the 200-msec standard pulse in all conditions? Or did they base their duration estimates on the last few standard pulses that occurred just before the test pulse? These questions are difficult to answer conclusively using only the psychophysical results. One clue, however, is that the shifts of the psychometric functions during the repeat condition were usually greater than of the oddball condition (Figure 1D). This asymmetry suggests that the internal representation of the standard is closer to the oddball and farther from the repeat test pulse durations and would be unlikely if the internal representation of the 200-msec standard had been an average of the recently observed standard pulses. Regardless of the exact internal representation of the standard duration, however, the relative shift of the psychometric functions between the oddball and repeat conditions suggests that the test pulse representation had been different in these two conditions. Thus, we will focus on the oddball-repeat difference in test pulse representation when we compare our psychophysical and electrophysiological results below.
Our results suggest that the perceived duration of a novel motion direction is larger and less variable than the perceived duration of a repeated direction. Many neurophysiological studies have demonstrated that neural activity in area MT of visual cortex underlies the perception of random dot motion (Britten et al., 1992). This area has also been suggested to be involved in temporal processing in humans (Bosco, Carrozzo, & Lacquaniti, 2008; Bueti, Bahrami, & Walsh, 2008). Because it is possible that changes in the population state created by adaptation may have affected the temporal dynamics of the neural response, we wanted to know whether the modulation of neural activity in area MT was consistent with the duration distortion experienced by our human observers.
We recorded from 49 multiunit sites in area MT of two monkeys. The animals viewed a stimulus that was identical to that of the humans in the fixed task, and the size, speed, and direction of the stimulus was matched to the RF properties of each recording site. Because of the difficulties associated with training animals to reliably report subjective durations, the monkeys did not perform the same task as the humans. Instead, we trained them to report a small change (125–437 msec) in one of the interpulse intervals, which occurred at a random time after the test pulse. This design ensured the animals directed their attention to the dot patch and until the interpulse interval change occurred, the trials were the same as those viewed by the human subjects. The test pulse occurred at any time from the second to the seventh pulse. The same three stimulus conditions (oddball, random, and repeat) were used in the electrophysiology, with the exception that the test motion pulse was presented in both the preferred and null direction of the recording site. Thus, the balance of observed motion energy was zero, and neurons were not cumulatively adapted to either preferred or null directions during an experiment.
The trial-averaged response of an example recording site to preferred and null test pulses is shown in Figure 2A for the three conditions. The test pulse arrived on the seventh pulse in this example and shows that the neural activity in response to the preferred direction for the oddball test (red) was larger compared with the other two conditions. We mimicked the neural population response from two direction-selective pools in MT by computing the preferred minus null response (P–N, Figure 2B; Britten et al., 1992). Because firing rates differed among recording sites, the P–N neural responses were normalized to that of the random condition. Consistent with neural adaptation in visual cortex, across all our recordings, the magnitude of the P–N neural activity for the oddball test pulse was reliably greater than that of the random condition (Figure 2D, filled red circle, p < .0001, t test). For the repeat condition, P–N activity was less than the random condition (filled blue circle, p < .0001, t test).
To test the hypothesis that the duration of the neural representation varied across conditions, we compared the duration of the neural responses to that recorded during the random condition. We used a statistical approach (see Methods) to determine the onset and offset of the average P–N response using the same threshold for all conditions. We found that relative to the random condition, the duration for the oddball neural response was slightly longer (p < .0001) whereas the duration to the repeat was slightly shorter (Figure 2D, open triangles, p = .002, t tests).
We also mimicked the integrative properties of higher visual areas by convolving the P–N neural responses with a leaky integrator (100 msec time constant; Figure 2C). Such an integrative mechanism has been used in a variety of studies that link neural activity in visual cortex to perception (Cook & Maunsell, 2002; Hanes & Schall, 1996). Although we do not know the exact shape of the integrator used to accumulate MT activity, our exponential integrator was based on a recent study that reported an integration time constant between 60 and 175 msec used by monkeys when detecting the occurrence of short motion pulses (Masse & Cook, 2010). The integrated P–N responses were consistently longer (p < .0001) for the oddball motion and shorter (p = .0001, t tests) for the repeat condition (Figure 2D and E, open circles). Thus, a higher brain area integrating a population of MT neurons would register longer response durations for novel compared with repeated motion directions. Note that the average oddball-repeat difference for the duration of the integrated response approached average duration distortions of the humans in the variable task but were still much less than that observed for humans in the fixed task (compare the human behavioral and monkey neural oddball-repeat difference in Figures 1D and 2D).
Our human experiments found that the slope of the psychometric function was usually larger for the oddball condition than for the repeat condition (Figure 1F). We wanted to know if a similar effect was observable in our neural recordings. Thus, we computed the CV as a normalized measure of trial-to-trial variability of the neural responses to the preferred direction only. As the scatter plot and marginal histogram in Figure 2F shows, there was significantly less variability for the oddball compared with repeated motion pulses across our sites (p < .0001, paired sample t test). Therefore, the variability of psychophysical and electrophysiological effects shared the same trend across our conditions.
It is known that the effects of motion adaptation become stronger the longer an adapting stimulus is shown. If adaption of the neural representation in area MT contributed to the distortion of duration estimates at the behavioral level, then we should observe similar trends in the time course in both the psychophysics and neurophysiology as a function of when the test pulse occurred. Using the human behavioral data, we calculated the time course of the DDF for each condition in the variable task by computing the psychometric function curves using a sliding window of four standard pulses (Figure 3A and B). For the electrophysiology, the integrated P–N response duration was calculated for each condition as a function of when the test pulse occurred (Pulses 2–7). In this analysis, we did not normalize to the random condition. Instead, raw spikes were convolved with the exponential integrator (time constant = 100 msec). To allow a direct comparison between DDF and the monkey neural data, we normalized the neural durations by 200 msec to produce an equivalent “neural DDF” (Figure 3C and D). Both measures are essentially equivalent and express the behavioral or neural durations of the test pulse relative to the 200-msec standard pulse.
Comparing Figure 3A and C shows that both the behavioral and neural durations of the test pulse expanded and contracted gradually for the oddball and repeat conditions, respectively, as a function of when the test pulse occurred. However, there was much less change for either during the random condition. Importantly, the relative change between the oddball and repeat condition increased steadily for both measures (compare Figure 3B and D). These results suggest that both behavioral and neural effects increase in time with comparable trends, which strengthens the link between our psychophysical and electrophysiological observations.
There are two important differences between the behavioral and neurophysiological data in Figure 3 that should be noted. First, the duration of the P–N integrated neural responses was, for the most part, greater than 200 msec (neural duration/200 > 1) compared with the duration estimates by humans (DDF < 1). We did not optimize the threshold used for detecting onset and offset of neural response to center durations around 200 msec. This is because the value of the test pulse's duration does not affect our analysis since it is only the relative change between conditions that is comparable for psychophysics and electrophysiology. The human DDFs are the result of a comparison with an unknown standard; hence, the same DDFs could be obtained as long as the relative relationship between the test and standard is maintained.
A second notable difference in the data in Figure 3 is that the rate of change between the oddball and repeat neural durations increased slightly faster than the comparable changes in human DDF. Potential explanations include a 50-msec shorter interpulse interval in the monkey's task that may increase neural adaptation or task differences that may affect the relative changes in duration estimates. An example of how the task can affect the strength of the duration distortion is illustrated by the fixed versus variable task effects on the human duration estimates in Figure 1.
Although the source of LFPs is not fully understood, one generally accepted proposition is that low-frequency LFPs reflect the synaptic input and intracortical processing of a cortical area (Logothetis, 2003). Thus, we wanted to know if the test pulse LFPs we recorded differed between our oddball and repeat conditions. Figure 4A shows the multitapered PSD estimates for the example recording in Figure 2A for the repeat and oddball conditions in response to a preferred direction test pulse. This particular recording site had increased power in the high-frequency ranges for the oddball versus the repeat conditions and a slightly reduced power in the low frequencies in the oddball compared with repeat condition.
Figure 4B shows the oddball-repeat percent change of power for different frequencies in response to a preferred test pulse, averaged across all our sites. This analysis shows opposite directions of change for frequencies above and below 80 Hz in response to the test pulse. The histograms in Figure 4C show the distribution of the effect across all sites, averaged for different frequency bands. Overall, these results show that power in very high frequencies (above about 100 Hz) was larger in the oddball and power in lower frequencies (about 20–50 Hz) was larger in the repeat condition. Thus, power in higher-frequency bands was correlated with the degree of test pulse novelty and power in lower-frequency bands was anticorrelated with novelty. Although we had measurable LFP power in our different conditions, it is important to note that our electrode impedance (∼500 kΩ) and high-pass cutoff (5 Hz) did not necessarily represent ideal conditions and some LFP components may not have been captured.
Our LFP results suggest that correlations between duration perception and neural activity arose for the first time in area MT. Care must be exercised in this interpretation, however, because the source of fluctuations in LFPs is not fully understood. The LFPs derived from the same electrode that is used to sort spikes can be considered only as an approximation of the true low-frequency fluctuations of a mass of neurons around the electrode tip. Similar results have previously been observed for LFP behavioral correlations (Liu & Newsome, 2006) and the effects of attention on LFPs (Khayat, Niebergall, & Martinez-Trujillo, 2010), namely, that changes in power in the higher and lower frequency bands is correlated and anticorrelated, respectively, with changes in the spike rate. However, the effects of adaptation on LFPs has not been studied before, and a systematic approach is needed to characterize our findings more clearly.
We examined the link between adaptation of neural responses in visual cortex and distortions in duration perception. We observed both expansion and contraction of human duration perception using a repeated motion stimulus. Although our subjects' estimate of duration was veridical when viewing random direction motion, repeated motion directions produced estimates of duration that were shorter than the physical duration whereas a novel motion direction (oddball) produced estimates of duration that were longer than the physical duration. The magnitude of this duration distortion was greater when the subjects expected the test pulse (fixed task) compared with a condition where the subjects did not know when the test pulse would occur (variable task). In both tasks, discriminability was better for the novel motion direction compared with the repeated motion direction.
Neural recordings from area MT while monkeys viewed the same stimulus revealed modulations that were correlated with the changes in human duration perception. However, the amplitude of the modulations, especially the duration of the neural responses, was much smaller than the behavioral effects observed in our human subjects. Analogous to the effects on human discriminability, neural responses to the novel motion direction were less variable compared with the repeated motion direction. Analysis of LFPs in the neural response suggested that these modulations were not inherited from preceding visual areas. As would be expected from the known properties of sensory adaptation, the magnitude of the neural modulation in MT increased when more repeated motion pulses were presented and was comparable to the effects of the number of repeated motion pulses on human duration estimates.
How might we reconcile the relatively small modulations of MT activity due to motion direction novelty compared with the larger effects on human perception? First, there might also be increased adaptation in higher visual areas that would potentially increase the neural modulations between conditions. In addition, task differences (duration discrimination versus interval change detection) could have affected the amount of modulation between the human behavior and animal neurophysiology. Finally, based on the large difference in DDF between the two human tasks (fixed versus variable), it is possible that a lack of expectation of the test pulse by the monkeys reduced neural modulations.
A key aspect of our neurophysiological analysis is the use of a leaky integrator to mimic the known integrative properties of higher visual areas. It is well appreciated that simple integration of neural activity converts increased magnitude into longer durations. Using a longer (or shorter) leaky integrator will produce greater (or less) modulation of duration of our neural data (data not shown). Although we do not know how MT activity is integrated, the leaky integrator used here is consistent with models that account for sensory to motor transformations of neural activity (Masse & Cook, 2010; Cook & Maunsell, 2002; Hanes & Schall, 1996).
Our results suggest that stimulus-induced changes in neural response duration may underlie perceived duration illusions. In this regard, if novelty is created by changing low-level stimulus characteristics that may independently affect subjective durations, then it can be predicted that temporal illusions may depend not only on novelty but also on other stimulus-induced temporal distortions. For example, a static disk presented after a series of repeated looming disks has been perceived to be shorter (van Wassenhove, Buonomano, Shimojo, & Shams, 2008), whereas an expanding disk presented after static ones has been perceived to be longer than the standards (Tse et al., 2004). These findings suggest that other factors in addition to adaptation can affect subjective durations and supports the hypothesis that the duration of the neural representation of the stimulus, affected by different stimulus factors, may contribute to temporal illusions (Schindel, Rowlands, & Arnold, 2011; Eagleman & Pariyadath, 2009).
Although at first glance it may seem that our approach confounds response duration by response magnitude, it has been shown that the two are often correlated for a single cortical area as a simple result of the functional characteristics of neurons (Raiguel, Xiao, Marcar, & Orban, 1999). One reason may be that the cellular mechanisms responsible for the increase in firing rate (such as depolarization, shorter postsynaptic integration time, higher convergence, and higher input) can also cause the membrane potential to pass the spike threshold sooner. Further disambiguation of the two may be possible by studying intermodal temporal illusions in two separate cortical areas that have a different native firing rate. In this manner, a change in neural duration becomes independent of response magnitude.
Another approach to more tightly link neural activity in visual cortex to duration perception would be to have the animals perform a duration discrimination task while recording the neural activity. Trial-by-trial covariation of behavioral choice and neural activity at threshold behavioral conditions has been taken as an indirect evidence for a causal perceptual role of the sensory area under study (Parker & Newsome, 1998, but see Nienborg & Cumming, 2009). In such paradigms, the subject's response is directly related to a stimulus characteristic, such as discriminating the direction of coherent random dot motion.
In our paradigm, however, the subject's behavior is not solely based on the perceived duration of the test pulse but depends on a comparison with an internal representation of the standard pulse duration. Possible changes in the internal representation of the standard could introduce an additional source of variability that affects the correlation of the test pulse's neural activity with behavior in unpredictable manners. A two-alternative forced-choice duration discrimination paradigm employing other types of stimulus-induced temporal illusions that do not involve having an unknown standard would be a more reasonable choice for such analysis. However, because different aspects of neural activity including response duration and amplitude may be correlated with each other, they will also be correlated with behavior if only one of them is causally linked to perception (a third variable problem). For these reasons, we did not train the monkeys to perform a duration estimation task for this initial set of experiments. However, this is an important future direction to better understand the contribution of sensory areas of the cortex to distortions in duration perception.
In conclusion, our results suggest that depending on the integrative properties of the brain, adaptation of sensory representations in visual cortex contributes to the subsecond modulations of duration perception of a visual stimulus (but see Schindel, Rowlands, & Arnold, 2011). Thus, changes in clock speed brought about by stimulus novelty are not necessarily required to explain duration distortions. A change in the duration of sensory responses is a particularly attractive mechanism because it provides the most parsimonious explanation of the duration distortion; novel stimuli are perceived to have longer durations because their integrated neural representations are themselves longer (Eagleman & Pariyadath, 2009; Eagleman, 2008; Pariyadath & Eagleman, 2007). This is analogous to the effects of attention on neural responses and behavior: Attending to visual stimuli makes those stimuli more behaviorally salient because their neural representations in visual cortex are themselves enhanced (McAdams & Maunsell, 1999; Desimone & Duncan, 1995).
This work was funded by the Canadian Institutes of Health Research Operating Grants (E. P. C.) and the EJLB Foundation Scholar Research Program (E. P. C.). We would like to thank Chang'an Zhan for useful discussions on the initial psychophysical studies, Mario Ruiz-Ruiz for help in animal experiments, and Walter Kucharski and Stephen Nuara for technical support.
Reprint requests should be sent to Navid G. Sadeghi, Department of Physiology, McGill University, 3655 Promenade Sir William Osler, Montreal, QC H3G 1Y6, Canada, or via e-mail: firstname.lastname@example.org.