An important class of psychological models of decision making assumes that evidence is accumulated by a diffusion process to a response criterion. These models have successfully accounted for reaction time (RT) distributions and choice probabilities from a wide variety of experimental tasks. An outstanding theoretical problem is how the integration process that underlies diffusive evidence accumulation can be realized neurally. Wang (2001, 2002) has suggested that long timescale neural integration may be implemented by persistent activity in reverberation loops. We analyze a simple recurrent decision making architecture and show that it leads to a diffusive accumulation process. The process has the form of a time-inhomogeneous Ornstein-Uhlenbeck velocity process with linearly increasing drift and diffusion coefficients. The resulting model predicts RT distributions and choice probabilities that closely approximate those found in behavioral data.
A growing body of research from the past two decades has used single-cell recording techniques to investigate decision making in awake, behaving monkeys. Many of these studies have used saccade-to-target decision tasks that are analogs of the kinds of simple decision tasks studied in the human choice reaction time (RT) literature. In these tasks, a monkey makes a saccadic eye movement to one of two targets on a display screen to indicate a decision. The aim of these studies has been to link neural firing rates in decision-related brain structures to behavioral measures of decision making, like choice probability and RT. These studies have found evidence for a view of decision making that is in good agreement with the sequential-sampling models that have been studied in cognitive and mathematical psychology for the past 50 years (Luce, 1986). They therefore offer the prospect of a direct theoretical link between the psychology and the neurobiology of choice and decision making. Our aim in this letter is to contribute to our understanding of this link.
Like other statistical decision models, sequential-sampling models assume that the evidence on which decision processes operate is inherently noisy because of moment-to-moment stochastic variability in the neural populations that encode it. Decisions are made by accumulating successive samples of noisy evidence until a criterion quantity of evidence needed for a decision is reached. This results in an improvement in the signal-to-noise ratio because moment-to-moment fluctuations in evidence due to noise are averaged out. The time needed for the accumulating evidence total to reach criterion determines the RT, and the particular criterion that is reached first determines the choice that is made. These models provide a good account of the main behavioral measures in simple, speeded decision tasks (Ratcliff & Smith, 2004) and, under the assumptions that neural firing rates are a correlate of accumulating evidence, predict activity in decision-related brain structures in the interval prior to the overt response. A review of this literature may be found in Smith and Ratcliff (2004); recent examples are the papers by Purcell et al. (2010) and Ratcliff, Hasegawa, Hasegawa, Smith, and Segraves (2007).
Of the various models proposed in the psychological literature for two-choice decision making, one of the most successful is the diffusion model of Ratcliff (1978; see Ratcliff & McKoon, 2008, for a tutorial review). This model represents decision making as diffusive information accumulation between absorbing barriers. The process of evidence accumulation is modeled as a Wiener diffusion process, or Brownian motion with drift, and the absorbing barriers represent the amount of evidence needed for a decision. Figure 1 summarizes the main features of the model.
Presentation of a stimulus initiates a process of noisy evidence accumulation, depicted in the figure by the irregular lines (the so-called sample paths of the process). Presentation of one stimulus, s1, causes the process to drift in an upward direction; presentation of the other stimulus, s2, causes it to drift in a downward direction, with the magnitude of the drift depending on the discriminability of the stimulus alternatives. Accumulation continues until the process reaches one of two absorbing barriers, located at a1 and a2, which represent the evidence criteria for the two responses. If the first criterion reached is a1, response R1 is made; if the first criterion reached is a2, response R2 is made. The RT on any trial is the time required to first reach one of the barriers.
The irregularity of the sample paths is a reflection of the cumulative effects of noise, assumed to be broad-spectrum gaussian noise, or white noise. Because the accumulation process is noisy, the time required for it to reach an absorbing barrier will differ from one presentation of a stimulus to the next. On some trials, the cumulative effects of noise will be such that the first barrier crossed by the process will be the wrong one, and an error will result. These features of the model allow it to predict trial-to-trial variation in RT and the occurrence of errors. Mathematically, the predicted distributions of RT for correct responses and errors are given by the first-passage-time distributions of a Wiener diffusion process through the absorbing boundaries. The predicted probabilities of the two responses are the associated first-passage-time probabilities (Gardiner, 1985; Karlin & Taylor, 1981). One of the attractive features of diffusion models is that they naturally predict the shapes of the families of RT distributions for correct responses and errors that are found empirically, together with the associated choice probabilities (Ratcliff & McKoon, 2008; Ratcliff & Smith, 2004).
The diffusion model and its variants are psychological models, developed to account for behavioral measures of choice probability and RT. However, the recent success in linking diffusion models of evidence accumulation to neural firing rates empirically (Ditterich, 2006; Gold & Shadlen, 2001; Hanes & Schall, 1996; Huk & Shadlen, 2005; Purcell et al., 2010; Ratcliff, Cherian, & Segraves, 2003; Ratcliff et al., 2007; Schall & Hanes, 1998) encourages us to try to characterize the link theoretically and mathematically in a more precise way. Our goal in this letter is to elucidate this link. The research we report builds on, and attempts to unify, two recent developments in models of decision making. The first is a recent article by Smith (2010a) that links diffusive evidence accumulation mathematically to a Poisson shot noise model of the flux in postsynaptic potentials induced by a sequence of action potentials. The second is a series of articles by Wang and colleagues modeling long-range neural integration processes using reverberation loops in networks of spiking neurons (Lee & Wang, 2009; Lo & Wang, 2006; Wang, 2001, 2002; Wong & Wang, 2006).
Wang's work addresses the important question of how neural activity can be integrated over the relatively long time intervals (on the order of 1 second or so) needed to support decision making behaviorally. Wang has pointed out that the integration processes found at the cellular level operate on too short a timescale to account for the kind of information integration assumed in behavioral models. He has proposed that long timescale neural integration could be carried out by reverberation loops in neural circuits. He has shown that a network of spiking neurons consisting of a pair of neural populations that implement reverberation loops can simulate many of the statistical properties of speeded decision making observed empirically. In this letter, we link the analysis of behavioral-level diffusion that Smith made to the kinds of recurrent neural architectures that Wang proposed. In so doing, we seek to shed light on how neural systems may implement the kind of diffusive information accumulation that has been shown to provide a good account of decision making experimentally.
Whereas Wang has investigated decision-making dynamics in a fairly large and complex network, our approach is different from, and complementary to, his. Our goal is to show that the main features of diffusive information accumulation that are found behaviorally are also exhibited by the simplest recurrent loop architectures. Unlike Wang, our aim is not to link decision-making dynamics to the properties of any particular network architecture but, rather, to show the robustness and generality of recurrence as a computational principle for long-range neural integration. We seek to do this in a way that is independent of the implementation details of any particular model or architecture. Because we investigate the statistical properties of information accumulation in very simple systems, which are characterized by the fewest possible assumptions, we refer to the resulting models as minimal recurrent architectures or minimal Wang architectures.
2. Neural Basis of Diffusive Information Accumulation
2.1. Theoretical Results
Ratcliff and Smith (2004) compared the performance of several sequential-sampling decision models on three sets of benchmark behavioral data. They found that models that represent evidence accumulation as a diffusion process provided a better account of RT distributions and accuracy data than do models that represented evidence accumulation using other sorts of stochastic process. This was true regardless of the decision rule implemented by the model architecture. They obtained similarly good fits for the model of Figure 1, which assumes a single diffusive accumulation process, and a model that assumes a pair of independent, racing diffusion processes (Smith, 2000; Ratcliff et al., 2007), and the model of Usher and McClelland (2001), which assumes a pair of mutually inhibitory, racing diffusion processes. What these models have in common, and what distinguishes them from less successful models, is their assumption of diffusive information accumulation.
In a similar vein, Purcell et al. (2010) have recently shown that diffusive accumulation models can predict both RT and neural activity in frontal eye field (FEF) motor neurons from activity in FEF visual neurons. The success of these various applications of diffusion process models begs the question of, Why diffusion? The fundamental diffusion process, from which other kinds of diffusion can be derived, is the Wiener, or Brownian motion, process, Wt. The process Wt is a continuous-time, continuous-state stochastic process whose sample paths are (almost everywhere) nondifferentiable. Its mean, E[Wt], and its variance, var(Wt), both increase linearly with time: E[Wt] = μt and var(Wt) = σ2t, where μ and σ2 are the drift and diffusion coefficient of the process respectively, respectively. Smith (2010a) showed how diffusive information could arise at a macro, or behavioral, level in populations of neurons in which activity is modeled as a Poisson shot noise process. The class of models that Smith analyzed is shown in Figure 2.
In these models, stimulus information is coded by opponent processes, represented by pairs of Poisson processes—one excitatory and one inhibitory. The excitatory process has a rate, or intensity, of λ1(t), and the inhibitory process has a rate of λ2(t). The time dependency in the notation allows the possibility that these processes may be time inhomogeneous, a property that may be important in applications (Smith & Ratcliff, 2009), but in this letter, we confine our attention to the simpler time-homogeneous case, λ1(t) ≡ λ1 and λ2(t) ≡ λ2 (constant). Each arriving Poisson spike creates a random disturbance or perturbation in the underlying neural population. We assume that each of these perturbations has an instantaneous random amplitude, Z, that decays exponentially with rate α. The random variable Z is assumed to be positively distributed with finite fourth moment.1 The excitatory and inhibitory shot noise processes, denoted Et and It, respectively, cumulate the successive local perturbations. These processes have a sawtooth appearance, as shown in Figure 2. Although this is not the most general shot noise process, it is a theoretically interesting one because it leads to an Ornstein-Uhlenbeck (OU) diffusion approximation. In the models that Smith (2010a) analyzed, stimulus information is coded by a process Xt, which is the difference between the excitatory and inhibitory processes, Xt = Et − It, as shown in Figure 2. We refer to the difference process as a Poisson shot noise pair.
Smith (2010a) showed how a shot noise representation of stimulus encoding leads to diffusive information accumulation at the behavioral level, using a two-step mathematical approximation. The steps in this approximation are shown in Figure 3. At high intensities of the Poisson processes, the shot noise pair converges to an OU velocity process, denoted Vt. This process represents the instantaneous evidence state at time t. Successive states are accumulated, or integrated, over time to make a decision. With this identification, accumulating stimulus information is represented by an integrated OU process, or OU displacement process, Ut. The macro, or long timescale, statistics of the OU displacement process approximate those of a Wiener process. This is the process that is used to model behavioral data. Smith's argument was that diffusive evidence accumulation, assumed in Ratcliff's (1978) model to be represented by a Wiener process, may be better viewed as an integrated OU process, whose properties are indistinguishable from the Wiener process at the level of resolution of behavioral data.
The model that Smith (2010a) analyzed is a statistical decision model whose properties were purposely not tied to the properties of any particular neural network or architecture. It is neurally plausible to the extent that the Poisson processes can be viewed as sequences of action potentials and the associated perturbations can be viewed as the flux in the postsynaptic potential induced by synapses impinging on the body of a neuron. In the neural modeling literature, this model is known as the Stein model (Gerstner & Kistler, 2002; Stein, 1965; Tuckwell, 1988) and provides an idealized model of a leaky integrate-and-fire neuron. In the Stein model, the first-passage-time distribution of the process through an absorbing barrier describes the distribution of firing times of the cell. In the psychological model, it represents the distribution of times of the accumulating evidence state through a response criterion, which gives the predicted distribution of RT.
The psychological model assumes that information is integrated over time to make a decision. In typical RT applications, the integration times range from a few hundred milliseconds to a second or more. However, as noted in section 1, Wang has pointed out that these times are much longer (by an order of magnitude) than any integration process that is found at a cellular level (Lee & Wang, 2009; Lo & Wang, 2006; Wang, 2001, 2002). Wang and colleagues have proposed that such integration might be carried out by reverberation loops and have investigated a model network of spiking neurons that implement such loops. His model network exhibits many of the properties that are found behaviorally and in single-cell recordings from decision-related brain structures. Our goal in this analysis is to provide a statistical analysis of the simplest form of recurrent decision architectures, to extend the analysis of Smith (2010a) and complement that of Wang.
The model that Smith (2010a) analyzed is related to a neural model that Mazurek, Roitman, Ditterich, and Shadlen (2003) proposed, in which sensory signals that are coded as filtered point processes are integrated over time to make a decision. Like Smith, Mazurek et al. assumed neural integration as a theoretical primitive of the model and did not consider how integration on the timescale involved in decision making might be realized neurally. Our analysis of recurrent decision architectures attempts to shed light on the computational and statistical properties of recurrence as a mechanism for neural integration. Bogacz, Brown, Moehlis, Holmes, and Cohen (2006) noted the similarities between Wang's recurrent network neural model and the psychological model of Usher and McClelland (2001), which assumes mutually inhibitory, racing diffusion processes. Our primary focus here is not on these global features of model architecture (i.e., number of evidence accumulators or presence of inhibition) but, rather, on the link between recurrence and diffusive accumulation, as the latter appears to be critical to explaining behavioral data (Ratcliff & Smith, 2004).
2.2. The Recurrent Loop Model
Our recurrent loop model comprises two different kinds of integration process operating on two different timescales. One is an intracellular process, which operates on a short timescale that is set by the value of the membrane potential decay constant. We follow Wang (2001, 2002) and assume that this process has an upper bound of 50 ms. The other is an intercellular (i.e., network) process, which operates on the long timescale (around 1 second or so) of behavioral decision making. The intracellular process is amplitude coded; information is carried by the difference between the cumulative sum of excitatory and inhibitory postsynaptic potentials. The intercellular process is frequency coded; information is carried by the density of action potentials in spike trains. Unlike psychological models—like the diffusion model (Ratcliff, 1978), the dual-diffusion model (Ratcliff & Smith, 2004), the Ornstein-Uhlenbeck model (Busemeyer & Townsend, 1993), the Usher and McClelland (2001) model, and the integrated Ornstein-Uhlenbeck model (Smith, 2010a)—we explicitly prohibited long timescale, algebraic integration of the amplitude-coded postsynaptic process as a design feature of the model. We imposed this constraint in order to show that we could recover a form of diffusive information accumulation, even in the absence of any form of long timescale stochastic integration of the kind assumed by psychological, and many neurobiological, models. Our ability to do this is the key theoretical result in this letter.
In our model, long timescale, diffusive information accumulation is not assumed as a theoretical primitive of the model. Rather, it emerges from the interaction between the short timescale intracellular process and the long timescale intercellular process. The short timescale intracellular process, although it is formally similar to the stochastic accumulation processes assumed in psychological models and analyzed in detail by Smith (2010a), is limited in its range by the value of the membrane decay constant, and consequently cannot provide the kind of extended information integration needed to make a decision. Instead, it operates more like a threshold logic device, signaling when the local difference in excitatory and inhibitory spike densities exceeds a threshold. This process carries out the frequency-to-amplitude-to-frequency recoding needed for interneuron signaling.
The general form of the model is shown in Figure 4. The evidence for each stimulus alternative is coded as a shot noise pair. Evidence for s1 is coded by the pair λ1 and λ2, and evidence for s2 is coded by the pair λ3 and λ4 (all now assumed time homogeneous). In the models that Smith (2010a) analyzed, these processes are integrated over time to make a decision. Here we assume instead that they form the inputs to sets of leaky integrate-and-fire neurons. These neurons cumulate values of the shot noise process to a firing threshold, emit a spike, and are then instantly reset to 0. Mathematically, the firing time is a random variable, T, defined as , where Xt is the input shot noise pair and θ is the firing threshold. The distribution of T can be obtained from an expression in Gerstner and Kistler (2002) for the Ornstein-Uhlenbeck approximation to a leaky integrate-and-fire neuron. The moments of the firing time distribution can be obtained from an expression given by Wan and Tuckwell (1982). In our implementation of the model, we obtained sequences of firing times directly by applying a threshold to simulated pairs of shot noise processes, as shown in Figure 4.
As shown in Figure 4, there are two kinds of leaky integrate-and-fire neurons in the model. Those in the boxes on the left initiate activity in the recurrent loops; those in the boxes on the right signal that a decision has been made. We focus first on the neurons that initiate loop activity.
The output of an integrate-and-fire neuron is a sequence of spikes at random times, T1, T2, T3, … Mathematically, this sequence forms a renewal process in which the emission of a spike by a neuron is a renewal event. These spikes are then cumulated in the recurrent loops shown on the right of the figure. The assumptions we make about the loop are the simplest ones possible. We assume that a spike entering the loop at time Ti initiates a process that leads to another spike being registered at the beginning of the loop after a further random time, ξ(1)i. This initiates a similar process that results in a further spike at time ξ(2)i, and so on. The effect of the initial spike at Ti is thus to initiate a nonterminating process of spike generation that produces spikes at times Ti, Ti + ξ(1)i, Ti + ξ(2)i, … There are therefore two kinds of spikes in the loop at any time: input spikes, which are generated by the integrate-and-fire process, and autonomously generated spikes, which preserve the input activity in recurrent form. These spikes occur at times Ti + ξ(j)i, i = 1, 2, …, j = 0, 1, 2, …, where we define ξ(0)i = 0.
The assumption that any spike in the loop generates a new spike after a random time with probability one ensures the activity within the loop is persistent. The alternative assumption, that spikes within the loop generate new spikes with probability less than one, would endow the loop with leakage. Authors such as Smith (1995) and Usher and McClelland (2001) have argued for leakage on the grounds of biological plausibility, but Ratcliff and Smith (2004) found no evidence that model fit was improved by the inclusion of leakage. For this reason, in this letter we have confined our attention to loops that act as perfect (i.e., leak-free) integrators.2 Note that leakage within a loop is functionally distinct from leakage in the membrane potential in the integrate-and-fire neurons that initiate loop activity. Leakage in the membrane potential limits the integration time of a neuron; leakage in a loop means that the accumulating evidence state converges to a stationary distribution, which sets the limits of information accumulation. We assume leakage of the former, but not the latter, kind.
To analyze the properties of this model, we assume that the distribution of the random variables ξ(j)i is exponential with rate δ. That is, the mean time for a spike to travel around the loop is 1/δ. As we discuss in relation to our simulation study subsequently, the exponential assumption is not critical to the performance of the model, but it is convenient, as it allows us to obtain a simple analytic characterization of the mean firing rate, which would be much less tractable in the general case. The assumption of exponentiality is motivated by the fact that the model is a population model, in which a loop is an aggregation of multiple, similar neural pathways. A basic theorem in renewal theory states that the superposition of independent renewal processes approaches a Poisson process as the number of processes becomes large, regardless of the distribution of the individual renewal events (Cox & Miller, 1965). This theorem would apply if the recurrent activity is carried by multiple, parallel, independent pathways. In the same spirit, we assume that the two loops in Figure 4 are each driven by a population of independent integrate-and-fire neurons. The sequence of input events in a loop, Ti, i = 1, 2, 3, … is then the superposition of the spikes generated by its associated population of neurons. Although the distribution of interspike intervals of an individual integrate-and-fire neuron is not exponential, the superposition of these processes approximates an exponential distribution fairly closely with even a small number of processes. We denote the rates of the superposition input processes by γk, k = 1, 2. The subscript k expresses the fact that the firing rates in the two loops, which reflect the strength of evidence for the two stimulus alternatives, will in general be different.
As shown in Figure 4, the evidence for the two responses, R1 and R2, is coded in frequency form in separate reverberation loops. We found in our simulation study that we needed a two-loop model to reproduce the RT distributions that are found experimentally. We were not able to reproduce these with any form of single-loop model we studied. To combine the activity in the two loops into a neurally plausible decision model, we compute two signed difference processes, as shown on the right of Figure 4. Like the integrate-and-fire neurons on the left side of Figure 4, the difference processes are exponential shot noise pairs with decay rate α.
To combine the activity in the loops into a decision model, we adopt the neuron-antineuron coding scheme of Gold and Shadlen (2001; see also Mazurek et al., 2003). If Γ1(t) denotes the activity in the upper loop and Γ2(t) denotes the activity in the lower loop, the process coding evidence for response R1 has excitatory rate Γ1(t) and inhibitory rate Γ2(t). The process coding evidence for response R2 has excitatory rate Γ2(t) and inhibitory rate Γ1(t). That is, the two processes are additive inverses of each other. In the absence of barriers or thresholds, this leads to two perfectly anticorrelated shot noise pairs—one with input Γ1(t) − Γ2(t) and one with input Γ2(t) − Γ1(t). We further constrain these processes both to be positive (cf. Usher & McClelland, 2001); if the inputs are such that a process becomes negative, it is truncated at zero. (In the diffusion process limit, this is equivalent to constraining the processes with reflecting barriers at zero.) The shot noise pairs cumulate the signed differences in the firing rates in the loops. Response R1 is made if the Γ1(t) − Γ2(t) process exceeds a threshold or criterion a1; response R2 is made if the Γ2(t) − Γ1(t) process exceeds a criterion a2. Like the integrate-and-fire neurons on the left of Figure 4, which initiate persistent activity in the recurrent loops, the integration carried out by the decision neurons is restricted to a short timescale set by the value of the decay constant, α. These neurons respond when the signed difference in the activity in the two loops, which represents the accumulated evidence at a given time, exceeds a threshold value. They are not themselves able to accumulate evidence over the long times needed for a decision.
We developed the decision model in this way because of its equivalence to the Ratcliff (1978) model in Figure 1. Because the two processes are anticorrelated, discriminative information is carried effectively by a single, signed decision variable, as it is in the Ratcliff model. The positive part of the decision variable is carried by the upper process; the negative part is carried by the lower process. Truncation at zero means the anticorrelation is approximate rather than exact, but the decision process nevertheless behaves much like a single, signed diffusion process. Because the processes are both constrained to be positive, the decision model is not vulnerable to the criticism sometimes made of the Ratcliff model: that its assumption of a signed decision variable is neurally implausible because firing rates code unsigned quantities only. Indeed, it was to circumvent this problem that Gold and Shadlen (2001) developed their neuron-antineuron coding scheme.
We also investigated a second decision model, which assumed that the information for the decision is carried by a single shot noise pair, driven by the Γ1(t) − Γ2(t) difference process. In this model, response R1 is made if the process exceeds an upper criterion a1, and response R2 is made if the process falls below a lower criterion, a2, with a2 < 0 < a1. This model corresponds more directly to the Ratcliff (1978) model in Figure 1, but is susceptible to the criticism of neural implausiblity because the decision variable can take on both positive and negative values (i.e., the decision variable is signed). We found that the model of Figure 4, which assumes a pair of anticorrelated decision processes, and the model with a single, signed process gave very similar patterns of predictions and fits to behavioral data. We report only the fit of the more neurally plausible model of Figure 4 in this letter.
2.3. Simulation of a Recurrent Loop
Figure 5 shows simulated firing rates for a single recurrent loop, together with predictions from the exponential approximation, equation 2.12. The panels on the left show histograms of interspike intervals for the integrate-and-fire neurons shown on the left in Figure 4, together with the best-fitting (maximum likelihood) exponential approximation. The exponential rate parameters estimate γk in equation 2.12. The three panels show, in descending order, the interspike intervals for a superposition of n = 1, 3, 5 neurons. The exponential approximation is fair at n = 3 and good at n = 5. The parameters used in the simulations were λ1 = 50, λ2 = 25, α = 20, and θ = 3 (firing threshold), with the change in membrane potential (Z in equations 2.1 to 2.3) caused by a spike set to unity (i.e., P[Z = 1] = 1). The units of λ1 and λ2 are s−1; the units of θ are an arbitrary multiple of mV. The units of α are (multiples of) mV.s−1. The value of the decay constant, α = 20, corresponds to an upper bound on the intracellular integration process of 50 ms, as Wang stipulated.
The panels in the center and on the right show the mean and variance of the firing rates in the recurrent loop, when driven by input sequences with the same parameters as were used to generate the histograms on the left. The irregular gray lines are the simulated results; the black lines are exponential approximations from equations 2.12 and 2.13. The means and variances of the simulated firing rates were computed on bins of 5 ms in width. The two pairs of functions shown in each of the panels in the center and on the right of Figure 5 are for different values of the loop recurrence parameter: δ = 20 s−1 (upper) and δ = 5 s−1 (lower). The important features of the results in Figure 5 are that (1) the mean and variance of the firing rates increase linearly over time and (2) the firing rates are well described by the exponential approximation of equations 2.12 and 2.13, even though the interspike intervals of the integrate-and-fire neurons are not exponential. The approximation improves with increasing numbers of neurons driving the loop and with increasing values of δ, but is fairly good even with n = 1.
In the model in Figure 4, the activities in the two recurrent loops, Γ1(t) and Γ2(t), drive a shot noise decision process, which is implemented by a pair of anticorrelated decision neurons. The value of the shot noise processes that drive the decision neurons depend on the momentary differences in the spike densities in the two loops. When this difference reaches one of two thresholds, which function as response criteria, the process emits a response. The analysis of equations 2.12 and 2.13 and the simulations in Figure 5 show that the loop processes are approximately Poisson, despite the presence of nonlinearities introduced by the input integrate-and-fire neurons and the overdispersion in the approximating compound Poisson process (see the appendix). The approximating Poisson processes are time inhomogeneous, with intensities γkδt that increase linearly with time.
The process in equation 2.19 should be contrasted with previous applications of the OU process in models of decision making, in which some form of equation 2.4 has typically been assumed (Busemeyer & Townsend, 1993; Ratcliff & Smith, 2010; Smith, 1995; Usher & McClelland, 2001). The usual interpretation of the OU process in decision models is that it describes a bounded accumulation process, in the sense that the process possesses a stationary distribution. In contrast, the linear time dependency in the drift and diffusion coefficients of the process in equation 2.19 endows it with similar properties to the integrated OU process and the Wiener diffusion process, despite the presence of a decay term.
3. Fit to Behavioral Data
3.1. The Ratcliff and Smith (2004) Data
Like the diffusion model, and the integrated OU model, the mean and variance of the accumulating evidence state in the recurrent loop model increase linearly over time. Based on the analysis of Smith (2010a), who showed that the RT and accuracy predictions of the diffusion model and the integrated OU model were virtually indistinguishable, we conjectured that the recurrent loop model would make similar behavioral predictions. The key data that must be accounted for by a psychological model of decision making are the families of RT distributions for correct responses and errors, and the associated choice probabilities, as a function of stimulus discriminability. RT distributions have a characteristic unimodal, positively skewed form (Luce, 1986), which is found in a variety of simple decision tasks (e.g., Ratcliff & Smith, 2004). Reducing the discriminability of the stimulus increases the mean and variance of the RT distribution and increases the proportion of error responses, but has little effect on the distribution shape. The results of Ratcliff and Smith (2010) provide a good example of this: they reported RT distributions from a task requiring discrimination between pairs of letters embedded in dynamic noise. They reported plots in which sets of distribution quantiles from different experimental conditions were plotted against each other. (The distribution quantiles are the values of time that cut off specified proportions of the probability mass in the distribution.) The resulting plots were remarkably linear, providing evidence of the relative invariance of distributional shape. Ratcliff and McKoon (2008) reported similar data.
Figure 7 shows examples of the RT distributions predicted by the recurrent loop model as a function of stimulus discriminability. As described below, discriminability in the model is a function of the relative levels of excitatory and inhibitory activity in the loops. Figure 7 shows that the recurrent loop model generates RT distributions that have the right properties. The distributions are unimodal and positively skewed and have the right kinds of invariance properties, as the quantile plots on the right of the figure show. These distributions are similar to those found in empirical data. Indeed, the parameters we used to generate these distributions were obtained by fitting the model to experimental data, as we describe next.
As a more direct test of the recurrent loop model, we fitted it to one of the benchmark data sets that Ratcliff and Smith (2004) showed in their evaluation of sequential sampling models. The data were from a signal detection experiment in which subjects judged whether the separations between pairs of vertically aligned dots on a computer screen were small or large. Choice probabilities and RT distributions for correct responses and errors were collected from 17 undergraduates and combined by quantile averaging. The data are distributions of correct responses and errors, and the associated choice probabilities, for four levels of dot separation. We did not attempt to carry out a detailed evaluation of the recurrent loop model, because its predictions must be evaluated by time-consuming Monte Carlo methods that are unsuitable for use with the kinds of nonlinear optimization methods used to fit RT models. Instead, we simply sought to show that the choice probabilities and RT distributions predicted by the model agreed with those predicted by the diffusion model and its variants for a representative experimental task.
The full diffusion model includes a number of sources of trial-to-trial variability, in addition to the moment-by-moment variability in diffusive noise shown in Figure 1. The most important of these is variability in the drift parameter, μ. Variability in drift expresses the idea that there will be variation in the quality of the information encoded from nominally equivalent stimuli on different trials. This variability is an important feature of the model because it allows the model to predict mean RTs for error responses that are longer than those for correct responses. This pattern is usually found when stimulus discriminability is low and accuracy is stressed (slow errors). The opposite pattern (fast errors) is found when discriminability is high and speed is stressed (Luce, 1986). To minimize the effect of these trial-to-trial sources of variability, we used only the speed condition from the signal detection task described by Ratcliff and Smith (2004) in our evaluation. The error RTs in this condition are longer than those for correct responses, but the difference is smaller than in a second, accuracy, condition also reported by Ratcliff and Smith.
Figure 8 shows the fit of the recurrent loop model to the experimental data. The figure also shows, for comparison purposes, fits of the full and restricted versions of the diffusion model. The only source of variability in the restricted model is moment-to-moment diffusive variability; the other, trial-to-trial, sources of variability have been set to zero. This makes the model psychologically comparable to the recurrent loop model, although the latter has greater model freedom because it has two more free parameters. The full diffusion model includes three additional sources of trial-to-trial variability: variability in drift, variability in starting point, and variability in the nondecisional component of RT. One or more of these additional sources of variability is typically used when fitting the model to data.
To fit the recurrent loop model, we again used a version of the Gold and Shadlen (2001) neuron-antineuron coding scheme, in which the activities in the two channels are additive inverses of each other. In Figure 4, stimulus information is carried by four independent Poisson processes. The associated Poisson rates are the excitatory (λ1 and λ3) and inhibitory (λ2 and λ4) rates for the shot noise pairs that drive the leaky integrate-and-fire neurons. We fixed the sum of the excitatory and inhibitory Poisson rates in each of the shot noise pairs to a constant and allowed the difference in the rates to vary as a function of stimulus discriminability. We denote the difference in the excitatory and inhibitory rates in a shot noise pair by Δλ. We set λ1 + λ2 = c (constant), λ1 − λ2 = Δλ, λ3 + λ4 = c, and λ3 − λ4 = −Δλ. That is, evidence for one response is evidence against the other. With this coding scheme, the effective stimulus information is represented by a difference of differences, that is, by 2Δλ. Gold and Shadlen argued that the neuron-antineuron coding scheme provides a plausible mechanism for the neural computation of log-likelihood ratios.
Fixing the sum of the excitatory and inhibitory rates fixes the variance of the shot noise pairs (e.g., equation 2.2), which we set to an arbitrary value of 80 spikes per second. This is analogous to the practice of setting σ in the diffusion model to a constant value. (In Ratcliff's papers, this parameter is denoted by s and set to a value of s = 0.1.) The parameters of the diffusion process are identified only to a constant multiple of the standard deviation of the Wiener process, which acts a fundamental scaling parameter in the model. In our fits, we treated the variance of the shot noise process in a similar way. In addition, we constrained the shot noise decay parameter to α = 20 s−1 and allowed the remaining parameters listed in Table 1 to vary. This value of α is consistent with our interpretation of the shot noise process as a short-range, intracellular integration process, as discussed previously. As shown in Table 1, eight parameters were estimated in fitting the model to data: four values of Δλ, the difference in the excitatory and inhibitory activity in the loops, one for each level of stimulus discriminability. The other parameters were the firing threshold in the integrate-and-fire neurons, θ; the exponential loop delay, δ; the decision criterion, a; and the time for nondecisional processes, Ter. The estimated values of these parameters are shown in Table 1.
|Parameter .||Symbol .||Value .|
|Sum of firing rates||λ1 + λ2||80|
|(excitatory plus inhibitory)||λ3 + λ4||80|
|Shot noise decay rate||α||20|
|Parameter .||Symbol .||Value .|
|Sum of firing rates||λ1 + λ2||80|
|(excitatory plus inhibitory)||λ3 + λ4||80|
|Shot noise decay rate||α||20|
Notes: The quantities λ1 and λ3 are excitatory rates; λ2 and λ4 are inhibitory rates. The argument i in Δλ(i), i = 1, …, 4, the excitatory-inhibitory difference, denotes the stimulus condition. The excitatory-inhibitory difference was the only parameter that varied across stimulus conditions.
Nine parameters were estimated in fitting in the full diffusion model: four values of drift, μ, one for each stimulus condition; the decision criterion, a; the between-trial drift variance, η; the starting point variability, sz; the nondecision time, Ter; and the nondecision time variability, st. A detailed account of the function of each of the model parameters may be found in Ratcliff and McKoon (2008). In the restricted model, the between-trial variance parameters, η, sz, st, were all set to zero. The estimated values of the diffusion model parameters were very similar to those reported by Ratcliff and Smith (2004).
The fits in Figure 8 are shown in the form of a quantile probability plot. The quantile probability plot is a parametric plot that shows how the shapes of the RT distributions and choice probabilities depend on the stimulus. Selected quantiles of the RT distributions are plotted against the choice probabilities for correct responses and errors. We used five quantiles in fitting the data: the .1, .3, .5, .7, and .9 quantiles. The .1 quantile characterizes the fastest responses in the distribution (the leading edge), the .5 quantile characterizes the central tendency (the median), and the .9 quantile characterizes the slowest responses (the distribution tail). For each stimulus condition, there are two RT distributions—one for correct responses and one for errors. Figure 8 shows the eight different RT distributions from the four different stimulus conditions.
To construct a quantile probability plot, the distribution quantiles are plotted on the y-axis against the choice probabilities on the x-axis. If, in a particular experimental condition, the probability of a correct responses is p, then the probability of an incorrect response will be 1 − p. The quantiles of the distribution of correct responses are plotted on the y-axis against p, and the quantiles of the error distribution are similarly plotted against 1 − p. The five sets of symbols in Figure 8 are, ascending from bottom to top, the .1, .3, .5, .7, and .9 quantiles. Each pair of distributions for correct responses and errors is plotted similarly. In the resulting plot, the distributions of correct responses appear (usually) to the right of the 0.5 point on the x-axis, and the distributions of errors appear on the left. The two innermost sets of points are the distributions for the lowest level of stimulus discriminability, and the two outermost sets are the distributions for the highest level of discriminability.
The effects of experimental manipulations on RT and accuracy are represented compactly in plots of this kind. They show how accuracy changes as a function of stimulus discriminability and the associated changes in the mean, variance, and shape of the RT distribution. They also show the relationship between the distributions of RT for correct responses and errors.
We fitted the diffusion model using the Nelder-Mead Simplex algorithm, following the procedure described in Ratcliff and Smith (2004). It was not practicable to fit the recurrent loop model in the same way because we evaluated its predictions by Monte Carlo methods. Instead, we did a grid search of the parameter space to obtain an approximation to the best-fitting model. The fits are shown in Figure 8. The fit statistics for each of the models were χ2(35) = 8.6 (full diffusion model), χ2(38) = 57.3 (restricted diffusion model), and χ2(36) = 9.1 (recurrent loop). These fit statistics were calculated using n = 288 trials per condition (N = 1152 trials overall), consistent with our interpretation of the quantile-averaged data as describing the performance of an average subject.
As expected, the performance of the full diffusion model is appreciably better than that of the restricted model. Both versions of the models capture the unimodal, positively skewed shape of the RT distributions and the associated choice probabilities, but the inclusion of additional sources of variability—most important, trial-to-trial variability in drift—allows the model also to predict slower error RTs. The relative speed of correct responses and errors is reflected in the degree of asymmetry of the plot across its vertical midline. Without variability in drift, the model predicts a symmetrical quantile probability plot in which RTs for correct responses and errors are equal.
For our purposes, the important result is the performance of the recurrent loop model, shown in the bottom panel of the figure. Like the diffusion model, the model captures most of the main features of the data. It predicts positively skewed RT distributions and choice probabilities that approximate those in the data. Interestingly, it also predicts slow errors, despite having only a single source of variability. This result is surprising in the light of the formal similarities between the recurrent loop model and Ratcliff's (1978) diffusion model. The latter is able to predict slow errors only when it is augmented with between-trial variation in drift rate; otherwise it predicts the same distributions of RT for correct responses and errors. In contrast, the recurrent loop model predicts slow errors without between-trial variation in drift. Its ability to do so appears to be a function of the extra-diffusive variability that arises from the compound Poisson nature of the recurrent loop process (see the appendix). Because both the mean and variance in a loop depend on the outputs of integrate-and-fire neurons that initiate loop activity, they exhibit some degree of trial-to-trial correlation, in which low means are associated with low variances, and vice versa. This variability endows the model with similar properties to trial-to-trial variability in the diffusion model. As a result, on these data, the model performs almost as well as the full diffusion model, without the need to assume between-trial variability in drift.
3.2. Extensions of the Model
The recurrent loop model is a neurally plausible version of the Ratcliff (1978) two-barrier, single-process diffusion model. In the model, evidence for the two response alternatives is represented by a pair of positively valued, anticorrelated shot noise processes. Ratcliff and Smith (2004) considered a different but related model and subsequently applied it to neural data (Ratcliff et al., 2007, 2011). In this dual diffusion model, evidence for competing responses is represented by a pair of independent, parallel, racing diffusion processes. The model is like the Usher and McClelland (2001) model but does not include inhibitory interactions between the accumulating evidence totals. Ratcliff et al. (2007) used this model to predict firing rates in superior colliculus (SC) buildup neurons in a saccade-to-target decision task using parameters estimated from monkeys’ choice probabilities and RT distributions. Ratcliff et al. (2011) subsequently reported the results of simultaneous recordings from neurons coding the two response alternatives and found no evidence of a negative correlation in firing rates across trials. If, as hypothesized, such neurons provide a continuous readout of the accumulating evidence for the response alternatives, and if, as Usher and McClelland assumed, evidence for the alternatives is mutually inhibitory, a negative correlation would have been predicted. In the absence of such a correlation, Ratcliff et al. (2011) argued that the data were better described by the dual diffusion model. Purcell et al. (2010) reported a similar lack of inhibition in their analysis of FEF neurons.
In its assumption of independent accumulation processes, the dual diffusion model differs from our model, in which the accumulation processes are anticorrelated (see note 2). A recurrent version of the dual diffusion model could be developed in a straightforward way using the framework we have presented here. We have not attempted to do so because it is peripheral to the main purpose of this letter. To obtain the required independence of the decision processes, such a model would need two pairs of recurrent loops instead of the single pair assumed in the model of Figure 4.
Although recordings in SC and FEF suggest a model in which evidence for the two response alternatives is accumulated independently, leading to a dual diffusion model, a different picture emerges from recordings in other areas, notably, lateral interparietal area (LIP). Roitman and Shadlen (2002) recorded from motion-sensitive neurons in LIP in a direction-of-motion discrimination task and found evidence for anticorrelated accumulation processes. Firing rates in neurons coding the chosen direction increased, while firing rates in the neurons coding the nonchosen direction decreased. In our model, anticorrelated accumulation is represented by the activity in the decision neurons on the right-hand side of Figure 4. Because the loop differences that drive these neurons are additive inverses of each another, increases in one process are accompanied by corresponding decreases in the other. Such anticorrelated accumulation occurs, however, only in the short-range intracellular process, but not in the spike trains that represent the long-range intercellular process. The activity in the recurrent loops, which code evidence for the two responses, increases over time; at different rates for the chosen and nonchosen alternatives. Although this differs from the findings of Roitman and Shadlen (possibly because it represents a different stage of the decision process), changing the assumptions about the decision stage allows the model to reproduce the observed patterns of neural firing.
Currently a decision is assumed to be made as soon as the membrane potential in one of the leaky integrate-and-fire decision neurons on the right in Figure 4 exceeds a threshold. An alternative is to assume that each of these neurons emits not one spike but a sequence of spikes, which is thresholded later in the processing sequence, possibly by motor neurons terminating at muscle end plates, to make a choice response. Under this identification a “decision” ceases to be a single-point event and instead becomes the cumulative effect of a sequence of motor events that together comprise an overt action. Unlike the integrate-and-fire neurons on the left of Figure 4, whose firing rates reflect the instantaneous evidence for the two response alternatives, in this modified model, the sequences of spikes emitted by the neurons on the right of Figure 4 would reflect the cumulative evidence for the alternatives. Moreover, because the loop difference processes that drive these neurons are additive inverses of each other, the firing rate in one will go up while the other will go down, as Roitman and Shadlen found.
The extensions to the model discussed in the previous paragraphs seek to emphasize that many variations on the basic architecture are possible. The dual diffusion model and the recurrent loop model of Figure 4 represent two extremes of a theoretical continuum, in which the accumulation processes are either completely independent or perfectly anticorrelated, respectively. Intermediate models, like the Usher and McClelland (2001) model, are also possible if we assume that the loop difference processes that drive the decision neurons are partially, rather than perfectly, anticorrelated. We have chosen here to focus on the link with the Ratcliff (1978) model as the best-known and most intensively studied member of this class. However, the computational principles we have identified, which link Poisson shot noise processes to behavioral-level diffusion via recurrence, are general ones.
In this letter, we investigated the performance of a recurrent loop model of decision making. Our goal in this analysis was to link the shot noise representation of diffusive information accumulation of Smith (2010a) to the network analysis of Wang and colleagues (Wang, 2001, 2002; Lee & Wang, 2009; Lo & Wang, 2006; Wong & Wang, 2006). Wang has pointed out that a critical element of any attempt to link behavioral models of decision making to their neural substrate is to understand how information is integrated over time. He has argued that reverberation loops provide a plausible mechanism for carrying out such integration. He has also shown that a network of spiking neurons, consisting of pairs of populations implementing reverberation loops, can capture many of the behavioral and neural features of decision making. Here we investigated the statistical properties of a minimal recurrent architecture in which each spike entering a loop generates a new spike after a random delay. This captures the essential statistical features of recurrence in an abstract way that is not tied to the features of any particular network.
Our analysis showed that a model architecture in which the decision variable is computed from the difference in the activity in a pair of recurrent loops results in an information accumulation function in which the mean and variance of the accumulating evidence increase linearly over time. This analysis parallels that of Smith (2010a), which assumed algebraic integration as a basic feature of the model. Smith's analysis led to a representation of accumulating stimulus information as an integrated OU, or OU displacement, process. Our recurrent loop analysis led to an alternative representation, as an inhomogeneous OU velocity process with linear drift and diffusion coefficients. In either case, the model predicts approximately linear growth in the mean and variance of the accumulating evidence over time, like the diffusion model. The models therefore provide a theoretical account of how diffusive evidence accumulation can arise at the behavioral level, through the spatiotemporal aggregation of the local effects of action potentials in the underlying neural population. Whereas Smith's analysis was entirely statistical, we have sought to give further neural substance to the analysis by linking it to the properties of simple integrate-and-fire neurons in a recurrent loop architecture. The important finding in either instance is that behavioral-level diffusion emerges as a natural consequence of the underlying neural integration processes.
We used exponentiality freely in our analysis so we could derive simple analytical results. We justified the exponential assumption by arguing that the recurrent loops and their inputs are likely realized by multiple, parallel neural pathways. Exponentiality arises theoretically as a consequence of the superposition of independent renewal processes, with arbitrary renewal time distributions. Arguably, however, the assumption of independence is questionable in neural settings. For example, Zohary, Shadlen, and Newsome (1994) have shown that populations of weakly connected neurons do not show the kinds of law of large numbers statistical properties exhibited by populations of independent neurons. Fortunately, the assumption of exponentiality, while convenient, was not an essential feature of our analysis. The firing rate equations, equations 2.12 and 2.13, which we derived from exponential assumptions, provided a reasonable account of the simulations even when the loops were driven by a single integrate-and-fire process. We also found that if we replaced the assumption of an exponential loop delay with a fixed delay, the model became noisier, but its essential features were not materially altered. This suggests our results are likely to be reasonably robust across changes in assumptions. In sum, then, our results help explain the success of diffusion models of decision making in psychology, and why they tend to perform better than models that assume other kinds of evidence accumulation processes. We have also supported Wang's argument that recurrent activity in reverberation loops may provide the basis for long-range neural integration in evidence accumulation. They therefore contribute to our understanding of how psychological and behavioral mechanisms of decision making can be realized computationally in neural populations.
Appendix: Firing Rate of a Recurrent Loop
The preceding derivation implies that the firing rate in a recurrent loop can be well approximated by an inhomogeneous Poisson process with intensity parameter, γkδt, that increases linearly in time. The derivation ignores the probability that an initiating spike arrives in the interval (t, t + h] and then triggers a new recurrent sequence whose first spike is also observed during this interval. These are independent events whose joint probability equals γδh2 + o(h). This probability is of the order o(h) and can be neglected in the derivation of the instantaneous firing rates.
The research in this letter was supported by ARC Discovery Grant 0880080 to P.L.S. and R. Ratcliff. A paper based on this research was presented at the Fourth Australian Workshop on Computational Neuroscience at the Queensland Brain Institute, University of Queensland, in November 2010. We thank Scott Brown, Roger Ratcliff, James Townsend, and an anonymous reviewer for helpful comments on an earlier version of the manuscript.
The assumption that the fourth moment is finite is a technical one, which ensures weak covergence of the shot noise pair to a limiting Ornstein-Uhlenbeck velocity process (Smith, 2010a). It is not required for any of the properties of the processes used in this letter, except for the diffusion approximation in equation 2.19.
The claim that model fit is not improved by the addition of a leakage or decay term refers specifically to models with a single, signed decision variable, such as the two-barrier Wiener and two-barrier Ornstein-Uhlenbeck models. An alternative model architecture was considered by Smith (2000), Ratcliff and Smith (2004), and Ratcliff et al. (2007, 2011), in which the evidence for competing responses is represented by a pair of parallel, independent, racing diffusion processes. Ratcliff et al. called this model the dual diffusion model. In applications, fits of the dual diffusion model are sometimes improved by the addition of moderate amounts of leakage (Smith & Ratcliff, 2009). The model of Figure 4 differs from the dual diffusion model in that the decision processes are anticorrelated rather than independent. It is therefore functionally more like a single process model than a dual diffusion model.