The primate visual system has an exquisite ability to discriminate partially occluded shapes. Recent electrophysiological recordings suggest that response dynamics in intermediate visual cortical area V4, shaped by feedback from prefrontal cortex (PFC), may play a key role. To probe the algorithms that may underlie these findings, we build and test a model of V4 and PFC interactions based on a hierarchical predictive coding framework. We propose that probabilistic inference occurs in two steps. Initially, V4 responses are driven solely by bottom-up sensory input and are thus strongly influenced by the level of occlusion. After a delay, V4 responses combine both feedforward input and feedback signals from the PFC; the latter reflect predictions made by PFC about the visual stimulus underlying V4 activity. We find that this model captures key features of V4 and PFC dynamics observed in experiments. Specifically, PFC responses are strongest for occluded stimuli and delayed responses in V4 are less sensitive to occlusion, supporting our hypothesis that the feedback signals from PFC underlie robust discrimination of occluded shapes. Thus, our study proposes that area V4 and PFC participate in hierarchical inference, with feedback signals encoding top-down predictions about occluded shapes.
In natural scenes, objects rarely appear in isolation; rather, animals often have to discriminate and recognize partially occluded objects. While recognition under occlusion is difficult for even the best computer vision system, animals seldom have trouble. But the neural basis of this capacity is poorly understood. Here, we study the physiological underpinnings of a special case of the general problem, where occluders can be detected as distinct stimulus features.
Feedback projections from higher cortices are hypothesized to be important for successful recognition of occluded objects (Rust & Stocker, 2010; Gregoriou, Rossi, Ungerleider, & Desimone, 2014), and there are abundant feedback connections in the visual stream. Despite this, models of object recognition are typically hierarchical feedforward circuits (Fukushima, 1980; Riesenhuber & Poggio, 1999; Serre, Oliva, & Poggio, 2007; Cadieu et al., 2007; Yamins et al., 2014). This is partly because of the complexity of including feedback signals, but also because little is known about where the relevant feedback signals originate, where they terminate in visual cortex, and how they contribute to recognition. Developing a computational framework explaining how feedback facilitates shape recognition under occlusion therefore is a prominent challenge for visual neuroscience.
Recent experimental results provide key insights into how interactions between area V4, a fundamental stage in the primate shape processing pathway (Roe et al., 2012; Pasupathy & Connor, 1999, 2001), and the prefrontal cortex, important for the control of complex behavior (Miller & Cohen, 2001), may underlie the ability to recognize partially occluded objects (Kosai, El-Shamayleh, Fyall, & Pasupathy, 2014; Pasupathy, Fyall, & Choi, 2015; Fyall, El-Shamayleh, Choi, Shea-Brown, & Pasupathy, 2017). Specifically, in monkeys trained to discriminate pairs of shapes under varying degrees of occlusion, dynamics of V4 and PFC activity suggest that feedback signals from PFC to area V4 may serve to discount the effect of occlusion on the responses of V4 neurons, thereby increasing shape selectivity. This raises the question of how the feedback signals in V4-PFC circuitry perform the computation necessary for shape recognition. In this article, we propose and test the hypothesis that this occurs via hierarchical predictive coding. With the proposed model based on predictive coding, we successfully explain the dynamics of a subpopulation of neurons in V4 that exhibit delayed peak of responses (Pasupathy et al., 2015; Fyall et al., 2017), presumably induced by feedback signals from PFC.
Predictive coding has been proposed as a method to create efficient neural codes and has successfully described neural responses in a variety of different sensory systems (Bogacz, 2017; Bastos et al., 2012; Friston & Kiebel, 2009a, 2009b; Srinivasan, Laughlin, & Dubs, 1982; Rao & Ballard, 1999; Spratling, 2017; Rao, 1997, 1999, 2004, 2005; Lee & Mumford, 2003; Yuille & Kersten, 2006). Notably, the predictive coding framework reproduces center-surround antagonism in the retina (Srinivasan et al., 1982) and end-stopping effects in V1 (Rao & Ballard, 1999). In these studies, feedforward signals from each cortical area represent the residual errors between the feedback predictions and the encoding expectation. This interpretation of feedforward signals, however, has met the criticism (Koch & Poggio, 1999) that it implies reduced firing when familiar sensory inputs are encountered, differing from the common view in which sensory neurons respond strongly to preferred features. Here, we introduce a novel implementation of predictive coding, where the responses in V4 and PFC correspond to their most likely (or optimal) values given the stimulus and a hierarchical representation of its likelihood. Furthermore, the hierarchical inference is implemented in two steps, initially reflecting only the feedforward sensory signals and later integrating the feedback predictions, to explain the dynamic shape-selective responses in V4.
In addition to assigning an algorithmic role to the feedback signals, our model makes further predictions on the structure of the network, representation of the stimuli, and prior expectations encoded in V4 and PFC. Previous studies have shown that shapes can be discriminated based on V4 activity at the population level (Meyers, Freedman, Kreiman, Miller, & Poggio, 2008; Pasupathy & Connor, 2002), and shape identity information is already available at the level of V4. However, in our model, feedback predictions effectively remap the population responses and amplify the shape identity information reduced by partial occlusion. Furthermore, our model predicts that such amplification of the shape identity information following feedback predictions occurs only when the occlusion is salient and distinct from the shape.
In sum, our model suggests that feedback signals to V4 during the representation of occluded shapes can be interpreted in the context of predictive coding. These results shed light on how prior expectations contribute to the recognition of complex images in V4 and higher cortical areas.
Animals were trained on a sequential shape discrimination task, where two stimuli were presented in sequence and the animal had to report whether they were the same or different with a rightward or a leftward saccade, respectively. The second stimulus in the sequence was presented in the receptive field of the V4 neuron under study and was partially occluded. During recordings in area V4, all task details were customized to the preferences of the single neuron under study. Specifically, one of the two discriminanda was a preferred shape that elicited strong responses from the neuron while the other was a nonpreferred shape. Both shapes were presented in a preferred color for the cell, and the occluding dots were in a nonpreferred color so they provided only a modulatory influence. For recordings in the PFC, we studied many neurons simultaneously and did not customize stimulus shape or color to individual neuronal preferences, as is customary in the field. Each day the experimental session began as follows. We chose two stimuli to serve as the discriminanda. This was followed by two phases. First, during the training phase, animals performed the sequential discrimination task with the unoccluded versions of the discriminanda. This typically included 20 attempts and was to ensure that the unoccluded versions of the discriminanda were discriminable on the periphery. This was followed by the test phase during which the discriminanda were occluded to different levels with a field of randomly positioned dots. The level of occlusion was titrated by varying dot diameter while the number of dots was held constant, and was quantified as the percentage of the shape area that remained visible (% visible area).
All animal procedures conformed to NIH guidelines and were approved by the Institutional Animal Care and Use Committee at the University of Washington.
2.2 Coding Assumptions
We explain the response dynamics of V4 and PFC neurons during the shape discrimination task by building a computational model based on a few coding principles.
First, we assume that average firing rates of the neuronal populations recorded in experiments reflect the most likely representation of the neuronal responses given the input visual stimulus and a specific hierarchical model of the responses that we define below. Thus, assuming the sensory system seeks to infer the most likely representation of neuronal responses of hierarchical areas ranging from the lowest area 1 to the highest area , we simply find the set of responses that maximizes the posterior probability , where represents the sensory input. We refer to these as the optimal firing rates.
We have described the general and classical framework for hierarchical representation of a stimulus via a sequence of firing rates. In summary, we assume that the brain aims to have neuronal activity in every layer get as close as possible to the prediction made by the responses of the next higher layer, where the discrepancy is given by a noise term . Then the neural firing rates adjust to those that are most consistent (i.e., most likely) given the stimulus . We next describe the specific form of the representation that we use here.
2.3 Model Architecture
Our model is composed of two layers, a V4 layer and a PFC layer (see Figure 1A). We designate the higher cortical area as PFC based on the experimental evidence indicating feedback from PFC as a likely precursor of the delayed responses in V4 (Pasupathy et al., 2015; Fyall et al., 2017; see section 3). Furthermore, previous experimental studies have found anatomical and physiological evidence for direct feedforward (Ninomiya, Sawamura, Inoue, & Takeda, 2012) and feedback (Barbas & Mesulam, 1985; Ungerleider, Galkin, Desimone, & Gattass, 2008) connections between V4 and PFC in the primate brain.
The V4 layer is composed of three units: two that are selective for each of the two visual shapes that are being discriminated, namely, shape A and shape B (see Figure 1A, V4 unit 1 (green) and V4 unit 2 (blue), respectively), and a third V4 unit that responds selectively to the occluder-specific features, such as color (see Figure 1A, V4 unit 3 (red)). Shape selectivity has been previously demonstrated in area V4 (Pasupathy & Connor, 1999). While the existence of V4 cells that are selective exclusively for occluders has not been confirmed experimentally, a recent experimental study has found such strictly occluder-selective cells in the IT cortex (Namima & Pasupathy, 2016). Furthermore, we do not require that neurons corresponding to V4 unit 3 would be exclusively selective for occluders independent of other stimulus features; rather, they could respond preferentially to any occluder-specific features. The V4 cells that preferentially respond to the color of the occluders are a good candidate, as in the experiment, occluders were presented in a different color from the shape or the background. Supporting this, many V4 neurons are known to have color selectivity (Zeki, 1973; Schein & Desimone, 1990; Bushnell, Harding, Kosai, Bair, & Pasupathy, 2011; Bushnell & Paupathy, 2012), and many are sensitive simply to stimulus area rather than shape (Eghbali, Pasupathy, & Bair, 2016). Indeed, in Figure A.1B, we present example V4 cells that respond strongly to the presence of occluders regardless of whether these occluders are presented with the preferred or the nonpreferred shape. Each V4 unit can be interpreted as a subpopulation of V4 neurons with similar tuning properties.
The two PFC units in the model represent two distinct neuronal populations in PFC. While the roles of PFC neurons are not well understood, PFC is believed to be involved in planning complex behavior and tasks involving short-term memory (Miller & Cohen, 2001). Experimental recordings (Pasupathy et al., 2015; Fyall et al., 2017) from PFC also show that a subset of PFC neurons has mild shape selectivity while also responding strongly to occluders.
The sum of PFC activities weighted by the connection weights between V4 and PFC units (see Figure 1A) is represented as the feedback signal to V4 units. The initial feedback connection weights between V4 units and PFC units are chosen so that the PFC units show appropriate selectivity after training. Namely, one of the PFC units in the model is designated to be weakly shape A selective and the other PFC unit is weakly shape B selective. Both PFC units respond strongly to partially occluded shapes and only weakly to unoccluded shapes.
In this way, PFC neurons of the model respond strongly to both the task-relevant visual features (shape identity) and nuisance variables (occlusion level), while each of V4 populations responds preferentially to a single feature of the input visual stimulus. Thus, although the V4 responses are already modulated by both shape and occlusion level, the signals become even more mixed as they go up the hierarchy. Previous studies have shown that such mixed selectivity in the PFC plays an important computational role in a high-dimensional population encoding of task-relevant information (Rigotti et al., 2013; Fusi, Miller, & Rigotti, 2016).
2.4 Probabilistic Network Model
As we detail below, the responses of the neuronal units evolve toward values that maximize the posterior probability of these responses given the input shape stimulus. In other words, the neuronal activities, and synaptic weights at a slower timescale, are found by estimating the most likely values given the shape stimulus.
The mean and the covariance matrix are determined by the input shape identity and the occlusion level . Changes in and describe the sensory-input-driven responses of the V4 populations to different shapes under various degrees of occlusion. In other words, for each occlusion level and the shape identity, there is a most likely firing rate of each V4 unit given by , and that likelihood falls off according to the covariance .
Here we describe how we modulate and based on the sensory input . Let us assume the animal is presented with shape A as the test shape. With shape A presented, , the gaussian mean of the firing rate distribution of V4 unit 1 in Figure 1A (the shape A-selective V4 population), decreases as occlusion increases (see Figure 1C, green). However, of the V4 population, preferring shape B (V4 unit 2 in Figure 1A), stays constant at a baseline firing rate, independent of the change in occlusion level. That is, the V4 unit 2 does not prefer shape A; it responds with a low firing rate regardless of the occlusion level (see Figure 1C, blue). The standard deviation of the preferred V4 unit increases as occlusion increases in order to capture the increasing uncertainty of the shape identity under higher degrees of occlusion (see Figure 1C, where the green distribution widens as occlusion increases). The standard deviation of the nonpreferred V4 population (V4 unit 2) is assumed to be constant.
A justification for increasing the input variance but not with occlusion is as follows. These terms represent uncertainty in shape identity signals. We hypothesize that occlusion introduces the most uncertainty for neuronal responses to preferred shapes, as randomly placed occluders may either hide critical features of the preferred shapes or fail to hide these features. In the first case, shape signals will be strongly suppressed; in the second, they will be maintained. For nonpreferred shapes, which lack critical features, we hypothesize that shape signals for different occlusion patterns will be less volatile. Supporting this, while random placement of occluders may form accidental contours, a previous experimental study in V4 has shown that responses to preferred contours are suppressed when those contours are accidentally formed at the junction between the occluded and occluding objects (Bushnell, Harding, Kosai, & Pasupathy, 2011). Accordingly, the variance should be roughly constant with added occlusion or, if increasing, only by a small amount. In section 3, we explore which trends in variances are consistent with the data in more detail (see Figure 9).
Finally, for V4 unit 3, the relevant stimuli (occluding dots) are present on every trial but slightly shifted in position. As we assume that this unit responds to the presence of occluders but not their specific configuration, the variance is taken to be constant across occlusion levels (see Figure 1C, red). As the occlusion level increases, of the occluder-selective V4 population (V4 unit 3) also increases.
The dependence of the means and the variances on the occlusion level was set to be linear: and with , , , and . The slopes (, ) and the values defining the response distributions when the shape is unoccluded (, ) at were manually chosen to match the peak firing rates observed in experiments. With this choice of , as occlusion level increases, the peak of the response distribution decreases, stays constant at a low baseline firing rate, and increases, for V4 units 1, 2, and 3, respectively. Thus, V4 units 1 and 2 reproduce response patterns of V4 neurons to preferred and nonpreferred shapes under varying degrees of occlusion in experiments (see Figures 2 and A.1A), and unit 3 replicates V4 neurons that respond strongly to occlusion (see Figure A.1B). The values chosen for , on the other hand, indicate that the ambiguity of the stimulus feature increases only for the test shape-preferred V4 unit 1. In this way, input stimuli—shapes A and B with various degrees of occlusion—are represented by the response distributions of three different V4 populations given rather than by using actual pixel images.
The standard deviation of the response distribution of each V4 unit given the PFC responses determines the relative significance of the top-down predictive contribution on shaping the V4 responses. Specifically, a smaller standard deviation leads to smaller noise terms, forcing closer matches between PFC and V4 responses. These standard deviations were chosen as , and . Thus, the top-down component is more strongly emphasized for V4 unit 3, the V4 neuronal population selective for occluders. We found that such emphasis on the predictive component for the occluder-selective V4 population was necessary to reproduce the experimentally observed PFC response characteristics, an increase in PFC responses with a rise in occlusion level (see section 3).
Note that this cost function is the sum of the squared error between the V4 responses and the sensory-input imposed representation, and the squared error between the V4 responses and the top-down prediction made by PFC, weighted by their inverse variances.
The optimal “parameters”—the neuronal responses and the connection weights—are thus found by minimizing this cost function with respect to the parameters , , and . The initial V4 responses in experiments, which presumably depend only on the feedforward sensory input, are found by minimizing only the first term of equation 2.14. The initial responses are therefore equal to the sensory-driven representation . However, the delayed V4 responses, which we hypothesize to depend on both the feedfoward sensory input and the feedback prediction, are found by minimizing the entire cost function, equation 2.14.
2.5 Training Protocol: Weight Adjustment during the Preliminary Phase
We divide the optimization process into two phases based on the experimental setup: the preliminary phase and the test phase. In this section, we discuss how the synaptic weight matrix between PFC and V4 is found during the preliminary phase. To find these weights, we minimized the cost function with respect to and , as well as with respect to the connection weight matrix , over a series unoccluded trials. Then, during the test phase, the optimal estimates of the neuronal responses to shapes under varying degrees of occlusion are determined by minimizing the cost function with respect to and , with the connection weights fixed at the learned values.
The preliminary phase corresponds to the stage at the beginning of the experiment where the animal is exposed to a pair of unoccluded shapes used for the experimental session for about 20 times. We introduced its equivalent in the simulation, during which the cost function is minimized by gradient descent with respect to the firing rates of the V4 units and PFC units , as well as the connection weight matrix . During this phase, unoccluded shapes A and B are randomly chosen and used as inputs to the model for up to 30 trials.
The learning rate of was a significantly smaller value compared to that of and , which was . This models the relatively faster dynamics of firing rates and slower dynamics of synaptic plasticity. For each selected shape, we carried out gradient descent until the firing rates reach steady states after a minimum 20 iterations or until the iteration exceeds the maximum of 500 iterations. While and rapidly converge to a fixed point for each of the sampled shapes, the connection matrix gradually converges over the course of multiple samples of shapes A and B. In this way, the weight matrix is tuned over the course of the preliminary phase, which corresponds to the animal's familiarization with the pair of the shapes at the beginning of the experiment.
We set initial weights for and smaller than the initial values of other connection weights, to slightly bias one of the PFC populations (PFC unit 1) to be shape A selective and the other (PFC unit 2) to be shape B selective.
We acknowledge a limitation of the gradient descent method on in equation 2.15, which is that it requires nonlocal computation. In other words, the activities and the synaptic strengths of all the neuronal units in the system must be known in order to take a gradient descent step, a requirement that is not physiologically realistic. This issue also exists in previous models of predictive coding and sparse coding in the visual system (Rao & Ballard, 1999; Olshausen & Field, 1996, 1997), as pointed by Bogacz (2017) and Zylberberg, Murphy, and DeWeese (2011). While we do not pursue this matter further here, we note that Zylberberg et al. (2011) show that in the limit that the neuronal activity is sparse and uncorrelated, the nonlocal gradient descent rule is approximately equivalent to a synaptically local rule.
2.6 Optimal Stimulus Representation during the Test Phase
Once the weight matrix has converged over the course of the preliminary phase, it is fixed at the learned values during the test phase. The test phase corresponds to the recording session where the animal performs the matching task while test shapes with varying degrees of occlusion are displayed. We hypothesize that the V4 and PFC recordings from the experiment are represented by the average firing rates of the V4 and PFC populations in the model network, and , that minimize the cost function . Either shape A or shape B can be used as the input to the network. In this article, however, without loss of generality, we show only the simulations with shape A as the test shape so that the V4 unit selective for shape A (V4 unit 1) is the preferred population and the shape B-selective V4 unit (V4 unit 2) is the nonpreferred population. The weight matrix is fixed at the learned values from the preliminary phase.
includes a term that depends on the difference between and the top-down predictions made by PFC, , in addition to the error term between the and the V4 responses predicted by the input visual stimulus. Therefore, that minimizes this cost function is interpreted as the V4 responses shaped by both the feedforward and the feedback signals. This is compared to the delayed responses in V4 neurons in experiments that we hypothesize to be induced by feedback from PFC.
and are minimized using gradient descent and Matlab fminsearch with respect to and , starting from the initial value at 10 (spikes/s) for all neuronal units.
In section 3.1, we present experimental evidence that supports the hypothesis that feedback signals from PFC modulate shape representations in V4. In section 3.2, we compare the outcomes in our probabilistic network model to physiology and explain in sections 3.3 and 3.4 how robust shape recognition can be achieved in our model. Subsequently, we identify necessary assumptions on the network structure (in section 3.5) and the signal structure (in sections 3.6 and 3.7) of the model to capture the key trends in the experimental results. Finally, using our model, we make predictions in section 3.8 on shape-selective neuronal responses to a new type of reduced stimulus clarity.
3.1 Experimental Evidence for Feedback Signals in Area V4
Recent experiments demonstrated that neurons in V4 and PFC show strikingly different response patterns in monkeys performing a sequential shape discrimination task. Specifically, a class of V4 neurons shows evidence of feedback signals from PFC, supported by interesting response patterns in these V4 neurons and PFC neurons. Our goal in this study is to provide a normative model describing these experimental results.
Figure 2A shows the response dynamics of an example V4 cell to a preferred shape (left) and a nonpreferred shape (right). The V4 neuron exhibits two transient peaks when the preferred shape was presented, but only one smaller peak for the nonpreferred shape. In the initial transient at the onset of the preferred shape stimulus, the V4 neuron responded strongly to the unoccluded shape (black), and an increase in occlusion weakened the shape-selective responses (color). While the first peak shows a dramatic dependence on occlusion, the latter peak of responses shows a weaker dependence. Figure 2B shows the averaged responses of the V4 neuron during the initial transient (50–125 ms) and the delayed transient (175–250 ms), illustrating the differential effects of occlusion on V4 responses over time. The reduced effect of occlusion on V4 responses to the preferred shape during the second transient leads to enhanced shape selectivity, as previously observed in Pasupathy et al. (2015), Kosai et al. (2014), and Fyall et al. (2017). Such response patterns were observed in many other V4 neurons in experiments. In Figure A.1A, we show a few more example V4 cells that exhibit shape-selective responses that are less sensitive to occlusion during the delayed response peak.
In contrast to V4 neurons, PFC neurons exhibit one peak and show their strongest responses to occluded stimuli and weakest responses to unoccluded stimuli, as shown for an example PFC neuron in Figure 2C (Pasupathy et al., 2015). Figure 2D shows the time-averaged responses of the PFC neuron as a function of occlusion level for both the preferred and the nonpreferred shapes. As occlusion increases, the PFC responses increase, which is the opposite trend for V4. Moreover, the timing of the peak PFC responses is between the initial and the delayed transients of V4 responses, consistent with the hypothesis that the PFC responses, which arise from feedforward transmission of sensory information, in turn send feedback inputs and drive the second peak of responses in V4. These experimental observations led us to the hypothesis that the feedback inputs from PFC and other higher cortices underlie delayed improvement of shape-selective responses under occlusion in V4. (For more details on the experimental results, see Kosai et al., 2014, Pasupathy et al., 2015, and Fyall et al., 2017.)
3.2 Structure and Design of Probabilistic Network Model
We sought to understand the response dynamics of V4 and PFC neurons in the context of predictive coding, a hierarchical encoding of stimuli widely used to probe interactions of lower- and higher-sensory areas. We first pose a probabilistic network model of the V4-PFC circuitry with the presumptive feedback based on predictive coding and introduce an innovation that differentiates our model from previous predictive coding models.
In each layer of our V4-PFC network model, there are distinct units, each representing a neuronal population with similar tuning properties. The V4 layer is composed of three units that respond preferentially to different features of the visual stimulus (see Figure 1A): unit 1 to shape A, unit 2 to shape B, and unit 3 to an occluder-specific feature (e.g., the color of the occluders). In PFC, two units respond strongly to occlusion while also exhibiting some degree of shape selectivity. The representation of a population of neurons as a single unit is a common simplification, but we find that each unit replaced by a population of multiple neurons with mild heterogeneity yields qualitatively the same response trends as with the single-unit model (see appendix B).
In the model, V4 receives feedforward sensory inputs and seeks to match the responses imposed by the sensory inputs. At the same time, feedback predictions from PFC bias the V4 responses. The weighted sums of PFC responses provide top-down predictions conditioned on underlying visual stimulus and are regarded as the feedback from PFC to V4. With hierarchical Bayesian inference assumed, the most likely representation of the responses is obtained by finding a set of responses that maximize the posterior probability given the visual stimulus, which is equivalent to the product of conditional probabilities of the neuronal activities given only the activities of the next higher area (see equation 2.3). Note that we are finding not only the optimal responses of V4 units but also the optimal responses of PFC units to minimize the cost function. Therefore, the V4 neuronal responses drive the PFC responses, while the PFC predictions drive V4 neurons, enacting feedforward and feedback connections between V4 and PFC. Here, the visual input to each V4 unit is represented as a gaussian distribution, whose mean and variance change according to the shape identity and the occlusion level (see Figure 1C). Similarly, the feedback from PFC to each V4 unit is described by a gaussian distribution with the peak at a sum of the PFC responses weighted by the synaptic strengths (see Figure 1B).
In this way, the optimal representation of the neuronal responses integrates both the bottom-up sensory input and the top-down prediction. This is done by minimizing a cost function composed of the difference between the V4 activities and the top-down predictions, as well as the difference between the V4 activities and the V4 responses predicted by the sensory input, with each term inversely weighted by its respective variance (see equation 2.14). We compare this optimal representation directly to the neuronal responses in experiments; this differs from previous studies (Rao & Ballard, 1999; Srinivasan et al., 1982) where the residual error between the prediction and the neuronal activity was associated with physiologically measured responses. With this reformulation, neural activity conveys both the sensory input and the internal prediction, preventing the situation in original implementations of predictive coding in which neurons have depressed activity when familiar stimuli are presented regardless of the sensory tuning properties.
3.3 Network Training and Synaptic Weight matrix
The convergence of the weight matrix depends on the choice of initial conditions, given the nonconvex and underconstrained nature of the cost function , as there are multiple combinations of the connection weights and neuronal responses that minimize . However, this does not limit our main results, as we can regard the biased initial values as the connections between a subset of PFC populations and the V4 population of interest before learning the shapes, which may have either weak negative values or positive values, among a wide range of random initial connection weights between PFC and V4. Depending on the initial connection weights, the connections will become either stronger or weaker over the course of training, and shape selectivity in PFC neurons emerges.
Our simulations with synaptic weights starting from different initial values show that the neuronal responses of the model are robust to precise choices of these initial weights. In Figure 3A, we randomly choose different initial weights, under the constraint that and start from stronger values (in the range from 0.5 to 3.5) than and (in the range from 1 to 1). The initial weights between the occluder-selective V4 units and PFC units, and , are randomly chosen in the range from 0 to 2. For all of these choices, the connection weights and converge to higher values than and , resulting in mild shape selectivity in PFC units, with PFC unit 1 preferring the test shape (see Figure 3A i and iv). If the initial connection weights of and are at larger values than and , the shape preferences in PFC units switch (see Figure 3B i and iv), but the response characteristics of V4 neurons remain unchanged (see Figure 3B ii and iii). Interestingly, responses of V4 units are highly robust to differences in initial weights, converging to almost indistinguishable identical values in each case, as shown in Figures 3A, ii and iii and 3B ii and iii.
The obtained connection weight matrix is interpreted as a stored template or memory of the shape pair and is fixed during the following test phase. The memory of the shapes encoded in the connection weights is similar to the idea proposed in Mumford (1992), suggesting that descending pathways store templates in the weights of their synapses.
3.4 Two-Step Inference on Neuronal Dynamics
With the trained connection weights, we find that the model responses of each unit to partially occluded stimuli are comparable to neuronal responses in experimental recordings during the sequential shape discrimination task described above. In particular, we separate the responses inferred strictly by feedforward sensory inputs from those generated by integrated signals of both feedforward inputs and feedback predictions and show that the model responses capture the temporal dynamics in the electrophysiological recordings.
The optimal representations of the neuronal responses and that minimize either the first term ( from equation 2.16) or the full representation of the cost function ( from equation 2.17) are computed at each occlusion level. As explained in section 2, these are equivalent to the optimal responses in hierarchical Bayesian inference that maximize the posterior probability of the V4 neuronal responses given the shape identity and the occlusion level. Here we assume that the occluders are of a color different from that of the shape or the background, that is, occlusion is salient and distinct (see Figure 4B). The occluders therefore activate V4 unit 3, the occluder-selective neuronal population in the model.
We make the inference on the neuronal responses in two steps. First, only the bottom-up sensory input is considered, so that the posterior distribution depends only on the stimulus (see Figure 4A, solid box). In other words, the optimal representations of the activities of the V4 units, , are found by minimizing only the first term of the cost function in equation 2.14 or, equivalently, maximizing equation 2.9. We hypothesize that these optimal responses are modulated only by the bottom-up sensory inputs, to correspond to the initial transient in recorded V4 responses. Thus, only feedforward signals are present at this stage.
The delayed transients in V4 responses following the peak of responses in PFC are compared to the optimal responses that integrate both the bottom-up and the top-down inputs. The model representations of the delayed V4 responses and the PFC responses therefore are obtained by finding and minimizing the full cost function (see equation 2.14), which is equivalent to maximizing the full posterior distribution in equation 2.4 composed of both the feedforward, -dependent distribution and the feedback, prediction-driven distribution. In this way, the model draws a connection between the response dynamics of V4 and PFC neurons and different computational stages in the feedforward-feedback loop.
The inferred optimal responses of each neuronal unit in V4 and PFC across a range of occlusion levels, before and after the feedback from PFC, are shown in Figures 4C to 4E. Both PFC unit 1 and unit 2 responses increase with added occlusion (see Figure 4C), in agreement with the experiments where PFC neurons respond strongly to occluded stimuli and weakly to unoccluded stimuli (see Figure 2D). Such increased PFC responses to occlusion result from the PFC connections to the occluder-selective V4 unit 3. Through the synaptic connections, PFC predictions are compelled to match the responses of V4 unit 3, which responds preferentially to occluders. The model PFC units also show shape selectivity, with PFC unit 1 showing higher responses than PFC unit 2 to the test shape A across occlusion levels. This agrees with physiological evidence for shape selectivity in PFC (Pasupathy et al., 2015).
The two-step inference on the V4 responses accurately predicts the response characteristics of the initial and the delayed peaks in experimental recordings of V4 neurons. While the responses of V4 unit 2 (the neuronal unit not preferring the test shape A) stay constant at a low rate across the occlusion levels, V4 unit 1 (the preferred V4 unit) shows a decreasing response pattern as occlusion increases (i.e., as unoccluded area decreases). Compared to the responses inferred only based on the feedforward sensory input (see Figure 4D, solid green), the firing rates are less dependent on occlusion level when the feedback predictions are included (see Figure 4D, dotted green). Thus, with the feedback, an increase in occlusion does not as extensively degrade the preferred V4 responses. The model predictions therefore agree with the experimental observation on the two transients in V4 (see Figure 2B) and are in accordance with our hypothesis that the initial V4 responses reflect the feedforward signals from the afferent areas and the delayed peak responses in V4 are computed based on both the feedforward sensory signals and the feedback predictions from PFC. Because the response of the preferred V4 unit becomes resistant to occlusion when the feedback prediction is included, we say that the feedback enables V4 neurons to have enhanced shape discriminability under partial occlusion.
We note that in contrast to other sensory areas (Rao & Ballard, 1999; Srinivasan et al., 1982) where this comparison has been successfully made, the responses of shape-selective V4 neurons are not accurately described by a direct comparison to the residual errors between the feedback predictions and the neuronal responses underlying the current estimate of the sensory signals. Here, the residual error of V4 units 1 and 2 increases with added occlusion (see Figure 5), unlike the activity of shape-selective V4 neurons in experiments, which were strongest for unoccluded stimuli and weaker with occlusion (see Figure 2B; Kosai et al., 2014). Instead, we identify the optimal estimates shaped by the feedforward input and the feedback predictions as the neuronal responses measured in V4, which do replicate response characteristics in experiments. Unlike the residual errors, which reflect novelty of the sensory inputs, the optimal response representation conveys both sensory stimulus features and stimulus novelty.
Finally, the group of neurons that are hypothesized to respond preferentially to occluder saliency exhibit increasing responses as occlusion increases, both with and without the feedback (see Figure 4E). Although this class of neurons has not been systematically recorded in experiments, we identified several neurons with increasing responses to added occlusion regardless of the copresented shapes (see Figure A.1B) similar to the response patterns of V4 unit 3 (see Figure 4E).
In the above, we have compared the steady-state representation of neuronal responses in the model to transient peaks of responses in the experiments. The two-step inference does not have a mechanism for the shape of the transient activities observed in experiments. Specifically, instead of having the brief suppression of responses between the initial peak and the delayed peak (see Figure 2A), the gradient descent on (see equation 2.16) and (see equation 2.17) with respect to simply predicts the V4 response dynamics to reach and stay at the respective steady-state firing rates, which minimize and .
The gradient descent dynamics are shown in Figure C.1, where the feedback prediction term from PFC is included in the cost function after neuronal responses reach the steady-state firing rates minimizing . Before optimization of the full cost function starts, the responses may be brought down to the baseline firing rate for a brief interval rather than being continued from the values optimizing . In either case, the optimal responses measured at the end of optimization process of do not change (see Figures C.1B and C.2D), indicating that those values are robust within the range of firing rates we consider. Note that unless the responses are deliberately suppressed, the gradient descent dynamics do not exhibit transient peaks as observed in experiments. This implies that there may be additional physiological mechanisms in the cortical circuitry responsible for the transient dynamics. In principle, it is also possible that such temporal effects could be interpreted by extending the predictive coding to the temporal domain (Rao & Ballard, 1999; Friston & Kiebel, 2009a, 2009b).
In summary, in this section, we asked how the responses in a hierarchical predictive coding model compare to physiology. We find that upon training, the model indeed predicts the observed responses in V4 and PFC, when the dynamics unfold over an initial feedforward and a second feedback stage.
3.5 Parsimony of the Network Structure
In the simulations above, we have assumed a specific network structure. This poses the question of whether these assumptions were necessary and, in general, what aspects of network structure are required to reproduce the observed physiological responses.
Shape selectivity in V4 and PFC neurons is supported by experiments (Fyall et al., 2017); thus, we included the test shape preferred and nonpreferred V4 and PFC units: V4 units 1 and 2 and PFC units 1 and 2. Our model also includes an additional group of V4 cells that responds strongly to occlusion. We found that such occluder-selective V4 neurons are necessary to capture the response characteristics of PFC neurons observed in the experiments. Since the second term in the cost function, equation 2.14, is the squared difference between the PFC predictions—a linear combination of PFC responses—and the actual V4 responses, the PFC responses minimizing the cost function tend to follow the response trends of the afferent V4 neurons. The shape A (test shape)–preferred V4 unit 1 exhibits monotonically decreasing firing rates as the occlusion level increases, while the activity of the shape B–selective V4 unit 2 stays constant across degrees of occlusion as a consequence of the bottom-up stimulus-dependent inputs. With only these two types of neuronal populations, therefore, the PFC responses cannot capture the firing rate increase induced by occlusion. Given our model architecture without any additional mechanisms, there has to be a class of V4 neurons that responds strongly to occlusion but only weakly to unoccluded stimuli, so that PFC follows the similar response trends. Moreover, we found that the increase in PFC responses with occlusion cannot be obtained by including a simple prior distribution of PFC responses in the cost function instead of the third class of V4 neurons in question (the only way to have a prior implement the observed changes in PFC responses would be to have that prior itself change with occlusion level). There are several candidates for types of V4 neurons represented by unit 3. These include populations of neurons that respond preferentially to the color of the occluding dots.
Another feature of our architecture—the convergence of the signals, with each of the PFC cells connected to multiple afferent V4 neurons from different populations—is also critical to replicate the shape-selective responses that become more robust to occlusion after the PFC feedback. We experimented with different architectures and found that such convergence is crucial for transmitting information between different V4 units. Unless the same PFC unit makes predictions about both the shape-preferred V4 unit (V4 unit 1) and the occluder-selective V4 unit (V4 unit 3), the information about the occlusion level encoded by the occluder-selective V4 unit will not be transmitted to the shape-selective V4 population, which is crucial for maintaining robust shape discrimination and weaker dependence on occlusion. This structure, where the neurons of the lower cortical areas with different tuning properties send convergent signals to neurons in higher cortices, agrees with physiological findings in which signals become more mixed as they travel along the hierarchy (Felleman & Van Essen, 1991; Rigotti et al., 2013; Fusi et al., 2016).
Another feature of our model is that fewer units in PFC (two units) combine to make linear predictions about the responses of a larger number (three units) of V4 units. This is also necessary to capture the experimental data. Without such convergence, the V4 responses imposed by the bottom-up sensory input can be matched perfectly by the top-down predictions made by PFC units, leading the optimal predictive coding solution to make identical copies of the sensory input at each stage along the hierarchy—which clearly does not occur in experiments. Translating this constraint into biology does not mean there must be fewer neurons in higher areas of brain, but rather that there are fewer functional or active populations that can be grouped as single units in the higher area during the task.
In our model, information about the shape identity and the occlusion level is input to V4. The system implements a feedforward-feedback loop involving the higher-area PFC to enhance shape discriminability under occlusion, as illustrated in a state-space view in Figure 6. Without the feedback predictions, during the initial responses, high occlusion moves noisy versions of the responses close to or even above the unity line, obscuring the shape identity (see Figure 6A). However, when the feedback from PFC is included, the responses move away from the unity line, thus clarifying the shape identity under partial occlusion (see Figure 6B). The convergent structure of the network is the key for this effect to occur. Although information about occlusion is initially present at the level of V4, it does not affect the shape-selective V4 units without feedback from PFC. In other words, PFC predictions remap the information about the shape identity and the occlusion level onto the shape-selective space in V4, enhancing the shape discriminability there.
We note that recurrent connections among V4 populations—rather than the feedback described—in principle could also transmit information about the occlusion level to the shape-selective neurons. Which mechanism is more effective and efficient is an open question. However, the current experimental evidence showing the delayed peak of responses in V4 arising after PFC responses peak, as well as the strong PFC responses to occlusion, is suggestive of feedback.
In summary, the proposed network, composed of two PFC units and three V4 units, has a parsimonious structure to explain the neuronal responses in the experiments under predictive coding principles.
3.6 Structure of Inputs to V4
In the simulations, we have assumed a simple input structure, where the sensory input is determined by probability distributions of V4 responses conditioned on shape identity and occlusion level . Based on experiments, we model for the test shape–preferred V4 unit 1 to decrease from a high firing rate as occlusion levels grow, for the nonpreferred V4 unit 2 to stay at a low baseline firing rate and for the occlusion-preferring V4 unit 3 to increase. To provide a firmer basis for this, we show several example V4 neurons in Figure A.1. Example cells in Figure A.1A behave like V4 units 1 and 2 with decreasing responses to preferred shapes under added occlusion and overall low responses to nonpreferred shapes. The cells shown in Figure A.1B may correspond to V4 unit 3, which displays relatively low, shape-selective responses to unoccluded shapes and increasing responses to both preferred and nonpreferred shapes as occlusion level increases. The population-averaged initial peak responses of V4 neurons to preferred and nonpreferred shapes further support our implementation of (see Figure 7A). Specifically, the averaged responses of V4 neurons with clear two transient peaks and shape selectivity exhibit a decreasing response pattern to preferred stimuli with added occlusion, but responses to nonpreferred shapes stay at a constantly low firing rate across the range of occlusion levels.
In addition to the peak firing rate , another component that forms the input signals is the variance of the V4 response distributions given the sensory input. As discussed earlier, we hypothesize that increases with added occlusion as high degrees of occlusion obscure the shape identity and that is constant across degrees of occlusion, since random placements of occluding dots on a nonpreferred shape will not introduce as much variability in responses as on a preferred shape. We also modeled to stay constant regardless of occlusion level, as this unit responds to the presence of occluders but not their specific configuration. To test this hypothesis, we examined the consequences of other plausible assumptions for how variance depends on occlusion. First, when variances for all three V4 units increase at the same rate (see Figure 8A), the response characteristics of the shape-preferred V4 unit 1 remain unchanged, but the nonpreferred V4 unit 2 shows increasing delayed responses to added occlusion. Such increasing delayed responses of unit 2 are also obtained when the input variances for both V4 units 1 and 2 are increased with occlusion, while the variance for V4 unit 3 is kept constant (see Figures 8B and 8C). When the variances of all V4 units are decreased with added occlusion, we observe very different response patterns (see Figure 8D). Specifically, feedback does not improve the shape discriminability, as the initial and the delayed responses of V4 unit 1 are identical. Based on these simulations, we limit our model to the cases where introducing occlusion increases the input variances for shape-selective V4 units.
Our simulations show that the experimental results (see Figures 2A, 2B, 7A, and A.1A) are best captured by different rates of variance increase for different V4 units (see Figures 4A and 8C), in particular, when the variances for V4 unit 1 increase with added occlusion, the variances for V4 unit 2 stay constant or increase by a smaller amount compared to V4 unit 1, and the variances for V4 unit 3 stay constant. As a result, shape selectivity under occlusion is consistently improved in the delayed signals (see Figure 8C).
The dependence of variance on occlusion may not be uniquely defined and likely varies among V4 neurons. Indeed, neurons in V4 show a spectrum of different response patterns to nonpreferred stimuli, indicating that different V4 neurons encode input variances in more than one way. For example, the top two rows in Figure A.1A show example V4 neurons whose delayed responses to nonpreferred stimuli do not increase with added occlusion, corresponding to our model simulation with a constant variance for V4 unit 2. The last row in Figure A.1A, however, shows an example V4 neuron with increased delayed responses to occluded nonpreferred shapes. The example V4 cells thus suggest that neurons in V4 may respond to partially occluded, nonpreferred shapes with constant variances or slightly increased variances.
Our original model has only two V4 units of shape selectivity, one tuned for the test shape and the other not preferring the test shape. A more biologically realistic model would consist of a population of V4 neurons with diverse response properties. To construct this population, we first examined response profiles of a population of 109 neurons in V4 previously reported in Pasupathy and Connor (2002). For some shapes, disparity between the responses of the neurons preferring the shapes and those of the nonpreferred neurons is noticeable, as illustrated in the sorted population responses to a given shape in Figure 7B (top panel). However, to many other shapes, the population of V4 neurons show more graded responses, as in Figure 7B (bottom panel).
We next expanded the model network to a larger network with 2 PFC units and 30 V4 units (see Figure 9A). Among the V4 units, 10 are occluder-preferred units and the remaining 20 units are shape selective. Instead of dividing the shape-selective V4 units into test shape-preferred and nonpreferred groups, we modeled the V4 units to have a spectrum of peak firing rates and variances for the input-driven responses (see Figure 9B). For the V4 units that are more tuned to the test shape (corresponding to higher values of ), the input-driven variance increases by larger amounts with added occlusion (see Figure 9B). The connection weights from PFC units to the V4 units are also adjusted accordingly (i.e., so that in Figure 9A, the green PFC unit has stronger connections to the green V4 units compared to the blue V4 units and vice versa).
This expanded model yields qualitatively the same results as the simple network with two shape-selective V4 units. The PFC units and the occluder-selective V4 units show increasing responses with added occlusion (see Figure 9C i and ii), and the shape-selective V4 units yield decreasing response patterns with an increase in occlusion (see Figure 9C iii). Moreover, the delayed responses of the shape-selective V4 units obtained by optimizing the full cost function exhibit reduced sensitivity to occlusion, and the effect is stronger for the units with stronger test shape preference. To see this more clearly, Figure 9C iv presents responses of a selected number of the shape-selective V4 units with high, intermediate, and low degrees of preference for the test shape. In sum, the delayed increase in responses to stimuli under occlusion induced by the feedback, in neurons that respond preferentially to the test shape, is maintained in a population of V4 units with graded response properties, validating the predictions made by our simplified model.
3.7 Differential Weighting of Feedforward and Feedback Inputs
In our model, the relative strengths of feedback and feedfoward interactions are determined by assumptions about levels of variability in the inference errors (the noise terms in equations 2.7 and 2.11) at each network layer (see equations 2.8 and 2.12). Here we ask how these assumptions affect the ability of the model to reproduce trends in experimental data.
Recall that the cost function in our model has two terms: one based on bottom-up sensory inputs and the other based on top-down predictions (see equation 2.4). The contribution of each of these components is weighted by the inverse variance of the respective probability distribution. The pattern of the optimal responses to occlusion can therefore be modulated by these variances. Here we examine how this occurs and show that the trade-off between feedforward and feedback components achieved by the variances in Figure 4 is necessary to capture the response characteristics observed in experiments.
We first discuss the effects of the variances for the bottom-up input-driven distributions. In the original model (see Figure 4), for the bottom-up component, variances are set equal to 1 for all three V4 populations when the input shape is unoccluded. We also set the variance for the test shape-preferred V4 population (V4 unit 1) to increase as occlusion level increases, to capture the increase in uncertainty of the shape identity in the presence of occlusion. We found that this increase in variance for the preferred V4 unit is necessary to mimic its weaker sensitivity to occlusion when feedback inputs are included. Without the increase in variance, this V4 unit depends relatively more on the bottom-up inputs under high degrees of occlusion, and as a result, it shows a steep decrease in its responses as occlusion increases (see Figure 10A, left panel, green). By increasing the variance of the sensory input-dependent distribution, therefore, the optimal response of this V4 population becomes more dependent on the top-down predictions made by PFC. As the PFC populations respond strongly to occluded stimuli, weighting the bottom-up component less will result in a more gradual decrease in V4 responses to increasing occlusion, as in the original model in Figure 4D.
Next, we examine the choice of the top-down variances in the original model that successfully captures experimental data. In the initial model, Figure 4, the variances of the top-down component do not depend on the occlusion level and stay at constant values. However, the top-down effect is differentially weighted for each of the V4 populations; it is weighted more for the occluder-selective V4 population () compared to the shape A- and B-selective neurons (). This is needed to reproduce the rise in PFC responses at higher levels of occlusion. The smaller variance, or equivalently, more weight, on the top-down predictions of the occluder-selective V4 unit drives the PFC unit to follow the same increasing response pattern as the occluder-selective V4. The smaller variance imposed on the top-down prediction for the occluder-selective V4 unit can be interpreted as the top-down predictions having more significance for occlusion than for identity of the shape.
We investigated effects of changes in the top-down variances on the response patterns. When the feedback prediction-driven distributions for all V4 units are uniformly weighted with unit variance, the top-down effect becomes more pronounced (see Figure 10B) compared to the case with the variances at the original values (see Figure 4D). As a consequence, the delayed responses of the test shape-preferred V4 (V4 unit 1) increase with added occlusion, reflecting strong modulation by PFC (see Figure 10B, left panel, green dotted line). Similarly, when the top-down variances on all three V4 units are set to be larger than the bottom-up variances, relatively more influence is exerted by the bottom-up drive (see Figure 10C). As a result, the feedback no longer increases the robustness of V4 unit 1 responses under partial occlusion (see Figure 10C, left panel, green dotted line).
In sum, we have shown that the ability to reproduce trends in experimental recordings in our predictive coding model requires the balance of top-down and the bottom-up influences that is given by the increase in the input-dependent variance with added occlusion for the test shape-selective neurons and the smaller variance in the top-down prediction on the occluder-selective neurons.
3.8 Model Prediction for Responses to Nonsalient Occlusion, Noise, or Reduced Contrast
We have assumed that occlusion is salient and that there is a separate population of cells in V4 that responds preferentially to occlusion. But what happens to predictions of the model when the occlusion is nonsalient—that is, indistinct from the shape? To answer this, we consider the case where the occluder reduces the shape signal but does not activate a dedicated class of V4 neurons. For example, when the occluders are of the same color as the shape or the background, occlusion would increase the ambiguity of the shape identity but would not induce responses in a V4 population separately responsive to a distinct color. Other examples include a decrease in shape clarity by white noise or reduced contrast (illustrated in Figure 11B).
We simulated such nonsalient occlusion and ambiguity in our model by setting , and therefore the peak of the response distribution for the occluder-selective V4 conditioned on sensory stimulus, to a constant. Therefore, an increase in occlusion or ambiguity in the shape stimulus does not increase the responses of V4 unit 3, as shown in Figure 11E. The peak for the shape A-preferring V4 unit, however, is assumed to decrease with occlusion, as for previous simulations. Note that such neuronal behaviors are assumed because the occluder-selective V4 unit 3 is not modeled to specifically detect occlusion, but rather to respond to some occluder-specific feature such as a distinct color with contrast relative to background. This results in a decrease in the preferred PFC responses with occlusion or ambiguity and only a slight increase in the nonpreferred PFC responses (see Figure 11C). Therefore, the feedback predictions made by PFC do not increase the preferred V4 unit 1 responses when the shape ambiguity (occlusion level) is high. In Figure 11D, the preferred V4 responses after the feedback (dotted green) are therefore indistinguishable from the responses before the feedback (solid green). Our model thus predicts that when the shape signal is reduced in a way that is not salient, the feedback from PFC does not improve shape discriminability.
From the point of view of perception, this prediction seems plausible since we often have more difficulty recognizing an object when the obscurant is not distinct from the object. Moreover, preliminary experimental observations show that PFC neurons do not respond strongly to occluders of the same color as the background. In addition, the responses in the second peak were not observed in V4 neurons when the shapes were obscured by reducing their contrast. While these preliminary observations are in accordance with our model predictions, more data should certainly be collected before conclusions can be drawn.
In this study, we have proposed that robust shape-selective V4 responses under partial occlusion can be explained in the framework of predictive coding and hierarchical Bayesian inference. We have used this framework to construct a model of V4 and PFC in which signals converge as they travel up the hierarchy. In particular, we suggest that top-down predictions made by PFC neurons with mixed selectivity for shape identity and occlusion play a significant role in maintaining robust shape discriminability under salient partial occlusion in V4. In this model, PFC neurons make linear predictions on V4 activities in the form of feedback signals, and the connection weights are interpreted to store the memory of the shape identities. We reformulated the traditional framework of predictive coding, so that the optimal representation of the internal states of the model V4 and PFC units, rather than residual errors, are comparable to the electrophysiological recordings in these areas.
Our model suggests that the initial responses in experimental recordings of a class of V4 neurons are purely feedforward and computed solely based on the bottom-up sensory input, while the delayed responses are modulated by both the bottom-up sensory signals and the top-down predictions. The model further shows that the feedback signals in V4 improve the shape discriminability under occlusion by reducing ambiguity in the population representation of the shape identity and that this is achieved by transmission of the occlusion information via a feedforward-feedback loop. This can be viewed as an extension of the concept proposed in Rao and Ballard (1999) where predictions made by higher visual areas with larger receptive fields enable neurons encoding the surround and the center in V1 to share information. In our model of V4, neurons encoding different features of a shape stimulus such as curvature or color share information by predictions made by the higher areas.
The increase in the shape-selective responses of V4 induced by the feedback depends on asymmetric weighting of the top-down and the bottom-up effects, so that the top-down prediction is weighted more strongly for the occluder-selective neurons and the dependency of the shape-selective neuronal responses on the sensory input decreases with added occlusion. Interesting future work could more directly test this weighting of the top-down and the bottom-up effects. For example, the top-down predictive component of our model would be weakened by training with a larger set of noisy shape stimuli under various degrees of occlusion, which will introduce larger variance terms in the feedback prediction. Model predictions for experiments where partially occluded shapes are used for training are given in appendix D. Our simulations predict that when training is done with partially occluded shapes, the V4 neurons do not exhibit a delayed increase in shape-selective responses, underlining the significance of initial exposure to unoccluded shapes (see appendix D). If the variances (, ) are allowed to be learned as well, using a noisy stimuli set under various degrees of occlusion for training will weaken the top-down influence by increasing the top-down variance . Alternatively, cooling PFC is another way that more emphasis would be placed on the bottom-up sensory input. Overall, our model predicts a smaller or no increase in shape selectivity during delayed V4 responses in these cases where the effect of the top-down predictive signals is reduced.
For the input-driven response distributions, we have assumed the mean and the covariance to depend linearly on occlusion level. While this assumption keeps our model simple, detailed neuronal responses may be captured more accurately by implementing nonlinear dependence. For example, the example V4 cell in Figure 2 shows the maximum delayed increase in shape discriminability when the occlusion level is intermediate. However, in our model, the separation between the initial and the delayed V4 responses increases monotonically with added occlusion, more resembling example V4 cell 3 in Figure A.1A. The response pattern in Figure 2 can be reproduced by nonlinear dependence on occlusion of the bottom-up mean and covariance (data not shown). Thus, the variability in detailed response patterns across cells may indicate heterogeneous occlusion-dependence functions of individual V4 neurons.
In this way, our model contributes to a new understanding of both neurophysiological and computational mechanisms underlying discrimination of partially occluded shapes in V4, suggesting a possible functional contribution of feedback signals.
4.1 Relationship to Previous Models
Several previous theoretical studies investigated the computational mechanisms for recognition of partially occluded shapes, patterns, and objects (Fukushima, 1987, 2001, 2005; Rao, 1997). However, these are strictly feedforward and often overlook feedback computation, in stark contrast to biological networks that feature abundant feedback and recurrent connections. One approach is based on an extended version of neocognitron—a hierarchical, multilayered, and feedforward neural network model (Fukushima, 1987, 2001, 2005). This extended neocognitron has an additional masker layer that detects occluders by difference in brightness and suppresses them at an early state. A study by Rao (1997, 1999) uses a Kalman filter model and Bayesian optimal estimation theory of maximizing the posterior probability of the internal states. With robust optimization method that clips large residual errors, the model effectively segments the occluders from the image, treating the occluders as the outlier. The physiological mechanisms underlying the robust optimization method, however, are not known.
There have been a number of other modeling studies of V4 tuning to shape contours based on hierarchical feedforward models of object categorization, which have structural similarity to the ventral visual pathway (Fukushima, 1980; Riesenhuber & Poggio, 1999; Serre et al., 2007; Cadieu et al., 2007; Yamins et al., 2014). These models are also purely feedforward, and while they have had successes in reproducing V4 shape selectivity (Cadieu et al., 2007; Yamins et al., 2014), they lack separate mechanisms to account for occlusion. Unlike these previous models, our model bridges hierarchical predictive coding and experimentally recorded response dynamics in area V4 and PFC.
Our model is focused primarily on the encoding of the partially occluded stimulus, while the underlying behavioral task required animals to report whether the two stimuli presented were the same or different. Other literature has proposed how the sensory representation of the test stimulus may be compared to the memory representation of the reference stimulus (Hayden & Gallant, 2013; Murray, Jaramillo, & Wang, 2017; Romo & Salinas, 2003) to derive a behavioral decision, but this is beyond the scope of this article.
4.2 Learning the Shape Templates with Connection Weights
Our model modifies the synaptic weights between V4 and PFC neurons during the preliminary phase, which consists of a few presentations of unoccluded shapes. This step corresponds to the initial learning phase in experiments where the animal discriminates an unoccluded pair of shapes used for the session. In this setup, the fast learning of the shape pair after exposure to the shapes for just a few times is achieved by the memory stored in the synaptic weights between V4 and PFC neurons. When partially occluded shapes are used during the preliminary phase, the system learns different values of synaptic weights and the feedback does not improve shape discriminability (see appendix D). Fast learning, as attested by the shape discrimination task here, has been observed widely, where new sensory stimuli are easily learned with just a few presentations (Seitz, 2010; Rubin, Nakayama, & Shapley, 1997).
Physiological recordings in cortical cells in vitro, however, show only small changes in synaptic strength after a pair of pre- and postsynaptic spikes (Markram, Lubke, Frotscher, & Sakmann, 1997; Bi & Poo, 1998; Gerstner, Kempter, van Hemmen, Wagner, & Hemmen, 1996), suggesting that neurons learn a repeated stimulus more gradually, after a large number of presentations. Such seemingly contradictory evidence from physiology and behavioral observations can be reconciled by introducing stronger synaptic changes than usually observed in vitro, possibly aided by neuromodulation (Fusi, Drew, & Abbot, 2005). More recently, it has been proposed that even weak synaptic plasticity can support fast learning in the balanced regime of excitation and inhibition (Yger, Stimberg, & Brette, 2015). Due to the leverage effect from the excitatory and inhibitory balance in this regime, small synaptic modifications applied to many synapses onto a given neuron result in a large effect (Yger, Stimberg, & Brette, 2015).
4.3 Mapping Computational Units in Predictive Coding to Cortical Circuitry
Different algorithms implementing hierarchical predictive coding share the general principle of a generative model: the brain has an internal representation of the world that is actively compared to the actual sensory inputs. However, the precise computational procedures employed by these algorithms, as well as their connections to neuronal populations, are controversial and vary widely across different studies (Spratling, 2008, 2017; Bastos et al., 2012; Bogacz, 2017; Rao & Ballard, 1999; Mumford, 1992).
For example, in our model, the variances of the response distributions of different V4 units given the sensory input or the higher cortical activity are predefined to capture the response characteristics in experiments. However, they can also be treated as parameters to be optimized and are assigned to the most likely values, with a slight modification on the network structure as done in a few other models of hierarchical predictive coding. In these studies, the variances are interpreted as synaptic weights and are obtained by minimizing the free energy (Bogacz, 2017; Friston & Kiebel, 2009a, 2009b).
There are varied interpretations on the connections between predictive coding algorithms and computations done by cortical circuitry. Cortical areas have laminar structures, and different layers or populations within the cortical area may correspond to different local computational nodes that arise in predictive coding algorithms. However, there is no unifying description of the intracortical connectivity and the local computations within a cortical area. For example, inhibitory feedback connection implemented in the model proposed by Rao & Ballard (1999), is modified in Spratling (2008, 2017) to reflect excitatory feedback signals observed in physiology. In order to avoid negative responses, Spratling (2008, 2017) also replaced additive excitation and subtractive inhibition in Rao and Ballard (1999) by multiplicative and divisive modulations, respectively. In our model, we follow the approach in Rao and Ballard (1999) and implement additive excitation and subtractive inhibition for simplicity.
Within area V4, there surely are multiple neuronal populations across the laminar structures, and each neuronal node may perform different computations as suggested by earlier studies. In this work, we have shown that V4 neuronal responses to partially occluded shapes are better explained by the optimal representation of responses than the residual errors between the current estimates and the predictions, and thus correspond to the node that encodes the current estimates. However, neurons whose responses are less dependent on their tuning to stimuli features but more sensitive to novelty of the stimulus may correspond to the unit that computes residual errors. Investigations of specific neuronal populations within V4-PFC circuitry in the context of the corresponding computational nodes in the predictive coding algorithm will provide a better understanding and validation of our model.
Appendix A: More Example V4 cells
Figure A.1 presents more example V4 cell responses from experimental data.
Appendix B: Population Average Responses
In this appendix, the model is extended to include populations of neurons with slight heterogeneity. In our main model, each unit in the model V4 and PFC is considered as a population of neurons with similar tuning properties. For example, V4 unit 1 represents a population of V4 neurons that respond preferentially to shape A, and V4 unit 3 is interpreted as a population of V4 neurons responding strongly to some salient features of the occluders. With each unit representing a neuronal population, the optimal response inferred by minimizing the cost function depicts the average response of each neuronal population. Since the cost function in equation 2.14 increases linearly with added neuronal units that share the same properties with the existing populations, representing a neuronal population as a single unit seems a reasonable simplification.
We now test this simplification explicitly. We performed further numerical simulations with a slightly heterogeneous group of neurons for each neuronal unit in V4 and PFC. The heterogeneity is introduced to the V4 neurons by assigning , the mean vector of the feedforward sensory input-driven response distribution, from a normal distribution with a unit standard deviation for each neuron within the population. Therefore, the bottom-up sensory input drives neurons within the same group to converge to slightly different optimal responses. In addition, the initial connection weights and the initial firing rates of the neurons are also slightly heterogeneous, chosen from normal distributions with the means at the initial values used in previous simulations and the standard deviations of 0.1 for initial weights and 0.5 for initial firing rates. The PFC neurons in each population therefore also show weakly heterogeneous optimal representations as a result. Each neuronal population is composed of 10 slightly heterogeneous neurons. Moreover, each PFC neuron sends the prediction signals to one neuron from each of the three V4 populations, and each V4 neuron receives the feedback that is a weighted sum of two PFC neurons, one from each PFC population. Therefore, the convergence ratio from V4 to PFC is preserved as in the previous simulations where populations are represented as single units. We have tested a couple of other convergence ratios (e.g., two neurons per each V4 population connected to a single PFC neuron) and found that they produce qualitatively similar results.
The connection weight matrix is learned during the preliminary phase, and the optimal responses of the V4 and PFC neurons with the learned weights are obtained by minimizing the cost function using the same method as in the previous simulations with single unit representation. Figure B.1 shows the averaged inferred responses (dots) and the standard deviation (bars) within the population of the shape A-selective V4 neurons (green) and the shape B-selective V4 neurons (blue) before (solid line) and after (dotted line) the feedback predictions, as a function of unoccluded area. Figure B.1C illustrates the same results in a state-space view for the responses before (left) and after (right) the feedback. The inferred responses of the neurons in the shape A-selective (V4 unit 1) and the shape B-selective (V4 unit 2) populations, predicted by the common PFC neurons, are projected onto the 2D space of the shape A- and shape B-selective population responses. The level of occlusion is indicated by the color bar, and the yellow line represents the population average responses. The population responses shown in Figures B.1B and B.1C match the results from the single-unit representation model in Figures 4 and 6; the shape discriminability increases during the delayed responses when the feedback predictions are included. Although not shown here, the population average responses of the PFC populations and the occluder-selective V4 population also agree with the previous results in Figure 4.
Treating each population as a single unit as done in Figure 1 is therefore a reasonable simplification of the model, which expedites computation while maintaining the core mechanisms of the model. Furthermore, there may be recurrent connections among the neurons within the same group, which reduce the variances among these neurons and further validate representation of these neurons as a single unit.
Appendix C: Gradient Descent Dynamics of Neuronal Firing Rates
Figure C.1 displays the gradient descent dynamics of responses of each neuronal unit in the model network.
Appendix D: Connection Weights Learned with Partially Occluded Shapes
In this appendix, we make predictions on shape discriminability when the synaptic weights store templates of partially occluded shapes instead of unoccluded shapes. This represents the case where the animal has memory of partially occluded shapes rather than being exposed to unoccluded shapes.
In the simulations of the previous sections, the connection weight matrix is learned based on presentations of the pair of unoccluded shapes in order to mimic the experimental procedure where the animals discriminated a pair of unoccluded shapes at the beginning of each trial. Here we test our model with the weight matrix learned from partially occluded shapes. We train the weight matrix on the pair of the shapes under and occlusion (see Figures D.1B and D.1C) and compare the results to the simulation with the weights learned based on unoccluded shapes (see Figure D.1A).
The stronger connection weights between the PFC units and the occluder-selective V4 unit 3 that emerge from training on partially occluded shapes change the response pattern of the preferred V4. Due to the stronger weights, the PFC responses are relatively lower overall. Then the delayed responses of shape A-preferred V4 unit 1 induced by PFC predictions are moved to lower values when the stimulus has no or a low degree of occlusion. As the occlusion level increases, the standard deviation of the preferred V4 increases, weakening the bottom-up influence, which suppresses the V4 responses under occlusion. As a result, under high occlusion, the optimal representation of the preferred V4 unit 1 responses depends more on the top-down PFC prediction reflecting the occluder-selective V4 response pattern.
When the training is based on a pair of unoccluded shapes, the responses of the preferred V4 unit are never lower with the feedback than without the feedback across all occlusion levels, and thus, the feedback enhances the responses under higher degrees of occlusion. When trained on shapes with occlusion, the delayed responses are lower than the initial responses under low degrees of occlusion. For occlusion levels higher than about to occlusion, the delayed responses are higher than the initial responses. When trained on occlusion, the delayed V4 responses are always lower than the initial responses in the occlusion range of to . However, the differences between the initial and the delayed responses of the test shape-preferred V4 unit 1 are very small; the initial and the delayed responses are almost identical to the parameter set used here. In addition, across the range of occlusion levels, the errors between the top-down predictions and the inferred V4 activities are smaller when the weights are trained on partially occluded shapes (see Figure D.1, right column).
The deviation from the initial responses (see the solid line in Figure D.1) is on average smaller for the simulations with weights trained on partially occluded shapes. Since the variance is smaller than and , the prediction tends to follow the response patterns of the occluder-selective V4 unit, which increases with added occlusion. When the weights are trained on unoccluded shapes, the connection from a PFC unit to the V4 unit 1 with the same shape preference is the strongest, while its connection to the occluder-selective V4 unit 3 is weaker. Then the increase in the PFC responses with added occlusion is relatively large, compensating for the effects of the small weights and . The large increase in PFC responses induced by added occlusion can then evoke a larger deviation in the test shape-selective V4 unit 1 from its initial responses. When the weights are trained on partially occluded shapes, compared to the case with training on unoccluded shapes, the weights to the test shape-selective V4 unit 1 are reduced by a little and the weights to the occluder-selective V4 unit 3 increase significantly. Then the PFC responses do not increase as much as the occlusion level increases (the preferred PFC unit responses decrease slightly when trained on occlusion or stay constant when trained on occlusion, while the other PFC unit exhibits increasing responses as occlusion increases; data not shown), and thus exert milder effects on the shape-selective V4 units.
In brief, when the connection weights of the network are trained on partially occluded shapes, the feedback from PFC does not improve the shape discriminability. Our model predicts that seeing the unoccluded shapes and learning them prior to the occluded shape-discrimination task may be a necessary step to benefit from the delayed enhancement of shape discriminability induced by the feedback predictions. Testing this hypothesis will be an interesting future experimental study.
We thank Rajesh Rao, Wyeth Bair, and Joel Zylberberg for many helpful discussions and comments. This work was supported by the Washington Research Foundation Innovation Postdoctoral Fellowship in Neuroengineering to H.C., NEI grant R01EY018839 to A.P., NSF Career Award DMS-1056125 to E.S-B, Vision Core grant P30EY01730 to the University of Washington, and P51 grant OD010425 to the Washington National Primate Research Center. E.S-B wishes to thank the Allen Institute for Brain Science founders, Paul G. Allen and Jody Allen, for their vision, encouragement, and support.