Temporal Binding and Segmentation in Visual Search: A Computational Neuroscience Analysis

The ability to efficiently search and select a target for action is crucial to human survival. Search across space has been much studied, and the conditions supporting both efficient and inefficient search have been established in terms of the similarity and combinatorial relationships that distinguish targets from distractors (Treisman, 1998; Wolfe, 1994; Duncan & Humphreys, 1989; Treisman & Gelade, 1980). Search across time has been studied largely using the preview procedure. Under preview conditions, participants may be asked to carry out a conjunction search task (blue H target vs. blue O green H distractors), but unlike standard conjunction search, one set of distractors is presented before the second set of distractors plus the target (when present; see Watson & Humphreys, 1997). Although standard conjunction search is typically difficult, with search slopes of the order of 225 msec/item or more, preview search can be highly efficient—with slopes equated to when the displays contain only the second set of distractors and the target. Temporal differences between stimuli can be used to guide attention efficiently to just the new set of items.

Although preview search is a relatively simple procedure, the evidence suggests that several factors contribute to performance. For example, mere temporal segmentation alone is insufficient to explain target selection because it requires relatively long time intervals between the preview and the search display to optimize search (e.g., of the order of 400 msec or more; Humphreys, Olivers, & Braithwaite, 2006; Watson & Humphreys, 1997)—much longer than those required for temporal segmentation (Yantis & Gibson, 1994). On top of temporal segmentation then, the data suggest that participants actively suppress the previewed items when they prioritize search for the target. This means that probes are difficult to detect when presented on old previewed stimuli compared with when they fall on new items and even relative to when they fall in the background of the search displays (Humphreys, Stalmann, & Olivers, 2004; Watson & Humphreys, 2000). This suppression of old items was not found when the primary task was probe detection and search for the new targets was not prioritized, consistent with the effect stemming from a top–down set to search for the new items and ignore the old. There is also evidence that preview search can benefit from active expectancies developed for the features of the upcoming search targets (Braithwaite & Humphreys, 2003). For example, if the target is in a small set of new items carrying the same color as the preview, target selection can be difficult but this effect is reduced if participants know the color of the upcoming target. Another factor is attentional capture by the onsets that define the new search items. Donk and Theeuwes (2001) argued that the benefit to preview search was eliminated when the new stimuli were not defined by luminance onsets—although further work has shown that new onsets are not critical providing there is a sufficiently long time interval between the preview and search display (Braithwaite, Humphreys, Watson, & Hulleman, 2005).

Other work highlighting the role of visual onsets in preview search comes from studies examining how the temporal relations between the old and new items modulate preview search. Kunar, Humphreys, and Smith (2003a) presented preview displays for around 450 msec and followed them by a brief offset of the items before all the search display appeared together (the initial items and the new search stimuli, respectively at old and new locations). Although the temporal interval between the old and new items was sufficient to generate temporal segmentation and to enable the old items to be suppressed, the preview benefit was abolished. Kunar, Humphreys, and Smith (2003b) argued that the benefit was eliminated because all the display items grouped by temporal onset when they reappeared together and that this disrupted effects of temporal segmentation and/or the inhibition of old distractors. Interestingly, though, Kunar et al. (2003b) also showed that the reintroduction of a brief preview before the new search items reinstated the search benefit, though the brief preview itself was insufficient to generate a gain in search efficiency relative to a standard conjunction search. Thus, the second, brief preview seemed to reinstate the latent presence of the old preview, perhaps by breaking up the onset-based grouping of the old and new items.

These studies of the preview benefit in search provide important constraints for models of how visual attention operates in time as well as space. One attempt to model such effects using a biologically plausible framework was put forward by Mavritsaki, Heinke, Allen, Deco, and Humphreys (2011). The spiking search over time and space (sSoTS) model used biased competition to select targets. The framework for the original model is set out in Figure 1. sSoTS employed spiking level neural dynamics, with time constants matching those found in real neural systems. The model contained initial feature maps, which we assume exist at intermediate stages of vision, that are activated by the presence of particular visual features at particular locations. The maps were composed of both excitatory units and inhibitory units, and the inhibitory units acted to damp down activity when there were multiple items containing the same features (similar to lateral inhibition). These feature maps interacted with a saliency map, which reflected the presence of any feature at different locations, with the strength of the activity based on the strength of the sensory signal. In addition, expectancies for given target properties were set by selectively preactivating feature maps or by selectively activating a location in the saliency map. These expectancies bias selection to favor either particular features or a particular location. Target detection was based on activity within the saliency map reaching a set threshold when compared with the activity in the locations occupied by distractors. For this, we calculate an attention index (IndA) based on Luce's (1959, 1977) choice theorem (for more details, please see Mavritsaki et al., 2011, p. 40).

Targets differing in their features from distractors can be detected efficiently based on the selective activation of their feature map and mutual inhibition within the feature maps for distractors. Selection of a target defined by a conjunction of features is less efficient, however, because there is no differential inhibition of any of the feature maps which are mutually activated by targets and distractors. Mavritsaki, Heinke, Humphreys, and Deco (2006) showed that the time for a conjunction target to be detected increased with the number of items competing for selection and the slope of the search function for conjunction targets was around twice that for feature targets, matching data from human search (Treisman & Gelade, 1980).

One interesting aspect of sSoTS is that the spiking neurons show adaptation after firing, reflecting the build-up of Ca²⁺ over time. This means that, within the saliency map, neurons should reduce in activation after reaching their threshold, effecting a drop in “attentional interest.” The sluggish time course of this effect predicts the slow time course of preview search, where old items lose their attentional interest more slowly than predicted by temporal segmentation alone (Watson & Humphreys, 1997). In addition, to model the evidence for top–down suppression of old items, Mavritsaki et al. (2006) introduced an inhibitory parameter that could be applied to the locations of items that were to be ignored in search. Armed with the adaptation and suppression mechanisms, Mavritsaki et al. (2006) showed that preview benefits emerged in search for conjunction targets, with the slope of the search function for preview targets matching that obtained when only the new items were presented. The preview benefits followed a similar time course to that found in human search. In addition, by convolving activity in the model with an assumed hemodynamic response function, sSoTS was able to simulate fMRI data on preview search (e.g., Allen, Humphreys, & Matthews, 2008), with activity in the posterior parietal cortex correlating with the predicted activity within the saliency map (Mavritsaki et al., 2011).

In the present article, we ask whether the sSoTS model is able to capture the finer-grained aspects of the time course of search, as studied by Kunar et al. (2003a). Can the effects of introducing a temporal gap between a preview and the presentation of all the search items disrupt the preview benefit? Note that it is not clear that this will necessarily be the case, given that the adaptation process should be set in play even when items are removed from the visual field, so a preview benefit may still be predicted. This was assessed in Part 1 of the article, and we show that a new temporal binding process needs to be built into the model to capture the effects of the temporal gap (the binding-spiking search over time and space [b-sSoTS] model). In Part 2, we examined how well this extended model was able to simulate an additional set of data based on the effects of brain lesions on preview search. Olivers and Humphreys (2004) evaluated the effects on search of lesions to the posterior parietal cortex. They found that not only was conjunction search relatively disrupted when compared with feature search (see also Eglin, Robertson, & Knight, 1989; Riddoch & Humphreys, 1987) but preview search was also impaired. Interestingly, for targets falling on the contralesional side, preview search was worse even than search when a temporal gap was introduced between the preview and the search display. This is the opposite of the result found in normal participants. Olivers and Humphreys (2004) argued that the posterior parietal patients had reduced sensitivity to the onsets of the new items and showed weaker effects of temporal segmentation. In this case, the temporal gap enhanced segmentation without performance being disrupted by the old and new items on setting together. In Part 2, we evaluated if the b-sSoTS model could capture this opposite pattern of performance after brain lesion. We compare the data patterns generated by the model with those found in human participants and reported in Kunar et al. (2003a) and Olivers and Humphreys (2004)—focusing on changes in the effects of display size across the different conditions.

The sSoTS model is composed of excitatory spiking neurons that represent pyramidal cells in the brain and inhibitory spiking neurons that represent interneurons. There are three layers in the network; in each layer, the number of excitatory (pyramidal cells) and inhibitory (interneurons) neurons follows the ratio that is usually found in human brain (80:20; Abeles, 1991). Two layers represent the feature maps for the search asks we used, one representing the shape of the items and the other the color. The third layer represents the saliency map, where the units respond to outputs coming from particular locations in both feature maps. Each map has inhibitory and excitatory neurons separated in different pools; the inhibitory neurons provide global inhibition to the neurons in the excitatory. Here we can think that the shape feature map has layers for the H and A stimuli and the color map has layers for the colors blue and green, matching the items in the experiments (see Olivers & Humphreys, 2004). For each dimension, there were six pools that represented the six possible positions that an item could be presented on the visual field. Each layer in the maps also has one pool of neurons that add noise into the system (the “nonspecific” pool). The saliency map received forward connections from each feature map and also had projections back to the feature maps. More details on the organization of the model can be found in Figure 5 in Mavritsaki et al. (2011).

The sSoTS model is composed of 5000 neurons distributed across the three layers. Because of limited computational power, it would have been impossible to be able to identify the model's parameter with so many equations to solve for each neuron. To solve this problem, a “meanfield approach” was employed (Mavritsaki et al., 2011; Deco & Rolls, 2005; Brunel & Wang, 2001). In this approach, the activation of groups of neurons is represented by a transfer function based on a number of approximations (these approximations can be found in Figure 1B). The mean-field approach reduces significantly the computational power needed to identify the model's parameters, because there is no longer a need to solve so many equations for each neuron but just one for a group of neurons.

The basic network parameters (e.g., the weight from the inhibitory pool to the feature dimension pools) were identified using the mean-field method, and then the remaining parameters were set by hand in the spiking level. These are set out in Table 1.

The model was set to simulate three search tasks: single feature search (blue H target vs. blue A distractors), conjunction search (blue H target vs. green H and blue A distractors), and preview search (preview: green H distractors followed by blue A distractors and blue H target). There were two display sizes (4 and 6 items), and the target was positioned equally often in the six potential locations. There were also four preview conditions: (1) the preview was presented for 450 msec and remained in place when the new search items appeared (standard preview); (2) the preview was presented for 450 msec and then removed for 450 msec before being presented again at the same locations with the new search items (preview gap); (3) the preview was presented for 300 msec and remained when the new items appeared (short preview); (4) the preview was presented for 450 msec, removed for 450 msec, and then re-presented for 300 msec, and it remained in place when the search items appeared (the “top-up” preview condition; see Kunar et al., 2003a). In our prior work, we showed that the preview needed to be represented for at least 450 msec to generate a preview benefit in search (Mavritsaki et al., 2011). The 300-msec preview should thus be too short to fully establish a preview benefit. The question then is whether the effects of this short preview may be enhanced from the earlier memory of the preview, when presented for 450 msec. In addition, we assessed if the preview benefit was lost if a temporal gap was inserted before the presentation of the new search items. There were 100 runs of each simulation (note that the results will vary as a function of the noise in the model), and from these simulations, we randomly created groups of 20 runs that were grouped to form a single “participant”

The results were analyzed using both the RTs and a response efficiency index based on the mean RT/accuracy (see Townsend & Ashby, 1983) to take the accuracy of response into account alongside response latencies. The basic pattern of the results did not change fundamentally as a function of the measure used.

The model was run in four conditions: single feature (blue H target vs. blue A distractors), conjunction (blue H target vs. blue A and green H distractors), standard preview (450 msec green H followed by blue H target and blue A distractors), and preview gap (450 msec green H, 450 msec interval, then blue H target vs. blue A and green H distractors). The results were analyzed by contrasting the RTs for the conditions of interest. In all but the few cases noted below, the efficiency data followed the RTs. Where the efficiency data did not (e.g., due to the error data contrasting with the RT effects), we report the results for efficiency alongside those for RTs. The figures show the efficiency data to make data reporting compact (combining RTs and accuracy).

RTs were faster for the single feature condition (F(1, 9) = 269.04, p < .001), and there was a smaller effect of display size for the single feature versus the conjunction condition (F(1, 9) = 13.98, p < .005). For the RTs, the slopes of the functions were 17.96 and 57.03 msec/item, respectively.

RTs were overall slower for the standard preview condition (F(1, 9) = 63.07, p < .001), but the conditions did not differ as a function of the display size (F(1, 9) = 2.02, p = .189, for the interaction of condition and display size). The slope of the search function for RTs was 21.56 msec/item for the standard preview condition.

There was an advantage for the standard preview over the conjunction condition, both in terms of overall RTs and the slopes of the search functions (F(1, 9) = 356.17 and 14.01, p < .001 and p < .005, respectively).

There was an overall RT advantage for the single feature condition (F(1, 9) = 65.40, p < .001), but no difference in terms of the search slopes (F < 1.0). The search slope for RTs in the preview gap condition was 34.56 msec/item.

There was a benefit for the preview gap condition both in terms of overall RTs and in terms of the slopes of the search functions (F(1, 9) = 423.0 and 18.29, p < .001 and .005, respectively).

There was a small advantage in terms of overall RTs for the preview gap condition and a trend also for search to be more efficient in the preview condition (F(1, 9) = 7.02, p < .05, and F(1, 9) = 4.62, p > .05).

The simulations show that the sSoTS model was able to simulate human visual search for single feature, conjunction, and preview targets, replicating prior data (Mavritsaki et al., 2011). This is important because it demonstrates how, in a model using biologically plausible parameters and spatially parallel processing, both efficient and inefficient search patterns can be captured, based on differential competition for visual selection. The conjunction condition is more difficult than the single feature condition because the green H distractors compete for selection along with the blue A distractors. In the preview condition, the competition from the green H distractor is reduced when the interval between the items is sufficiently long for adaptation to occur and also for top–down inhibition to be applied. The result is that the slope of the search function is matched to the single feature baseline, even though overall RTs are slower. This matches data from preview search in humans (Watson & Humphreys, 1997).

The new result here is in the preview gap condition, when a temporal interval is introduced between the preview and the search display. In the model, performance was slightly but not dramatically disrupted in the gap condition compared with the single feature and standard preview conditions, and the slopes of the search functions did not differ across these conditions. This occurred because the processing units remained adapted and because top–down inhibition continued to be applied so that previewed distractors were effectively ignored. Given that in the human studies the conditions are typically blocked so that participants know that the previewed items would remain in place and still be irrelevant when the search display appeared, there seems no reason to think that top–down inhibition would stop being applied. There is also evidence that this inhibition is not reset simply by the old items offsetting. Kunar et al. (2003b) examined preview search when previewed items were briefly occluded before appearing again with the new search items. A preview benefit was found, despite the previews offsetting and then onsetting again with the new items. Offsetting the previews is not sufficient to eliminate the benefit to preview search. The difficulty here, however, is that the result in the gap condition does not match that found in human search, where the temporal gap was sufficient to disrupt preview search (and indeed push search in this condition back to the conjunction baseline). sSoTS, as originally construed, fails to simulate this aspect of human performance.

In a second version of the sSoTS model (b-sSoTS), we incorporated a mechanism for binding together visual elements with common onset time signals. The proposal here is that early visual areas can register onsets, sending excitatory signals forward to higher-level feature and saliency maps. The parameter for this is given in Table 1. The excitation from two simultaneously activated locations in different feature map further excites the saliency map, making new onsets salient, by increasing their connection through the increase in the feed-backward weight. This is highlighted in Figure 1 by the increased line width between the two activated pools in the blue and A maps and the corresponding position in the saliency map. Applied here it means that, under standard preview conditions, there is enhanced activation from the onsets of the new items, further biasing search against the old (previewed) stimuli. Under the preview gap condition, when the old items are re-presented, there are new onsets for the old as well as the new stimuli. The extra excitatory inputs from the onsets then acts against the adaptation and top–down inhibition effects, so that the preview advantage should decrease.

Unless otherwise mentioned, the conditions and parameters exactly matched those for the sSoTS model, with the single difference being that we introduced temporal binding between the feature maps and the saliency map. The binding parameter (wbind, please see Table 1) was fine-tuned to simulate the behavioral results for single feature, conjunction, standard preview, short preview, preview gap, and “top-up” preview conditions.

The mean RT/accuracy data for the single feature, conjunction, standard preview, and preview gap conditions when binding is used in the model are presented in Figures 2 and 3. The data for the short preview and top-up preview conditions are shown in Figure 4, along with the single feature and conjunction baselines.

We first present the contrasts between the conditions illustrated in Figure 3 before presenting those relating to the conditions in Figure 4.

RTs were faster for the single feature condition (F(1, 9) = 348.63, p < .001), and there was also an effect of display size for the single feature versus the conjunction condition (F(1, 9) = 740.018, p < .001). The slopes of the RT functions were 17.87 and 65.29 msec/item, respectively.

RTs were overall slower for the standard preview condition (F(1, 9) = 120.45, p < .001), but the conditions were not differentially affected by the display sizes (F(1, 9) = 4.45, p > .05, for the interaction of condition and display size). The slope of the search function was 13.23 msec/item for the standard preview condition.

There was an advantage for the standard preview over the conjunction condition, both in terms of overall RTs and the slopes of the search functions (F(1, 9) = 390.57 and 389.66, both p < .001, respectively).

There was an overall RT advantage for the single feature condition (F(1, 9) = 109.76, p < .001) but no difference in terms of the search slopes (F < 1.0). However, in this case there was a confound with search accuracy. For the efficiency measure (RT/accuracy), there was a clear interaction of Condition × Display size (F(1, 9) = 37.90, p < .001). The search slope for the preview gap condition was 60.93 msec/item and greater than for the single feature baseline.

There was a benefit for the preview gap condition both in terms of overall RTs and in terms of the slopes of the search functions (F(1, 9) = 454.68 and 416.44, p < .001, respectively).

There was no overall difference between the preview gap and the standard preview conditions (F(1, 9) = 3.77, p > .05), but there was an interaction of conditions and display size (F(1, 9) = 6.97, p = .03).

The data indicate that, when temporal binding was introduced into sSoTS, the conditions more closely matched those found in human search (Kunar et al., 2003a). The single feature and standard preview search conditions differed in overall RTs but not in terms of search efficiency, and both were more efficient than the conjunction baseline. This agrees with human search data, where single feature and preview search conditions, though they differ in RTs, are both more efficient than the conjunction baseline (Watson & Humphreys, 1997). The introduction of temporal binding did not disrupt the advantages from reduced competition either when one set of distractors was omitted (the single feature baseline) or when distractors were suppressed (by adaptation and top–down suppression, in the preview condition). However, there was a disruption to search in the preview gap condition, when an interval was introduced between the preview and the search display and all the items in the search display onset together. Although there was no difference in overall RTs, there was a big difference in the slopes; the slope for standard preview was 13.23 msec/item, and the slope for the preview gap condition was 60.93 msec/item. This result comes about because re-presenting all the search items again, after the interval, creates a set of common new onsets, counteracting effects of adaptation and top–down inhibition, which otherwise bias search against the old items. The preview benefit decreases. Figure 5 shows activation in the original sSoTS and in b-sSoTS in the preview gap condition. Note that the gain in activity for the target location is slowed in b-sSoTS, reflecting the greater competition for selection when the old items are reactivated by their onsetting together. Figure 6 shows the activation for sSoTS and b-sSoTS for the preview, “short preview,” and “top-up preview” conditions.

In a second set of contrasts, we examined the performance of b-sSoTS when the interval between the preview and the search display was reduced (short preview) and when the short preview was “topped up” by the earlier presentation of the preview gap conditions (top-up preview). Search in these conditions was compared with the single feature and conjunction baselines. In human search, the short preview leads to worse search performance relative to when the preview is displayed for longer (Watson & Humphreys, 1997), and the top-up preview then reinstates the preview benefit (Kunar et al., 2003a).

RTs were overall slower for the short preview condition (F(1, 9) = 1283.67, p < .001), and search was also less efficient (F(1, 9) = 102.54, p < .001, for the interaction of condition and display size). The slope of the search function was 41.74 msec/item for the short preview condition.

There was an advantage for the short preview over the conjunction condition, both in terms of overall RTs and the slopes of the search functions (F(1, 9) = 28.87 and 30.99, p < .01, respectively).

There was an overall RT advantage for the single feature condition (F(1, 9) = 50.11, p < .001) and a small difference in terms of search slopes (F(1, 9) = 5.14, p = .05, p > .05). This slope difference was eradicated when the efficiency measure (RT/accuracy) was employed to correct for speed–error trade-offs (F(1, 9) = 1.79, p = .213). The search slope for the top-up preview condition was 13.88 msec/item.

There was a benefit for the preview gap condition both in terms of overall RTs and in terms of the slopes of the search functions (F(1, 9) = 723.3 and 411.24, p < .001, respectively).

There was an advantage in terms of overall RTs and search slopes for the top-up preview versus the short preview condition (F(1, 9) = 814.00 and 66.77, both p < .001).

The results indicate that, although there was some advantage for the short preview condition over the conjunction baseline, there was then a cost relative to the single feature baseline in terms of search slopes; this is also demonstrated in human search data (Kunar et al., 2003a). In this case, the preview duration was insufficient for the full effects of adaptation and top–down inhibition to modulate search (see Watson & Humphreys, 1997, for data on human search). However, the effects of the short preview could be enhanced by the earlier presentation of stimuli equivalent to the preview gap conditions. Here the “trace”of the initial preview (generating adaptation and top–down suppression) could enhance the effects generated by the short preview under conditions in which the preview items did not onset with the new distractors and target—that is, under conditions in which temporal binding across all the display items was eliminated. The results provide an existence proof that temporal binding may place a significant part in visual search over time and matches data reported by Kunar et al. (2003a).

The lesioning applied in sSoTSb is an extended approach of the lesioning method that was applied in earlier studies of the sSoTS model (Mavritsaki, Heinke, Deco, & Humphreys, 2009). Lesioning was implemented by removing at random a number of neurons from one side of the saliency map (to simulate a unilateral brain lesion; note that, with this procedure, the lesion could be unequally distributed across the different pools). The magnitude of the lesion was varied, from 18.33% of the total number of neurons in each pool removed to 22.78% of the units removed, to simulate different magnitudes of brain lesion. The percentages used were calculated on the basis of the number of units removed from each side, for example, 18.33% means that we removed in average 22 neurons from each pool from one part side of the saliency map. These results do not directly mimic the percentage of lesioned regions found in studies of human neuropsychological patients, but they provide a more general reflection of the effects that can be expected as lesion size varies. The data were compared qualitatively with results from Olivers and Humphreys (2004).

The simulations were the same as those presented in Part 1, except that different numbers of processing units were removed from one side of the saliency map (unilateral lesions). Lesioning was only applied here to the b-sSoTS model, given that this model better accounted for the time course of normal search than the original sSoTS model. The search conditions were (i) single feature, (ii) conjunction, (iii) standard preview (450 msec preview), and (iv) preview gap (450 msec preview and 450 temporal interval), matching the conditions reported in Olivers and Humphreys (2004).

The results are presented in Table 2, where we compare the data for each lesioned case against 10 batches where the model was unlesioned. The data reveal that, in all cases, there were costs to performance after lesioning, particularly for targets falling on the contralesional side of a unilateral lesion (e.g., compared with when targets fell on the ipsilesional side) and particularly in the conjunction and standard preview conditions (please see Figure 7A for contralesional and Figure 7B for ipsilesional). Essentially unilateral lesions produce a spatial imbalance, which means that stimuli on the ipsilesional side dominate the competition for selection. In addition, the lesioning of the saliency map means that the model is less sensitive to the temporal differences between the old and new items on the contralesional side, showing weaker adaptation and suppression of old stimuli. The net effect is that the preview condition is disrupted alongside the conjunction condition. The patterns of activity generating these results are shown in Figure 8A and B.

We also contrasted performance between the standard preview and preview gap conditions for the “control” runs of the model and for the lesioned versions. For this, we took the difference between the slopes of these conditions for the controls and compared it with the difference in the slopes of the conditions for contralesional and then ipsilesional targets when the model was lesioned using a modified t test to compare individual cases against a group (Crawford & Howell, 1998). This was done both for the smallest (18.33) and largest lesions (22.78) imposed on the model to test for generalization of the effects across the different lesion sizes. For contralesional targets, the contrast between the conditions for the lesioned model (where the slope for preview gap < standard preview) differed relative to that found for the control (nonlesioned) version (where standard preview slope < preview gap slope) (t(9) = 6.96 and 6.48, p < .001, for lesions 18.33 and 22.78, respectively). For ipsilesional targets, there were no differences relative to control versions of the model (t < 1.0 and t(9) = 1.01, p > .05 for lesions 18.33 and 22.78, respectively). Here slopes for standard preview < preview gap. The results demonstrate contrasting patterns of performance in these two preview conditions before and after lesioning.

The changes in preview search are of considerable interest. For ipsilesional targets, the pattern of performance matched that found with nonlesioned “control” simulations. There was a benefit to performance in the standard preview condition (relative to the temporal gap condition) due to the suppression of the old, ipsilesional items and the binding of old to new items in the temporal gap case. In contrast, there was a cost to performance for the standard preview relative to the temporal gap condition when targets fell in the contralesional field. This cost reflects the poor temporal segmentation of the preview and new search items in the contralesional field, under the standard preview condition. In this case, temporal segmentation can be aided by the time interval introduced between the preview and the new search items, giving a longer time for the weak adaptation effect to take place. In addition to this, stimuli at lesioned locations in the model show weaker temporal binding, due to the general reduction in activation on the lesioned side. The net result is that there is less of a cost to performance for contralesional targets in the preview gap condition. A double dissociation emerges in which, for controls and for ipsilesional targets in the lesioned model, the standard preview search is better than search under the preview gap condition; in contrast, for contralesional targets in the lesioned model, the opposite occurs. This reversal of the standard pattern of performance matches that found in posterior parietal patients by Olivers and Humphreys (2004).

In this article, we examined whether a model using biologically plausible activation functions could capture the details of the time course of visual search under preview conditions (Watson & Humphreys, 1997). We focused on the patterns of results reported when temporal intervals are introduced between preview and search displays. Kunar et al. (2003a) reported two results of primary interest. First, introducing a temporal gap after a preview display and re-presenting the old and new stimuli together (in a preview gap condition), severely disrupted search (compared with a standard preview condition in which there was no gap). Second, when a preview is too short to generate a full benefit to performance, a benefit can be reintroduced by “topping up” the short preview (by having the preview gap condition precede the short preview). Kunar et al. (2003a) account for the results by suggesting that search was affected by temporal binding between the preview and new items, when these items appear together (in the preview gap condition but not in the top-up preview). This binding brings the old items into competition for selection, disrupting search in the preview gap but not in the top-up condition.

In Part 1 here, we contrasted simulations run with the sSoTS neural network model with a new variant of that model in which temporal binding was introduced (the b-sSoTS model). Binding took place when there was new input activity given to the feature units, which triggered excitatory activation between these units and units in the saliency map (Figure 1). The original sSoTS model made the incorrect prediction that search should remain efficient in the preview gap condition, because both adaptation following the firing of the saliency neurons and top–down inhibition of the old stimuli should still operate across the temporal gap. In contrast, the b-sSoTS model did correctly predict that the preview gap condition should disrupt performance, because binding triggered by the common onsets of the old and new items worked against the effects of adaptation and top–down suppression, once again introducing competition from distractors for selection.

The b-sSoTS model was also successful in accounting for the relations between a short preview and top-up preview conditions (when the short preview was preceded by the preview gap conditions). When the preview is too short, search is not maximally efficient because insufficient time has passed for the full effects of adaptation and top–down suppression to take effect in the model. However, the initial presentation of the preview gap condition means that there is lingering adaptation and top–down suppression in the model, provided that it is not offset by temporal binding between the old and new items. The top-up condition prevents this temporal binding because the reintroduction of a short preview means that the old and new items do not bind. The model matches the data reported by Kunar et al. (2003a).

In Part 2, we applied the b-sSoTS model to data reported by Olivers and Humphreys (2004) on the effects of posterior parietal damage on preview search. Olivers and Humphreys (2004) found that unilateral parietal lesions disrupted preview as well as conjunction search, particularly when targets fell on the side of space contra to the lesion. Moreover, although the patients showed the normal pattern of performance in which there was somewhat better search for ipsilesional targets in the standard preview relative to the preview gap condition, the opposite occurred with contralesional targets.

b-sSoTS was lesioned by the random removal of processing units on one side of the saliency map. This selectively disrupted conjunction compared with single feature search, a standard finding in the neuropsychological literature (Eglin et al., 1989; Riddoch & Humphreys, 1987). In addition, there was disruption of search for contralesional targets under preview conditions, whereas performance benefitted from a temporal gap introduced between the preview and the search display (even though both the old and new items then onset together). The model provides one explicit account of these results. Damage to units on one side of the saliency map in the model disrupts the dynamics of processing, weakening temporal adaptation, reducing top–down inhibition, and also temporal binding when items onset together. On the one hand, the disruption to adaptation and top–down inhibition makes it more difficult to generate a benefit to search under standard preview conditions. On the other hand, the introduction of a temporal gap (and a longer time interval between the preview and the search display) enables even weak adaptation to have some effect, and the reduction in temporal binding means that the advantage from the initial preview is not overridden by the re-presented old items binding with the new search stimuli. The counterintuitive pattern of results reported for patients by Olivers and Humphreys (2004) were simulated.

The temporal binding we have introduced to sSoTS can be thought similar to the notion of binding by temporal synchrony, which has received some neurophysiological and psychological support (Elliott & Muller, 2004; Usher & Donnelly, 1998; Singer & Gray, 1995). The idea here is that common temporal onsets trigger reciprocal activity between neurons, binding the activity that the neurons are coding. In this prior work, temporal binding has been based around neurons having learned associations—perhaps reflecting the statistics of the visual world (basic Gestalt grouping). Other researchers have additionally demonstrated that there is rapid and relatively effortless binding of color and shape when objects have diagnostic colors (e.g., yellow banana, orange carrot), so that targets defined by these colors “pop out” in visual search (but not when the targets carry unfamiliar colors; Wildegger, Riddoch, & Humphreys, 2015; Rappaport, Humphreys, & Riddoch, 2013). In accounting for such results, the authors argue that learning establishes temporal links between different attributes of stimuli, so the common occurrence of the attributes generates stronger neural activity. On the other hand, the present results are built around the idea that there is temporal binding based on common onsets even between stimuli that are not grouped by common Gestalt properties or stored knowledge. Some neuropsychological evidence for grouping by common onsets was presented by Humphreys, Riddoch, Nys, and Heinke (2002). These authors study the phenomenon of “antiextinction” whereby patients can be better at reporting two stimuli relative to their report of a single stimulus. Humphreys et al. proposed that antiextinction reflected grouping of two items by common onset, which then enabled patients to select both items together—similar to when other forms of grouping have been shown to positively modulate extinction (e.g., Humphreys, 1998; Ward, Goodrich, & Driver, 1994). Consistent with the argument for grouping by common onset, Humphreys et al. (2002) reported that there were no benefits for two-item trials when one item was defined by an onset and the other by an offset. In our simulations, the mere onset of two stimuli together was sufficient to enable them to be bound and to modulate subsequent processing.

The account we present is in terms of preview search being dependent on a number of mechanisms. Other accounts stress the role of single mechanisms—for example, the role of onset capture by the new search items (Donk & Theeuwes, 2001). The current simulations are built on the idea that onsets do play a role in preview search, and grouping by common onset is a critical factor in capturing results on the timing of preview search (and why a temporal gap between the preview and search displays disrupts performance). On the other hand, behavioral evidence for a role of top–down suppression (Humphreys et al., 2004) and evidence for a restoration of efficient search when a short preview is “topped up” by earlier its presentation (Kunar et al., 2003a) provide clear support for the involvement of additional mechanisms, such as stimulus adaptation and active suppression; these additional mechanisms are incorporated here, enabling the model to simulate a wide body of data.

The results on the effects of lesioning the saliency map also fit with neuropsychological studies. For example, unilateral neglect, a clinical deficit often linked to damage to posterior parietal cortex, is known to be linked not only to impaired spatial processing but also to slow temporal selection of targets (e.g., an increased attentional blink; Husain, Shapiro, Martin, & Kennard, 1997). Battelli and colleagues (Battellil, Walsh, Pascual-Leone, & Cavanagh, 2008; Battelli, Pascual-Leone, & Cavanagh, 2007) have reported that posterior parietal lesions disrupt the ability to segment between onsets and offsets, and Roberts, Lau, Chechlacz, and Humphreys (2012) used lesion symptom mapping to demonstrate a relation between posterior parietal damage and reduced ability to discriminate the temporal order of events. More recently, Howard, Chechlacz, and Humphreys (2015) documented that posterior parietal patients had an impaired ability to update visual stimuli as they changed over time. Our simulations indicate that the effects of damage to the saliency map are to disrupt the dynamics of visual selection so that adaptation takes more time; there are reduced effects of top–down inhibition and also a reduced impact of new onsets (e.g., in the preview gap condition). According to the b-sSoTS model, visual selection is based on a dynamic saliency map where activity levels over time determine the efficiency of selection. Damage to the map affects temporal as well as spatial aspects of selection.

Although relevant neuropsychological data have yet to be reported, the current simulations indicate that the disruption to search should increase as lesion size increases. This is perhaps not surprising but should be assessed. In addition, recent neuroanatomical studies have proposed that impaired attention after posterior parietal damage may reflect the disconnection of fiber tracts between parietal and frontal cortices, rather than the loss of parietal gray matter per se (e.g., Lunven et al., 2015). Here we simulated the existing neuropsychological data (Olivers & Humphreys, 2004) by removing processing units—essentially an analog of gray matter lesioning. Whether similar patterns of data arise after disconnecting different parts of the model is a further question for future research.

A final point concerns the exact relations between the present simulations and the human search data. Our stress here has been on qualitative similarities between the data sets—for example, whether there are changes in the slopes of the search functions across different conditions. One of the interesting aspects of b-sSoTS is that it attempts to mirror biological processing systems more directly than approaches using more traditional, connectionist architectures, but there remain many approximations that mean that qualitative matches are more important than matching effects of the exact timing between stimuli or the exact slopes of the search functions. Here we show that the qualitative patterns of human search over time can be captured, providing an existence proof that dynamic processes of binding, neural adaption, and top–down suppression are sufficient to account for human search.

This work was supported by a grant from the European Research Council (PePe 333833).

Reprint requests should be sent to Eirini Mavritsaki, Department of Psychology, Birmingham City University, The Curzon Building, 4 Cardigan Street, Birmingham B4 7BD, UK, or via e-mail: [email protected] or Glyn Humphreys, Department of Experimental Psychology, Oxford University, South Parks Road, Oxford OX1 3UD, UK, or via e-mail: [email protected].