For individuals with grapheme–color synesthesia, achromatic letters and digits elicit vivid perceptual experiences of color. We report two experiments that evaluate whether synesthesia influences overt visual attention. In these experiments, two grapheme–color synesthetes viewed colored letters while their eye movements were monitored. Letters were presented in colors that were either congruent or incongruent with the synesthetes' colors. Eye tracking analysis showed that synesthetes exhibited a color congruity bias—a propensity to fixate congruently colored letters more often and for longer durations than incongruently colored letters—in a naturalistic free-viewing task. In a more structured visual search task, this congruity bias caused synesthetes to rapidly fixate and identify congruently colored target letters, but led to problems in identifying incongruently colored target letters. The results are discussed in terms of their implications for perception in synesthesia.
Synesthesia is a fascinating condition in which ordinary stimuli elicit unusual sensory experiences. For some synesthetes, sounds can have a specific color (Ward, Huckstep, & Tsakanikos, 2006) or tastes can elicit specific tactile and shape impressions (Cytowic, 1989, 1993). Synesthesia can even involve multiple extraordinary sensory experiences for a single inducing stimulus (Dixon, Smilek, & Merikle, 2004). An interesting and important characteristic of synesthesia is that the unusual sensory associations synesthetes experience are not simply epiphenomenal, but rather appear to profoundly influence the synesthete's cognition. Studies have shown synesthesia can have a substantial impact on memory (Baron-Cohen et al., 2007; Smilek, Dixon, Cudahy, & Merikle, 2002; Luria, 1968), emotional experiences (Callejas, Acosta, & Lupiáñez, 2007; Smilek, Malcolmson, et al., 2007; Ramachandran & Hubbard, 2001), perception (e.g., Nikolić, Lichti, & Singer, 2007; Palmeri, Blake, Marois, Flanery, & Whetsell, 2002; Ramachandran & Hubbard, 2001; Smilek, Dixon, Cudahy, & Merikle, 2001), and covert attention (Smilek, Callejas, Dixon, & Merikle, 2007; Smilek, Dixon, & Merikle, 2003; Palmeri et al., 2002).
In the present article, we further explored the functional impact of synesthesia by evaluating whether synesthesia can influence overt exploratory behavior as indexed by eye movements. There are several reasons for our interest in assessing the influence of synesthesia on overt eye movements aside from its already documented influence on covert attention. First, it has been shown that overt and covert attention can be dissociated (e.g., Klein, 1980), and so demonstrations that synesthesia influences covert attention do not necessarily imply that synesthesia will also influence overt eye movements. Second, our goal was to investigate spatial shifts of attention that operate in everyday life and these often involve fixating objects so that they can be processed by the high acuity portion of the retina (i.e., the fovea; see Findlay & Gilchrist, 2003). Finally, demonstrating that synesthesia influences overt eye movements would also provide evidence that synesthesia can have a direct impact on the oculomotor system.
To evaluate whether synesthesia influences eye movements, two individuals with grapheme–color synesthesia were studied. D. E. is a 20-year-old male undergraduate student at the University of Waterloo. For D. E., seeing black letters and numbers elicits the highly specific color experiences typical of grapheme–color synesthesia. For example, D. E. describes the letter D as having an unsightly shade of light pink, whereas the letter E and the number 3 are both a similar bright green. These color photisms are perceived as though emanating from, or overlaid on top of, the letters and numbers themselves, and are present even when the graphemes are in his peripheral vision. K. S. is a 23-year-old female undergraduate student also at the University of Waterloo. Like D. E., when K. S. sees black letters and numbers she experiences highly specific colors that appear to be overlaid on top of the letters and numbers. Thus, K. S.'s and D. E.'s synesthetic experiences fit the definition of a “projector” synesthete as defined by Dixon et al. (2004). Illustrating the variability one finds when comparing experiences across synesthetes (see Rich, Bradshaw, & Mattingley, 2005; Simner et al., 2005; Calkins, 1893), unlike D. E., K. S. experiences a deep maroon color for the letter D and a dark green color for the letter E; as well, she reports having to look directly at a grapheme in order to see its color. Nonetheless, for any given synesthete, the grapheme–color pairings reported are remarkably invariant over time (Dixon, Smilek, Cudahy, & Merikle, 2000; Odgaard, Flowers, & Bradman, 1999; Baron-Cohen, Harrison, Goldstein, & Wyke, 1993; Svartdal & Iversen, 1989) and both D. E. and K. S. report that for as long as they can remember, they have consistently experienced the same color for a given letter every time they saw it.
The purpose of Experiment 1 was to show that grapheme–color synesthesia can influence overt attention, as measured by eye movements. To demonstrate this, we capitalized on the fact that people, in general, prefer to look at things they like, and avoid looking at things they do not (Shimojo, Simion, Shimojo, & Scheier, 2003). In order to understand how our preference to look at things we like and avoid things we do not would play out in grapheme–color synesthesia, however, one must first understand synesthetic–color congruity. For K. S., seeing a black E elicits the experience of the black E along with a dark green overlay. Because text is most frequently presented as black characters on a white background (books, newspapers, Microsoft Word documents, etc.), this combination is the standard combination of physical color (black) and photism color (green) that K. S. is used to experiencing. Synesthetes report that. on those rare occasions when instead of a black letter the letter is presented in the “right” (i.e., congruent) color, this combination of physical color (green) and synesthetic color (green) looks and somehow “feels” right. By contrast, if the letter is presented in the “wrong” (incongruent) color (e.g., red), synesthetes experience two conflicting colors: the “wrong” color (red) in which the letter is physically presented, and the “right” color (green)—the color of their synesthetic photism induced by the letter. Synesthetes describe these incongruently colored graphemes as “jarring,” “nausea inducing,” and “difficult to look at.” In short, synesthetes report they like looking at congruently colored stimuli and dislike looking at incongruently colored stimuli. If our propensity to spend more time looking at things we like, and avoid looking at things we dislike, holds for synesthesia, then we should find a color congruity bias in the synesthetes' eye movements. We used a free-viewing task where synesthetes were asked to view an array of colored letters until each item had been “seen” (i.e., they were free to either look directly at the letters, or “see” them using their peripheral vision). Importantly, half of the letters were presented in colors that were congruent with the synesthete's colors for those letters, and half were shown in colors that were incongruent with the synesthete's colors. We therefore anticipated that synesthetes would fixate congruently colored letters significantly more often and for a longer duration, on average, than incongruently colored letters.
Two synesthetes first identified the colors they associate with each letter of the English alphabet. The two synesthetes and two groups of nonsynesthetic participants then viewed displays containing colored letters. Half of the letters were congruently colored (for the synesthetes) and half were incongruently colored. The participants were required simply to view each display until they felt they had seen all of the letters it contained. Eye movements were monitored throughout the free-viewing task in order to evaluate the specific prediction that synesthetes would show a congruity bias in their viewing behavior. To recap, for synesthetes, congruently colored letters should be fixated significantly more often and for a longer duration, on average, than incongruently colored letters.
A 20-year-old man with grapheme–color synesthesia (D. E.) participated in the approximately 1-hr free-viewing task. As well, a 23-year-old woman with grapheme–color synesthesia (K. S.) participated in the same 1-hr free-viewing session. Six nonsynesthetic students from the University of Waterloo served as yoked controls for each synesthete (i.e., they were presented with exactly the same letter displays in the same order, and were required to perform the same task as the synesthete to whom they were yoked). Yoked control participants received partial credit toward their Introductory Psychology course as compensation. One of the nonsynesthete participants yoked to D. E. was removed because of a disproportionately large number of fixations falling on entirely blank portions of the screen. This participant was replaced.
Displays were created using colors previously identified by the synesthetes as the colors that corresponded to each letter of the alphabet. These colors were applied either to the “right” letter (congruent) or a different letter (incongruent). Each display always contained an equal number of congruently and incongruently colored letters. The letters used for any given display were selected such that all letters in the display carried a different color. The letters shown on the free-viewing displays were presented against a standard black background, so the maximum number of letters available to be presented was constrained by being able to select only those colors that were discernable against a black screen (24 for D. E., 23 for K. S.). For both synesthetes, there were several letters of the alphabet that elicited photisms with similar colors. To ensure a unique color of presentation for each letter on a given display, however, letters with similar colors were not presented in any of the displays. Given these constraints, the displays used in the free-viewing task included a maximum of 14 of the 24 letters available for D. E. and 12 of the 23 letters available for K. S.
Each display was constructed by dividing the screen into an imaginary 6 × 6 grid and randomly assigning each letter to one of the resulting 36 locations. All letters were presented in 36-point Lucida Console font for D. E. To ensure the results would generalize beyond this specific font size, for K. S. font size was randomly adjusted such that on any given display up to four items were presented at a size 45% larger than the 36-point Lucida Console font used for the other letters. As the size adjustment was irrespective of color, any effects of size would serve only to add noise to the eye movement data. Displays were viewed from a distance of approximately 81 cm. Items measured from 0.6 cm (0.4°) to 1.5 cm (0.9°) horizontally and from 1.2 cm (0.7°) to 2.0 cm (1.2°) vertically depending on the letter and size of presentation (see Figure 1). Each item was presented in the center of its grid location (cell), which measured approximately 6.9 cm (4.2°) horizontally and 5.2 cm (3.2°) vertically. A letter was considered fixated when the pixel coordinates of the averaged fixation location landed inside its cell.
An SR Research EyeLink II head-mounted eye tracking system was used to display the stimuli and collect response time and eye movement data. The EyeLink II head band contains three cameras for simultaneous tracking of both eyes and of head position for head-motion compensation. Pupils and corneal reflections were tracked on the most accurately calibrated eye for each participant. The system was calibrated using a randomly ordered nine-dot pattern, and the mean error in the computation of gaze position was less than 0.55° visual angle on average. The system's default settings for acceleration and velocity thresholds were used for saccade detection.
Two display screens were used. The stimulus displays were presented to the participants on a Dell P1230 22-in. CRT flat screen color monitor with a medium short phosphor persistence; the monitor resolution was set at 1024 × 768. The displays were also presented to the experimenter on a second monitor so that real-time feedback could be given about gaze position. This allowed the experimenter to evaluate system accuracy throughout the experiment and to initiate a recalibration if necessary.
Prior to the free-viewing task, the synesthetes had identified the colors they associate with each letter of the English alphabet. For this process, black letters were presented on a white paper. Synesthetes then matched the photism that was elicited by the letter to a color on a computer screen using the color palette included in Microsoft Paint. Once a color was identified, the RGB values were recorded. This process was repeated until a color had been selected for each letter. For some letters, the synesthetes indicated the letter had more than one color, and in these cases, they were asked to identify the primary color associated with that letter; only the primary color was subsequently used for the free-viewing displays.
For the free-viewing task, participants were instructed to view each display until they believed they had seen everything on the display, and that they could view the displays for as long as they wished. Participants were further instructed to press a button when they were finished viewing a display and ready to proceed to the next display. In order to ensure a similar mean viewing time to that of D. E. and K. S., nonsynesthete participants were also instructed to attempt to view the displays for at least 8 sec, on average, but to do so without counting the time in their heads.
The free-viewing task was split into two blocks: a practice block of 12 trials and an experimental block of 180 trials. Nonsynesthetes viewed the same displays in the same randomized order as the synesthete to whom they were yoked. A trial began with presentation of the stimulus display, which then remained on screen until the participant pressed a button. After the button press, a blank screen was presented for 2000 msec before the onset of the next stimulus display. Following every set of 12 trials, participants had a brief rest period. A drift correction, performed after this rest period, also took place once every 12 trials.
Results and Discussion
Recursive outlier removal analyses were first performed for each display based on the participants' average fixation duration for congruently and incongruently colored items. Fixations were removed in which fixation durations were greater than 3 standard deviations from the mean for that display. This removed 1.26% of the total number of fixations for D. E. (0.31% incongruent, 0.95% congruent), 1.74% of the total number of fixations for K. S. (0.67% incongruent, 1.07% congruent), and an average of 1.67% of the total number of fixations for the nonsynesthetes (mean 0.83% incongruent, 0.98% SD; mean 0.85% congruent, 1.20% SD).
Figure 2 illustrates the mean number of fixations per display for incongruently and congruently colored letters. Figure 2A shows that D. E. fixated congruently colored letters significantly more than incongruently colored letters [t(179) = 8.41, p < .001]. By comparison, none of the nonsynesthetes showed a significant difference on this measure (all t values <1.5, all p values >.13). D. E.'s viewing performance was compared to the nonsynesthetes using difference scores. D. E.'s difference score (average congruent fixations minus average incongruent fixations) was 12.5 standard deviations greater than the mean of the same difference scores computed for his yoked nonsynesthetes. The data clearly support the presence of a congruity bias in D. E.'s viewing behavior.
Figure 2B shows that the performance of K. S. parallels that of D. E. She fixated congruently colored letters significantly more often than incongruently colored letters [one-tailed t(179) = 2.13, p < .05]. Again, none of the nonsynesthetes who completed the same free-viewing task showed a significant difference between conditions, with all one-tailed t values <1.4, all p values >.08. K. S. was also compared to her yoked nonsynesthetes using difference scores. The difference between her average number of fixations on congruent and incongruent letters (congruent minus incongruent) was 3.3 standard deviations greater than the mean of her yoked nonsynesthete participants. Thus, both K. S. and D. E. show evidence of a congruity bias in their overt viewing behavior.1
In addition to the number of fixations made on congruently and incongruently colored letters, the congruity bias was also expected to be evident in the synesthetes' average fixation duration. Specifically, the synesthetes were predicted to show significantly longer fixation durations for congruently colored letters than incongruently colored letters. Figure 3A shows D. E. fixated congruently colored letters significantly longer than incongruently colored letters in the free-viewing task [t(179) = 3.00, p < .01], whereas none of D. E.'s yoked nonsynesthetes showed significant differences between congruent and incongruent duration times (all t values <1.6, all p values >.12). This suggests that D. E.'s fixation durations were not attributable to a simple stimulus bias (i.e., his congruently colored letters were not somehow more “attention holding” to nonsynesthetes). K. S. also showed a congruity bias in her viewing behavior, fixating congruently colored letters significantly longer than incongruently colored letters (Figure 3B) [one-tailed t(179) = 1.87, p < .05]. None of her yoked nonsynesthetes showed significant differences between these conditions (all one-tailed t values <1.0, all p values >.16). As can be seen by comparing the standard error bars for the nonsynesthetes in Figures 2 and 3, there was much more variance in the fixation duration measure. Likely because of this variability, difference-score measures akin to the ones computed for number of fixations failed to show that the synesthetes were outliers from the control distributions. That said, the fact that both synesthetes showed significant differences between fixation durations for congruent and incongruent letters, while none of the nonsynesthetes showed such differences, provides converging evidence for a congruity bias in grapheme–color synesthesia.
Taken together, the mean numbers of fixations and the mean fixation durations clearly show the presence of a color congruity bias in the synesthetes' eye movements, and are consistent with the original prediction that the synesthetes would show a tendency to fixate congruently colored letters both more frequently and longer than incongruently colored letters. As such, the present findings support the conclusion that, for synesthetes, the manner in which graphemes are colored can rapidly and systematically influence overt visual attention even at the level of eye fixations. They also validate the subjective reports of a preference for congruently colored information.
The goal of Experiment 2 was to evaluate whether a congruity bias occurs in a more traditional visual search task. In this task, participants were to search for a predefined target grapheme (e.g., an E) embedded among a number of distractor graphemes and make “target present” or “target absent” judgments. Participants were presented with a single achromatic letter that was to be the target of their search for that trial and then searched for this target within a display of congruently and incongruently colored letters. Participants were instructed to ignore the color of the letters and respond based on the identity of the letter alone (i.e., if the target is a T, then either a green T or a red T should lead to a “target detected” button press).
On trials in which the target was present, the congruity bias was predicted to influence response times. That is, when searching through a display of congruent and incongruent stimuli, synesthetes should be biased to respond more quickly to congruently colored targets relative to incongruently colored ones. Thus, we expected to find significant overall search performance differences between congruently and incongruently colored target letters for synesthetes but not nonsynesthetes. As a corollary of this prediction, relative to their yoked nonsynesthetes, the synesthetes were predicted to show enhanced search performance for congruently colored targets but poorer performance for incongruently colored targets.
On target-present trials, the congruity bias should also influence the eye movements of synesthetes. We predicted that their attention should be drawn to congruently colored targets, more so than incongruently colored targets. To test this hypothesis, we looked at the first two eye movements of synesthetes and controls. We used the first two eye movements rather than the first eye movement because pilot work indicated that the first eye movement on a large proportion of trials landed in a blank space between graphemes. We anticipated that for target-present trials, synesthetes would show an increased likelihood of quickly fixating congruently colored target letters within the first two fixations of their search.
In addition to the congruity bias effects predicted above, the subjective reports of synesthetes also lead to interesting predictions concerning performance when the targets are incongruently colored. A number of synesthetes have described incongruently colored stimuli as being “hard to interpret.” The synesthete C., for example, has described a grapheme presented in an incongruent color as being like “a square circle—it just doesn't make any sense.” Because synesthetic color is such a dominant aspect of a synesthetes' perception (especially for projector synesthetes such as D. E. and K. S.; Dixon et al., 2004), it may take synesthetes extra time to determine exactly what it is that they are seeing when faced with an incongruently colored stimulus. If so, then incongruently colored targets may take longer to identify than congruently colored targets in this highly constrained visual search task. Importantly, response times alone do not differentiate between behavior attributable to the congruity bias and behavior attributable to difficulty interpreting incongruent stimuli. In adjudicating between these two alternatives, it is the eye movements that provide the touchstone. If synesthetes are having trouble making sense of incongruent graphemes (i.e., identifying them as the target of the search), they may actually fixate on the target, fail to identify it, and continue searching. To see if this were the case, we tabulated the number of times synesthetes (and nonsynesthetes) fixated targets, yet failed to press the “target detected” button and kept searching. We predicted that for synesthetes this would occur much more often for incongruently colored targets than for congruently colored targets, which, once fixated, should be identified as the goal of the search.
D. E. and K. S. were the synesthetes, and a different group of six nonsynesthetic students from the University of Waterloo served as yoked controls for each synesthete. The nonsynesthetes received partial credit toward their Introductory Psychology course as compensation.
The letters and colors used were the same as those in Experiment 1 (24 letters for D. E. and 23 letters for K. S., allowing a maximum of 14 unique colors per display for D. E. and 12 unique colors for K. S.). Thus, in order to maintain a linear increase in the number of items displayed across three set sizes, any given search display contained 6, 10, or 14 letters for D. E. and 8, 10, or 12 letters for K. S. The same 36 location grid was used to display the letters. Figure 1 illustrates an example of a search display for set size 12 with K. S.
All search displays contained an equal number of congruently and incongruently colored letters, arranged randomly within the 36 cells of an imaginary 6 × 6 grid. The displays were designed to create two key search conditions: target present and congruently colored, and target present but incongruently colored. Target-absent trials were also presented. On half of the target-absent trials, a distractor item carried the color of the synesthete's photism for the object of the search (e.g., if the target was an E, and E elicited a pink photism, there would be a pink grapheme such as a pink T or F, but the target letter E was not present). To ensure that font typeface was not responsible for any observed differences in they eye movement data, typeface was varied across synesthetes. For D. E., letters were presented in 32-point Times New Roman font, and for K. S., they were presented in the same, equivalently sized, 36-point Lucida Console font as used in Experiment 1. Displays were viewed from a distance of approximately 81 cm. Individual letters measured 0.5 cm (0.3°) to 1.7 cm (1.0°) horizontally and 1.2 cm (0.7°) to 1.6 cm (1.0°) vertically depending on the letter and typeface. Each letter was located centrally within its corresponding cell, which measured approximately 6.9 cm (4.2°) horizontally and 5.2 cm (3.2°) vertically. A letter was considered fixated when the pixel coordinates of the averaged fixation location landed inside its cell.
The same SR Research EyeLink II head-mounted eye tracking system was used to track eye movements during the visual search. The system was calibrated using a randomly ordered nine-dot pattern, and the mean error in the computation of gaze position was less than 0.5° of visual angle on average. The system's default settings for acceleration and velocity thresholds were used for saccade detection.
For the visual search task, participants were asked to search for a target letter among a display of randomly selected colored letters. The target item was initially shown achromatically, and participants were instructed to identify the target as present if it appeared in the display, regardless of its color. The visual search task consisted of 360 trials, divided into blocks of 12 trials with equal representation of each of three set sizes (6, 10, 14 for D. E. or 8, 10, 12 for K. S.) and the four search conditions: target present and congruently colored, target present but incongruently colored, target absent with a distractor carrying the target's color, and target absent with no target color equivalent available. The first block of trials was considered practice and was not analyzed. As the search conditions were represented equally in each block, 87 trials had a congruently colored target, 87 had an incongruently colored target, 87 had no target but a key distractor carrying the target's photism, and 87 had no target or letters carrying the target's photism available; thus including 29 trials per set size in each condition. As in Experiment 1, yoked nonsynesthete participants were presented with the same search displays, presented in the same randomized order, as for the synesthetes to whom they were yoked.
As indicated in Figure 4, a trial began with the presentation of an achromatic target letter located in the center of the screen. The target letter remained on screen for 2000 msec before being replaced by a blank screen for 1000 msec and then the search display, which remained on screen until a “target detected” button press was made, or a “target absent” button press was made. Following every block of 12 trials, participants had a brief rest period. A drift correction, performed after this rest period, also took place once every 12 trials.
Results and Discussion
Trials on which a participant responded incorrectly with respect to the actual presence of the target on the search display (search errors) were removed from the dataset prior to all analyses. Recursive outlier removal analyses were then performed for each search condition based on the participants' reaction time and the number of fixations recorded. Trials were removed in which participants' reaction times and total number of fixations fell outside 3 standard deviations from that individual's mean for that condition and set size. Collapsing across set size and search condition, search error and outlier trial removals eliminated 5.7% of trials for D. E. (8.0% target present; 3.4% target absent), 3.4% of trials for K. S. (4.6% target present; 2.3% target absent), and an average of 5.8% of trials for nonsynesthete participants (8.3% mean, 3.1% SD target present; 3.4% mean, 2.8% SD target absent).
Figure 5 illustrates search efficiency on target present trials for D. E. and his yoked nonsynesthetes (Figure 5A), and K. S. and her yoked nonsynesthetes (Figure 5B). The figure shows D. E. was substantially faster to find congruently colored targets than incongruently colored targets. The data were submitted to a 2 (congruent vs. incongruent) × 3 (6, 10, 14 letters) independent-samples ANOVA, which confirmed that target congruity had a significant main effect for D. E. [F(1, 154) = 147.02, MSE = 120,616.7, p < .001]. The ANOVA also confirmed there was a significant main effect of set size for D. E. [F(2, 154) = 6.53, MSE = 120,616.7, p < .01], and a small but marginally significant interaction [F(2, 154) = 3.03, MSE = 120,616.7, p < .052]. None of D. E.'s yoked nonsynesthetes showed a significant main effect of congruity (all F < 2.3, all p > .13), although one of the nonsynesthetes did show a significant interaction [F(1, 163) = 3.68, MSE = 96,406.9, p < .05], making D. E.'s interaction difficult to interpret. Therefore, the remainder of the reaction time analyses focused primarily on the main effect of congruity. D. E.'s average reaction times were compared to nonsynesthete reaction times for key conditions relating to our congruity bias predictions. First, compared to his yoked nonsynesthetes, D. E. took significantly longer to find incongruently colored target letters (his mean search time for finding incongruent targets was 2.25 standard deviations slower than the mean of the nonsynesthetes). Second, he took substantially less time to find congruently colored targets than his yoked nonsynesthetes (his mean search time for finding congruent targets was 2.95 standard deviations faster than the mean of the nonsynesthetes). Thus, these findings suggest that D. E.'s congruity bias may have improved search performance for congruent targets while hindering search for incongruent targets.
K. S. showed the same general trend in her performance as D. E. As can be seen in Figure 5B, K. S. found congruently colored targets substantially faster than incongruently colored targets. Again, the data were submitted to a 2 (congruent vs. incongruent) × 3 (8, 10, 12 letters) independent-samples ANOVA, which confirmed that target congruity had a significant main effect for K. S. [F(1, 160) = 145.99, MSE = 163,978.6, p < .001]. The ANOVA also confirmed there was a small but marginally significant main effect of set size for K. S. [F(2, 160) = 2.92, MSE = 163,978.6, p < .058], and a small but significant interaction [F(2, 160) = 3.09, MSE = 163,978.6, p < .05]. In this case, the fairly small effect of set size is understandable given that only two letters were added at each increase. As with D. E., none of K. S.'s yoked nonsynesthetes showed faster reaction times for congruent versus incongruent target detection times (all main effects F < 1.4, all p > .24). Neither did any of the nonsynesthetes show a significant interaction of target congruity with set size (all F < 2.4, all p > .09). It is worth noting, however, that one nonsynesthete did show a small but significant main effect of congruity, but it was in the opposite direction as was found with K. S. and D. E. (i.e., this nonsynesthete was faster for incongruently colored targets) [F(1, 155) = 5.44, MSE = 120,602.6, p < .05].
The comparison of K. S.'s mean search times with those of her yoked nonsynesthetes was complicated by the fact that one of these nonsynesthetes was much slower overall than the others, substantially inflating the variability around the nonsynesthetes' mean search times. As such, when comparing K. S.'s search times to those of her control group, the overall performance of K. S. did not fall significantly outside the nonsynesthetes' distribution for either incongruently or congruently colored targets. Individual comparisons, however, revealed that even with this one participant in the distribution, K. S. was still significantly slower to find incongruently colored targets at set sizes 8 and 12, performing more than 2.15 standard deviations from the mean of the nonsynesthetes in each case. Furthermore, if the variability of the nonsynesthetes is reduced by removing the slowest-performing nonsynesthete for congruent target-present trials (i.e., the nonsynesthete who performed least like K. S. in this condition), then K. S.'s mean search time does fall outside the distribution of the nonsynesthetes, more than 2.6 standard deviations faster than the mean of the nonsynesthetes. Thus, like D. E., relative to her yoked nonsynesthetes, K. S. does show some degree of both speeded search for congruently colored targets and slowed search for incongruently colored targets.
We first examined whether D. E. showed an increased likelihood of finding congruently colored targets within the first two fixations of his search. If D. E. exhibited a congruity bias in his searching behavior, then relative to the nonsynesthete participants, it should be possible to evaluate whether this bias affected his ability to locate congruently and incongruently colored targets. Indeed, Figure 6A shows that within the first two fixations, D. E. was twice as likely to fixate congruent targets relative to incongruent targets. He fixated the target within two fixations on 60.5% of trials when the target was presented in a congruent color but fixated the target letter within the first two fixations of his search on only 27.8% of trials where the target was presented in an incongruent color—a difference of 32.6% (congruent − incongruent). This difference was more than 4 standard deviations greater than the mean difference of the yoked nonsynesthetes (6.1% mean, SD = 6.6%). More fine-grained comparisons with D. E.'s yoked nonsynesthetes indicate that it is D. E.'s performance on congruent trials that differentiates him from the nonsynesthetes. Relative to the nonsynesthetes, D. E. showed an increased likelihood of finding congruently colored targets within two fixations (he was 2.85 standard deviations away from the mean of the nonsynesthetes in this condition). By contrast, he was well within the range of the nonsynesthetes on trials where the target was incongruently colored. Once again, K. S.'s performance (Figure 6B) was remarkably similar to that of D. E. K. S. fixated the target within two fixations on 63.4% of trials where targets were congruently colored compared to 36.9% of trials where the target was presented in an incongruent color. This difference of 26.5% was 3.2 standard deviations greater than the mean difference of K. S.'s yoked nonsynesthetes (2.8% mean, SD = 7.3%). Like D. E., the magnitude of K. S.'s congruity effect was carried by her performance on congruent trials. Her mean for congruent targets was 2.25 standard deviations greater than the mean of the nonsynesthetes, whereas her mean for incongruent targets was indistinguishable from the nonsynesthetes.
The combination of response time and eye-movement data provides more insight into the search performance of synesthetes than either measure alone. Specifically, the two measures provide converging evidence for better-than-normal performance by synesthetes for congruent targets. That is, for congruent targets, eye movements indicate that targets are found more quickly by synesthetes than controls, and response times follow suit (synesthetes are significantly faster to detect congruently colored targets than controls). For incongruently colored targets, eye movements indicate that the synesthetes fixate on the target as quickly as controls, yet they take significantly longer to respond to these targets than controls. This data pattern makes sense if synesthetes fixate incongruently colored targets, but fail to identify these targets as the goal of the search and continue searching. Such a finding would correspond to the subjective reports of synesthetes who claim that often they have trouble interpreting what they are seeing when faced with letters in the “wrong” color.
To provide additional evidence for this postulate, we examined the number of refixations for congruently and incongruently colored targets. Refixations occur when the target is fixated, but the target is not identified. In this case, search continues and participants fixate elsewhere, and (eventually) refixate the target. As can be seen in Figure 7, D. E. refixated targets on more than 20% of the trials in which the target was incongruently colored (Figure 7A); more than 12 standard deviations greater than the mean proportion of refixations made by the nonsynesthetes on these same trials. By comparison, D. E. did not refixate any of the congruently colored targets and, as expected, the nonsynesthetes showed no effect of target congruence on the number of target refixations they committed.
K. S. showed a remarkably similar pattern of results (Figure 7B). As with D. E., K. S. refixated targets on more than 20% of the trials in which they were incongruently colored; more than 9.7 standard deviations greater than the mean proportion of refixations made by the nonsynesthetes. Finally, just like D. E., K. S. did not refixate a single congruently colored target. Taken together, these findings suggest that although the synesthetes had no difficulty identifying congruently colored targets, they did have trouble identifying incongruent targets—even when they were looking directly at them. Importantly, these refixation data explain the relatively long RTs for incongruent trials. That is, incongruently colored targets were initially looked at but not recognized, and thus, had to be refixated before the synesthete could identify these items as the goal of their search.
Ruling Out an Alternative Explanation
There is an alternative explanation for the finding that both synesthetes performed significantly faster when searching for congruently colored targets than incongruently colored targets, without relying on a synesthetic color congruity bias. Namely, the synesthetes could have employed a simple color search strategy. Recall that prior to search trials, participants were shown an achromatic target letter that indicated the goal of the upcoming search. Synesthetes could have noted the color of their photism for the target and simply looked through the display for the video color that best matched the color of their photism for the target item. This is unlikely for a number of reasons. First, this strategy is inefficient, leading to the correct target only 25% of the time (on congruent target trials, not on incongruent trials, nor on target absent trials). By contrast, matching the photism color stored in memory with the photism colors on the screen would lead to correct identifications 100% of the time (as would matching the shapes of the letters). Second, employing a photism color–video color matching strategy predicts that initial eye movements would be made away from incongruent targets. That is, synesthetes should be less likely than controls to make eye movements to incongruent targets because they carry a color that does not match the photism color stored in memory. The obtained data indicate that synesthetes were as likely as controls to fixate the incongruent targets within the first two fixations and rule out a simple color matching strategy as a viable explanation.
Combining all the results of the visual search task, we find clear support for the presence of a color congruity bias in the synesthetes' search performance. As well, we find behavioral support for the synesthetes' subjective reports of difficulty dealing with the visual conflict of incongruently colored information. Together, these findings suggest the synesthetes' search was substantially facilitated by presenting target letters in a color congruent with their synesthetic photisms and substantially hampered by presenting targets in a color incongruent with their photisms.
Our investigation of the influence of grapheme–color synesthesia on eye movements showed that grapheme–color congruity affected viewing behavior in several key respects. In the free-viewing task in Experiment 1, both D. E. and K. S. exhibited a color congruity bias in their overt viewing behavior, fixating congruently colored letters more often and for longer durations than incongruently colored letters. In so far as fixations index attention, we conclude that, based on a lifetime of accrued associations between specific graphemes and specific colors, synesthetes are habitually used to experiencing, and therefore prefer, congruently colored graphemes and pay more attention to congruently than incongruently colored graphemes. Note that in the free-viewing task there were no specific requirements to identify any of these stimuli.
In the visual search task used in Experiment 2, participants did have to identify the graphemes that they were shown in order to respond appropriately. In this task, synesthetes located and identified congruently colored target letters substantially faster than incongruently colored target letters. This superior search performance for congruently colored targets was indexed by faster-than-normal search times and by a greater-than-normal likelihood of fixating these congruent targets within the first two fixations. In addition to a congruity bias, the synesthetes also showed an incongruity deficit when identifying incongruently colored targets in the visual search task. That is, although the synesthetes were just as likely as nonsynesthetes to fixate incongruently colored target letters within the first two fixations, they had greater difficulty identifying those letters as the goal of the search for that trial. This resulted in both longer-than-normal search times and necessitated more refixations of incongruent targets before they could be identified. By contrast, the synesthetes never had to refixate congruent targets—once fixated, these targets were immediately recognized and responded to as the goal of the search.
The present findings showing that synesthetic colors influence eye movements are consistent with previous studies demonstrating the impact of synesthesia on covert attention (Smilek, Callejas, et al., 2007; Smilek et al., 2003; Palmeri et al., 2002). In contrast, the present findings challenge several recent reports that synesthesia does not influence attention during visual search (e.g., Gheri, Chopping, & Morgan, 2008; Edquist, Rich, Brinkman, & Mattingley, 2006; Sagiv, Heer, & Robertson, 2006). As we have noted before (Dixon & Smilek, 2005; Dixon et al., 2004; Smilek & Dixon, 2002), we believe this discrepancy in conclusions across studies is likely due to differences in synesthetic experiences among the individuals tested.
Implications for Perception in Synesthesia
The difficulty D. E. and K. S. had in interpreting the identity of incongruently colored target letters in the visual search task highlights the potential for interactive effects of meaning and synesthetic color prior to full conscious awareness of the identity of an object. This assumption builds upon previous research on congruency effects in synesthesia and research showing that photisms can be generated even though the synesthetes are unaware of the graphemes that induced these photisms. Multiple studies have shown that, for synesthetes, incongruently colored graphemes pose difficulties in Stroop-type naming tasks (Mattingley, Rich, Yelland, & Bradshaw, 2001; Dixon et al., 2000). Recently Nikolić et al. (2007) have shown that these synesthetic interference effects are not limited to semantic knowledge (e.g., “I know that C is red”), but involve conflict between the real colors and the perceptual experience of the photisms. Their findings show that photisms involve color-opponent processing channels in V4/V8. Additionally, a number of studies have shown that synesthetic photisms can influence visual search efficiency (Palmeri et al., 2002; Smilek et al., 2001), whereas others have shown that photisms can support perceptual grouping (Hubbard, Arman, Ramachandran, & Boynton, 2005; Ramachandran & Hubbard, 2001), and still others have shown that photisms can protect the ability to identify graphemes from different types of masking (Wagar, Dixon, Smilek, & Cudahy, 2002; Smilek et al., 2001). These studies empirically support the view that photisms can emerge before the synesthete is aware of the grapheme that is triggering them (but see Mattingley et al., 2001 for an alternative view).
In the current visual search experiment, we manipulated video and synesthetic color congruency in the context of a visual search task. Recall that for all participants the (achromatic) target letter was shown in advance, followed by the search display. Search was conducted by comparing the letter that was held in memory to the letters within the search display. For nonsynesthetes, letter shape was the only dimension relevant to making this comparison. For synesthetes, however, the initial achromatic letter would generate a photism associated with that grapheme. Thus, for synesthetes, there were three possible dimensions on which comparisons could be made: letter shape, the video color of the graphemes in the search display, and the synesthetic colors of the graphemes in the search display. Based on previous research, we assume that when looking at the search displays, D. E. and K. S. experienced photisms prior to their being fully aware of the graphemes that generated them. For congruently colored targets, there were three sources of information that were diagnostic of target identity—the letter, the synesthetic color induced by the letter, and the video color of the letter (although the video color was available to the nonsynesthetes, it carried no relevance to target identity). The fact that for synesthetes these latter two dimensions were both available and relevant to target identity enabled them to make eye movements to and identify (i.e., make an accurate response to) these targets more quickly than controls. Thus, for synesthetes, all three distinct dimensions facilitated their ability to localize the targets.
For incongruent targets, however, the video color of the target and the synesthetic color of the target conflicted. Behaviorally, synesthetes “missed” these targets even when they were fixated. If, as suggested by the work of Nikolić et al. (2007), the video color and synesthetic color both use opponent-color channels early in V4/V8, then the conflict arising from these competing signals may have caused difficulties in interpreting exactly what the synesthete was looking at. Thus, even when they fixated on an incongruent target, the processing of the video color interfered with the interpretation of the photism color (which for synesthetes is a highly salient cue to the meaning of the grapheme) and resulted in the synesthete failing to realize that the letter being fixated was the goal of visual search. The end result of this conflict is that the synesthete failed to identify the target grapheme, and continued searching. In nonsynesthetes, there was no conflict—only letter shape provides a cue to target identity; hence, nonsynesthetes fixated these targets as quickly as synesthetes and immediately identified them as the goal of the search.
In terms of neural architecture, these findings can be accounted for by a re-entrant model of grapheme–color synesthesia (Smilek & Dixon, 2002; Smilek et al., 2001). In this cascade model, perception does not occur all at once, but is built up over successive cyclical iterations of activity flowing from low-level (e.g., form, color) areas to higher-level (i.e., meaning) areas and then from higher-level areas back to lower-level areas. Importantly, a conscious percept emerges only after multiple iterations of this interactive activity. Smilek and Dixon (2002) proposed how a black grapheme could lead to a color experience. Graphemic form is processed in posterior fusiform areas which, in turn, activate anterior areas of the fusiform processing relatively rudimentary aspects of graphemic meaning. In the model, these early-stage meaning areas are akin to the way stations described by Dehaene (1992) for numbers—areas that process the number form, but pass on information to inferior parietal areas that support more complex calculations involving magnitude comparisons and approximate calculation (Dehaene, 1992). Similarly for letters, these meaning areas recognize a particular letter and pass on information to other areas involving grapheme-to-sound correspondences. Importantly, however, these meaning areas do not only pass information to later-stage areas. They also back-activate areas in V4/V8 corresponding to the synesthetic color. Eventually, over successive iterations of such feedforward and feedbackward activation, the percept of a specific black grapheme with a colored overlay would emerge.
Consider now how such a model would account for the processing of congruently and incongruently colored graphemes. First, consider a congruently colored grapheme (e.g., the letter D, presented in pink). The form of the D would activate V1, V2, and then activate posterior (form) and then anterior (meaning) areas of the fusiform, respectively. Anterior areas of the fusiform (associated with the letter D) would back-activate color-opponent channels in V4/V8 corresponding to the color of the photism for that grapheme (pink). Importantly, these areas would feed forward to reciprocally activate the anterior areas of the fusiform selective for the letter D. That is, form activates meaning, which back-activates (photism) color, which reactivates meaning. Color experiences are not only activated by meaning, but also by the fact that the grapheme itself was physically colored. The video color of the grapheme (pink) would pass through V1 and V2 and activate areas in V4/V8 associated with the color pink. These areas, in turn, would activate anterior fusiform areas that code the letter D. Thus, neurons in V4/V8 that are selective for pink would be activated by bottom–up opponent-color channels, and by top–down pathways from the anterior fusiform. Feedforward pathways from V4/V8 would increase the activation in the anterior fusiform associated with the letter D. The resulting patterns of cyclical activation would result in the percept of a pink D with a pink photism.
Consider next an incongruently colored grapheme—a D presented in blue. As above, the form of the D would activate neurons that code for this letter, and there would be back-activation to V4/V8 associated with pink (the color of the photism for D). The video color of the D (blue), however, would activate different neurons in V4/V8 (those selective for blue). Activation from these neurons would feed forward to the anterior fusiform and activate neurons that code for the synesthete's blue grapheme (in D. E.'s case an H). In this scenario, we have two pools of neurons in V4/V8 being activated: pink (via back-activation from anterior fusiform areas coding for D) and blue (bottom–up activation from the video color on the screen). We also have two pools of neurons being activated in the anterior fusiform (activation for neurons associated with the letter D and activation associated with the letter H). Such competition could account for the synesthetes' self reports that when presented with an incongruent stimulus they have trouble making sense of what they are seeing. More concretely, these patterns of activation can account for the current findings where synesthetes fail to identify incongruently colored graphemes even when they are looking at them.
Prior instantiations of this model lacked feedforward links between areas that process color (V4/V8), and areas that process the meaning of a grapheme (that the grapheme presented was an E or a 5). The current finding that synesthetes have trouble identifying incongruent graphemes even when they fixate them, in our view, necessitates the addition of these feedforward links between color and brain areas involved in grapheme identification. Such links are consistent with the findings of Cohen Kadosh and Henik (2006), which demonstrate that color ultimately influences magnitude judgments of digits, even when participants are asked to ignore these colors.
Although the present data do not help to adjudicate between our re-entrant model and cross-activation models, our revised re-entrant model is supported by a number of recent findings. Rouw and Scholte (2007) showed that the brains of grapheme–color synesthetes exhibit hyperconnectivity in the white matter areas of the right inferior temporal cortex near the fusiform gyrus—the area that the re-entrant activation model predicts would play an important role in the experience of grapheme–color synesthesia. Hubbard et al. (2005) showed that for synesthetes, seeing black graphemes lead to greater activation of V4/V8 than in nonsynesthetes—a finding in our model which we explain via back-activation from early stage meaning areas back to color areas. Dixon, Smilek, Duffy, Zanna, and Merikle (2006) showed that identically shaped graphemes (a scoreboard 5 and a scoreboard S) can elicit differently colored photisms when context biases the synesthete to interpret these graphemes as a number on one occasion and as a letter on a different occasion—a finding that must be explained by feedback from meaning areas to human color areas.
We believe the present findings have direct bearing on recent debates concerning the influence of meaning on the allocation of visual attention in synesthesia. Some studies have previously shown the importance of feedbackward (or top–down) influences in synesthesia, whereas other studies suggest that feedforward (or bottom–up) influences are more important. The present findings make an important contribution to this debate by showing that both a congruity bias and deficits in identifying incongruently colored letters influence the eye movements of grapheme–color synesthetes. As such, these findings strengthen the general conclusion that theories of the perceptual and cognitive influences of synesthesia on behavior ought to incorporate a substantial degree of interaction between low-level perception and higher-level interpretation of meaning. As well, we believe further study of the congruity bias observed in the viewing behavior of grapheme–color synesthetes may provide valuable insights into the cognitive and brain mechanisms underlying synesthesia.
This research was supported by the Natural Sciences and Engineering Research Council (NSERC) with a grant to D. S., a postdoctoral fellowship to M. G. R. and a graduate scholarship to J. S. A. C.
Reprint requests should be sent to Jonathan S. A. Carriere, Department of Psychology, University of Waterloo, 200 University Avenue West, Waterloo, ON, Canada N2L 3G1, or via e-mail: firstname.lastname@example.org.
It is notable that both synesthetes made a larger number of fixations than the nonsynesthetes on the free-viewing task. This observation is similar to our previous findings with a free viewing task (see Smilek, Malcolmson, et al., 2007) and we believe reflects that the displays of congruently and incongruently colored letters are more interesting for synesthetes than for nonsynesthetes.