Active vision is a dynamic process involving the flexible coordination of different gaze strategies to achieve behavioral goals. Although many complex behaviors rely on an ability to efficiently switch between gaze-control strategies, few studies to date have examined mechanisms of task level oculomotor control in detail. Here, we report five experiments in which subjects alternated between conflicting stimulus-saccade mappings within a block of trials. The first experiment showed that there is no performance cost associated with switching between pro and anti saccades. However, follow-up experiments demonstrate that whenever subjects alternate between arbitrary stimulus-saccade mappings, latency costs are apparent on the first trial after a task change. More detailed analysis of switch costs showed that latencies were particularly elevated for saccades directed toward the same location that had been the target for a saccade on the preceeding trial. This saccade “inhibition of return” effect was most marked when unexpected error feedbacks cued task switches, suggesting that saccade selection processes are modulated by reward. We conclude that there are two systems for saccade control that differ in their characteristics following a task switch. The “reflexive” control system can be enabled/disabled in advance of saccade execution without incurring any performance cost. Switch costs are only observed when two or more arbitrary stimulus-saccade mappings have to be coordinated by a “symbolic” control system.