There exists a long-standing debate regarding whether small and large numerosities engage different networks of processing. The ability to rapidly enumerate small (1–4) numerosities is referred to as “subitizing” and is thought to be qualitatively different from large numerosity processing. Functional neuro-imaging studies have attempted to dissociate neural correlates of small and large number processing by contrasting subitizing with counting of numerosities just outside the subitizing range. In the present study, we used functional magnetic resonance imaging (fMRI) to contrast the processing of numerosities in the “subitizing range” with numerosities requiring estimation. Participants compared sequentially presented slides of either dots or Arabic numerals for their relative magnitude. We show that comparison of nonsymbolic numerosities in the subitizing range led to activation of the right temporo-parietal junction, while at the same time this region was found to be suppressed during large numerosity processing. Furthermore, relative suppression of this region was strongly associated with faster response times. In previous studies, this region has been implicated in stimulus-driven attention.We therefore contend that activation of the temporo-parietal junction during small number processing and the suppression thereof during large numerosity comparisons reflects differential reliance on stimulus-driven versus goal-directed attentional networks in the brain.