It is well established that, when comparing nonsymbolic magnitudes (e.g., dot arrays), adults can use both numerical (i.e., the number of items) and nonnumerical (density, total surface areas, etc.) magnitudes. It is less clear which of these magnitudes is more salient or processed more automatically. In this fMRI study, we used a nonsymbolic comparison task to ask if different brain areas are responsible for the automatic processing of numerical and nonnumerical magnitudes, when participants were instructed to attend to either the numerical or the nonnumerical magnitudes of the same stimuli. An interaction of task (numerical vs. nonnumerical) and congruity (congruent vs. incongruent) was found in the right TPJ. Specifically, this brain region was more strongly activated during numerical processing when the nonnumerical magnitudes were negatively correlated with numerosity (incongruent trials). In contrast, such an interference effect was not evident during nonnumerical processing when the task-irrelevant numerical magnitude was incongruent. In view of the role of the right TPJ in the control of stimulus-driven attention, we argue that these data demonstrate that the processing of nonnumerical magnitudes is more automatic than that of numerical magnitudes and that, therefore, the influence of numerical and nonnumerical variables on each other is asymmetrical.