Whether the human brain is equipped with a special neural substrate for numbers, or rather with a common neural substrate for processing of several types of magnitudes, has been the topic of a long-standing debate. The present study addressed this question by using functional magnetic resonance imaging (fMRI) and event-related potentials (ERPs) together with the size-congruity paradigm, a Stroop-like task in which numerical values and physical sizes were varied independently. In the fMRI experiment, a region-of-interest analysis of the primary motor cortex revealed interference effects in the hemisphere ipsilateral to the response hand, indicating that the stimulus-stimulus conflict between numerical and physical magnitude is not completely resolved until response initiation. This result supports the assumption of distinct comparison mechanisms for physical size and numerical value. In the ERP experiment, the cognitive load was manipulated in order to probe the degree to which information processing is shared across cognitive systems. As in the fMRI experiment, we found that the stimulus-stimulus conflict between numerical and physical magnitude is not completely resolved until response initiation. However, such late interaction was found only in the low cognitive load condition. In contrast, in the high load condition, physical and numerical dimensions interacted only at the comparison stage. We concluded that the processing of magnitude can be subserved by shared or distinct neural substrates, depending on task requirements.