How do our abilities to process number and other continuous quantities such as time and space relate to each other? Recent evidence suggests that these abilities share common magnitude processing and neural resources, although other findings also highlight the role of dimension-specific processes. To further characterize the relation between number, time, and space, we first examined them in a population with a developmental numerical dysfunction (developmental dyscalculia) and then assessed the extent to which these abilities correlated both behaviorally and anatomically in numerically normal participants. We found that (1) participants with dyscalculia showed preserved continuous quantity processing and (2) in numerically normal adults, numerical and continuous quantity abilities were at least partially dissociated both behaviorally and anatomically. Specifically, gray matter volume correlated with both measures of numerical and continuous quantity processing in the right TPJ; in contrast, individual differences in number proficiency were associated with gray matter volume in number-specific cortical regions in the right parietal lobe. Together, our new converging evidence of selective numerical impairment and of number-specific brain areas at least partially distinct from common magnitude areas suggests that the human brain is equipped with different ways of quantifying the outside world.