The ability to process the numerical magnitude of sets of items has been characterized in many animal species. Neuroimaging data have associated this ability to represent nonsymbolic numerical magnitudes (e.g., arrays of dots) with activity in the bilateral parietal lobes. Yet the quantitative abilities of humans are not limited to processing the numerical magnitude of nonsymbolic sets. Humans have used this quantitative sense as the foundation for symbolic systems for the representation of numerical magnitude. Although numerical symbol use is widespread in human cultures, the brain regions involved in processing of numerical symbols are just beginning to be understood. Here, we investigated the brain regions underlying the semantic and perceptual processing of numerical symbols. Specifically, we used an fMRI adaptation paradigm to examine the neural response to Hindu-Arabic numerals and Chinese numerical ideographs in a group of Chinese readers who could read both symbol types and a control group who could read only the numerals. Across groups, the Hindu-Arabic numerals exhibited ratio-dependent modulation in the left IPS. In contrast, numerical ideographs were associated with activation in the right IPS, exclusively in the Chinese readers. Furthermore, processing of the visual similarity of both digits and ideographs was associated with activation of the left fusiform gyrus. Using culture as an independent variable, we provide clear evidence for differences in the brain regions associated with the semantic and perceptual processing of numerical symbols. Additionally, we reveal a striking difference in the laterality of parietal activation between the semantic processing of the two symbols types.