How does our brain understand the number five when it is written as an Arabic numeral, and when presented as five fingers held up? Four facets have been implicated in adult numerical processing: semantic, visual, manual, and phonological/verbal. Here, we ask how the brain represents each, using a combination of tasks and stimuli. We collected fMRI data from adult participants while they completed our novel “four number code” paradigm. In this paradigm, participants viewed one of two stimulus types to tap into the visual and manual number codes, respectively. Concurrently, they completed one of two tasks to tap into the semantic and phonological/verbal number codes, respectively. Classification analyses revealed that neural codes representing distinctions between the number comparison and phonological tasks were generalizable across format (e.g., Arabic numerals to hands) within intraparietal sulcus (IPS), angular gyrus, and precentral gyrus. Neural codes representing distinctions between formats were generalizable across tasks within visual areas such as fusiform gyrus and calcarine sulcus, as well as within IPS. Our results identify the neural facets of numerical processing within a single paradigm and suggest that IPS is sensitive to distinctions between semantic and phonological/verbal, as well as visual and manual, facets of number representations.