The importance of the left occipitotemporal cortex for visual word processing is highlighted by numerous functional neuroimaging studies, but the precise function of the visual word form area (VWFA) in this brain region is still under debate. The present functional magnetic resonance imaging study varied orthographic familiarity independent from phonological-semantic familiarity by presenting orthographically familiar and orthographically unfamiliar forms (pseudohomophones) of the same words in a phonological lexical decision task. Consistent with orthographic word recognition in the VWFA, we found lower activation for familiar compared with unfamiliar forms, but no difference between pseudohomophones and pseudowords. This orthographic familiarity effect in the VWFA differed from the phonological familiarity effect in left frontal regions, where phonologically unfamiliar pseudowords led to higher activation than phonologically familiar pseudohomophones. We suggest that the VWFA not only computes letter string representations but also hosts word-specific orthographic representations. These representations function as recognition units with the effect that letter strings that readily match with stored representations lead to less activation than letter strings that do not.