Studies have shown that fusiform face area (FFA) activity increases with visual expertise. We present an fMRI study showing that faces from a social category made relevant by an experimental manipulation (members of an experimentally created in-group) preferentially recruited the FFA even when they were matched in exposure to face stimuli from a less significant social category (members of an experimentally created out-group). Faces were randomly assigned to groups and fully counterbalanced so that no perceptual cues allowed participants to visually distinguish category membership. The results revealed a pattern of in-group enhancement (not out-group disregard), such that the FFA was selectively engaged following the presentation of in-group compared with out-group or unaffiliated control faces even when the intergroup distinction was arbitrary, and exposure to in-group and out-group faces was equivalent and brief. In addition, individual differences in FFA activity for in-group versus out-group faces were correlated with recognition memory differences for in-group and out-group faces. The effects of group membership on the FFA were not affected by task instruction to respond to in-group or out-group members and were functionally dissociated from early visual processing in the primary visual cortex. This study provides evidence that the FFA is sensitive to top–down influences and may be involved in subordinate level (vs. superordinate level) encoding of stimuli in the absence of long-term exposure or explicit task instructions.