We show that five individuals with congenital prosopagnosia (CP) are impaired at face recognition and discrimination and do not exhibit the normal superiority for upright over inverted faces despite intact visual acuity, low-level vision and intelligence, and in the absence of any obvious neural concomitant. Interestingly, the deficit is not limited to faces: The CP individuals were also impaired at discriminating common objects and novel objects although to a lesser extent than discriminating faces. The perceptual deficit may be attributable to a more fundamental visual processing disorder; the CP individuals exhibited difficulty in deriving global configurations from simple visual stimuli, even with extended exposure duration and considerable perceptual support in the image. Deriving a global configuration from local components is more critical for faces than for other objects, perhaps accounting for the exaggerated deficit in face processing. These findings elucidate the psychological mechanisms underlying CP and support the link between configural and face processing.