The perception of facial gender has been found to be adaptively recalibrated: adaptation to male faces causes participants to perceive subsequent faces as more feminine and vice versa [Webster, M. A., Kaping, D., Mizokami, Y., & Duhamel, P. Adaptation to natural facial categories. Nature, 428, 557–561, 2004]. In an event-related brain potential (ERP) study, Kovács et al. [Kovács, G., Zimmer, M., Banko, E., Harza, I., Antal, A., & Vidnyanszky, Z. Electrophysiological correlates of visual adaptation to faces and body parts in humans. Cerebral Cortex, 16, 742–753, 2006] reported reduced N170 amplitudes and increased latencies for test faces following female gender adaptation compared to control stimulus (a phase randomized face) adaptation. We examined whether this N170 attenuation to test faces was related to the adaptor's gender, or to adaptation to face exposure in general. We compared N170 effects after adaptation to either male or androgynous faces. Additionally, we investigated cross-modal adaptation for the same test faces following male or androgynous voice adaptors. Visual adaptation to face gender replicated previously reported aftereffects in classifying androgynous faces, and a similar trend was observed following adaptation to voice gender. Strikingly, N170 amplitudes were dramatically reduced for faces following face adaptors (relative to those following voice adaptors), whereas only minimal gender-specific adaptation effects were seen in the N170. By contrast, strong gender-specific adaptation effects appeared in a centroparietal P3-like component (∼400–600 msec), which in the context of adaptation may reflect a neural correlate of the detection of perceptual novelty.