The ventral visual cortex has a modular organization in which discrete and well-defined regions show a much stronger response to certain object categories (e.g., faces, bodies) than to other categories. The majority of previous studies have examined the response of these category-selective regions to isolated images of preferred or nonpreferred categories. Thus, little is known about the way these category-selective regions represent more complex visual stimuli, which include both preferred and nonpreferred stimuli. Here we examined whether glasses (nonpreferred) modify the representation of simultaneously presented faces (preferred) in the fusiform face area. We used an event-related fMR-adaptation paradigm in which faces were presented with glasses either on or above the face while subjects performed a face or a glasses discrimination task. Our findings show that the sensitivity of the fusiform face area to glasses was maximal when glasses were presented on the face than above the face during a face discrimination task rather than during a glasses discrimination task. These findings suggest that nonpreferred stimuli may significantly modify the representation of preferred stimuli, even when they are task irrelevant. Future studies will determine whether this interaction is specific to faces or may be found for other object categories in category-selective areas.