There is substantial evidence that object representations in adults are dynamically updated by learning. However, it is not clear to what extent these effects are induced by active processing of visual objects in a particular task context on top of the effects of mere exposure to the same objects. Here we show that the task does matter. We performed an event-related fMRI adaptation study in which we derived neural selectivity from a release of adaptation. We had two training conditions: “categorized objects” were categorized at a subordinate level based on fine shape differences (Which type of fish is this?), whereas “control objects” were seen equally often in a task context requiring no subordinate categorization (Is this a vase or not?). After training, the object-selective cortex was more selective for differences among categorized objects than for differences among control objects. This result indicates that the task context during training modulates the extent to which object selectivity is enhanced as a result of training.