One fifth of neurons in the medial-temporal lobe of human epilepsy patients respond selectively to categories of images, such as faces or cars. Here we show that responses of hippocampal neurons are rapidly modified as subjects alternate (over 60 sec) between two tasks (1) identifying images from a category, or (2) playing a simple video game superimposed on the same images. Category-selective responses, present when a subject identifies categories, are eliminated when the subject shifts to playing the game for 87% of category-selective hippocampal neurons. By contrast, responses in the amygdala are present during both tasks for 72% of category-selective amygdalar neurons. These results suggest that attention to images is required to evoke selective responses from single neurons in the hippocampus, but is not required by neurons in the amygdala.