The ability to flexibly categorize object concepts is essential to semantic cognition because the features that make two objects similar in one context may be irrelevant and even constitute interference in another. Thus, adaptive behavior in complex and dynamic environments requires the resolution of feature-based interference. In the current case study, we placed visual and functional semantic features in opposition across object concepts in two categorization tasks. Successful performance required the resolution of functional interference in a visual categorization task and the resolution of visual interference in a functional categorization task. In Experiment 1, we found that patient D. A., an individual with bilateral temporal lobe lesions, was unable to categorize object concepts in a context-dependent manner. His impairment was characterized by an increased tendency to incorrectly group objects that were similar on the task-irrelevant dimension, revealing an inability to resolve cross-modal semantic interference. In Experiment 2, D. A.'s categorization accuracy was comparable to controls when lures were removed, indicating that his impairment is unique to contexts that involve cross-modal interference. In Experiment 3, he again performed as well as controls when categorizing simple concepts, suggesting that his impairment is specific to categorization of complex object concepts. These results advance our understanding of the anterior temporal lobe as a system that represents object concepts in a manner that enables flexible semantic cognition. Specifically, they reveal a dissociation between semantic representations that contribute to the resolution of cross-modal interference and those that contribute to the resolution of interference within a given modality.