Visual context influences our perception of target objects in natural scenes. However, little is known about the analysis of context information and its role in shape perception in the human brain. We investigated whether the human lateral occipital complex (LOC), known to be involved in the visual analysis of shapes, also processes information about the context of shapes within cluttered scenes. We employed an fMRI adaptation paradigm in which fMRI responses are lower for two identical than for two different stimuli presented consecutively. The stimuli consisted of closed target contours defined by aligned Gabor elements embedded in a background of randomly oriented Gabors. We measured fMRI adaptation in the LOC across changes in the context of the target shapes by manipulating the position and orientation of the background elements. No adaptation was observed across context changes when the background elements were presented in the same plane as the target elements. However, adaptation was observed when the grouping of the target elements was enhanced in a bottom-up (i.e., grouping by disparity or motion) or top-down (i.e., shape priming) manner and thus the saliency of the target shape increased. These findings suggest that the LOC processes information not only about shapes, but also about their context. This processing of context information in the LOC is modulated by figure–ground segmentation and grouping processes. That is, neural populations in the LOC encode context information when relevant to the perception of target shapes, but represent salient targets independent of context changes.