In a social setting, seeing Sally look at a clock means something different to seeing her gaze longingly at a slice of chocolate cake. In both cases, her eyes and face might be turned rightward, but the information conveyed is markedly different, depending on the object of her gaze. Numerous studies have examined brain systems underlying the perception of gaze direction, but less is known about the neural basis of perceiving gaze shifts to specific objects. During fMRI, participants observed an actor look toward one of two objects, each occupying a distinct location. Video stimuli were sequenced to obtain repetition suppression (RS) for object identity, independent of spatial location. In a control condition, a spotlight highlighted one of the objects, but no actor was present. Observation of the human actor's gaze compared with the spotlight engaged frontal, parietal, and temporal cortices, consistent with a broad action observation network. RS for gazed object in the human condition was found in posterior intraparietal sulcus (pIPS). RS for highlighted object in the spotlight condition was found in middle occipital, inferior temporal, medial fusiform gyri, and superior parietal lobule. These results suggest that human pIPS is specifically sensitive to the type object that an observed actor looks at (tool vs. food), irrespective of the observed actor's gaze location (left vs. right). A general attention or lower-level object feature processing mechanism cannot account for the findings because a very different response pattern was seen in the spotlight control condition. Our results suggest that, in addition to spatial orienting, human pIPS has an important role in object-centered social orienting.