Visual perception and mental imagery have been shown to share a hierarchical topological visual structure of neural representation, despite the existence of dissociation of neural substrate between them in function and structure. However, we have limited knowledge about how the visual hierarchical cortex is involved in visual perception and visual imagery in a unique and shared fashion. In this study, a data set including a visual perception and an imagery experiment with human participants was used to train 2 types of voxel-wise encoding models. These models were based on Gabor features and voxel activity patterns of high-level visual cortex (i.e., fusiform face area, parahippocampal place area, and lateral occipital complex) to predict activity in the early visual cortex (EVC, i.e., V1, V2, V3) during perception, and then tested with respect to the generalization of these models to mental imagery. Our results showed that during perception and imagery, activities in the EVC could be independently predicted by the Gabor features and activity of high-level visual cortex via voxel-wise encoding models, which suggested that perception and imagery might share neural representation in the EVC. We further found Gabor-specific and non-Gabor-specific patterns of neural response to stimuli in the EVC, which were shared by perception and imagery. These findings provide insight into the mechanisms of how visual perception and imagery share representation in the EVC.