fMRI was used to examine lexical processing in native adult Chinese speakers. A 2 task (semantics and phonology) × 2 modality (visual and auditory) within-subject design was adopted. The semantic task involved a meaning association judgment and the phonological task involved a rhyming judgment to two sequentially presented words. The overall effect across tasks and modalities was used to identify seven ROIs, including the left fusiform gyrus (FG), the left superior temporal gyrus (STG), the left ventral inferior frontal gyrus (VIFG), the left middle temporal gyrus (MTG), the left dorsal inferior frontal gyrus (DIFG), the left inferior parietal lobule (IPL), and the left middle frontal gyrus (MFG). ROI analyses revealed two modality-specific areas, FG for visual and STG for auditory, and three task-specific areas, IPL and DIFG for phonology and VIFG for semantics. Greater DIFG activation was associated with conflicting tonal information between words for the auditory rhyming task, suggesting this region's role in strategic phonological processing, and greater VIFG activation was correlated with lower association between words for both the auditory and the visual meaning task, suggesting this region's role in retrieval and selection of semantic representations. The modality- and task-specific effects in Chinese revealed by this study are similar to those found in alphabetical languages. Unlike English, we found that MFG was both modality- and task-specific, suggesting that MFG may be responsible for the visuospatial analysis of Chinese characters and orthography-to-phonology integration at a syllabic level.