It is commonly believed that, in right-handed individuals, words and faces are processed by distinct neural systems: one in the left hemisphere (LH) for words and the other in the right hemisphere (RH) for faces. Emerging evidence suggests, however, that hemispheric selectivity for words and for faces may not be independent of each other. One recent account suggests that words become lateralized to the LH to interact more effectively with language regions, and subsequently, as a result of competition with words for representational space, faces become lateralized to the RH. On this interactive account, left-handed individuals, who as a group show greater variability with respect to hemispheric language dominance, might be expected to show greater variability in their degree of RH lateralization of faces as well. The current study uses behavioral measures and ERPs to compare the hemispheric specialization for both words and faces in right- and left-handed adult individuals. Although both right- and left-handed groups demonstrated LH over RH superiority in discrimination accuracy for words, only the right-handed group demonstrated RH over LH advantage in discrimination accuracy for faces. Consistent with this, increased right-handedness was related to an increase in RH superiority for face processing, as measured by the strength of the N170 ERP component. Interestingly, the degree of RH behavioral superiority for face processing and the amplitude of the RH N170 for faces could be predicted by the magnitude of the N170 ERP response to words in the LH. These results are discussed in terms of a theoretical account in which the typical RH face lateralization fails to emerge in individuals with atypical language lateralization because of weakened competition from the LH representation of words.