The idea of a hierarchical structure of language constituents of phonemes, syllables, words, and sentences is robust and widely accepted. Empirical similarity differences at every level of this hierarchy have been analyzed in the form of confusion matrices for many years. By normalizing such data so that differences are represented by conditional probabilities, semiorders of similarity differences can be constructed. The intersection of two such orderings is an invariant partial ordering with respect to the two given orders. These invariant partial orderings, especially between perceptual and brain representations, but also for comparison of brain images of words generated by auditory or visual presentations, are the focus of this letter. Data from four experiments are analyzed, with some success in finding conceptually significant invariants.