The distinction between letter strings that form words and those that look and sound plausible but are not meaningful is a basic one. Decades of functional neuroimaging experiments have used this distinction to isolate the neural basis of lexical (word level) semantics, associated with areas such as the middle temporal, angular, and posterior cingulate gyri that overlap the default mode network. In two fMRI experiments, a different set of findings emerged when word stimuli were used that were less familiar (measured by word frequency) than those typically used. Instead of activating default mode network areas often associated with semantic processing, words activated task-positive areas such as the inferior pFC and SMA, along with multifunctional ventral occipitotemporal cortices related to reading, whereas nonwords activated default mode areas previously associated with semantics. Effective connectivity analyses of fMRI data on less familiar words showed activation driven by task-positive and multifunctional reading-related areas, whereas highly familiar words showed bottom–up activation flow from occipitotemporal cortex. These findings suggest that functional neuroimaging correlates of semantic processing are less stable than previously assumed, with factors such as word frequency influencing the balance between task-positive, reading-related, and default mode networks. More generally, this suggests that results of contrasts typically interpreted in terms of semantic content may be more influenced by factors related to task difficulty than is widely appreciated.