It has been found that mind wandering interferes with the sensory and cognitive processing of widespread stimuli. However, it remains unclear what factors can modulate the magnitude of the interference effects of mind wandering. Here, we investigate whether and how word familiarity modulates the interference effects of mind wandering on semantic and reafferent information processing. High- and low-frequency words were used as stimuli to induce high- and low-familiarity contexts in a sustained attention to response task, in which participants were required to respond to Chinese nonanimal words (nontarget) and withhold responses to Chinese animal words (target) as well as to intermittently report whether their state was “on task” or “off task.” Behavioral results revealed lower reaction stability for both high- and low-frequency nontarget words preceding “off-task” reports than those preceding “on-task” reports. However, ERP results revealed that low-frequency rather than high-frequency words elicited more negative N400, attenuated late positive complex, and attenuated reafferent potential for “off-task” reports than for “on-task” reports. The results suggest that mind wandering makes semantic extraction and integration more difficult for unfamiliar but not familiar two-character Chinese words and attenuates the reafferent feedback of the motor response. These findings are consistent with the decoupling hypothesis of mind wandering and provide the first neural evidence for how familiarity with external stimuli modulates the interference effects of mind wandering.