A flash is perceived to lag spatially behind a moving object even when the two retinal images are physically aligned (flash-lag effect, FLE). Here we show that this robust illusion can be diminished by a knowledge of letters in the observer's brain. When moving and flashed segments in the FLE made the shape of a Kanji letter (ideographic characters used in Japan), the magnitude of the illusory lag perceived by Japanese subjects was significantly reduced compared to when conventional geometric (nonletter) segments were used. This diminishment was not observed when a pseudo-Kanji letter was presented to Japanese subjects or when non-Japanese English-speakers (who do not have a knowledge of Kanji) saw a real Kanji letter, indicating that the reduction in the FLE was induced by a retrieval of the knowledge (shapes of letters) stored in the observer's brain. Furthermore, measurements of neural activities by magnetoencephalography showed that the initial brain response, in which the effect of the knowledge became evident, occurred as early as 160 msec after the appearance of the flashed segment. These results demonstrated a substantial influence of knowledge on the flash-lag illusion and further suggest a rapid response of the knowledge-based perceptual pathway in the human brain.