Results on MSR Sentence Completion. For our models (top), we show accuracies for Wikipedia (left) and in-domain (right) training. We compare with language models from prior work (left): Kneser-Ney 5-gram model (Mikolov, 2012), Word RNN (Zweig et al., 2012), Word LSTM and LdTreeLSTM (Zhang et al., 2016). We further report models incorporating distributional encodings of semantics (right): Skipgram(+RNNs) from Mikolov et al. (2013a), the PMI-based model of Woods (2016), and the Context-Embedding-based approach of Melamud et al. (2016).