From Word Types to Tokens and Back: A Survey of Approaches to Word Meaning Representation and Interpretation

Word representation in vector space lies in the core of distributional approaches to language processing. The idea that words’ collocations describe their meaning (Harris 1954; Firth 1957) underlies Distributional Semantic Models (DSMs) and the structure of the semantic space built by neural language models. Different approaches, however, address different units of meaning representation. DSMs represent words by aggregating over their usages in a corpus of documents (Landauer and Dumais 1997; Lund and Burgess 1996). Similarly, word embedding approaches such as word2vec, GloVe, and fastText generate a static vector per word type, which groups its different senses (Mikolov et al. 2013a; Pennington, Socher, and Manning 2014; Bojanowski et al. 2017). Contextual language models, on the contrary, generate dynamic representations that change for every new occurrence of a word in texts and directly encode the contextualized meaning of individual tokens (Peters et al. 2018; Devlin et al. 2019; Liu et al. 2019). Contrary to a static embedding model which would propose a single vector for a polysemous word like bug, a contextual model would generate different representations for instances of the word in context (e.g., “There is a bug in my soup”, “There is a bug in my code”).

Contextualized representations constitute a powerful feature of state-of-the-art language models, and contribute to their impressive performance in downstream tasks. Their flexibility confers them an undeniable advantage over static embeddings which, by aggregating information from different contexts in the same word vector, often lead to the “meaning conflation” problem (Pilehvar and Camacho-Collados 2020). Additionally, the dynamic nature of contextualized vectors provides a more straightforward way for capturing meaning variation than previous sense representation methodologies (Reisinger and Mooney 2010; Iacobacci, Pilehvar, and Navigli 2015; Camacho-Collados and Pilevar 2018). In DSMs, this type of contextualization was performed through word vector composition, where the basic vector for a word was adapted to a new context of use by being combined with the vectors of the words in the context (Mitchell and Lapata 2008; Erk and Padó 2008; Thater, Fürstenau, and Pinkal 2011; Dinu and Lapata 2010; Dinu, Thater, and Laue 2012). In deep contextual language models, every word is influencing every other word in a sequence and all the representations are getting updated in different layers of the model based on this distributional information.

The dynamic character of contextualized representations also poses some challenges for meaning representation. Although modeling word usage is one of their recognized merits and a highly useful methodological tool for studying linguistic structure (Linzen, Dupoux, and Goldberg 2016; Hewitt and Manning 2019), the observed context variation makes the study of the encoded semantic knowledge challenging (Ethayarajh 2019b; Mickus et al. 2020; Timkey and van Schijndel 2021). We, thus, witness in recent work a resurgence of interest towards more abstract, higher (word type) level, representations, deemed to provide a more solid basis for meaning exploration. Naturally, this trend is mainly observed in the lexical semantics field where the notion of lexical concept is central (Lauscher et al. 2020; Liu, McCarthy, and Korhonen 2020; Bommasani, Davis, and Cardie 2020; Vulić et al. 2020b; Garí Soler and Apidianaki 2021a).

The prevalence of contextual models in the field of computational linguistics has also brought about a shift from out-of-context word similarity and analogy tasks—used for evaluating static embedding quality (Mikolov et al. 2013b)—to interpretation tools common in human language learning studies (such as cloze tasks and probes) (Linzen, Dupoux, and Goldberg 2016; Kovaleva et al. 2019; Tenney et al. 2019; Ettinger 2020). These serve to assess the linguistic and world knowledge encoded in contextualized vectors, and are often complemented with methods that explore the models’ inner workings (Voita, Sennrich, and Titov 2019; Hewitt and Manning 2019; Clark et al. 2019; Voita et al. 2019; Tenney, Das, and Pavlick 2019). In lexical semantics, probing is used to explore the knowledge that the models encode about the semantic properties of words and their relationships (Petroni et al. 2019; Bouraoui, Camacho-Collados, and Schockaert 2020; Ravichander et al. 2020; Apidianaki and Garí Soler 2021), or their understanding of semantic scope and negation (Ettinger 2020; Lyu et al. 2022). Nevertheless, evaluations that rely on probing are not always indicative of the knowledge that is encoded by the models. Language models are brittle to small changes in the used prompts, and the output strongly depends on prompt quality and naturalness (Ettinger 2020; Ravichander et al. 2020; Apidianaki and Garí Soler 2021; Jiang et al. 2020). Furthermore, the output of semantic probes is difficult to evaluate, since there might be multiple valid answers and possible fillers (e.g., red, tasty, and fruits would all be good fillers for the masked slot in the query “Strawberries are [MASK]”). These issues have brought attention back to word similarity and analogy tasks, considered to be more established and mature for exploring the concept-related knowledge encoded in language model representations (Vulić et al. 2020b; Bommasani, Davis, and Cardie 2020).

The goal of this survey is to provide an overview of word meaning representation methodologies and evaluation practices. It will put current developments into perspective with respect to previous representation and evaluation paradigms, discuss their specificities, and highlight the issues that have been addressed and the challenges that remain. Special focus will be put to word type (static) and word token (dynamic) embedding approaches. We will also discuss methods for deriving word type-level vectors from contextualized representations. This back-and-forth between representation types and evaluation strategies nourishes active discussions in the community. Our goal is to clarify their respective strengths and weaknesses, and to open up perspectives for future research.

The overview of the methods proposed in this survey is not intended to be exhaustive. Our main concern has been to include work that is representative of the evolution and trends on the topic of word meaning representation in the past years. Nevertheless, given the pace in which the field evolves and the actual space constraints this publication needs to abide by, it is practically impossible to include a full account of existing work. Furthermore, the majority of the methods and datasets that will be presented have been developed for the English language. We include a discussion of results obtained in other languages when needed in order to highlight the cross-lingual generalization potential of the presented methods—or their limitations in this respect—as well as the methodological differences and design choices that apply in a multilingual setting.

An overview table of the survey contents is given in Figure 1. Section 2 presents methodologies that generate embeddings at the level of word types and word tokens. We discuss their strengths and limitations, as well as solutions that have been proposed to address the latter, including the generation of embeddings at the level of senses. The section includes a critical presentation of benchmarks commonly used for evaluating word embedding quality. Section 3 presents methods that specialize static and contextualized word embeddings for semantic relationships during (pre-)training or at a post-processing stage. Section 4 presents interpretation and evaluation methods aimed at exploring the semantic knowledge that is encoded in contextual embedding representations. We discuss the challenges posed by probing methodologies for lexical semantic analysis. We also explain how the geometry of the vector space that is built by contextual language models can provide insights into the quality of the representations, and highlight factors that might complicate the derivation of high quality similarity estimates. In Section 5, we present methods that generate word type-level representations from contextualized vectors, and methods that combine static and dynamic embeddings in order to leverage their respective strengths and address their limitations. The Conclusion includes a discussion of perspectives for future work in word meaning representation.

This section provides an overview of word and meaning representation methodologies that rely on language models. We present approaches that generate distributed representations (embeddings) at the level of word types, senses, and tokens. Links with distributional approaches are established when needed in order to better understand the evolution of embedding representations, or to explain their advantages over count-based distributional models. For a full account of distributional approaches and their origins, we point the reader to the survey paper by Turney and Pantel (2010). The interaction between distributional and formal semantics is explained in Boleda and Herbelot (2016). For a thorough look into embeddings generated by different types of language models, we refer the reader to the book by Pilehvar and Camacho-Collados (2020).

Word embedding models leverage neural networks to learn low-dimensional word representations from corpora (Bengio et al. 2003; Collobert and Weston 2008; Collobert et al. 2011; LeCun, Bengio, and Hinton 2015; Mikolov et al. 2013a). These “self supervision” models are trained on raw text and rely on the optimization of a language modeling objective. The vector estimation problem is framed directly as a supervised task, where the weights in a word vector are set to maximize the probability of the contexts in which the word is observed in the corpus used for training. The popular Continuous Bag-of-Words (CBOW) word2vec model architecture (Mikolov et al. 2013a) is based on the feedforward neural network language model (Bengio et al. 2003); the task is to predict the current word (w_t) using its surrounding context (W_t = w_t−n, …, w_t, …, w_t+n) minimizing a loss function. In word2vec Skip-gram, on the contrary, the goal is to predict the words in the surrounding context given the target word (w_t).

The idea underlying word embedding models is that contextual information can provide a good approximation to word meaning since semantically similar words tend to have similar contextual distributions (Harris 1954; Firth 1957; Miller and Charles 1991). This is also the guiding principle of DSMs (Turney and Pantel 2010; Erk 2012; Clark 2015), while the idea of meaning as distribution goes back to Wittgenstein (1953) who wrote that ‘the meaning of a word is its use in the language’.¹ In count-based methods, vectors keep track of the contexts where words appear in a large corpus (i.e., their co-occurrences) as proxies for meaning representation. In both cases, word similarity can be measured by applying geometric techniques (e.g., cosine similarity or Euclidean distance) to the obtained (embedding or count-based) vectors. Additionally, similar to DSMs, the self-supervised embedding learning approach requires no manual annotations and the models can be trained on raw text. Both methodologies can thus be applied to different languages given that large-scale unannotated corpora are available.

Low dimensionality is considered an advantage of word embeddings over count-based vectors (Baroni, Dinu, and Kruszewski 2014). In DSMs, vector dimensions correspond to words in the vocabulary (V) so their number can easily reach hundreds of thousands or even millions, depending on the corpus the vectors are trained on. Storing each word w ∈ V in a |V|-dimensional vector results in a very large matrix with |V|² cells. Additionally, the generated vectors are sparse, containing a small number of non-zero elements. The high dimensionality and sparseness of distributional vectors challenge both the scalability of the models and their computational efficiency.

A common approach to alleviate the sparseness of distributional representations and improve their performance in semantic tasks is to apply some type of transformation to the raw vectors. This involves reweighting the counts for context informativeness and smoothing them with dimensionality reduction techniques (e.g., Singular Value Decomposition [SVD]) (Turney and Pantel 2010). The applied optimization process is generally unsupervised and based on independent (for example, information-theoretic) considerations (Baroni, Dinu, and Kruszewski 2014). Such transformations are not needed with word embedding techniques which involve a single supervised learning step. Word embedding models generate low-dimensional vectors which are more compact than count-based vectors. Consequently, similarity calculations and other operations on these vectors are fast and efficient.² Much of the power of word embedding models derives from their ability to compress distributional information into a lower-dimensional space of continuous values.

Pretrained word embeddings outperform count-based representations in intrinsic evaluations (i.e., word similarity and relatedness tasks) (Mikolov, Yih, and Zweig 2013; Baroni, Dinu, and Kruszewski 2014), and can be successfully integrated in downstream applications due to their high generalization potential. They also present limitations. Models like word2vec (Mikolov et al. 2013a), GloVe (Pennington, Socher, and Manning 2014), and fastText (Bojanowski et al. 2017), for example, are by design unable to model polysemy, since they build a single representation for each word in the vocabulary of a language. The contextual evidence for different word meanings is thus conflated into a single vector.

Modeling a word type as a single point in the semantic space is considered as a major deficiency of static embedding models. Not distinguishing between different meanings of a polysemous word (e.g., plant, mouse, bug) can negatively impact the semantic understanding of NLP systems that rely on these representations. Additionally, meaning conflation has consequences on the structure of the obtained semantic space and on semantic modeling accuracy, since the vectors of unrelated words are pulled closer together (Neelakantan et al. 2014; Chen, Liu, and Sun 2014; Camacho-Collados and Pilevar 2018). This is illustrated in Figure 2 by the proximity of rat, cat, and keyboard, due to their similarity to different senses of the noun mouse. A careful analysis shows that multiple word senses reside in linear superposition within word2vec and GloVe word embeddings, and that vectors that approximately capture the senses can be recovered using simple sparse coding (Arora et al. 2018). In the distributional semantics literature, context-specific representations for words were generated through vector composition (Schütze 1998; Mitchell and Lapata 2008; Baroni and Zamparelli 2010; Zanzotto et al. 2010), sometimes taking into consideration the syntactic role and selectional preferences of words in the sentence (Padó and Lapata 2007; Erk and Padó 2008; Thater, Fürstenau, and Pinkal 2011).³

Another shortcoming of the dense continuous-valued vector representations that are learned by word embedding models is that they lack interpretable dimensions, limiting our understanding of the semantic features they actually encode (Chersoni et al. 2021; Petersen and Potts 2022). This is in contrast to co-occurrence-based distributional vectors, where features can deliver direct and interpretable insights. In spite of their low dimensionality, word embeddings are still able to capture word similarity due to the objective used for training, which makes them create similar vectors for similar words.

The next section describes the methodology most commonly used for evaluating word type-level embeddings.

Word embeddings have often been intrinsically evaluated against manually compiled word analogy, similarity, and relatedness datasets, which test their capability to represent word meaning. This section presents the most common approaches and datasets used in this goal. Although these datasets are not perfect and their use as a test bed for evaluation has often been criticized, they still remain interesting and relevant for this survey.⁴

Word analogy has been extensively used for evaluating the quality of static word embeddings. It is usually framed as a relational similarity task, and models the idea that pairs of words may hold similar relations to those that exist between other pairs of words (Turney 2006). In the equation a : b :: c : d (which reads as “a is to b as c is to d”), the first three terms (a, b, c) are given and the tested model needs to predict the word that stands for d. Mikolov, Yih, and Zweig (2013) showed that such relations are reflected in vector offsets between word pairs.⁵ In the famous example “man is to king as woman is to X”, the embedding for the word queen can be roughly recovered from the representations of king, man, and woman using the following equation: $queen→≈king→−man→+woman→$⁠. Benchmarks commonly used for this type of evaluation include the Google analogy set (Mikolov et al. 2013a),⁶ the Microsoft Research Syntactic (MSR) analogies dataset (Mikolov, Yih, and Zweig 2013), and the SemEval 2012 Task 2 “Measuring Degrees of Relational Similarity” dataset (Jurgens et al. 2012).

In spite of their popularity, word analogies have been progressively discredited as a test bed for evaluation due to numerous concerns regarding their validity. First, the accuracy of the vector offset method depends on the proximity of the target vector to its source (e.g., $queen→$ and $king→$⁠), limiting its applicability to linguistic relations that happen to be close in the vector space (Rogers, Drozd, and Li 2017). Reliance on cosine similarity also conflates offset consistency with largely irrelevant neighborhood structure (Linzen 2016). Linzen also notes that results are inconsistent when the direction of the analogy is reversed, even though the same offset is involved in both directions. Moreover, linguistic relations might not always translate to linear relations between vectors but to more complex correspondence patterns (Drozd, Gladkova, and Matsuoka 2016; Ethayarajh 2019b). The classic implementation of the analogy task is also problematic; examples are structured in such a way that given the first three terms, there is one specific correct fourth term. This might be the case with factual queries involving morpho-syntactic and grammatical alternations (e.g., high : higher :: long : X), but for semantic queries there might be several equally plausible correct answers (e.g., man:doctor :: woman:X) (Nissim, van Noord, and van der Goot 2020).⁷ The usual implementation of this type of evaluation, which excludes premise vectors from predictions, is also problematic (Schluter 2018).⁸ Finally, the queries often reflect subjective biases that compromise the value of analogies as a bias detection tool.

Another way to evaluate the quality of word representations is to compare their similarity against human semantic similarity and relatedness judgments. A high correlation between human judgments on word pairs and the cosine of the corresponding vectors is perceived as an indication of the quality of the constructed space. Similarity describes a tighter relationship between synonyms or words linked with “IS-A” (hypernymy) relations (e.g., a car IS-A vehicle), while related words have some other type of connection (they might be meronyms or holonyms) or are topically associated (Agirre et al. 2009; Bruni, Tran, and Baroni 2014). For example, house is similar to building, and is also related to brick and garden.

The bulk of these datasets have been compiled in the context of linguistic and psycholinguistic studies (Rubenstein and Goodenough 1965; Miller and Charles 1991; Hodgson 1991; Finkelstein et al. 2001; Bruni et al. 2012; Hill, Reichart, and Korhonen 2015; Gerz et al. 2016; Pilehvar et al. 2018; Vulić et al. 2020a). Such datasets also serve to assess the proficiency of English language learners (e.g., the TOEFL dataset), and to evaluate distributional models in dedicated shared tasks (Jurgens et al. 2012).⁹ However, there are some issues with this type of evaluation too. First, the same word pairs may be rated differently in similarity and relatedness datasets (Bruni et al. 2012; Hill, Reichart, and Korhonen 2015). Second, judgments for related word classes (cat-dog) are more reliable than for unrelated words (cat-democracy) (Kabbach and Herbelot 2021). Another downside of this type of evaluation is that similarity scores are assigned to pairs of words in isolation. Consequently, the comparison of static embeddings to these scores does not allow to assess the capability of the models to capture polysemy and word meaning in context.

Multi-prototype methods were proposed as a solution to the meaning conflation problem of static word embeddings. These methods generate separate vectors for the different senses of a word, which are often discovered from text corpora using unsupervised Word Sense Induction methods. The contexts where a word occurs are clustered, and cluster centroids are used as prototype vectors. The multi-prototype method of Reisinger and Mooney (2010) is illustrated in Figure 3.

Multi-prototype methods vary with respect to the vector representations, the clustering algorithm, and the context used. Reisinger and Mooney (2010) use count-based vectors composed of features that correspond to unigrams in a 10-word context window around a target word w_t, while Huang et al. (2012) and Neelakantan et al. (2014) use word embeddings. For clustering, Reisinger and Mooney apply a mixture of von Mises-Fisher distributions (movMF) clustering method. Huang et al. (2012) use the K-means algorithm to decompose word embeddings into multiple prototypes. In the Multiple-Sense Skip-Gram (MSSG) method of Neelakantan et al. (2014), clustering and sense embedding learning are performed jointly during training. The multi-prototype Skip-gram model of Tian et al. (2014) has fewer parameters and is trained using the Expectation-Maximization algorithm. In contrast to methods where senses are induced from words’ local context, Liu et al. (2015) propose Topical Word Embeddings (TWE). This method allows each word to have different embeddings under different topics computed globally using latent topic modeling (Blei, Ng, and Jordan 2003).

Multi-prototype embedding methods offer a way to capture and represent senses, but also face a number of challenges. In early methods, the number of clusters (or senses) k was a parameter that had to be pre-defined. This number was sometimes chosen arbitrarily and used for all words, independently of their polysemy (Huang et al. 2012). Moreover, these methods are generally offline and difficult to adapt to new data and domains, or to capture new senses (Chen, Liu, and Sun 2014). An alternative has been to use non-parametric clustering which allows to dynamically adjust the number of senses to each word. The method of Neelakantan et al. (2014) precisely relies on the notion of “facility location” (Meyerson 2001); a new cluster is created online during training with probability proportional to the distance from the context to the nearest cluster (sense). The bigger this distance, the higher the probability that the context describes a new sense of the word. Similarly, the method of Li and Jurafsky (2015) learns embeddings for senses of a word induced using the Chinese Restaurant Processes (Blei et al. 2003), a practical interpretation of Dirichlet Processes (Ferguson 1973) for non-parametric clustering. In this approach too, a word is associated with a new sense vector when evidence in the context (its neighboring words) suggests that it is sufficiently different from its previously identified senses.

Other concerns that have been expressed with respect to multi-prototype methods are that the clusters are not always interpretable (i.e., it is difficult to identify the senses they correspond to), and the representations obtained for rare senses are unreliable (Pilehvar and Collier 2016). Finally, the usefulness of using this type of sense embeddings in downstream tasks is unclear. These have been shown to outperform previous word embedding representation methods in intrinsic evaluations, but when tested in real NLP applications they seem to benefit some tasks (part-of-speech tagging and semantic relation identification) and harm others (sentiment analysis and named entity extraction) (Li and Jurafsky 2015).

Seeking a more stable criterion than clustering for sense identification, several studies have proposed to use translations as proxies for senses. This idea dates back to work by Gale, Church, and Yarowsky (1992), where it was put forward as a solution to the knowledge acquisition bottleneck, and has since been adopted in numerous word sense induction and disambiguation approaches (Dagan and Itai 1994; Dyvik 1998, 2002, 2005; Resnik and Yarowsky 1999; Ide, Erjavec, and Tufis 2002; Resnik 2004; Diab and Resnik 2002; Apidianaki 2008, 2009; Lefever, Hoste, and De Cock 2011; Carpuat 2013). The underlying assumption is that the senses of a polysemous word in a source language (w_s) are translated with different words (T = t₁, …, t_n) in other languages. Clustering is still relevant in this context since sets of synonymous translations may describe the same sense of word w_s (Apidianaki 2008, 2009).

Translations have also served to create embeddings for word senses. Guo et al. (2014) project clusters of English translations describing senses onto Chinese words in a parallel corpus, in order to create the labeled data needed for training a neural network model that generates sense embeddings. The sense embedding method of Šuster, Titov, and van Noord (2016) also exploits monolingual and translation information. Their model consists of an encoding part which assigns a sense to a given word (called “pivot”), and a decoding (or reconstruction) part that predicts context words based on the pivot word and its sense. Parameters of encoding and reconstruction are jointly optimized, the goal being to minimize the error in recovering context words based on the pivot word and its assigned sense. Finally, methods that form rich context-aware features and vectors for source language words and phrases have also served to improve translation quality in Phrase-Based Statistical Machine Translation and Neural Machine Translation systems (Carpuat and Wu 2007; Apidianaki et al. 2012; Liu, Lu, and Neubig 2018).

Sense embedding methods produce vectors for senses found in lexicographic resources, sometimes combining this knowledge with information from large text corpora. A merit of this approach is that the generated sense vectors are more interpretable than clustering-induced senses (Camacho-Collados and Pilevar 2018). A typical sense embedding procedure is illustrated in Figure 4.

The SensEmbed method of Iacobacci, Pilehvar, and Navigli (2015) and the Senses and Words to Vector (SW2V) method of Mancini et al. (2017) both learn sense representations from disambiguated texts.¹⁰ A difference between the two approaches is that the former produces sense representations only, while the latter jointly learns word and sense embeddings which share the same unified vector space. In both methods, the quality of the generated sense representations strongly depends on the success of the disambiguation step. The method of Chen, Liu, and Sun (2014) alleviates this dependence by learning representations from sense definitions (glosses) in WordNet (Fellbaum 1998). Each sense is represented by the average of the vectors of the content words in the gloss that are most similar to the target word. The training objective of Skip-gram is then modified in order to obtain vectors that are good at predicting not only a word’s context words, but also its senses. Similarly, Rothe and Schütze (2017) propose a model called AutoExtend which learns embeddings for WordNet synsets. The embedding for a word corresponds to the sum of the embeddings of its senses, and the vector for a synset corresponds to the sum of the senses contained in the synset.

Sense embedding approaches provide a clear solution to the meaning conflation problem of word embeddings, but they are tied to an external semantic lexicon. State of the art contextual language models, on the contrary, produce vectors that capture the meaning of individual tokens in a more straightforward way. The next section describes different contextual language models with special focus on the widely used Transformer-based BERT model.

Contextual language models constitute a new representation paradigm where the generated embeddings encode the meaning of individual word tokens (Peters et al. 2018; Devlin et al. 2019; Liu et al. 2019). Contrary to static embeddings which describe word types (e.g., there is only one word2vec vector for the noun bug), contextual models assign different vectors to different instances of the same word depending on the context of use (e.g., “There is a bug in my soup”, “There is a bug in my code”). These vectors are dynamic and can capture subtle meaning nuances expressed by word instances, alleviating, at the same time, the meaning conflation problem of static embeddings and sense embeddings’ reliance on lexicographic resources.

Vector contextualization has been extensively studied with respect to DSMs, well before the appearance of contextual language models. This was achieved using vector composition methods, which build representations that go beyond individual words to obtain word meanings in context (Mitchell and Lapata 2008; Erk and Padó 2008; Dinu and Lapata 2010; Thater, Fürstenau, and Pinkal 2011). Specifically, the contextualized meaning of a target word w_t in context c was obtained by creating a vector that combined the vectors of w_t and of the words {w₁, …, w_n} in c, using some operation such as component-wise multiplication or addition. Some models also use latent semantic dimensions. The model of Dinu and Lapata (2010), for example, represents word meaning as a probability distribution over a set of latent senses reflecting the out-of-context likelihood of each sense. The contextualized meaning of a word is then modeled as a change in the original sense distribution.¹¹ The model of Van de Cruys, Poibeau, and Korhonen (2011) exploits the latent space to determine the features that are important for a particular context and adapts the out-of-context (dependency-based) feature vector of the target word accordingly, allowing for a more precise and more distinct computation of word meaning in context. Thater, Fürstenau, and Pinkal (2011), on the contrary, use no explicit sense representation but rather derive a contextualized vector from the basic meaning vector of a target word by reweighting its components on the basis of the context of occurrence.¹² They observe that retaining only the dimensions that correspond to the word’s syntactic neighbors results in an extremely sparse vector (with zero values for most of its dimensions). They thus propose to leverage semantic similarity information about the context words and to also retain dimensions that are distributionally similar to them, weighted by their similarity score.

ELMo (Peters et al. 2018) relies on a bidirectional LSTM (biLSTM) (Hochreiter and Schmidhuber 1997; Graves and Schmidhuber 2005) trained on a large corpus with a language modeling objective. The original model consists of three layers: A character n-gram convolutional layer which is followed by a two-layer bidirectional LSTM, as shown in Figure 5(a). ELMo representations are a linear combination of the internal layers of the model. The representation that is generated for each token is a combination of the hidden states of the two BiLSTM layers which encode the context-sensitive representation of the word, and the static representation of the word which is character-based. When ELMo is integrated into task-specific architectures, the task and the linear combination of different layers are simultaneously learned in a supervised way.

BERT (Devlin et al. 2019) is a very widely used contextual language model. It relies on the Transformer architecture (Vaswani et al. 2017) which was initially developed for sequence-to-sequence (seq2seq) tasks such as machine translation. The goal was to simplify the Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) architectures previously used. These encoder-decoder models typically used an attention mechanism. The Transformer removed recurrence and convolutions, and relied entirely on the “self-attention” mechanism. This fully attention-based approach, where the representation of a sequence is computed by relating different words (positions) in the same sequence, shows improved performance compared to previous architectures in numerous NLP tasks. Additionally, attention is a useful interpretation tool which shows how the model assigns weight to different input elements when performing specific tasks (Raganato and Tiedemann 2018; Voita, Sennrich, and Titov 2019; Kovaleva et al. 2019; Rogers, Kovaleva, and Rumshisky 2020).

Contrary to ELMo, where a forward and a backward language model are separately trained (cf. Figure 5(a)), BERT relies on a bidirectional model which jointly conditions on the left and right context in all layers (cf. Figure 5(b)). BERT is pre-trained using two objectives, Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). MLM is similar to a Cloze task (Taylor 1953). In MLM, a portion of the input tokens is masked at random (e.g., The cat [MASK] on the mat) and the model has to predict them based on the context.¹³ In NSP, the model needs to predict whether two segments follow each other in the original text. The goal of this objective is to improve performance in downstream tasks that require reasoning about the relationships between pairs of sentences (e.g., NLI or Question Answering). Sentence pairs are grouped into a single sequence and separated with a special token ([SEP]).

BERT receives as input a combination of token embeddings, position embeddings, and segment embeddings, as shown in Figure 6. The position embedding shows where the token occurs in the input string, and the segment embedding indicates whether it occurs in the first or the second sentence (A or B). These three vectors are added element-wise to deliver the representation of the word which will be passed through the Transformer layers. Each word in the sentence influences every other word, and the representations are updated based on this contextual information due to the dense inter-connections inside the Transformer. The first token of every sequence is a special token ([CLS]), the final hidden state of which is used as the aggregate sequence representation. BERT can be fine-tuned for different tasks by simply adding a classification or regression head on top of the [CLS] token.

Two pre-trained English BERT models are available (BERT_base and BERT_large) which were trained on the BooksCorpus (800M words) (Zhu et al. 2015) and the English Wikipedia (2,500M words).¹⁴ BERT models are trained with a specific kind of tokenization where words are split into smaller units called WordPieces (Wu et al. 2016). For example, the word playing in Figure 6 is split into two pieces, play and ##ing. Word vectors can be derived from these subword-level representations using different mechanisms, described in the next section.

Word embedding methods often operate at the subword level. The embedding for a word corresponds, in this case, to the average of its subword embeddings. This subword pooling operation is a common and necessary first step for generating a representation for a word that has been broken into smaller pieces, but is not always explicitly stated as a separate step in research papers.

BERT-like models specifically use WordPiece tokenization (Wu et al. 2016). The most frequent words in the training corpus are represented as a single token, while other less frequent words might be split into multiple wordpieces. This process yields w¹, …, w^k pieces for a word w, which can be concatenated in order to form the word (cat(w¹, …, w^k) = w) (Bommasani, Davis, and Cardie 2020; Vulić et al. 2020b). The final representation for the word is constructed by taking the average over the subword encodings, further averaged over n ≤ N layers where N is the number of Transformer layers.¹⁵ Apart from the arithmetic mean (mean(·)), other mechanisms for aggregating these vectors include element-wise min or max pooling (min(·), max(·)). It is also possible to use the last vector of a word (last(·)), discarding the representations of earlier layers (Bommasani, Davis, and Cardie 2020).

Subword units were used in earlier models as well, since they provide the flexibility needed to account for rare, unknown or out-of-vocabulary (OOV) words (i.e., not seen in the training data). This subword information allows the models to improve the representation of morphologically complex words (formed via compounding, affixation, or inflection) and to capture the explicit relationship among morphological variants (Luong and Manning 2016). This is especially important in the case of morphologically rich languages where a word (verb or noun) might have a high number of inflected forms or cases, the majority of which occur rarely in the corpora used for model training (Bojanowski et al. 2017). In the context of NMT, Sennrich, Haddow, and Birch (2016) proposed to encode unknown words as sequences of subword units in order to enable open-vocabulary translation. The idea was that the morphemes of morphologically complex words can be translated separately, and that character-level translation rules can be used for cognates and loanwords with a common origin (Tiedemann 2012). The segmentation into subword units allows the model to take into account morphology when learning word representations, and to learn translations that it can generalize to unseen words. Their segmentation techniques included simple character n-gram models and a variant of the byte pair encoding (BPE) compression algorithm, which merges frequent character n-grams into a single symbol (Gage 1994).

Character-level embedding models like fastText (Bojanowski et al. 2017) learn representations directly from characters, which also allows to form robust representations for OOV tokens. Similarly, the Charagram model (Wieting et al. 2016) embeds a character sequence (word or sentence) by adding the vectors of its character n-grams. ELMo representations are also character-based (Peters et al. 2018; Jozefowicz et al. 2016). The model uses a character CNN. The produced contextualized representations are a function of the internal states of a deep bidirectional language model (biLM). In the model of Kim et al. (2016), C is the vocabulary of characters, d is the dimensionality of character embeddings, and Q ∈ ℝ^d×|C| is the matrix of character embeddings. If a word k ∈ V of length l is made up of a sequence of characters [c₁, …, c_l], then the character-level representation of k is given by the matrix C^k ∈ ℝ^d×l, where the j-th column corresponds to the character embedding for c_j (i.e., the c_j-th column of Q). Words are then represented as the sum of their character n-gram vectors followed by an elementwise nonlinearity.¹⁶ A character-based variant of BERT has also been proposed as an alternative to re-training BERT for specialized domains, where the general-domain wordpiece vocabulary might not be optimal (El Boukkouri et al. 2020). CharacterBERT uses a Character-CNN module (Peters et al. 2018) to produce a single embedding representation for a word, which is then added to position and segment embeddings. Pre-training is carried out as in BERT but in MLM, the model predicts entire words instead of wordpieces. Each input token is assigned a single final contextual representation by the model.

BERT-type models trained on monolingual text exist in several languages (e.g., Martin et al. 2020; Le et al. 2020; Cañete et al. 2020; Koutsikakis et al. 2020; Virtanen et al. 2019). The multilingual BERT (mBERT) model was (pre-)trained on the 104 languages with the largest Wikipedias.¹⁷ mBERT uses a 110k shared WordPiece vocabulary which is mostly English-driven, often resulting in arbitrary partitionings in other languages. This suboptimal tokenization has a negative impact on the quality of the lexical knowledge that is encoded in the representations (Garí Soler and Apidianaki 2021a, b). Language-specific monolingual models generally perform better and contain more linguistic information for a particular language than their multilingual counterparts (Vulić et al. 2020b). This is due to the trade-off that is observed when the number of languages scales up but model capacity remains fixed, also described as the “curse of multilinguality”. As noted by Conneau et al. (2020a), encompassing more languages leads to better cross-lingual performance in low-resource languages up to some point, after which the overall performance on both monolingual and cross-lingual benchmarks degrades. In other words, the models tend to sacrifice monolingual information coverage for a wider language coverage. Still, very large multilingual models (e.g., XLMR-L) perform on par with language-specific BERT models in some tasks such as multilingual WSD (Pasini, Raganato, and Navigli 2021), mainly because of the difference in model size.¹⁸

Lighter BERT-inspired models also exist. DistilBert (Sanh et al. 2019) and ALBERT (A Lite BERT) (Lan et al. 2020) have significantly fewer parameters than BERT, and still yield high performance in Natural Language Understanding (NLU) tasks. RoBERTa (Liu et al. 2019) is trained longer and with larger batches than BERT, over more data and on longer sequences. The NSP objective is removed, and a dynamic masking pattern is applied to the training data. SpanBERT (Joshi et al. 2020) masks random contiguous spans of variable length instead of individual tokens. It replaces BERT’s MLM objective by a span-boundary objective, where the model learns to predict the entire masked span from the observed tokens at its boundary. Also, SpanBERT is pre-trained on single segments, allowing the model to learn longer-range features. AMBERT (Zhang, Li, and Li 2021) adopts a multi-grained tokenization approach and generates representations for words, sub-word pieces, and phrases. Fine- and coarse-grained representations are learned in parallel using two encoders with shared parameters and MLM. The model is fine-tuned for classification using the [CLS] representations created by both encoders. Fine-tuning is defined as optimization of a regularized loss of multi-task learning.

Other high performing Transformer-based models are the OpenAI GPT-2 and GPT3 models (Radford et al. 2019) which deliver high performance on several benchmarks in a zero-shot setting. Finally, the ELECTRA model (Clark et al. 2020) is trained using the “replaced token detection” procedure, which corrupts the input by replacing some tokens with plausible alternatives sampled from a small generator network. A discriminative model is then trained that predicts whether a token in the corrupted input was replaced by a generator sample or not, instead of predicting masked tokens as in MLM.

New datasets aimed at evaluating the capability of contextual models to capture in-context similarity and the meaning of individual word instances have been created. Such datasets existed since the era of DSMs but their coverage was limited. The Usage Similarity (Usim) dataset (Erk, McCarthy, and Gaylord 2009, 2013), for example, contains ten instances of 56 target words manually annotated with graded pairwise usage similarity judgments on a scale from 1 to 5 (from less to more similar). The Stanford Contextual Word Similarity (SCWS) dataset (Huang et al. 2012) includes pairs of sentences that contain instances of different target words, or of homographs with different part of speech (e.g., pack as noun and verb).¹⁹ The Concepts in Context (CoInCo) corpus (Kremer et al. 2014) contains substitute annotations for all content words in a sentence. The similarity of word instances is modeled through the overlap of their substitutes, similar to the SemEval-2007 lexical substitution dataset (McCarthy and Navigli 2007).²⁰ Datasets with automatically assigned substitute annotations have also been created. The ukWaC-subs dataset (Garí Soler and Apidianaki 2020b), for example, contains sentences automatically annotated with lexical substitutes from the Paraphrase Database (PPDB) (Ganitkevitch, Van Durme, and Callison-Burch 2013; Pavlick et al. 2015) using the context2vec model (Melamud, Goldberger, and Dagan 2016).

Word-in-Context (WiC) (Pilehvar and Camacho-Collados 2019) is another automatically created dataset which contains binary similarity judgments for pairs of word instances in context.²¹ Sentences are labeled as true (T) or false (F) based on whether they are listed under the same sense in WordNet. Automatic pruning was applied in order to remove related instances with subtle sense distinctions, avoid replicating the fine sense granularity of WordNet and reduce errors, but some noisy annotations still remain (Garí Soler, Apidianaki, and Allauzen 2019). An analysis of the target word and context biases in this and other in-context similarity datasets has been performed by Liu, McCarthy, and Korhonen (2022). A smaller and more focused dataset for studying regular (or systematic) polysemes (Apresjan 1974), the Contextualized Polyseme Word Sense Dataset, has been proposed by Haber and Poesio (2020, 2021). The dataset covers ten types of regular metonymic polyseme alternations (e.g., animal/meat lamb: chicken, pheasant; food/event: lunch, dinner).²² It contains three measures of word sense similarity including graded similarity judgments for the word instance pairs used, co-predication acceptability judgments,²³ and categorical word class judgments.

Benchmarks addressing in-context similarity in a multilingual setting also exist. The XL-WiC dataset addresses twelve languages (Raganato et al. 2020).²⁴ It was automatically created by leveraging information from Multilingual WordNet (Bond and Paik 2012) and Wiktionary. In XL-WiC, positive (True) examples correspond to the same sense in the underlying resource, and negative (False) examples correspond to different senses. The MCL-WiC dataset (Martelli et al. 2021) was manually annotated using lexemes from the multilingual BabelNet network (Navigli and Ponzetto 2010) and addresses five languages.²⁵ MCL-WiC enables a multilingual and a cross-lingual evaluation scenario. The cross-lingual AM2iCo dataset addresses 14 language pairs where English is paired with a target language (Liu et al. 2021).²⁶ AM2iCo was created using Wikipedia’s cross-lingual links which served to identify cross-lingual concept correspondences. A sample of examples for each language pair was then validated through crowdsourcing.²⁷

The representations that have been presented in this section describe the meaning of words at the level of word types, senses, and individual instances. The advantages of each representation method have been discussed as well as their shortcomings, which often incite the development of new approaches. The majority of the presented methods derive meaning representations from text data in an unsupervised or self-supervised way. This data-driven knowledge is refined or occasionally combined with sense information from external lexicons.

Another type of methods aims at improving the information that is learned from corpora by injecting different types of external knowledge in the representations during pre-training or fine-tuning. These semantic specialization methods integrate external knowledge in the form of linguistic constraints, and improve the quality of word representations to better reflect word meaning compared to vanilla word vectors. This makes them highly relevant for this survey. We devote the next section to these knowledge integration methods.

Semantic specialization methods infuse knowledge about different types of lexical relationships into word embeddings. The motivation behind these methods is that the rich information that is present in knowledge graphs and other handcrafted resources can complement the incomplete, and sometimes ambiguous, information that is extracted from texts (Xu et al. 2014). The linguistic and factual information used is often difficult to capture with conventional distributional training. Semantic specialization methods can also serve to adapt generic embedding representations to a specific task, by feeding into them information from resources constructed for that task (Yu and Dredze 2014). We can distinguish semantic specialization methods across three axes:

• (i) 

  the type of embeddings they modify: static or contextualized;

• (ii) 

  the stage in which they intervene: during training or post-processing;

• (iii) 

  whether they address knowledge about individual words (“intra-word” approaches) or about the relationships between words (“inter-word” approaches);

• (iv) 

  the type of knowledge (e.g., linguistic, factual) used for vector enrichment.

Joint models integrate external linguistic constraints into the training procedure of word embedding algorithms. The RC-NET framework of Xu et al. (2014) leverages relational and categorical knowledge extracted from the Freebase graph (Bollacker et al. 2008). This knowledge is transformed into two separate regularization functions that are combined with the original objective function of Skip-gram (Mikolov et al. 2013b). The combined optimization problem is solved using back propagation neural networks, and the generated word representations encode the knowledge found in the graph. The Relation Constrained Model (RCM) of Yu and Dredze (2014) uses a learning objective that incorporates the CBOW objective and linguistic constraints from WordNet and the Paraphrase Database. The combined objective maximizes the language model probability and encourages embeddings to encode semantic relations present in the resources.

Some specialization methods address both synonymy and antonymy. This distinction is hard to make for word embedding models that have been trained using co-occurrence information, given that antonyms (e.g., east/west, happy/sad, interesting/boring) appear in near-identical contexts and get highly similar distributional vectors. Pham, Lazaridou, and Baroni (2015) propose an extension of the Skip-gram method in their multi-task Lexical Contrast Model which optimizes embedding vectors on the joint tasks of predicting corpus contexts and making the representations of WordNet synonyms closer than that of WordNet antonyms. Similarly, Ono, Miwa, and Sasaki (2015) combine information from WordNet and the Roget’s thesaurus (Kipfer 2009) with distributional data to train embeddings specialized for capturing antonymy, by modifying the objective function of Skip-gram. Nguyen, Schulte im Walde, and Vu (2016) also integrate lexical contrast information in the objective function of the Skip-gram model in order to capture antonymy. Contrary to the above described methods which apply lexical contrast information from WordNet to each of the target words, the model of Nguyen, Schulte im Walde, and Vu (2016) applies lexical contrast to every single context of a target word in order to better capture this semantic relationship. The method of Schwartz, Reichart, and Rappoport (2015) uses automatically discovered symmetric patterns indicative of antonymy such as “from X to Y” and “either X or Y” (e.g., “from bottom to top”, “either high or low”) to assign dissimilar vector representations to antonyms.

Embeddings augmented by joint models generally perform better than vanilla Skip-gram embeddings on intrinsic semantic tasks such as word similarity and analogy, and in downstream settings. A limitation of joint models is that they are tied to the distributional objective of the embedding model. As a result, any change to the underlying distributional model induces a change to the joint model too.

Retrofitting methods tune word vector spaces post-hoc according to external linguistic constraints (Faruqui et al. 2015; Mrkšić et al. 2016; Vulić and Mrkšić 2018; Vulić 2018). An advantage of retrofitting methods compared to joint models—which inject information during training—is that they are more versatile and applicable to any input space (Glavaš and Vulić 2018). Some approaches, however, combine joint learning and retrofitting techniques in order to specialize embeddings for semantic constraints (Kiela, Hill, and Clark 2015).

Early retrofitting methods (Faruqui et al. 2015; Jauhar, Dyer, and Hovy 2015; Wieting et al. 2015; Rothe and Schütze 2017) used synonymy constraints to bring the vectors of semantically similar words closer in the vector space. Specifically, the methods encouraged words that were linked in a semantic resource (e.g., paraphrases, synonyms, or hypernyms) to have similar vector representations. Let V = {w₁, w₂, …, w_N} be the vocabulary or set of word types, {$w1→$⁠, $w2→$⁠, …, $wN→$} the corresponding vectors in the original vector space, and D a resource (lexicon, ontology, or dictionary) which encodes semantic relationships between words in V. The goal of retrofitting methods is to produce new word vectors {$w1′→$⁠, $w2′→$⁠, …, $wN′→$} that observe the constraints present in D. These are often sets of synonymous words (w_i, w_j), or they can describe other relationships such as antonymy and entailment. In the case of synonymous word pairs, the goal is to bring the vectors of these words closer together in the vector space. A way to achieve this is to reduce their cosine distance (d(⁠$wi→$⁠, $wj→$⁠) = 1 − cos(⁠$wi→$⁠, $wj→$⁠)) (Mrkšić et al. 2016).

For methods that leverage antonymy relations, the goal is to push the vectors of antonymous words away from each other in the transformed vector space V′ (i.e., increase their cosine distance), similar to the objective in the Lexical Contrast model of Pham, Lazaridou, and Baroni (2015). Using both similarity and dissimilarity constraints further improves performance, compared to methods that only inject similarity constraints in the vector space built by language models. The counter-fitting approach of Mrkšić et al. (2016) and the ATTRACT-REPEL algorithm (Mrkšić et al. 2017) use both synonymy and antonymy constraints. ATTRACT-REPEL makes it possible to embed vector spaces of multiple languages into a single vector space using constraints from the multilingual BabelNet network (Navigli and Ponzetto 2010). This allows to exploit information from high-resource languages in order to improve the word representations in lower-resource ones.

The above described retrofitting and counter-fitting methods address symmetric relationships (synonymy and antonymy). The LEAR (Lexical Entailment Attract-Repel) and GLEN (Generalized Lexical ENtailment) models (Vulić and Mrkšić 2018; Glavaš and Vulić 2019) address the asymmetric lexical entailment relationship. This relationship was considered by joint models that used hypernymy constraints for learning hierarchical embeddings (Yu et al. 2015; Nguyen et al. 2017), instead of injecting knowledge to modify an existing vector space. Contrary to similarity-focused specialization models that tune only the direction of distributional vectors (Mrkšić et al. 2017; Glavaš and Vulić 2018; Ponti et al. 2018), a specialization procedure that addresses hierarchical relationships between concepts needs to also rescale the vector norms. Two words can be differentiated by means of their Euclidean norms so that the norm of the hypernym is larger than that of the hyponym (Nguyen et al. 2017). By injecting external linguistic constraints (‘IS-A’ WordNet links) into the original vector space, LEAR brings true hyponymy-hypernymy pairs closer together in the transformed Euclidean space, and adjusts their norms to reflect the actual hierarchy of concepts in WordNet. At the same time, a joint objective enforces semantic similarity using the symmetric cosine distance, yielding a vector space specialized for both lexical relations at once.

A limitation of early retrofitting methods is that they specialize only the vectors of words seen in the constraints, leaving unchanged the vectors of unobserved words. More recent methods address this issue by specializing the full vocabulary of the original embedding space. The adversarial post-specialization method of Ponti et al. (2018), for example, propagates the external lexical knowledge to the full distributional space. Words seen in the resources serve as training examples for learning a global specialization function, by combining a distance loss with an adversarial loss. The Explicit Retrofitting (ER) models of Glavaš and Vulić (2018, 2019) also learn a global specialization function which can specialize the vectors of words unseen in the training data. The model proposed in the earlier study accounts for symmetric relations (synonymy and antonymy), while the more recent one (GLEN) addresses asymmetric relations (entailment). An important advantage of these models is that they can be extended to specialize vector spaces of languages unseen in the training data, by coupling ER with shared multilingual embedding spaces. Word vectors of a new language can be projected to the space of the language for which the specialization function has been learned using some cross-lingual projection method (e.g., Artetxe, Labaka, and Agirre 2018; Lample et al. 2018).

Contextual language models (e.g., ELMo, BERT, GPT-2, GPT-3) are trained on large amounts of raw text using self-supervision. Consequently, although more powerful than static embeddings, they also encode only the distributional knowledge that is available in the corpora used for training. Without explicit grounding to real-world entities, they have difficulty recovering factual knowledge (Peters et al. 2019; Logan et al. 2019). Numerous knowledge injection methods aimed at augmenting the knowledge that is encoded in contextualized vectors have been proposed.

Similar to joint optimization methods for static embeddings, knowledge about semantic relationships can be integrated during contextual language models’ pre-training. This can be done by adding an auxiliary word relationship classification task, as in the Lexically-Informed BERT (LIBERT) model of Lauscher et al. (2020). In LIBERT, knowledge about semantic similarity (i.e., synonymy and hypernymy) is infused into BERT vectors in a multi-task learning setting, where the MLM and NSP objectives are coupled with an auxiliary binary word relation classification task.

The SenseBERT model (Levine et al. 2020) injects information about senses into contextualized representations using an auxiliary masked word sense prediction task, alongside BERT’s usual training tasks (MLM and NSP). The language model that predicts the missing words’ sense is trained jointly with the standard word form-level language model, without need for sense-annotated data. Specifically, information from WordNet serves as weak supervision for self-supervised learning; the masked word’s supersenses form a set of possible labels for the sense prediction task.³⁰

The joint learning of words and knowledge from pre-crafted resources has also been attempted in other studies (Wang et al. 2014; Toutanova et al. 2015; Han, Liu, and Sun 2016; Cao et al. 2017; Zhang et al. 2019). These mainly address entities and relationships found in large knowledge graphs such as the Freebase (Bollacker et al. 2008), or in in-domain specific resources (Liu et al. 2020). The proposed models generally embed words, entities, and their relationships in the same continuous latent space. Operating simultaneously on relations observed in text and in pre-existing structured databases permits to reason about structured and unstructured data in mutually-supporting ways (Riedel et al. 2013). Such knowledge-enhanced language models are created by integrating fixed entity embeddings into pre-trained language representation models, or by jointly optimizing a knowledge embedding objective and a language modeling objective during pre-training.

In KnowBert (Peters et al. 2019), an integrated entity linker is used to retrieve relevant entity embeddings, and contextual word representations are updated via a form of word-to-entity attention. In this model, the entity linker and self-supervised language modeling objective are jointly trained end-to-end in a multi-task setting. The KEPLER (Knowledge Embedding and Pre-trained LanguagERepresentation) model of Wang et al. (2021) encodes entity descriptions as entity embeddings, and jointly optimizes a knowledge embedding objective and a masked language modeling objective during pre-training by means of a multi-task loss. ERNIE (Enhanced Language RepresentatioN with Informative Entities) (Zhang et al. 2019) uses a pre-training task where token-entity alignments are masked and the system is required to predict both. The reported experimental results show that jointly embedding this information improves performance in downstream tasks involving fact and entity relation prediction. Yamada et al. (2020) propose the LUKE (Language Understanding with Knowledge-based Embeddings) model, which is trained on a large entity-annotated corpus retrieved from Wikipedia, and integrates an “entity-aware” self-attention mechanism which considers the types of tokens (words or entities) when computing attention scores. LUKE achieves impressive empirical performance on a wide range of entity-related tasks.

The above-mentioned approaches use hand-crafted knowledge resources. The “Symbolic knowledge distillation” paradigm is a conceptual framework where large general language models (e.g., GPT-3) author commonsense knowledge graphs to train commonsense models (Hwang et al. 2021; West et al. 2022). The approach is motivated by knowledge distillation (Hinton, Vinyals, and Dean 2014) where a larger teacher model transfers knowledge to a compact student model.

These knowledge enrichment methods mainly address factual knowledge about entities and relations, not lexical semantic relationships (such as synonymy, hyponymy, or entailment). However, our proposed overview addresses approaches that enrich embeddings with lexico-semantic knowledge, with the aim to improve the representations of words and their relationships. This section will thus not go deeper into this analysis. The interested reader may consult the references given above for a more thorough presentation of the proposed methods.

These methods retrofit external semantic knowledge into contextualized embeddings through a process similar to fine-tuning. Shi et al. (2019) tune ELMo embeddings (Peters et al. 2018) using the Microsoft Research Paraphrase Corpus (MRPC) (Dolan, Quirk, and Brockett 2004). They propose an orthogonal transformation for ELMo that is trained to bring representations of word instances closer when they appear in meaning-equivalent contexts (paraphrases). The idea is the same as the one underlying retrofitting methods. Arase and Tsujii (2019) use paraphrase data for fine-tuning BERT, and then fine-tune the model again for the tasks of interest (paraphrase identification and semantic equivalence assessment). The model that is first exposed to paraphrase data is better at performing these tasks, compared to a model directly fine-tuned on task data. Garí Soler and Apidianaki (2020b) also show that BERT fine-tuned on usage similarity and paraphrasing datasets (Erk, McCarthy, and Gaylord 2013; Kremer et al. 2014; Creutz 2018) performs better on the Graded Word Similarity in Context (GWSC) task (Armendariz et al. 2020).

The LexFit model of Vulić et al. (2021) relies on dual-encoder network structures in order to extract the lexical knowledge that is stored in pre-trained encoders, and turn language models into static decontextualized word encoders. The procedure involves fine-tuning pre-trained language models on lexical pairs from an external resource. These comprise positive pairs where a particular relationship holds between two words, and negative pairs, where the first word is paired with a random word with which it is not related.

The knowledge enrichment methods described in this section aim at improving the quality of the lexical semantic knowledge that is encoded in language model representations. All studies report improved results in intrinsic and extrinsic evaluations compared to vanilla models with no access to external knowledge. Naturally, these methods rely on the assumption that some lexical semantic knowledge is already encoded in language model representations and can be refined. The next section presents methods that attempt to decipher the encoded knowledge which is generally acquired through exposure to large volumes of data during language model pre-training.

Interpretation studies demonstrate that language model representations encode rich knowledge about language and the world (Voita, Sennrich, and Titov 2019; Clark et al. 2019; Voita et al. 2019; Tenney, Das, and Pavlick 2019; Linzen 2019; Rogers, Kovaleva, and Rumshisky 2020). The intense exploration of language models’ inherent knowledge has been motivated by their impressive performance in NLU tasks. Interpretation studies attempt to understand what this high performance is due to, by deciphering the knowledge that is encoded inside the representations. The bulk of this interpretation work relies on probing tasks which serve to predict linguistic properties from the representations that are generated by vanilla models, before integration of any external knowledge. Success in these tasks indicates that the model’s representations encode the addressed linguistic knowledge.

Early probing studies explored surface linguistic phenomena pertaining to grammar and syntax, which are directly accessible in contextualized (token-level) representations (Linzen, Dupoux, and Goldberg 2016; Hewitt and Manning 2019; Hewitt and Liang 2019). The first studies addressing semantic knowledge explored phenomena in the syntax-semantics interface such as semantic role labeling and coreference (Tenney, Das, and Pavlick 2019; Kovaleva et al. 2019), and the symbolic reasoning potential of LM representations (Talmor et al. 2020). Lexical polysemy is more challenging to study using token-level representations since it is encoded at a higher level of abstraction than individual instances, that of word types. Representations extracted from pools of sentences allow to abstract away from individual context variation. They are more informative about words’ semantic properties (Vulić et al. 2020b), and can serve to model abstract semantic notions (e.g., intensity) (Garí Soler and Apidianaki 2020a, 2021b). Semantic relationships like hypernymy and entailment are also usually encoded at the word type-level (e.g., cat ⊧ animal, tulip ⊧ flower), although they are context-dependent in the case of polysemous words.³¹

We hereby present methods that have been proposed for exploring the semantic information that is encoded in contextualized representations. These include visualization and probing tasks, as well as studies that investigate the semantic properties of the words in the constructed space by relying on its geometry.

The capability of language models to represent polysemy was initially studied by visualizing sense distinctions found in lexicons. Reif et al. (2019) and Wiedemann et al. (2019) generate BERT representations from Wikipedia sentences and the SemCor corpus (Miller et al. 1993). They show that the representations of polysemous words’ usages are organized in the semantic space in a way that reflects the meaning distinctions present in the data. They also demonstrate BERT’s WSD capabilities when leveraging sense-related information from these resources. Similarly, in a more recent study, Loureiro et al. (2021) show that BERT representations are precise enough to allow for effective disambiguation. Illustrating the semantic distinctions made by the BERT model in a visualization experiment, they show that it is able to group instances of polysemous words according to sense. The dataset used in these experiments is CoarseWSD-20, a benchmark that targets the ambiguity of 20 nouns and illustrates easily interpretable sense distinctions.

All these studies rely on sense annotated data and do not directly address the semantic knowledge that is encoded in contextualized representations. This type of investigation can be done using probing.

Prompting methods are widely used for few-shot and zero-shot learning, but have also been quite popular in interpretability studies that explore the linguistic and common sense knowledge which is encoded in pre-trained LMs. This framework does not involve any model training or fine-tuning. Instead, prompting methods use a template that contains some unfilled slots, which serves to modify the original input x into a textual string prompt x′ (Liu et al. 2022). In a sentiment analysis task, for example, the input “I love this movie” may be transformed into “I love this movie. Overall, it was a [Z] movie” using the template “[X] Overall, it was a [Z] movie”. The language model is then used to fill the slot [Z] in prompt x′ with the highest scoring answer $z^$⁠. This answer, or the obtained final string (⁠$x^$⁠), can then be mapped to a class (e.g., positive or negative) in a classification setting.

A correct filler selected by the model can also be seen as an indication of the existence of a specific type of knowledge in the pre-trained LM representations. In the above example, it would be that the model knows “love” is a verb with positive sentiment. Consequently, prompting methods are also commonly used in interpretation studies (Petroni et al. 2019; Bouraoui, Camacho-Collados, and Schockaert 2020; Ravichander et al. 2020; Ettinger 2020; Apidianaki and Garí Soler 2021). The input is again transformed into a prompt that contains a slot to be filled by the model. Cloze-style prompts that contain blanks that need to be filled (e.g., I love this movie, it is a [Z] movie.) are a good fit for exploring masked LMs because they closely match the form of the pre-training task. Prefix prompts, which continue a string prefix (e.g., “I love this movie. What’s the sentiment of the review? [Z]”), can also be used. Naturally, due to their form, prefix prompts are a better fit for generation tasks.

In this section, we present prompting methods that have been used for lexical semantics. For an extensive overview of work on prompt-based methods, we refer the reader to the survey of Liu et al. (2022). Query reformulation methods are also relevant from a semantics perspective. These methods modify the input (using query mining and paraphrasing) in order to produce semantically similar prompts that serve to promote lexical diversity and improve knowledge extraction from the language model (Jiang et al. 2020). These may also involve prompt ensembling methods that combine multiple prompts, as well as end-to-end re-writing models for generating rephrased queries (Haviv, Berant, and Globerson 2021). Lastly, there is evidence that prompt-based models do not understand the meaning of the prompts used (Webson and Pavlick 2022), which opens up new interesting research avenues for semantics.

Cloze task queries (Taylor 1953) are a good fit for BERT-type models which are trained using the MLM objective. These prompts contain a “[MASK]” token at the position that needs to be filled and serve to query the model for different types of knowledge, such as encyclopedic knowledge (e.g., “Dante was born in [MASK]”) (Petroni et al. 2019), relational knowledge (e.g., “Recessions are caused by [MASK]”) (Bouraoui, Camacho-Collados, and Schockaert 2020), hypernymy relationships (e.g., “A car is a [MASK]”) (Ravichander et al. 2020), noun properties (e.g., “Strawberries are [MASK]”) (Apidianaki and Garí Soler 2021), and others.

Cloze task probing is often criticized as an interpretation method because language models are brittle to small changes in the used prompts (Jiang et al. 2020). For example, plural queries (Strawberries are [MASK]) are more efficient than singular queries (A strawberry is [MASK]) in retrieving noun properties (Apidianaki and Garí Soler 2021). The naturalness of the query is also important. There are higher chances that the model has seen natural statements in the training data and can thus handle them better than unnatural queries (Ettinger 2020). Concerns are also expressed regarding the systematicity of the knowledge that is identified using probes. Ravichander et al. (2020) showed that affirmative factual or relationship knowledge extracted from BERT does not systematically generalize to novel items. Consequently, BERT’s capabilities as discovered through probes may not correspond to some systematic general ability.

Importantly, some types of information are difficult to retrieve using cloze tasks due to the “reporting bias” phenomenon which poses challenges to knowledge extraction (Gordon and Van Durme 2013; Shwartz and Choi 2020). According to this phenomenon, the frequency with which people write about actions and properties is not necessarily a reflection of real-world frequencies, or of the degree to which a property is characteristic of a class of individuals. Hence, exceptional actions or properties (e.g., A person was killed) are over-represented in texts (for example, in newspaper articles) and amplified by the models that are trained on these data, at the expense of more common or trivial ones which are obvious to the participants in the communication (e.g., A person is breathing). As a consequence, low results in a probing experiment might suggest either that the tested model has encoded marginal knowledge about the studied phenomenon, or that it has not seen the relevant information during training. A model might encode lexical and encyclopedic knowledge which is available in Wikipedia texts used for training (e.g., a banana is a fruit, Obama was President of the United States), and miss other types of perceptual knowledge (e.g., bananas are yellow, silk is soft). Finally, another issue with semantic cloze task evaluations is that—contrary to queries addressing structural properties (e.g., number agreement or syntactic dependencies)—there might be multiple correct answers for a query. These are only partially covered by the resources used for evaluation which are often developed in (psycho-)linguistic studies following different annotation protocols (McRae et al. 2005; Devereux et al. 2014). Misra, Rayz, and Ettinger (2022) highlight the risk of taking the absence of evidence in the resource used for evaluation as evidence, and stress the need for more comprehensive evaluation datasets.³²

In the work of Aina, Gulordava, and Boleda (2019), probing serves to explore the word type information that is present in the representations of a biLSTM language model, and how this interacts with contextual (token-level) information in the hidden representations of the model. They specifically train diagnostic classifiers on the tasks of retrieving the input embedding for a word (Adi et al. 2017; Conneau et al. 2018) and a representation of its contextual meaning, as reflected in its in-context lexical substitutes. The results show that the information about the input word is not lost after contextualization.

More recent probing methods for lexical semantics rely on word type-level embeddings that are derived from contextualized representations using vector aggregation techniques (Vulić et al. 2020b; Bommasani, Davis, and Cardie 2020; Garí Soler and Apidianaki 2021a).³³ Using this type of word type-level embeddings has become standard practice in studies that address the models’ knowledge about lexical meaning. Reasons for this are the strong impact of context variation on the quality of the representations and the similarity estimates that can be derived from them (Ethayarajh 2019a; Mickus et al. 2020). This situation is problematic given that vector similarity calculations are key in lexical semantic tasks. In the next paragraph, we explain that the similarity estimates which are drawn from the semantic space of contextual models are not that reliable for reasons mainly related to the geometry of the vector space.

The results reported by Ethayarajh (2019a) highlight the high anisotropy of the space that is constructed by contextual language models, which has a strong negative impact on the quality of the similarity estimates that can be drawn from it. Anisotropic word representations occupy a narrow cone in the vector space instead of being uniformly distributed with respect to direction (Gao et al. 2019; Cai et al. 2021; Rajaee and Pilehvar 2021). In this anisotropic cone, even unrelated words have excessively positive correlations (Ethayarajh 2019a; Rajaee and Pilehvar 2021). This is illustrated in Figure 7. A highly isotropic space is shown on the left, where the similarity of instances of a monosemous word (the noun dog) is high compared to other words represented in the space. The anisotropic space on the right suggests the opposite; since all word instances are found in a narrow cone, any two word instances have high cosine similarity. As a result, similarity estimates are distorted because of the geometry of the space, so the high similarity of dog is not important.³⁴ For example, instance representations obtained with GPT-2 for randomly sampled words are as close in the space as instances of the same word.

The Self Similarity (SelfSim) metric introduced by Ethayarajh (2019a) for measuring contextualization and exploring the geometry of the embedding space (presented in the previous section) is a useful tool for studying words’ semantic properties. Garí Soler and Apidianaki (2021a) use SelfSim to study lexical polysemy. They form four sentence pools from the SemCor corpus (Miller et al. 1993) that reflect different sense distributions:

• mono groups instances of a monosemous word (e.g., camping, aircraft);

• poly-bal contains a balanced distribution of each sense of a polysemous word;

• poly-same groups instances of a single sense of the polysemous word;

• poly-rand is composed of randomly sampled instances of the word.

The poly-rand pool is expected to be highly biased towards the most frequent sense due to the skewed frequency distribution of word senses (Kilgarriff 2004; McCarthy et al. 2004), and to reflect natural occurrence in texts. The controlled composition of the pools offers the possibility to explore (a) the knowledge about lexical polysemy that is acquired by BERT during pre-training, and (b) the influence of new contexts on token representations. The answer to question (a) is given by comparing the mono and poly-same pools, which contain instances of the same sense. Question (b) is answered by observing whether it is possible to distinguish the pools that contain different sense distributions for polysemous words using contextualized representations.

The four pools are compared by means of the average pairwise SelfSim of the ten instances they each contain. Figure 8 shows the results obtained using monolingual BERT-type (uncased) models in English, French, Spanish, and Greek (Devlin et al. 2019; Le et al. 2020; Cañete et al. 2020; Koutsikakis et al. 2020). The distinctions that are established between the pools are significant in all layers of the models, and show that BERT representations encode an impressive amount of knowledge about polysemy.³⁵ The distinction between mono and poly-same is particularly important. Both pools contain instances of a single sense of the studied words (i.e., there is no meaning-related variation across instances inside each pool). Hence, their distinction across layers shows that information about polysemy is acquired by the models during pre-training. Polysemous words should be encountered in more varied contexts than monosemous words, and this variation must be encoded with BERT’s language modeling objective when the model is exposed to large amounts of data during training.

This prior knowledge about lexical meaning is combined with information from new contexts of use, as shown by the finer-grained distinctions made between different poly pools. As expected, average SelfSim is higher between instances of the same sense (poly-same) than in poly-bal and poly-rand which contain instances of different senses. The poly-rand pool naturally has higher average SelfSim than poly-bal due to the larger representation of the most frequent sense in randomly sampled sentences. The decreasing trend observed in the plots and the peak in layer 11 confirm the phases of context encoding and token reconstruction observed by Voita, Sennrich, and Titov (2019). Other results reported by Garí Soler and Apidianaki (2021a) show that average SelfSim is higher for monosemous words and words with low polysemy than for highly polysemous words, even when controlling for grammatical category and word frequency. Additionally, BERT representations can serve to determine the partitionability of a word’s semantic space into senses (McCarthy, Apidianaki, and Erk 2016).

These findings about BERT’s knowledge of lexical polysemy are consistent with results reported by Pimentel et al. (2020), who investigate the relationship between lexical ambiguity and contextual uncertainty using an information theoretic approach. This approach relies on the assumption that speakers compensate lexical ambiguity by making contexts more informative in order to facilitate disambiguation and communication (Piantadosi, Tily, and Gibson 2011). As a corollary, the ambiguity of a word type should correlate with how much information the context provides about it, and negatively correlate with contextual uncertainty. Furthermore, they expect the contextualized representations of polysemous wordforms to be more spread out and to occupy larger regions of the meaning space built by BERT than the representations of wordforms with fewer senses. With respect to the mBERT model, they also observe that it can serve to estimate lexical ambiguity in English, but the quality of the estimates deteriorates in lower-resource languages. A similar type of investigation is performed by Xypolopoulos, Tixier, and Vazirgiannis (2021), who examine the geometry of the ELMo embedding space (Peters et al. 2018). Using multiresolution grids, the authors observe the volume covered by the cloud of points that correspond to different instances of a word, which they consider to be representative of its polysemy. Specifically, they construct a hierarchical discretization of the space where each level corresponds to a different resolution, and the same number of bins are drawn along each dimension. The polysemy score for a word is based on the volume (i.e., the proportion of bins) covered by its vectors at each level.

Word representations are more dissimilar in upper layers of the models, where contextualization is higher. Voita, Sennrich, and Titov (2019) investigate the evolution of token representations in Transformers trained with different training objectives (LM, MLM, and a Machine Translation objective) using an information-theoretic approach. They explore the flow of information inside the Transformer by specifically estimating the mutual information between a token representation at a certain layer and the input token. Their results show that, with the MLM objective, information about the input token is initially lost during a “context encoding” phase, but is recovered at the last layer (just before prediction) during a “token reconstruction” phase.

Luo, Kulmizev, and Mao (2021) consider position embeddings responsible for the presence of outliers which they view as a major cause of anisotropy. The outliers are a very small number of dimensions that regularly appear in the same position in the pre-trained encoder layers of a Transformer model (e.g., BERT, RoBERTa, GPT-2, BART, XLNet, ELECTRA). As shown by Kovaleva et al. (2021), disabling these outliers dramatically disrupts model performance. It degrades the quality of the language model as reflected in the MLM loss and in the quality of masked predictions. Luo, Kulmizev, and Mao (2021) train RoBERTa–base models from scratch without using position embeddings, and show that the outliers disappear.

Isotropy is a desirable property of word embedding spaces (Huang et al. 2018; Cogswell et al. 2016). As explained in Section 4.4.1, from a geometric point of view, a space is called isotropic if the vectors within that space are uniformly distributed in all directions (Rajaee and Pilehvar 2021). Low isotropy affects the expressiveness of the embedding space and the optimization procedure (models’ accuracy and convergence time).

Methods for reducing the anisotropy of the vector space and improving the quality of the obtained similarity estimates have been proposed, initially addressing static embedding representations. Mu and Viswanath (2018) showed that the static word vectors generated by algorithms such as GloVe, Skip-gram, and CBOW, share a large common vector and the same dominating directions. After removing the common mean vector during a post-processing operation, the representations become more “isotropic”, that is, more distinct and uniformly distributed within the vector space. By eliminating the top principal components of all words, word representations become stronger and deliver improved performance in intrinsic and downstream tasks.³⁸ Raunak, Gupta, and Metze (2019) also apply this procedure as the first step in their vector dimensionality reduction method. The resulting embeddings are lower dimension and perform on par or better than the original embeddings in similarity and classification tasks.

In more recent work, Bihani and Rayz (2021) also propose a method that renders off-the-shelf representations isotropic and semantically more meaningful, called “Low Anisotropy Sense Retrofitting” (LASeR). Their method also resolves the representation degeneration problem at a post-processing stage, and conducts sense-enrichment of contextualized representations. Rajaee and Pilehvar (2021) combine the method of Mu and Viswanath (2018) with a cluster-based approach, in order to mitigate the degeneration issue that occurs in contextual embedding spaces and increase their isotropy. Their approach relies on the observation that the embedding space of contextual models is extremely anisotropic in all non-input layers from a global sight, but significantly more isotropic from a local point of view (i.e., when embeddings are clustered). Isotropy is measured in their experiments using the method of Arora et al. (2016). They apply PCA to embedding clusters in order to find the principal components (PCs) that indicate the dominant directions for each specific cluster. Dominant directions in clusters of verb representations, for example, encode tense information. As a result, representations for different instances of a verb with the same tense and different meaning are closer to each other in the space than representations for instances with the same meaning and different tense. Removing these directions increases the isotropy of the BERT and RoBERTa spaces, makes them more suitable for semantic applications, and improves performance on semantic tasks.

Rajaee and Pilehvar (2022) show that the spaces that are created by mBERT in different languages are also massively anisotropic.³⁹ However, unlike monolingual models, there is no dominant dimension that has high contribution to the anisotropic distribution.⁴⁰ This contradicts the finding of Luo, Kulmizev, and Mao (2021) about the role of position embeddings in the emergence of outliers, since monolingual and multilingual spaces are constructed using the same training procedure which involves position embeddings. Methods aimed at increasing the isotropy of a multilingual space could, however, significantly improve its representation power and performance.

Most often, representation similarity is estimated using the cosine or the Euclidean distance. Interestingly, Timkey and van Schijndel (2021) call into question the informativity of these measures for contextual language models. They show that these measures are often dominated by 1–5 “rogue dimensions” in GPT-2, BERT, RoBERTa, and XLNET, regardless of the pre-training objective. They explain that it is this small subset of dimensions that drives anisotropy, low self-similarity, and the drop in representational quality in later layers of the models. Their presence can cause cosine similarity and Euclidean distance to rely on less than 1% of the embedding space. Finally, they show that there is a striking mismatch between these dimensions and those that are important to the behavior of the model.

In the “intra-word” approaches described in the previous section, the context varies across sentences but the target word remains the same. The polysemy pools in the study of Garí Soler and Apidianaki (2021a), for example, contain ten instances of each target word in different contexts. Additionally, the SelfSim score for a word corresponds to the average cosine similarity between its contextualized representations across a number of contexts (Ethayarajh 2019a). The exploration of “inter-word” relationships using contextualized representations is more challenging because the target words are also different. A way to control for the imprint of context on the representations would be to substitute the words to be compared in the same sentence (Garí Soler and Apidianaki 2020a, 2021b). Naturally, this process can only be applied to words whose substitution leads to natural sentences. This is often the case with synonyms (e.g., “I am [glad, happy, excited, delighted] to let you know that your paper will be published”), but can also be the case with hypernyms (e.g., “I like begonias/flowers]”).⁴¹ By keeping the context stable, the comparison of the contextualized vectors of the words can reveal their similarities and differences. Cloze task probing has also been used for exploring this type of lexical relationships (Ravichander et al. 2020).

An alternative for probing contextualized representations for inter-word relationships is to create some sort of word type-level embedding by aggregating over the representations of individual instances (Vulić et al. 2020b; Bommasani, Davis, and Cardie 2020). This transformation reduces the strong imprint of context variation on the obtained vectors and improves the quality of similarity estimates. Using this type of transformation, Vulić et al. conduct extensive experiments and show that monolingual pre-trained LMs store rich type-level lexical knowledge. Other alternatives include feeding the word without context, or using the embedding before contextualization. Methods that derive word-level representations have, thus, started gaining popularity since they provide a more solid basis for meaning exploration. A detailed account of such methods is given in Section 5.

In this section, we presented methods that have been proposed for probing contextual language model representations for lexical semantic knowledge. We explained how visualization has been used to this purpose, and we presented prompting methods for semantics and other interpretation methods that serve to analyze the geometry of the vector space. We dedicated a section to the challenges for semantic exploration that are posed by the space itself mainly due to its high anisotropy, and discussed solutions that have been proposed for improving the quality of the similarity estimates that can be drawn from it.

The next section provides an overview of methods aimed at deriving static embeddings from contextualized ones, and at combining the two types of vectors in order to leverage their respective strengths and address their limitations.

Contextualized representations pose challenges to the investigation of lexical semantic knowledge that is situated at the level of word types (cf. Section 4.5). Additionally, static word embeddings are more interpretable than their contextualized counterparts, and the knowledge they encode is easier to analyze due to the wide availability of evaluation datasets. Word type-level embeddings present several advantages over token-level ones in application settings as well, specifically in terms of speed, computational resources, and ease of use (Bommasani, Davis, and Cardie 2020).

In general, continuous real-valued representations give rise to a large memory footprint and slow retrieval speed. This hinders their applicability to low-resource (memory and computation-wise) platforms, such as mobile devices (Shen et al. 2019; Gupta and Jaggi 2021). Tools for utilizing and processing static embeddings efficiently and in a more lightweight fashion have also been proposed (Patel et al. 2018). However, the situation becomes even more complicated with contextualized embeddings. Gupta and Jaggi (2021) report that even when ignoring the training phase, the computational cost of using static word embeddings is typically tens of millions times lower than using contextual embedding models. Furthermore, Strubell, Ganesh, and McCallum (2019) characteristically highlight the environmental cost of contextualized representations; training a BERT model on GPU is roughly equivalent to a trans-American flight in terms of carbon emissions.

Given the above observations and the need to integrate embeddings in low-resource settings and devices, word type-level representations are progressively being brought back to the foreground of representation learning. They are also becoming popular in probing studies, since they serve to study the encoded lexical semantic knowledge (Vulić et al. 2020b; Bommasani, Davis, and Cardie 2020). However, this recent interest does not involve the use of “traditional” static embeddings (i.e., word2vec, fastText, or GloVe), but a new type of word type-level embeddings that are derived from contextual language models. The possibility to derive a lexicon at a higher level of abstraction than individual word instances is a valuable tool for linguistic and semantic analysis. Different strategies have been proposed for performing this operation.

A simple way to generate a decontextualized vector for a word w is to feed it “in isolation” (i.e., without any context) into the pre-trained language model, and to use the output vector as its representation. If the word is split into multiple pieces (w¹, …, w^k), these can be concatenated to form a representation for the word (cat(w¹, …, w^k) = w) (Bommasani, Davis, and Cardie 2020; Vulić et al. 2020b). Vulić et al. (2021) prepend and append the special tokens [CLS] and [SEP] to the word or subword sequence before they pass it through BERT.

The decontextualized approach is simple but it presents an unnatural input to the pre-trained encoder which is not trained on out-of-context words. Furthermore, the approach is not very efficient compared with methods that perform pooling over different contexts, described hereafter. Recent work, however, shows that this approach can serve to turn contextual language models into effective decontextualized (static) word encoders through contrastive learning techniques (Vulić et al. 2021). Static type-level vectors can be extracted from any Transformer-based LM, directly from the representations that are generated by the model or after fine-tuning using linguistic constraints from an external resource.

Another simple strategy for extracting context-agnostic representations is to use BERT’s embedding layer (L₀) before contextualization (Vulić et al. 2020b; Conneau et al. 2020b). Contrary to the decontextualized approach described in Section 5.1.1—where the word is fed to the model in isolation—in this case, the input is a contextualized word instance but its representation is extracted from the embedding layer (L₀) before the context influences the representation. Word vector parameters from this layer can be directly used as static embeddings. However, this method produces suboptimal representations when tested in a wide range of setups and languages (Vulić et al. 2020b; Wang, Cui, and Zhang 2020).

Aggregating representations across multiple contexts is the most common approach for creating word type-level embeddings from contextualized representations. In this case, the encoder receives more natural input than with the decontextualized approach. Context pooling for a word w basically involves sampling a number n of sentences that contain instances of w, as shown in Figure 9. Vectors are computed for each of these contextualized instances (w_c1, …, w_cn) and a pooling strategy (f ∈ {min, max, mean}) is applied to yield a single representation for the word (w = f(w_c1, …, w_cn)) (Bommasani, Davis, and Cardie 2020). The number of contexts n to be pooled is a parameter that needs to be set. Interestingly, Vulić et al. (2020b) observe marginal albeit consistent performance gains in all their evaluation tasks when using a larger over a smaller number of contexts (100 vs. 10).⁴² This suggests that a few sentences are sufficient for building a good quality word type-level representation. Figure 10 illustrates the results obtained by different static vector derivation approaches and the BERT (mono) and Multilingual BERT (mbert) models on the English lexical semantic similarity task of the Multi-SimLex benchmark (Vulić et al. 2020a).⁴³ The representations that are built from 10 contexts capture similarity better on this task than those derived from 100 contexts, and they both perform better than mbert and fastText embeddings. The results for words encoded in isolation are shown with the [mono ∣ mbert].iso.[nospec ∣ withcls ∣ all] lines in Figure 10. Interestingly, the representations built using words in isolation are only marginally outscored by their counterparts which aggregate representations across different contexts in several tasks.

Representation pooling has also been applied to ELMo embeddings (Peters et al. 2018). Schuster et al. (2019) showed that “static anchors” created by pooling over ELMo embeddings can serve for aligning cross-lingual spaces. This is not straightforward in the case of token-level embeddings because context variation makes the alignment of cross-lingual spaces difficult. Operating at the anchor level compresses the space and makes possible the use of a dictionary for cross-lingual space alignment.

Methods that combine static and contextualized representations aim to build on their respective strengths and address their limitations. Similar to knowledge injection methods (cf. Section 3), such combination methods can be applied during vector training or at a post-processing stage. The combined vectors are high quality and tend to perform better than static (word2vec, GloVe, fastText) embeddings and contextualized representations. They thus indicate a viable option for replacing compute-heavy contextual embedding models in application settings where resources are limited. Combination is directional with contextualized representations being used to enhance the quality of static embeddings, and vice versa.

These methods integrate contextualized representations into the training process of static embeddings. Wang, Cui, and Zhang (2020) precisely use words’ BERT representations for training a Skip-gram model. Contextualized vectors serve, in this case, to resolve ambiguities, and to inject rich syntactic and semantic information in the generated vectors.

Gupta and Jaggi (2021) also propose a distillation method (called X2Static) that generates word type-level representations from contextualized vectors by extending CBOW training. Their method relies on the Sent2vec embedding method (Pagliardini, Gupta, and Jaggi 2018) which uses the entire sentence to predict the masked word instead of a fixed-size context window (used by CBOW), and allows to learn higher n-gram representations. Contrary to Sent2vec, which encodes context as the sum of static vectors, in X2Static context is represented as the average of all embeddings returned by BERT. This more refined context representation accounts for word order and interaction.

Static embeddings have also been used for enhancing contextualized representations for lexical semantics. The method of Liu, McCarthy, and Korhonen (2020) learns a transformation of contextualized vectors through static anchors. The anchors might be word type-level embeddings (such as word2vec Skip-gram, GloVe, and fastText), or the average of the contextualized representations for a word across instances (similar to the representation pooling methods described in Section 5.1.3).⁴⁴

The contextualized embeddings are represented as a source matrix and the static model representations as the target matrix. In order to combine them, an orthogonal alignment matrix is found which rotates the target space to the source space by solving the least squares linear regression problem. A linear mapping is learned that serves to transform the source space towards the average of source and the rotated target space. Three contextual models are used in the experiments, BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019), and XL-Net (Yang et al. 2019). The quality of the transformed vectors is evaluated on three “Within Word” tasks that address the similarity of same-word instances: Usage Similarity (Usim) (Erk, McCarthy, and Gaylord 2013), Word-in-Context (WiC) (Pilehvar and Camacho-Collados 2019), and Cosimlex-I (Armendariz et al. 2020); and two “Inter Word” tasks which address the similarity of different words: Stanford Contextual Word Similarity (SCWS) (Huang et al. 2012) and Cosimlex-II (Armendariz et al. 2020). The evaluation results show that the proposed transformation improves the performance of contextual model representations in all tasks, with the largest and most consistent gains coming from aligning them towards static (especially fastText) embeddings. Overall, the transformation results in better within word contextualization by increasing the similarity of same word instances.⁴⁵ It also contributes to a better overall inter-word semantic space, as shown by the improvements in Inter Word tasks. The contextualization power of the original space is preserved and enhanced by the transformation.

Table 1 presents the results reported by Liu, McCarthy, and Korhonen (2020) for the RoBERTa model before and after the adjustment. We show results for the combination of RoBERTa representations with static embeddings (fastText, SGNS, GloVe) and with contextualized embeddings generated by other models (BERT, XLNet), and compare to a fastText embedding baseline. Performance on Usim, CoSimlex-II, and SCWS is measured using Spearman correlation (ρ). Accuracy is used for WiC and uncentered Pearson correlation for Cosimlex-I. The combination of static and contextualized embeddings has also been shown to benefit downstream tasks, such as Bilingual Lexicon Induction (BLI) (Zhang et al. 2021a) and social media categorization (Alghanmi, Espinosa Anke, and Schockaert 2020).

The methods that have been presented in this section are aimed at deriving word type-level embeddings from contextualized representations, or at mutually enriching static and contextualized vectors through their combination. The former type of transformation is useful for reducing the impact of context variation, and for exploring the word type-level information that is encoded in contextualized representations. As for vector combination, it serves to improve the quality of each type of embeddings, allowing them to benefit from the strengths of the representations with which they are being combined.

The goal of this survey has been to trace the evolution of word meaning representations from static to contextualized word embeddings, and to present current approaches for vector transformation and static embedding derivation. Despite the superiority of contextualized over word type-level vectors in Natural Language Understanding tasks, static representations present several advantages in application settings in terms of speed, computational resources, and ease of use. From a theoretical standpoint, the resurgence of strategies aimed at obtaining vectors at the word type-level from vectors built for individual tokens has been motivated by the distorted similarity estimates that are derived from contextual models’ space. Observations about the intense contextualization of token representations—which complicates reasoning at a higher, more abstract, level—have also contributed to this direction.

Several questions around word meaning representation remain open for exploration. Further investigation of the degeneration issue in contextual embedding spaces is needed, as well as the development of mitigation methods aimed at increasing the isotropy of contextual spaces and improving the quality of the similarity estimates that can be derived from them (Ethayarajh 2019a; Mu and Viswanath 2018; Rajaee and Pilehvar 2022). It is also important to revisit the metrics commonly used for measuring semantic similarity which might not be very informative in highly anisotropic contextual spaces (Timkey and van Schijndel 2021). Furthermore, although contextualized representations contain rich information about words and their context, the representation of longer text sequences remains a challenge. The BERT [CLS] token, which is commonly used to represent sentences in classification settings, hardly captures their meaning (Reimers and Gurevych 2019). Notably, it produces representations of inferior quality compared to the ones created by averaging over sentences’ subword tokens (Vulić et al. 2020b). Such a simple method, which represents a sentence by the weighted average of the pre-computed word vectors (slightly modified using PCA/SVD), was also shown to work particularly well for static embeddings (Arora, Liang, and Ma 2017). However, one drawback of this approach is that it does not account for word order and for the interaction between words in a sentence, which are important for generating a proper interpretation. An interesting new research avenue would be to explore strategies that account for these factors, possibly drawing inspiration from methods that have been proposed in the distributional semantics literature for capturing compositionality (Mitchell and Lapata 2008, 2009; Baroni, Bernardi, and Zamparelli 2014; Boleda and Herbelot 2016; Baroni 2019). A proper representation of the meaning of longer text segments is crucial for improving the models’ language understanding and reasoning capabilities.

As far as interpretability is concerned, the initial goal of “BERTology” studies has been to explain the high performance of language models in NLU tasks. It has, however, been shown that the models might not actually use the rich knowledge that is encoded in their representations for performing these tasks (Ravichander, Belinkov, and Hovy 2021; Feder et al. 2021; Belinkov 2022). An interesting and challenging topic for further exploration is to identify the semantic knowledge that is actually used to understand and reason about language. This could be done using counterfactual and adversarial approaches that study the impact of specific types of information on model performance (Elazar et al. 2021; Goodfellow, Shlens, and Szegedy 2015; Jia and Liang 2017; Alzantot et al. 2018).

Adversarial methods constitute a valuable tool for assessing the knowledge that language models encode, their understanding of it, and the extent to which it is used for accomplishing specific tasks. Generating adversarial examples for text data is, however, challenging due to the discrete nature of word tokens (as opposed to the continuous nature of image pixel values). In addition, a natural language attacking system is expected to satisfy three utility-preserving properties: human prediction consistency, semantic similarity, and language fluency (Jin et al. 2020). Future work should focus on developing adversarial methods that satisfy these utility-preserving constraints, in order to increase the usefulness of such methods. Adversarial methods addressing semantics would be highly useful, especially given the brittleness of language models to small changes in the ways they are queried (Jiang et al. 2020).

An additional limitation of models that are trained with the Masked Language Modeling objective is that they learn to predict a single word (or wordpiece) that is missing from a query. As a result, they cannot propose fillers for more than one slot. Combining the probabilities given by BERT to individual wordpieces for tokens composed of multiple subword tokens (or multi-word expressions) is non-trivial, since it would require running BERT several times with the same number of masked tokens as are the wordpieces, increasing computational requirements (Pimentel et al. 2020). In order to estimate the probability distribution over the entire vocabulary, an arbitrary number of MASKs would be needed at each position and the probability values would have to be normalized. Although methods that generate contextual representations at different granularity exist (e.g., SpanBERT and AMBERT), the lexical knowledge encoded in these representations has not yet been analyzed. This type of investigation could provide useful insights regarding the models’ understanding of the meaning of longer sequences, and about compositionality processing in contextual language models.

Finally, the semantic knowledge that language models encode depends on the data they were exposed to during training. The impact of reporting bias on the amount and quality of this knowledge is important, since people do not tend to state trivial perceptual or commonsense information in texts (Gordon and Van Durme 2013; Shwartz and Choi 2020). This can, however, be captured using different modalities. Language grounding is a challenging and highly interesting perspective in this respect that could enhance the models’ commonsense reasoning potential. Knowledge drawn from images (Lazaridou, Pham, and Baroni 2015; Li et al. 2020, 2021; Zhang et al. 2021b; Yang et al. 2022) could serve to complement the often incomplete information derived from texts, and to reduce the biases that are present, and often amplified, in language model representations.

Table A1 contains information about existing word similarity datasets that are commonly used for static (word type-level) embedding evaluation. We give the number of word pairs contained in each dataset, the grammatical category of the words included in the dataset, the scale of similarity scores used, and the number of annotators. For most of these datasets, annotators were asked to assign an absolute similarity score to each word pair, while for MEN (Bruni et al. 2012) they were asked to make comparative judgments between words. Given two candidate word pairs (e.g., wallet-moon and car-automobile), they had to pick the pair whose words were most related in meaning. Each pair was rated against 50 comparison pairs, resulting in a final score on a 50-point scale.

Most of these datasets are in English, except for Multi Simlex which is multilingual and addresses twelve typologically diverse languages: Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese. The dataset contains 1,888 semantically aligned concept pairs in each language.

Table A2 contains examples extracted from English in-context similarity datasets commonly used for evaluating contextualized representations. CoInCo (Kremer et al. 2014) and Usim (Erk, McCarthy, and Gaylord 2009) contain manual annotations, while ukWaC-subs (Garí Soler and Apidianaki 2020b) and WiC (Pilehvar and Camacho-Collados 2019) have been automatically created. We provide examples of sentence pairs from each dataset with their annotation and the target word highlighted. For CoInCo, the annotations are in-context substitutes proposed by the annotators. In ukWaC-subs, instance pairs are categorized as true (T) or false (F) depending on the overlap of substitutes that have been automatically assigned to them using context2vec (Melamud, Goldberger, and Dagan 2016). ukWaC-subs contains sentence pairs where a word w is replaced by (a) a good substitute, (b) a synonym of a different sense of w (i.e., not a good in-context substitute), (c) a random word of the same part of speech. The WiC dataset contains T/F labels that have been automatically assigned to instance pairs based on their proximity in WordNet. Finally, Usim and the Stanford Contextual Word Similarity (SCWS) dataset (Huang et al. 2012) contain manual usage similarity annotations with scores in the range from 1 (low similarity) to 5 (high similarity) for Usim, and 1 to 10 for SCWS. Usim targets instances of the same word. SCWS contains annotations for instance pairs corresponding to different words, and for homographs with different part of speech (e.g., pack as a verb and a noun).

This research is based on work supported in part by the European Research Council (ERC) FoTran project (grant agreement 771113), the NSF (award No. 1928631), and the Office of the Director of National Intelligence (ODNI) via the IARPA HIATUS Program (contract No. 2022-22072200005). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, NSF, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.