Unsupervised Compositionality Prediction of Nominal Compounds

It is a universally acknowledged assumption that the meaning of phrases, expressions, or sentences can be determined by the meanings of their parts and by the rules used to combine them. Part of the appeal of this principle of compositionality¹ is that it implies that a meaning can be assigned even to a new sentence involving an unseen combination of familiar words (Goldberg 2015). Indeed, for natural language processing (NLP), this is an attractive way of linearly deriving the meaning of larger units from their components, performing the semantic interpretation of any text.

For representing the meaning of individual words and their combinations in computational systems, distributional semantic models (DSMs) have been widely used. DSMs are based on Harris’ distributional hypothesis that the meaning of a word can be inferred from the context in which it occurs (Harris 1954; Firth 1957). In DSMs, words are usually represented as vectors that, to some extent, capture cooccurrence patterns in corpora (Lin 1998; Landauer, Foltz, and Laham 1998; Mikolov et al. 2013; Baroni, Dinu, and Kruszewski 2014). Evaluation of DSMs has focused on obtaining accurate semantic representations for words, and state-of-the-art models are already capable of obtaining a high level of agreement with human judgments for predicting synonymy or similarity between words (Freitag et al. 2005; Camacho-Collados, Pilehvar, and Navigli 2015; Lapesa and Evert 2017) and for modeling syntactic and semantic analogies between word pairs (Mikolov, Yih, and Zweig 2013). These representations for individual words can also be combined to create representations for larger units such as phrases, sentences, and even whole documents, using simple additive and multiplicative vector operations (Mitchell and Lapata 2010; Reddy, McCarthy, and Manandhar 2011; Mikolov et al. 2013; Salehi, Cook, and Baldwin 2015), syntax-based lexical functions (Socher et al. 2012), or matrix and tensor operations (Baroni and Lenci 2010; Bride, Van de Cruys, and Asher 2015). However, it is not clear to what extent this approach is adequate in the case of idiomatic multiword expressions (MWEs). MWEs fall into a wide spectrum of compositionality; that is, some MWEs are more compositional (e.g., olive oil) while others are more idiomatic (Sag et al. 2002; Baldwin and Kim 2010). In the latter case, the meaning of the MWE may not be straightforwardly related to the meanings of its parts, creating a challenge for the principle of compositionality (e.g., snake oil as a product of questionable benefit, not necessarily an oil and certainly not extracted from snakes).

In this article, we discuss approaches for automatically detecting to what extent the meaning of an MWE can be directly computed from the meanings of its component words, represented using DSMs. We evaluate how accurately DSMs can model the semantics of MWEs with various levels of compositionality compared to human judgments. Since MWEs encompass a large amount of related but distinct phenomena, we focus exclusively on a subcategory of MWEs: nominal compounds. They represent an ideal case study for this work, thanks to their relatively homogeneous syntax (as opposed to other categories of MWEs such as verbal idioms) and their pervasiveness in language. We assume that models able to predict the compositionality of nominal compounds could be generalized to other MWE categories by addressing their variability in future work. Furthermore, to determine to what extent these approaches are also adequate cross-lingually, we evaluate them in three languages: English, French, and Portuguese.

Given that MWEs are frequent in languages (Sag et al. 2002), identifying idiomaticity and producing accurate semantic representations for compositional and idiomatic cases is of relevance to NLP tasks and applications that involve some form of semantic processing, including semantic parsing (Hwang et al. 2010; Jagfeld and van der Plas 2015), word sense disambiguation (Finlayson and Kulkarni 2011; Schneider et al. 2016), and machine translation (Ren et al. 2009; Carpuat and Diab 2010; Cap et al. 2015; Salehi et al. 2015). Moreover, the evaluation of DSMs on tasks involving MWEs, such as compositionality prediction, has the potential to drive their development towards new directions.

The main hypothesis of our work is that, if the meaning of a compositional nominal compound can be derived from a combination of its parts, this translates in DSMs as similar vectors for a compositional nominal compound and for the combination of the vectors of its parts using some vector operation, that we refer to as composition function. Conversely we can use the lack of similarity between the nominal compound vector representation and a combination of its parts to detect idiomaticity. Furthermore, we hypothesize that accuracy in predicting compositionality depends both on the characteristics of the DSMs used to represent expressions and their components and on the composition function adopted. Therefore, we have built 684 DSMs and performed an extensive evaluation, involving over 9,072 analyses, investigating various types of DSMs, their configurations, the corpora used to train them, and the composition function used to build vectors for expressions.²

This article is structured as follows. Section 2 presents related work on distributional semantics, compositionality prediction, and nominal compounds. Section 3 presents the data sets created for our evaluation. Section 4 describes the compositionality prediction framework, along with the composition functions which we evaluate. Section 5 specifies the experimental setup (corpora, DSMs, parameters, and evaluation measures). Section 6 presents the overall results of the evaluated models. Sections 7 and 8 evaluate the impact of DSM and corpus parameters, and of composition functions on compositionality prediction. Section 9 discusses system predictions through an error analysis. Section 10 summarizes our conclusions. Appendix A contains a glossary, Appendix B presents extra sanity-check experiments, Appendix C contains the questionnaire used for data collection, and Appendices D, E, and F list the compounds in the data sets.

The literature on distributional semantics is extensive (Lin 1998; Turney and Pantel 2010; Baroni and Lenci 2010; Mohammad and Hirst 2012), so we provide only a brief introduction here, underlining their most relevant characteristics to our framework (Section 2.1). Then, we define compositionality prediction and discuss existing approaches, focusing on distributional techniques for multiword expressions (Section 2.2). Our framework is evaluated on nominal compounds, and we discuss their relevant properties (Section 2.3) along with existing data sets for evaluating compositionality prediction (Section 2.4).

Distributional semantic models (DSMs) use context information to represent the meaning of lexical units as vectors. These vectors are built assuming the distributional hypothesis, whose central idea is that the meaning of a word can be learned based on the contexts where it appears—or, as popularized by Firth (1957), “you shall know a word by the company it keeps.”

Formally, a DSM attempts to encode the meaning of each target word w_i of a vocabulary V as a vector of real numbers v(w_i) in ℝ^|V|. Each component of v(w_i) is a function of the co-occurrence between w_i and the other words in the vocabulary (its contexts w_c). This function can be simply a co-occurrence count c(w_i, w_c), or some measure of the association between w_i and each w_c, such as pointwise mutual information (PMI, Church and Hanks [1990]; Lin [1999]) or positive PMI (PPMI, Baroni, Dinu, and Kruszewski [2014]; Levy, Goldberg, and Dagan [2015]).

In DSMs, co-occurrence can be defined as two words co-occurring in the same document, sentence, or sentence fragment in a corpus. Intrasentential models are often based on a sliding window; that is, a context word w_c co-occurs within a certain window of W words around the target w_i. Alternatively, co-occurrence can also be based on syntactic relations obtained from parsed corpora, where a context word w_c appears within specific syntactic relations with w_i (Lin 1998; Padó and Lapata 2007; Lapesa and Evert 2017).

The set of all vectors v(w_i), ∀w_i ∈ V can be represented as a sparse co-occurrence matrix V × V → ℝ. Given that most word pairs in this matrix co-occur rarely (if ever), a threshold on the number of co-occurrences is often applied to discard irrelevant pairs. Additionally, co-occurrence vectors can be transformed to have a significantly smaller number of dimensions, converting vectors in ℝ^|V| into vectors in ℝ^d, with d ≪ |V|.³ Two solutions are commonly employed in the literature. The first one consists in using context thresholds, where all target–context pairs that do not belong to the top-d most relevant pairs are discarded (Salehi, Cook, and Baldwin 2014; Padró et al. 2014b). The second solution consists in applying a dimensionality reduction technique such as singular value decomposition on the co-occurrence matrix where only the d largest singular values are retained (Deerwester et al. 1990). Similar techniques focus on the factorization of the logarithm of the co-occurrence matrix (Pennington, Socher, and Manning 2014) and on alternative factorizations of the PPMI matrix (Salle, Villavicencio, and Idiart 2016).

Alternatively, DSMs can be constructed by training a neural network to predict target–context relationships. For instance, a network can be trained to predict a target word w_i among all possible words in V given as input a window of surrounding context words. This is known as the continuous bag-of-words model. Conversely, the network can try to predict context words for a target word given as input, and this is known as the skip-gram model (Mikolov et al. 2013). In both cases, the network training procedure allows encoding in the hidden layer semantic information about words as a side effect of trying to solve the prediction task. The weight parameters that connect the unity representing w_i with the d-dimensional hidden layer are taken as its vector representation v(w_i).

There are a number of factors that may influence the ability of a DSM to accurately learn a semantic representation. These include characteristics of the training corpus such as size (Mikolov, Yih, and Zweig 2013) as well as frequency thresholds and filters (Ferret 2013; Padró et al. 2014b), genre (Lapesa and Evert 2014), preprocessing (Padó and Lapata 2003, 2007), and type of context (window vs. syntactic dependencies) (Agirre et al. 2009; Lapesa and Evert 2017). Characteristics of the model include the choice of association and similarity measures (Curran and Moens 2002), dimensionality reduction strategies (Van de Cruys et al. 2012), and the use of subsampling and negative sampling techniques (Mikolov, Yih, and Zweig 2013). However, the particular impact of these factors on the quality of the resulting DSM may be heterogeneous and depends on the task and model (Lapesa and Evert 2014). Because there is no consensus about a single optimal model that works for all tasks, we compare a variety of models (Section 5) to determine which are best suited for our compositionality prediction framework.

Before adopting the principle of compositionality to determine the meaning of a larger unit, such as a phrase or multiword expression (MWE), it is important to determine whether it is idiomatic or not.⁴ This problem, known as compositionality prediction, can be solved using methods that measure directly the extent to which an expression is constructed from a combination of its parts, or indirectly via language-dependent properties of MWEs linked to idiomaticity like the degree of determiner variability and morphological flexibility (Fazly, Cook, and Stevenson 2009; Tsvetkov and Wintner 2012; Salehi, Cook, and Baldwin 2015; Köper and Schulte im Walde 2016). In this article, we focus on direct prediction methods in order to evaluate the target languages under similar conditions. Nonetheless, this does not exclude the future integration of information used by indirect prediction methods, as a complement to the methods discussed here.

For direct prediction methods, three ingredients are necessary. First, we need vector representations of single-word meanings, such as those built using DSMs (Section 2.1). Second, we need a mathematical model of how the compositional meaning of a phrase is calculated from the meanings of its parts. Third, we need the compositionality measure itself, which estimates the similarity between the compositionally constructed meaning of a phrase and its observed meaning, derived from corpora. There are a number of alternatives for each of the ingredients, and throughout this article we call a specific choice of the three ingredients a compositionality prediction configuration.

Regarding the second ingredient, that is, the mathematical model of compositional meaning, the most natural choice is the additive model (Mitchell and Lapata 2008). In the additive model, the compositional meaning of a phrase w₁w₂ … w_n is calculated as a linear combination of the word vectors of its components: ∑_iβ_iv(w_i), where v(w_i) is a d-dimensional vector for each word w_i, and the β_i coefficients assign different weights to the representation of each word (Reddy, McCarthy, and Manandhar 2011; Schulte im Walde, Müller, and Roller 2013; Salehi, Cook, and Baldwin 2015). These weights can capture the asymmetric contribution of each of the components to the semantics of the whole phrase (Bannard, Baldwin, and Lascarides 2003; Reddy, McCarthy, and Manandhar 2011). For example, in flea market, it is the head (market) that has a clear contribution to the overall meaning, whereas in couch potato it is the modifier (couch).

The additive model can be generalized to use a matrix of multiplicative coefficients, which can be estimated through linear regression (Guevara 2011). This model can be further modified to learn polynomial projections of higher degree, with quadratic projections yielding particularly promising results (Yazdani, Farahmand, and Henderson 2015). These models come with the caveat of being supervised, requiring some amount of pre-annotated data in the target language. Because of these requirements, our study focuses on unsupervised compositionality prediction methods only, based exclusively on automatically POS-tagged and lemmatized monolingual corpora.

Alternatives to the additive model include the multiplicative model and its variants (Mitchell and Lapata 2008). However, results suggest that this representation is inferior to the one obtained through the additive model (Reddy, McCarthy, and Manandhar 2011; Salehi, Cook, and Baldwin 2015). Recent work on predicting intra-compound semantics also supports that additive models tend to yield better results than multiplicative models (Hartung et al. 2017).

The third ingredient is the measure of similarity between the compositionally constructed vector and its actual corpus-based representation. Cosine similarity is the most commonly used measure for compositionality prediction in the literature (Schone and Jurafsky 2001; Reddy, McCarthy, and Manandhar 2011; Schulte im Walde, Müller, and Roller 2013; Salehi, Cook, and Baldwin 2015). Alternatively, one can calculate the overlap between the distributional neighbors of the whole phrase and those of the component words (McCarthy, Keller, and Carroll 2003), or the number of single-word distributional neighbors of the whole phrase (Riedl and Biemann 2015).

Instead of covering compositionality prediction for MWEs in general, we focus on a particular category of phenomena represented by nominal compounds. We define a nominal compound as a syntactically well-formed and conventionalized noun phrase containing two or more content words, whose head is a noun.⁵ They are conventionalized (or institutionalized) in the sense that their particular realization is statistically idiosyncratic, and their constituents cannot be replaced by synonyms (Sag et al. 2002; Baldwin and Kim 2010; Farahmand, Smith, and Nivre 2015). Their semantic interpretation may be straightforwardly compositional, with contributions from both elements (e.g., climate change), partly compositional, with contribution mainly from one of the elements (e.g., grandfather clock), or idiomatic (e.g., cloud nine) (Nakov 2013).

The syntactic realization of nominal compounds varies across languages. In English, they are often expressed as a sequence of two nouns, with the second noun as the syntactic head, modified by the first noun. This is the most frequently annotated POS-tag pattern in the MWE-annotated DiMSUM English corpus (Schneider et al. 2016). In French and Portuguese, they often assume the form of adjective–noun or noun–adjective pairs, where the adjective modifies the noun. Examples of such constructions include the adjective–noun compound FRpetite annonce (lit. small announcement ‘classified ad’) and the noun–adjective compound PTburaco negro (lit. hole black ‘black hole’).⁶ Additionally, compounds may also involve prepositions linking the modifier with the head, as in the case of FRcochon d’Inde (lit. pig of India ‘guinea pig’) and PTdente de leite (lit. tooth of milk ‘milk tooth’). Because prepositions are highly polysemous and their representation in DSMs is tricky, we do not include compounds containing prepositions in this article. Hence, we focus on 2-word nominal compounds of the form noun₁–noun₂ (in English), and noun–adjective and adjective–noun (in the three languages).

Regarding the meaning of nominal compounds, the implicit relation between the components of compositional compounds can be described in terms of free paraphrases involving verbs, such as flu virus as virus that causes/creates flu (Nakov 2008),⁷ or prepositions, such as olive oil as oil from olives (Lauer 1995). These implicit relations can often be seen explicitly in the equivalent expressions in other languages (e.g., FRhuile d’olive and PTazeite de oliva for ENolive oil).

Alternatively, the meaning of compositional nominal compounds can be described using a closed inventory of relations which make the role of the modifier explicit with respect to the head noun, including syntactic tags such as subject and object, and semantic tags such as instrument and location (Girju et al. 2005). The degree of compositionality of a nominal compound can also be represented using numerical scores (Section 2.4) to indicate to what extent the component words allow predicting the meaning of the whole (Reddy, McCarthy, and Manandhar 2011; Roller, Schulte im Walde, and Scheible 2013; Salehi et al. 2015). The latter is the representation that we adopted in this article.

The evaluation of compositionality prediction models can be performed extrinsically or intrinsically. In extrinsic evaluation, compositionality information can be used to decide how a compound should be treated in NLP systems such as machine translation or text simplification. For instance, for machine translation, idiomatic compounds need to be treated as atomic phrases, as current methods of morphological compound processing cannot be applied to them (Stymne, Cancedda, and Ahrenberg 2013; Cap et al. 2015).

Although potentially interesting, extrinsic evaluation is not straightforward, as results may be influenced both by the compositionality prediction model and by the strategy for integration of compositionality information into the NLP system. Therefore, most related work focuses on an intrinsic evaluation, where the compositionality scores produced by a model are compared to a gold standard, usually a data set where nominal compound semantics have been annotated manually. Intrinsic evaluation thus requires the existence of data sets where each nominal compound has one (or several) numerical scores associated with it, indicating its compositionality. Annotations can be provided by expert linguist annotators or by crowdsourcing, often requiring that several annotators judge the same compound to reduce the impact of subjectivity on the scores. Relevant compositionality data sets of this type are listed below, some of which were used in our experiments.

• • 

  Reddy, McCarthy, and Manandhar (2011) collected judgments for a set of 90 English noun–noun (e.g., zebra crossing) and adjective–noun (e.g., sacred cow) compounds, in terms of three numerical scores: the compositionality of the compound as a whole and the literal contribution of each of its parts individually, using a scale from 0 to 5. The data set was built through crowdsourcing, and the final scores are the average of 30 judgments per compound.This data set will be referred to as Reddy in our experiments.

• • 

  Farahmand, Smith, and Nivre (2015) collected judgments for 1,042 English noun–noun compounds. Each compound has binary judgments regarding non-compositionality and conventionalization given by four expert annotators (both native and non-native speakers). A hard threshold is applied so that compounds are considered as noncompositional if at least two annotators say so (Yazdani, Farahmand, and Henderson 2015), and the total compositionality score is given by the sum of the four binary judgments. This data set will be referred to as Farahmand in our experiments.

• • 

  Kruszewski and Baroni (2014) built the Norwegian Blue Parrot data set, containing judgments for modifier-head phrases in English. The judgments consider whether the phrase is (1) an instance of the concept denoted by the head (e.g., dead parrot and parrot) and (2) a member of the more general concept that includes the head (e.g., dead parrot and pet), along with typicality ratings, with 5,849 judgments in total.

• • 

  Roller, Schulte im Walde, and Scheible (2013) collected judgments for a set of 244 German noun–noun compounds, each compound with an average of around 30 judgments on a compositionality scale from 1 to 7, obtained through crowdsourcing. The resource was later enriched with feature norms (Roller and Schulte im Walde 2014).

• • 

  Schulte im Walde et al. (2016) collected judgments for a set of 868 German noun–noun compounds, including human judgments of compositionality on a scale of 1 to 6. Compounds are judged by multiple annotators, and the final compositionality score is the average across annotators. The data set is also annotated for in-corpus frequency, productivity, and ambiguity, and a subset of 180 compounds has been selected for balancing these variables. The annotations were performed by the authors, linguists, and through crowdsourcing. For the balanced subset of 180 compounds, compositionality annotations were performed by experts only, excluding the authors.

For a multilingual evaluation, in this work, we construct two data sets, one for French and one for Portuguese compounds, and extend the Reddy data set for English using the same protocol as Reddy, McCarthy, and Manandhar (2011).

In Section 3.1, we describe the construction of data sets of 180 compounds for French (FR-comp) and Portuguese (PT-comp). For English, the complete data set contains 280 compounds, of which 190 are new and 90 come from the Reddy data set. We use 180 of these (EN-comp) for cross-lingual comparisons (90 from the original Reddy data set combined with 90 new ones from EN-comp₉₀), and 100 new compounds as held-out data (EN-comp_Ext), to evaluate the robustness of the results obtained (Section 6.3). These data sets containing compositionality scores for 2–word nominal compounds are used to evaluate our framework (Section 4), and we discuss their characteristics in Section 3.2.⁸

For each of the target languages, we collected, via crowdsourcing, a set of numerical scores corresponding to the level of compositionality of the target nominal compounds. We asked non-expert participants to judge each compound considering three sentences where the compound occurred. After reading the sentences, participants assess the degree to which the meaning of the compound is related to the meanings of its parts. This follows from the assumption that a fully compositional compound will have an interpretation whose meaning stems from both words (e.g., lime tree as a tree of limes), while a fully idiomatic compound will have a meaning that is unrelated to its components (e.g., nut case as an eccentric person).

Our work follows the protocol proposed by Reddy, McCarthy, and Manandhar (2011), where compositionality is explained in terms of the literality of the individual parts. This type of indirect annotation does not require expert linguistic knowledge, and still provides reliable data, as we show later. For each language, data collection involved four steps: compound selection, sentence selection, questionnaire design, and data aggregation.

For each data set, we manually selected nominal compounds from dictionaries, corpus searches, and by linguistic introspection, maintaining an equal proportion of compounds that are compositional, partly compositional, and idiomatic.⁹ We considered them to be compositional if their semantics are related to both components (e.g., benign tumor), partly compositional if their semantics are related to only one of the components (e.g., grandfather clock), and idiomatic if they are not directly related to either (e.g., old flame). This preclassification was used only to select a balanced set of compounds and was not shown to the participants nor used at any later stage. For all languages, all compounds are required to have a head that is unambiguously a noun, and additionally for French and Portuguese, all compounds have an adjective as modifier.

Compounds may be polysemous (e.g., FRbras droit may mean most reliable helper or literally right arm). To avoid any potential sense uncertainty, each compound was presented to the participants with the same sense in three sentences. These sentences were manually selected from the WaC corpora: ukWaC (Baroni et al. 2009), frWaC, and brWaC (Boos, Prestes, and Villavicencio 2014), presented in detail in Section 5.

For each compound, after reading three sentences, participants are asked to:

• • 

  provide synonyms for the compound in these sentences. The synonyms are used as additional validation of the quality of the judgments: if unrelated words are provided, the answers are discarded.

• • 

  assess the contribution of the head noun to the meaning of the compound (e.g., is a busy bee always literally a bee?)

• • 

  assess the contribution of the modifier noun or adjective to the meaning of the compound (e.g., is a busy bee always literally busy?)

• • 

  assess the degree to which the compound can be seen as a combination of its parts (e.g., is a busy bee always literally a bee that is busy?)

For English and French, we collected answers using Amazon Mechanical Turk (AMT), manually removing answers that were not from native speakers or where the synonyms provided were unrelated to the target compound sense. Because AMT has few Brazilian Portuguese native speakers, we developed an in-house web interface for the questionnaire, which was sent out to Portuguese-speaking NLP mailing lists.

For a given compound and question we calculate aggregated scores as the arithmetic averages of all answers across participants. We will refer to these averaged scores as the human compositionality score (hc)s. We average the answers to the three questions independently, generating three scores: hc_H for the head noun, hc_M for the modifier, and hc_HM for the whole compound. In our framework, we try to predict hc_HM automatically (Section 5). To assess the variability of the answers (Section 3.2.1), we also calculate the standard deviation across participants for each question (σ_H, σ_M, and σ_HM).

The list of compounds, their translations, glosses, and compositionality scores are given in Appendices D (EN-comp₉₀ and EN-comp_Ext), E (FR-comp), and F (PT-comp).¹⁰

In this section, we present different measures of agreement among participants (Section 3.2.1) and examine possible correlations between compositionality scores, familiarity, and conventionalization (Section 3.2.2) in the data sets created for this article.

To assess the quality of the collected human compositionality scores, we use standard deviation and inter-annotator agreement scores.

The standard deviation (σ) of the participants’ answers can be used as an indication of their agreement: for each compound and for each of the three questions, small σ values suggest greater agreement. In addition, if the instructions are clear, σ can also be seen as an indication of the level of difficulty of the task. In other words, all other things being equal, compounds with larger σ can be considered intrinsically harder to analyze by the participants. For each data set, we consider two aggregated metrics based on σ:

• • 

  $σ¯$ — The average of σ in the data set.

• • 

  P_σ>1.5 — The proportion of compounds whose σ is higher than 1.5.

Table 1 presents the result of these metrics when applied to our in-house data sets, as well as to the original Reddy data set. The column $n¯$ indicates the average number of answers per compound, while the other six columns present the values of $σ¯$ and P_σ>1.5 for compound (HM), head-only (H), and modifier-only (M) scores.

These values are below what would be expected for random decisions (⁠$σ¯$_rand ≃ 1.71, for the Likert scale). Although our data sets exhibit higher variability than Reddy, this may be partly due to the application of filters done by Reddy, McCarthy, and Manandhar (2011) to remove outliers.¹¹ These values could also be due to the collection of fewer answers per compound for some of the data sets. However, there is no clear tendency in the variation of the standard deviation of the answers and the number of participants n. The values of $σ¯$ are quite homogeneous, ranging from 1.05 for EN-comp₉₀ (head) to 1.27 for EN-comp_Ext (head). The low agreement for modifiers may be related to a greater variability in semantic relations between modifiers and compounds: these include material (e.g., brass ring), attribute (e.g., black cherry), and time (e.g., night owl).

Figure 1(a) shows standard deviation (σ_HM, σ_H, and σ_M) for each compound of FR-comp as a function of its average compound score hc_HM.¹² For all three languages, greater agreement was found for compounds at the extremes of the compositionality scale (fully compositional or fully idiomatic) for all scores. These findings can be partly explained by end-of-scale effects, that result in greater variability for the intermediate scores in the Likert scale (from 1 to 4) that correspond to the partly compositional cases. Hence, we expect that it will be easier to predict the compositionality of idiomatic/compositional compounds than of partly compositional ones.

To measure inter-annotator agreement of multiple participants, taking into account the distance between the ordinal ratings of the Likert scale, we adopt the α score (Artstein and Poesio 2008). The α score is more appropriate for ordinal data than traditional agreement scores for categorical data, such as Cohen’s and Fleiss’ κ (Cohen 1960; Fleiss and Cohen 1973). However, due to the use of crowdsourcing, most participants rated only a small number of compounds with very limited chance of overlap among them: the average number of answers per participant is 13.6 for EN-comp₉₀, 10.2 for EN-comp_Ext, 33.7 for FR-comp, and 53.5 for PT-comp. Because the α score assumes that each participant rates all the items, we focus on the answers provided by three of the participants, who rated the whole set of 180 compounds in PT-comp.

Using a linear distance schema between the answers,¹³ we obtain an agreement of α = .58 for head-only, α = .44 for modifier-only, and α = .44 for the whole compound. To further assess the difficulty of this task, we also calculate α for a single expert annotator, judging the same set of compounds after an interval of one month. The scores were α = .69 for the head and α = .59 for both the compound and for the modifier. The Spearman correlation between these two annotations performed by the same expert is ρ = 0.77 for hc_HM. This can be seen as a qualitative upper bound for automatic compositionality prediction on PT-comp.

Figure 1(b) shows the average scores (hc_HM, hc_H, and hc_M) for the compounds ranked according to the average compound score hc_HM. Although this figure is for FR-comp, similar patterns were found for the other data sets. For all three languages, the human compositionality scores provide additional confirmation that the data sets are balanced, with the compound scores (hc_HM) being distributed linearly along the scale. Furthermore, we have calculated the average hc_HM values separately for the compounds in each of the three compositionality classes used for compound selection: idiomatic, partly compositional and compositional (Section 3.1). These averages are, respectively, 1.0, 2.4, and 4.0 for EN-comp₉₀; 1.1, 2.4, and 4.2 for EN-comp_Ext; 1.3, 2.7, and 4.3 for FR-comp; and 1.3, 2.5, and 3.9 for PT-comp, indicating that our attempt to select a balanced number of compounds from each class is visible in the collected hc_HM scores.

Additionally, the human scores also suggest an asymmetric impact of the non-literal parts over the compound: whenever participants judged an element of the compound as non-literal, the whole compound was also rated as idiomatic. Thus, most head and modifier scores (hc_H and hc_M) are close to or above the diagonal line in Figure 1(b). In other words, a component of the compound is seldom rated as less literal than the compositionality of the whole compound hc_HM, although the opposite is more common.

To evaluate if it is possible to predict hc_HM from the hc_H and hc_M, we calculate the arithmetic and geometric means between hc_H and hc_M for each compound. Figure 2 shows the linear regression of both measures for FR-comp. The goodness of fit is $rarith2$ = .93 for the arithmetic mean, and $rgeom2$ = .96 for the geometric mean, confirming that they are good predictors of hc_HM.¹⁴ Thus, we assume that hc_HM summarizes hc_H and hc_M, and focus on predicting hc_HM instead of hc_H and hc_M separately. These findings also inspired the pc_arith and pc_geom compositionality prediction functions (Section 4).

To examine whether there is an effect of the familiarity of a compound on hc scores, in particular if more idiomatic compounds need to be more familiar, we also calculated the correlation between the compositionality score for a compound hc_HM and its frequency in a corpus, as a proxy for familiarity. In this case we used the WaC corpora and calculated the frequencies based on the lemmas. The results, in Table 2, show a statistically significant positive Spearman correlation of ρ = 0.305 for EN-comp₉₀, ρ = 0.384 for EN-comp_Ext, and ρ = 0.598 for FR-comp, indicating that, contrary to our expectations, compounds that are more frequent tend to be assigned higher compositionality scores. However, frequency alone is not enough to predict compositionality, and further investigation is needed to determine if compositionality and frequency are also correlated with other factors.

We also analyzed the correlation between compositionality and conventionalization to determine if more idiomatic compounds correspond to more conventionalized ones. We use PMI (Church and Hanks 1990) as a measure of conventionalization, as it indicates the strength of association between the components (Farahmand, Smith, and Nivre 2015). We found no statistically significant correlation between compositionality and PMI.

This section describes the common setup used for evaluating compositionality prediction, such as corpora (Section 5.1), DSMs (Section 5.2), and evaluation metrics (Section 5.3).

In this work we used the lemmatized and POS-tagged versions of the WaC corpora not only for building DSMs, but also as sources of information about the target compounds for the analyses performed (e.g., in Sections 3.2.2, 9.1, and 9.2):

• • 

  for English, the ukWaC (Baroni et al. 2009), with 2.25 billion tokens, parsed with MaltParser (Nivre, Hall, and Nilsson 2006);

• • 

  for French, the frWaC with 1.61 billion tokens preprocessed with TreeTagger (Schmid 1995); and

• • 

  for Brazilian Portuguese, a combination of brWaC (Boos, Prestes, and Villavicencio 2014), Corpus Brasileiro,¹⁸ and all Wikipedia entries,¹⁹ with a total of 1.91 billion tokens, all parsed with PALAVRAS (Bick 2000).

For all compounds contained in our data sets, we transformed their occurrences into single tokens by joining their component words with an underscore (e.g., ENmonkey business → monkey_business and FRbelle-mère → belle_mère).^20,21 To handle POS-tagging and lemmatization irregularities, we retagged the compounds’ components using the gold POS and lemma in our data sets (e.g., for ENsitting duck, sit/verb_duck/noun → sitting/adjective_duck/noun). We also simplified all POS tags using coarse-grained labels (e.g., verb instead of vvz). All forms are then lowercased (surface forms, lemmas, and POS tags); and noisy tokens, with special characters, numbers, or punctuation, are removed. Additionally, ligatures are normalized for French (e.g., œ → oe) and a spellchecker²² is applied to normalize words across English spelling variants (e.g., color → colour).

To evaluate the influence of preprocessing on compositionality prediction (Section 7.3), we generated four versions of each corpus, with different levels of linguistic information. We expect lemmatization to reduce data sparseness by merging morphologically inflected variants of the same lemma:

• 1. 

  surface⁺: the original raw corpus with no preprocessing, containing surface forms.

• 2. 

  surface: stopword removal, generating a corpus of surface forms of content words.

• 3. 

  lemma_PoS: stopword removal, lemmatization,²³ and POS-tagging; generating a corpus of content words distinguished by POS tags, represented as lemma/POS-tag.

• 4. 

  lemma: stopword removal and lemmatization; generating a corpus containing only lemmas of content words.

In this section, we describe the state-of-the-art DSMs used for compositionality prediction.

In the models based on the PPMI matrix, the representation of a target word is a vector containing the PPMI association scores between the target and its contexts (Bullinaria and Levy 2012). The contexts are nouns and verbs, selected in a symmetric sliding window of W words to the left/right and weighted linearly according to their distance D to the target (Levy, Goldberg, and Dagan 2015).²⁴ We consider three models that differ in how the contexts are selected:

• • 

  In PPMI–thresh, the vectors are |V|-dimensional but only the top d contexts with highest PPMI scores for each target word are kept, while the others are set to zero (Padró et al. 2014a).²⁵

• • 

  In PPMI–TopK, the vectors are d-dimensional, and each of the d dimensions corresponds to a context word taken from a fixed list of k contexts, identical for all target words. We chose k as the 1,000 most frequent words in the corpus after removing the top 50 most frequent words (Salehi, Cook, and Baldwin 2015).

• • 

  In PPMI–SVD, singular value decomposition is used to factorize the PPMI matrix and reduce its dimensionality from |V| to d.²⁶ We set the value of the context distribution smoothing factor to 0.75, and the negative sampling factor to 5 (Levy, Goldberg, and Dagan 2015). We use the default minimum word count threshold of 5.

Word2vec²⁷ relies on a neural network to predict target/context pairs (Mikolov et al. 2013). We use its two variants: continuous bag-of-words (w2v–cbow) and skip-gram (w2v–sg). We adopt the default configurations recommended in the documentation, except for: no hierarchical softmax, 25 negative samples, frequent-word down-sampling rate of 10⁻⁶, execution of 15 training iterations, and minimum word count threshold of 5.

GloVe²⁸ implements a factorization of the logarithm of the positional co-occurrence count matrix (Pennington, Socher, and Manning 2014). We adopt the default configurations from the documentation, except for: internal cutoff parameter x_max = 75 and processing of the corpus in 15 iterations. For the corpora versions lemma and lemma_PoS (Section 5.1), we use the minimum word count threshold of 5. For surface and surface⁺, due to the larger vocabulary sizes, we use thresholds of 15 and 20.²⁹

The LexVec model³⁰ factorizes the PPMI matrix in a way that penalizes errors on frequent words (Salle, Villavicencio, and Idiart 2016). We adopt the default configurations in the documentation, except for: 25 negative samples, subsampling rate of 10⁻⁶, and processing of the corpus in 15 iterations. Due to the vocabulary sizes, we use a word count threshold of 10 for lemma and lemma_PoS, and 100 for surface and surface⁺.³¹

In addition to model-specific parameters, the DSMs described above have some shared DSM parameters. We construct multiple DSM configurations by varying the values of these parameters. These combinations produce a total of 228 DSMs per language (see Table 3). In particular, we evaluate the influence of the following parameters on compositionality prediction:

• • 

  WindowSize: Number of context words to the left/right of the target word when searching for target-context co-occurrence pairs. The assumption is that larger windows are better for capturing semantic relations (Jurafsky and Martin 2009) and may be more suitable for compositionality prediction. We use window sizes of 1 + 1, 4 + 4, and 8 + 8.³²

• • 

  Dimension: Number of dimensions of each vector. The underlying hypothesis is that, the higher the number of dimensions, the more accurate the representation of the context is going to be. We evaluate our framework with vectors of 250, 500, and 750 dimensions.

• • 

  WordForm: One of the four word-form and stopword removal variants used to represent a corpus, in Section 5.1: surface⁺, surface, lemma, and lemma_PoS. They represent different levels of specificity in the informational content of the tokens, and may have a language-dependent impact on the performance of compositionality prediction.

To evaluate a compositionality prediction configuration, we calculate Spearman’s ρ rank correlation between the predicted compositionality scores (pc)s and the human compositionality scores (hc)s for the compounds that appear in the evaluation data set. We mostly use the rank correlation instead of linear correlation (Pearson) because we are interested in the framework’s ability to order compounds from least to most compositional, regardless of the actual predicted values.

For English, besides the evaluation data sets presented in Section 3, we also use Reddy and Farahmand (see Section 2.4) to enable comparison with related work. For Farahmand, since it contains binary judgments³³ instead of graded compositionality scores, results are reported using the best F₁ (BF₁) score, which is the highest F₁ score found using the top n compounds classified as noncompositional, when n is varied (Yazdani, Farahmand, and Henderson 2015). For Reddy, we sometimes present Pearson scores to enable comparison with related work.

Because of the large number of compositionality prediction configurations evaluated, we only report the best performance for each configuration over all possible DSM parameter values. The generalization of these analyses is then ensured using cross-validation and held-out data. To determine whether the difference between two prediction results are statistically different, we use nonparametric Wilcoxon’s sign-rank test.

In this section, we present the overall results obtained on the Reddy, Farahmand, EN-comp, FR-comp, and PT-comp data sets, comparing all possible configurations (Section 6.1). To determine their robustness we also report evaluation for all languages using cross-validation (Section 6.2) and for English using the held-out data set EN-comp_Ext (Section 6.3). All results reported in this section use the pc_uniform function.

Table 4 shows the highest overall values obtained for each DSM (columns) on each data set (rows). For English (Reddy, EN-comp, and Farahmand), the highest results for the compounds found in the corpus were obtained with w2v and PPMI–thresh, shown as the first value in each pair in Table 4. Not all compounds in the English data sets are present in our corpus. Therefore, we also report results adopting a fallback strategy (the second value). Because its impact depends on the data set, and the relative performance of the models is similar with or without it, for the remainder of the article we discuss only the results without fallback.³⁴

The best w2v–cbow and w2v–sg configurations are not significantly different from each other, but both are different from PPMI–thresh (p < 0.05). In a direct comparison with related work, our best result for the Reddy data set (Spearman ρ = .812, Pearson r = .814) improves upon the best correlation reported by Reddy, McCarthy, and Manandhar (2011) (ρ = .714), and by Salehi, Cook, and Baldwin (2015) (r = .796). For Farahmand, these results are comparable to those reported by Yazdani, Farahmand, and Henderson (2015) (BF₁ = .487), but our work adopts an unsupervised approach for compositionality prediction. For both FR-comp and PT-comp, the w2v models are outperformed by PPMI–thresh, whose predictions are significantly different from the predictions of other models (p < 0.05).

In short, these results suggest language-dependent trends for DSMs, by which w2v models perform better for the English data sets, and PPMI–thresh for French and Portuguese. While this may be due to the level of morphological inflection in these languages, it may also be due to differences in corpus size or to particular DSM parameters used in each case. In Section 7, we analyze the impact of individual DSM and corpus parameters to better understand this language dependency.

Table 4 reports the best configurations for the EN-comp, FR-comp, and PT-comp data sets. However, to determine whether the Spearman scores obtained are robust and generalizable, in this section we report evaluation using cross-validation. For each data set, we partition the 180 compounds into 5 folds of 36 compounds (f₁, f₂, …, f₅). Then, for each fold f_i, we exhaustively look for the best configuration (values of WindowSize, Dimension, and WordForm) for the union of the other folds (∪_j≠if_j), and predict the 36 compositionality scores for f_i using this configuration. The predicted scores for the 5 folds are then grouped into a single set of predictions, which is evaluated against the 180 human judgments.

The partition of compounds into folds is performed automatically, based on random shuffling.³⁵ To avoid relying on a single arbitrary fold partition, we run cross-validation 10 times, with different fold partitions each time. This process generates 10 Spearman correlations, for which we calculate the average value and a 95% confidence interval.

We have calculated cross-validation scores for a wide range of configurations, focusing on the following DSMs: PPMI–thresh, w2v–cbow, and w2v–sg. Figure 4 presents the average Spearman correlations of cross-validation experiments compared with the best results reported in the previous section, referred to as oracle. In the top left panel the x-axis indicates the DSMs for each language using the best oracle configuration, Figure 4(a). In the other panels, it indicates the best oracle configuration for a specific DSM and a fixed parameter for a given language. We present only a sample of the results for fixed parameters, as they are stable across languages. Results are presented in ascending order of oracle Spearman correlation. For each oracle datapoint, the associated average Spearman from cross-validation is presented along with the 95% confidence interval.

The Spearman correlations obtained through cross-validation are comparable to the ones obtained by the oracle. Moreover, the results are quite stable: increasingly better configurations of oracle tend to be correlated with increasingly better cross-validation scores. Indeed, the Pearson r correlation between the 9 oracle points and the 9 cross-validation points in the top-left panel is 0.969, attesting to the correlation between cross-validation and oracle scores.

For PT-comp, the confidence intervals are quite wide, meaning that prediction quality is sensitive to the choice of compounds used to estimate the best configurations. Probably a larger data set would be required to stabilize cross-validation results. Nonetheless, the other two data sets seem representative enough, so that the small confidence intervals show that, even if we fix the value of a given parameter (e.g., d = 750), the results using cross-validation are stable and very similar to the oracle.

The confidence intervals overlapping with oracle data points also indicate that most cross-validation results are not statistically different from the oracle. This suggests that the highest-Spearman oracle configurations could be trusted as reasonable approximations of the best configurations for other data sets collected for the same language constructed using similar guidelines.

As an additional test of the robustness of the results obtained, we calculated the performance of the best models obtained for one of the data sets (EN-comp), on a separate held-out data set (EN-comp_Ext). The latter contains 100 compounds balanced for compositionality, not included in EN-comp (that is, not used in any of the preceding experiments). The results obtained on EN-comp_Ext are shown in Table 5. They are comparable and mostly better than those for the oracle and for cross-validation. As the items are different in the two data sets, a direct comparison of the results is not possible, but the equivalent performances confirm the robustness of the models and configurations for compositionality prediction. Moreover, these results are obtained in an unsupervised manner, as the compositionality scores are not used to train any of the models. The scores are used only for comparative purposes for determining the impact of various factors in the ability of these DSMs to predict compositionality.

In this section, we analyze the influence of DSM parameters on compositionality prediction. We consider different window sizes (Section 7.1), numbers of vector dimensions (Section 7.2), types of corpus preprocessing (Section 7.3), and corpus sizes. For each parameter, we analyze all possible values of other parameters. In other words, we report the best results obtained by fixing a value and considering all possible configurations of other parameters. Results reported in this section use the pc_uniform function.

DSMs build the representation of every word based on the frequency of other words that appear in its context. Our hypothesis is that larger window sizes result in higher scores, as the additional data allows a better representation of word-level semantics. However, as some of these models adopt different weight decays for larger windows,³⁶ variation in their behavior related to window size is to be expected.

Contrary to our expectations, for the best models in each language, large windows did not lead to better compositionality prediction. Figure 5 shows the best results obtained for each window size.³⁷ For English, w2v is the best model, and its performance does not seem to depend much on the size of the window, but with a small trend for smaller sizes to be better. For French and Portuguese, PPMI–thresh is only the best model for the minimal window size, and there is a large gap in performance for PPMI–thresh as window size increases, such that for larger windows it is outperformed by other models. To assess which of these differences are statistically significant, we have performed Wilcoxon’s sign-rank test on the two highest Spearman values for each DSM in each language. All differences are statistically significant (p < 0.05), with the exception of PPMI–SVD.

The appropriate choice of window size has been shown to be task-specific (Lapesa and Evert 2017), and the results above suggest that, for compositionality prediction, it depends also on the DSM used. Overall, the trend is for smaller windows to lead to better compositionality prediction.

When creating corpus-derived vectors with a DSM, the question is whether additional dimensions can be informative in compositionality prediction. Our hypothesis is that the larger the number of dimensions, the more precise the representations, and the more accurate the compositionality prediction.

The results shown in Figure 6 for each of the comparable data sets confirm this trend in the case of the best DSMs: w2v and PPMI–thresh. Moreover, the effect of changing the vector dimensions for the best models seems to be consistent across these languages. The results for PPMI–SVD, lexvec, and glove are more varied, but they are never among the best models for compositionality prediction in any of the languages.³⁸ All differences between the two highest Spearman correlations are statistically significant (p < 0.05), with the exception of PPMI–SVD for FR-comp, according to Wilcoxon’s sign-rank test.

In related work, DSMs are constructed from corpora with various levels of preprocessing (Bullinaria and Levy 2012; Mikolov et al. 2013; Pennington, Socher, and Manning 2014; Kiela and Clark 2014; Levy, Goldberg, and Dagan 2015; Salle, Villavicencio, and Idiart 2016). In this work, we compare four levels: WordForm = surface⁺, surface, lemma_PoS and lemma, described in Section 5.1, corresponding to decreasing amounts of information. Testing different varieties of corpus preprocessing allows us to explore the trade-off between informational content and the statistical significance related to data sparsity for compositionality prediction.

Figure 7 presents the impact of different types of corpus preprocessing on the quality of compositionality prediction. In EN-comp, all differences between the two highest Spearman values for each DSM were significant, according to Wilcoxon’s sign-rank test, except for PPMI–thresh, whereas in FR-comp and PT-comp they were significant only for PPMI–TopK and lexvec. However, note that the top two results are often both obtained on representations based on lemmas. If we compare the highest lemma-based result with the highest surface-based result for the same DSM, we find a statistically significant difference in every single case (p < 0.05).

When considering the results themselves, although the results for English are heterogeneous, for French and Portuguese, the lemma-based representations consistently allow a better prediction of compositionality scores. This may be explained by the fact that these two languages are morphologically richer than English, and lemma-based representations reduce the sparsity in the data, allowing more information to be gathered from the same amount of data. Moreover, adding POS information (lemma_PoS vs. lemma) does not seem to bring consistent improvements that are statistically significant. This suggests that words that share the same lemma are semantically close enough that any gains from disambiguation are masked by the sparsity of a higher vocabulary size. Finally, the impact of stopword removal is also inconclusive (surface vs. surface⁺), considering the best models for each language.

If we assume that the bigger the corpus, the better the DSM, this could explain why the results for English are better than those for French and Portuguese, although it does not explain why Portuguese is behind French.³⁹ In this section, we examine the impact of corpus size on prediction quality by incrementally increasing the amount of data used to generate the DSMs while monitoring the Spearman correlation (ρ) with the human annotations. We use only the best DSMs for these languages, PPMI–thresh and w2v–sg, with the configurations that produced highest Spearman scores for each full corpus.

As expected, the results in Figure 8 show a smooth, roughly monotonic increase of the ρ values with corpus size, for PPMI–thresh and w2v–sg for each language and data set.⁴⁰ In all cases there is a clear saturation behavior, so that we can safely say that after one billion tokens, the quality of the predictions reaches a plateau and additional corpus fragments do not bring improvements. This suggests that differences in compositionality prediction performance for these languages cannot be totally explained by differences in corpus sizes.

Up to this point, the predicted compositionality scores for the compounds were calculated using a uniform function that assumes that each component contributes 50% to the meaning of the compound (pc_uniform). However, this might not accurately capture a faithful representation of compounds whose meaning is more semantically related to one of the components (e.g., crocodile tears, which is semantically closer to the head tears; and night owl, which is semantically closer to the modifier night). As this may have an impact on the success of compositionality prediction, in this section we evaluate how different compositionality prediction functions model these compounds. In particular, we proposed pc_maxsim, (Section 4) for dynamically determining weights that assign maximal similarity between the compound and each of its components. We have also proposed pc_geom, which favors idiomatic readings through the geometric mean of the similarities between a compound and its components. Our hypotheses are that pc_maxsim will be better correlated with human scores for compositional and partly compositional compounds, while pc_geom can better capture the semantics of idiomatic ones (Section 8.1).

First, to verify whether other prediction functions improve results obtained for the best pc_uniform configurations reported up to now, we have evaluated every strategy on all DSM configurations. Table 6 shows that the functions that combine both components (columns pc_uniform to pc_arith) generate better compositionality predictions than functions that ignore one of the individual components (columns pc_head and pc_mod). There is some variation among the combined scores, with the best score indicated in bold. Every best score is statistically different from all other scores in its row (p < 0.05). The results for pc_arith and pc_uniform are very similar, reflecting their similar formulations.⁴¹

Here we focus on the issue of adjusting β in the compositionally constructed vector; that is, we consider the use of pc_maxsim instead of pc_uniform. This score seems to be beneficial in the case of English (EN-comp), but not in the case of French or Portuguese. Table 7 presents the best pc_maxsim model for each data set, along with the average weights assigned to head and modifier for every compound in the data set. Before analyzing the results in Table 7, we have to verify whether the data sets are balanced for the influence of each component to the meaning of the whole, or if there is any bias towards heads/modifiers. The influence of the head, estimated as the average of hc_H/(hc_H + hc_M) over all compounds of a data set, is 0.50 for EN-comp, 0.52 for FR-comp, and 0.52 for PT-comp. This indicates that the data sets are balanced in terms of the influence of each component, and neither head nor modifier predominates as more compositional or idiomatic than the other.

As for the average β weights in pc_maxsim, while the weights that maximize compositionality are fairly similar for EN-comp, they strongly favor the head for both FR-comp and PT-comp. This may be explained by the fact that, for the latter, the modifiers are all adjectives, while EN-comp has mostly nouns as modifiers. Surprisingly, this seemingly more realistic weighting of the compound components for French and Portuguese does not reflect in better compositionality scores, and does not correspond to the average influence of modifiers in these data sets, estimated as 0.48 on average. One possible explanation could be that, in these cases, the adjectives may be contributing to some specific more idiomatic meaning that is not found in isolated occurrences of the adjective itself, such as FRbeau (lit. beautiful), which is used in the translation of most in-law family members, such as FRbeau-frère (lit. beautiful-brother ‘brother-in-law’). In the next section, we investigate which compounds are affected the most by these different scores.

We perform a cross-lingual analysis, grouping the hc_HM scores of the EN-comp, FR-comp, and PT-comp into a unique data set (henceforth ALL-comp), containing 540 compounds. Figure 9 presents the values of rank improvement for the best PPMI–thresh and w2v–sg configurations, ranked according to hc_HM (rk_human): compounds that are better predicted by pc_maxsim have positive rank movements (above the 0 line).⁴² The density of movement on either side of the 0 (no movement) line appears to be similar for both models with pc_maxsim performing as well as pc_uniform.

Figure 9 also marks the outlier compounds with the highest improvements (numbers from 1 to 8) and those with the lowest improvements (letters from A to H), and Table 8 shows their improvement scores. In the case of these outliers, the adjustment seems to be more beneficial to compositional compounds than to idiomatic cases. This is confirmed by a linear regression of the movement of the 8 +8 outliers as a function of the compositionality scores hc_HM, where we obtain a positive coefficient of r = 0.73 and r = 0.72 for PPMI–thresh and w2v–sg, respectively. There are more outlier compounds for Portuguese and French (particularly the former), suggesting that pc_maxsim has a stronger impact on those languages than on English. Moreover, some compounds had a similar improvement under both DSMs, with, for example, high improvement for PTcaixa forte literally box strong ‘safe’ and low improvement for PTcoração partido ‘broken heart’. In addition, pc_maxsim also affected some equivalent compounds in different languages, as in the case of PTcaixa forte and FRcoffre fort. Overall, pc_maxsim does not present a considerable impact on the predictions, obtaining an average improvement of $improvmaxsim¯$ = +0.41 across all compounds in ALL-comp.

Figure 10 shows the same analysis for f = geom, showing the improvement score of pc_geom over pc_uniform. We hypothesized that pc_geom should more accurately represent idiomatic compounds. From the previous sections, we know that pc_geom has lower performance than pc_uniform when used to estimate the compositionality of the entire data sets (cf. Table 6). This is confirmed by an average score of $improvgeom¯$ = −7.87. As in Figure 9, Figure 10 shows a random distribution of improvements. However, the outliers have the opposite pattern, indicating that large reclassifications due to pc_geom tend to favor idiomatic instead of compositional compounds. The linear regression of the movement of the outliers as a function of the compositionality scores results in r = −0.73 and r = −0.82 for PPMI–thresh and w2v–sg, respectively. These confirm our hypothesis for the behavior of pc_geom.

Table 9 lists the outlier compounds indicated in Figure 10 along with their improvement values. Here again, the majority of the outliers belong to PT-comp. Some of the compounds that were found as outliers in pc_maxsim re-appear as outliers for pc_geom with inverted polarity in the improvement score, such as the ranks predicted by PPMI–thresh for PTprato feito literally plate made ‘blue-plate special’ (improv_maxsim = +58, improv_geom = −234) and by w2v–sg for FRbras droit literally arm right ‘assistant’ (improv_maxsim = −68, improv_geom = +228). This suggests that, as future work, we should consider combining both approaches into a single prediction that decides which score to use for each compound as a function of pc_uniform.

In the previous sections, we examined the performance of the compositionality prediction framework in terms of the correlation between automatic predictions and human judgments across languages. We now investigate the relation between predicted scores and other variables that may have an impact on results, such as familiarity (Section 9.1) and conventionalization (Section 9.2). We also compare the predicted compositionality scores with trends previously found in human scores (Section 9.3). The experiments focus on the ALL-comp data set, which groups the predicted scores from the best configurations on EN-comp, FR-comp, and PT-comp (cf. Table 4).

Results from Section 3.2.2 show that the familiarity of compounds measured as frequency in large corpora is associated with the compositionality scores assigned by humans. We would like to know whether this correlation also holds true to system predictions: Are the most frequent compounds being predicted as more compositional? As expected, the rank correlation between frequency and pc_uniform shows medium to strong correlation (see Table 10, column ρ[pc, freq]), though the level of correlation is somewhat DSM-dependent, are in line with the correlation observed between frequency and human scores, and with the high correlation between predicted and human scores.

Another hypothesis we test is whether frequent compounds are easier to model. A first intuition would be that this hypothesis is true, as a higher number of occurrences is associated with a larger amount of data, from which more representative vectors can be built. To test this hypothesis, we define a compound’s difficulty as the difference between the predicted score and the normalized human score, diff = |pc − (hc_HM/5)|, where high values indicate a compound whose compositionality is harder to predict.⁴³

We found a weak (though statistically significant) correlation between frequency and difficulty for some of the DSMs (Table 10, column ρ[diff, freq]). They are mostly positive, indicating that frequency is correlated with difficulty, which is a surprising result, as it implies that the compositionality of rarer compounds was mildly easier to predict for these systems, disproving the hypothesis above. These results either point to an overall lack of correlation between frequency and difficulty, or indicate mild DSM-specific behavior, which should be investigated in further research.

PMI is not only a well-known estimator of the level of conventionalization of a multiword expression (Church and Hanks 1990; Evert 2004; Farahmand, Smith, and Nivre 2015), but it is also used in some DSMs as a way to estimate the strength of association between target and context words. To assess if what our models are implicitly measuring is the association between the component words of a compound rather than compositionality, we now examine the correlation between compositionality scores and PMI.

We found only a weak but statistically significant correlation between predicted compositionality and PMI (Table 10, column ρ[pc, PMI]), which suggests that these DSMs preserve some information regarding conventionalization. However, given that no significant correlation between PMI and human compositionality scores was found (Section 3.2.2) and as DSM predictions are strongly correlated to human predictions, these results indicate that our models capture more than conventionalization. They may also be a feature of this particular set of compounds, as even the compositional cases are also conventional to some extent (e.g., white/?yellow wine). Therefore, further investigation of possible links between idiomaticity and conventionalization is needed.

We also calculated the correlation between PMI and the human–prediction difference (diff), to determine if DSMs build less precise vectors for less conventionalized compounds (approximated as those with lower PMI). However, no statistically significant correlation was found for most DSMs (Table 10, column ρ[diff, PMI]).

Spearman correlation assesses the performance of a given configuration by providing a single numerical value. This facilitates the comparison between configurations, but it hides the internal distribution of predictions. By splitting the data sets into ranges, we obtain a more fine-grained view of possible patterns linked to compositionality prediction.

To determine if the compounds that humans agree more on are also more accurately predicted, we divided ALL-comp into three equally sized subsets, according to the standard deviation among human annotators (low, mid-range, and high values of standard deviation, σ_HM). As high standard deviation indicates disagreement among annotators, it may be an indicator of the difficulty of the annotation. Table 11 presents the best DSMs, according to Spearman’s ρ evaluated separately on each of the subsets. Indeed, for the compounds that had low σ_HM, the Spearman values were the highest (between 0.73 and 0.75), while for those with high σ_HM, the Spearman correlation with human judgments was the lowest (between 0.35 and 0.43). These results confirm that higher scores are achieved for the compounds for which humans agree more, and suggest that part of the difficulty of this task for automatic systems is also related to difficulties for humans.

To determine if compositional compounds would be more precisely predicted than idiomatic compounds, we divide ALL-comp into three equally sized subsets based on the level of human compositionality scores (low, mid-range, and high values of hc_HM). Table 11 presents the correlation obtained on each subset for the best configuration of each DSM. The more idiomatic compounds have the lowest Spearman values (from 0.16 to 0.29) while the more compositional have the highest ones (from 0.32 to 0.37). These results confirm that the predictions are better for compositional than for idiomatic compounds. Moreover, these scores are much lower than those from the full data set (from 0.63 to 0.66), suggesting that it may be harder to make fine-grained distinctions (e.g., between two compositional compounds like access road and subway system) than to make inter-range distinctions (e.g., between idiomatic and compositional compounds like ivory tower and access road). However, further investigation would be needed to verify this hypothesis.

We proposed a framework for compositionality prediction of multiword expressions, focusing on nominal compounds and using DSMs for meaning representation. We investigated how accurately DSMs capture idiomaticity compared to human judgments and examined the impact of several variables in the accuracy of the predictions. In order to determine how language dependent the results are, we evaluated the compositionality prediction framework in English, French, and Portuguese, using data sets containing human-rated compositionality scores, some of which were specifically constructed as part of this work.⁴⁴ Using these data sets, we presented a large-scale evaluation involving 228 DSMs for each language, and we evaluated more than 9,000 framework configurations to determine the impact of possible factors that may influence compositionality prediction.

Our experiments confirmed that our framework is able to capture idiomaticity accurately, obtaining a strong correlation with human judgments for all three languages. Comparing the performance of different DSMs, the particular choice of DSM had a noticeable impact on the results, with differences over 0.10 Spearman ρ points for all languages. For the comparable data sets (EN-comp, FR-comp, and PT-comp), the best models were w2v and PPMI–thresh.⁴⁵ Results differed according to language: although for English w2v were the best models, for French and Portuguese, PPMI–thresh outperformed the other models. Moreover, the results for the three languages varied considerably, with those for English outperforming by 0.10 and 0.20 Spearman ρ points those for French and Portuguese, respectively. The latter are morphologically richer than the former, and a closer examination of the type of preprocessing adopted for best results reveals that both languages benefit from less sparse representations resulting from lemmatization and stopword removal, while for English no preprocessing was particularly beneficial.

Although corpus size is often assumed to play a fundamental role in the quality of DSMs, so that the bigger the corpus the better the results, prediction quality stabilized at around one billion tokens for all languages. This may reflect the point where the minimum frequency was reached for producing reliable representations for all compounds in these data sets, even the rare cases, and larger corpora did not lead to better predictions. Moreover, for the best models in each language, DSMs with more dimensions resulted in more accurate predictions confirming our hypothesis. We also found a trend for small window sizes leading to better results for the best models in all three languages, contrary to our hypothesis. A typically good configuration used vectors of 750 dimensions built from minimal context windows of one word to each side of the target.

DSMs were also robust regarding the choice of compositionality prediction function, with a uniform combination of the head and modifier producing the best results for all languages. Other functions like pc_maxsim and pc_geom, which modify these scores to account for different contributions of each component, produced at best similar results.

A deeper analysis of the predicted compositionality scores revealed that, similarly to human-rated scores, familiarity measured as frequency was positively correlated with predicted compositionality. In the case of conventionalization measured as PMI, no correlation was found with human-rated scores and only a mild correlation was found with some predicted scores, suggesting that our models capture more than compound conventionalization, as they have a strong agreement with human scores. Intra-compound standard deviation on human scores was also found to be related to predicted scores, indicating that DSMs have difficulties on those compounds that humans also found difficult. Moreover, predictions were found to be more accurate for compositional compounds.

Although there are many questions that still need to be solved regarding compositionality, we believe that the results presented here advance significantly its understanding and computational modeling. Furthermore, the proposed framework opens important avenues of research that are ready to be pursued. First, the role of morphological inflection could be clarified by extending this investigation to even more inflected languages, such as Turkish. Moreover, other categories of MWEs such as verb +noun expressions should be evaluated to determine the interplay between compositionality prediction and syntactic flexibility of MWEs. The ultimate test would be to use predicted compositionality scores in downstream applications and tasks involving some degree of semantic processing, ranging from MWE identification to parsing, and word-sense disambiguation. In particular, it would be interesting to predict compositionality in context, in order to distinguish idiomatic from literal usages in sentences.

composition function

is a function that takes as input a sequence of vectors v(w_i) to v(w_j) and outputs a compositionally constructed vector v_⊕(w_i … w_j) representing the compositional meaning of the sequence, where ⊕ indicates the function used to compose the vectors. Example: $vβ(w1,w2)=βv(w1)||v(w1)||+(1−β)v(w2)||v(w2)||$⁠. 1, 23

compositionality prediction configuration

is the combination of a particular DSM configuration with a given compositionality prediction function, fully specifying how a predicted compositionality score is calculated for a given word sequence w_i … w_j. 1

compositionality prediction framework

is the set of all possible compositionality prediction configurations available. 1, 22

compositionality prediction function

is a function that takes as input corpus-based vectors for a sequence of words v(w_i … w_j) and for the individual words composing that sequence v(w_i) … v(w_j), and outputs a predicted compositionality score, usually proportional to the similarity between the corpus-based vector v(w_i … w_j) and a compositionally constructed vector v(w_i) to v(w_j) derived from v(w_i) … v(w_j) using a composition function. Example: maxsim. 1

compositionally constructed vector

is the output of a composition function, that is, a vector v_⊕(w_i … w_j) derived from the individual words’ corpus-derived vectors v(w_i) to v(w_j). 1, 23

corpus-derived vector

is the output of a DSM for a given element w_i of the vocabulary V, that is, a corpus-derived D-dimensional real-numbered vector v(w_i) that represents the meaning of w_i. A corpus-derived vector of a word sequence v(w_i … w_j) is built by treating it as a single token in the corpus. 1, 22

distributional semantic model (DSM)

is a function that takes as input a vocabulary V and a (large) corpus, and outputs a corpus-derived vector v(w_i) for each element w_i of V based on the distributional profile of w_i’s occurrences in the corpus. The vocabulary V can be automatically derived from the input corpus. Example: w2v–cbow. 1

DSM configuration

is a set of DSM parameters and their values, fully specifying how corpus-derived vectors are built from a given corpus. Example: w2v–cbow using lemma_PoS.W₈.d₇₅₀. 1, 28

DSM parameter

is a variable in a DSM whose value influences the way corpus-derived vectors are built from a corpus. Example: WordForm. 1, 28

human compositionality score (hc)

is a real value representing the compositionality assigned by human annotators to a word sequence w_i … w_j. The correlation between predicted compositionality (pc) and human compositionality (hc) scores is used to evaluate a compositionality prediction configuration. When subscripted, indicates the question used to obtain the score. Example: hc_H. 1, 17, 29

predicted compositionality score (pc)

is the output of a compositionality prediction function, that is, a real value representing the predicted compositionality of a word sequence w_i … w_j. The correlation between predicted compositionality (pc) and human compositionality (hc) scores is used to evaluate a compositionality prediction configuration. When subscripted, indicates the compositionality prediction function used to obtain the score. Example: pc_uniform. 1, 29

The number of possible DSM configurations grows exponentially with the number of internal variables in a DSM, forestalling the possibility of an exhaustive search for every possible parameter. We have evaluated in this article the set of variables that are most often manually tuned in the literature, but a reasonable question would be whether these results can be further improved through the modification of some other often-ignored model-specific parameters. We thus perform some sanity checks through a local search of such parameters around the highest-Spearman configuration of each DSM.

Some of the DSMs in consideration on this paper are iterative: they re-read and reprocess the same corpus multiple times. For those DSMs, we present the results of running their best configuration, but using a higher number of iterations. This higher number of iterations is inspired by the models found in parts of the literature, where, for example, the number of glove iterations can be as high as 50 (Salle, Villavicencio, and Idiart 2016) or even 100 (Pennington, Socher, and Manning 2014). The intuition is that most models will lose some information (due to their probabilistic sampling), which could be regained at the cost of a higher number of iterations.

Table 12 presents a comparison between the baseline ρ for 15 iterations and the ρ obtained when 100 iterations are performed. For all DSMs, we see that the increase in the number of iterations does not improve the quality of the vectors, with the relatively small number of 15 iterations yielding better results. This may suggest that a small number of iterations can already sample enough distributional information, with further iterations accruing additional noise from low-frequency words. The extra number of iterations could also be responsible for overfitting of the DSM to represent particularities of the corpus, which would reduce the quality of the underlying vectors. Given the extra cost of running more iterations,⁴⁶ we refrained from building further models with as many iterations in the rest of the article.

Minimum-count thresholds are often neglected in the literature, where a default configuration of 0, 1, or 5 being presumably used by most authors. An exception to this trend is the threshold of 100 occurrences used by Levy, Goldberg, and Dagan (2015), whose toolkit we use in PPMI–SVD. No explicit justification has been found for this higher word-count threshold. A reasonable hypothesis would be that higher thresholds improve the quality of the data, as it filters rare words more aggressively.

Table 13 presents the result from the highest-Spearman configurations along with the results for an identical configuration with a higher occurrence threshold of 50.⁴⁷ The results unanimously agree that a higher threshold does not contribute to the removal of any extra noise. In particular, for PPMI–SVD, it seems to discard enough useful information to considerably reduce the quality of the compositionality prediction measure. The results strongly contradict the default configuration used for PPMI–SVD, suggesting that a lower word-count threshold might yield better results for this task.

For many models, the best window size found was either WindowSize = 1 + 1 or WindowSize = 4 + 4 (see Section 7.1). It is possible that a higher score could be obtained by a configuration in between. While a full exhaustive search would be the ideal solution, an initial approximation of the best 2 + 2 configuration could be obtained by running the experiments on the highest Spearman configurations, with the window size replaced by 2 + 2.

Results shown in Table 14 for a window size of 2 + 2 are consistently worse than the base model, indicating that the optimal configuration is likely the one that was obtained with window size of 1 + 1 or 4 + 4. This is further confirmed by the fact that most DSMs had the best configuration with window size of 1 + 1 or 8 + 8, with few cases of 4 + 4 as best model, which suggests that the quality of most configurations in the space of models is either monotonically increasing or decreasing with regards to these window sizes, favoring thus the configurations with more extreme WindowSize parameters.

As seen in Section 7.2, some DSMs obtain better results when moving from 250 to 500 dimensions, and this trend continues when moving to 750 dimensions. This behavior is notably stronger for PPMI–thresh, which suggests that an even higher number of dimensions could have better predictive power.

Table 15 presents the result of running PPMI–thresh for increasing values of of the Dimension parameter. The baseline configuration (indicated as ^★ in Table 15) was the highest-scoring configuration found in Section 7.2: lemma_PoS.W₁.d₇₅₀ for PT-comp and FR-comp, and surface.W₈.d₇₅₀ for Reddy. As seen in Section 7.2, results for 250 and 500 dimensions have lower scores than the results for 750 dimensions. Results for 1,000 dimensions were mixed: they are slightly worse for FR-comp and EN-comp, and slightly better for PT-comp. Increasing the number of dimensions generates models that are progressively worse. These results suggest that the maximum vector quality is achieved between 750 and 1,000 dimensions.

The word vectors generated by the glove and w2v models have some level of non-determinism caused by random initialization and random sampling techniques. A reasonable concern would be whether the results presented for different parameter variations are close enough to the scores obtained by an average model. To assess the variability of these models, we evaluated three different runs of every DSM configuration (the original execution ρ₁, used elsewhere in this article, along with two other executions ρ₂ and ρ₃) for glove, w2v–cbow, and w2v–sg. We then calculate the average ρ_avg of these three executions for every model.

Table 16 reports the highest-Spearman configurations of ρ_avg for the Reddy and EN-comp data sets. When comparing ρ_avg to the results of the original execution ρ₁, we see that the variability in the different executions of the same configuration is minimal. This is further confirmed by the low sample standard deviation⁴⁸ obtained from the scores of the three executions. Given the high stability of these models, results in the rest of the article were calculated and reported as ρ₁ for all data sets.

Along with the verification of parameters, we also evaluate whether data set variations could yield better results. In particular, we consider the use of filtering techniques, which are used in the literature as a method of guaranteeing data set quality. As per Roller, Schulte im Walde, and Scheible (2013), we consider two strategies of data removal: (1) removing individual outlier compositionality judgments through z-score filtering; and (2) removing all annotations from outlier human judges. A compositionality judgment is considered an outlier if it stands at more than z standard deviations away from the mean; a human judge is deemed an outlier if its Spearman correlation to the average of the others ρ_oth is lower than a given threshold R.⁴⁹ These methods allow us to remove accidentally erroneous annotations, as well as annotators whose response deviated too much from the mean (in particular, spammers and non-native speakers).

Table 17 presents the evaluation of raw and filtered data sets regarding two quality measures: The average of the standard deviations for all NCs (⁠$σ¯$⁠); and the proportion of NCs in the data set whose standard deviation is higher than 1.5 (P_σ>1.5), as per Reddy, McCarthy, and Manandhar (2011). The results suggest that filtering techniques can improve the overall quality of the data sets, as seen in the reduction of the proportion of NCs with high standard deviation, as well as in the reduction of the average standard deviation itself. We additionally present the data retention rate (DRR), which is the proportion of NCs that remained in the data set after filtering. While the DRR does indicate a reduction in the amount of data, this reduction may be considered acceptable in light of the improvement suggested by the quality measures.

On a more detailed analysis, we have verified that the improvement in these quality measures is heavily tied to the use of z-score filtering, with similar results obtained when it is considered alone. The application of R-filtering by itself, on the other hand, did not show any noticeable improvement in the quality measures for reasonable amounts of DRR. This is the opposite from what was found by Roller, Schulte im Walde, and Scheible (2013) on their German data set, where only R-filtering was found to improve results under these quality measures. We present our findings in more detail in Ramisch, Cordeiro, and Villavicencio (2016).

We then consider whether filtering can have an impact on the performance of predicted compositionality scores. For each of the 228 model configurations that were constructed for each language, we launched an evaluation on the filtered EN-comp₉₀, FR-comp, and PT-comp data sets (using z-score filtering only, as it was responsible for most of the improvement in quality measures). Overall, no improvement was observed in the results of the prediction (values of Spearman ρ) when we compare raw and filtered data sets. Looking more specifically at the best configurations for each DSM (see Table 18), we can see that most results do not significantly change when the evaluation is performed on the raw or filtered data sets. This suggests that the amount of judgments collected for each compound greatly offsets any irregularity caused by outliers, making the use of filtering techniques superfluous.

The questionnaire was structured in five subtasks, presented to the annotators through these instructions:

• 1. 

  Read the compound itself.

• 2. 

  Read 3 sentences containing the compound.

• 3. 

  Provide 2 to 3 synonym expressions for the target compound seen in the sentences, preferably involving one of the words in the compound. We ask annotators to prioritize short expressions, with 1 to 3 words each, and to try to include the MWE components in their reply (eliciting a paraphrase).

• 4. 

  Using a Likert scale from 0 to 5, judge how much of the meaning of the compound comes from the modifier and the head separately. Figure 11 shows an example for the judgment of the head.

• 5. 

  Using a Likert scale from 0 to 5, judge how much of the meaning of the compound comes from its components.

We require answers in an even-numbered scale (there are 6 possibilities between 0 and 5), as otherwise the participants could be biased toward the middle score. In order to help participants visualize the meaning of their reply, whenever their mouse hovers over a particular score, we present a guiding tooltip, as can be seen in Figure 11.

The order of subtasks has also been taken into account. During a pilot test, we found that presenting the multiple-choice questions (subtasks 4–5) before asking for synonyms (subtask 3) yielded lower agreement, as users were often less self-consistent in the multiple-choice questions (e.g., replying “non-compositional” for subtask 4 but “compositional” for subtask 5), even if they carefully selected their synonyms in response to subtask 3.

The request for synonyms before the multiple-choice questions prompts the participants to focus on the meaning of the compound. These synonyms can then also be taken into account when considering the semantic contribution of each element of the compound—we leave this for future work.

We present below the 90 nominal compounds in EN-comp₉₀ and the 100 nominal compounds in EN-comp_Ext, along with their human-rated compositionality scores. We refer to Reddy, McCarthy, and Manandhar (2011) for the other 90 compounds belonging to Reddy which, together with the former two sets, represent 280 nominal compounds in total.