Aligning Human and Computational Coherence Evaluations

Topic modeling is an important tool in the analysis and exploration of text corpora in terms of their salient topics (Blei, Ng, and Jordan 2003). To evaluate the effectiveness of topic models, the preponderance of topic modeling literature relies on automated coherence metrics. A key benefit is convenience, allowing researchers to sidestep expensive and time-consuming user studies. The basis for this reliance is the assumption that the coherence metrics correlate with human judgment (Mimno et al. 2011a; Lau, Newman, and Baldwin 2014; Röder, Both, and Hinneburg 2015).

The presumed correlation with human judgment should not be taken for granted. There are recent studies that challenge the assumption. Doogan and Buntine (2021) highlight the inconsistencies of automated coherence metrics via correlation analysis within each metric. Hoyle et al. (2021) claim some disagreement between human judgment and automated coherence metrics.

We postulate that the reasons behind such a mixed picture could be the differences in the topic samples and underlying corpora-dependent statistics, resulting in localized “biases” that affect the conclusions reached by respective studies. Given the importance of these metrics, we seek to conduct an extended analysis of automated coherence metrics on a larger scale than anything previously attempted. This study includes orders of magnitude greater than the number of topics typically analyzed, covering three large corpora, utilizing a comprehensive user study with extensive labels, across most of the widely used metrics.

There is a strong case for quantity. Given a vocabulary, a combinatorially large number of possible topics exist. If each topic representation is a vector of its scores on different metrics, the resulting curse of dimensionality (Bellman and Kalaba 1959) necessitates a larger sample size. We claim that evaluating thousands of topics might not be sufficient, and a larger sample size is required to approximate a diverse distribution, where sampled topics are representative of the corpus and the metrics. We surmise that the previous practice of using topic models to generate topics could introduce a biased result in the analysis. Topic models vary in performance, with a lengthy list of models compiled by Hoyle et al. (2021). There is also emerging debate on the performance between traditional and neural topic models (Doogan and Buntine 2021). For instance, Hoyle et al. (2022) find evidence that neural topic models are more inconsistent than traditional topic models, producing more variant topic sets across different runs. The stability of a topic in traditional topic models across different runs has been shown to correlate to its quality (Xing and Paul 2018). Because different topic models have different characteristics and performances, we propose generating candidate topic representations independent of topic models to evaluate the usability of coherence metrics.

We have three contributions in our previous work (Lim and Lauw 2023b). First, we begin by analyzing the inter-metric correlations (see Section 7). We propose a novel approach to sample representations of “topics” for the purpose of evaluating automated coherence metrics (see Section 4.1). Compared with prior works, we sample these representations free from topic model bias, and in a meaningful and diverse manner. With evaluations on three large corpora, we reaffirm that selected metrics do not contradict each other. We also highlight the underestimated effects (see Section 7.1) of ϵ = 1e−12, used in the calculation of Normalized Pointwise Mutual Information (NPMI) (see Section 3.1) to avoid undefined logarithm zero.

Second, we extend our analysis to investigate inter-corpus correlations (see Section 7.2). We examine the understated differences of corpora statistics on the metrics by comparing the correlations across corpora. While such correlations exist to some degree, these metrics strongly depend on the evaluation corpus. Thus, any expectation that these metrics would correlate uniformly with human judgment on all possible corpora may be misplaced.

Finally, pivotal to any interpretability research, we design and conduct a user study, which is the keystone of our work (see Section 5). Compared with prior work, its design is more complex as we seek to benchmark human judgment at a finer granularity across different random user study groups (see Section 5.1). We analyze the user study results via a few novel proxy measures, revealing that human judgment is nuanced and varies between individuals. The metric correlation to human judgment is corpus-dependent, with the average participant likely to be attuned to the generic corpora (see Section 6). Accompanying the work, we also released our toolkit to enable convenient coherence evaluation of topic representations and to advance interpretability research. Our implementation and releasable resources can be found here.¹

In this article, motivated by the results reported in our previous work, we conduct additional investigations into our previous findings to account for the effect of possible confounding factors. We also take this opportunity to include extra details and explanations that further elucidate the findings. Our previous work focuses more on automated coherence metrics, using human judgment as its benchmark. In contrast, this article reverses the direction of analysis, placing a heavier interest in human judgment while using automated coherence metrics as a tool for human judgment analysis.

We notice one of the automated coherence metrics, C_UMass (Mimno et al. 2011b; Röder, Both, and Hinneburg 2015) (defined in Section 3.1), exhibits an unexpectedly lower correlation with human judgment, compared with other shortlisted automated coherence metrics. As C_UMass remains a popular metric, with recent studies (such as Meng et al. 2022; Zhang et al. 2022) using it as an evaluation criterion, we believe that further investigation is required to ascertain its efficacy. As the selected metrics are count-based and reliant on the corpus, different hyperparameters controlling window size wsz and minimum frequency mf of word pairs might affect the scores. Deviating from recommended hyperparameter settings, we evaluate the correlation of automated metrics against human judgment at different hyperparameter settings, showing the importance of setting a low wsz and mf (see Section 6.3). Additionally, some metrics might be order-dependent, with the ordering of vocabulary in the topic representation affecting the scores. C_UMass is one of these order-dependent metrics. We conduct additional analysis to determine the effect of ordering in these metrics. Our findings from this analysis show that the effect of ordering might be overstated and unnecessary (see Section 6.4). Using the newfound information, we update the results of our previous work and further strengthen its findings (see Section 7).

Among the eight user study groups, one of the user study groups has low to no correlation with automated coherence metrics. We find a correlation between user study groups’ inter-rater reliability and their self-reported English proficiency (see Section 6.1). Included in their responses, we collected our study participants’ self-rated English proficiency, which we can use as a proxy measure of their true English proficiency. Via statistical analyses, we show that their reported English proficiency affects the correlation between automated coherence metrics and human judgment (see Section 6.2). When we exclude the outlier group, there is a significant increase in correlation between automated coherence metrics and human judgment. Beyond reporting correlation scores, we also conduct novel analyses to examine the human perception of coherence benchmark against computed coherence. In addition to quantifying surface-level differences (see Section 8.1), we analyze individual participants’ responses to derive insights into their perception of coherence, building a framework to estimate subjectivity between individuals based on corpus statistics (see Section 8.2). With the framework, we investigate the possible strategies our study participants used to complete the task (see Section 8.3) and the effect of our study participants’ subject matter experience (see Section 8.5).

There are many approaches for topic modeling (Blei, Ng, and Jordan 2003), from non-neural based (Hoffman, Blei, and Bach 2010; Zhao, Tan, and Xu 2017), to many other neural-based methods, via autoencoders (Kingma and Welling 2014) such as Miao, Yu, and Blunsom (2016); Srivastava and Sutton (2017), Dieng, Ruiz, and Blei (2020), Zhang and Lauw (2020), and Bianchi et al. (2021), via graph neural networks (Yang et al. 2020; Shen et al. 2021; Zhang and Lauw 2022), and hierarchical methods (Meng et al. 2020). A common factor among these works is the usage of automated coherence metrics in their evaluation of topic representations produced. We select several popular metrics for evaluation as listed in Section 3. Coherent topic representations allow us to interpret the topic embeddings and reduces the opacity of models.

Topic models are applicable in numerous downstream tasks and play a supporting role in augmenting other models. Lau, Baldwin, and Cohn (2017) use topic modeling in neural language modeling. Wang et al. (2019) and Xu et al. (2023) proposed different topic-guided variational autoencoders for text generation. For abstractive document summarization, Wang et al. (2020) generate topic embeddings to improve the performance of transformer-based models. On stance detection (Mohammad et al. 2016), Arakelyan, Arora, and Augenstein (2023) use topic sampling and contrastive learning to achieve state-of-the-art results. On multimodal relation extraction (Zheng et al. 2021; see Section 2.3), Wu et al. (2023) use multimodal topic features in their framework to reduce reliance on internal data and exploit external data.

A knowledge graph is defined as a set of Entities E, Relations R, and Facts F, with a fact being represented as a triple of {head, relation, tail}, where head ∈ E and tail ∈ E (Ji et al.2022). Compared with knowledge graphs, the associations within a topic representation are implicit rather than explicit. For example, WordNet (Miller 1995) is a knowledge graph that organizes words using their semantic relations (such as synonyms). Between vocabulary pairs, they may have multiple explicit relations. In topic modeling, when we consider their relation using corpus statistics, the single implicit relation assumes multiple explicit relations to varying degrees depending on its context. From the structured knowledge graphs, we can learn knowledge graph embeddings (Wang et al. 2017), which can be used in downstream tasks such as knowledge graph completion tasks to identify missing facts (Bordes et al. 2013). In contrast, we obtain topic embeddings from corpora that do not have an explicit graph structure but seek to interpret it via topic representations with the quality of its implicit graph structure measured on automated coherence metrics.

The field of Mechanistic Interpretability attempts to reverse the computations within models in pursuit of explainability (Olah et al. 2020), with efforts directed towards transformer-based models (Elhage et al. 2021; Geva et al. 2022; Cunningham et al. 2023; Bricken et al. 2023). We believe there is potential to apply topic modeling methodologies and evaluation in this area. In Lim and Lauw (2023a), we proposed an approach that assigns multiple topic representations to single neurons residing in the multilayer perceptron layers in decoder-only transformer models. Automated coherence metrics enable the construction and evaluation of millions of topic representations, optimized for human coherence, from millions of neurons.

Mimno et al. (2011a) utilize expert annotators to independently label 148 topic representations, using another ten expert annotators to evaluate topic representations via intruder word detection tasks. Röder, Both, and Hinneburg (2015) benchmark topics against different permutations of metrics with an evaluation set containing 900 topics with human ratings aggregated from prior studies (Aletras and Stevenson 2013; Lau, Newman, and Baldwin 2014; Rosner et al. 2014). In Hoyle et al. (2021), a minimum of 15 crowd workers were employed in simple rating and word intrusion tasks evaluating 40 topic-model-generated (Griffiths and Steyvers 2004; Burkhardt and Kramer 2019; Dieng, Ruiz, and Blei 2020) and 16 synthetic random topics. In Doogan and Buntine (2021), their largest user study required four subject matter experts to create 3,120 labels across 390 topics generated via topic models (Blei, Ng, and Jordan 2003; Zhao et al. 2017). In comparison, our study has large quantities in both topic representations and study participants, annotating 800 unbiasedly sampled topic representations split between 40 user study participants with at least an undergraduate level of education, generating 180K word pairs. Each question has 45 possible combinations of word pairs, with binary labels denoting coherence relations between word pairs. Our automated experiments deal with hundreds of thousands of unique topics.

Garimella, Banea, and Mihalcea (2023) use word models with topic-based features to analyze the relationship between language and demographics. Similar to our work, they also utilized context windows in text. Beyond the demographic level, we investigate the relationship with language at the individual level, specific to the perception of coherence in topic representations. Bo, Fu, and Lim (2023) extensively analyzed the relationship between English language proficiency and academic performance, showing that proficiency scores strongly predict academic results. Their study is particularly relevant as our user study participants are from a similar population in terms of culture, geography, and education level.

In this section, we define the selected automated coherence metrics that utilize corpus statistics and describe the corpora we use to obtain the word probabilities.

For indirect confirmation measure $m~$⁠, instead of directly using word probabilities, it uses m to create a vector of features v (Aletras and Stevenson 2013) that represent a word w from the topic t it belongs to, distorted by hyperparameter γ Equation (11). We will evaluate γ at 1 and 2, denoted by superscript $CVγ$⁠.² Subscript ∉ denotes the absence of epsilon. In this article, ∉-variants are $CV,∉γ$ and C_NPMI,∉.

We use word co-occurrences statistics obtained from three large corpora: ArXiv,³ PubMed,⁴ and Wiki.⁵Table 1 numerically describes the corpora. We further elaborate on each corpus:

1. ArXiv. We use the ArXiv abstracts dataset, considering each abstract as a document. These abstracts mainly constitute research work related to non-medical science disciplines.

2. PubMed. We use the PubMed Central Open Access Subset that contains journal articles and preprints related to medical research and information. We consider each article body as a document, removing citations within it.

3. Wiki. We use the English-Wikipedia dump of August 2022 processed using Attardi (2015). We consider the content of the article as a document. To evaluate for correctness of our toolkit, we use the popular benchmark Palmetto (Röder, Both, and Hinneburg 2015), which uses a subset of Wikipedia 2011.

For each corpus, we apply processing steps suggested in Hoyle et al. (2021), retaining up to 40K frequently occurring words.⁶ Additionally, we generate a lemmatized (denoted with the suffix -lemma) and unlemmatized variant (original) for further analysis. Appendix B contains additional information on the shared vocabulary between corpora.

Intuitively, if two different metrics are to correlate with human judgment, we would expect correlations between the scores of these metrics. However, it is claimed in Doogan and Buntine (2021) that these metrics do not correlate well. There are a few tested methods to generate topics: from topic models (Aletras and Stevenson 2013; Lau, Newman, and Baldwin 2014), beam search optimized on coherence (Rosner et al. 2014), and random sampling of words (Hoyle et al. 2021). Considering only optimized and random (incoherent) topic representations will result in a skewed distribution. In contrast, we seek to mine topic representations that emulate a balanced distribution for a meaningful comparison. Furthermore, there is also a desire for uniqueness among topics, which avoids repetition and is representative of the corpus. We propose a new non-topic modeling approach to sample topics to evaluate these metrics.

The problem of mining topics of k words can be mapped to the classical k-clique listing problem (Chiba and Nishizeki 1985; Danisch, Balalau, and Sozio 2018). To generate a meaningful distribution of topic representations, we map the corpus-level information as a graph, treating each word from its vocabulary set V as a vertex. Each word will share an edge with every other word. m_nlr is used to determine the value of the edges between two vertices as its normalized range is intuitive, allowing easy identification of the value ranges for the generation of sub-graphs. Using m_lc and m_f, on the other hand, increases the sampling’s complexity as they are order-dependent. As a result, their sub-graph contains bi-directional edges. Conducting sampling runs using any m, not only m_nlr, might introduce sampling bias, favoring certain topic representations, which our approach seeks to mitigate. Figure 1 illustrates an overview of our approach.

The initial graph will be a complete graph of |V | vertices, where its k-sized sub-graph is a topic representation of k words. Combinatorially, there are $Ck∣V∣$ possible unique topics. It is practically infeasible and unnecessary to list all k-cliques. For a more tractable approach, we modify the routine from Yuan et al. (2022) (Algorithm 1, 2) to include the following properties:

1. Sub-graphs of varying quality. This routine seeks to generate smaller graphs from the original complete graph to cover the spectrum of topic quality. Edges on the graph are eliminated conditionally via their value, and the remaining edges and connected vertices constitute the new sub-graph, enabling the creation of three different kinds of sub-graphs: pos where edge-values are above a given lower-bound, mid where edge-values are between threshold values, and neg where edges are below an upper-bound (see Appendix B).

2. Topic extraction. Inspired by Perozzi, Al-Rfou, and Skiena (2014), instead of iterating through all the neighboring nodes or searching for the next best node, we randomly select its unexplored neighbor node, which shares an edge with all explored nodes, to explore. We extract the explored k-path as our sampled topic.

3. Topic uniqueness. To attain a variety of topics, we remove all edges in a mined clique, making it impossible to sample a similar topic representation from the same sub-graph. Figure 2 illustrates this feature.

4. Balanced distribution of topics. For a given corpus, we introduce additional common topics sampled from different corpora, which differ in word distribution. We refer to this segment of external topic representations as ext. Lastly, random segment comprises groups of random words, included to represent a distribution absent from the other sampled segments. Table 2 shows the result from this mining approach. The final distribution would thus be more balanced, comprising topics of varying scores along the spectrum.

In Algorithm 2, the pre-processing step, PRE-CORE, to reduce complexity remained unchanged. This step is skippable when the graph is large and dense, such as during neg sub-graphs generation. Our modification in Algorithm 1 and Algorithm 2 introduces randomness via permutations and early stopping when Algorithm 1 finds a k-clique and a desired number of k-cliques found in Algorithm 2. The sub-graph reduction routine is in Algorithm 1.

From Table 3, we have three different thresholds for pos. As the lower bound increases for pos, it becomes increasingly difficult to sample cliques that meet the criteria, with some sampling runs returning no samples. Sampling for mid is easier, in terms of speed and quantity sampled, compared to pos as more edges fit the relaxed criteria. For ArXiv and Wiki, we use bounds (−0.05,0.15) while for PubMed, we use bounds (0,0.15). We use a stricter bound for PubMed as relaxing the bounds will result in a sample similar to a random distribution. While neg seems easier to sample, compared with pos or mid, it is not always guaranteed to be easier. When we set neg’s upper bound to −0.4 for PubMed, sampling cliques became challenging. Selected hyperparameter thresholds should return feasible samples while allowing us to avoid emulating a random distribution, which we can trivially obtain.

Since our approach (see Section 4.1) does not produce P(w|t), we can locally optimize the word positions within a topic to obtain the best possible score for position-sensitive metrics C_UMass and C_P. This additional step ensures fairness in our evaluation of order-dependent metrics. We use subscript s to denote alphabetical order and subscript o to denote optimized positions. In our correlation evaluations, we use topic representation t of size k = 10, with words w arranged based on P(w|t) in descending order.

Given a set of k words as a topic, we seek to optimize the position-dependent score (see Algorithm 3). This problem is reducible to a weighted activity selection problem, akin to finding a max-weight independent set in an interval graph, and solvable in polynomial time (Bar-Noy et al. 2001). Consider a word w at the j^th position, with index j starting from 0, its j incoming edges and k −j + 1 outgoing edges representing its rank in the order. The incoming edges indicate the subset of preceding words to w, while the outgoing edges indicate the subset of the remaining succeeding words. We define an activity’s position in the ordering using its preceding and succeeding activities. Each activity has an equal interval with its weight determined by the difference of outgoing and incoming edges to all other words scored via m. We can transform the activities into an interval graph, with $∣Cjl∣·∣Cl−j+1l∣$ combinatorial number of possible instances for each word per interval in the schedule. Our transformation will result in an interval graph of k disjoint graphs. While the number of activities might seem combinatorially explosive, selecting the first activity only involves k activities. Each selection will prune multiple branches of possible orderings, resulting in k −j choices after j selection. Hence, we are only required to select the best activity within each disjoint graph conditioned on availability (for unselected words).

Previous studies measure human judgment through simple evaluation tasks such as rating the coherence of a topic on a few-point ordinal scale (Mimno et al. 2011a; Aletras and Stevenson 2013), identifying the intruder word introduced into the topic representation (Chang et al. 2009), or both (Lau, Newman, and Baldwin 2014; Hoyle et al. 2021). For word intrusion, outlier detection signals the cohesiveness of the topic representation, which is similar to rating topics on an ordinal scale. However, for both tasks, qualitative gaps might exist. In word intrusion, study participants are restricted to just one outlier per topic, assuming perfect coding, resulting in an exponential drop in scoring. For simple ratings, topic representations of differing qualities might get the same score as its ordinal scale is rigid in its score increments.

Additionally, while the decisions between human annotators might be equivalent, it is not evident if their thought processes are similar. The key reason for this line of inquiry stems from the observation that everyone is different in some aspects, such as knowledge, culture, and experiences. Assuming our prior beliefs influence our understanding of words, what and how we perceive similarity and coherence might differ from person to person. For these reasons, we are motivated to design a user study that combines both word intrusion and topic rating tasks but is differentiated at a finer granularity such that we can quantify the decision-making process. We instruct study participants to cluster word groups that indicate coherent and outlier word groups. We then examine the relationships between automated coherence metrics and different proxy tasks derived from the user study.

User studies are typically not replicated precisely. While the methodology may be similar, the participants and questions asked are likely to differ. We designed our study to encompass multiple groups to investigate replicability. By examining the variation resulting from the difference in participants and question sets, we hope to ensure some degree of confidence in the study.

For our study, we recruit eight user study groups U = {U₁,…, U₈}, with five study participants per group, totaling 40 study participants. Most of our study participants recruited have completed or are pursuing an undergraduate program from tertiary institutions where English is the primary language of instruction, requiring sufficient English competency for admissions (specified in Bo, Fu, and Lim 2023). As such, we expect our applicants to have adequate competency in English, sufficient to understand the tasks, and to recognize most, if not all, of the words presented. Before distributing the study, participants are allocated randomly to only one study group and disallowed to attempt another set of questions.

For each study group, we prepared eight unique question sets T = {T₁,…, T₈}, each containing 100 10-word topic representations, T_i = {t_{1, i},…, t_{100, i}} and t = {w_{0, j, i},…, w_{9, j, i}}. For each participant U_{u, i} ∈ U_i, we present each t_{j, i} ∈ T_i individually and sorted alphabetically. We ask participants to cluster words in t_{j, i} that they deem similar to form coherent word groups g, where their response R_{u, j, i} to t_{j, i} is a set of unique g. To limit the complexity of the given task, we constrain each word to belong to one coherent word group. Additionally, when a study participant determines that the word is unrelated to the given topic representation, it forms its group of one. We format the response as a Likert matrix grouping coherent words (see Figure 3), mandating a response for each word w_{k, j, i} ∈ t_{j, i}. Refer to Table 4 for the table of notations and Appendix A for actual instructions given to the study participants.

We construct an initial pool of 1,000 topics. We randomly sampled 400 common topic representations from Wiki, ArXiv, and PubMed for parity between corpora. To represent non-scientific topics, we randomly sampled 200 topics from Wiki that do not appear in ArXiv/PubMed. For ArXiv/PubMed exclusive topics, we randomly sampled 200 topic representations each, with these topics also appearing in Wiki. Since most coherence evaluations occur in the positive domain of C_NPMI, we conduct sampling in a 7:1:1:1 ratio of pos/mid/neg/random segments of the corpus, seeking to emulate a uniform score distribution. To account for word familiarity, we select lemmatized topics with words found in 20K most frequently used words, based on Corpus of Contemporary American English. The 14K most frequently used words, including proper nouns, will account for 99% of running words (Nation 2006; Beglar 2010), where 99 out of 100 consecutive words in any text will belong to the 14K set. In our manual review of the topic representations, we do not find any unrecognizable words. However, if there were difficult words, the study allowed participants to look up the definitions of words.⁷ We pair each user study group with a set of randomly sampled 100 topics from the pool without replacement. For topic representations not found in ArXiv or PubMed, we exclude them during the evaluation of those corpora.

The interpretation of P₃ is the inverse of P₂. While these group-wise measures might seem similar, they measure different nuances of human-annotated data. P₁ evaluates the sizes of multi-word groups, weighted towards larger groups. P₂ only accounts for the maximum word group size, which ignores the properties of the other remaining group. P₃ disregards group sizes to a certain extent and includes single-word “outlier” groups. We evaluate these measures’ correlation against various C(t_{j, i}).

From our previous work (Lim and Lauw 2023b), scores from the proxy tasks correlate to automated coherence metrics. In Figure 4, the mean results of study groups on the three proxy tasks indicate correlations between human judgment and some automated coherence metrics (see Lim and Lauw [2023b] for individual group results). These results include user study group U₃, which exhibits outlier results, prompting us to investigate the possible reason for the observations (see Section 5). Since most of our study participants have some science-related background, we are surprised by ArXiv’s lower correlation scores relative to Wiki in each proxy task. Additionally, ArXiv’s correlation scores have a higher variance when compared to PubMed and Wiki. Lastly, we also observed a weak correlation between C_Umass and the proxy tasks. Corpus hyperparameters could be a confounding factor, and we examine its effect (see Section 6.3).

Many factors will affect the variation for IRR (Belur et al. 2021). For our user study, we attempted to mitigate some of these factors. Considering the framing and education factors, we include a short introductory primer and some example questions. Study participants will view these materials before starting the user study (Appendix A). To alleviate fatigue, we allowed the study participants up to a week to work on the task, pausing and continuing at their own pace. We were not concerned about the learning effect, as topic representations of various themes exist in our question sets. Additionally, the correctness of the task is subjective to their personal preference. As our objective is to poll for their beliefs, with many possible valid answers, reviewing their responses to enforce consistency between study participants is unnecessary. While there are correlations between proxy measures and coherence metrics, it does not imply that individual study participants have similar responses. IRR allows us to quantify the difference in responses across different study groups.

To measure IRR, we use Krippendorf’s α (Krippendorff 2011), defining pair-wise rater similarity with two different metrics to measure common answers between raters: Jaccard distance (Jaccard 1912) and Measuring Agreement on Set-valued Items (MASI) distance (Passonneau 2006). We treat each w_{k, j, i} ∈ t_{j, i} as a multi-classification question, comprising other words (in t_{j, i}) and “not related” as categories, producing Boolean vector representations. When given two sets A and B, Jaccard distance calculates their intersection over their union, (A ∩ B)/(A ∪ B), while MASI distance gives fixed scores: 0 for disjoint sets, 1/3 for intersecting sets where neither sets are a subset of the other, 2/3 when a set is a subset of another, and 1 for identical sets. Comparing the two metrics, they are equivalent in the cases of disjoint and identical sets. However, MASI rewards subsets but penalizes other non-empty intersections.

From Table 5, except U₅ and U₇, the ordering of the study groups is similar for both metrics. U₁ scored the highest for both distance metrics, while U₃ scored the lowest. α (Jaccard) is higher than α (MASI) across the different user study groups. A completely random study response will have an α (Jaccard) of 0.12 and α (MASI) of 0.092, significantly less than the study’s respective $α-$⁠, giving us some confidence about the reliability of the responses.

There might be two plausible reasons for the difference in the α metrics observed across the different user study groups. When two responses have a non-empty intersection with neither being a subset of the other, the results suggest that most intersections of such nature are more than 1/3 of the union of sets. Alternatively, when either response is a subset of the other, the results also suggest that the subset is likelier to have more than 2/3 of the elements in the superset. Considering the exponential number of possible combinations for each topic response, these values show that our study participants have varied responses but with some overlapping similarities.

Even though we recruit from an English-speaking population, there could still be some variation in writing and reading abilities. Before the study, we asked prospective applicants to rate their English proficiency using a sliding scale from 0 to 100. Self-ratings of English proficiency can estimate the study participants’ English competency. Xu (1991) found that self-rated English Proficiency was the most significant predictor of the perceived level of academic difficulty for reading tasks. In a study conducted by Takahashi (2009), students who rated themselves higher also scored better academically. Our participants would have had some form of standardized English test before being given admission into a tertiary education program. However, for privacy reasons, we refrained from asking participants for their academic grades or test scores for English. As anonymity is guaranteed, reporting self-assessed English proficiency is a reasonable way to assess their opinion of their abilities relative to their peers, giving us a viable proxy estimate of their true English proficiency. We reiterate that self-reporting a low score does not mean they are incompetent in English.

Figure 5 shows the distribution of participants’ self-rated English proficiency. The distribution is left-skewed, with many participants rating themselves proficient in English. From Figure 6, out of all the user groups, U₃ seems to buck the trend, reporting lower to no correlation with the proxy tasks. Its correlation scores are two to three times the standard deviation less than the mean, a clear outlier. No participants from U₃ self-rate their proficiency score above 80, whereas the rest of the groups contain multiple participants highly confident in their mastery of English.

Using the mean self-rated proficiency scores (see Table 6), we conduct a simple linear regression analysis against their respective IRR scores and the correlation scores of C_NPMI from the three different proxy tasks. We select C_NPMI as it has moderate to good correlation with other automated coherence metrics on Wiki (Lim and Lauw 2023b). For the regression analysis, we set the bias parameter to 0 as we expect user groups with zero English proficiency to have low IRR and no correlation to the various proxy tasks. The mean self-rated English proficiency of the group (X) is assumed to be normally distributed, with the Shapiro-Wilk test (Shapiro and Wilk 1965) giving a p-value of 81.61%. We analyze the relationship of X with different dependent variables (Y): IRR and the different correlation scores from the various proxy tasks. From Table 7, the adjusted R² of the various analyses is very high at 98%, implying a strong correlation between X and Y with most of the variance explained. The different linear regression tests report very low p-values, confirming that the effect from X is significant, suggesting goodness of fit. For validation, we examine the residual normality and variance homogeneity. We can assume that the residuals are normally distributed, with the Shapiro-Wilk test giving a p-value >5%. We can similarly assume that the variance is homogeneous, with the White test (White 1980) giving p-value >5%.

The linear regression analysis (Section 6.2.2) shows that group mean self-rated English proficiency (X) has a significant effect on the correlation scores. As X is assumed to have a normal distribution, it suggests that our user study simulates drawing correlation studies from an English-competent population with this effect significant among such a population. Some of these study groups with higher self-rated English proficiency also contain participants who rated their proficiency low. Fortunately, averaging responses may have reduced outlier effects from individual study participants.

There are two corpus hyperparameters: window size and minimum frequency. Window size influences the word pair counts as it determines which neighboring words to count as occurring together. Minimum frequency is a count threshold that excludes word pairs with a lower word pair count. Intuitively, window size determines the locality of the context, that is, sentence, paragraph, or document scope. Minimum frequency regulates and excludes rare word pairs. As these hyperparameters may affect the results obtained, it is imperative that we verify their effect.

Using our collected responses, we can map human judgment against various automated coherence scores computed on Wiki at different corpus hyperparameters. From Figure 7, for all metrics, correlation scores improve or remain consistent as minimum frequency decreases to 0 and window size decreases to 10. We exclude U₃ in the hyperparameter search and we obtained similar hyperparameter search results in ArXiv and PubMed (see Appendix B).

When we decrease the minimum frequency, scores of rare-occurring word pairs increase, as we had previously treated them as non-occurring, increasing the overall score of the topic representation. We focus our observations on ArXiv benchmarks (see Table 8), as its corpus size is smaller than Wiki, with many of the topic representations independently sampled for the user study from Wiki. An increase in correlation scores indicates a likelihood that the previously omitted scores are informational. We see an improvement in correlation scores across most ArXiv-based automated coherence metrics, compared to those based on Wiki. Since the topic representations selected for the user study comprise common vocabulary between the corpora, this may positively influence the informativeness of the observed rare-occurring word pairs. We use the Wilcoxon signed rank test (Wilcoxon 1945) to test the significance of the difference between pairs of correlation scores between different automated coherence metrics and human judgment from the same study group.

Even though there might be different optimal settings for the various automated coherence metrics, the difference is marginal and within error bars. The only exception is C_UMass,o where there is a gap between the recommended settings and its optimal settings, possibly explaining its weakness in correlation to human judgment and other automated coherence metrics. The advantage of setting a minimum frequency is to prevent rare word pairs from skewing the scores. However, this omitted information may negatively affect the score. Setting this hyperparameter is about deciding which scenario is likelier. For window sizes, word pair occurrences in close proximity suggest greater relevance. With larger document sizes, having large window sizes might result in associating irrelevant word pairs and penalizes frequently occurring word pairs within the document. For larger corpora, our results demonstrate that it is unnecessary to set a large minimum frequency or window size.

By the definitions of C_P and C_UMass (Section 3.1), different ordering of words might produce different scores. Our approach (Section 4.1) samples unordered topic representations, which we then optimize for ordering (Section 4.3). However, our user study presented these topic representations alphabetically, necessitating further investigation.

Table 9a and 9b show the respective mean correlation scores of optimally ordered and sorted variants of C_P and C_UMass across different corpora and proxy tasks. Excluding U₃, we compare the pairs of related correlation scores across variants, using the Wilcoxon signed rank test, to test whether the optimally ordered variant (subscript o) has a stronger correlation score than the alphabetically sorted variant (subscript s). Most tests report a high p-value, unable to reject the null hypothesis that the s-variant has a stronger or similar correlation score to the o-variant. Even for those tests that are statistically significant, it appears that the difference in magnitude is marginal.

From our results, it seems that ordering of words for automated coherence metrics does not significantly affect the correlation with human judgment. Presenting the topic representation alphabetically in our user study is essential from a user experience perspective, allowing our participants to locate specific words easily when entering their response. The marginal difference might imply that the benefits of ordering only applies to a minority of topic representations.

In this section, we identified and analyze four different factors: English proficiency, window size, minimum frequency, and ordering of words. Figure 8 visualizes the individual group correlation scores against the various automated coherence scores on Wiki on the newly selected corpus hyperparameters. Excluding U₃, the mean correlation score increases with a reduction in variation. For future experiments, we standardize corpus hyperparameter settings, unless otherwise specified, to window size =10 and minimum frequency =0, as all of our selected automated coherence metrics seem to work well in those settings. We recommend setting a low minimum frequency of [0,100] and a window size =10 for large corpora.

Previously, we examined the relation between human and metric. In this section, we examine the behavior of automated coherence metrics, in relation to each other and across corpora.

In our previous work (Lim and Lauw 2023b), we determined that inter-metric correlations exist between automated coherence metrics, identifying how including or excluding ϵ = 1e−12 affects these correlations. Considering the effects of hyperparameters, we re-evaluate the correlation (Pearson’s r⁹) between different automated metrics measured on Wiki (see Figure 9), PubMed, and ArXiv (see Appendix B). We expect a high positive correlation score between metrics if they are both purportedly measuring for coherence.

Our first inter-metric analysis (see Figure 9a) on the better ϵ variant uses the entire sampled set of topic representations. We observe that C_NPMI and C_P correlate well with other metrics. However, ∉-variant metrics do not correlate strongly with C_{UMass, o}. The difference in correlation scores between C_NPMI and C_UMass and C_NPMI,∉ and C_UMass suggests that the weaker correlation is due to the removal of ϵ. When ϵ is removed from C_NPMI, edges between word pairs with no occurrence will have a C_NPMI score of 0. This indifference to non-occurring word pairs decreases the correlation score. This explanation is applicable to C_V,∉ as it uses C_NPMI,∉. The removal of ϵ is important for C_V, as non-occurring word pairs have similar negative NPMI vectors, resulting in a high C_V score that is contradicting.

Our second inter-metric analysis (see Figure 9) uses a subset of coherent topic representations with C_NPMI > 0. We observe that C_NPMI and C_P have similar weaker correlation with C_UMass compared to ∉-variants. Building on the previous analysis, we can attribute the better correlation scores of C_NPMI and C_P to C_UMass on incoherent topic representations, suggesting that these metrics agree on incoherence. In both analyses (see Table 9), for position-dependent metrics C_P and C_UMass, we observe that their correlation between their position-optimized (subscript o) and sorted (subscript s) variants have very high correlation. This high correlation suggests that word positioning in the topic representation might be unimportant for a larger corpus (see Section 6.4). The C_V, C_NPMI, C_NPMI,∉, and C_P of both analyses have a similarly strong correlation, albeit stronger with coherent topics.

Comparing our updated analyses to Lim and Lauw (2023b), we see an improvement of inter-metric correlation scores between C_UMass and the other metrics on Wiki, ArXiv, and PubMed. We attribute this improvement to the standardization of corpus hyperparameters: window size and minimum frequency. Choosing C_NPMI, C_V,∉, or C_P for coherence evaluation does not seem to matter as they are highly correlated with each other and have similar correlation scores with human judgment. C_UMass has a weaker correlation to human judgment and moderately correlates with other metrics. However, it could be meaningful to use C_UMass in conjunction with another metric after accounting for its sensitivity to corpus hyperparameters.

A natural extension after inter-metric comparison is to compare metrics measured on different corpora. It is a common expectation that research works would use multiple corpora, with the differences between corpora quantified superficially (such as in Section 3.2). Between corpora, we can quantify their differences at a topical level, using common topics evaluated on automated coherence metrics. If the corpora are thematically similar, we expect a high correlation. Again, using the new corpus hyperparameters, we update our findings.

We posit that variance in scores measured on different corpora lowers correlation scores due to the missing themes within the shared vocabulary space in either corpus. Using the common topics from the paired corpora, we conduct a correlation analysis on the scores measured on each corpus per metric. Different from Lim and Lauw (2023b), where we only include topics evaluated on both corpus pairs, we seek out all topics present in either corpus that can exist in the other. As such, we increased the number of topic representations evaluated in each corpus pair. Compared with Lim and Lauw (2023b), Table 10 shows a wider range of correlations between each corpus pair on different automated coherence metrics. While these correlation scores are positive, these correlations do not imply identical statistics in various corpora. The control analysis in Table 11 a shows a strong correlation score. Compared with the scores found in Table 10, the lower correlation scores confirm the presence of topical differences between the various corpora.

While we know how lemmatization affects topic modeling (Schofield and Mimno 2016), its effect on evaluation is unclear. We carried out two additional ablations simulating lemmatizing topics for evaluation. We evaluate these topics on pairs of corpora differing in lemmatization while originating from the same documents. These corpus pairs would be thematically identical in knowledge and organization while superficially different in corpus statistics, with the lemmatized vocabulary having larger counts when compared to the unlemmatized counterpart. For the first ablation, we shortlist topics containing at least one unlemmatized word, where if lemmatized, the word exists in the same unlemmatized corpus. We compare correlations of the original set of topic representations and its corresponding lemmatized set, with their scores measured on the same unlemmatized corpus. In the second ablation, we use the same shortlisting process but with lemmatized topics evaluated on the lemmatized corpus statistics.

For the first ablation, their scores exhibit a strong correlation (see Table 11b), suggesting that the difference between lemmatized and unlemmatized sets of topics is marginal. For the second ablation, our results (see Table 11c) show a strong correlation across the various metrics, implying that it is viable to post-process topics for evaluation.

When we compare corpora to human judgment, we indirectly quantify the difference in knowledge representation and organization. Similarly, comparing between corpora achieves the same goal. In Section 7.2.1, the low to moderate correlation between corpus pairs using the same automated coherence metrics quantifies the thematic differences between corpora. This result implies that automated coherence metrics calculated on some corpora might not be able to approximate human judgment. To evaluate for coherence, we can choose between conducting a user study or using a different reference corpus, already benchmarked to human judgment. For large corpora, the impact of text lemmatization is minimal, possibly from the law of averages, where common words have similar occurrence frequencies in both lemmatized and unlemmatized corpora. Both lemmatized and unlemmatized corpus statistics report similar correlations of automated coherence to human judgment.

Our experiments suggest that the Wiki corpus can emulate the average human mental model of coherence. We can analogize the human mental model as a corpus, each individual having a unique mental model. Previously, we benchmarked human-corpus and inter-corpus coherence relations. In this section, we seek to investigate human-human coherence relations. Most user studies, including ours, analyze human responses in a group setting, obtaining an average representation of the group. Regrettably, we claim that in doing so, we are removing a defining feature of humans, which is their individuality. The question of who is important as different individuals have different knowledge and perceptions of coherence. Our IRR (see Section 6.1) shows that the responses are different but still correlate well with Wiki (Figure 8). Given that it is possible to compare the knowledge representation between corpora (see Section 7.2), we extend the analysis to individual study participants’ perception of coherence. For these analyses, we reverse the roles, using Wiki as the benchmark frame of reference and human judgment as the subject of interest.

The user study design allows us to investigate our study participants’ responses, where we attempt to gain additional insights into their thought processes and how they decide which words within the topic representation are coherent.

To get a sense of the difference in responses, we select four quantifiable metrics and visualize their results in Figure 10.

1. Self-rated English proficiency (see Table B14, appeared in Figure 5, reappears in Figure B14). Study participants rated their English Proficiency on a slider, ranging from 0 to 100. We use this as a proxy measure for their true English proficiency, and their group-mean self-ratings have a strong correlation with IRR and C_NPMI on the three proxy tasks (see Table 7).

2. Total number of outliers (see Table B14). The propensity of deciding which words are outliers can indicate a difference in the thought processes of our study participants. The decision of deciding that the word is not coherent with any word groups is influenced by which linguistic content they consume. There are possibly many factors influencing the choice of linguistic content, such as but not limited to education, culture, interests, and so forth.

3. Total number of coherent groups selected (see Table B14). Similar to measuring the total number of outliers, this metric is an alternative difference measure. The definition of what is coherent might differ between individuals. Assuming similar outlier counts, some may have stricter criteria for coherence and thus have many smaller coherent groups. Others may have a loose interpretation of coherence, seeking global themes and building fewer but larger coherent groups.

4. Word-edge score agreement within coherent groups (see Table B14). We measure the percentage of words in our study participants’ chosen coherent groups that share the group with the word’s highest-scoring C_NPMI paired word, quantifying how frequently our user study participants group frequently occurring word pairs together. Anchoring the statistics on Wiki as a reference point, we can observe the differences between individual study participants.

From the visualization in Figure 10, within each study group, the overall responses of its individual study participants are very different. Additionally, since the agreement will be affected by the change in window sizes, we focus our examination on two groups, U₁ and U₃ (Figure 11), as the remaining user study groups are similar to U₁. We observe a trend where word-edge score agreement at larger window sizes (70, 110, bd) and lower window sizes (10, 40) are very similar. Since the difference is marginal, it suggests that the grouping of best word pairs is consistent. However, from our previous results in our hyperparameter search (see Figure 7), we observe that changing the window size does have a noticeable change in correlation scores, specifically bd, where the difference in its scores are widest when compared to other window sizes. As this measurement only accounts for a portion of possible word pair relations, we have to extend the analysis to include word group relations and to account for outliers.

In our previous work (Lim and Lauw 2023b), we did a correlation analysis on word pair scores. However, measuring individual correlations might not be meaningful due to the binary mode of choice as the user study design enforces tie-breaking. Moreover, the topic representations consist of ten words, which may provide additional context for the study participants to work with. Nevertheless, these word pair scores may provide additional information on how study participants decide which word is an outlier or belongs to a group.

We chose C_NPMI as it correlates well with both ϵ and ∉ metrics (see Section 7). From the sets of selected word pair edges, we derive two means, $ν-u,imin$ and $ν-u,imax$⁠. Considering that $ν-u,imax>ν-u,imin$⁠, we define mean ambiguity gap $[ν-u,imin,ν-u,imax]$ to map out a range of scored word pair edges where there is some level of ambiguity, indicating the degree of disagreement between the individual participant and corpus statistic. For word pair edges that lie outside the gaps, study participants align with Wiki on the relatedness within the word pairs. Below the gap, study participants agree with Wiki that the word pairs are unrelated, while above the gap, the word pairs are coherent.

From the results in Table 12, study group U₃ reported the widest group mean ambiguity gap. Intuitively, a wider gap might imply more uncertainty, explaining U₃’s lower correlation with automated coherence metrics. However, when we examine other groups, there are also individual study participants with equal or higher differences in the mean gap. We also examined the correlations of these measurements with the self-rated English proficiency of individual study participants and found that $ν-u,imax$ significantly correlates, averaging r = 0.55 across the various window sizes. We also observe that the different study participants have a wide range of scores for $νu,imin$ (see Appendix Table B11 and Figure 12), even within the same group, suggesting differences in opinions on grouping coherent words.

When deciding which words to group, one has to consider many possible combinations as there are 5¹⁰ possible answers using the Likert matrix format. Because our participants cannot process all possibilities, a realistic approach to the task will consist of entity-to-entity comparisons. These comparisons can be word-to-word, word-to-group, or group-to-group, with group comparisons reducing the task’s overall complexity. It is also likely that these comparisons are done linearly and will result in some local optimum, where no further changes will result in a better response where its difference is detectable by the participant. Since we have already established that automated coherence scores correlate to human judgment at topic representations of size 10, the observations should be extendable to groups of smaller sizes. The ambiguity gap can quantify the preferences of each individual study participant with respect to the corpus statistics. Excluding U₃, these other study groups have participants with high and low mean gap differences, evidence that there are outlier opinions in the group. Since these groups correlate better with automated coherence metrics, these outlier opinions might be productive within the group as they introduce a statistical gradation that helps in the coherence evaluation of word pairs.

We further analyze the results in a word-to-group setting using swaps inspired by the classical 2-opt algorithm (Croes 1958). $ν-u,imax$ serves as an estimate for the respective study participant’s coherence tipping point, an intuition that helps them decide to include or exclude words. If their responses are consistent, we expect a low percentage of swaps that produce an observable difference from their perspective, suggesting 2-optimality in their response.

A swap consists of transferring a word from one group to another. After the swap, we score both groups, using C_NPMI for this analysis, and determine whether the sum of the change in scores for both groups is noticeable, where it exceeds a threshold personalized to each study participant. We use $ν-u,imax$ as the threshold to decide if the benefit is detectable to study participant u in study group j. For a given pair of groups g₁ and g₂, we conduct swaps from g₁ to g₂, each resulting in $g1^$ and $g2^$ pair. We examine four different kinds of swaps:

1. Cluster-to-cluster. We find coherent group set I_{u, j, i}, where |I_{u, j, i}|≥ 2, that belongs to the same response to a topic representation R_{u, j, i} for topic representation t_{j, i}. We consider all 2-permutations group-wise in I_{u, j, i}. We expect |g₁| + |g₂| number of swaps for each group pair. We consider the scenario $(C(g1^)+C(g2^))−(C(g1)+C(g2))>ν-u,imax$ as an improvement in finding a better locally optimal solution within a swap. if $∣g1^∣=1$⁠, $C(g1^)=0$⁠. This swap tests which group is the word better off in.

2. Outlier-to-cluster. We find outlier group set O_{u, j, i} belonging to response R_{u, j, i} for topic representation t_{j, i}. We select combinations of g₁ from O_{u, j, i} and g₂ from I_{u, j, i}. Since |g₁| = 1, there is only one swap between each pair. After the swap, $g1^=∅$⁠, we set $C(g1^)=0$⁠. Since $∣g1^∣=1$⁠, its $C(g1^)=0$ as well. Hence, we consider the scenario $C(g2^)−C(g2)>ν-u,imax$ as an improvement. This swap tests if we can obtain bigger and better coherent topic groups by including outliers.

3. Cluster-to-outlier. We conduct swaps of each g₁ from I_{u, j, i} with g₂ = ∅. For each g₁, there will be |g₁| number of swaps. As $∣g2^∣=1$⁠, its $C(g2^)=0$⁠. If $∣g1^∣=0$⁠, its $C(g1^)=0$⁠. We consider the scenario $C(g1^)−C(g1)>ν-u,imax$ as an improvement. This swap tests if we can obtain a smaller but more coherent topic by excluding words as outliers.

4. Outlier-to-outlier. As |g₁| = 1 and |g₂| = 1, let w_g be the word from g. We conduct swaps between permutations of O_{u, j, i}, where |O_{u, j, i}|≥ 2, resulting in |O_{u, j, i}|² number of swaps. As the swap combines two outlier words, the change is equivalent to its edge score $C(g2^)>ν-u,imax$⁠. This swap tests if we can get better 2-sized coherent groups from outliers. Among all the different swaps, this swap is the easiest to show improvement as it only compares the edge.

From the results in Figure 13, most swaps do not result in a better overall state. Since $ν-u,imax$ is the threshold, larger $ν-u,imax$ are expected to have fewer improving swaps. We benchmark 15 random sampling runs at three different $ν-u,imax$ of 0.10, 0.15, and 0.20. If we treat the corpus statistics as the oracle, the low rate of better swaps in the random group, serving as baselines, suggests some difficulty in the task. Comparing the study participants to their corresponding random baseline with the next higher $ν-u,imax$⁠, we find that the majority of the participants have a lower rate of improvement in outlier-to-cluster and outlier-to-outlier swaps. Whereas for cluster-to-cluster and cluster-to-outlier swaps, fewer participants have a lower rate of improvement than the random baseline.

We do not expect study participants to have perfect responses as there are many possible swaps. A low rate of improving swaps suggests that most study participants reached some decision local optima on certain kinds of swaps during their interpretation of the topic representations presented. Naturally, since the user study task is on grouping words, a low rate of improvement in outlier-to-cluster and outlier-to-outlier confirms that our participants’ primary focus is on assigning words to a group. In contrast, a low rate of improvement in cluster-to-cluster and cluster-to-outlier for some participants either suggests that they compared clusters and improved on their initial response or had coherent initial word clusters. Whereas for other participants, the higher rate on certain swaps suggests that they might not have considered them. The analysis of local optimality shows the difference in answering style, reinforcing the case for investigating individual study participants.

From our user study, $⌞ν^min⌟$⁠, $⌜ν^min⌝$⁠, $⌞ν^max⌟$⁠, and $⌜ν^max⌝$ are 0.005, 0.085, 0.109, and 0.152, respectively. Considering two topic representations with a slight qualitative difference, quantifiable through some metric, we may be unable to discern between the two topics. If we could not perceive the minor difference, then perhaps that difference might not be as meaningful. An ordinal grading system might be able to highlight the qualitative difference, but determining such a system is challenging. An alternative approach might involve capping the extremes. Consider two topics that have a noticeable qualitative difference, with both scoring well above $⌜ν^max⌝$ or well below $⌞ν^min⌟$⁠. In the case of clearly coherent topics, it is unlikely that we will consider the lower-scoring topic as incoherent. To ensure robustness, we can examine multiple different caps and floors. The efficacy of using grades to score topic representations is an area for future investigation.

An important factor that influences one’s decision will be their expertise and experience. Deciding whether to recruit subject matter experts is pivotal and might affect the user study results. User studies on specific domains necessitate recruiting related experts. In this section, we investigate the effect of individual expertise on our user study.

It is challenging to judge expertise in Wikipedia as it encompasses a plethora of subjects. However, we can easily classify relevant expertise on domains commonly found in ArXiv. From our study participants, we identify individuals who we believe have experience with these domains. Our shortlisting criteria require these individuals to be graduates with working research experience or pursuing postgraduate studies in relevant scientific domains. We use public LinkedIn information to ascertain their experiences at the point of the study. From our pool of 40 study participants, we identified 12 potential experts. We conduct three different analyses to analyze their impact on our user study. To determine if the experts identified responded differently, we analyze their ambiguity gaps using ArXiv corpus statistics, comparing their results to non-experts’. As different user study groups have different numbers of experts, we identified the expert-dominant group U₁ and compared their responses to other groups using their responses’ correlation to ArXiv corpus statistics. Lastly, since there are varying levels of expertise even among experts, we examine the experts in U₁ as a case study.

From Figure 14, of the seven study groups with experts, we have two experts having the widest ambiguity gap within their respective study group, implying that they have the greatest disagreement with the ArXiv corpus statistic. The mean ambiguity gap for experts is 0.141, and the mean ambiguity gap for non-experts is 0.149. A Mann-Whitney U test (Mann and Whitney 1947), with a null hypothesis where the ambiguity gap of experts is more than or equal to that of non-experts, returning a p-value of 0.25, and we are unable to reject the null hypothesis.