A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice

Classification evaluation is ubiquitous: We have a system that predicts some classes of interest (aka classifier) and want to assess its prediction skill. We study a widespread and seemingly clear-cut setup of multi-class evaluation, where we compare a classifier’s predictions against reference labels in two steps. First, we construct a confusion matrix that has a designated dimension for each possible prediction/label combination. Second, an aggregate statistic, which we denote as metric, summarizes the confusion matrix as a single number.

Already from this it would follow that ‘the perfect’ metric can’t exist, since important information is bound to get lost when reducing the confusion matrix to a single dimension. Still, we require a metric to rank and select classifiers, and thus it should characterize a classifier’s ‘skill’ or ‘performance’ as well as possible. But exactly what is ‘performance’ and how should we measure it? Such questions do not seem to arise (as much) in other ‘performance’ measurements that are known to humans. For example, a marathon’s result derives from a clear and broadly accepted criterion (time over distance) that can be measured with validated instruments (a clock). However, in machine learning, criterion and instrument are often less clear and lie entangled in the term ‘metric’.

Since metric selection can influence which system we consider better or worse in a task, one would think that metrics are selected with great care. But when searching through papers for reasons that would support a particular metric choice, we mostly (at best) find only weak or indirect arguments. For example, it is observed (Table 1^,1) that ‘labels are imbalanced’ or it is wished for that ‘all class labels have equal weight’. These perceived problems or needs are then often supposedly addressed with a ‘macro’ metric (Table 1).

However, what is meant with phrases like ‘imbalanced data’ or ‘macro’ is rarely made explicit, and how the metrics that are then selected in this context are actually addressing a perceived ‘imbalance’ is unclear. According to a word etymology, evaluation with a ‘macro’ metric may involve the expectation that we are told a bigger picture of classifier capability (Greek: makrós, ‘long/large’), whereas a smaller picture (Greek: mikrós, ‘small’) would perhaps bind the assessment to a more local context. Regardless of such musings, it is clear that blurry terminology in the context of classifier evaluation can lead to results that may be misconceived by readers and authors alike.

This paper aims to serve as a handy reference for anyone who wishes to better understand classification evaluation, how evaluation metrics align with expectations expressed in papers, and how we might construct an individual case for (or against) selecting a particular evaluation metric.

After introducing Preliminaries (§2) and five Metric Properties (§3), we conduct a thorough Metric Analysis (§4) of common classification measures: ‘Accuracy’ (§4.1), ‘Macro Recall and Precision’ (§4.2, §4.3), two different ‘Macro F1’ (§4.4, §4.5), ‘Weighted F1’ (§4.6), as well as ‘Kappa’ and ‘Matthews Correlation Coefficient (MCC)’ in §4.7. We then show how to create simple but meaningful Metric Variants (§5). We wrap up the theoretical part with a Discussion (§6) that includes a short Summary (§6.1) of our main analysis results in Table 5. Next we study Metric Selection in Shared Tasks (§7) and give Recommendations (§8). Finally, we contextualize our work against some Background and Related Work (§9), and finish with Conclusions (§10).

We introduce a set of intuitive concepts as a basis.

For any classifier $f:D→C={1,…,n}$ and finite set S ⊆ D × C, let $mf,S∈R≥0n×n$ be a confusion matrix where $mijf,S=∣{s∈S∣f(s1)=i∧s2=j}∣$⁠.² We omit superscripts whenever possible. So generally, m_{i, j} contains the mass of events where the classifier predicts i, and the true label is j. That is, on the diagonal of the matrix lies the mass of correct predictions, and all other fields indicate the mass of specific errors. A $metric:R≥0n×n→(−∞,1]$ then allows us to order confusion matrices, respectively, rank classifiers (bounds are chosen for convenience). We say that (for a data set S) a classifier f is better than (or preferable to) a classifier g iff metric(m^{f, S}) > metric(m^{g, S}), i.e., a higher score indicates a better classifier.

Let us now define five basic quantities:

To understand and distinguish metrics in more precise ways, we define five metric properties: Monotonicity, class sensitivity, class decomposability, prevalence invariance, and chance correction.

Take a classifier that receives an input. If the prediction is correct, we would naturally expect that the evaluation score does not decrease, and if it is wrong, the evaluation score should not increase. We cast this clear expectation into

i.e., diagonal fields of the confusion matrix (correct mass) should yield a non-negative ‘gradient’ in the metric, while for all other fields (containing error mass) it should be non-positive. PI assumes differentiability of a metric, but it can be simply extended to the discrete case.³

A ‘macro’ metric needs to be sensitive to classes, or else it could not yield a ‘balanced measurement’ for ‘classes having different sizes’ (c.f. Table 1). By contrast, a ‘micro’ metric should care only about whether predictions are wrong or right, which would bind its score more to a local context of a specific data set and its class distribution. This means for macro metrics that they should possess

A metric without PII is not a macro metric.

‘Macro’ metrics are sometimes named ‘macro- average’ metrics. This suggests that they may be perceived as an average over classes. We introduce

i.e., as an unweighted mean over class-specific scores from inputs examples related to a specific class (c = i∨$f→i$⁠). For example, later we will see that ‘macro F1’ is a specific parameterization of Eq. 4.

A common argument for using metrics other than the ratio of correct predictions is that we want to ‘not neglect rare classes’ and ‘show classifier performance equally [w.r.t.] all classes’.⁵

Nicely, with the assumption of class prevalence being the only important difference across data sets, we could then even say that classifier f is generally better than (or preferable to) g iff metric(m^f) > metric(m^g), further disentangling a classifier comparison from a specific data set. At first glance, PIII seems to capture this wish already, by virtue of an unweighted mean over classes. However, the score w.r.t. one class is still influenced by the prevalence of other classes, and thus the result of the mean can change in non-transparent ways if class frequency is varied.

Therefore it makes sense to define such an expectation (‘treat all classes equally’) more strictly. We simulate different class prevalences with a

Two simple ‘baseline’ classifiers are: Predicting classes uniformly randomly, or based on observed prevalence. A macro metric can be expected to show robustness against any such chance classifier and be chance corrected, assigning a clear and comparable baseline score. Thus, it should have

Here, ω returns an upper-bound baseline score from the number of classes n^S alone. If it also holds that $max{...}=min{...}=ω(nS)$⁠, we say that metric is strictly chance corrected, and in the case where ∀S, r : metric(m^{r, S}) = Ω (constant) we speak of complete chance correction.

Less formally, chance correction means that the metric score attached to any chance baseline has a bound that is known to us (the bound generalizes over data sets but not over the number of classes). Strict chance correction means additionally that any chance classifier’s score will be the same, and just depends on the number of classes. Finally, complete chance correction means that every chance classifier always yields the same score, regardless of the number of classes. Note that strictness or completeness may not always be desired, since they can marginalize empirical overall correctness in a data set. Any chance correction, however, increases the evaluation interpretability by contextualizing the evaluation with an interpretable baseline score.

Equipped with the appropriate tools, we are now ready to start the analysis of classification metrics. We will study ‘Accuracy’, ‘Macro Recall’, ‘Macro Precision’, ‘Macro F1’, ‘Weighted F1’, ‘Kappa’, and ‘Matthews Correlation Coefficient’ (MCC).

As a ‘micro’ metric, Accuracy has only PI (monotonicity). This is expected, since PII-V aim at macro metrics. Interestingly, in multi-class evaluation, accuracy equals ‘micro Precion, micro Recall and micro F1’ that sometimes occur in papers. See Appendix A for proofs.

Accuracy is an important statistic, estimating the probability of observing a correct prediction in a data set. But this means that it is strictly tied to the class prevalences in a specific data set. And so, in the pursuit of some balance or a more generalizable score, researchers seem interested in other metrics.

Macro Recall has all five properties (Proofs in Appendix B). It is also strictly chance corrected with ω(n) = 1/n.

Since macro Recall has all five properties, including prevalence invariance (PIV), it may be a good pick for evaluation, particularly through a ‘macro’ lens. It also offers three intuitive interpretations: Drawing an item from a random class, Bookmaker metric, and prevalence-calibrated Accuracy.

In the first interpretation, we draw a random item from a randomly selected class. What’s the probability that it is correctly predicted? MacR estimates the answer $∑i1n·P(f→i∣c=i)$⁠.

While properties I, II, III, V are fulfilled, macro Precision does not have prevalence invariance (Proofs in Appendix C). With some λ, the max. score difference (macP(m) vs. macP(mλ)) approaches $1−1n$⁠.⁷ Like macro Recall, it is strictly chance corrected (ω(n) = 1/n).

Macro Precision wants to approximate the probability to see a correct prediction, given we randomly draw one out of n different predictions. Hence, macP seems to provide us with an interesting measure of ‘prediction trustworthiness’. An issue is that the score does not generalize across different class prevalences, since $bias(i)∝P(f→i)=∑jP(f→i,c=j)$ is subject to change if prevalences of other classes j ≠ i vary (∝ means approximately proportional to). Therefore, even though macP is decomposed over classes (PIII), it is not invariant to prevalence changes (PIV), and if we have f, f′ with different biases, score differences are difficult to interpret, particularly with an underlying ‘macro’ expectation that a metric be robust to class prevalence.

Again, all properties except PIV are fulfilled (Proofs in Appendix D). Interestingly, while macro F1 has PV (chance correction), the chance correction isn’t strict, differentiating it from other macro metrics: Indeed, its chance baseline upper bound ω(n) = 1/n is achieved only if $P(f→i)=P(c=i)$⁠, meaning that macro F1 not only corrects for chance, but also factors in more data set accuracy (like a ‘micro’ score). Additionally, the second line of the formula shows that macro F1 is invariant to the false-positive and false-negative error spread for a specific class.

Macro F1 wants the distribution of prediction and class prevalence to be similar (a micro feature), but also high correctness for every class, by virtue of the unweighted mean over classes (a macro feature). Thus it seems useful to find classifiers that do well in a given data set, but probably also in others, a ‘balance’ that could explain its popularity. However, macro F1 inherits an interpretability issue of Precision. It doesn’t strictly ‘treat all classes equally’ as per PIV, at least not without prevalence calibration (Eq. 6).

In contrast to its name twin, one less property is given (PIII), since it cannot be decomposed over classes (Proofs in Appendix E), and it is strictly chance corrected with ω(n) = 1/n. Opitz and Burst (2019) prove that Eq. 11 and Eq. 10 can diverge to a large degree of up to 0.5.

Weighted F1 is sensible to classes (PII). The other four properties are not featured, which means that it is also non-monotonic. See Appendix F for proofs.

While measuring performance ‘locally’ for each class, the results are weighted by class-prevalence. Imagining metrics on a spectrum from ‘micro’ to ‘macro’, weightF1 sits next to Accuracy, the prototypical micro metric. This is also made obvious by its featured properties, where only one would mark a ‘macro’ metric (PII). Due to its lowered interpretability and non-monotonicity, we may wonder why weightF1 would be preferred over Accuracy. Finally, with prevalence calibration, it reduces to macro F1, weightF1(mλ^∼) = macF1(mλ^∼), similar to how calibrated Accuracy reduces to macro Recall.

MCC and Kappa have PII and PV (complete chance correction: Ω = 0). However, they are non-monotonic (PI), not class-decomposable (PIII), and not prevalence-invariant (PIV); Proofs in Appendix G. Further note that sgn(MCC) = sgn(KAPPA) and |MCC|≥|KAPPA|, since p^Tb ≤{b^Tb, p^Tp}.

Kappa and MCC are similar measures. Since $chance≈∑iP(c=i)·P(f→i)$ allows the interpretation of observing a prediction that is correct just by chance, Kappa and MCC can be viewed as a standardized Accuracy.

Property PIV (prevalence invariance) is rare, but we saw that it can be artificially enforced. Indeed, if we standardize the confusion matrix by making sure every class has the same prevalence (Eq. 5, Eq. 6), we ensure prevalence invariance (PIV) for a measure. As an effect of this, we found that Kappa and Accuracy reduce to macro Recall, and weighted F1 becomes the same as macro F1. For a more detailed interpretation of prevalence calibration, see our discussion for macro Precision (§4.3).

When does a prevalence calibration make sense? Since prevalence calibration offers a gain in ‘macro’-features, it can be used with the aim to push a metric more towards a ‘macro’ metric.

Table 5 shows an overview of the visited metrics. We make some observations: i) macro Recall has all five properties, including class prevalence invariance (PIV), i.e., ‘it treats all classes equally’ (in a strict sense). However, through prevalence calibration, all metrics obtain PIV. ii) Kappa, MCC, and weighted F1 do not have property PI. Under some circumstances, errors can increase the score, possibly lowering interpretability. iii) All metrics except Accuracy and weighted F1 show chance baseline correction. Strict chance baseline correction isn’t a feature of Macro F1, and complete (class-count independent) chance correction is only achieved with MCC and Kappa.

Macro Recall and Accuracy seem to complement each other. Both have a clear interpretation, and relate to each other with a simple prevalence calibration. Indeed, macro Recall can be understood as a prevalence-calibrated version of Accuracy. On the other hand, macro F1 is interesting since it does not strictly correct for chance (as in macro Recall) but also factors in more of the test set correctness (as in a ‘micro’ score).

MCC and Kappa are similar measures, where Kappa tends to be slightly more interpretable and shows more robustness to classifier biases. Somewhat also similar are Accuracy and weighted F1, both are greatly affected by class-prevalence. As discussed in §4.6, we could not determine clear reasons for favoring weighted F1 over Accuracy.

The concept of ‘balance’ seems positively flavored, and thus we may wish to reflect on more ‘balances’ other than prevalence invariance (PIV).

Another type of ‘balance’ is introduced by GmacR (or HmacR). By virtue of the geometric (harmonic) mean that puts more weight on low outliers, they favor a classifier that equally distributes its correctness over classes. This is also reflected by macR being strictly chance corrected with 1/n, while its siblings have 1/n as the upper bound only achieved by the uniform random baseline, and the metrics’ gradients that scale with low-recall outliers (Appendix B.5, B.6).

Yet again another type of ‘balance’ we saw in macro F1 (macF1) that selects a classifier with high recall over many classes (as featured by a ‘macro’ metric) and maximizes empirical data set correctness (as featured by a ‘micro’ metric), an attribute that is also visible in its chance baseline upper bound (1/n) that is only achieved by a prevalence-informed baseline.

Finally, a ‘meta balance’ could be achieved when we are unsure which metric to use, by ensembling a score from a set of selected metrics.

So far, we had focused on theory. Now we want to take a look at applied evaluation practice. We study works from the SemEval series, a large annual NLP event where teams compete in various tasks, many of which are classification tasks.

As an example, we first study the popular SemEval shared task (Rosenthal et al., 2017) on tweet sentiment classification (positive/negative/neutral) with team predictions thankfully made available.

In Table 6 we see the results of the eight best of 37 systems.⁹ The two ‘winning’ systems (A, B, Table 6) were determined with macR, which is legitimate. Yet, system E also does quite well: It obtains highest Accuracy, and it achieves a better balance over the three classes (R₁ = 69.8, R₂ = 64.0, R₃ = 66.8, max. Δ = 5.8) as opposed to, e.g., system B (R₁ = 87.8, R₂ = 51.4, R₃ = 65.2 max. Δ = 36.4), indicated, on average, also by GmacR and HmacR metrics. So if we want to ensure performance under high uncertainty of class prevalence (as expected in Twitter data?), we may prefer system E, a system that would be also ranked higher when using an ensemble of metrics.

Figure 1 shows a pair-wise Spearman ρ correlation of team rankings of all 37 teams according to different metrics. There are only 4 pairs of metrics that are in complete agreement (ρ = 100): Macro Recall agrees with calibrated Kappa and calibrated Accuracy. This makes sense: As we noted before (Eq. 8, Eq. 16), they are equivalent (the same applies to weighted F1 and macro F1, after calibration). Looking at single classes, it seems that the second class is the one that can tip the scale: R₂ disagrees in its team ranking with all other metrics (ρ ≤ 14).

For the eight best systems in Table 6, we retrieve their citation count from Google scholar as a (very coarse) proxy for their popularity. The result invites the speculation that a superficial ordering could already be effectual: While both team 1 (A) and team 2 (B) were explicitly selected as the winner of the shared task, the first-listed system appears to have almost double the amount of citations (even though B performs better as per most metrics). The citations of other systems (C–H) do not exceed lower double digits, although we saw that a case can even be made for E (rank 6), which achieves stable performance over all classes.

Our selected example task used macro Recall for winner selection. While the measure had been selected with care (its value became also evident in our analysis), we saw that systems were very close and arguments could have been made for selecting a slightly different (set of) winner(s). To our surprise, in a large proportion of shared tasks, the situation seems worse. Annotating 42 classification shared task overview papers from the recent 5 years,¹⁰ we find that

• only 23.8% of papers provide a formula, and

• only 10.9% provide a sensible argument for their metric. 14.3% use a weak argument similar to the ones shown in Table 1. A large number (73.8%) do not state an argument or employ a ‘trope’ like “As is standard, ...”.

The most frequent metric in our sample is ‘macro F1’ (macF1). In some cases, its doppelganger macF1′ seems to have been used, or a ‘balanced Accuracy’.¹¹ Sometimes, in the absence of further description or formula, it can be hard to tell which ‘macro F1’ metric has been used, also due to deviating naming (macro-average F1, mean F1, macro F1, etc.). In at least one case, this has led to a disadvantage: Fang et al. (2023) report that “During model training and validation, we were not aware that the challenge organizers used a different method for calculating macro F1, namely by using the averages of precision and recall”, a measure that is much different (cf. §4.4).

We also may wonder about the situation beyond SemEval. While a precise characterization of a broader picture would escape the scope of this paper, a cursory glance shows the observed unclarities in research papers from all kinds of domains.

Overall, we would like to refrain from making bold statements as to which metric is ‘better’, since different contexts may easily call for different metrics. Still, from our analyses, we can synthesize some general recommendations:

• State the evaluation metric clearly, best use a formula. This also helps protect against ambiguity, e.g., as induced through homonyms such as macF1 and macF1′.

• Try building a case for a metric: As a starting point we can think about how the class distribution in our data set would align with the distribution that we could expect in an application. With greater uncertainty, more macro metric features may be useful. For finer selection, consider viewing our analyses, checking any desirable metric features. Our summary (§6.1) can provide guidance.

• Consider presenting more than a single number. In particular, complementary metrics such as Accuracy and macro Recall can be indicative about i) a classifier’s empirical data set correctness (as in a micro metric) and ii) its robustness to class distribution shifts (as in a macro metric). If the amount of classes n is low, consider presenting class-wise recall scores for their generalizability. If n is larger and a metric is decomposable over classes (PIII), reporting the variance of a ‘macro’ metric over classes can also be of value.

• Consider admitting multiple ‘winners’ or ‘best systems’: If we are not able to build a strong case for a single metric, then it may be sensible to present a set of well-motivated metrics and select a set of best systems. If one ‘best’ system needs be selected, an average over such a set could be a useful heuristic.

Surveys or dedicated book chapters can provide a useful introduction to classification evaluation (Sokolova and Lapalme, 2009; Manning, 2009; Tharwat, 2020; Grandini et al., 2020). Deeper analysis has been provided mostly in the two-class setting: In a series of articles by David M. W. Powers, we find the (previously mentioned) Bookmaker’s perspective on metrics (Powers, 2011), a critique of the F1 (Powers, 2015) and Kappa (Powers, 2012). Delgado and Tibau (2019) study binary MCC and Kappa (favoring MCC), while Sebastiani (2015) defines axioms for binary evaluation, including a monotonicity axiom akin to a stricter version of our PI, advocating a ‘K-metric’.¹² Luque et al. (2019) analyze binary confusion matrices, and Chicco and Jurman (2020) compare F1, Accuracy and MCC, concluding that “MCC should be preferred (…) by all scientific communities”. The mathematical relationship between the two macro F1s is further analyzed by Opitz and Burst (2019).

Overall, we want to advocate for a mostly agnostic stance as to what metric might be picked in a case (if it is sensibly done so), remembering our premise from the introduction: the perfect metric doesn’t exist. Thus, we aimed at balancing intuitive interpretation and analysis of metrics, while acknowledging desiderata as worded in papers.

These can be required when class labels reside on a nominal ‘Likert’ scale (O’Neill, 2017; Amigo et al., 2020), or in a hierarchy (Kosmopoulos et al., 2015), or they are ambiguous and need be matched (e.g., ‘none’/‘null’/‘other’) across annotation styles (Fu et al., 2020), or their number is unknown (Ge et al., 2017). Classifiers are also evaluated with P-R curves (Flach and Kull, 2015) or a receiver-operator characteristic (Fawcett, 2006; Honovich et al., 2022). The CheckList (Ribeiro et al., 2020) proposes behavioral testing of classifiers, and the NEATCLasS workshop series (Ross et al., 2023) is an effort to find novel ways of evaluation.

Starting from a definition of the two basic and intuitive concepts of classifier bias and class prevalence, we examined common classification evaluation metrics, resolving unclear expectations such as those that pursue a ‘balance’ through ‘macro’ metrics. Our metric analysis framework, including definitions and properties, can aid in the study of other or new metrics. A main goal of our work is to provide guidance for more informed decision making in metric selection.

We are grateful to three anonymous reviewers and Action Editor Sebastian Gehrmann for their valuable comments. We are also thankful to Julius Steen for helpful discussion and feedback.

i ≠ j: $∂Acc(m)∂mi,j=−∑kcorrect(k)∣S∣2=−Acc∣S∣≤0$⁠; else $∂Acc(m)∂mi,i=∣S∣−∑kcorrect(k)∣S∣2=1−Acc∣S∣≥0$⁠.

It is easy to see that properties other than Monotonicity are not featured by Accuracy.

If i ≠ j: $∂macR(m)∂mi,j=−correct(j)n·prevalence(j)2≤0$⁠; else $∂macR(m)∂mi,i=prevalence(i)−correct(i)n·prevalence(i)2≥0$⁠.

Follows from above.

In Eq. 4 set $g(row,col,x)=rowx∑icoli$ and p = 1.

$Ri′=λi,imi,i∑jλi,imj,i=λi,imi,iλi,i∑jmj,i=Ri$⁠.

If i ≠ j: $∂macP(m)∂mi,j=−correct(i)n·bias(i)2≤0$⁠; else $∂macP(m)∂mi,i=bias(i)−correct(i)n·bias(i)2≥0$⁠.

Follows from above.

In Eq. 4 set $g(row,col,x)=rowx∑irowi$ and p = 1.

A counter-example $Pi′=λi,imi,i∑jλj,jmi,j≠Pi$ is easily found. E.g., in Table 2, 3, 4: $macP=0.534+0.512=58≠macP′=0.535+0.523=1930$⁠.

Follows from above.

In Eq. 4 set p = 1, $g(row,col,x)=2rowx∑irowi+coli$⁠.

It is not prevalence-invariant, c.f. C.4.

Let (macR + macP −macP · macR) = ϵ ≥ 0. We have $∂macF1′(m)∂mi,j=2xϵmacR+macP$ where $x=(∂macR(m)∂mi,j+∂macP(m)∂mi,j)$⁠. Since macR and macP are monotonic, macF1′ also has monotonicity.

Follows from above.

Not possible.

It is not prevalence-invariant, c.f. C.4.

Since macF1′ is the HM of (strictly chance corrected) macro Precision and macro Recall, we also have strictly chance correction with $1n$⁠.

Follows from above.

Not possible.

Trivial.

Trivial.

We resort back to Kappa and MCC formulas for non-normalized confusion matrices, multiplying numerator and denominator by s², where $s=∑(i,j)mi,j$ is the number of data examples, and write r for $∑icorrect(i)$⁠.