Results over documents annotated consistently by all 5 annotators, where annotations can be the same as or different from gold-standard.
| . | Wiki→Wiki . | CNN→Wiki . | ||
|---|---|---|---|---|
| . | Acc (%) . | F1 (%) . | Acc (%) . | F1 (%) . |
| Majority-class | 70.6 | 0.0 | 70.6 | 0.0 |
| BERT-Large | 76.7 | 42.0 | 75.4 | 36.9 |
| XLNet-Large | 79.1 | 57.0 | 76.6 | 35.4 |
| RoBERTa-Large | 82.0 | 59.6 | 77.3 | 37.4 |
| ALBERT-xxLarge | 85.9 | 68.8 | 78.8 | 42.9 |
| Human | 79.5 | 45.4 | 79.5 | 45.4 |
| . | Wiki→Wiki . | CNN→Wiki . | ||
|---|---|---|---|---|
| . | Acc (%) . | F1 (%) . | Acc (%) . | F1 (%) . |
| Majority-class | 70.6 | 0.0 | 70.6 | 0.0 |
| BERT-Large | 76.7 | 42.0 | 75.4 | 36.9 |
| XLNet-Large | 79.1 | 57.0 | 76.6 | 35.4 |
| RoBERTa-Large | 82.0 | 59.6 | 77.3 | 37.4 |
| ALBERT-xxLarge | 85.9 | 68.8 | 78.8 | 42.9 |
| Human | 79.5 | 45.4 | 79.5 | 45.4 |