Inter-annotator agreement for our datasets calculated using Fleiss’ kappa (κ) at the token and entity level. Disagreement from type refers to the proportion of all entity-level disagreements, which are due only to type mismatch.
| Dataset . | Token Fleiss’ κ . | Entity Fleiss’ κ . | Disagreement from Type . |
|---|---|---|---|
| amh | 0.987 | 0.959 | 0.044 |
| hau | 0.988 | 0.962 | 0.097 |
| ibo | 0.995 | 0.983 | 0.071 |
| kin | 1.000 | 1.000 | 0.000 |
| lug | 0.997 | 0.990 | 0.023 |
| luo | 1.000 | 1.000 | 0.000 |
| pcm | 0.989 | 0.966 | 0.048 |
| swa | 1.000 | 1.000 | 0.000 |
| wol | 1.000 | 1.000 | 0.000 |
| yor | 0.990 | 0.964 | 0.079 |
| Dataset . | Token Fleiss’ κ . | Entity Fleiss’ κ . | Disagreement from Type . |
|---|---|---|---|
| amh | 0.987 | 0.959 | 0.044 |
| hau | 0.988 | 0.962 | 0.097 |
| ibo | 0.995 | 0.983 | 0.071 |
| kin | 1.000 | 1.000 | 0.000 |
| lug | 0.997 | 0.990 | 0.023 |
| luo | 1.000 | 1.000 | 0.000 |
| pcm | 0.989 | 0.966 | 0.048 |
| swa | 1.000 | 1.000 | 0.000 |
| wol | 1.000 | 1.000 | 0.000 |
| yor | 0.990 | 0.964 | 0.079 |