Table 1
Item, value, and pair weight comparisons in the case of five annotators with missing values (denoted by “.”)
| item 1 | item 2 | item 3 | item 4 | |
|---|---|---|---|---|
| Mary | noun | noun | noun | noun |
| Paul | noun | noun | noun | noun |
| Suzan | noun | noun | verb | . |
| Jack | noun | noun | . | . |
| Robert | noun | . | . | . |
| nv = number of values | 5 | 4 | 3 | 2 |
| np = number of pairs | 10 | 6 | 3 | 1 |
 | ||||
| wu(VL) : item weight in VL | 5/2 | 2 | 3/2 | 1 |
| wu(PL) : item weight in PL | 10 | 6 | 3 | 1 |
| wv(IL) = 2wu(IL) / nv : value weight in IL | 2/5 | 1/2 | 2/3 | 1 |
 | ||||
| wv(PL) = 2wu(PL) / nv : value weight in PL | 4 | 3 | 2 | 1 |
| wp(IL) = wu(IL) / np : pair weight in IL | 1/10 | 1/6 | 1/3 | 1 |
| wp(VL) = wu(VL) / np : pair weight in VL | 1/4 | 1/3 | 1/2 | 1 |
 | ||||
| item 1 | item 2 | item 3 | item 4 | |
|---|---|---|---|---|
| Mary | noun | noun | noun | noun |
| Paul | noun | noun | noun | noun |
| Suzan | noun | noun | verb | . |
| Jack | noun | noun | . | . |
| Robert | noun | . | . | . |
| nv = number of values | 5 | 4 | 3 | 2 |
| np = number of pairs | 10 | 6 | 3 | 1 |
| | ||||
| wu(VL) : item weight in VL | 5/2 | 2 | 3/2 | 1 |
| wu(PL) : item weight in PL | 10 | 6 | 3 | 1 |
| wv(IL) = 2wu(IL) / nv : value weight in IL | 2/5 | 1/2 | 2/3 | 1 |
| | ||||
| wv(PL) = 2wu(PL) / nv : value weight in PL | 4 | 3 | 2 | 1 |
| wp(IL) = wu(IL) / np : pair weight in IL | 1/10 | 1/6 | 1/3 | 1 |
| wp(VL) = wu(VL) / np : pair weight in VL | 1/4 | 1/3 | 1/2 | 1 |
| | ||||