Skip to Main Content

One of our goals is to assess whether FactBank’s seven-category scheme is the right one for the task. To this end, we also evaluated whether a five-tag version would increase agreement and perhaps provide a better match with readers’ intuitions. Logically, PR − is equivalent to PS+, and PS − to PR+, so it seemed natural to try to collapse them into a two-way division between “probable” and “possible”. We thus ran the MTurk experiment again with the five-point scheme in Table 3.

Table 3. 

An alternative five-tag annotation scheme.

Category
Original
yes CT+ 
probable PR+/PS− 
possible PS+/PR− 
no CT− 
unknown Uu 
Category
Original
yes CT+ 
probable PR+/PS− 
possible PS+/PR− 
no CT− 
unknown Uu 

Close Modal

or Create an Account

Close Modal
Close Modal