Skip to Main Content
Table 1 

Previous works on annotating argumentation. IAA = Inter-annotator agreement; N/A = not applicable.

SourceArg. ModelDomainSizeIAA
Newman and Marshall (1991) Toulmin (1958) legal domain (People vs. Carney, U.S. Supreme Court) qualitative N/A 
Bal and Dizier (2010) proprietary socio-political newspaper editorials 56 documents Cohen's κ (0.80) 
Feng and Hirst (2011) Walton, Reed, and Macagno (2008) (top 5 schemes) legal domain (AracuariaDB corpus, 61% subset annotated with Walton scheme) ≈ 400 arguments not reported claimed to be small 
Biran and Rambow (2011) proprietary Wikipedia Talk pages, blogs 309 + 118 Cohen's κ (0.69) 
Georgila et al. (2011) proprietary general discussions (negotiations between florists) 21 dialogs Krippendorff's α (0.37-0.56) 
Mochales and Moens (2011) Claim-Premise based on Freeman (1991) legal domain (AracuariaDB corpus, European Human Rights Council) 641 documents w/ 641 arguments (AracuariaDB) 67 documents w/ 257 arguments (EHRC) not reported 
Walton (2012) Walton, Reed, and Macagno (2008) (14 schemes) political argumentation 256 arguments not reported 
Rosenthal and McKeown (2012) opinionated claim, sentence level blog posts, Wikipedia discussions 4000 sentences Cohen's κ (0.50-0.57) 
Conrad, Wiebe, and Hwa (2012) proprietary (spans of arguing subjectivity) editorials and blog post about ObamaCare 84 documents Cohen's κ (0.68) on 10 documents 
Schneider and Wyner (2012) proprietary, argumentation schemes camera reviews N/A (proposal/position paper) N/A 
Schneider, Davis, and Wyner (2012) Dung (1995) + Walton, Reed, and Macagno (2008) unspecified social media N/A (proposal/position paper) N/A 
Villalba and Saint-Dizier (2012) proprietary, RST hotel reviews, hi-fi products, political campaign 50 documents not reported 
Peldszus and Stede (2013a) Freeman (1991) + RST Potsdam Commentary Corpus N/A (proposal/position paper) N/A 
Florou et al. (2013) none public policy making 69 argumentative segments / 322 non-argumentative segments not reported 
Peldszus and Stede (2013b) based on Freeman (1991) not reported, artificial documents created for the study 23 short documents Fleiss' κ multiple results 
Sergeant (2013) N/A Car Review Corpus (CRC) N/A (proposal/position paper) N/A 
Wachsmuth et al. (2014) none hotel reviews 2100 reviews Fleiss' κ (0.67) 
Procter, Vis, and Voss (2013) proprietary (Claim, Counter-claim) Riot Twitter Corpus 7729 tweets under ‘Rumors’ category percentage agreement (89% – 96%) 
Stab and Gurevych (2014a) Claim-Premise based on Freeman (1991) student essays 90 documents Kripp. αU (0.72) Kripp. α (0.81) 
Aharoni et al. (2014) proprietary (claims, evidence) Wikipedia 104 documents Cohen's κ (0.40) 
Park and Cardie (2014) proprietary (argument propositions) policy making (passenger rights and consumer protection) 1047 documents Cohen's κ (0.73) 
Goudas et al. (2014) proprietary (premises) social media 204 documents not reported 
Faulkner (2014) none (“supporting argument”) student essays 8176 sentences Cohen's κ (0.70) 
SourceArg. ModelDomainSizeIAA
Newman and Marshall (1991) Toulmin (1958) legal domain (People vs. Carney, U.S. Supreme Court) qualitative N/A 
Bal and Dizier (2010) proprietary socio-political newspaper editorials 56 documents Cohen's κ (0.80) 
Feng and Hirst (2011) Walton, Reed, and Macagno (2008) (top 5 schemes) legal domain (AracuariaDB corpus, 61% subset annotated with Walton scheme) ≈ 400 arguments not reported claimed to be small 
Biran and Rambow (2011) proprietary Wikipedia Talk pages, blogs 309 + 118 Cohen's κ (0.69) 
Georgila et al. (2011) proprietary general discussions (negotiations between florists) 21 dialogs Krippendorff's α (0.37-0.56) 
Mochales and Moens (2011) Claim-Premise based on Freeman (1991) legal domain (AracuariaDB corpus, European Human Rights Council) 641 documents w/ 641 arguments (AracuariaDB) 67 documents w/ 257 arguments (EHRC) not reported 
Walton (2012) Walton, Reed, and Macagno (2008) (14 schemes) political argumentation 256 arguments not reported 
Rosenthal and McKeown (2012) opinionated claim, sentence level blog posts, Wikipedia discussions 4000 sentences Cohen's κ (0.50-0.57) 
Conrad, Wiebe, and Hwa (2012) proprietary (spans of arguing subjectivity) editorials and blog post about ObamaCare 84 documents Cohen's κ (0.68) on 10 documents 
Schneider and Wyner (2012) proprietary, argumentation schemes camera reviews N/A (proposal/position paper) N/A 
Schneider, Davis, and Wyner (2012) Dung (1995) + Walton, Reed, and Macagno (2008) unspecified social media N/A (proposal/position paper) N/A 
Villalba and Saint-Dizier (2012) proprietary, RST hotel reviews, hi-fi products, political campaign 50 documents not reported 
Peldszus and Stede (2013a) Freeman (1991) + RST Potsdam Commentary Corpus N/A (proposal/position paper) N/A 
Florou et al. (2013) none public policy making 69 argumentative segments / 322 non-argumentative segments not reported 
Peldszus and Stede (2013b) based on Freeman (1991) not reported, artificial documents created for the study 23 short documents Fleiss' κ multiple results 
Sergeant (2013) N/A Car Review Corpus (CRC) N/A (proposal/position paper) N/A 
Wachsmuth et al. (2014) none hotel reviews 2100 reviews Fleiss' κ (0.67) 
Procter, Vis, and Voss (2013) proprietary (Claim, Counter-claim) Riot Twitter Corpus 7729 tweets under ‘Rumors’ category percentage agreement (89% – 96%) 
Stab and Gurevych (2014a) Claim-Premise based on Freeman (1991) student essays 90 documents Kripp. αU (0.72) Kripp. α (0.81) 
Aharoni et al. (2014) proprietary (claims, evidence) Wikipedia 104 documents Cohen's κ (0.40) 
Park and Cardie (2014) proprietary (argument propositions) policy making (passenger rights and consumer protection) 1047 documents Cohen's κ (0.73) 
Goudas et al. (2014) proprietary (premises) social media 204 documents not reported 
Faulkner (2014) none (“supporting argument”) student essays 8176 sentences Cohen's κ (0.70) 
Close Modal

or Create an Account

Close Modal
Close Modal