Skip to Main Content

To assess the predictive power of the five feature types (LX, DS, TM, AC, and PH) we exclude one type at a time and compare the performance of the resulting set to that of the full model. Table 10 displays the error rate of each ML classifier on the general task, classifying any ACW into any of the most frequent discourse/pragmatic functions (Agr, BC, CBeg, PEnd, Mod, Chk). Table 11 shows the same results for the other two tasks: the detection of a discourse boundary function—cue beginning (CBegPBeg), cue ending (CEnd, PEnd), or no-boundary (all other labels); and the detection of an acknowledgment function—Agr, BC, PBeg, or PEnd, vs. all other labels.10

Table 10

Error rate of each classifier on the general task using different feature sets; F-measures of the SVM classifier; and error rate and F-measures of two baselines and human labelers. For the classifier error rates: Significantly different from full model. § Significantly different from SVM. (Wilcoxon signed rank sum test, p < 0.05.) Significance was not tested for the classifier F-measures.


Error Rate
SVM F-Measure
Feature Set
C4.5
Ripper
SVM
Agr
BC
CBeg
PEnd
Mod
Chk
LXDSTMACPH 16.6%§ 16.3%§ 14.3% .86 .81 .89 .50 .97 .39 
DSTMACPH 21.3% †§ 17.2%  16.5%  .84 .82 .87 .44 .94 
LXTMACPH 20.3% †§ 20.1% § 17.0%  .84 .80 .83 .16 .97 .21 
LXDSACPH 17.1% § 18.1% †§ 14.8%  .86 .81 .89 .38 .97 .35 
LXDSTMPH 15.2%  16.3% 16.2%  .85 .80 .86 .16 .97 .33 
LXDSTMAC 17.0% § 16.9% § 14.7% .86 .80 .89 .48 .97 .35 
LX 23.7% †§ 22.7%  22.3%  .79 .80 .65 .96 .03 
DS 22.8% †§ 24.0% †§ 25.3%  .76 .67 .82 .87 
TM 29.5% †§ 27.3% †§ 36.2%  .70 .57 .83 
AC 44.8% †§ 29.8% †§ 41.3%  .67 .66 .14 .58 
PH 56.4% †§ 26.5% †§ 45.4%  .65 .08 .13 .64 
Majority class baseline ER 56.4% .61 
Word-based baseline ER 27.7% .75 .79 .94 .13 
Human labelers ER (estimate 1)  9.3% .92 .91 .94 .51 .99 .67 
Human labelers ER (estimate 2) 11.0% .90 .89 .93 – .99 – 

Error Rate
SVM F-Measure
Feature Set
C4.5
Ripper
SVM
Agr
BC
CBeg
PEnd
Mod
Chk
LXDSTMACPH 16.6%§ 16.3%§ 14.3% .86 .81 .89 .50 .97 .39 
DSTMACPH 21.3% †§ 17.2%  16.5%  .84 .82 .87 .44 .94 
LXTMACPH 20.3% †§ 20.1% § 17.0%  .84 .80 .83 .16 .97 .21 
LXDSACPH 17.1% § 18.1% †§ 14.8%  .86 .81 .89 .38 .97 .35 
LXDSTMPH 15.2%  16.3% 16.2%  .85 .80 .86 .16 .97 .33 
LXDSTMAC 17.0% § 16.9% § 14.7% .86 .80 .89 .48 .97 .35 
LX 23.7% †§ 22.7%  22.3%  .79 .80 .65 .96 .03 
DS 22.8% †§ 24.0% †§ 25.3%  .76 .67 .82 .87 
TM 29.5% †§ 27.3% †§ 36.2%  .70 .57 .83 
AC 44.8% †§ 29.8% †§ 41.3%  .67 .66 .14 .58 
PH 56.4% †§ 26.5% †§ 45.4%  .65 .08 .13 .64 
Majority class baseline ER 56.4% .61 
Word-based baseline ER 27.7% .75 .79 .94 .13 
Human labelers ER (estimate 1)  9.3% .92 .91 .94 .51 .99 .67 
Human labelers ER (estimate 2) 11.0% .90 .89 .93 – .99 – 
Table 11

Error rate of each classifier on the detection of discourse boundary functions and acknowledgment functions, using different feature sets. Significantly different from full model. § Significantly different from SVM. (Wilcoxon signed rank sum test, p < 0.05.)


Disc. Boundary
Acknowledgment

{CBeg, PBeg} vs. {CEnd, PEnd} vs. Rest
{Agr, BC, PBeg, PEnd} vs. Rest
Feature Set
C4.5
Ripper
SVM
C4.5
Ripper
SVM
LXDSTMACPH  6.9%  8.1% §  6.9%  5.8%  5.9% §  4.5% 
DSTMACPH  7.6%   8.0%  7.6%   8.5% †§  5.5% §  6.4%  
LXTMACPH 10.4%  10.1%   9.5%   8.7% †§  8.7% †§  6.5%  
LXDSACPH  8.0%   8.7% §  7.5%   5.3%  5.7% §  4.9% 
LXDSTMPH  6.6% §  7.9%  8.9%   5.4%  5.4%  5.1% 
LXDSTMAC  7.1%  8.3% §  7.0%  5.8% §  5.6% §  4.6% 
LX 14.2%  14.5% †§ 13.9%  11.4%  11.4%  11.7%  
DS  7.8% §  8.6% § 10.9%   8.4% †§  8.9%   9.4%  
TM 12.2% †§ 11.2% †§ 14.7%  12.8% †§ 13.5%  14.5%  
AC 17.3% †§ 14.3% †§ 18.5%  26.7%  16.6% †§ 28.4%  
PH 18.6%  17.6%  18.6%  36.5% †§ 14.1% †§ 25.4%  
Majority class baseline ER 18.6% 36.5% 
Word-based baseline ER 18.6% 15.3% 
Human labelers ER (est. 1)  5.3%  2.9% 
Human labelers ER (est. 2)  5.6%  3.0% 

Disc. Boundary
Acknowledgment

{CBeg, PBeg} vs. {CEnd, PEnd} vs. Rest
{Agr, BC, PBeg, PEnd} vs. Rest
Feature Set
C4.5
Ripper
SVM
C4.5
Ripper
SVM
LXDSTMACPH  6.9%  8.1% §  6.9%  5.8%  5.9% §  4.5% 
DSTMACPH  7.6%   8.0%  7.6%   8.5% †§  5.5% §  6.4%  
LXTMACPH 10.4%  10.1%   9.5%   8.7% †§  8.7% †§  6.5%  
LXDSACPH  8.0%   8.7% §  7.5%   5.3%  5.7% §  4.9% 
LXDSTMPH  6.6% §  7.9%  8.9%   5.4%  5.4%  5.1% 
LXDSTMAC  7.1%  8.3% §  7.0%  5.8% §  5.6% §  4.6% 
LX 14.2%  14.5% †§ 13.9%  11.4%  11.4%  11.7%  
DS  7.8% §  8.6% § 10.9%   8.4% †§  8.9%   9.4%  
TM 12.2% †§ 11.2% †§ 14.7%  12.8% †§ 13.5%  14.5%  
AC 17.3% †§ 14.3% †§ 18.5%  26.7%  16.6% †§ 28.4%  
PH 18.6%  17.6%  18.6%  36.5% †§ 14.1% †§ 25.4%  
Majority class baseline ER 18.6% 36.5% 
Word-based baseline ER 18.6% 15.3% 
Human labelers ER (est. 1)  5.3%  2.9% 
Human labelers ER (est. 2)  5.6%  3.0% 

Close Modal

or Create an Account

Close Modal
Close Modal