Skip to Main Content
Table 2: 

Ranker performance (top-5) on dev set. Asterisk (*) indicates our best ranker used in Table 1.

IR MethodEMROUGE-L
Baseline Rankers 
BM25 18.99 47.48 
BERT DS-ranker (Mou et al., 2020) 24.26 52.68 
 - ROUGE-L filtering 22.63 51.02 
 Repl BERT w/ BiDAF 21.88 50.64 
 Repl BERT w/ MatchLSTM 21.97 50.39 
 
Our Rankers 
BERT ICT-ranker 21.29 50.35 
BERT DS-ranker 
 + Hard EM 22.45 50.50 
 + ICT pre-training* 24.83 53.19 
 
Oracle Conditions 
Upperbound (BM25 top-32) 30.81 61.40 
Oracle (BM25 w/ Q+A) 35.75 63.92 
IR MethodEMROUGE-L
Baseline Rankers 
BM25 18.99 47.48 
BERT DS-ranker (Mou et al., 2020) 24.26 52.68 
 - ROUGE-L filtering 22.63 51.02 
 Repl BERT w/ BiDAF 21.88 50.64 
 Repl BERT w/ MatchLSTM 21.97 50.39 
 
Our Rankers 
BERT ICT-ranker 21.29 50.35 
BERT DS-ranker 
 + Hard EM 22.45 50.50 
 + ICT pre-training* 24.83 53.19 
 
Oracle Conditions 
Upperbound (BM25 top-32) 30.81 61.40 
Oracle (BM25 w/ Q+A) 35.75 63.92 
Close Modal

or Create an Account

Close Modal
Close Modal