Skip to Main Content
Table 3: 

End-to-end OpenQA results, reporting exact-match (EM) on the open-domain test sets. For SQuAD, we report EM on the main 2018 Wikipedia dump (left) and on the 2016 Wikipedia dump (right). Underlined are the best results using only a BERT-base reader; bold is overall best.

NameRetriever (# parameters)Reader (# parameters)NQTQSQuAD
Mainstream OpenQA 
BM25 + BERT Karpukhin et al. (2020) BM25 BERT-base (110M) 32.6 52.4 38.1 / – 
GraphRetriever Min et al. (2019b) BM25 +graph-based retr. BERT-base (110M) 34.5 56.0 – 
PathRetriever Asai et al. (2020) TFIDF +graph-based retr. BERT-WWM (330M) 32.6 – – / 56.5 
 
Learned-Retrieval OpenQA 
ORQA Lee et al. (2019) ORQA (2×110M) BERT-base (110M) 33.3 45.0 20.2 / – 
REALM Guu et al. (2020)Guu et al. (2020) REALM (2×110M) BERT-base (110M) 40.4 – – 
DPR (best variant) Karpukhin et al. (2020) DPR (2×110M) BERT-base (110M) 41.5 57.9 36.7 / – 
RAG (generative) Lewis et al. (2020) DPR (joint) (2×110M) BART-large (406M) 44.5 56.1 – 
 
Our Models 
ColBERT-QA (base) ColBERT-QA3 (110M) BERT-base (110M) 42.3 64.6 47.7 / 53.5 
ColBERT-QA (large) ColBERT-QA3 (110M) BERT-WWM (330M) 47.8 70.1 54.7 / 58.7 
NameRetriever (# parameters)Reader (# parameters)NQTQSQuAD
Mainstream OpenQA 
BM25 + BERT Karpukhin et al. (2020) BM25 BERT-base (110M) 32.6 52.4 38.1 / – 
GraphRetriever Min et al. (2019b) BM25 +graph-based retr. BERT-base (110M) 34.5 56.0 – 
PathRetriever Asai et al. (2020) TFIDF +graph-based retr. BERT-WWM (330M) 32.6 – – / 56.5 
 
Learned-Retrieval OpenQA 
ORQA Lee et al. (2019) ORQA (2×110M) BERT-base (110M) 33.3 45.0 20.2 / – 
REALM Guu et al. (2020)Guu et al. (2020) REALM (2×110M) BERT-base (110M) 40.4 – – 
DPR (best variant) Karpukhin et al. (2020) DPR (2×110M) BERT-base (110M) 41.5 57.9 36.7 / – 
RAG (generative) Lewis et al. (2020) DPR (joint) (2×110M) BART-large (406M) 44.5 56.1 – 
 
Our Models 
ColBERT-QA (base) ColBERT-QA3 (110M) BERT-base (110M) 42.3 64.6 47.7 / 53.5 
ColBERT-QA (large) ColBERT-QA3 (110M) BERT-WWM (330M) 47.8 70.1 54.7 / 58.7 
Close Modal

or Create an Account

Close Modal
Close Modal