Overview of existing work on EBHD of NLP models. We use abbreviations as follows: Task: TC = Text Classification (single input), VQA = Visual Question Answering, TQA = Table Question Answering, NLI = Natural Language Inference / Model: NB = Naive Bayes, SVM = Support Vector Machines, LR = Logistic Regression, TellQA = Telling QA, NeOp = Neural Operator, CNN = Convolutional Neural Networks, BERT* = BERT and RoBERTa / Bug sources: AR = Natural artifacts, SS = Small training subset, WL = Wrong label injection, OD = Out-of-distribution tests / Exp. scope: G = Global explanations, L = Local explanations / Exp. method: SE = Self-explaining, PH = Post-hoc method / Feedback (form): LB = Label, WO = Word(s), WS = Word(s) Score, ES = Example Score, FE = Feature, RU = Rule, AT = Attention, RE = Reasoning / Update: M = Adjust the model parameters, D = Improve the training data, T = Influence the training process / Setting: SP = Selected participants, CS = Crowdsourced participants, SM = Simulation, NR = Not reported.
| Paper . | Context . | Workflow . | Setting . | |||||
|---|---|---|---|---|---|---|---|---|
| Task . | Model . | Bug sources . | Exp. scope . | Exp. method . | Feedback . | Update . | ||
| Kulesza et al. (2009) | TC | NB | AR | G,L | SE | LB,WS | M,D | SP |
| Stumpf et al. (2009) | TC | NB | SS | L | SE | WO | T | SP |
| Kulesza et al. (2010) | TC | NB | SS | G,L | SE | WO,LB | M,D | SP |
| Kulesza et al. (2015) | TC | NB | AR | G,L | SE | WO,WS | M | SP |
| Ribeiro et al. (2016) | TC | SVM | AR | L | PH | WO | D | CS |
| Koh and Liang (2017) | TC | LR | WL | L | PH | LB | D | SM |
| Ribeiro et al. (2018b) | VQA | TellQA | AR | G | PH | RU | D | SP |
| TC | fastText | AR,OD | ||||||
| Teso and Kersting (2019) | TC | LR | AR | L | PH | WO | D | SM |
| Cho et al. (2019) | TQA | NeOp | AR | L | SE | AT | T | NR |
| Khanna et al. (2019) | TC | LR | WL | L | PH | LB | D | SM |
| Lertvittayakumjorn et al. (2020) | TC | CNN | AR,SS,OD | G | PH | FE | T | CS |
| Smith-Renner et al. (2020) | TC | NB | AR,SS | L | SE | LB,WO | M,D | CS |
| Han and Ghosh (2020) | TC | LR | WL | L | PH | LB | D | SM |
| Yao et al. (2021) | TC | BERT* | AR,OD | L | PH | RE | D,T | SP |
| Zylberajch et al. (2021) | NLI | BERT | AR | L | PH | ES | D | SP |
| Paper . | Context . | Workflow . | Setting . | |||||
|---|---|---|---|---|---|---|---|---|
| Task . | Model . | Bug sources . | Exp. scope . | Exp. method . | Feedback . | Update . | ||
| Kulesza et al. (2009) | TC | NB | AR | G,L | SE | LB,WS | M,D | SP |
| Stumpf et al. (2009) | TC | NB | SS | L | SE | WO | T | SP |
| Kulesza et al. (2010) | TC | NB | SS | G,L | SE | WO,LB | M,D | SP |
| Kulesza et al. (2015) | TC | NB | AR | G,L | SE | WO,WS | M | SP |
| Ribeiro et al. (2016) | TC | SVM | AR | L | PH | WO | D | CS |
| Koh and Liang (2017) | TC | LR | WL | L | PH | LB | D | SM |
| Ribeiro et al. (2018b) | VQA | TellQA | AR | G | PH | RU | D | SP |
| TC | fastText | AR,OD | ||||||
| Teso and Kersting (2019) | TC | LR | AR | L | PH | WO | D | SM |
| Cho et al. (2019) | TQA | NeOp | AR | L | SE | AT | T | NR |
| Khanna et al. (2019) | TC | LR | WL | L | PH | LB | D | SM |
| Lertvittayakumjorn et al. (2020) | TC | CNN | AR,SS,OD | G | PH | FE | T | CS |
| Smith-Renner et al. (2020) | TC | NB | AR,SS | L | SE | LB,WO | M,D | CS |
| Han and Ghosh (2020) | TC | LR | WL | L | PH | LB | D | SM |
| Yao et al. (2021) | TC | BERT* | AR,OD | L | PH | RE | D,T | SP |
| Zylberajch et al. (2021) | NLI | BERT | AR | L | PH | ES | D | SP |