Ablation study on MultiWOZ2.0. There are three types of ablation study. The first is to analyze the effects of the pretrained data. The second is to validate the effects of the designed pretrained tasks. The last is to figure out the effects of IE tools. Results are significant (p < 0.01) comparing the OPAL model and BART model as the initialized TOD model.
| Model . | MultiWOZ2.0 . | |||
|---|---|---|---|---|
| Inform . | Success . | BLEU . | Combined . | |
| OPAL | 89.40 | 81.10 | 18.60 | 103.85 |
| Effect of Pretrained Corpora | ||||
| WIKI | 88.40 | 79.50 | 18.28 | 102.23 |
| TOD | 89.00 | 78.20 | 17.55 | 101.15 |
| REDD | 86.90 | 77.10 | 16.93 | 98.93 |
| Effect of Pretrained Tasks | ||||
| w/o NTG | 87.00 | 80.80 | 16.88 | 100.79 |
| w/o OR | 85.20 | 79.50 | 17.52 | 99.88 |
| Effect of IE Tools | ||||
| OpenIE-Stanford | 88.40 | 79.20 | 17.34 | 101.14 |
| BART | 87.50 | 72.20 | 16.67 | 96.52 |
| Model . | MultiWOZ2.0 . | |||
|---|---|---|---|---|
| Inform . | Success . | BLEU . | Combined . | |
| OPAL | 89.40 | 81.10 | 18.60 | 103.85 |
| Effect of Pretrained Corpora | ||||
| WIKI | 88.40 | 79.50 | 18.28 | 102.23 |
| TOD | 89.00 | 78.20 | 17.55 | 101.15 |
| REDD | 86.90 | 77.10 | 16.93 | 98.93 |
| Effect of Pretrained Tasks | ||||
| w/o NTG | 87.00 | 80.80 | 16.88 | 100.79 |
| w/o OR | 85.20 | 79.50 | 17.52 | 99.88 |
| Effect of IE Tools | ||||
| OpenIE-Stanford | 88.40 | 79.20 | 17.34 | 101.14 |
| BART | 87.50 | 72.20 | 16.67 | 96.52 |