Skip to Main Content
Table 5:
Test Error Rates (%) on the 20 bAbI QA Tasks for Models Using 10,000 Training Examples with the GRU Controller and Representations of Facts Obtained with BoW Using Positional Encoding.
TaskSoft D-NTM (1-step)Discrete D-NTM (1-step)Soft D-NTM (3-steps)Discrete D-NTM (3-steps)
1: One supporting fact 0.00 0.00 0.00 0.00 
2: Two supporting facts 61.04 59.37 56.87 55.62 
3: Three supporting facts 55.62 57.5 62.5 57.5 
4: Two argument relations 27.29 24.89 26.45 27.08 
5: Three argument relations 13.55 12.08 15.83 14.78 
6: Yes/no questions 13.54 14.37 21.87 13.33 
7: Counting 8.54 6.25 8.75 14.58 
8: Lists/sets 1.69 1.36 3.01 3.02 
9: Simple negation 17.7 16.66 37.70 17.08 
10: Indefinite knowledge 26.04 27.08 26.87 23.95 
11: Basic coreference 20.41 3.95 2.5 2.29 
12: Conjunction 0.41 0.83 0.20 4.16 
13: Compound coreference 3.12 1.04 4.79 5.83 
14: Time reasoning 62.08 58.33 61.25 60.62 
15: Basic deduction 31.66 26.25 0.62 0.05 
16: Basic induction 54.47 48.54 48.95 48.95 
17: Positional reasoning 43.75 31.87 43.75 30.62 
18: Size reasoning 33.75 39.37 36.66 36.04 
19: Path finding 64.63 69.21 67.23 65.46 
20: Agent motivation 1.25 0.00 1.45 0.00 
Average error (%) 27.02 24.98 26.36 24.05 
Falied (err. 5%) 15 14 13 14 
TaskSoft D-NTM (1-step)Discrete D-NTM (1-step)Soft D-NTM (3-steps)Discrete D-NTM (3-steps)
1: One supporting fact 0.00 0.00 0.00 0.00 
2: Two supporting facts 61.04 59.37 56.87 55.62 
3: Three supporting facts 55.62 57.5 62.5 57.5 
4: Two argument relations 27.29 24.89 26.45 27.08 
5: Three argument relations 13.55 12.08 15.83 14.78 
6: Yes/no questions 13.54 14.37 21.87 13.33 
7: Counting 8.54 6.25 8.75 14.58 
8: Lists/sets 1.69 1.36 3.01 3.02 
9: Simple negation 17.7 16.66 37.70 17.08 
10: Indefinite knowledge 26.04 27.08 26.87 23.95 
11: Basic coreference 20.41 3.95 2.5 2.29 
12: Conjunction 0.41 0.83 0.20 4.16 
13: Compound coreference 3.12 1.04 4.79 5.83 
14: Time reasoning 62.08 58.33 61.25 60.62 
15: Basic deduction 31.66 26.25 0.62 0.05 
16: Basic induction 54.47 48.54 48.95 48.95 
17: Positional reasoning 43.75 31.87 43.75 30.62 
18: Size reasoning 33.75 39.37 36.66 36.04 
19: Path finding 64.63 69.21 67.23 65.46 
20: Agent motivation 1.25 0.00 1.45 0.00 
Average error (%) 27.02 24.98 26.36 24.05 
Falied (err. 5%) 15 14 13 14 

Note: The number in bold indicates the best performance.

Close Modal

or Create an Account

Close Modal
Close Modal