Skip to Main Content
Table 2:
Test Error Rates (%) on the 20 bAbI QA Tasks for Models Using 10,000 Training Examples with the Feedforward Controller.
TaskContinuous D-NTMDiscrete D-NTMDiscrete D-NTMDiscrete D-NTM
1: One supporting fact 4.38 81.67 14.79 72.28 
2: Two supporting facts 27.5 76.67 76.67 81.67 
3: Three supporting facts 71.25 79.38 70.83 78.95 
4: Two argument relations 0.00 78.65 44.06 79.69 
5: Three argument relations 1.67 83.13 17.71 68.54 
6: Yes/no questions 1.46 48.76 48.13 31.67 
7: Counting 6.04 54.79 23.54 49.17 
8: Lists/sets 1.70 69.75 35.62 79.32 
9: Simple negation 0.63 39.17 14.38 37.71 
10: Indefinite knowledge 19.80 56.25 56.25 25.63 
11: Basic coreference 0.00 78.96 39.58 82.08 
12: Conjunction 6.25 82.5 32.08 74.38 
13: Compound coreference 7.5 75.0 18.54 47.08 
14: Time reasoning 17.5 78.75 24.79 77.08 
15: Basic deduction 0.0 71.42 39.73 73.96 
16: Basic induction 49.65 71.46 71.15 53.02 
17: Positional reasoning 1.25 43.75 43.75 30.42 
18: Size reasoning 0.24 48.13 2.92 11.46 
19: Path finding 39.47 71.46 71.56 76.05 
20: Agent motivation 0.0 76.56 9.79 13.96 
Average error 12.81 68.30 37.79 57.21 
Failed (err. 5%) 20 19 20 
TaskContinuous D-NTMDiscrete D-NTMDiscrete D-NTMDiscrete D-NTM
1: One supporting fact 4.38 81.67 14.79 72.28 
2: Two supporting facts 27.5 76.67 76.67 81.67 
3: Three supporting facts 71.25 79.38 70.83 78.95 
4: Two argument relations 0.00 78.65 44.06 79.69 
5: Three argument relations 1.67 83.13 17.71 68.54 
6: Yes/no questions 1.46 48.76 48.13 31.67 
7: Counting 6.04 54.79 23.54 49.17 
8: Lists/sets 1.70 69.75 35.62 79.32 
9: Simple negation 0.63 39.17 14.38 37.71 
10: Indefinite knowledge 19.80 56.25 56.25 25.63 
11: Basic coreference 0.00 78.96 39.58 82.08 
12: Conjunction 6.25 82.5 32.08 74.38 
13: Compound coreference 7.5 75.0 18.54 47.08 
14: Time reasoning 17.5 78.75 24.79 77.08 
15: Basic deduction 0.0 71.42 39.73 73.96 
16: Basic induction 49.65 71.46 71.15 53.02 
17: Positional reasoning 1.25 43.75 43.75 30.42 
18: Size reasoning 0.24 48.13 2.92 11.46 
19: Path finding 39.47 71.46 71.56 76.05 
20: Agent motivation 0.0 76.56 9.79 13.96 
Average error 12.81 68.30 37.79 57.21 
Failed (err. 5%) 20 19 20 

Notes: The discrete D-NTM model bootstraps the discrete attention with the continuous attention, using the curriculum method introduced in section 3.2. The discrete D-NTM model is the continuous-attention model that uses discrete attention at test time. The number in bold indicates the best performance.

Close Modal

or Create an Account

Close Modal
Close Modal