Skip to Main Content
Table 1:
Test Error Rates (%) on the 20 bAbI QA Tasks for Models Using 10,000 Training Examples.
TaskLSTM1-Step LBA+CBA NTM1-Step CBA NTM1-Step Continuous D-NTM1-Step Discrete D-NTM3-Step LBA+CBA NTM3-Step CBA NTM3-Step Continuous D-NTM3-Step Discrete D-NTM
1: One supporting fact 0.00 16.30 16.88 5.41 6.66 0.00 0.00 0.00 0.00 
2: Two supporting facts 81.90 57.08 55.70 58.54 56.04 61.67 59.38 46.66 62.29 
3: Three supporting facts 83.10 74.16 55.00 74.58 72.08 83.54 65.21 47.08 41.45 
4: Two argument relations 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 
5: Three argument relations 1.20 1.46 20.41 1.66 1.04 0.83 1.46 1.25 1.45 
6: Yes/no questions 51.80 23.33 21.04 40.20 44.79 48.13 54.80 20.62 11.04 
7: Counting 24.90 21.67 21.67 19.16 19.58 7.92 37.70 7.29 5.62 
8: Lists/sets 34.10 25.76 21.05 12.58 18.46 25.38 8.82 11.02 0.74 
9: Simple negation 20.20 24.79 24.17 36.66 34.37 37.80 0.00 39.37 32.50 
10: Indefinite knowledge 30.10 41.46 33.13 52.29 50.83 56.25 23.75 20.00 20.83 
11: Basic coreference 10.30 18.96 31.88 31.45 4.16 3.96 0.28 30.62 16.87 
12: Conjunction 23.40 25.83 30.00 7.70 6.66 28.75 23.75 5.41 4.58 
13: Compound coreference 6.10 6.67 5.63 5.62 2.29 5.83 83.13 7.91 5.00 
14: Time reasoning 81.00 58.54 59.17 60.00 63.75 61.88 57.71 58.12 60.20 
15: Basic deduction 78.70 36.46 42.30 36.87 39.27 35.62 21.88 36.04 40.26 
16: Basic induction 51.90 71.15 71.15 49.16 51.35 46.15 50.00 46.04 45.41 
17: Positional reasoning 50.10 43.75 43.75 17.91 16.04 43.75 56.25 21.25 9.16 
18: Size reasoning 6.80 3.96 47.50 3.95 3.54 47.50 47.50 6.87 1.66 
19: Path finding 90.30 75.89 71.51 73.74 64.63 61.56 63.65 75.88 76.66 
20: Agent motivation 2.10 1.25 0.00 2.70 3.12 0.40 0.00 3.33 0.00 
Average error 36.41 31.42 33.60 29.51 27.93 32.85 32.76 24.24 21.79 
Failed (err. 5%) 16 16 18 16 14 15 14 16 12 
TaskLSTM1-Step LBA+CBA NTM1-Step CBA NTM1-Step Continuous D-NTM1-Step Discrete D-NTM3-Step LBA+CBA NTM3-Step CBA NTM3-Step Continuous D-NTM3-Step Discrete D-NTM
1: One supporting fact 0.00 16.30 16.88 5.41 6.66 0.00 0.00 0.00 0.00 
2: Two supporting facts 81.90 57.08 55.70 58.54 56.04 61.67 59.38 46.66 62.29 
3: Three supporting facts 83.10 74.16 55.00 74.58 72.08 83.54 65.21 47.08 41.45 
4: Two argument relations 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 
5: Three argument relations 1.20 1.46 20.41 1.66 1.04 0.83 1.46 1.25 1.45 
6: Yes/no questions 51.80 23.33 21.04 40.20 44.79 48.13 54.80 20.62 11.04 
7: Counting 24.90 21.67 21.67 19.16 19.58 7.92 37.70 7.29 5.62 
8: Lists/sets 34.10 25.76 21.05 12.58 18.46 25.38 8.82 11.02 0.74 
9: Simple negation 20.20 24.79 24.17 36.66 34.37 37.80 0.00 39.37 32.50 
10: Indefinite knowledge 30.10 41.46 33.13 52.29 50.83 56.25 23.75 20.00 20.83 
11: Basic coreference 10.30 18.96 31.88 31.45 4.16 3.96 0.28 30.62 16.87 
12: Conjunction 23.40 25.83 30.00 7.70 6.66 28.75 23.75 5.41 4.58 
13: Compound coreference 6.10 6.67 5.63 5.62 2.29 5.83 83.13 7.91 5.00 
14: Time reasoning 81.00 58.54 59.17 60.00 63.75 61.88 57.71 58.12 60.20 
15: Basic deduction 78.70 36.46 42.30 36.87 39.27 35.62 21.88 36.04 40.26 
16: Basic induction 51.90 71.15 71.15 49.16 51.35 46.15 50.00 46.04 45.41 
17: Positional reasoning 50.10 43.75 43.75 17.91 16.04 43.75 56.25 21.25 9.16 
18: Size reasoning 6.80 3.96 47.50 3.95 3.54 47.50 47.50 6.87 1.66 
19: Path finding 90.30 75.89 71.51 73.74 64.63 61.56 63.65 75.88 76.66 
20: Agent motivation 2.10 1.25 0.00 2.70 3.12 0.40 0.00 3.33 0.00 
Average error 36.41 31.42 33.60 29.51 27.93 32.85 32.76 24.24 21.79 
Failed (err. 5%) 16 16 18 16 14 15 14 16 12 

Notes: LBA: location-based addressing; CBA: content-based addressing. D-NTM models use a GRU controller. In this table, we compare multistep versus single-step addressing, original NTM with location-based + content-based addressing versus only content-based addressing, and discrete versus continuous addressing D-NTM on bAbI. The number in bold indicates the best performance.

Close Modal

or Create an Account

Close Modal
Close Modal