Skip to Main Content
Table 10: 
Manual error analysis of the Copynet model predictions. Lower examples are of high-level QDMRs.
QuestionGoldPrediction (Copynet)Analysis
“How many objects other than cylinders are there?” (1) objects; (2) cylinders; (3) #1 besides #2; (4) number of #3. (1) objects; (2) cylinders; (3) number of #1; (4) number of #2; (5) difference of #3 and #4. sem. equiv. (30%) 
“Where is the youngest teacher from?” (1) teachers; (2) the youngest of #1; (3) where is #2 from. (1) youngest teacher; (2) where is #1. incorrect (46%) 
“Kyle York is the Chief Strategy Officer of a company acquired by what corporation in 2016?” (1) company that Kyle York is the Chief Strategy Officer of; (2) corporation that acquired #1 in 2016. (1) company that Kyle York is the Chief Strategy Officer of; (2) corporation in 2016 that #1 was acquired by. sem. equiv. (46%) 
“Dayton’s Devils had a cameo from the ‘MASH’ star who played what role on the show?” (1) MASH star that Dayton ’s Devils had a cameo from; (2) role that #1 played on the show. (1) the MASH that Dayton ’s Devils had a cameo; (2) what role on the show star of #1 played. incorrect (46%) 
QuestionGoldPrediction (Copynet)Analysis
“How many objects other than cylinders are there?” (1) objects; (2) cylinders; (3) #1 besides #2; (4) number of #3. (1) objects; (2) cylinders; (3) number of #1; (4) number of #2; (5) difference of #3 and #4. sem. equiv. (30%) 
“Where is the youngest teacher from?” (1) teachers; (2) the youngest of #1; (3) where is #2 from. (1) youngest teacher; (2) where is #1. incorrect (46%) 
“Kyle York is the Chief Strategy Officer of a company acquired by what corporation in 2016?” (1) company that Kyle York is the Chief Strategy Officer of; (2) corporation that acquired #1 in 2016. (1) company that Kyle York is the Chief Strategy Officer of; (2) corporation in 2016 that #1 was acquired by. sem. equiv. (46%) 
“Dayton’s Devils had a cameo from the ‘MASH’ star who played what role on the show?” (1) MASH star that Dayton ’s Devils had a cameo from; (2) role that #1 played on the show. (1) the MASH that Dayton ’s Devils had a cameo; (2) what role on the show star of #1 played. incorrect (46%) 
Close Modal

or Create an Account

Close Modal
Close Modal