Skip to Main Content
Table 1: 

Example summaries with the corresponding averaged expert and crowd-sourced annotations for coherence, consistency, fluency, and relevance. Expert annotations better differentiate coherence, consistency, and fluency among the examples when compared to the crowd-sourced annotations.

Generated SummariesExpert scores (avg.)Crowd-worker scores (avg.)
the queen’s guard was left red-faced after he slipped on a he Coh: 5.0 Coh: 3.4 
manhole cover lost his footing and slid sideways, knocking his Con: 5.0 Con: 3.8 
bearskin on the side . the embarrassed soldier quickly scrambled Flu: 5.0 Flu: 3.4 
to his feet as his colleagues marched past as if nothing had Rel: 5.0 Rel: 3.8 
happened. tourist david meadwell recorded the unscheduled   
manouevre outside buckingham palace on thursday afternoon.   
 
holidaymaker david meadwell recorded the unscheduled Coh: 2.7 Coh: 3.2 
manouevre outside buckingham palace. he lost his footing and Con: 2.0 Con: 3.4 
slid sideways, knocking bearskin on the side of the box. queen ’s Flu: 4.7 Flu: 3.4 
guard was left red-faced after he slipped on manhole cover. Rel: 3.7 Rel: 4.0 
the entire incident was caught on a manhole cover. the embarrassed   
soldier quickly scrambled to his feet as his colleagues marched past.   
 
buckingham palace guard slipped on manhole cover in front Coh: 3.3 Coh: 3.0 
of hundreds of horrified tourists. the queen ’s guard was left Con: 5.0 Con: 3.2 
red-faced after he slipped on a manhole cover. he lost his footing Flu: 1.7 Flu: 2.8 
and dropped his rifle on the side of the box and dropping his rifle. Rel: 4.3 Rel: 3.2 
the incident was caught on camera camera camera. the guard is   
thought to have slipped because of metal shutters nailed to the   
soles of his boots.   
 
(a) Generated summary examples illustrate common problems found in model outputs, such as ambiguous pronouns, incorrect references, and repetitive content. 
Reference Summaries Expert scores (avg.) Crowd-worker scores (avg.) 
river plate admit they ‘ dream ’ of manchester united striker Coh: 3.0 Coh: 3.0 
radamel falcao. the colombia international spent eight years Con: 2.0 Con: 3.6 
with the argentine club. falcao has managed just four goals in Flu: 5.0 Flu: 3.0 
19 premier league appearances. read : falcao still ‘ has faith ’ Rel: 2.3 Rel: 4.4 
that he could continue at man utd next season. click here for   
the latest manchester united news.   
 
the incident occurred on april 7 north of poland in the baltic Coh: 2.0 Coh: 4.0 
sea . u.s. says plane was in international airspace. russia says Con: 1.7 Con: 3.4 
it had transponder turned off and was flying toward russia Flu: 3.0 Flu: 4.2 
 Rel: 2.3 Rel: 3.6 
 
(b) Reference summaries highlight issues found in theCNN/DailyMail dataset, such as click-baits and references to other articles as well as unreferenced dates and lowcoherence caused by concatenating bullet-point summaries. 
Generated SummariesExpert scores (avg.)Crowd-worker scores (avg.)
the queen’s guard was left red-faced after he slipped on a he Coh: 5.0 Coh: 3.4 
manhole cover lost his footing and slid sideways, knocking his Con: 5.0 Con: 3.8 
bearskin on the side . the embarrassed soldier quickly scrambled Flu: 5.0 Flu: 3.4 
to his feet as his colleagues marched past as if nothing had Rel: 5.0 Rel: 3.8 
happened. tourist david meadwell recorded the unscheduled   
manouevre outside buckingham palace on thursday afternoon.   
 
holidaymaker david meadwell recorded the unscheduled Coh: 2.7 Coh: 3.2 
manouevre outside buckingham palace. he lost his footing and Con: 2.0 Con: 3.4 
slid sideways, knocking bearskin on the side of the box. queen ’s Flu: 4.7 Flu: 3.4 
guard was left red-faced after he slipped on manhole cover. Rel: 3.7 Rel: 4.0 
the entire incident was caught on a manhole cover. the embarrassed   
soldier quickly scrambled to his feet as his colleagues marched past.   
 
buckingham palace guard slipped on manhole cover in front Coh: 3.3 Coh: 3.0 
of hundreds of horrified tourists. the queen ’s guard was left Con: 5.0 Con: 3.2 
red-faced after he slipped on a manhole cover. he lost his footing Flu: 1.7 Flu: 2.8 
and dropped his rifle on the side of the box and dropping his rifle. Rel: 4.3 Rel: 3.2 
the incident was caught on camera camera camera. the guard is   
thought to have slipped because of metal shutters nailed to the   
soles of his boots.   
 
(a) Generated summary examples illustrate common problems found in model outputs, such as ambiguous pronouns, incorrect references, and repetitive content. 
Reference Summaries Expert scores (avg.) Crowd-worker scores (avg.) 
river plate admit they ‘ dream ’ of manchester united striker Coh: 3.0 Coh: 3.0 
radamel falcao. the colombia international spent eight years Con: 2.0 Con: 3.6 
with the argentine club. falcao has managed just four goals in Flu: 5.0 Flu: 3.0 
19 premier league appearances. read : falcao still ‘ has faith ’ Rel: 2.3 Rel: 4.4 
that he could continue at man utd next season. click here for   
the latest manchester united news.   
 
the incident occurred on april 7 north of poland in the baltic Coh: 2.0 Coh: 4.0 
sea . u.s. says plane was in international airspace. russia says Con: 1.7 Con: 3.4 
it had transponder turned off and was flying toward russia Flu: 3.0 Flu: 4.2 
 Rel: 2.3 Rel: 3.6 
 
(b) Reference summaries highlight issues found in theCNN/DailyMail dataset, such as click-baits and references to other articles as well as unreferenced dates and lowcoherence caused by concatenating bullet-point summaries. 
Close Modal

or Create an Account

Close Modal
Close Modal