Example summaries with the corresponding averaged expert and crowd-sourced annotations for coherence, consistency, fluency, and relevance. Expert annotations better differentiate coherence, consistency, and fluency among the examples when compared to the crowd-sourced annotations.
Generated Summaries . | Expert scores (avg.) . | Crowd-worker scores (avg.) . |
---|---|---|
the queen’s guard was left red-faced after he slipped on a he | Coh: 5.0 | Coh: 3.4 |
manhole cover lost his footing and slid sideways, knocking his | Con: 5.0 | Con: 3.8 |
bearskin on the side . the embarrassed soldier quickly scrambled | Flu: 5.0 | Flu: 3.4 |
to his feet as his colleagues marched past as if nothing had | Rel: 5.0 | Rel: 3.8 |
happened. tourist david meadwell recorded the unscheduled | ||
manouevre outside buckingham palace on thursday afternoon. | ||
holidaymaker david meadwell recorded the unscheduled | Coh: 2.7 | Coh: 3.2 |
manouevre outside buckingham palace. he lost his footing and | Con: 2.0 | Con: 3.4 |
slid sideways, knocking bearskin on the side of the box. queen ’s | Flu: 4.7 | Flu: 3.4 |
guard was left red-faced after he slipped on manhole cover. | Rel: 3.7 | Rel: 4.0 |
the entire incident was caught on a manhole cover. the embarrassed | ||
soldier quickly scrambled to his feet as his colleagues marched past. | ||
buckingham palace guard slipped on manhole cover in front | Coh: 3.3 | Coh: 3.0 |
of hundreds of horrified tourists. the queen ’s guard was left | Con: 5.0 | Con: 3.2 |
red-faced after he slipped on a manhole cover. he lost his footing | Flu: 1.7 | Flu: 2.8 |
and dropped his rifle on the side of the box and dropping his rifle. | Rel: 4.3 | Rel: 3.2 |
the incident was caught on camera camera camera. the guard is | ||
thought to have slipped because of metal shutters nailed to the | ||
soles of his boots. | ||
(a) Generated summary examples illustrate common problems found in model outputs, such as ambiguous pronouns, incorrect references, and repetitive content. | ||
Reference Summaries | Expert scores (avg.) | Crowd-worker scores (avg.) |
river plate admit they ‘ dream ’ of manchester united striker | Coh: 3.0 | Coh: 3.0 |
radamel falcao. the colombia international spent eight years | Con: 2.0 | Con: 3.6 |
with the argentine club. falcao has managed just four goals in | Flu: 5.0 | Flu: 3.0 |
19 premier league appearances. read : falcao still ‘ has faith ’ | Rel: 2.3 | Rel: 4.4 |
that he could continue at man utd next season. click here for | ||
the latest manchester united news. | ||
the incident occurred on april 7 north of poland in the baltic | Coh: 2.0 | Coh: 4.0 |
sea . u.s. says plane was in international airspace. russia says | Con: 1.7 | Con: 3.4 |
it had transponder turned off and was flying toward russia | Flu: 3.0 | Flu: 4.2 |
Rel: 2.3 | Rel: 3.6 | |
(b) Reference summaries highlight issues found in theCNN/DailyMail dataset, such as click-baits and references to other articles as well as unreferenced dates and lowcoherence caused by concatenating bullet-point summaries. |
Generated Summaries . | Expert scores (avg.) . | Crowd-worker scores (avg.) . |
---|---|---|
the queen’s guard was left red-faced after he slipped on a he | Coh: 5.0 | Coh: 3.4 |
manhole cover lost his footing and slid sideways, knocking his | Con: 5.0 | Con: 3.8 |
bearskin on the side . the embarrassed soldier quickly scrambled | Flu: 5.0 | Flu: 3.4 |
to his feet as his colleagues marched past as if nothing had | Rel: 5.0 | Rel: 3.8 |
happened. tourist david meadwell recorded the unscheduled | ||
manouevre outside buckingham palace on thursday afternoon. | ||
holidaymaker david meadwell recorded the unscheduled | Coh: 2.7 | Coh: 3.2 |
manouevre outside buckingham palace. he lost his footing and | Con: 2.0 | Con: 3.4 |
slid sideways, knocking bearskin on the side of the box. queen ’s | Flu: 4.7 | Flu: 3.4 |
guard was left red-faced after he slipped on manhole cover. | Rel: 3.7 | Rel: 4.0 |
the entire incident was caught on a manhole cover. the embarrassed | ||
soldier quickly scrambled to his feet as his colleagues marched past. | ||
buckingham palace guard slipped on manhole cover in front | Coh: 3.3 | Coh: 3.0 |
of hundreds of horrified tourists. the queen ’s guard was left | Con: 5.0 | Con: 3.2 |
red-faced after he slipped on a manhole cover. he lost his footing | Flu: 1.7 | Flu: 2.8 |
and dropped his rifle on the side of the box and dropping his rifle. | Rel: 4.3 | Rel: 3.2 |
the incident was caught on camera camera camera. the guard is | ||
thought to have slipped because of metal shutters nailed to the | ||
soles of his boots. | ||
(a) Generated summary examples illustrate common problems found in model outputs, such as ambiguous pronouns, incorrect references, and repetitive content. | ||
Reference Summaries | Expert scores (avg.) | Crowd-worker scores (avg.) |
river plate admit they ‘ dream ’ of manchester united striker | Coh: 3.0 | Coh: 3.0 |
radamel falcao. the colombia international spent eight years | Con: 2.0 | Con: 3.6 |
with the argentine club. falcao has managed just four goals in | Flu: 5.0 | Flu: 3.0 |
19 premier league appearances. read : falcao still ‘ has faith ’ | Rel: 2.3 | Rel: 4.4 |
that he could continue at man utd next season. click here for | ||
the latest manchester united news. | ||
the incident occurred on april 7 north of poland in the baltic | Coh: 2.0 | Coh: 4.0 |
sea . u.s. says plane was in international airspace. russia says | Con: 1.7 | Con: 3.4 |
it had transponder turned off and was flying toward russia | Flu: 3.0 | Flu: 4.2 |
Rel: 2.3 | Rel: 3.6 | |
(b) Reference summaries highlight issues found in theCNN/DailyMail dataset, such as click-baits and references to other articles as well as unreferenced dates and lowcoherence caused by concatenating bullet-point summaries. |