Example summaries with the corresponding averaged expert and crowd-sourced annotations for coherence, consistency, fluency, and relevance. Expert annotations better differentiate coherence, consistency, and fluency among the examples when compared to the crowd-sourced annotations.
| Generated Summaries . | Expert scores (avg.) . | Crowd-worker scores (avg.) . |
|---|---|---|
| the queen’s guard was left red-faced after he slipped on a he | Coh: 5.0 | Coh: 3.4 |
| manhole cover lost his footing and slid sideways, knocking his | Con: 5.0 | Con: 3.8 |
| bearskin on the side . the embarrassed soldier quickly scrambled | Flu: 5.0 | Flu: 3.4 |
| to his feet as his colleagues marched past as if nothing had | Rel: 5.0 | Rel: 3.8 |
| happened. tourist david meadwell recorded the unscheduled | ||
| manouevre outside buckingham palace on thursday afternoon. | ||
| holidaymaker david meadwell recorded the unscheduled | Coh: 2.7 | Coh: 3.2 |
| manouevre outside buckingham palace. he lost his footing and | Con: 2.0 | Con: 3.4 |
| slid sideways, knocking bearskin on the side of the box. queen ’s | Flu: 4.7 | Flu: 3.4 |
| guard was left red-faced after he slipped on manhole cover. | Rel: 3.7 | Rel: 4.0 |
| the entire incident was caught on a manhole cover. the embarrassed | ||
| soldier quickly scrambled to his feet as his colleagues marched past. | ||
| buckingham palace guard slipped on manhole cover in front | Coh: 3.3 | Coh: 3.0 |
| of hundreds of horrified tourists. the queen ’s guard was left | Con: 5.0 | Con: 3.2 |
| red-faced after he slipped on a manhole cover. he lost his footing | Flu: 1.7 | Flu: 2.8 |
| and dropped his rifle on the side of the box and dropping his rifle. | Rel: 4.3 | Rel: 3.2 |
| the incident was caught on camera camera camera. the guard is | ||
| thought to have slipped because of metal shutters nailed to the | ||
| soles of his boots. | ||
| (a) Generated summary examples illustrate common problems found in model outputs, such as ambiguous pronouns, incorrect references, and repetitive content. | ||
| Reference Summaries | Expert scores (avg.) | Crowd-worker scores (avg.) |
| river plate admit they ‘ dream ’ of manchester united striker | Coh: 3.0 | Coh: 3.0 |
| radamel falcao. the colombia international spent eight years | Con: 2.0 | Con: 3.6 |
| with the argentine club. falcao has managed just four goals in | Flu: 5.0 | Flu: 3.0 |
| 19 premier league appearances. read : falcao still ‘ has faith ’ | Rel: 2.3 | Rel: 4.4 |
| that he could continue at man utd next season. click here for | ||
| the latest manchester united news. | ||
| the incident occurred on april 7 north of poland in the baltic | Coh: 2.0 | Coh: 4.0 |
| sea . u.s. says plane was in international airspace. russia says | Con: 1.7 | Con: 3.4 |
| it had transponder turned off and was flying toward russia | Flu: 3.0 | Flu: 4.2 |
| Rel: 2.3 | Rel: 3.6 | |
| (b) Reference summaries highlight issues found in theCNN/DailyMail dataset, such as click-baits and references to other articles as well as unreferenced dates and lowcoherence caused by concatenating bullet-point summaries. | ||
| Generated Summaries . | Expert scores (avg.) . | Crowd-worker scores (avg.) . |
|---|---|---|
| the queen’s guard was left red-faced after he slipped on a he | Coh: 5.0 | Coh: 3.4 |
| manhole cover lost his footing and slid sideways, knocking his | Con: 5.0 | Con: 3.8 |
| bearskin on the side . the embarrassed soldier quickly scrambled | Flu: 5.0 | Flu: 3.4 |
| to his feet as his colleagues marched past as if nothing had | Rel: 5.0 | Rel: 3.8 |
| happened. tourist david meadwell recorded the unscheduled | ||
| manouevre outside buckingham palace on thursday afternoon. | ||
| holidaymaker david meadwell recorded the unscheduled | Coh: 2.7 | Coh: 3.2 |
| manouevre outside buckingham palace. he lost his footing and | Con: 2.0 | Con: 3.4 |
| slid sideways, knocking bearskin on the side of the box. queen ’s | Flu: 4.7 | Flu: 3.4 |
| guard was left red-faced after he slipped on manhole cover. | Rel: 3.7 | Rel: 4.0 |
| the entire incident was caught on a manhole cover. the embarrassed | ||
| soldier quickly scrambled to his feet as his colleagues marched past. | ||
| buckingham palace guard slipped on manhole cover in front | Coh: 3.3 | Coh: 3.0 |
| of hundreds of horrified tourists. the queen ’s guard was left | Con: 5.0 | Con: 3.2 |
| red-faced after he slipped on a manhole cover. he lost his footing | Flu: 1.7 | Flu: 2.8 |
| and dropped his rifle on the side of the box and dropping his rifle. | Rel: 4.3 | Rel: 3.2 |
| the incident was caught on camera camera camera. the guard is | ||
| thought to have slipped because of metal shutters nailed to the | ||
| soles of his boots. | ||
| (a) Generated summary examples illustrate common problems found in model outputs, such as ambiguous pronouns, incorrect references, and repetitive content. | ||
| Reference Summaries | Expert scores (avg.) | Crowd-worker scores (avg.) |
| river plate admit they ‘ dream ’ of manchester united striker | Coh: 3.0 | Coh: 3.0 |
| radamel falcao. the colombia international spent eight years | Con: 2.0 | Con: 3.6 |
| with the argentine club. falcao has managed just four goals in | Flu: 5.0 | Flu: 3.0 |
| 19 premier league appearances. read : falcao still ‘ has faith ’ | Rel: 2.3 | Rel: 4.4 |
| that he could continue at man utd next season. click here for | ||
| the latest manchester united news. | ||
| the incident occurred on april 7 north of poland in the baltic | Coh: 2.0 | Coh: 4.0 |
| sea . u.s. says plane was in international airspace. russia says | Con: 1.7 | Con: 3.4 |
| it had transponder turned off and was flying toward russia | Flu: 3.0 | Flu: 4.2 |
| Rel: 2.3 | Rel: 3.6 | |
| (b) Reference summaries highlight issues found in theCNN/DailyMail dataset, such as click-baits and references to other articles as well as unreferenced dates and lowcoherence caused by concatenating bullet-point summaries. | ||