Skip to Main Content
MethodBLEUCHRFCIDErMETEORLengthStats (cov/comp/den)Repeated (1/2/3)
Extractive Models 
M0 - LEAD-3 11.4270 0.3892 0.2125 0.2141 87.4475 0.9825 / 9.6262 / 57.8001 0.2086 / 0.0310 / 0.0310 
M1 - textbfNEUSUM 12.7784 0.3946 0.2832 0.2183 84.4075 0.9819 / 9.8047 / 32.8574 0.2325 / 0.0531 / 0.0531 
M2 - BanditSum 12.9761 0.3897 0.3305 0.2124 78.5279 0.9836 / 10.2810 / 40.4265 0.2384 / 0.0573 / 0.0573 
M3 - LATENT 12.9725 0.3897 0.3305 0.2123 78.5279 0.9834 / 10.2809 / 40.4095 0.2384 / 0.0573 / 0.0573 
M4 - REFRESH 10.6568 0.4526 0.0677 0.2395 114.5684 0.9850 / 7.1059 / 53.1928 0.2127 / 0.0289 / 0.0289 
M5 - RNES 11.2203 0.4062 0.1559 0.2300 99.9199 0.9938 / 7.9032 / 67.7089 0.2451 / 0.0540 / 0.0540 
M6 - JECS 12.5659 0.4310 0.3090 0.2122 79.7797 0.9874 / 10.1111 / 26.6943 0.2041 / 0.0327 / 0.0327 
M7 - STRASS 7.8330 0.3330 0.2945 0.1607 76.4859 0.9969 / 12.7835 / 59.9498 0.1864 / 0.0343 / 0.0343 
 
Abstractive Models 
M8 - Pointer Generator 13.8247 0.3567 0.5065 0.1860 63.5211 0.9957 / 13.1940 / 26.0880 0.2015 / 0.0375 / 0.0375 
M9 - Fast-abs-rl 12.9812 0.3778 0.4329 0.2014 70.8600 0.9860 / 11.0141 / 9.9859 0.2157 / 0.0370 / 0.0370 
M10 - Bottom-Up 15.1293 0.3523 0.6176 0.1887 56.5715 0.9811 / 14.7771 / 12.6181 0.1856 / 0.0211 / 0.0211 
M11 - Improve-abs 11.9816 0.3715 0.3356 0.2005 75.9512 0.9674 / 10.6043 / 8.9755 0.2499 / 0.0542 / 0.0542 
M12 - Unified-ext-abs 12.8457 0.3786 0.3851 0.2017 74.4663 0.9868 / 10.7510 / 33.1106 0.2177 / 0.0493 / 0.0493 
M13 - ROUGESal 13.8882 0.3668 0.4746 0.1936 66.5575 0.9853 / 13.0369 / 25.2893 0.2102 / 0.0458 / 0.0458 
M14 - Multi-task (Ent + QG ) 14.5276 0.3539 0.5749 0.1831 60.0294 0.9853 / 14.1828 / 22.2296 0.1985 / 0.0411 / 0.0411 
M15 - Closed book decoder 13.4158 0.3675 0.4648 0.1925 68.2858 0.9866 / 12.0588 / 27.3686 0.2074 / 0.0444 / 0.0444 
M16 - SENECA 13.7676 0.3660 0.5233 0.1966 64.9710 0.9880 / 12.3610 / 16.7640 0.2146 / 0.0303 / 0.0303 
M17 - T5 19.3891 0.3833 0.7763 0.2140 59.5288 0.9775 / 14.2002 / 12.9565 0.1810 / 0.0209 / 0.0209 
M18 - NeuralTD 12.9241 0.3783 0.3543 0.2038 74.4033 0.9830 / 10.7768 / 12.4443 0.2645 / 0.0901 / 0.0901 
M19 - BertSum-abs 14.9525 0.3649 0.6240 0.1876 60.8893 0.9517 / 13.9197 / 12.3254 0.1697 / 0.0156 / 0.0156 
M20 - GPT-2 (supervised) 13.9364 0.3678 0.5787 0.1759 51.8352 0.9791 / 15.9839 / 15.4999 0.1875 / 0.0362 / 0.0362 
M21 - UniLM 15.5736 0.4230 0.5294 0.2084 67.1960 0.9685 / 11.5672 / 11.7908 0.1722 / 0.0180 / 0.0180 
M22 - BART 17.1005 0.4271 0.7573 0.2105 62.2989 0.9771 / 12.8811 / 15.2999 0.1627 / 0.0127 / 0.0127 
M23 - Pegasus (dynamic mix) 18.6517 0.4261 0.7280 0.2131 64.1348 0.9438 / 13.7208 / 11.6003 0.1855 / 0.0355 / 0.0081 
M23 - Pegasus (huge news) 17.8102 0.3912 0.6595 0.2189 66.7559 0.9814 / 12.9473 / 14.9850 0.1883 / 0.0251 / 0.0251 
(b) Model scores from other text generation evaluation metrics. 
MethodBLEUCHRFCIDErMETEORLengthStats (cov/comp/den)Repeated (1/2/3)
Extractive Models 
M0 - LEAD-3 11.4270 0.3892 0.2125 0.2141 87.4475 0.9825 / 9.6262 / 57.8001 0.2086 / 0.0310 / 0.0310 
M1 - textbfNEUSUM 12.7784 0.3946 0.2832 0.2183 84.4075 0.9819 / 9.8047 / 32.8574 0.2325 / 0.0531 / 0.0531 
M2 - BanditSum 12.9761 0.3897 0.3305 0.2124 78.5279 0.9836 / 10.2810 / 40.4265 0.2384 / 0.0573 / 0.0573 
M3 - LATENT 12.9725 0.3897 0.3305 0.2123 78.5279 0.9834 / 10.2809 / 40.4095 0.2384 / 0.0573 / 0.0573 
M4 - REFRESH 10.6568 0.4526 0.0677 0.2395 114.5684 0.9850 / 7.1059 / 53.1928 0.2127 / 0.0289 / 0.0289 
M5 - RNES 11.2203 0.4062 0.1559 0.2300 99.9199 0.9938 / 7.9032 / 67.7089 0.2451 / 0.0540 / 0.0540 
M6 - JECS 12.5659 0.4310 0.3090 0.2122 79.7797 0.9874 / 10.1111 / 26.6943 0.2041 / 0.0327 / 0.0327 
M7 - STRASS 7.8330 0.3330 0.2945 0.1607 76.4859 0.9969 / 12.7835 / 59.9498 0.1864 / 0.0343 / 0.0343 
 
Abstractive Models 
M8 - Pointer Generator 13.8247 0.3567 0.5065 0.1860 63.5211 0.9957 / 13.1940 / 26.0880 0.2015 / 0.0375 / 0.0375 
M9 - Fast-abs-rl 12.9812 0.3778 0.4329 0.2014 70.8600 0.9860 / 11.0141 / 9.9859 0.2157 / 0.0370 / 0.0370 
M10 - Bottom-Up 15.1293 0.3523 0.6176 0.1887 56.5715 0.9811 / 14.7771 / 12.6181 0.1856 / 0.0211 / 0.0211 
M11 - Improve-abs 11.9816 0.3715 0.3356 0.2005 75.9512 0.9674 / 10.6043 / 8.9755 0.2499 / 0.0542 / 0.0542 
M12 - Unified-ext-abs 12.8457 0.3786 0.3851 0.2017 74.4663 0.9868 / 10.7510 / 33.1106 0.2177 / 0.0493 / 0.0493 
M13 - ROUGESal 13.8882 0.3668 0.4746 0.1936 66.5575 0.9853 / 13.0369 / 25.2893 0.2102 / 0.0458 / 0.0458 
M14 - Multi-task (Ent + QG ) 14.5276 0.3539 0.5749 0.1831 60.0294 0.9853 / 14.1828 / 22.2296 0.1985 / 0.0411 / 0.0411 
M15 - Closed book decoder 13.4158 0.3675 0.4648 0.1925 68.2858 0.9866 / 12.0588 / 27.3686 0.2074 / 0.0444 / 0.0444 
M16 - SENECA 13.7676 0.3660 0.5233 0.1966 64.9710 0.9880 / 12.3610 / 16.7640 0.2146 / 0.0303 / 0.0303 
M17 - T5 19.3891 0.3833 0.7763 0.2140 59.5288 0.9775 / 14.2002 / 12.9565 0.1810 / 0.0209 / 0.0209 
M18 - NeuralTD 12.9241 0.3783 0.3543 0.2038 74.4033 0.9830 / 10.7768 / 12.4443 0.2645 / 0.0901 / 0.0901 
M19 - BertSum-abs 14.9525 0.3649 0.6240 0.1876 60.8893 0.9517 / 13.9197 / 12.3254 0.1697 / 0.0156 / 0.0156 
M20 - GPT-2 (supervised) 13.9364 0.3678 0.5787 0.1759 51.8352 0.9791 / 15.9839 / 15.4999 0.1875 / 0.0362 / 0.0362 
M21 - UniLM 15.5736 0.4230 0.5294 0.2084 67.1960 0.9685 / 11.5672 / 11.7908 0.1722 / 0.0180 / 0.0180 
M22 - BART 17.1005 0.4271 0.7573 0.2105 62.2989 0.9771 / 12.8811 / 15.2999 0.1627 / 0.0127 / 0.0127 
M23 - Pegasus (dynamic mix) 18.6517 0.4261 0.7280 0.2131 64.1348 0.9438 / 13.7208 / 11.6003 0.1855 / 0.0355 / 0.0081 
M23 - Pegasus (huge news) 17.8102 0.3912 0.6595 0.2189 66.7559 0.9814 / 12.9473 / 14.9850 0.1883 / 0.0251 / 0.0251 
(b) Model scores from other text generation evaluation metrics. 
Close Modal

or Create an Account

Close Modal
Close Modal