Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices
Publication date
2024-09
Authors
Schmidtová, Patrícia
Mahamood, Saad
Balloccu, Simone
Dušek, Ondřej
Gatt, Albert
Gkatzia, Dimitra
Howcroft, David M.
Plátek, Ondřej
Sivaprasad, Adarsa
Editors
Mahamood, Saad
Minh, Nguyen Le
Ippolito, Daphne
Advisors
Supervisors
Document Type
Part of book
Metadata
Show full item recordCollections
License
cc_by
Abstract
Automatic metrics are extensively used to evaluate natural language processing systems. However, there has been increasing focus on how they are used and reported by practitioners within the field. In this paper, we have conducted a survey on the use of automatic metrics, focusing particularly on natural language generation (NLG) tasks. We inspect which metrics are used as well as why they are chosen and how their use is reported. Our findings from this survey reveal significant shortcomings, including inappropriate metric usage, lack of implementation details and missing correlations with human judgements. We conclude with recommendations that we believe authors should follow to enable more rigour within the field.
Keywords
Information Systems, Software, Computational Theory and Mathematics, Computer Science Applications
Citation
Schmidtová, P, Mahamood, S, Balloccu, S, Dušek, O, Gatt, A, Gkatzia, D, Howcroft, D M, Plátek, O & Sivaprasad, A 2024, Automatic Metrics in Natural Language Generation : A Survey of Current Evaluation Practices. in S Mahamood, N L Minh & D Ippolito (eds), INLG 2024 - 17th International Natural Language Generation Conference, Proceedings of the Conference. INLG 2024 - 17th International Natural Language Generation Conference, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 557-583, 17th International Natural Language Generation Conference, INLG 2024, Tokyo, Japan, 23/09/24. https://doi.org/10.18653/v1/2024.inlg-main.44, conference