Skip to main content

Research Repository

Advanced Search

GramError: a quality metric for machine generated songs.

Davies, Craig; Wiratunga, Nirmalie; Martin, Kyle

Authors

Craig Davies



Contributors

Max Bramer
Editor

Miltos Petridis
Editor

Abstract

This paper explores whether a simple grammar-based metric can accurately predict human opinion of machine-generated song lyrics quality. The proposed metric considers the percentage of words written in natural English and the number of grammatical errors to rate the quality of machine-generated lyrics. We use a state-of-the-art Recurrent Neural Network (RNN) model and adapt it to lyric generation by re-training on the lyrics of 5,000 songs. For our initial user trial, we use a small sample of songs generated by the RNN to calibrate the metric. Songs selected on the basis of this metric are further evaluated using ”Turinglike” tests to establish whether there is a correlation between metric score and human judgment. Our results show that there is strong correlation with human opinion, especially at lower levels of song quality. They also show that 75% of the RNN-generated lyrics passed for human-generated over 30% of the time.

Citation

DAVIES, C., WIRATUNGA, N. and MARTIN, K. 2018. GramError: a quality metric for machine generated songs. In Bramer, M. and Petridis, M. (eds.) Artificial intelligence XXXV: proceedings of the 38th British Computer Society's Specialist Group on Artificial Intelligence (SGAI) International conference on innovative techniques and applications of artificial intelligence (AI-2018), 11-13 December 2018, Cambridge, UK. Lecture notes in computer science, 11311. Cham: Springer [online], pages 184-190. Available from: https://doi.org/10.1007/978-3-030-04191-5_16

Conference Name 38th British Computer Society's Specialist Group on Artificial Intelligence (SGAI) International conference on innovative techniques and applications of artificial intelligence (AI-2018)
Conference Location Cambridge, UK
Start Date Dec 11, 2018
End Date Dec 13, 2018
Acceptance Date Sep 3, 2018
Online Publication Date Nov 16, 2018
Publication Date Dec 31, 2018
Deposit Date Jan 21, 2019
Publicly Available Date Jan 21, 2019
Publisher Springer
Pages 184-190
Series Title Lecture notes in computer science
Series Number 11311
Series ISSN 0302-9743
Book Title Artificial intelligence XXXV
ISBN 9783030041908
DOI https://doi.org/10.1007/978-3-030-04191-5_16
Keywords Natural language generation; Quality metric; Recurrent neural; Network
Public URL http://hdl.handle.net/10059/3271

Files





You might also like



Downloadable Citations