GramError: a quality metric for machine generated songs.

Davies, Craig; Wiratunga, Nirmalie; Martin, Kyle

Abstract

This paper explores whether a simple grammar-based metric can accurately predict human opinion of machine-generated song lyrics quality. The proposed metric considers the percentage of words written in natural English and the number of grammatical errors to rate the quality of machine-generated lyrics. We use a state-of-the-art Recurrent Neural Network (RNN) model and adapt it to lyric generation by re-training on the lyrics of 5,000 songs. For our initial user trial, we use a small sample of songs generated by the RNN to calibrate the metric. Songs selected on the basis of this metric are further evaluated using ”Turinglike” tests to establish whether there is a correlation between metric score and human judgment. Our results show that there is strong correlation with human opinion, especially at lower levels of song quality. They also show that 75% of the RNN-generated lyrics passed for human-generated over 30% of the time.

Citation

DAVIES, C., WIRATUNGA, N. and MARTIN, K. 2018. GramError: a quality metric for machine generated songs. In Bramer, M. and Petridis, M. (eds.) Artificial intelligence XXXV: proceedings of the 38th British Computer Society's Specialist Group on Artificial Intelligence (SGAI) International conference on innovative techniques and applications of artificial intelligence (AI-2018), 11-13 December 2018, Cambridge, UK. Lecture notes in computer science, 11311. Cham: Springer [online], pages 184-190. Available from: https://doi.org/10.1007/978-3-030-04191-5_16

• Conference Name: 38th British Computer Society's Specialist Group on Artificial Intelligence (SGAI) International conference on innovative techniques and applications of artificial intelligence (AI-2018)
• Conference Location: Cambridge, UK
• Start Date: Dec 11, 2018
• End Date: Dec 13, 2018
• Acceptance Date: Sep 3, 2018
• Online Publication Date: Nov 16, 2018
• Publication Date: Dec 31, 2018
• Deposit Date: Jan 21, 2019
• Publicly Available Date: Nov 17, 2019
• Publisher: Springer
• Pages: 184-190
• Series Title: Lecture notes in computer science
• Series Number: 11311
• Series ISSN: 0302-9743
• Book Title: Artificial intelligence XXXV
• ISBN: 9783030041908
• DOI: https://doi.org/10.1007/978-3-030-04191-5_16
• Keywords: Natural language generation; Quality metric; Recurrent neural; Network
• Public URL: http://hdl.handle.net/10059/3271