Music recommenders: user evaluation without real users?

Craw, Susan; Horsburgh, Ben; Massie, Stewart

Abstract

Good music recommenders should not only suggest quality recommendations, but should also allow users to discover new/niche music. User studies capture explicit feedback on recommendation quality and novelty, but can be expensive, and may have difficulty replicating realistic scenarios. Lack of effective offline evaluation methods restricts progress in music recommendation research. The challenge is finding suitable measures to score recommendation quality, and in particular avoiding popularity bias, whereby the quality is not recognised when the track is not well known. This paper presents a low cost method that leverages available social media data and shows it to be effective. Not only is it based on explicit feedback from many users, but it also overcomes the popularity bias that disadvantages new/niche music. Experiments show that its findings are consistent with those from an online study with real users. In comparisons with other offline measures, the social media score is shown to be a more reliable proxy for opinions of real users. Its impact on music recommendation is its ability to recognise recommenders that enable discovery, as well as suggest quality recommendations.

Citation

CRAW, S., HORSBURGH, B. and MASSIE, S. 2015. Music recommenders: user evaluation without real users? In Yang, Q. and Woolridge, M. (eds.) Proceedings of the 24th International joint conference on artificial intelligence (IJCAI-15), 25-31 July 2015, Buenos Aires, Argentina. Palo Alto: AAAI Press [online], pages 1749-1755. Available from: https://www.ijcai.org/Proceedings/15/Papers/249.pdf

• Conference Name: 24th International joint conference on artificial intelligence (IJCAI-15)
• Conference Location: Buenos Aires, Argentina
• Start Date: Jul 25, 2015
• End Date: Jul 31, 2015
• Acceptance Date: Apr 16, 2015
• Online Publication Date: Jul 25, 2015
• Publication Date: Dec 31, 2015
• Deposit Date: Aug 10, 2015
• Publicly Available Date: Aug 10, 2015
• Publisher: Association for the Advancement of Artificial Intelligence
• Pages: 1749-1755
• ISBN: 9781577357384
• Public URL: http://hdl.handle.net/10059/1265
• Publisher URL: https://www.ijcai.org/Proceedings/15/Papers/249.pdf