Skip to main content

Research Repository

Advanced Search

Music recommenders: user evaluation without real users?

Craw, Susan; Horsburgh, Ben; Massie, Stewart


Ben Horsburgh


Qiang Yang

Michael Woolridge


Good music recommenders should not only suggest quality recommendations, but should also allow users to discover new/niche music. User studies capture explicit feedback on recommendation quality and novelty, but can be expensive, and may have difficulty replicating realistic scenarios. Lack of effective offline evaluation methods restricts progress in music recommendation research. The challenge is finding suitable measures to score recommendation quality, and in particular avoiding popularity bias, whereby the quality is not recognised when the track is not well known. This paper presents a low cost method that leverages available social media data and shows it to be effective. Not only is it based on explicit feedback from many users, but it also overcomes the popularity bias that disadvantages new/niche music. Experiments show that its findings are consistent with those from an online study with real users. In comparisons with other offline measures, the social media score is shown to be a more reliable proxy for opinions of real users. Its impact on music recommendation is its ability to recognise recommenders that enable discovery, as well as suggest quality recommendations.


CRAW, S., HORSBURGH, B. and MASSIE, S. 2015. Music recommenders: user evaluation without real users? In Yang, Q. and Woolridge, M. (eds.) Proceedings of the 24th International joint conference on artificial intelligence (IJCAI-15), 25-31 July 2015, Buenos Aires, Argentina. Palo Alto: AAAI Press [online], pages 1749-1755. Available from:

Conference Name 24th International joint conference on artificial intelligence (IJCAI-15)
Conference Location Buenos Aires, Argentina
Start Date Jul 25, 2015
End Date Jul 31, 2015
Acceptance Date Apr 16, 2015
Online Publication Date Jul 25, 2015
Publication Date Dec 31, 2015
Deposit Date Aug 10, 2015
Publicly Available Date Aug 10, 2015
Publisher Association for the Advancement of Artificial Intelligence
Pages 1749-1755
ISBN 9781577357384
Public URL
Publisher URL


You might also like

Downloadable Citations