Skip to main content

Research Repository

Advanced Search

Application of aboutness to functional benchmarking in information retrieval.

Wong, Kam-Fai; Song, Dawei; Bruza, Peter; Cheng, Chun-Hung

Authors

Kam-Fai Wong

Dawei Song

Peter Bruza

Chun-Hung Cheng



Abstract

Experimental approaches are widely employed to benchmark the performance of an information retrieval (IR) system. Measurements in terms of recall and precision are computed as performance indicators. Although they are good at assessing the retrieval effectiveness of an IR system, they fail to explore deeper aspects such as its underlying functionality and explain why the system shows such performance. Recently, inductive (i.e., theoretical) evaluation of IR systems has been proposed to circumvent the controversies of the experimental methods. Several studies have adopted the inductive approach, but they mostly focus on theoretical modeling of IR properties by using some meta-logic. In this paper, we propose to use inductive evaluation for functional benchmarking of IR models as a complement of the traditional experimental based performance benchmarking. We define a functional benchmark suite in two stages: (a) the evaluation criteria based on the notion of aboutness; and (b) the formal evaluation methodology using the criteria. The proposed benchmark has been successfully applied to evaluate various well-known classical and logicbased IR models. The functional benchmarking results allow us to compare and analyze the functionality of the different IR models.

Journal Article Type Article
Publication Date Oct 31, 2001
Journal ACM transactions on information systems
Print ISSN 1046-8188
Electronic ISSN 1558-5868
Publisher Association for Computing Machinery
Peer Reviewed Peer Reviewed
Volume 19
Issue 4
Pages 337-370
Institution Citation WONG, K.-F., SONG, D., BRUZA, P. and CHENG, C.-H. 2001. Application of aboutness to functional benchmarking in information retrieval. ACM transactions on information systems, 19(4), pages 337-370. Available from: https://doi.org/10.1145/502795.502796
DOI https://doi.org/10.1145/502795.502796
Keywords Functional benchmarking; Aboutness; Logic based information retrieval; Inductive evaluation

Files





You might also like



Downloadable Citations

;