No.10（2015）10.Development of metrics for quality evaluation of information retrieval systems for incomplete evaluation data set

Norihiro Ohira, Shinichi Tomiyama

  Until now, numerous information retrieval (IR) systems have been developed and utilized. Nevertheless, it is uncommon that they are objectively compared with other systems. In academic studies, the qualities of IR systems are numerically measured based on a complete evaluation set that gives relations between search results and query keywords. However, these methods have never been popular in general, because they cost too much to evaluate the relevance of all the relations by hand.
  In this study, we developed two methods to evaluate IR systems using an incomplete evaluation set. The first method uses machine learning. We have obtained favorable results in experiments using only 50 manually-evaluated relations. The second is a new metric for the quality of IR systems. It differs from conventional inf-nDCG in the point that it can handle the query intent. In experiments, it was more robust against the incomplete evaluation of relationships than inf-nDCG.

Keywords
Search engine, Information retrieval, Big data, Metric, Ranking

Back to Contents