| | |
Summary: On the Effectiveness of Evaluating Retrieval Systems
in the Absence of Relevance Judgments #
Javed A. Aslam
Department of Computer Science
Dartmouth College
jaa@cs.dartmouth.edu
Robert Savell
Department of Computer Science
Dartmouth College
rsavell@cs.dartmouth.edu
ABSTRACT
Soboro#, Nicholas and Cahan recently proposed a method
for evaluating the performance of retrieval systems without
relevance judgments. They demonstrated that the system
evaluations produced by their methodology are correlated
with actual evaluations using relevance judgments in the
TREC competition. In this work, we propose an explana
tion for this phenomenon. We devise a simple measure for
quantifying the similarity of retrieval systems by assessing
the similarity of their retrieved results. Then, given a collec
|
