Summary: Where to Stop Reading a Ranked List?
Archives and Information Studies, Faculty of Humanities, University of Amsterdam
ISLA, Informatics Institute, University of Amsterdam
Abstract: We document our participation in the
TREC 2008 Legal Track. This year we focused
solely on selecting rank cut-offs for optimizing the
given evaluation measure per topic.
In recall-oriented retrieval setups, such as the Legal Track,
ranked retrieval has a particular disadvantage in comparison
with traditional Boolean retrieval: there is no clear cut-off
point where to stop consulting results. It is expensive to give
a ranked list with too many results to litigation support pro-
fessionals paid by the hour. This may be one of the reasons
why ranked retrieval has been adopted very slowly in pro-
fessional legal search.1