Summary: Copyright is held by the author/owner(s)
SIGIR '03, July 28-August 1, 2003, Toronto, Canada.
Using Manually-Built Web Directories for Automatic
Evaluation of Known-Item Retrieval
Steven M. Beitzel, Eric C. Jensen, Abdur Chowdhury, David Grossman, Ophir Frieder
Illinois Institute of Technology
10 W 31 St.
Chicago, IL 60616
Information retrieval system evaluation is complicated by the
need for manually assessed relevance judgments. Large
manually-built directories on the web open the door to new
evaluation procedures. By assuming that web pages are the
known relevant items for queries that exactly match their title, we
use the ODP (Open Directory Project) and Looksmart directories
for system evaluation. We test our approach with a sample from
a log of ten million web queries and show that such an evaluation