'Research Insider': «Feeling Lucky? Multi-armed Bandits for Ordering Judgements in Pooling-based Evaluation»

Evaluation is crucial in Information Retrieval. The Cranfield paradigm allows reproducible system evaluation by fostering the construction of standard and reusable benchmarks. Each benchmark or test collection comprises a set of queries, a collection of documents and a set of relevance judgements. Relevance judgements are often done by humans and thus expensive to obtain.

Consequently, relevance judgements are customarily incomplete. Only a subset of the collection, the pool, is judged for relevance. In TREC-like campaigns, the pool is formed by the top retrieved documents supplied by systems participating in a certain evaluation task. With multiple retrieval systems contributing to the pool, an exploration/exploitation trade-off arises naturally. Exploiting effective systems could find more relevant documents, but exploring weaker systems might also be valuable for the overall judgement process.

In this paper, we cast document judging as a multi-armed bandit problem. This formal modelling leads to theoretically grounded adjudication strategies that improve over the state of the art. We show that simple instantiations of multi-armed bandit models are superior to all previous adjudication strategies.

About the speaker

Dr. David E. Losada is an Associate Professor in Computer Science & Artificial Intelligence at the University of Santiago de Compostela (Spain). He is currently the Director of the Master's Programme on Big Data Analytics.

David E. Losada received his BS in Computer Science (with honors) in 1997, and his PhD in Computer Science (with honors) in 2001, both from the University of A Coruña (Spain). From 2001 to 2002, he was a lecturer in the San Pablo-CEU University (Spain) and, in 2003, he joined the Univ. of Santiago de Compostela as a senior research fellow ("Ramón y Cajal" R&D programme).

His current research interests include a wide range of Information Retrieval (IR) and related areas such as: IR probabilistic models, summarization, novelty detection, sentence retrieval, patent search and opinion mining. Losada is an active member of the IR community and he regularly serves in the Programme Committee of prestigious international conferences such as SIGIR or ECIR. He has also led several R&D projects and contracts in the area of search technologies. In 2011, Losada was recognized with an ACM senior member award.