Dynamic Trade-Off Prediction in Multi-Stage Retrieval Systems
release_cwjhgbvynvaptetvvbiwqwr2z4
by
J. Shane Culpepper and Charles L. A. Clarke and Jimmy Lin
2016
Abstract
Modern multi-stage retrieval systems are comprised of a candidate generation
stage followed by one or more reranking stages. In such an architecture, the
quality of the final ranked list may not be sensitive to the quality of initial
candidate pool, especially in terms of early precision. This provides several
opportunities to increase retrieval efficiency without significantly
sacrificing effectiveness. In this paper, we explore a new approach to
dynamically predicting two different parameters in the candidate generation
stage which can directly affect the overall efficiency and effectiveness of the
entire system. Previous work exploring this tradeoff has focused on global
parameter settings that apply to all queries, even though optimal settings vary
across queries. In contrast, we propose a technique which makes a parameter
prediction that maximizes efficiency within a effectiveness envelope on a per
query basis, using only static pre-retrieval features. The query-specific
tradeoff point between effectiveness and efficiency is decided using a
classifier cascade that weighs possible efficiency gains against effectiveness
losses over a range of possible parameter cutoffs to make the prediction. The
interesting twist in our new approach is to train classifiers without requiring
explicit relevance judgments. We show that our framework is generalizable by
applying it to two different retrieval parameters - selecting k in common top-k
query retrieval algorithms, and setting a quality threshold, ρ, for
score-at-a-time approximate query evaluation algorithms. Experimental results
show that substantial efficiency gains are achievable depending on the dynamic
parameter choice. In addition, our framework provides a versatile tool that can
be used to estimate the effectiveness-efficiency tradeoffs that are possible
before selecting and tuning algorithms to make machine learned predictions.
In text/plain
format
Archived Files and Locations
application/pdf 545.0 kB
file_2fespqobrndd5ibza6cyadpdfe
|
arxiv.org (repository) web.archive.org (webarchive) |
1610.02502v1
access all versions, variants, and formats of this works (eg, pre-prints)