A Relative Information Gain-based Query Performance Prediction Framework with Generated Query Variants release_l6z7uo3xj5h5bokhnfzmq3vqii

by Suchana Datta, Debasis Ganguly, Mandar Mitra, Derek Greene

Published in ACM Transactions on Information Systems by Association for Computing Machinery (ACM).

2022  

Abstract

Query performance prediction (QPP) methods, which aim to predict the performance of a query, often rely on evidences in the form of different characteristic patterns in the distribution of Retrieval Status Values (RSVs). However, for neural IR models, it is usually observed that the RSVs are often less reliable for QPP because they are bounded within short intervals, different from the situation for statistical models. To address this limitation, we propose a model-agnostic QPP framework that gathers additional evidences by leveraging information from the characteristic patterns of RSV distributions computed over a set of <jats:italic>automatically-generated</jats:italic> query variants, relative to that of the current query. Specifically, the idea behind our proposed method - Weighted Relative Information Gain (WRIG), is that a substantial relative decrease or increase in the standard deviation of the RSVs of the query variants is likely to be a relative indicator of how easy or difficult the original query is. To cater for the absence of human-annotated query variants in real-world scenarios, we further propose an automatic query variant generation method. This can produce variants in a controlled manner by substituting terms from the original query with new ones sampled from a weighted distribution, constructed either via a relevance model or with the help of an embedded representation of query terms. Our experiments on the TREC-Robust, ClueWeb09B and MS MARCO datasets show that WRIG, by the use of this relative changes in QPP estimate, leads to significantly better results than a state-of-the-art baseline method which leverages information from (manually created) query variants by the application of additive smoothing [64]. The results also show that our approach can improve the QPP effectiveness of neural retrieval approaches in particular.
In application/xml+jats format

Archived Files and Locations

application/pdf  2.3 MB
file_hwyiifkvbvhblhuw27dqemhgty
dl.acm.org (publisher)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article-journal
Stage   published
Date   2022-06-23
Language   en ?
Container Metadata
Not in DOAJ
In Keepers Registry
ISSN-L:  1046-8188
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 8b50d00f-0fe4-4b54-9d54-366fc2469580
API URL: JSON