The Curious Case of IR Explainability: Explaining Document Scores within and across Ranking Models

Abstract

It is often useful for an IR practitioner to analyze the similarity function of an IR model, or for a non-technical search engine user to understand why a document was shown at a certain rank, in terms of the three fundamental aspects of a similarity function, namely the a) frequency of a term in a document, b) frequency of a term in a collection and c) the length of a document. We propose a general methodology of approximating an IR model as the coefficients of a linear function of these three fundamental aspects (and an additional aspect of semantic similarity between terms for neural models), which potentially can help IR practitioners to optimize the relative importance of each aspect on specific document collection and types of queries. Our analysis shows that the coefficients, which represent the relative importance of the three fundamental aspects, are useful to compare a model's different parametric instantiations or compare across different models.