- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies whether trained rerankers can improve accuracy and efficiency when combined in a cascade. It introduces two lightweight scorers over cached embeddings, trained using pointwise or Shapley-based objectives, to filter candidates before a more expensive reranker. Both scorers use independent sentence scores at deployment. Experiments examine evidence retention, answer quality, latency, and memory. The study provides useful evidence about when cascades help, although the justification for Shapley training and the benefit of adding further stages remain less convincing.
Critical strengths
-
Cached embeddings reduce repeated computation, while separate training allows existing rerankers to be combined without retraining the entire pipeline.
-
The paper includes negative results, confidence intervals, and seed sensitivity. The ColBERT experiments also demonstrate promising improvements in both accuracy and pruning speed.
Critical weaknesses
-
The architecture combines frozen embeddings, feature fusion, DeepSets, and staged filtering, while Shapley training follows SCP. The contribution is primarily the empirical study rather than a substantially new ranking method.
-
Shapley training ranks marginal contributions, but inference ranks singleton values. No consistency constraint or analysis establishes when these two rankings should agree.
-
Each sentence receives an independent score. Its value cannot change when another selected sentence duplicates, contradicts, or complements it, limiting the practical role of the set-based formulation.
-
Encoding sentences alone can obscure pronouns, references, and supporting relationships. The coarse scorer cannot use neighboring text to resolve these dependencies before filtering.
-
All unselected MS MARCO passages become negatives. Some may contain useful supporting information, but the paper does not examine how this possible annotation noise affects learned selection.
-
Later rerankers cannot recover useful sentences discarded by the first stage. The method provides no uncertainty-based fallback for candidates whose usefulness is difficult to judge independently.
-
The greedy procedure follows relevance scores without optimizing value relative to sentence length or joint usefulness. A highly ranked long sentence can consume budget needed for several useful shorter sentences.
-
Increasing deployment samples from 5 to 50 lowers retention from 70.23% to 49.77% (Table 31). This substantial decline needs a clearer explanation of what the learned coalition scores capture.
-
Pointwise–ColBERT reaches 34.14% accuracy versus 31.49% for Pointwise alone. However, the paper does not report paired uncertainty for this comparison, leaving the additional stage’s benefit insufficiently established.
-
Both learned heads underperform frozen cosine on MuSiQue. The stronger BGE control also improves NoLiMa retention without improving QA, showing that better learned relevance scores do not reliably improve downstream answers.
-
Pointwise–ColBERT obtains identical QA accuracy with hard selection and RRF. The retention difference is also statistically inconclusive, so the added fusion mechanism lacks clear empirical justification.
Questions for authors
See the Weaknesses
Dear reviewers, as the deadline is approaching, please check the rebuttal and consider participating in the discussion.