- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies incremental neural reranking: after a query revision, how few of the previously scored candidates must be rescored to reproduce the ordered top-k of a full rerank? The authors formalize minimum recomputation with an exact oracle formulated as a binary linear program, finding that only 2.65 of 64 evaluations are needed on average, and show that revision-induced score changes are largely shared across candidates, so top-k crossings concentrate near the previous cutoff. From this analysis, they derive differential reranking: score all new candidates, rescore a block at the top of the old ranking, optionally expanded by a margin rule. On fixed pools, the policy achieves ~6× reranking-stage GPU speedup at fidelity nDCG@10 ≥ 0.99; on 250 held-out TopiOCQA conversations it cuts end-to-end latency by 14.4% and beats a same-budget BM25 baseline. However, the deployed policy reduces to a simple prefix-rescoring heuristic with the formal machinery unused, and the end-to-end gain is demonstrated under a single retrieval configuration chosen to maximize pool overlap.
Critical strengths
- Framing query revision as minimum recomputation, with an exact oracle solved as a binary linear program including deterministic tie handling, is novel for cross-encoder reranking and provides an upper bound on savings that most efficiency papers lack.
- Careful mechanism analysis. The shared/residual decomposition (Fig. 3, Table 7), the 8.7× gap versus independent score noise of equal variance, and the rank-(k+1) analyses (Tables 5–6) are informative, and the two-parameter crossing model transfers across rerankers and controlled benchmarks (Table 9) where richer models fail to.
- I also appreciate the transparency. Section 9 gives an honest account of why the live savings are much smaller than the fixed-pool ones, and several negative results (adaptive fallbacks, learned routers, deeper pools) are reported in the appendices instead of being dropped.
Critical weaknesses
- The writing really needs to be improved; more background discussion should be added.
- The literature review / related work section is missing. I would suggest including this part to give a clearer positioning against existing efficient reranking methods.
- My main concern is that the method actually evaluated is much simpler than what the paper develops. In the live experiment the policy just scores the new candidates, rescores the top of the old ranking, and reranks fully on the following turn. The crossing model, conformal calibration, and expansion rule are not used there at all, so what is really being tested amounts to rescoring a prefix of the previous ranking. The crossing model was also only checked on the controlled revision sets, not on the conversational queries that produced the main numbers.
- I also find the end-to-end gain modest given the overhead. The savings in the live setting are much smaller than in the fixed-pool experiments, in part because the policy reranks fully after every partial update. The system has to keep ranking state across turns, and old scores are only reusable when the query representation stays the same between turns (Sec. 7). The authors did try adaptive rules for choosing when to rescore, but none passed their validation, which makes me question how robust the fixed configuration is.
Questions for authors
Could you report the crossing model's ROC-AUC and calibration on the real TopiOCQA warm turns, and compare the threshold-based depth selection against the fixed m=32 prefix rule in the live setting? If the fixed-depth rule does just as well, it is unclear to me what the formal analysis adds beyond motivating a simple heuristic.
2 comments
Response to Reviewer 4Yu4 (2/2)
With the discussion period closing soon, we wanted to briefly highlight the main updates:
- Adaptive and conformal policies on real conversations. We expanded the 250-conversation TopiOCQA evaluation beyond fixed depth to include capped adaptive depth and conformal refinement. Even with TopiOCQA's frequent topic switches, both outperform fixed depth in call count, latency, and fidelity nDCG—cutting warm-turn Qwen calls by 17.9% and 23.6% while keeping fidelity above 0.999 (Section 9, Table 2).
- Savings persist under query rewriting. Rather than just noting the failure mode under query rewriting, we used our analysis to explain why a stored prefix misses newly promoted passages and how adding the current BM25 head fixes it. This variant preserves 1,037 of 1,049 gold hits while still saving about 5% of calls (Section 9, Appendix F).
Together, these results demonstrate that the method works end-to-end and show that our ranking-shift analysis gives readers a generalizable guide for tailoring candidate selection across retrieval settings. Please let us know if you have any remaining questions, and we hope you will consider raising your score.
Below is the query rewriting response that has been copied over now that there is room:
Does query representation have to say the same?
The T5 analysis was meant to probe our method's limits, not to restrict it to one retrieval strategy. The same analysis adapts our method to rewriting.
Appendix F shows that rewriting can promote a retained passage from deep in the stored ranking into the new top-10, even when current BM25 ranks it near the top. We therefore pair the stored prefix with the top of the current BM25 ranking (Algorithm 3). We tested this on the 250 held-out conversations using identical queries and BM25 pools for full and differential reranking (Section 9; Appendix F, Table 20). All turns:
| Pipeline | Qwen Calls | Latency | Gold Hit@10 | MRR@10 |
|---|---|---|---|---|
| Question history, full | 64.00 | 8.29 s | 14.30% | 0.0939 |
| Question history, differential | 53.55 | 6.88 s | 14.19% | 0.0933 |
| T5 rewriting, full | 64.00 | 6.36 s | 36.25% | 0.2390 |
| T5 rewriting, differential | 60.86 | 6.07 s | 35.95% | 0.2385 |
On T5, the method preserves 1,037 of full reranking's 1,049 gold hits at warm-turn fidelity nDCG@10 of 0.9987 while saving about 5% of calls and 0.3 s per request.
Response to Reviewer 4Yu4 (1/2)
Thank you for your careful review and constructive questions. We aim to have the analysis of how rankings shift as the main contribution for most readers, and the multiple methods as direct derivations of it.
1. What does the analysis add?
Fixed depth is simple only in hindsight. Which candidates to rescore, how deep, and what to change are all novel and come from the analysis:
The same analysis also drives the two policies below.
2. Do adaptive depth and conformal calibration work on real conversations?
Yes. We first reported fixed depth on TopiOCQA for simplicity and have since expanded the 250-conversation evaluation to adaptive depth, with all settings fixed before the run (Section 9, Table 2).
Our crossing analysis ties top-10 entry to a candidate's cutoff margin and local density (Section 5), so summing that risk sets depth from stored scores alone (Section 6). On live pools, the model still ranks crossings well (ROC-AUC 0.767) but overestimates them (9.5% predicted vs. 4.6% observed; Appendix E.3), so uncapped it rescores deeper than fixed depth (mean 41.7 vs. 32).
Warm turns:
Both beat fixed depth outright, with fewer calls, lower latency, and higher fidelity nDCG. Capped adaptive also raises gold Hit@10 and exact-set agreement (89.96% vs. 89.69%), and conformal refinement cuts latency the most.
TopiOCQA is an end-to-end stress test of frequent topic switching, so its 17.9–23.6% warm-turn call reductions are the conservative floor, while fixed pools mark the ceiling (75–80% fewer calls, about 6× faster; Section 7). Adaptive's margin over fixed depth narrows as turns become less predictable. At the ceiling, adaptive matches fixed m=20 accuracy with 26% fewer calls, and most conversations lie in between, where its margin is larger.
Those are external methods (Appendix D.3), distinct from our policies above; our conclusions do not depend on them, and we report them so others can avoid the pitfalls.
3. How large is the gain after overhead?
Every live number already includes the overhead. Latency is end-to-end request latency, covering retrieval, scoring every new candidate, the full refresh after each partial update, and candidate selection.
Measured this way, capped adaptive saves about 1.5 s per warm request at fidelity nDCG@10 of 0.9997, and conformal refinement about 2.0 s at 0.9996. These savings are a floor, since both policies fully refresh after each partial update (Section 10).
If you mean the query rewriting limitation, it weakens reuse rather than breaking it. We expanded that appendix (Section 9; Appendix F); our follow-up comment shows it still works.
4. Additions
The paper now has clearer background and an expanded Related Work section (Section 2). The new sections are in blue. Thank you again, and we hope you will reconsider your score.