acceptodds
Under review as a conference paper at ICLR 2027

Differential Neural Reranking: Updating Rankings Without Reranking Everything

Abstract

A revised query may need a new ranking, but not a rescore of every candidate. We ask how few candidates must be rescored to reproduce the top-k of a full rerank, and propose differential neural reranking, which uses the previous ranking to choose them. Across eight retrieval domains and two reranker architectures, revisions shift nearly every candidate's score. Much of this movement is shared across candidates and preserves their order, while the candidate-specific changes that alter the top-k concentrate near the cutoff. On 273 successive session revisions, an exact oracle needs only 2.65 of 64 fresh evaluations on average to recover the ordered top-10. A two-parameter model predicts cutoff crossings across rerankers and benchmarks without refitting, capturing the rank structure that determines which candidates require fresh evaluation. On fixed candidate pools, differential reranking speeds up the reranking stage by about 6× on GPU at fidelity nDCG@10 above 0.99. The effect persists in end-to-end pipelines, with savings large enough to offset the added scoring cost as the candidate pool changes. On 250 held-out TopiOCQA conversations, with the policy and evaluation protocol fixed in advance, differential reranking reduced mean end-to-end latency by 14.40%, outperformed a BM25 baseline with the same reranker budget, and kept fidelity nDCG@10 at or above 0.98 on every later turn in 249 of 250 conversations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
neural reranking, information retrieval, efficient inference, query reformulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4Yu4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies incremental neural reranking: after a query revision, how few of the previously scored candidates must be rescored to reproduce the ordered top-k of a full rerank? The authors formalize minimum recomputation with an exact oracle formulated as a binary linear program, finding that only 2.65 of 64 evaluations are needed on average, and show that revision-induced score changes are largely shared across candidates, so top-k crossings concentrate near the previous cutoff. From this analysis, they derive differential reranking: score all new candidates, rescore a block at the top of the old ranking, optionally expanded by a margin rule. On fixed pools, the policy achieves ~6× reranking-stage GPU speedup at fidelity nDCG@10 ≥ 0.99; on 250 held-out TopiOCQA conversations it cuts end-to-end latency by 14.4% and beats a same-budget BM25 baseline. However, the deployed policy reduces to a simple prefix-rescoring heuristic with the formal machinery unused, and the end-to-end gain is demonstrated under a single retrieval configuration chosen to maximize pool overlap.

Critical strengths
  • Framing query revision as minimum recomputation, with an exact oracle solved as a binary linear program including deterministic tie handling, is novel for cross-encoder reranking and provides an upper bound on savings that most efficiency papers lack.
  • Careful mechanism analysis. The shared/residual decomposition (Fig. 3, Table 7), the 8.7× gap versus independent score noise of equal variance, and the rank-(k+1) analyses (Tables 5–6) are informative, and the two-parameter crossing model transfers across rerankers and controlled benchmarks (Table 9) where richer models fail to.
  • I also appreciate the transparency. Section 9 gives an honest account of why the live savings are much smaller than the fixed-pool ones, and several negative results (adaptive fallbacks, learned routers, deeper pools) are reported in the appendices instead of being dropped.
Critical weaknesses
  • The writing really needs to be improved; more background discussion should be added.
  • The literature review / related work section is missing. I would suggest including this part to give a clearer positioning against existing efficient reranking methods.
  • My main concern is that the method actually evaluated is much simpler than what the paper develops. In the live experiment the policy just scores the new candidates, rescores the top of the old ranking, and reranks fully on the following turn. The crossing model, conformal calibration, and expansion rule are not used there at all, so what is really being tested amounts to rescoring a prefix of the previous ranking. The crossing model was also only checked on the controlled revision sets, not on the conversational queries that produced the main numbers.
  • I also find the end-to-end gain modest given the overhead. The savings in the live setting are much smaller than in the fixed-pool experiments, in part because the policy reranks fully after every partial update. The system has to keep ranking state across turns, and old scores are only reusable when the query representation stays the same between turns (Sec. 7). The authors did try adaptive rules for choosing when to rescore, but none passed their validation, which makes me question how robust the fixed configuration is.
Questions for authors

Could you report the crossing model's ROC-AUC and calibration on the real TopiOCQA warm turns, and compare the threshold-based depth selection against the fixed m=32 prefix rule in the live setting? If the fixed-depth rule does just as well, it is unclear to me what the formal analysis adds beyond motivating a simple heuristic.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4Yu4 (1/2)

Thank you for your careful review and constructive questions. We aim to have the analysis of how rankings shift as the main contribution for most readers, and the multiple methods as direct derivations of it.

1. What does the analysis add?

Fixed depth is simple only in hindsight. Which candidates to rescore, how deep, and what to change are all novel and come from the analysis:

  • Which candidates to rescore. Because shared shifts confine crossings near the old cutoff while top incumbents reorder (Sections 4–5), the stored prefix is the right set to rescore. The obvious alternative, selecting by BM25 at the same budget, loses 13 points of exact order (61% vs. 74%; Section 9).
  • How deep to rescore. Modeling shared shifts and cutoff-margin risk sets depth per turn (Section 5), so shift refinement matches m=20m=20 accuracy with 26% fewer calls on fixed pools (14.8 vs. 20; Section 7), and on conversations both capped adaptive and conformal refinement beat fixed depth. Without the analysis, depth is a costly guess, as m=8m=8 keeps only 25 of 250 conversations at fidelity ≥ 0.98 while m=32m=32 keeps all (Appendix D.4, Table 16).
  • What to change when retrieval changes. A prefix alone misses the deep stored passages that query rewriting promotes (Appendix F). The analysis identifies the fix, adding the current BM25 head, which preserves 1,037 of 1,049 gold hits at 0.9987 fidelity (Section 9).

The same analysis also drives the two policies below.

2. Do adaptive depth and conformal calibration work on real conversations?

Yes. We first reported fixed depth on TopiOCQA for simplicity and have since expanded the 250-conversation evaluation to adaptive depth, with all settings fixed before the run (Section 9, Table 2).

Our crossing analysis ties top-10 entry to a candidate's cutoff margin and local density (Section 5), so summing that risk sets depth from stored scores alone (Section 6). On live pools, the model still ranks crossings well (ROC-AUC 0.767) but overestimates them (9.5% predicted vs. 4.6% observed; Appendix E.3), so uncapped it rescores deeper than fixed depth (mean 41.7 vs. 32).

  • Capped adaptive caps depth at 32, where our pool-overlap analysis found every retained top-10 entrant (Section 8). This removes the excess depth while keeping shallower depths on low-risk turns.
  • Conformal refinement measures the shift instead of predicting risk. Because most score movement is shared (Section 5), the shift on the retained top 11 predicts the rest, and the prefix extends only where a calibrated margin (ε = 0.5%) reaches the cutoff. When that first block misses a top-10 document, the extension recovers it in 90.2% of cases (checkpoint; Appendix E.4).

Warm turns:

Warm-Turn PolicyQwen CallsLatencyFidelity nDCG@10Gold Hit@10
Full64.008.52 s1.0000010.12%
Fixed depth (m=32m=32)53.547.10 s0.999109.97%
Capped adaptive52.526.97 s0.9996710.01%
Uncapped adaptive59.587.98 s0.9997710.09%
Conformal refinement48.886.56 s0.999579.88%

Both beat fixed depth outright, with fewer calls, lower latency, and higher fidelity nDCG. Capped adaptive also raises gold Hit@10 and exact-set agreement (89.96% vs. 89.69%), and conformal refinement cuts latency the most.

TopiOCQA is an end-to-end stress test of frequent topic switching, so its 17.9–23.6% warm-turn call reductions are the conservative floor, while fixed pools mark the ceiling (75–80% fewer calls, about 6× faster; Section 7). Adaptive's margin over fixed depth narrows as turns become less predictable. At the ceiling, adaptive matches fixed m=20m=20 accuracy with 26% fewer calls, and most conversations lie in between, where its margin is larger.

"adaptive rules for choosing when to rescore ..."

Those are external methods (Appendix D.3), distinct from our policies above; our conclusions do not depend on them, and we report them so others can avoid the pitfalls.

3. How large is the gain after overhead?

Every live number already includes the overhead. Latency is end-to-end request latency, covering retrieval, scoring every new candidate, the full refresh after each partial update, and candidate selection.

Measured this way, capped adaptive saves about 1.5 s per warm request at fidelity nDCG@10 of 0.9997, and conformal refinement about 2.0 s at 0.9996. These savings are a floor, since both policies fully refresh after each partial update (Section 10).

"old scores are only reusable when the query representation stays the same"

If you mean the query rewriting limitation, it weakens reuse rather than breaking it. We expanded that appendix (Section 9; Appendix F); our follow-up comment shows it still works.

4. Additions

The paper now has clearer background and an expanded Related Work section (Section 2). The new sections are in blue. Thank you again, and we hope you will reconsider your score.

AuthorsOpenReview ↗

Response to Reviewer 4Yu4 (2/2)

With the discussion period closing soon, we wanted to briefly highlight the main updates:

  • Adaptive and conformal policies on real conversations. We expanded the 250-conversation TopiOCQA evaluation beyond fixed depth to include capped adaptive depth and conformal refinement. Even with TopiOCQA's frequent topic switches, both outperform fixed depth in call count, latency, and fidelity nDCG—cutting warm-turn Qwen calls by 17.9% and 23.6% while keeping fidelity above 0.999 (Section 9, Table 2).
  • Savings persist under query rewriting. Rather than just noting the failure mode under query rewriting, we used our analysis to explain why a stored prefix misses newly promoted passages and how adding the current BM25 head fixes it. This variant preserves 1,037 of 1,049 gold hits while still saving about 5% of calls (Section 9, Appendix F).

Together, these results demonstrate that the method works end-to-end and show that our ranking-shift analysis gives readers a generalizable guide for tailoring candidate selection across retrieval settings. Please let us know if you have any remaining questions, and we hope you will consider raising your score.

Below is the query rewriting response that has been copied over now that there is room:

Does query representation have to say the same?

The T5 analysis was meant to probe our method's limits, not to restrict it to one retrieval strategy. The same analysis adapts our method to rewriting.

Appendix F shows that rewriting can promote a retained passage from deep in the stored ranking into the new top-10, even when current BM25 ranks it near the top. We therefore pair the stored prefix with the top of the current BM25 ranking (Algorithm 3). We tested this on the 250 held-out conversations using identical queries and BM25 pools for full and differential reranking (Section 9; Appendix F, Table 20). All turns:

PipelineQwen CallsLatencyGold Hit@10MRR@10
Question history, full64.008.29 s14.30%0.0939
Question history, differential53.556.88 s14.19%0.0933
T5 rewriting, full64.006.36 s36.25%0.2390
T5 rewriting, differential60.866.07 s35.95%0.2385

On T5, the method preserves 1,037 of full reranking's 1,049 gold hits at warm-turn fidelity nDCG@10 of 0.9987 while saving about 5% of calls and 0.3 s per request.

Reviewer_zbjDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to update a neural ranking after a query revision while scoring fewer candidates. It defines an exact minimum-recomputation oracle, analyzes shared score changes and cutoff crossings, and uses the previous ranking to select candidates for rescoring. Controlled experiments cover eight domains and two reranker architectures. A held-out evaluation on 250 TopiOCQA conversations reports a 14.40% mean end-to-end latency reduction while closely matching full reranking. The evaluated quality target is fidelity to the full reranker, rather than relevance to human labels.

Critical strengths
  1. The paper studies a useful source of wasted computation. It separates large shared score changes from the smaller changes that alter the top results. The exact oracle and cross-reranker analysis provide useful evidence for reusing the previous ranking. This goes beyond proposing a cache based only on query similarity.

  2. The realistic evaluation includes important costs and failure cases. The 250-conversation study counts first turns, new candidates, cache updates, and full reranks after partial updates. It reports 14.40% end-to-end latency savings and compares with BM25 selection using the same reranker budget. The paper also distinguishes approximately 6× fixed-pool reranking speedup from the smaller conversational benefit. This makes the practical result credible within its stated setup.

Critical weaknesses
  1. Practical usefulness is not fully established outside the chosen retrieval formulation. The final experiment measures agreement with full Qwen reranking using concatenated question history. Appendix E.3 reports that query rewriting retrieves the gold passage more often but produces less overlap. Thus, the most favorable setting for score reuse may not be the best retrieval setting. High fidelity to the chosen reference does not by itself show that the method preserves gold-passage relevance or offers a good relevance–latency tradeoff under stronger retrieval. This limits the practical significance, although it does not invalidate the reported fidelity result.
Questions for authors
  1. On the same 250 held-out conversations, what are the gold-passage retrieval metrics for full and differential reranking? How do their relevance–latency tradeoffs compare when using the T5 query-rewrite pipeline already examined in the appendix?
  2. How much of the observed fidelity loss changes relevant-document inclusion, rather than only the ordering of nearly tied documents? This would help interpret the reported live exact-order rate of 74.15% alongside the very high fidelity nDCG.
2 comments
AuthorsOpenReview ↗

Response to Reviewer zbjD (1/2)

Thank you for your careful review and questions. You noted that the method was evaluated in the retrieval setting most favorable to reuse, and that high fidelity alone does not show that gold relevance is preserved. Below, we report gold relevance for every policy on the same 250 held-out conversations, and use the paper's analysis of how rankings change to adapt the method to stronger T5 rewriting.

1. Does differential reranking work with T5 rewriting?

The T5 analysis was meant to probe our method's limits, not to restrict it to one retrieval strategy. The same analysis adapts our method to rewriting.

Appendix F shows that rewriting can promote a retained passage from deep in the stored ranking into the new top-10, even when current BM25 ranks it near the top. We therefore pair the stored prefix with the top of the current BM25 ranking (Algorithm 3). We tested this on the same 250 held-out conversations with matched queries and pools (Section 9; Appendix F, Table 20). All turns:

PipelineQwen CallsLatencyGold Hit@10MRR@10
Question history, full64.008.29 s14.30%0.0939
Question history, differential53.556.88 s14.19%0.0933
T5 rewriting, full64.006.36 s36.25%0.2390
T5 rewriting, differential60.866.07 s35.95%0.2385

On T5, it preserves 1,037 of 1,049 gold hits at warm-turn fidelity nDCG@10 of 0.9987 while saving about 5% of calls and 0.3 s per request.

2. Does differential reranking preserve gold relevance?

Yes. Fixed depth and capped adaptive stay within 0.15 percentage points of full reranking in gold Hit@10, and conformal refinement loses about 0.24 points, while cutting warm-turn Qwen calls by 16–24% (Section 9; Appendix E.3, Table 18). We first reported fixed depth for simplicity and have since expanded the evaluation to adaptive depth.

Our crossing analysis sets depth from stored scores by summing each candidate's risk of entering the top 10 (Sections 5–6). On live pools it overestimates that risk, so uncapped it rescores deeper than fixed depth.

  • Capped adaptive caps depth at 32, where our pool-overlap analysis found every retained top-10 entrant (Section 8).
  • Conformal refinement measures the shift instead: because most score movement is shared (Section 5), the shift on the retained top 11 predicts the rest, and the prefix extends only where a calibrated margin (ε = 0.5%) reaches the cutoff (Appendix E.4).

Warm turns:

PolicyQwen callsLatencyGold Hit@10MRR@10
Full64.008.52 s10.12%0.0616
Fixed depth (m=32m=32)53.547.10 s9.97%0.0608
Capped adaptive52.526.97 s10.01%0.0609
Conformal refinement48.886.56 s9.88%0.0616

Both beat fixed depth in calls and latency. Capped adaptive also raises exact-set agreement (89.96% vs. 89.69%), and conformal refinement cuts latency the most.

3. Do ranking disagreements affect relevant-document inclusion?

Relevant-document inclusion is rarely affected. Capped adaptive returns full reranking's exact top-10 set on 89.96% of warm turns and preserves 378 of its 382 pooled reference gold hits (379 for fixed depth; 376 for conformal refinement; Appendix E.3, Table 18). Under T5 rewriting, the set matches on 88.21% of warm turns (Appendix F, Table 20).

"ordering of nearly tied documents"

Most remaining disagreement is reordering within the same set. For fixed depth, exact-set agreement is 89.69% but exact-order agreement is 74.15%. In a fixed-depth end-to-end audit, 84 reversed document pairs across 49 order-only disagreements had an average live-score gap of 0.077 logits (median 0.0625; maximum 0.21875; Appendix E.3). Batch-dependent score variation can swap near ties, and using full-rerank reference logits on fixed depth's selected candidates raises exact order to 99.41% (Conformal checkpoint: 97.96%).

Together, the new results show how the analysis guides the selection rule in each retrieval setting and what that rule costs in relevance: about 0.24 points of gold Hit@10 under question history, and 12 of 1,049 gold hits under T5 rewriting. The paper is updated, and the new sections are in blue. Thank you again, and we hope you will reconsider your score.

AuthorsOpenReview ↗

Response to Reviewer zbjD (2/2)

With the discussion period closing soon, we wanted to briefly highlight the main updates:

  • Adaptive and conformal policies on real conversations. We expanded the 250-conversation TopiOCQA evaluation beyond fixed depth to include capped adaptive depth and conformal refinement. Even with TopiOCQA's frequent topic switches, both outperform fixed depth in call count, latency, and fidelity nDCG—cutting warm-turn Qwen calls by 17.9% and 23.6% while keeping fidelity above 0.999 (Section 9, Table 2).
  • Savings persist under query rewriting. Rather than just noting the failure mode under query rewriting, we used our analysis to explain why a stored prefix misses newly promoted passages and how adding the current BM25 head fixes it. This variant preserves 1,037 of 1,049 gold hits while still saving about 5% of calls (Section 9, Appendix F).

Together, these results demonstrate that the method works end-to-end and show that our ranking-shift analysis gives readers a generalizable guide for tailoring candidate selection across retrieval settings. Please let us know if you have any remaining questions, and we hope you will consider raising your score.

Reviewer_u3jAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to update a neural ranking after a query changes without rescoring every candidate. It first defines an oracle that finds the smallest set of old scores to refresh while reproducing the full rerank's ordered top-k. Controlled experiments across eight domains and two rerankers show that much of the score movement is shared across candidates, while candidates entering the top-k tend to be near the old cutoff. A simple crossing model motivates rescoring a prefix of the old ranking, with optional expansion based on newly observed scores. For conversational retrieval, the deployed policy scores all new candidates, rescores a fixed block of retained candidates, and performs a full rerank after each partial update. On fixed pools, the paper reports about sixfold reranking-stage speedups. On 250 held-out TopiOCQA conversations, it reports a 14.40% reduction in end-to-end latency and high fidelity to full reranking, while explicitly accounting for new candidates and full resets.

Critical strengths

1、The paper gives a clear way to study redundant computation in repeated retrieval. The minimum-recomputation oracle distinguishes the candidates involved in a ranking change from the scores that actually need refreshing, which matters when old and new scores are mixed. The analysis then connects this opportunity to measurable score behavior: shared shifts preserve much of the order, and cutoff crossings are concentrated near the previous top-k. On the 273 controlled session revisions, the oracle needs only 2.65 of 64 fresh evaluations on average. This does not provide an online guarantee, and the paper makes that clear, but it is a useful reference for understanding where a practical update rule spends extra work.

2、The live evaluation is a strong part of the paper. The final TopiOCQA policy is fixed before evaluating the 250 held-out conversations, and the reported cost includes retrieval, selection, new-candidate scoring, cold starts, and full reranks after partial updates. The 14.40% mean latency reduction is therefore more informative than a reranker-call count alone. The same-budget BM25 control also tests whether historical rankings add useful information beyond simply evaluating fewer current candidates. Reporting reliability for every later turn in a conversation, together with the difference between live BF16 rankings and a common-score replay, makes the practical result easier to assess. These details support a real, if setting-dependent, efficiency gain.

Critical weaknesses

1、The main practical result is established under a retrieval setup that favors candidate reuse, and the corresponding retrieval-quality tradeoff is still unclear. The conversational pipeline concatenates the current question with all previous user questions. Section 7 and Appendix E.3 show that current-question retrieval has almost no reusable candidates, while T5-based rewriting retrieves the gold passage more often than question history but has substantially less overlap. The final 250-conversation experiment reports fidelity to full reranking under the history-based setup, rather than a quality-versus-latency comparison across these alternatives. This does not invalidate the measured speedup. It does leave open whether a user would obtain a better retrieval system by adopting this setup, or whether a stronger query formulation would remove much of the available saving. That is important for judging the method's practical reach.

2、The connection between the detailed mechanism study and the final serving policy is only partly validated. The crossing model is evaluated on controlled pools built from judged candidates plus sampled or identifier-selected distractors. Those pools need not have the same concentration of difficult negatives as a real retriever's top results. The final conversational policy also drops the adaptive expansion and uses a development-selected fixed depth of 32, followed by full resets. The live results therefore validate a historical-prefix selection rule, but not the transferred crossing probabilities or the adaptive mechanism itself on realistic changing pools. I would like a clearer account of which conclusions are supported by the controlled experiments and which carry over to deployment. This limits the generality of the mechanism claim more than it affects the correctness of the reported serving result.

Questions for authors

1、On the held-out conversations, what is the gold-passage retrieval quality of full and differential reranking with question-history queries, compared with a stronger query-rewriting pipeline at its measured latency? Can the quality-versus-cost tradeoff be shown without treating high overlap as an objective in itself?

2、Does the two-parameter crossing model remain useful on the actual BM25-retrieved candidate pools and natural topic changes? Please report calibration error or observed crossing rates in addition to the calibration correlation used in Table 9.

3、Does m = 32 mean the best 32 retained candidates, or retained candidates whose rank in the entire previous pool is at most 32? Algorithm 1, the prefix-set equation, and the conversational descriptions are not fully consistent on this point, and the choice changes both cost and coverage.

4、How do the Mxbai-guided and EARM-style selectors compare on the held-out conversations when total latency, rather than only Qwen evaluations, is counted? The development comparisons are useful, but they do not establish that the selected prefix rule offers the best practical tradeoff.

3 comments
AuthorsOpenReview ↗

Response to Reviewer u3jA (1/2)

Thank you for your careful review and constructive questions. Your two main concerns ask whether our analysis carries over to deployment. Below, we show the analysis can help adapt to both query rewriting (W1) and real pools (W2).

1. W1 & Q1 — Does differential reranking work with T5 rewriting?

The T5 analysis was meant to probe our method's limits, not to restrict it to one retrieval strategy. The same analysis adapts our method to rewriting.

Appendix F shows that rewriting can promote a retained passage from deep in the stored ranking into the new top-10, even when current BM25 ranks it near the top. We therefore pair the stored prefix with the top of the current BM25 ranking (Algorithm 3). We tested this on the 250 held-out conversations using identical queries and BM25 pools for full and differential reranking (Section 9; Appendix F, Table 20). All turns:

PipelineQwen CallsLatencyGold Hit@10MRR@10
Question history, full64.008.29 s14.30%0.0939
Question history, differential53.556.88 s14.19%0.0933
T5 rewriting, full64.006.36 s36.25%0.2390
T5 rewriting, differential60.866.07 s35.95%0.2385

On T5, the method preserves 1,037 of full reranking's 1,049 gold hits at warm-turn fidelity nDCG@10 of 0.9987 while saving about 5% of calls and 0.3 s per request.

2. W2 & Q2 — Does the crossing model work on real retrieval pools?

Yes. We first reported fixed depth on TopiOCQA for simplicity and have since expanded the 250-conversation evaluation to adaptive depth (Section 9, Table 2).

Our crossing analysis ties top-10 entry to a candidate's cutoff margin and local density (Section 5), so summing that risk sets depth from stored scores alone (Section 6). On live pools, the model still ranks crossings well (ROC-AUC 0.767) but overestimates them (9.5% predicted vs. 4.6% observed; Appendix E.3), so uncapped it rescores deeper than fixed depth (mean 41.7 vs. 32).

  • Capped adaptive caps depth at 32, where our pool-overlap analysis found every retained top-10 entrant (Section 8). This removes the excess depth while keeping shallower depths on low-risk turns.
  • Conformal refinement measures the shift instead of predicting risk. Because most score movement is shared (Section 5), the shift on the retained top 11 predicts the rest, and the prefix extends only where a calibrated margin (ε = 0.5%) reaches the cutoff. When that first block misses a top-10 document, the extension recovers it in 90.2% of cases (checkpoint; Appendix E.4).

Warm turns:

Warm-Turn PolicyQwen CallsLatencyFidelity nDCG@10Gold Hit@10
Full64.008.52 s1.0000010.12%
Fixed depth53.547.10 s0.999109.97%
Capped adaptive52.526.97 s0.9996710.01%
Uncapped adaptive59.587.98 s0.9997710.09%
Conformal refinement48.886.56 s0.999579.88%

Both beat fixed depth outright, with fewer calls, lower latency, and higher fidelity nDCG. Capped adaptive also raises exact-set agreement (89.96% vs. 89.69%), and conformal refinement cuts latency the most.

"natural topic changes"

TopiOCQA is an end-to-end stress test of frequent topic switching, so its 17.9–23.6% warm-turn call reductions are the conservative floor, while fixed pools mark the ceiling (75–80% fewer calls, about 6× faster; Section 7). Even on topic-change turns, fidelity and gold Hit@10 hold (T5 run; Appendix F):

StratumCallsFidelityGold Hit@10 (Full → Diff.)
Topic-change turns62.150.999424.72% → 24.51%
Continuing turns60.440.998639.40% → 39.08%

"which conclusions are supported by the controlled experiments"

As you suggested, the evidence map (Appendix G, Table 21) and settings table (Appendix D.1, Table 11) separate controlled-experiment conclusions from deployment ones.

3. Q3 — Which 32 candidates are rescored?

The policy scores every new candidate, plus the first 32 retained candidates ordered by stored score. The paper now states this clearly.

4. Q4 — How do Mxbai and EARM compare on total latency?

  • Mxbai: It adds a full 64-candidate Mxbai pass to the prefix rule's Qwen calls every turn, so it can be slower. It trails in fidelity (0.99381 vs. 0.99975) and exact order (59.40% vs. 73.56%; Appendix D.2).

  • EARM: We have now timed an EARM-style selector (rank 8) end to end on an additional 80-conversation run (956 requests, all turns; Appendix D.2).

PolicyMean latencyQwen callsFidelity nDCG@10Exact order
Full8.82 s64.001.00000100.0%
Capped adaptive7.10 s53.250.9996770.0%
EARM-style7.17 s56.670.9402543.9%

EARM is slower than capped adaptive with far lower fidelity. The two can be complementary, since our analysis could choose which scores EARM refreshes.

The paper is updated, and the new sections are in blue. Thank you again, and we hope you will reconsider your score.

AuthorsOpenReview ↗

Response to Reviewer u3jA (2/2)

With the discussion period closing soon, we wanted to briefly highlight the main updates:

  • Adaptive and conformal policies on real conversations. We expanded the 250-conversation TopiOCQA evaluation beyond fixed depth to include capped adaptive depth and conformal refinement. Even with TopiOCQA's frequent topic switches, both outperform fixed depth in call count, latency, and fidelity nDCG, cutting warm-turn Qwen calls by 17.9% and 23.6% while keeping fidelity above 0.999 (Section 9, Table 2).
  • Savings persist under query rewriting. Rather than just noting the failure mode under query rewriting, we used our analysis to explain why a stored prefix misses newly promoted passages and how adding the current BM25 head fixes it. This variant preserves 1,037 of 1,049 gold hits while still saving about 5% of calls (Section 9, Appendix F).

Together, these results demonstrate that the method works end-to-end and show that our ranking-shift analysis gives readers a generalizable guide for tailoring candidate selection across retrieval settings. Please let us know if you have any remaining questions, and we hope you will consider raising your score.

Thank you for the additional experiments and for addressing most of my concerns. I am keeping my rating at 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.