acceptodds
Under review as a conference paper at ICLR 2027

Skip, Adapt, or Route? Certified Deployment of Test-Time Adaptive Rerankers under Limited Labels

Abstract

Test-time adaptation of a re-ranker's query representation from pseudo-relevance feedback improves some queries and damages others, so its average effect does not decide whether to deploy it. We treat this decision as a choice among frozen actions, made from a small labeled sample of the target traffic: keep the incumbent ranking, adapt every query, or route adaptation with a label-free gate. We prove three results. First, a selector that sees only label-free signals has regret bounded away from zero on conditions it cannot distinguish, so target labels are necessary. Second, an exact value ladder bounds what any gate on a signal z can add over the better simple action by min(E[mu(z)+], E[mu(z)-]), where mu(z) is the conditional effect of adaptation; four anchor regimes determine its sign. Third, certified policy selection (CPS) replaces the incumbent only when a confidence bound on the paired improvement clears a margin. It has a finite-sample safety guarantee, an explicit regret rate that includes the cost of fitting a gate, a closed-form label budget, and a sequential variant. We replay stored per-query outputs of 23 benchmark conditions, a production search log, and a new self-adaptation testbed. With 50 labeled queries, uncertified selection deploys a harmful action in 23–70% of calibrations and an uncorrected t-test in 6–28%; CPS with a normal bound does so in under 1% on average, and with a finite-sample bound never, at the price of deploying rarely. The label budget is predictable from the effect size. Score entropy sees at most about a fifth of the per-query routing headroom, because adaptation mainly pulls the scorer toward the retriever, and recommendations from a fixed configuration reverse once step sizes and the retriever's own ranking enter the action set.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, neural reranking, selective adaptation, safe policy improvement, confidence sequences, pseudo-relevance feedback

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9r1GOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to decide whether to deploy test-time adaptation in a search system using a small labeled sample. It compares no adaptation, adaptation on every query, and selective adaptation using a score-entropy threshold, replacing the incumbent only when statistical evidence supports an improvement in average performance. It also analyzes selection risk and the potential gains from routing. Offline replay shows fewer harmful deployments, although finite-sample guarantees are conservative and the tested entropy-threshold routing rules generally offer limited gains over the best fixed action.

Critical strengths

With limited labels, CPS reduces harmful deployments in offline experiments while retaining gains over the incumbent. Reporting both harmful-deployment rates and regret makes the tradeoff between safety and performance clear. The per-query caches also facilitate verification of the results.

Critical weaknesses
  1. The claimed upper bound on the routing value of the full entropy signal is unsupported. Section 5.5 uses cross-fitted and in-sample estimates over ten quantile bins to bracket the optimal routing gain achievable from entropy. These estimates underpin the claim that entropy captures at most about one fifth of the available per-query routing headroom. However, even if every binwise conditional mean were known exactly, the resulting routing value could be strictly below the optimum attainable from the full signal. The optimistic bias of in-sample fitting does not control this information loss. The empirical analysis fails to distinguish limited information in entropy from limited expressiveness of the routing rules. Consequently, the gains achieved by restricted rules do not justify the claimed ceiling on the value of the full signal. This undermines a central empirical conclusion.

  2. The contribution beyond existing safe-selection methods is not sufficiently established. Table 2 evaluates complete selection procedures containing both fixed and routed actions. The value added by entropy-based routing within these procedures remains unclear. A matched ablation should hold the fixed-action configurations, labeling budget, and certification procedure constant while comparing fixed-action-only selection, selection with entropy-based routing, and selection with shuffled-entropy routing. The best threshold fitted on the full data pool in Section 5.5 cannot substitute for this comparison. Given the reliance on existing statistical tools and the unsupported claim about signal value, the evidence does not establish a substantial contribution beyond those tools.

The presentation also overstates parts of the contribution and contains jumps in reasoning. The abstract prominently lists supporting analyses, including a standard sample-size formula, alongside the main results. Section 4 describes Proposition 2 as showing that routing gains are "often small", although the proposition provides no such conclusion. These statements blur the distinctions between existing tools, new results, and empirical observations. It is the authors’ responsibility to make these distinctions clear and ensure that their claims match the supporting evidence. Readers should not have to reconstruct them by repeatedly cross-checking the exposition, theorems, and experiments.

The central empirical conclusion lacks adequate support, and the remaining contribution is insufficient to justify acceptance on its own. I would reconsider my rating if the authors address these concerns with sufficient evidence or make substantive revisions that still establish a significant contribution. Changes limited to wording or other minor issues would not change my recommendation.

Questions for authors
  1. How does Section 5.5 control the routing value lost through binning, and thereby support the claim that the full entropy signal captures at most about one fifth of the available per-query routing headroom?

  2. Under the same fixed-action configurations, labeling budget, and certification procedure, how do harmful-deployment rates and regret differ among fixed-action-only selection, selection with entropy-based routing, and selection with shuffled-entropy routing? What additional value do these comparisons establish?

Reviewer_9faVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares several strategies for re-ranking candidates in retrieval -- a typical first stage model generates a ranked list of items, and then the system decide whether they should re-rank the top-k items (via query adaptation) from the first-stage generation or not. Specifically, this paper studies strategies including (1) use the original one every time (UPSTREAM), (2) apply re-ranking based on a frozen scorer decision (SKIP), (3) apply query adaptation all times (ALL), (3) apply pre-defined ranking depending on gating function decision (ROUTE). The paper also discusses if "Certified Policy Selection (CPS)", which replaces the action only when the calibration sample certifies an improvement can work. In the theoretical analysis, the paper derives how many additional samples will be need to perform CPS. The experiment on CPS demonstrate that CPS reduces the harm amount while also keeping the regret to be small on real-data experiments.

Critical strengths
  • The manuscript is clear about the point that the paper is not trying to propose a new method, rather try to understand how different re-ranking strategies works. Sufficient background context is provided, and the conclusion is clear.

  • Although I did not check every detail of the theoretical analysis (partly because I did not understand early stage of discussion including Proposition 3), the derived conclusion is intuitive, and appears to be reasonable.

  • The experiments are conducted on two real-data benchmark.

Critical weaknesses
  • Some parts of the draft are not easy to follow. For example. I believe the notion of "exchangable calibration" in limitation is not defined. I also found Proposition 3 is not well explained (see Questions for details), and Proposition 1 should have better presentation (See Questions).

  • Some datasets used in the experiments lack citation (especially Table 1). Credits of existing work (including benchmarks and datasets) should be properly acknowledged.

  • Confidence range or variance-related metric is not reported in the experiments.

Questions for authors
  • Can author(s)' explain the discrepancy between (b) scorer regret and (d) system regret in Figure 1?

  • Please provide derivation steps of μ(z)\mu(z) in Proposition 3. ("Then, ~") Also, more detailed explanation is needed what III. and IV. means in Proposition 3 (the manuscript explains that it intuitively mean that "if (ρ,γ,λ)(\rho, \gamma, \lambda)~" but do not explain how this interpretation is derived.)

  • Nits: Proposition 1: In my opinion, it is easier to understand if V(ROUTE)−V(a)V(ROUTE) - V(a) is written as (Gg−G)−(Lg−L)(G_g - G) - (L_g - L). I wonder why the order is flipped from V(ROUTE)−V(s)V(ROUTE) - V(s). This make things unnecessarily complicated.

  • Nits: typo in line 90 of page 2: fires and action -> fires action


Note that, although I scored weak acceptance as I did not have option for borderline, my exact evaluation is that this paper is on the borderline and I'm not fully understanding the paper in the initial review. Therefore, it is possible that it can be weak rejection later depending on the discussion.

Reviewer_P5reOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether a neural re-ranker should apply test-time adaptation (TTA), i.e., updating its query representation from pseudo-relevance feedback, to every query, to no query, or only to queries selected by a label-free gate. It frames this as choosing among fixed actions (keep the incumbent ranking, adapt all queries, or route adaptation) using a small labeled sample from the target traffic. It proposes Certified Policy Selection (CPS), which replaces the incumbent only when a confidence bound on the improvement exceeds a margin, and provides several propositions on the limits of label-free selection and gating. Experiments replay stored outputs from BEIR-based conditions, a search log (Qilin), and a self-adaptation testbed.

Critical strengths

None

Critical weaknesses

1. The experimental setup may not address the practical question the paper motivates.

The question that matters in practice is when to adapt a re-ranker that is actually worth deploying. The paper states that all re-rankers studied are weaker than their first-stage retriever. According to Appendix B, the scorers are BERT-family encoders without retrieval fine-tuning, using mean-pooled embeddings mapped to a Poincaré ball. In this regime the retriever’s own order is best in most conditions (Table 1), so the deployment decision largely reduces to “do not re-rank.” The proposed mechanism, that adaptation pulls the scorer toward the retriever, is also closely tied to this choice, since the anchors are the retriever’s top candidates. The self-adaptation testbed removes the weak scorer, but it does not test a re-ranker that improves on its first stage either. It is therefore unclear how far the observed behavior, including the conclusion that routing rarely pays, extends to realistic deployments. The paper should first explain why this configuration was chosen as the object of study.


2. The contribution is difficult to identify, partly because of presentation.

The paper states that it does not propose a new adaptation method, and Section 6 acknowledges that the statistical tools are established (high-confidence policy improvement, empirical-Bernstein and betting bounds, first-best treatment rules, paired-test sample sizes).

Therefore, It is unclear what the paper newly contributes beyond applying these tools to this setting. It is also unclear what message the routing analysis carries beyond this specific configuration, and how the propositions inform the procedure or a practitioner’s decision.

The presentation makes this harder to assess:

  • The numbers in the abstract and introduction (e.g., 23 benchmark conditions, 19 conditions, 30 of 56 cases) are hard to interpret without first reconstructing what is being counted and against what baseline.
  • SKIP and ALL are used in the Introduction before they are defined in Section 2.
  • The notion of “stored per-query outputs” is introduced only implicitly.
  • Table 1 packs many settings and values into each row, which makes comparison difficult.
Questions for authors
  1. Why did the authors choose re-rankers that are weaker than their first-stage retriever as the main object of study? How would they expect the conclusions to change for a re-ranker that improves on its first stage, such as a cross-encoder trained for retrieval? If the paper can show, or convincingly argue, what it tells practitioners about when to adapt a re-ranker worth deploying, I would reconsider my assessment.

  2. Beyond applying established selection procedures to TTA deployment, what do the authors consider the main new finding? For each proposition, could the authors state which decision in the procedure or in practice it informs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.