acceptodds
Under review as a conference paper at ICLR 2027

Disagreement-Guided Weak-to-Strong Routing

Abstract

Small language models are increasingly deployed on edge devices and robotic systems to meet cost and latency constraints, yet their limited capabilities can make reliable deployment challenging. Stronger models can provide assistance when needed, but deciding when to route a weak-model response to a stronger model often relies on signals of response correctness. In open-ended and agentic tasks, however, obtaining such signals can require costly ground-truth supervision that is difficult to obtain at scale. We revisit the foundations of weak-to-strong routing and ask whether routing decisions can instead be guided by signals that do not require ground-truth supervision. We first characterize the optimal weak-to-strong routing policy, while allowing for multiple weak-model responses generated through local inference. Building on this characterization, we derive a principled approximation to the optimal routing based on model-to-model comparison. In particular, we introduce a notion of model disagreement that can be measured directly from model-generated responses, without ground-truth supervision. Importantly, the approximation becomes increasingly accurate as the strong model becomes a better proxy for the ground-truth distribution. Across agentic and single-turn tasks, our approach effectively navigates the trade-off between routing cost and response accuracy without ground-truth supervision. Moreover, even when ground-truth correctness signals are available for training, disagreement-based routing can outperform methods trained directly to predict correctness.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Weak-to-strong, model routing, cascades

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9i7jOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This work studies weak-to-strong routing for language model cascades without relying on ground-truth correctness labels. It formalises the budgeted routing problem and characterises the Bayes-optimal routing rule in terms of the gain in correctness from escalating.

Other than that, the method learns predictors for these two terms using only model-generated data and a task-specific equivalence checker, and demonstrates strong performance on both agentic and single-turn reasoning tasks.

Critical strengths

The work provides a clean, unified formulation of budgeted weak-to-strong routing that considers weak and identifies the Bayes-optimal policy via an accuracy gain score.

Other than that, the corrected disagreement score is a simple, elegant derivation from the optimal gain by replacing the unknown ground truth distribution with the strong model's predictive distribution.

At last, the decompistion intro weak strong disagreement minus stong self disagreement is intuitive and actionable; it also connects to known failure modes of raw disagreement and explains when the approximation tightens.

Critical weaknesses

There seems to be some issue with work; the key approximation is justified conceptually but lacks quantitative error bounds; no theoretical guarantees are provided on the gap between corrected disagreement and the optimal gain as a function of model miscalibration or distributional shift.

The work assumes access to a reliable disagreement indicator; when this requires an LLM judge, it may introduce its own biases and cost and could implicitly leak supervision.

The work relies on the strong model being a better proxy of ground truth; when the strong model is weak or systematically biased, corrected disagreement can mislead and mode formal robustness analysis would be valuable.

Other than the above, plots in the work are of accuracy vs escalation budget but do not present the token/latency costs for k-sample baselines or for collecting offline strong model samples during training, making Pareto comparison less complete.

At last, recent baselines that optimise for signed escalation value or provide label-free cost control are missing, so the claim of outperforming correctness predictions may be overstated.

Questions for authors

The things mentioned in the weaknesses raise some questions, such as:

  1. Can you provide quantitative bounds for the approximation error under assumptions on the divergence between q and s, or at least empirical calibration/error analyses of s vs q on labelled benchmarks?

  2. What are the training-time costs to collect K_w/K_s samples and train the two predictors, and how do they amortise over deployment? Could you report end-to-end cost-accuracy Pareto comparisons including live k-sampling overheads for baselines?

  3. When A(x, y') requires an LLM judge, how sensitive are training targets to judge errors or style variation, and do you use any calibration of adversarial pairs to improve robustness?

  4. Could you compare against Supervised Signed-Gain Routers and/or C3PO on at least overlapping benchmarks to position performance relative to recent methods with similar objectives?

  5. In the agentic settings, did you measure rescue and harm rates for escalations? If so, how do corrected disagreements and raw disagreement differ on those diagnostics, especially when the teacher is imperfect?

I would or could raise the rating if the justifications seem convincing enough.

6 comments
AuthorsOpenReview ↗

Rebuttal: Part I

We thank the reviewer for their thorough reading and feedback and are glad that they found our formulation clean, the proposed corrected disagreement score elegant and simple, and the decomposition of the final approximation into two terms actionable and intuitive. Below we thoroughly address each of the reviewer's concerns:

Additional Baselines ( Supervised Signed-Gain routers + C3PO ): We thank the reviewer for pointing us to these recent works. We have now implemented both and we will add the results and both references (SRR [1] and C3PO [2]) to the revised manuscript: (Results: Comparison with SRR: https://imgur.com/a/D2aK7fb, Comparison with C3PO: https://imgur.com/a/ObB7rNe )

Supervised signed-gain routing: Signed Rescue Routing (SRR) trains its router with ground-truth correctness labels for both the weak and the strong model. It's target, rescue minus harm, is closely related to the optimal routing gain of our Theorem 1. To thoroughly cover this family of supervised baselines, we also include an optimal-target probe that learns the two terms of Theorem 1 directly from ground-truth labels. Both use the same training prompts, features, and probe head as our scores but utilize ground-truth labels. (Results: https://imgur.com/a/D2aK7fb). Our label-free approximation recovers the gains of these supervised routers without requiring any supervision during training. Corrected Disagreement matches or exceeds SRR on all three benchmarks, and comes very close to optimal-target probe.

That said, our central motivation is to approximate this quantity without ground-truth supervision. Learning the optimal target directly requires correctness labels for both weak and strong responses, which are unavailable in the label-free settings that motivate our paper. However we agree this is a great comparison to measure the gap to supervised routing. In our main experiments, we used the standard correctness probe simply as a common proxy for weak-model reliability. This result strengthens our main point by showing that our approximation can recover a substantial portion of the gains from the optimal target without needing supervision.

C3PO: C3PO is complementary in focus. It is a test-time procedure that thresholds the weak model’s self-consistency, with thresholds chosen from agreement with the strong model under a cost constraint, whereas our contribution is the routing signal itself. Because C3P builds on self-consistency, it requires several weak samples per query; we follow its paper’s setting and give our scores the same samples so that the comparison is matched (Results: https://imgur.com/a/ObB7rNe). We compare at the routing budgets that C3PO reaches. Corrected Disagreement is ahead at every such budget on Spider and AceCode, and on MATH-500 it is ahead at low and moderate budgets, with C3PO slightly ahead only at high budgets.This is consistent with our observation in the paper (Figure 3) that multi-sample baselines close the gap only on closed-form tasks such as MATH-500, where clustering answers is trivial. We also note that the two approaches can be combined, since C3PO’s cost-control procedure can be applied on top of any confidence score, including ours.

[1] Zheyuan Wang, Siyu Li, Peiqiao Song, Sijia Chen, Qianqian Song, and Qian Liu. Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference. arXiv:2609.07786, 2026. https://arxiv.org/abs/2609.07786

[2] Antonios Valkanas, Soumyasundar Pal, Pavel Rumiantsev, Yingxue Zhang, and Mark Coates. C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning. arXiv:2511.07396, 2025. https://arxiv.org/abs/2511.07396


W1 + Q1: Quantitative bounds on the approximation error: We thank the reviewer for this suggestion. We have added a bound on the approximation error to the paper (Proposition 3). Due to space constraints, we keep the statement and its proof in Appendix C.

Corrected disagreement replaces the ground truth with the strong model’s answers, so its error depends on how often the strong model is wrong. We show that for any prompt, the gap between corrected disagreement and the optimal routing score is at most twice the probability that the strong model answers that prompt incorrectly. The more accurate the strong model, the closer corrected disagreement is to the optimal score, and the two coincide when the strong model is always correct.

Empirically, we also added an experiment that varies the strong model’s accuracy on the labelled benchmarks (see our response in Part II, https://imgur.com/a/ZZFlDVQ ), and the results follow the same trend.

AuthorsOpenReview ↗

Rebuttal Continued: Part II

Robustness to strong models of varying capability: We thank the reviewer for raising this. We study a setting where no correctness labels are available, so the strong model is our only reference for the ground truth. If the strong model is itself unreliable, there is no way to recover the ground truth from the two models alone, and this limitation is shared by any label-free method. We therefore study how the scores behave as the strong model becomes weaker.

To address this directly, we ran additional targeted analysis where we fixed the weak model (Phi-3.5-mini), the prompts and the probe configuration, and varied only the strong model across four models of varying accuracy, from 51.5% to 97.5% (Results: https://imgur.com/a/ZZFlDVQ). Over the same sequence, the shared error with the weak model, i.e. the probability that the strong model is wrong when the weak model is wrong, falls from 66.2% to 3.6%. We report results at the population level, as in Section 4.2, and with learned probes.

We report results both at the population level, using Monte-Carlo estimates of the routing scores as in Section 4.2, and in the finite-sample setting with learned probes. Our findings are as follows:

Corrected disagreement tracks optimal routing for every strong model, and the gap shrinks as the strong model becomes more accurate. This is the behavior predicted in Section 4.1. The score approximates the unknown ground-truth distribution with the strong model’s predictive distribution, so it approaches optimal routing as the strong model improves. It remains close to optimal even for the least accurate strong model, which shares a large portion of the weak model’s errors.

Weak-strong disagreement alone is less robust to strong-model errors: The raw weak-strong disagreement falls short of optimal routing relative to corrected disagreement unless the strong model is almost always accurate. A less accurate strong model also disagrees with the weak model on prompts where escalation does not help, and the self-disagreement correction can recover these cases. When the strong model is highly accurate, it rarely disagrees with itself and the two scores coincide. Results are consistent in finite samples with learned probes as well, however the gains are smaller than at the population level, since the probes are learned from finite data.

Overall, corrected disagreement remains a reliable label-free approximation of the optimal routing across a range of strong models of varying capability. We note that Appendix A.1.1 compares the routing signals across several weak–strong pairs at the population level; and we will add this controlled comparison to the revision.


W4+ Q2: Training and test-time cost: We note that we find it helpful to look at the two type of costs separately. Test-time cost. Our contribution is the signal that a router learns, and each routing score is used within the same test-time procedure. When comparing baselines we tried to keep the weak-model sampling cost matched: in the one-shot setting (N = 1) every score uses a single weak response, and in the multi-sample setting every score is given the same N samples. Within each comparison, the escalation budget on the x-axis is therefore the main cost that varies across methods. Some baselines, such as self-consistency, require more than one sample by nature, which is why they appear only in the N > 1 comparisons, while our scores can also be used with a single sample. We will state this more clearly in the revision.

Training-time cost. Here the scores do differ in the data they need, and we have added a table that reports, for each dataset, the number of weak-model samples, strong-model samples, and ground-truth labels each score uses (Results: https://imgur.com/a/bxKWVeT). Disagreement-based scores use weak and strong samples but require no ground-truth labels, and the correction reuses the same strong samples at no additional cost. Supervised scores additionally require ground-truth labels. We have further added a training-time sensitivity analysis of the probes over the number of training prompts and the number of sampled responses per model (Results: https://imgur.com/a/p2gAE2c) for completeness.

Which method to use. As a rule of thumb, when ground-truth labels are readily available, one should use them. A supervised router that learns the optimal score directly is the natural choice, and we now include it in our comparisons as supervised-baseline. Our method is intended for the many practical settings where ground truth does not exist, is not well defined, or is costly to obtain, such as multi-step agentic tasks. In those settings supervised routers cannot be trained, and on the benchmarks where both can be evaluated, corrected disagreement comes close to them without using any labels.

AuthorsOpenReview ↗

Rebuttal: Part III

Q4: Rescue and Harm rates for escalation in agentic setting: We thank the reviewer for this interesting analysis, and we include it in the revision: (Results: https://imgur.com/a/zGFyhCM). Since step-level correctness is not defined in these environments, we measure rescue and harm per task, relative to the weak model acting alone on the same task: a task is rescued if the weak model alone fails it and the routed system solves it, and harmed in the opposite case. Our findings are as follows:

With both disagreement scores, escalation rescues far more tasks than it harms. Harm remains low at every budget on both Webshop and BFCL, and the number of of rescued tasks grows with the budget.

Corrected disagreement outperforms raw weak-strong disagreement in rescue counts. On WebShop, corrected disagreement rescues substantially more tasks than weak–strong disagreement at the same budget, with similar harm, and the difference is well beyond the variation across runs. On BFCL, where the teacher is imperfect and solves only about half of the tasks on its own, the two scores have similar rescue and harm rates, consistent with our observation in the paper. In both cases harm stays low, so an imperfect teacher reduces how much can be rescued but does not make escalation harmful.


Q3 + W2: Reliability of the disagreement indicator: To address this, we ran an additional experiment on AceCode, our most open-ended benchmark, where we vary the disagreement indicator used to build the training targets. We compare the execution-based indicator with two LLM judges (GPT-4o-mini and Gemini 2.5 Flash-Lite), a much smaller LLM judge ( the 1.5B weak model itself), and an NLI model, and we report the cascade accuracy fo the probes trained under each. ( Results: https://imgur.com/a/TFD6LKd).

We note that determining whether two responses agree is a much easier task than determining whether a response is correct, and in many tasks it can be done deterministically (e.g. exact match, or running both programs on the same inputs). In our experiment, the cascade accuracy was similar across the different indicators, with the two larger LLM judges slightly ahead at higher routing budgets. We expect judges of similar capability to perform similarly, and a more reliable indicator to give a better cascade in general.

Reviewer_GXtwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for routing between weak and strong models using predicted weak–strong disagreement minus predicted strong-model self-disagreement. Two probes predict these quantities from the weak model’s hidden states, using offline model-generated training targets rather than correctness labels. At inference, the routing score can be computed without querying the strong model, which is called only after escalation.

Critical strengths
  1. A relevant problem and an interpretable routing signal. The method considers whether the strong model can improve on the weak response, while accounting for uncertainty in the strong model itself. This provides a clear motivation for the self-disagreement correction. (pp. 4–5)

  2. Meaningful task diversity. Evaluation covers mathematical reasoning, SQL generation, coding, multi-turn tool use, and interactive shopping, including both answer-level and action-level routing. (pp. 8–9)

  3. Substantial improvements in some settings. For example, on MATH-500 with one weak sample and 30% escalation, corrected disagreement achieves 56.8% accuracy versus 51.9% for raw disagreement. On WebShop at approximately 40% escalation, success increases from 26.0% to 46.0%, with standard errors reported. (p. 16, Table 1; p. 19, Table 7)

Critical weaknesses
  1. Statistical reporting is incomplete for the incremental gains. Although the agent experiments average five runs and WebShop results include standard errors, Figure 3 and the corresponding single-turn tables do not report uncertainty. This matters particularly where corrected disagreement only slightly improves on raw disagreement: at one weak sample and 30% escalation, the gains are 0.2 percentage points on Spider and 0.1 percentage points on AceCode. The reliability of these differences is therefore unclear. (p. 8, Figure 2; p. 9, Figure 3; pp. 17–19, Tables 3, 5, and 7)

  2. Robustness to strong-model errors needs further characterization. The paper explicitly relates approximation quality to the strong model’s ability to represent the true-answer distribution and discusses differing benefits on BFCL and WebShop. However, these comparisons do not isolate the effects of strong-model accuracy and shared errors with the weak model. More controlled evidence would clarify when disagreement remains a reliable indicator of the benefit of escalation. (p. 5, §4.1; pp. 8–9, §6.1)

  3. Sensitivity to probe-training data and configuration is insufficiently characterized. The learned disagreement predictions are central to routing, but the paper does not systematically evaluate their sensitivity to the amount of training data, the number of sampled responses used to construct targets, or probe configurations. Agent experiments use 10 samples per model per visited state, while some feature-extraction and hyperparameter details are deferred to the code release. The deployment-time sampling ablations do not resolve these training questions. (pp. 16–18; pp. 19–21, Appendices A.5–A.6)

  4. The incremental benefit of weak-model hidden states is not isolated. The probes use weak-model representations, but no comparison with an independent text encoder trained on the same disagreement targets is reported. This leaves unclear whether internal access provides meaningful accuracy or efficiency benefits over a text-only alternative. I view this primarily as a limitation in the justification of the implementation and its deployment scope. (pp. 19–20, Appendix A.5)

Questions for authors
  1. Can the authors provide uncertainty estimates or appropriate paired comparisons for the main single-turn results, particularly the corrected-versus-raw disagreement differences in Figure 3?
  2. How does routing performance change with strong-model accuracy and the degree of shared error with the weak model, when other factors are held fixed?
  3. How do training-set size, offline response-sample counts, and probe configurations affect disagreement prediction and downstream routing performance?
  4. How would weak-model hidden states compare with an independent text encoder using the same training examples and targets, considering both routing accuracy and computational cost? A strong-model representation comparison could offer additional insight if its extra inference cost is explicitly accounted for.
4 comments

We thank the reviewer for their thorough reading and constructive feedback. We are glad they found the motivation for self-disagreement-based signals clear, the experiments diverse and meaningful, and the improvements substantial. Below, we address all of the reviewer's concerns and suggestions:

W1 + Q1: Statistical reporting for the single-turn results: We thank the reviewer for raising this. To address it directly, we computed the 95% confidence intervals for the paired difference between corrected disagreement and every other of the routing scores in Figure 3, using a bootstrap over the test prompts (2,000 resamples). (Results: https://imgur.com/a/VbpUQ19).

The gain of corrected over weak-strong disagreement is statistically significant on MATH-500 and Spider at all three budgets, and on AceCode at lower budgets. This pattern is expected from our analysis. both scores are disagreement-based signals proposed in this work, and they differ only in the strong model's self-disagreement correction. The correction comes from estimating the unknown ground-truth distribution with the strong model’s predictive distribution, so it adds the most on top of weak–strong disagreement when the strong model is accurate (Section 4.1). On AceCode the strong model is the least accurate of the three single-turn settings, so the correction is expected to add little there. We observe the same behavior in the agentic experiments (Section 6.1): the correction yields a large gain on WebShop, where the strong policy is highly capable, and a negligible one on BFCL, where the strong model itself struggles. The same figure reports the comparison with the proxy reward and with the best label-free baseline, where corrected disagreement outperforms both on all datasets.

We thank the reviewer for their suggestion, and in the revision, we will add these pairwise comparison and confidence intervals to Figure 3 and Tables 1, 3 and 5 in the Appendix.

W2 + Q2: Robustness to strong-model errors: We thank the reviewer for this question. To address this directly, we ran a controlled experiment where we fixed the weak model (Phi-3.5-mini), the prompts and the probe configurations, and varied only the strong models across four models of varying accuracy from 51.7% to 97.5%. ( Results: https://imgur.com/a/ZZFlDVQ ). Over the same sequence of increasing accuracy, the shared error with the weak model ( i.e the probability that the strong model is wrong when the weak model is wrong, falls from 66.2% to 3.6%). We report results both at the population level, using Monte-Carlo estimates of the routing scores as in Section 4.2, and in the finite-sample setting with learned probes. Our findings are as follows:

Corrected disagreement tracks optimal routing for every strong model, and the gap shrinks as the strong model becomes more accurate. This is the behavior predicted in Section 4.1. The score approximates the unknown ground-truth distribution with the strong model’s predictive distribution, so it approaches optimal routing as the strong model improves. It remains close to optimal even for the least accurate strong model, which shares a large portion of the weak model’s errors.

Weak-strong disagreement alone is less robust to strong-model errors: The raw weak-strong disagreement falls short of optimal routing relative to corrected disagreement unless the strong model is almost always accurate. A less accurate strong model also disagrees with the weak model on prompts where escalation does not help, and the self-disagreement correction can recover these cases. When the strong model is highly accurate, it rarely disagrees with itself and the two scores coincide. Results are consistent in finite samples with learned probes as well, however the gains are smaller than at the population level, since the probes are learned from finite data.

Overall, corrected disagreement remains a reliable label-free approximation of the optimal routing across a range of strong models of varying capability. We note that Appendix A.1.1 compares the routing signals across several weak–strong pairs at the population level; and we will add this controlled comparison to the revision, and we thank the reviewer for the suggestion.

AuthorsOpenReview ↗

Rebuttal Continued : Part II

W3 + Q3: Sensitivity-analysis for the probe training data and configuration: To address this, we have added a sensitivity analysis of the learned probes across the benchmarks: (Results: https://imgur.com/a/p2gAE2c). We vary one factor at a time while keeping the test set fixed and report probe quality and routing accuracy gains as a function of the number of training prompts, the number of sampled responses per model used to build the training targets, and the probe configuration (PCA dimension and regularization strength). The main findings are: a) Routing improves steadily with more training prompts, and is already useful with very few. b) The probes need few sampled responses per prompt. Using 8 samples per model gives nearly the same routing accuracy as 32. c) Results are stable across probe configurations, and the ordering of the methods remains unchanged. We thank the reviewer for the suggestion and include this analysis in the revised manuscript.

W4 + Q4: Comparison with independent text encoder: To address this, we ran the experiments using an external text embedding model, Qwen3-Embedding-0.6B, in place of the weak model's hidden states: Results: (https://imgur.com/a/V3u1Hac). The findings are consistent and our proposed Corrected Disagreement score remains effective regardless of the the representations used to train the probes. In the revision, we add these additional experiments in the Appendix.

Dear Reviewer,

We wanted to kindly follow up, as we’ve provided detailed responses to your concerns and would appreciate any further thoughts you may have. Thank you!

The additional experiments and analysis largely address my concerns for empirical performance.

Some limitations remain, particularly the separate effects of strong-model accuracy and correlated errors, as well as computational cost comparisons. However, these do not materially change my overall assessment.

I am maintaining my rating of 3 (Weak Accept), with increased confidence in the empirical findings.

Reviewer_nS9wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study when a small model should hand a query to a larger model. Existing cascades usually decide this by predicting whether the small model’s answer is correct, which requires ground truth labels that are costly or not well-defined in agentic tasks. For this, the authors first derive the optimal escalation rule under budgets on small-model samples and large-model calls, which is to escalate when the large model’s chance of being correct exceeds that of the best small-model answer. They then replace the unknown ground truth with the large model’s own answer distribution. This turns the rule into a label-free score. Specifically, how often the large model disagrees with the small model’s answer, minus how often the large model disagrees with itself. Across two agentic environments and three single-turn benchmarks, the score outperforms label-free baselines. At low budgets it also outperforms a probe trained on correctness labels. On WebShop at a 20% budget, it reaches 0.210 success versus 0.045 for self-consistency. The authors found that subtracting the large model’s self-disagreement helps most when the large model is strong.

Critical strengths

Prior cascade methods, such as FrugalGPT and AutoMix, depend on correctness labels to learn when to escalate. In agentic tasks, these labels might be hard to get. Further, LLM judges are unreliable on agent trajectories, and an action that looks reasonable can fail several steps later. In this paper, the authors propose a routing signal that only checks whether two answers match which removes this dependence on correctness labels. This makes routing possible even in settings where existing supervised cascades cannot be trained.

Critical weaknesses

The current experiments do not clearly isolate whether the disagreement target is what helping. The router is a probe trained on the small model’s hidden states. The only single-pass baseline it is compared against is token entropy, which involves no training. The authors should compare with two additional baselines: a) which trains a label-free probe on the same kind of hidden states, and b) a probe trained to predict the small model’s disagreement with itself, using the same pipeline but no large-model samples. Without these, the source of the gains is not clear.

Questions for authors
  1. Just curious, if the authors have any insight on ehy does random routing on WebShop outperform self-consistency, token entropy, and raw weak–strong disagreement at a 20% budget in Table 7?

  2. The supervised baseline learns only whether the small model is wrong, so it also escalates queries that the large model gets wrong. How does the method compare with a supervised probe trained to predict whether escalating turns a wrong answer into a right one, which Theorem 1 shows is the optimal target?

2 comments

We sincerely thank the reviewer for their thoughtful reading and constructive feedback. We are glad that they appreciated the core motivation and strength of our work, particularly its ability to enable effective routing without relying on ground-truth correctness labels. To thoroughly address the reviewer's concerns, we conducted additional experiments and address each point in detail below:

Comparison against a label-free weak-weak disagreement probe: We thank the reviewer for their suggestion. To address it directly, we implemented an additional baseline that learns to predict disagreement between two independently sampled responses from the weak model, using the same training pipeline and hidden states as our other probes. We evaluate this comparison at two levels:

a) At Population-Level Comparison ( Monte Carlo): (Results: https://imgur.com/a/LiKGdJS) At the population level (decoupled from finite-sample training noise), our corrected disagreement score significantly outperforms the weak-weak disagreement baseline across all benchmarks. Intuition: Our derivation approximates two quantities in the optimal routing signal: the weak model correctness and the strong model correctness. Weak-weak disagreement can be seen as approximating the ground-truth distribution with the weak model's own distribution, which by definition is a coarser approximation than the strong model's distribution.

b) Finite-Sample Probes: In the finite-sample setting, the probes trained to predict weak-weak disagreement generalize reasonably well and perform similarly to raw weak-strong disagreement. However our full approximation (corrected disagreement) outperforms both on all benchmarks. (Results with weak-model's hidden states as input: https://imgur.com/a/VrC9mPT; Results with a separate embedding model representations as input: https://imgur.com/a/qVsaFti).

Isolating the effect of representation: To isolate the effect of the representation, we repeated the experiments using an external text embedding model, Qwen3-Embedding-0.6B, instead of the weak model's hidden states: Results: (https://imgur.com/a/qVsaFti). The findings are consistent and our proposed corrected disagreement routing remains effective regardless of the the representations used to train the probes.

In the revision, we have added the weak-weak disagreement probe as an additional baseline in section A.10.1 and included the text-embedding experiments in A.10.2.

Q2: A Supervised probe on the optimal target: To address this, we trained a ground-truth supervised probe to directly predict the optimal correctness gain from escalation (Theorem 1) using the same learning pipeline (Results: https://imgur.com/a/xa0XrNo). Our label-free approximation recovers most of the gains without requiring any supervision during training, performing competitively to this fully supervised oracle. (Section A.10.3).

That said, our central motivation is to approximate this quantity without ground-truth supervision. Learning the optimal target directly requires correctness labels for both weak and strong responses, which are unavailable in the label-free settings that motivate our paper. However we agree this is a great comparison to measure the gap to supervised routing. In our main experiments, we used the standard correctness probe simply as a common proxy for weak-model reliability. This result strengthens our main point by showing that our approximation can recover a substantial portion of the gains from the optimal target without needing supervision.

Q1: Random routing on Webshop outperforming baselines at low budgets: We thank the reviewer for this sharp observation and we investigated this by examining the instances selected for escalation. The main reason is that high weak-model uncertainty does not necessarily imply high value from using the strong model. At low routing budgets, uncertainty-based methods such as self-consistency and token entropy prioritize the instances on which the weak model is most uncertain. However, these instances can also be difficult for the strong model, so escalating them does not necessarily improve the outcome. Random routing, by contrast, sometimes selects instances where the weak model is incorrect but relatively confident, and where the strong model is able to correct the mistake. This explains how random routing can outperform uncertainty-based policies at small budgets.

Our corrected disagreement score addresses this precisely, rather than relying on uncertainty in the weak model, it accounts for both weak–strong disagreement and uncertainty within the strong model and achieves a closer approximation to the expected optimal gain from escalation.

We also note that at small budgets, relatively few strong-model interventions are made, so the resulting performance estimates can be more sensitive to which instances are selected. We will clarify this behavior in our discussion of the WebShop results.

Dear Reviewer,

We wanted to kindly follow up, as we’ve provided detailed responses to your concerns and would appreciate any further thoughts you may have. Thank you!

Comments on the paper

1 comment

We thank all reviewers for their feedback. Our responses are now complete, and we kindly ask the reviewers to read them when they get a chance. We will also soon upload an updated manuscript that includes all the experiments, plots, and results already provided in the responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.