acceptodds
Withdrawn from ICLR 2027

Support Before Sign: Identifying Deferral Gain in LLM Cascades

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A cascade must decide whether replacing a small language model's answer with a larger model's would improve the graded outcome. Under answer-factored grading, this signed deferral gain factorizes into the probability of answer disagreement and the expected benefit conditional on disagreement. The factors require different supervision. Unlabeled paired outputs point-identify disagreement support but leave the signed margin undetermined. Under canonical exact-match grading and an unrestricted conditional gold-answer law, we characterize the sharp gain interval and show how observationally equivalent outcome laws induce opposite routing orders. Selective outcome labeling identifies the missing margin under conditional independence and positivity. This information structure motivates *Support Before Sign*: learn support from paired outputs without gold labels, learn the margin from labeled disagreements, and route on their product. On a fixed model pair, signed supervision at the main labeled budget improves integrated accuracy over call rate under both the original and equal-weight task mixtures. Its matched-token-cost benefit depends on the mixture, while observation-matched direct estimators perform better at some low-deferral operating points. The results distinguish three questions: what paired outputs identify, what signed labels add, and how the estimator converts that information into a routing policy.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM cascades, model routing, learning to defer, partial identification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KRtpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when to replace a small language model's response with that of a larger, potentially stronger model. Rather than directly estimating signed deferral gain, it factorizes that gain into the probability of answer disagreement and the expected signed gain conditional on disagreement. Under answer-factored grading, the authors characterize what paired outputs alone identify and fit a two-stage router using labels collected on sampled disagreements. Experiments routing from Qwen2.5-1.5B-Instruct to Qwen2.5-7B-Instruct show improvements in integrated accuracy versus call rate, while advantages at particular call rates and token costs vary. My primary concern is the practical significance and scope of the contribution, including its extension beyond tasks where answer disagreement is straightforward to define.

Critical strengths
  1. A useful approach to efficient supervision. Under answer-factored grading, paired outputs identify where replacement can matter, while outcome labels determine whether it helps. Agreement provides zero-gain supervision, allowing grading to focus on disagreements (§§3.2–3.4). The bounds and selective-label design are useful even though the factorization is elementary and the estimator does not dominate at every call or label budget.
Critical weaknesses
  1. The practical contribution remains insufficiently established. Label-budget experiments and observation-matched controls support learning a routing signal (§§5.2–5.3), but the signed-stage increment depends on the budget and operating point. The main evaluation uses one fixed model pair and readily normalized answers. It remains unclear whether simple query/confidence cues or hidden-state features drive the gains, and how the method performs when answer equivalence is harder to define. The imperfect-equivalence bound does not establish an empirical implementation. Unexplained benchmark exclusions further limit the reported scope.

  2. The figures and tables provide insufficient intuitive guidance. Figure 1 is difficult to follow, while dense tables and overlapping curves obscure the central routing comparisons (Figures 2–3).

Questions for authors
  1. Excluded conditions and selection rationale. Which benchmark was omitted from the primary collection after task-mixture and token-cost analyses, which family was omitted from the wider sweep, and why? What do their existing outcomes imply for the routing benefit, given the reuse of earlier test queries? Was the selection based on outcome-independent technical reasons?

  2. Presentation suggestion. Figure 1 needs a concrete link between disagreement probability, signed improvement and the routing decision. The result figures and tables need more distinguishable curves and clearer router–baseline comparisons at fixed call or label budgets.

4 comments

Comment 1/3: Feature contributions and the label budget

Thank you for asking which information drives the gain and when signed supervision is useful. We added feature ablations to separate the contribution of hidden states, followed by independent evaluations to test the label-budget dependence on new queries.

Query/confidence features versus hidden states

In these comparisons, hidden states add accuracy, and Stage 2 also improves over Support when only scalar features are available. We compared ten query/confidence summaries with the same summaries plus small-model hidden states, holding the model pair, data, revealed outcome labels, and training-only tuning procedure fixed. Labels come from benchmark gold and simulate a limited outcome-label budget. The scalar features describe query form, sequence length, token entropy, and token probabilities. Hidden states are extracted after the small model's final normalization and reduced using training data. Large-model states are never router inputs.

For Qwen3.5-9B to Qwen3.5-27B, with 1,000 outcome labels and 20% deferral:

ComparisonTaskAccuracy increment, percentage points [95% interval]
Combined features minus scalar-only Stage 2Natural Questions (NQ)+0.353 [0.060, 0.600]
Combined features minus scalar-only Stage 2TriviaQA+1.213 [0.727, 1.780]
Scalar-only Stage 2 minus scalar-only SupportNQ+0.560 [0.147, 1.000]

The first two comparisons measure the additional value of hidden states. The third establishes a signed-stage increment with hidden states removed. Query and confidence summaries are grouped, so this experiment does not isolate their individual contributions. These are exploratory, pointwise query-bootstrap intervals on previously examined evaluation data, conditional on five fitted label draws and without correction for multiple comparisons.

How much signed supervision is needed?

We then tested the budget dependence independently, using Qwen3.5-9B to Qwen3.5-27B on newly held-out NQ and TriviaQA queries. Each task used 8,000 training queries and 10,000 evaluation queries. The seven label budgets were 25, 50, 100, 250, 500, 1,000, and 2,000, with 20 label-sampling runs and subsets nested across budgets. Methods, tuning ranges, and evaluation criteria were fixed before test access; all training and tuning finished before evaluation.

At 20% deferral, Stage 2's accuracy increment over Support was:

TaskOutcome labelsIncrement, percentage points [simultaneous 95% interval]
NQ1,000+0.711 [0.198, 1.223]
NQ2,000+1.208 [0.677, 1.738]
TriviaQA1,000+0.765 [0.200, 1.329]
TriviaQA2,000+1.021 [0.384, 1.657]

These approximate simultaneous 95% bootstrap intervals are adjusted across the study's 96 prespecified comparisons. Budgets from 25 to 500 did not establish a positive increment. Resampling covers evaluation queries and label draws, conditional on the training pool, fitted representation and Support model, and fixed generations.

Comment 2/3: Model-pair coverage and practical routing outcomes

A separately trained cross-family cascade

To test whether the signed-stage benefit extends beyond one model pair, we compared Qwen3.5-9B to Qwen3.5-27B with Mistral-7B-Instruct-v0.3 to Qwen3.5-27B on another common set of 10,000 Natural Questions queries. Each pair used 8,000 training queries, 1,000 gold-free calibration queries, and its own fitted representation, Support model, and supervised heads. The methods and tuning ranges were frozen, and all training and tuning finished before test evaluation.

At 20% deferral:

Model pairOutcome labelsStage 2 minus Support, percentage points [simultaneous 95% interval]
Qwen to Qwen500+0.359 [-0.006, 0.723]
Qwen to Qwen2,000+1.737 [1.240, 2.234]
Mistral to Qwen500+2.931 [2.366, 3.495]
Mistral to Qwen2,000+3.819 [3.183, 4.454]

The Mistral-to-Qwen cascade shows a signed-stage benefit at both budgets; the Qwen-to-Qwen increment is positive at 2,000 labels and inconclusive at 500. The approximate simultaneous bootstrap intervals are adjusted across 24 prespecified utility/harm comparisons. They resample shared queries and 20 label-sampling runs, conditional on the training pool, fitted representation and Support model, and fixed generations. Each pair has a separately trained router. Model size, accuracy, and answer distributions also differ, so the larger Mistral increment cannot be attributed specifically to model family.

Net improvement and harmful substitutions

We counted rescued errors and harmful substitutions to determine what produces the accuracy increment. A rescue replaces an incorrect small-model answer with a correct large-model answer; harm makes the reverse substitution. Their difference is the net number of additional correct answers. At 2,000 labels and 20% deferral, averages per 1,000 total queries were:

PairMethodRescueHarmNet
QwenSupport13.8000.50013.300
QwenStage 234.8754.20530.670
MistralSupport25.3000.20025.100
MistralStage 263.9100.62563.285

The net increments arise because additional rescues exceed additional harm. Qwen's harm increase is positive under the simultaneous interval; Mistral's harm difference remains uncertain.

We also compared Stage 2 with a two-head direct estimator (twohead) that fits common correctness and large-minus-small difference components and routes using the difference. It receives the same revealed outcome labels and validation folds. For Stage 2 trained as specified in the paper versus fully tuned twohead, all four net-gain intervals include zero. A separate comparison matches coefficient counts in the margin and difference routing heads, preprocessing, final-gain validation, and ten tuning candidates per method. The matched two-stage variant improves Mistral accuracy by 0.459 points [0.282, 0.635] at 500 labels and 0.260 [0.106, 0.413] at 2,000, while reducing actual harm. The fitting losses still differ between conditional-margin and direct-gain estimation. These results concern the matched variants; the table above reports Stage 2 with the paper's training procedure.

Preparation and serving costs

To measure the resources required for these gains, we recorded paired generation, feature extraction, representation/Support tuning, head tuning, and calibration. At 2,000 labels, the preparation subtotal is 2.44 hours for Qwen and 2.66 hours for Mistral on one NVIDIA GB10. Support, Stage 2, and twohead share the paired pool and representation; the two supervised methods additionally require 2,000 outcome labels.

A separate replay of 512 common queries uses one prespecified 2,000-label fit, three timing repetitions, and batch size 1. Stage 2's mean service times are 1.004/1.432 seconds for Qwen/Mistral, versus Support's 1.189/1.570 seconds. The replay uses calibrated thresholds and separately graded outputs; actual call rates differ slightly. Shorter large-model generations among the selected queries account for much of this timing difference. Annotation labor and some acquisition/preprocessing costs remain unmeasured, and the repeated timings do not create additional independent quality samples.

Comment 3/3: Answer equivalence, excluded conditions, and presentation

Semantic equivalence beyond normalized short answers

The completed independent evaluations concern short-answer routing under normalized exact-match grading. They broaden task and model coverage, but they do not establish semantic equivalence for unrestricted answers. The imperfect-equivalence bound gives a criterion for assessing a matcher; it does not validate an implementation.

We have prepared 400 blinded answer-pair audit cases from Natural Questions and TriviaQA to assess this issue. The rubric separately asks whether the answers are semantically equivalent and whether each is correct. Stratified sampling weights are retained for estimating population rates. Human annotations are pending, so there is no completed semantic-validation result to report. These cases use previously generated short answers and do not constitute a long-form generation evaluation.

The relevant implementation criterion is grading consistency. Let HH be the large-model grade minus the small-model grade, DD indicate disagreement under the matcher, and ZZ denote router-visible features. If the matcher declares agreement despite different grades, the factorization omits E[H1D=0∣Z]\mathbb{E}[H\mathbf{1}\\{D=0\\}\mid Z]. For grades in [0,1][0,1], the magnitude of this term is bounded by the conditional probability of such grade-changing false agreements. The audit must therefore assess both equivalence and grading consistency. Equal correctness scores alone do not establish semantic equivalence.

Excluded benchmarks and selection rationale

The available records do not establish the names, outcomes, or selection rationale for the excluded benchmark and task family. Appendix B documents retaining two of three benchmarks and ten of eleven families after earlier mixture/token-cost analyses, with reuse of test queries. We cannot establish that those exclusions were outcome-independent and will state this selection limitation in the main text.

The new studies directly test the questions about feature contributions, label budgets, and model-pair coverage raised in your review. For the independent evaluations, methods, tuning ranges, label budgets, and evaluation criteria were frozen before test access, and all training and tuning finished before the results were opened. The partitions exclude exact and near matches to retained historical queries. They are drawn from official training splits and held out from router development; they are not the official benchmark test sets. This protocol strengthens the additional evidence while leaving the historical selection question unresolved.

Concrete routing decisions and readable comparisons

We will revise Figure 1 around two queries. For query A, the disagreement probability and conditional margin are (0.9,-0.2); for B they are (0.4,0.8). Support ranks A first because disagreement is more likely. Their expected gains are -0.18 and +0.32, so the product ranks B first. The example connects the learned quantities to the replacement decision.

The result figures will show differences at fixed call budgets, with intervals and distinct line styles. The submitted operational tables already make these comparisons possible: Support exceeds entropy by 0.797 and 1.148 accuracy points at 10% and 20% deferral, while Stage 2 with 2,000 labels exceeds Support by 0.842 and 0.771 points. We will bring these comparisons into the main presentation instead of asking readers to estimate gaps between overlapping curves. The original split/label-fit stability analyses will remain distinct from the new independent evaluations.

Thank you for the additional experiments and clarifications. They strengthen the evidence that signed supervision can improve routing. However, two concerns remain:

  • The incremental value of the factorization remains unclear. Improvements over Support establish the benefit of adding outcome labels. While the positive matched-control results are useful, the evidence for an advantage over direct gain estimation remains limited to particular settings.
  • The original evaluation remains insufficiently traceable. The statement that the excluded benchmarks, outcomes, and selection rationale cannot be established does not resolve my question. These are basic facts needed to assess the reported evidence. Maintaining and providing an accurate account of the experiments underlying the reported conclusions is a basic responsibility in scientific research. Please verify these details against the experimental records. If they cannot be recovered, please explain which original conclusions depend on this evaluation and which are independently supported by the new studies.

I appreciate the additional work, but these issues remain important to my assessment of the contribution and reliability.

Reviewer_YFJTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies LLM cascade routing by decomposing signed deferral gain into disagreement probability and conditional gain. It proposes a two-stage method that learns these components from unlabeled outputs and labeled disagreements, respectively. Experiments show modest improvements from signed supervision.

Critical strengths
  1. The authors propose a support–margin decomposition of signed deferral gain, providing a clear framework for understanding LLM cascade routing.

  2. The theoretical analysis characterizes the limitations of support-only routing and highlights the necessity of labeled data for identifying signed deferral gain.

Critical weaknesses
  1. Limited novelty. Previous work (e.g., Jitkrittum et al., 2023) has already studied signed deferral gain and established its optimality for two-model cascades. What is the main novel contribution of this paper? What new insights or practical takeaways do the theoretical results provide?

  2. Limited improvement from Stage 2. In Table 2, the Factorized router shows only marginal improvement over Support-only. However, Table 4 shows that the smaller model wins in 6.4% of all queries, suggesting potential benefits from learning signed gain. Could the authors report the prediction accuracy (e.g., AUROC) of the router after Stage 2 compared with Stage 1, particularly in identifying cases where deferral is harmful?

Questions for authors

Please refer to the weaknesses listed above.

3 comments

Comment 1/3: What the identification results add to signed-gain routing

Thank you for asking us to clarify the contribution relative to Jitkrittum et al. and to examine harmful deferrals directly.

Jitkrittum et al. [1] establish signed deferral gain as the optimal routing target and study direct estimation from correctness labels. Our contribution characterizes which supervision identifies that target. Paired model outputs reveal part of the required information without correctness labels; disagreement outcomes identify the remaining signed component under the conditions below.

Let H=Yℓ−YsH=Y_{\ell}-Y_s be the large-model grade minus the small-model grade, DD indicate answer disagreement, and ZZ denote features available before calling the large model. Under a common answer-factored grader, agreement gives H=0H=0 even when the shared answer is incorrect. Writing δ(z)\delta(z) for disagreement probability and m(z)m(z) for expected gain conditional on disagreement gives

g(z)=E[H∣Z=z]=Pr⁡(D=1∣Z=z)E[H∣D=1,Z=z]=δ(z)m(z).g(z)=\mathbb{E}[H\mid Z=z]=\Pr(D=1\mid Z=z)\mathbb{E}[H\mid D=1,Z=z]=\delta(z)m(z).

The theoretical results go beyond this conditioning identity:

  • The unlabeled paired-output distribution identifies δ\delta. Under canonical exact-match grading with unrestricted conditional gold-answer laws, the sharp compatible interval for gg is [−δ,δ][-\delta,\delta]. Thus, the same observable distribution permits opposite signed gains.
  • Among routing policies with 0≤π≤10\leq\pi\leq1 and call budget E[π(Z)]=B\mathbb{E}[\pi(Z)]=B, let UBU_B and LBL_B be the largest and smallest attainable E[π(Z)δ(Z)]\mathbb{E}[\pi(Z)\delta(Z)]. Let RB+(π)R_B^+(\pi) and RB−(π)R_B^-(\pi) denote regret relative to the optimal policy at that budget under the two endpoint laws. Then RB+(π)+RB−(π)=UB−LBR_B^+(\pi)+R_B^-(\pi)=U_B-L_B. When UB>LBU_B > L_B, a router based only on the unlabeled law incurs positive regret under at least one endpoint. More paired generations cannot remove this ambiguity.
  • Disagreement outcomes identify mm when label selection is independent of outcomes conditional on (Z,D)(Z,D), with positive selection probability wherever δ>0\delta > 0.

These statements specify the information gained from each supervision source. For answer-factored grades in [0,1][0,1], [−δ,δ][-\delta,\delta] remains an outer bound; sharpness has the exact-match and outcome-law assumptions stated above.

The practical consequence is a supervision design. We learn disagreement from the entire paired pool and concentrate outcome labeling on disagreements. Agreement supplies known zero-gain outcomes: the factorized estimator uses agreements when fitting Support, while our direct controls also use them in their gain-fitting loss. Support estimates δ\delta without gold outcomes; Stage 2 estimates the conditional margin mm and routes by their product. Deployment uses only information available before the large-model call. Our direct controls receive the same paired outputs, representation, and revealed outcome labels, allowing an empirical comparison of how the estimators use that information.

This also explains why a fitted Stage 2 increment can be small. If m(z)m(z) is a positive constant, the population Support score δ(z)\delta(z) already orders queries by expected gain, leaving no population ranking improvement to recover. When mm varies, a small label budget can still yield an inaccurate margin estimate, and direct regression can estimate the same target more accurately in finite samples. The identification results motivate the supervision design; the experiments below test its benefit at different budgets and against direct estimation.

[1] Jitkrittum et al. (2023). When Does Confidence-Based Cascade Deferral Suffice?. NeurIPS.

Comment 2/3: Requested harm diagnostic and the source of the Stage 2 gain

Stage 2 does not improve general harm discrimination in our new independent evaluations. We ran these diagnostics specifically to answer the requested comparison with Stage 1 and to determine how harm ranking relates to actual routing utility.

We evaluated Qwen3.5-9B to Qwen3.5-27B and Mistral-7B-Instruct-v0.3 to Qwen3.5-27B. Each pair used 8,000 training queries and 20 label-sampling runs, with subsets nested across budgets, followed by 10,000 shared held-out Natural Questions (NQ) short-answer queries. Methods and tuning ranges were frozen before test access; all training and tuning preceded evaluation. These are new cohorts, separate from the submitted cohort with 6.4% potential harm.

For H=Yℓ−YsH=Y_{\ell}-Y_s, harm means H=−1H=-1 under binary exact-match grading. Every method uses the negative routing score to detect harm. AUROC, the area under the receiver operating characteristic curve, measures how well the score ranks harmful against non-harmful cases. Support is Stage 1; Stage 2 uses the support-margin product. The two-head direct estimator (twohead) fits common-correctness and grade-difference components and routes with the latter; Comment 3 details the controls. Cells report all queries / disagreements only, averaged across label draws:

PairLabelsSupportStage 2Tuned twohead
Qwen to Qwen500.658 / .819.625 / .758.589 / .662
Qwen to Qwen2,000.658 / .819.573 / .636.580 / .635
Mistral to Qwen500.834 / .918.740 / .799.771 / .811
Mistral to Qwen2,000.834 / .918.698 / .736.763 / .798

Stage 2 is below Support in both evaluation populations in all four conditions. To explain what its utility improvement represents, we also counted actual rescues and harmful substitutions. For binary grades,

g(z)=Pr⁡(H=1∣Z=z)−Pr⁡(H=−1∣Z=z).g(z)=\Pr(H=1\mid Z=z)-\Pr(H=-1\mid Z=z).

A fixed-budget accuracy objective rewards rescues and penalizes harm. Even an exact gain score need not order queries by harm probability, because rescue probability also varies. For example, rescue/harm probabilities of 0.60/0.20 give higher expected gain than 0.25/0.01, despite higher harm risk.

At 2,000 labels and 20% deferral, counts per 1,000 total queries are:

PairMethodRescueHarmNet gain
QwenSupport13.8000.50013.300
QwenStage 234.8754.20530.670
QwenTuned twohead34.6854.73029.955
MistralSupport25.3000.20025.100
MistralStage 263.9100.62563.285
MistralTuned twohead61.9450.31061.635

Relative to Support, Qwen Stage 2 adds 21.075 rescues and 3.705 harmful substitutions per 1,000 queries; Mistral adds 38.610 and 0.425. Net accuracy improves by 1.737 percentage points [1.240, 2.234] and 3.819 [3.183, 4.454]. These approximate simultaneous 95% bootstrap intervals are adjusted across 24 prespecified utility/harm comparisons. Qwen's additional-harm interval is [2.047, 5.363] per 1,000 queries; Mistral's is [-0.233, 1.083]. The gain comes from additional rescues exceeding additional harm.

At 500 labels, the Stage 2 increments are 0.359 points [-0.006, 0.723] for Qwen and 2.931 [2.366, 3.495] for Mistral. To test label-budget dependence more broadly, a preceding independent study evaluated the same Qwen pair at 25, 50, 100, 250, 500, 1,000, and 2,000 labels. Each task used 8,000 training queries, 20 label-sampling runs, and 10,000 held-out queries from NQ or TriviaQA; tuning preceded test access. Increments were positive at 1,000 and 2,000 labels under simultaneous intervals; smaller budgets were inconclusive. At 2,000 outcome labels, the submitted experiment reports a routing AUC (area under accuracy versus deferral rate) increment of 0.00278 and a 20%-deferral accuracy increment of 0.771 points. These describe different operating summaries.

Using the negative raw margin to distinguish H=−1H=-1 from H=+1H=+1 among nonzero-gain disagreements, AUROC rises from .494 to .669 for Qwen and .662 to .735 for Mistral between 500 and 2,000 labels. This describes the margin component; it does not by itself explain the product's lower harm AUROC.

The utility/harm intervals resample queries and label draws, conditional on each training pool, fitted representation/Support, and model generations. AUROCs are exploratory. The findings distinguish predictable harm from potential harm: its prevalence alone does not establish that router-visible features identify it.

Comment 3/3: Fair direct-estimation controls and tests of agreement and capacity

To compare factorization with direct gain estimation, we evaluated Stage 2 and a two-head direct estimator (twohead) under the same available supervision. We then matched tuning and parameter counts in the margin and difference routing heads more strictly and tested two possible explanations for remaining differences.

The twohead control fits a common correctness component and a large-minus-small difference component; the latter routes queries. It receives the same paired outputs, representation, outcome-label IDs, and validation folds as Stage 2. Labels are revealed from benchmark gold, and agreement zeros enter the direct estimator's gain-fitting loss. In the linear implementation the two components decouple, so the common head supplies no extra gold outcomes to the difference head.

On the independent Natural Questions (NQ) evaluation described in Comment 2, Stage 2 following the paper's training procedure and tuned twohead use different selection objectives and search sizes: 10 candidates for the conditional-margin fit versus 50 for the final-gain fit. At 20% deferral:

PairLabelsStage 2 minus tuned twohead, accuracy percentage points [simultaneous 95% interval]
Qwen to Qwen500-0.411 [-0.828, 0.007]
Qwen to Qwen2,000+0.072 [-0.120, 0.263]
Mistral to Qwen500+0.126 [-0.182, 0.433]
Mistral to Qwen2,000+0.165 [-0.025, 0.355]

All four intervals include zero. Thus, the positive Stage 2 increments over Support establish the benefit of signed supervision in the reported conditions, while this comparison leaves its net-utility ordering against tuned direct estimation unresolved.

To compare the estimators under matched selection conditions, we used controls with the same number of coefficients in the margin and difference routing heads, preprocessing, final-gain validation objective, and ten candidates per method. The fitting losses still differ: Stage 2 fits a conditional margin on disagreements and multiplies it by Support, while the direct control fits gain and uses agreement zeros in its loss. Both variants were fixed before test access. For Mistral, the matched two-stage variant improves accuracy over the matched direct control by 0.459 points [0.282, 0.635] at 500 labels and 0.260 [0.106, 0.413] at 2,000 labels. It reduces harm by 0.985 [0.391, 1.579] and 0.545 [0.065, 1.025] substitutions per 1,000 total queries. For Qwen, net-gain differences remain unresolved, while harm is lower for the matched two-stage variant at both budgets. This supports the matched two-stage estimator on Mistral; it does not isolate the factorization identity from its fitting loss and induced function class.

We also investigated whether agreement constraints or insufficient Stage 2 capacity explained the earlier exploratory results:

  • Agreement constraints. Direct estimation uses agreement's known zero gain. To test whether using that constraint explicitly in the two-stage fit would help, we added an agreement penalty and compared it with a matched unpenalized fit. At 1,000 labels and 20% deferral on NQ, the mean accuracy difference was 0.000 points [-0.060, 0.060]. This change did not produce a detectable improvement in that setting.
  • Nonlinear Stage 2. To test the linear-margin restriction while retaining the decomposition, we used 128 random Fourier features approximating a radial-basis kernel. The linear and nonlinear versions had corresponding twohead controls with matched parameter counts for the margin and difference routing heads and ten tuning candidates each. On NQ at 1,000 labels, disagreement-only harm AUROC (area under the receiver operating characteristic curve) improved by 0.0436 over the matched linear Stage 2, while the 20%-deferral accuracy increment of 0.100 points [-0.073, 0.313] was inconclusive. The gap to the matched twohead did not clearly shrink. This tested increase in capacity improved harm ranking but did not establish a utility improvement.

Both checks used previously examined data and pointwise intervals without multiplicity correction. They assess specific proposed explanations; the independent comparisons above provide the main evidence for signed supervision and the matched estimators.

Reviewer_ZcGaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors study two-model LLM cascades (in which a small model answers all queries and a router escalates some to a larger model) and propose a method for how to best route answers to the large model.

The proposed method is conceptually simple, learning δ^\hat{\delta} from unlabeled paired outputs of both models, learning m^\hat{m} from labeled paired outputs where the models disagree, and routing based on the score g=δ^m^g = \hat{\delta}\hat{m}.

They apply their method to a single model-pair (small: Qwen2.5-1.5B-Instruct, large: Qwen2.5-7B-Instruct) on two datasets (GSM8K and MMLU, 15.3k queries in total).

Critical strengths
  • The studied problem of how to best route answers in a two-model LLM cascade is quite interesting. The proposed method is conceptually simple and makes intuitive sense overall.
Critical weaknesses
  • The experimental evaluation is quite limited with just a single model-pair and two datasets, the two evaluated models are also relatively small (1.5B and 7B parameters) and not overly recent.

  • The experimental results are not particularly impressive/convincing (the results for "support-only" is close to the baselines in Section 5.1 and Table 1, the results for "factorized" are very close to "support-only" in Table 2, Figure 2 and Figure 3), it is therefore somewhat unclear how much value the proposed method actually adds.

Questions for authors

Questions/suggestions:

  • Could you perhaps clarify why the theoretical results in Section 3 are important, why these are needed? That unlabeled pairs can be used to estimate δ\delta and labeled disagreement-pairs can be used to estimate mm feels rather intuitive, so do we really need all this theory?

  • Table 1, this comparison of "support only" with the other baselines is not really fair though, as "support only" requires both the small and large models to first be run on the entire pool of questions?

  • Table 1, not entirely clear to me what "realized-gain reference (clairvoyant)" means here. Is this an oracle where you for each question pick the best of the two models?

  • Table 1, the performance of support-only (0.668) is not that much better than the simplest possible baseline random (0.639)? Especially not given the quite substantial gap up to the oracle (0.758)?

  • Section 5.2: The results for support-only and factorized/product are basically identical (0.668 vs 0.670)? On GSM8K, support-only is even a tiny bit worse? Is this really working then? In the left of Figure 2, the blue "support" line is also very very close to 0? In the right of Figure 2 as well, given the y axis range? And also in Figure 3, the blue and black lines are very close?

  • Section 5, "Each of the seven fixed splits partitions the collection into a pool of 9,216 and a test set of 6,145": Why seven splits? Seems like a quite specific number, why not e.g. 5 or 10?

  • In Section 4.2 you write that m^\hat{m} should be clipped to [-1, 1]. In that case, you shouldn't need to use clip(m^\hat{m}, -1 ,1) (in Figure 1, Section 4.3, etc)?

  • At first read, I found the abstract a bit difficult to follow/understand. I also found Section 5.3 - 5.5 less clear than most of the rest of the paper.

Minor things:

  • Figure 1 is quite neat, could perhaps be pointed to earlier in the paper, in the Introduction?
  • Don't think you need to use that many digits in the results in Table 1 and 2.
4 comments

Comment 1/4: Stage 2's benefit at fixed call budgets and on larger models

Thank you for the detailed questions. The submitted full-range routing-area increment is 0.00278 at 2,000 outcome labels, consistent with the small gap between the rounded values 0.668 and 0.670. At the separately reported 10% and 20% call budgets, Stage 2 improves accuracy over Support by 0.842 and 0.771 percentage points. The respective 95% resampling stability intervals are [0.665, 1.015] and [0.613, 0.931] points. These summarize seven overlapping partitions and five label fits; the partitions are not independent replications.

The task-specific results also depend on the metric. For Grade School Math 8K (GSM8K), Stage 2 changes the full routing area under the accuracy-versus-deferral-rate curve (AUC) by -0.00047, while improving accuracy at 20% deferral by 0.103 percentage points. For Massive Multitask Language Understanding (MMLU), the latter improvement is 0.753 points. This analysis ranks the mixed-task routers within each task at separate call budgets. We will retain both task results and bring the fixed-budget comparisons into the main text.

To test whether the signed-stage increment extends beyond the original small-model pair, we evaluated Qwen3.5-9B to Qwen3.5-27B and Mistral-7B-Instruct-v0.3 to Qwen3.5-27B on a common new Natural Questions (NQ) collection. Each pair used 8,000 training queries, 1,000 calibration queries without gold labels, and 10,000 held-out evaluation queries. These are new partitions of the official training split, held out from router development. Each pair has its own representation, Support fit, and supervised heads. Methods, tuning ranges, budgets, and evaluation criteria were fixed before test access; all training and tuning finished before evaluation.

At 20% deferral, net gain is rescued errors minus harmful substitutions, divided by all queries. Its change equals the accuracy change:

Model pairOutcome labelsStage 2 minus Support, percentage points95% simultaneous interval
Qwen to Qwen500+0.359[-0.006, 0.723]
Qwen to Qwen2,000+1.737[1.240, 2.234]
Mistral to Qwen500+2.931[2.366, 3.495]
Mistral to Qwen2,000+3.819[3.183, 4.454]

The 2,000-label increments amount to approximately 174 and 382 additional correct answers per 10,000 queries, on average across label draws, at the same large-model call budget. Three interval lower bounds exceed our prespecified practical reference of 0.2 percentage points. The Qwen result at 500 labels remains inconclusive.

These approximate simultaneous 95% bootstrap intervals adjust for 24 prespecified accuracy/harm comparisons. Resampling covers shared evaluation queries and 20 label-sampling runs, with subsets nested across budgets, conditional on the training pool, fitted representation and Support, and fixed generations. The gain comes from additional rescues exceeding additional harmful substitutions; Qwen's actual harm also increases. We report these components separately.

We also tested label-budget dependence in a preceding frozen Qwen study on NQ and TriviaQA, with 10,000 held-out queries per task, seven budgets, and 20 label-sampling runs. At 20% deferral and 2,000 labels, Stage 2's increments over Support were +1.208 points on NQ [0.677, 1.738] and +1.021 on TriviaQA [0.384, 1.657]. Its 1,000-label intervals were positive; budgets from 25 to 500 were inconclusive under that study's simultaneous analysis.

With sufficient labels, Stage 2 improves fixed-budget accuracy in these new evaluations, including the cross-family cascade. The routers are retrained for each pair. Since model size and accuracy also differ, the larger Mistral increment cannot be attributed specifically to model family. These additional evaluations are reported separately from the submitted tables.

Comment 2/4: Support, simple baselines, and the role of the theory

In the submitted evaluation, Support's full routing AUC, the area under accuracy versus deferral rate, exceeds random routing by 0.02846; entropy exceeds random by 0.01483. At fixed 10% and 20% deferral, Support improves accuracy over entropy by 0.797 and 1.148 percentage points. In the submitted retrospective matched-accuracy frontier comparison, Support reaches the accuracy of entropy's 20% policy with 21.5% fewer large-model calls. The latter matches thresholds on the test frontier; it is distinct from evaluating a threshold calibrated before deployment.

To check whether Support's advantage over simple scores extends to the new Natural Questions evaluation, we included entropy, maximum probability, and a fixed random policy at the same 20% call budget. For maximum probability, lower confidence gives higher deferral priority. The Qwen3.5-9B to Qwen3.5-27B and Mistral-7B-Instruct-v0.3 to Qwen3.5-27B pairs give the following exploratory results, per 1,000 total queries:

PairPolicyNet additional correct answersHarmful substitutions
Qwen to QwenSupport13.30.5
Qwen to QwenEntropy15.62.5
Qwen to QwenMaximum probability13.93.0
Qwen to QwenRandom14.55.9
Mistral to QwenSupport25.10.2
Mistral to QwenEntropy20.81.9
Mistral to QwenMaximum probability22.61.5
Mistral to QwenRandom29.64.9

Support makes fewer harmful substitutions in these comparisons, but does not consistently achieve the highest net gain. At 2,000 labels, Stage 2 raises Support's net gain to 30.670 and 63.285 per 1,000 queries, respectively.

The clairvoyant reference answers a different question. It knows each query's realized grade difference, H=Yℓ−YsH=Y_{\ell}-Y_s, and selects the highest-gain queries subject to the same call budget. With an exact budget, it may select zero-gain or harmful cases after higher-gain cases are exhausted. It measures the remaining opportunity under the budget using unavailable outcomes; it is not unrestricted selection of the better answer on every query or a deployable comparator.

The theoretical question is what paired outputs can identify before correctness labels are collected. Let zz denote router-visible features. Under a shared answer-factored grader, agreement implies H=0H=0. Hence the expected gain g(z)g(z) equals disagreement probability δ(z)\delta(z) times the expected gain conditional on disagreement, m(z)m(z): g(z)=δ(z)m(z)g(z)=\delta(z)m(z). Paired outputs identify δ\delta. Under canonical exact-match grading and unrestricted conditional gold-answer laws, however, the same paired-output distribution is compatible with every gain in [−δ(z),δ(z)][-\delta(z),\delta(z)]. Opposite endpoint laws can require opposite routing decisions despite identical unlabeled observations. The regret result quantifies the consequence when their optimal rankings differ.

This identifies the limit of adding more unlabeled pairs and the information that outcome labeling must supply. It motivates learning disagreement from the paired pool, retaining agreement's known zero-gain information, and labeling disagreements to estimate the signed margin. Recovery from selective labels requires outcome-independent selection conditional on the routing features and disagreement, with positive labeling probability throughout the disagreement support. These are population identification results; the fixed-budget experiments test how well the resulting estimators use that information with finite data.

Comment 3/4: Preparation cost, online inference, and amortization

Table 1 compares serving-time routing quality across call budgets. To answer the concern about offline paired generation, we measured preparation and online inference separately. The new accounting includes paired generation and feature extraction, representation and Support selection, supervised-head tuning and refitting, and calibration.

For one deployment, the recorded preparation components are:

ComponentQwen3.5-9B to Qwen3.5-27BMistral-7B to Qwen3.5-27B
Paired-pool generation and feature extraction137.37 min149.38 min
Representation and Support selection2.53 min2.55 min
Calibration generation and processing6.56 min7.66 min
Recorded preparation subtotal2.44 h2.66 h

We compare Support and Stage 2 with a direct two-head control that fits a common correctness component and a large-minus-small grade-difference component, using the latter for routing. All three share the selected representation and paired pool in this comparison, so their preparation subtotals are nearly identical. The linear supervised-head search and refit adds less than 0.2 seconds per method. Stage 2 and the two-head control each receive 2,000 outcome labels from benchmark gold; Support receives none. Label acquisition remains a separate resource cost. Other direct estimators need not share this preparation dependency.

To connect serving cost to quality on the same workload, we replayed 512 common queries on one NVIDIA GB10 at batch size 1, using one prespecified 2,000-label fit and three timing repetitions. Thresholds were set on 1,000 calibration queries without gold labels. Times below are means per query; rescue/harm/net counts are totals over the 512 queries, using the actual generated outputs.

PairPolicySeconds/queryRescueHarmNet
Qwen to QwenSupport1.189716
Qwen to QwenStage 21.0041239
Qwen to QwenTwo-head0.99916214
Qwen to QwenEntropy0.89814113
Mistral to QwenSupport1.57018018
Mistral to QwenStage 21.43232032
Mistral to QwenTwo-head1.41928028
Mistral to QwenEntropy1.36115015

Stage 2's measured mean time is 15.6% lower than Support's for Qwen and 8.8% lower for Mistral, with 3 and 14 more net correct answers. Entropy is faster than Stage 2 in both pairs; it gives 4 more net correct answers on Qwen and 17 fewer on Mistral. Thus this replay does not establish an equal-quality cost advantage over entropy. Timing repetitions leave the quality sample at 512, and zero observed harm does not establish zero risk.

Representation transformation and routing take about 2.2 milliseconds. Shorter large-model generations among Stage 2's selected queries account for much of its time reduction relative to Support; calibrated call rates also differ. Stage 2 and the two-head control have close measured times.

Using recorded preparation OO and mean service time cc, the compute-time scenario for NN future queries is C(N)=O+NcC(N)=O+Nc. At 10,000 queries, the totals are:

PairSupportStage 2Two-head control
Qwen to Qwen5.74 h5.23 h5.21 h
Mistral to Qwen7.02 h6.64 h6.60 h

Always using the large model takes 4.70 hours in this accounting and has higher measured replay accuracy. Preparation has not yet amortized at this workload. Cold starts for the model pair add about 5.6 minutes. Data acquisition, some preprocessing, and annotation labor remain unmeasured, so the totals are recorded compute subtotals. Entropy needs no paired training pool; its standalone calibration preparation was not separately measured.

The submitted analysis projects 8.85% lower serving-token cost for Support at entropy's original-mixture 20% accuracy anchor. Its paired-pool token calculation requires roughly 262,000 to 422,000 future queries to break even there; two equal-weight cases have no finite positive break-even. These are retrospective test-frontier projections under a token proxy, distinct from the calibrated local time measurements above. We will report both cost models with their preparation, labeling, and traffic assumptions.

Comment 4/4: Seven splits, clipping, and clearer presentation

The original evaluation used seven partitions; the historical reason for that count cannot be established. We will describe these overlapping partitions and label fits as a sensitivity analysis. The new studies use the prespecified resampling procedures described above.

We will remove the redundant clipping notation. A raw margin estimate will be distinguished from its clipped value, m^(z)=clip⁡(m~(z),−1,1)\widehat m(z)=\operatorname{clip}(\widetilde m(z),-1,1), after which the routing score is simply g^(z)=δ^(z)m^(z)\widehat g(z)=\widehat\delta(z)\widehat m(z).

Figure 1 will be introduced earlier and will include a numerical routing example. Suppose queries A and B have (disagreement probability, conditional margin) equal to (0.9, -0.2) and (0.4, 0.8). Support ranks A first, while their expected gains of -0.18 and +0.32 make the product rank B first.

For Figures 2 and 3, we will show router-minus-baseline differences at fixed call budgets, with intervals and distinct line styles. The main tables will report the operating-point increments alongside integrated performance, including the task-specific negative result discussed above. We will reduce numerical precision in Tables 1 and 2, simplify the abstract, and organize Sections 5.3-5.5 around the question being tested and its result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.