acceptodds
Under review as a conference paper at ICLR 2027

Coverage Is Not Policy Value: An Audit of Test-Time Compute Oracles

Abstract

Test-time-compute studies often report an oracle that selects whichever stochastic action eventually succeeds on each problem. This measures retrospective coverage, which need not equal the value of choosing an action before its outcome is known. We distinguish retrospective coverage, the expected-action oracle (the best population action mean), finite-replication split-selected value, and held-out policy value. We show that coverage can grow with the number of exchangeable actions even when no decision-time policy improves over a fixed action. Our evaluation uses 771 model-problem states from three open models on MATH-500 and GPQA-Diamond, with five prompt actions and 16 outcomes per stochastic action. Under empirical product coupling, retrospective coverage exceeds split-selected value by 4.41 percentage points on MATH (95% problem-bootstrap interval [3.63, 5.26]) and 11.73 points on GPQA ([10.34, 13.17]); all six model-benchmark cells have positive gaps. Although split selection improves over a fixed prompt action, four router families evaluated on held-out problems show no statistically supported gain in either the prompt-action bank or a second bank that treats models as actions. Oracle headroom should therefore be reported with its outcome coupling and evaluated separately from decision-time policy value.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time compute, large language model reasoning, oracle evaluation, stochastic action selection, policy value, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7VgvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an empirical study on test-time compute in LLMs. Specifically, this paper focuses on the correlation between retrospective coverage and deployable policies. The major finding is that retrospective coverage can hardly be a strong indicator for determining action policies, although there are some correlations.

Critical strengths
  • This paper targets an interesting topic.
  • This paper does provide some interesting finding that may serve as a starting point for further investigation.
Critical weaknesses
  • This paper does not provide any conclusive results. Although I think the major finding of this paper interesting, it is unclear whether this finding can be used to improve test-time compute in LLMs.
  • The benefit of test-time compute for LLMs in general is typically limited as existing empirical results have demonstrated. Most exploration of test-time compute focuses on domains like maths and coding. However, this paper does not consider any domain-specific characteristics in test-time compute, although a mathematical benchmark is used in the study. In my opinion, this decision may make the findings of this paper less interesting and less useful.
  • The scope of the reported study is limited. It is unclear whether the findings of this paper are generalizable for cutting-the-edge LLMs.
  • Somewhere in this paper, the presentation seems to be difficult to follow. In particular, I do not quite understand what Section 4 aim to present.
Questions for authors

Q1: What are the takeaway messages from this paper? How can this paper be used to improve the practice? Q2: How is Section 4 related to the empirical study reported in this paper? Q3: If domain specialty is further considered, how would this impact the findings of this study?

5 comments

Thank you for asking us to make the practical purpose and empirical scope clearer. The paper is an evaluation audit, not a new algorithm for improving reasoning accuracy. Its practical contribution is to identify which claim an oracle measurement can support and which additional evaluation is necessary before choosing a deployed policy.

Q1. Takeaways and practical use

There are three different decisions.

  1. Does a successful candidate exist? Retrospective coverage answers this question, provided the action set and the coupling of stochastic outcomes are specified. It is relevant to generating several candidates, but does not by itself tell us which action to execute.
  2. Which single stochastic action has the highest mean utility? This requires expected action values. With limited observations, choosing the largest empirical mean adds selection uncertainty. Independent selection/evaluation panels measure the value of that finite-panel rule without reusing its selection outcomes. They do not identify the population best-action value exactly.
  3. Can a decision-time policy obtain an improvement? This requires a policy that sees only information available before the action, tested on held-out problems against a fixed action selected on disjoint fitting data. Routing calls and token costs must also be counted.

For a concrete example from the submitted model-action experiment on GPQA, the fixed model scores 46.31%, product-coupled retrospective coverage is 68.53%, and eight-outcome split-selected value is 60.06%. These correspond to roughly 22.22 and 13.75 points of two different diagnostic headrooms. Neither number is a measured deployable routing gain. The practitioner should therefore not select a routing system from either headroom alone; its held-out policy improvement and cost are separate measurements.

We have expanded the empirical check: GPQA fitting now contains 316 questions/model instead of 66, and evaluation has 64 outcomes per stochastic action on the same 132 problems. Expanded structured prompt routing gains 0.21 points, with nominal interval [-1.27, 1.70], relative to its fit-selected fixed action. Homogeneous MATH fitting, however, gives a small positive structured gain of 1.08 points [0.25, 1.97], which does not survive exploratory within-benchmark multiplicity adjustment. Thus the conclusion is not that routing never helps; it is that coverage and policy gains must be measured separately.

This reporting workflow makes the evidence for deployment decisions explicit. We do not claim to have demonstrated a new accuracy or efficiency improvement for test-time compute.

Q2. How Section 4 connects to the experiments

Section 4 supplies the interpretation of the measured quantities. The Introduction and the opening of Section 4 clarify its role as statistical context for the empirical targets, not a standalone theoretical contribution:

Statistical consequenceExperimental consequence
Equal-mean actions can have high coverage without a decision-time advantageA large coverage value alone is not evidence of a useful router
Marginal success rates do not determine joint coverageDeclare product coupling and report coupling sensitivity
Maximizing noisy empirical means is optimisticDo not label the plug-in maximum as population EAO
Independent holdout evaluates the selected ruleUse disjoint outcome panels and state their selection depth
No unrestricted finite-sample unbiased EAO estimatorUse explicit uncertainty/bounds rather than equating EAO with an observable maximum

The last point does not rule out useful estimation. We now provide panel-size curves and model-based component estimates. On GPQA with 64 outcomes/action, the coverage-minus-split-selected gap remains 11.60 points [10.22, 13.01] at selection size thirty-two. This is a more resolved finite-panel measurement, not a claim that the population component has been identified without assumptions.

For transparency, the revision also corrects the fitting count (748 total states, including 500 legacy MATH states, rather than 248 total), the model-router replication description (eight on MATH and sixteen on GPQA), and legacy MATH history-feature and stop tie inconsistencies. Figure 3 separates tie breadth from changes in maximizing sets; we withdraw the unsupported link between independent-choice disagreement and utility-router failure. Appendices C and F clarify scoring after truncated GPQA reasoning. Marking all truncated fresh continuations incorrect in a separate sensitivity, with prefix/stop utilities unchanged and no attempts omitted, gives an extended coverage-minus-V32 gap of 11.69 points [10.31, 13.13]. The primary labels and original coverage results are retained.

Q3. Domain characteristics and generalization

The distinction between candidate coverage and choosing an action applies beyond a particular domain. The quantitative gap does not have to be domain-invariant. Appendix F.3 reports metadata-defined groups, averaging models within each question before computing problem-level intervals.

On MATH, the original sixteen-outcome, eight-selection gap is 2.91 points for level-one problems (11 questions, interval [0.55, 6.42]) and 5.50 for level-five problems (32 questions, [3.91, 7.21]). This is descriptive evidence of different magnitudes, not a causal difficulty effect or a statistical test of their difference. The example is not a subgroup interaction test, and the small denominators remain explicit.

On the extended GPQA bank, the eight-selection gaps are 10.29 points for biology (12 questions, [7.12, 13.81]), 13.77 for chemistry (63, [11.48, 16.06]) and 13.25 for physics (57, [11.32, 15.21]). Biology has a particularly small denominator. These results support examining domain composition rather than presenting one pooled number as universal; they do not establish a significant domain interaction.

We also retain the near-ceiling qualification for the stronger MATH models. Neither these strata nor the additional Qwen3-8B routing test establish quantitative generalization to frontier LLMs. Section 9 distinguishes the general measurement definitions from the empirical findings for these three open models, these action definitions and these evaluation populations.

Thanks for the response. I think the response helps clarify some issues. However, I still think this paper may not be mature enough for publication in the current form, although I also think some findings are interesting.

Reviewer_LqwCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper distinguishes retrospective coverage—the expected best realized outcome across stochastic actions—from the expected-action oracle, finite-replication split-selected value, and held-out policy value. It establishes elementary relationships between these quantities and proposes independent selection/evaluation panels to avoid reusing outcomes when evaluating a selected action.

The empirical study evaluates five prompt actions across three models on MATH-500 and GPQA-Diamond, using 16 outcomes per stochastic action. Product-coupled coverage exceeds the value selected using eight outcomes per action by 4.41 percentage points on MATH and 11.73 points on GPQA. Four simple router families show no statistically supported improvement over fit-selected fixed actions. A second experiment treats models as actions.

Critical strengths
  1. The paper clearly separates quantities that answer different evaluation questions. Explicitly distinguishing outcome-dependent coverage, optimal expected action value, finite-panel selection, and deployable policy performance is useful. The treatment of outcome coupling also makes an often implicit assumption visible. These distinctions could improve the interpretation and reporting of oracle results.

  2. The replicated evaluation supports more careful comparisons than single-outcome action banks. Independent selection and evaluation panels, problem-level bootstrap intervals, retention of failed generations in the denominator, and comparisons against independently selected fixed actions are valuable methodological choices. The coupling and tie-handling sensitivity analyses further strengthen the descriptive findings.

Critical weaknesses

The headline differences combine coverage beyond the best expected action with regret from selecting an action using only eight observations. Likewise, the reported fraction of headroom “not retained” includes both effects. These results establish the performance of a particular finite-panel selection procedure, but do not establish how much coverage actually exceeds the population expected-action oracle in the evaluated systems.

The authors acknowledge this limitation, so my concern is insufficient empirical resolution rather than an incorrectly stated identity. Nevertheless, it limits the paper’s central contribution: the formal results largely follow from basic probability, convexity, concentration, and independent holdout evaluation, making the quantitative audit particularly important. Proposition 5 rules out an exactly unbiased finite-sample estimator; it does not rule out informative bounds, uncertainty-aware estimation, or replication analyses that constrain the two components. Without such evidence, the empirical magnitude of the central population distinction remains unresolved.

The routing experiments and label-stability diagnostic provide limited evidence about why the apparent headroom is not recovered. Three of the four routers are ridge-based variants, and model routing uses only 50 MATH or 66 GPQA fitting problems. Their failure to improve can therefore reflect limited training data or unsuitable predictors. The paper appropriately avoids an impossibility claim, but the experiments still provide relatively little understanding beyond the failure of these particular configurations.

More specifically, the agreement statistic in Appendix D is the inner product of two tied-maximizer distributions. If both panels identify exactly the same five-way tie, this statistic is only (1/5=20%), despite perfect agreement between the distributions. Consequently, the reported 26.9% agreement on MATH does not by itself demonstrate substantial instability in useful action choices; it mixes disagreement with the breadth of ties. The paper acknowledges ceiling effects, but does not adequately separate these explanations. Moreover, Appendix E states that the prompt routers already learn smoothed success-rate targets rather than hard argmax labels. The discussion suggesting that hard-label instability helps explain their failure is therefore not directly supported by the implemented training procedure.

Questions for authors
  1. Evidence needed to interpret the main result: Can you provide informative bounds or uncertainty-aware estimates for the two components of Equation (7), or show how the empirical coverage-minus-selected-value gap changes with selection-panel size? This would help determine whether the headline differences predominantly reflect coverage beyond EAO or finite-panel selection regret.

  2. Clarification of fitting data: Section 6 reports 250 MATH fitting problems for each of two models, 50 for the third, and 66 GPQA fitting problems per model. This appears to imply (250+250+50+3\times66=748) fitting model-problem states. The same section instead reports 248 fitting states and 16,368 generated outcomes. Please reconcile these numbers and specify the exact fitting data used by each fixed-action baseline and router.

  3. Clarification of the stability interpretation: How much of the low agreement in Figure 3 reflects changes in the maximizing action set, and how much reflects identical or substantially overlapping ties? Does an analysis based on set agreement or utility loss support the same conclusion? How does the proposed hard-label explanation apply to routers trained on smoothed utility targets?

  4. Evidence needed to interpret router failure: Do you have learning curves, repeated training splits, or a positive-control experiment showing that these router configurations can recover a known decision-time signal at the available fitting-set sizes? Such evidence would help distinguish limitations of the tested predictors from difficulty inherent in the action banks.

5 comments

Thank you for distinguishing the usefulness of the measurement framework from the unresolved empirical questions. We agree that the submitted coverage-minus-selected-value gap does not identify its population components, and that the tie statistic and utility-trained routers require different interpretations. We address your four questions below. The associated evidence is in Section 7 (pp. 7–8), Figure 3 (p. 8) and Appendix F (pp. 17–21).

Q1. Population components and selection-panel size

We extended the same 132 GPQA evaluation problems for every model from 16 to 64 outcomes per stochastic action. The original prefixes, action definitions and checkpoint revisions were retained. For each observed bank, we integrate the finite-panel selection rule exactly using independent hypergeometric panels across actions and uniform tie weights; evaluation uses outcomes outside each selection panel. Equal-model averages are bootstrapped by question, keeping the three models on a question together.

Outcomes/actionSelection outcomes/actionProduct coverage minus split-selected value, points95% problem-bootstrap interval
16811.73[10.34, 13.14]
64813.23[11.83, 14.65]
641612.17[10.79, 13.58]
643211.60[10.22, 13.01]
644811.36[9.99, 12.77]
646311.08[9.74, 12.46]

The 16-outcome row is an exact finite-bank re-evaluation. Its interval [10.34, 13.14] differs slightly from the original Monte Carlo interval [10.34, 13.17] retained in Table 3; these are distinct computations, not two intervals for the same computation.

Increasing selection-panel size within the 64-outcome bank reduces the gap, but does not remove it. The 63-outcome selection row has only one evaluation outcome per action and is a sensitivity, not a precise EAO estimate. The first two rows use different empirical marginal means, so their difference should not be interpreted as the effect of selection-panel size alone. For MATH, the original 16-outcome bank gives gaps of 6.42, 5.76, 5.01, 4.41, 4.13 and 3.92 points at selection sizes 1, 2, 4, 8, 12 and 15.

To examine Equation (7), we also compute both components coherently from joint draws of the action means. We use independent Beta(1/2,1/2) and Beta(1,1) priors and integrate population V8 exactly over its binomial selection panels. At 64 outcomes per GPQA action, posterior means are:

ModelCoverage component, Jeffreys / uniform, pointsSelection component, Jeffreys / uniform, points
Qwen3-8B7.60 / 8.202.37 / 2.55
R1-Distill-7B7.87 / 8.872.50 / 2.70
Phi-3.5-mini15.86 / 16.523.44 / 3.52

For example, Qwen's Jeffreys posterior gives 7.60 [7.27, 7.94] points for coverage minus EAO and 2.37 [2.23, 2.50] for EAO minus V8. These are 95% credible intervals conditional on the evaluated states, not problem-population confidence intervals. Both priors attribute more of the GPQA gap to the coverage component than to eight-outcome selection regret. That conclusion is model-based: conservative simultaneous Clopper–Pearson propagation remains broad. Its conditional 95% coverage-component bounds are [0.16, 30.41], [0.32, 41.30] and [0.28, 48.30] points for Qwen, R1 and Phi, respectively, with coverage guaranteed within each model's evaluated state set. MATH remains more prior-sensitive at 16 outcomes. Appendix F.1 presents these analyses with their assumptions, rather than as a distribution-free decomposition; Proposition 5 is not used to dismiss uncertainty-aware estimation.

Q2. Fitting populations and corrections

Your count is correct: 748 model-problem states were used across prompt-router fitting. The 248-state total describes only the current audit bank and omitted the distinction from 500 legacy MATH fitting states.

Fitting sourceQuestions/modelModelsFresh outcomes/action
Legacy MATH250Qwen and R18
Current MATH50Phi16
Current GPQA66All three16

Thus 248 current fitting states generate 16,368 responses, and 771 evaluation states generate 50,886, totaling 67,254 current-bank responses. Legacy routing additionally consumes 17,000 responses: 500 states times two prefixes plus four sets of eight fresh outcomes. The 44,000 retained legacy source responses are a different count, because not all retained base samples enter these routing analyses. Fractional stop utilities are deterministic evaluations of the prefix, not additional generations.

The original model-router fitting intersections contain 50 MATH and 66 GPQA questions, with executed minimum replication depths 8 and 16, respectively. The appendix's description of eight for both was inaccurate. We reproduced the original scores at their executed depths and evaluated GPQA at eight as a separate sensitivity.

Our checks also found legacy MATH history-feature coordinates that differed from evaluation and a different stop tie convention. We rebuilt both from the saved prefixes and reran the affected routers. The correction-only MATH structured-router gain becomes +1.35 points, nominal interval [0.23, 2.65], with Holm p=0.108 across the four original prompt-router families under a paired sign-randomization test. Section 7 and Table 9 report this positive nominal result rather than an unqualified null statement.

To address generation-protocol differences as well, we regenerated all 250 fitting states for each of Qwen and R1 under the evaluation action protocol, including fresh prefixes and eight outcomes per stochastic action. The resulting homogeneous MATH structured gain is +1.08 points [0.25, 1.97]. This is a separate comparison because the fitting states and fit-selected comparator can change. These corrections do not modify the original evaluation action banks or their coverage gaps.

We also found that the appendix's truncation wording did not describe the executed scoring convention precisely: GPQA can receive a correct constrained option readout after truncated reasoning. Marking all truncated fresh continuations incorrect in a separate sensitivity gives an extended GPQA coverage-minus-V32 gap of 11.69 points [10.31, 13.13], versus 11.60 [10.22, 13.01] with the executed scorer. Prefix/stop utilities are unchanged, and no attempts are omitted. The scoring description is corrected without changing the primary labels.

Q3. Tie breadth, set changes and utility targets

You are right: independent-choice agreement is not agreement between the maximizing distributions. Two identical five-way ties give 20% independent-choice agreement but 100% set and distribution agreement. Our 8/8 complementary-panel analysis now separates these quantities:

DiagnosticMATHGPQA
Independent-choice match26.93%45.02%
Identical maximizing sets65.47%57.88%
Mean Jaccard overlap83.75%70.89%
One minus total-variation distance84.14%71.34%
Identical non-singleton maximizing sets58.89%30.65%

These are panel-averaged descriptive quantities on the original 16-outcome banks. They show that broad ties explain much of the low MATH independent-choice match. Figure 3 and its discussion no longer infer unstable maximizing distributions from the independent-choice statistic alone.

The prompt routers already fit smoothed utility targets. Hard-label instability therefore does not directly explain their results. The revision removes that causal interpretation. Utility-vector diagnostics are descriptions, not evidence that a hard-label mechanism caused the router behavior.

Q4. Expanded fitting, controls and learning curves

We added 250 GPQA Main questions outside all 198 Diamond questions, checking both record identities and normalized question text. Together with the original 66 fitting questions, this gives 316 fitting questions per model; the 132 evaluation questions are unchanged. New questions have eight outcomes per stochastic action, while the original fitting questions have sixteen. Smoothed targets use each question's actual count, without padding or treating duplicated outcomes as independent observations. The Main-to-Diamond distribution shift remains a limitation.

On the extended 64-outcome evaluation bank, expanded-fitting pooled gains are +0.21 points [-1.27, 1.70] for structured ridge, -1.39 [-3.69, 0.76] for signed-hash ridge and -0.52 [-1.36, 0.31] for a tuned structured-feature tree ensemble. A separately frozen Qwen3-8B router using the original aggregate fitting means gives -0.15 [-1.02, 0.66] against its own original fixed comparator. None survives within-benchmark Holm correction; these are exploratory tests, not an equivalence result. The LLM comparison has a different fitting-information regime and is not a controlled ranking against the expanded trained routers.

Ten nested-subset learning-curve seeds at 32/66/100/200/316 questions do not show a uniform trend. For R1, mean structured-router gains increase from -0.74 points at 32 to +1.55 at 316; Qwen is non-monotonic and ends at -1.10; Phi ends at +0.18. These curves allow data insufficiency and predictor limitations, rather than ruling them out. They also mix fitting size with source composition. Synthetic controls confirm that the nonlinear learner recovers some known signal at the available sample sizes, but recovery of small signals is weak. Appendix F.3 uses these results to bound the interpretation, not argue that real routing signal is absent.

Reviewer_S1u3OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that retrospective oracle coverage overestimates the benefit of test-time action selection. It distinguishes five things: fixed-action performance, learned-policy value, expected-action oracle (EAO), split-selected value VmV_m, and retrospective coverage oracle (RCO).

The theoretical results establish standard relationships between these quantities, including coupling dependence, selection bias, and unbiased split-panel evaluation, whereas the experiments on MATH-500 and GPQA-Diamond show that RCO exceeds VmV_m by 4.41 and 11.73 points, respectively.

Critical strengths
  1. Model-action experiment is arguably the strongest result presented here. On GPQA, split-selected value and RCO far exceed fixed-action baseline, yet all four learned routers perform worse. This illustrates how a large oracle headroom can coexist with negative realized routing gains. Though, the tiny fitting sets substantially weaken the conclusion.

  2. Best-action labels from disjoint outcome agree only 26.9% on MATH and 45.0% on GPQA. Moreover, 51.5% of GPQA states lack an action with posterior optimality probability above 0.8. These results do show an instability of argmax-based supervision and motivate using utility vectors or posterior probs instead. Even though this is a useful practical insight for training routers, it is not a fundamentally new method.

Critical weaknesses
  1. The distinction between retrospective coverage and achievable policy value is already established by pass@k versus pass@1 (Chen et al., 2021; Brown et al., 2024) and prior work on oracle evaluation (Stroebl et al., 2026; Hu, 2026).

    All five propositions is just standard statistics, i.e., joint-failure probability, elementary coupling bounds, maximum-of-noisy-estimates bias, sample-splitting unbiasedness, and finite-sample non-estimability of non-polynomial Bernoulli functionals. Renaming these quantities does not constitute a theoretical contribution.

    The headline result, RCO > VmV_m, follows via construction. Its direction is guaranteed, while its magnitude depends on the chosen action bank. The paper offers little beyond verifying expected relationships on a small benchmark.

  2. Claiming that oracle headroom does not translate into a learnable policy value rests on four basic ridge/kNN routers trained on only 50–66 problems in key comparisons. Best-action labels agree across outcome panels only 26.9% on MATH and 45.0% on GPQA, making these small-data routing exps. uninformative.

    Two models already achieve roughly 94–96% accuracy on MATH, leaving little room for improvement. Failure of weak routers in this near-ceiling, low-data setting does not establish that better policies cannot exploit the headroom.

    Several key analyses were also specified after the outcome banks were collected. Overall, to me, empirical evidence supports only a narrow negative result about simple routers, far from a meaningful general conclusion about test-time compute.

References

  • Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).
  • Brown, Bradley, et al. "Large language monkeys: Scaling inference compute with repeated sampling." arXiv preprint arXiv:2407.21787 (2024).
  • Stroebl, Benedikt, Sayash Kapoor, and Arvind Narayanan. "The limits of inference scaling through resampling." International Conference on Learning Representations. Vol. 2026. 2026.
  • Snell, Charlie, et al. "Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning." International Conference on Learning Representations. Vol. 2025. 2025.
  • Yang, Chih-Hsuan, et al. "LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks." arXiv preprint arXiv:2608.14927 (2026).
  • Hu, Jie. "Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration." arXiv preprint arXiv:2607.17531 (2026).
Questions for authors
  1. What specific result is new relative to Snell et al. (2025), Stroebl et al. (2026), Yang et al. (2026), and Hu (2026)? The propositions and central distinctions appear to prety much follow directly from prior work.

  2. Since RCO >= EAO >= VmV_m by construction, what hypothesis does the observed ordering actually test?

  3. Why not evaluate LLM-based routing, model-confidence signals, or substantially larger routing datasets before drawing conclusions about learnable policy value?

  4. What would be the smallest routing gain the experiment setup could reliably detect? Perhaps learning curves over training-set size to distinguish insufficient data from genuinely unlearnable action selection could be informative.

5 comments

Thank you for identifying the distinction between a useful evaluation audit and a general negative claim about learnable routing. We agree that standard ordering relationships and unsuccessful small-data predictors are not sufficient for the latter. The revision addresses empirical resolution and predictor scope while retaining the original title, three research questions, propositions and action banks. Section 7 summarizes the additional evidence; Appendix F provides details.

Q1. Contribution relative to the closest work

We do not claim that the coverage-versus-achievable-value distinction is new, or that the five propositions introduce new statistical principles. Pass@k already separates candidate existence from single-response accuracy. The Introduction and Section 4 present the propositions as measurement consequences that specify the experimental targets, rather than as headline theoretical novelty.

The relevant overlap is substantial. Snell et al. study adaptive allocation of inference computation, including dependence on problem difficulty. Stroebl et al. establish limitations of resampling with imperfect verifiers. Yang et al. directly study cost-aware protocol routing with trained and frozen routers; the cited study also has a Findings of EMNLP 2026 proceedings record. Hu diagnoses fixed-pool selection through recoverable mass, verification signals and harm. We do not attribute to our paper the general observation that oracle headroom may fail to become deployed gains.

Our narrower contribution is a replicated audit of the sampling axis within each problem and stochastic action. It distinguishes the realized-outcome coupling, the best conditional action mean, selection from a finite labeled panel and a held-out policy. The empirical question is how much the measured gap changes with replication and selection depth, how its population components can be constrained, and which conclusions survive tie-aware and policy evaluations. We now provide a 64-outcome GPQA replication analysis, coherent component estimates under explicit priors, corrected and homogeneous fitting comparisons, and expanded-data routing diagnostics. The value of the paper rests on this quantitative audit and its reporting consequences, not on renaming familiar quantities or asserting a universal failure of routing.

Q2. What does an ordering guaranteed by construction test?

The direction is not a hypothesis test. At the population level, coverage is at least EAO and EAO is at least the value of the finite-panel rule. The experiments quantify the magnitude and sensitivity for a declared action bank; they do not provide evidence for the inequality itself.

In particular, the original 11.73-point GPQA gap combines a coverage component with selection regret. We extended all three models on the same 132 evaluation questions from 16 to 64 outcomes per stochastic action while preserving prefixes and prompts. Within that bank, coverage minus split-selected value decreases from 13.23 points with eight selection outcomes/action to 12.17 with sixteen and 11.60 with thirty-two (95% problem-bootstrap interval [10.22, 13.01] for the latter). This shows that the finite-panel gap persists at a larger selection depth, not that it equals coverage minus EAO.

For population components, coherent Beta(1/2,1/2) posterior means of coverage minus EAO are 7.60, 7.87 and 15.86 points for Qwen, R1 and Phi; corresponding EAO-minus-V8 means are 2.37, 2.50 and 3.44. Uniform-prior sensitivities retain the same qualitative component ordering. These estimates condition on the evaluated states and on the Bernoulli mean model; conservative simultaneous confidence bounds remain wide. We therefore distinguish empirical finite-bank curves from model-based population resolution and do not make a distribution-free dominance claim.

Q3. Stronger predictors and more fitting data

We have added nonlinear routing, a frozen reasoning-based Qwen3-8B router, larger fitting populations and a homogeneous MATH fitting evaluation. All policies retain disjoint fitting/evaluation questions and only decision-time features.

Section 6 and Appendix E clarify that prompt-router fitting used 748 model-problem states: 248 from the current bank plus 500 legacy MATH states. The model-router fitting replication depths were eight on MATH and sixteen on GPQA, rather than eight for both. These accounting and protocol corrections leave the original evaluation action banks unchanged.

First, we found and corrected history-feature and stop tie inconsistencies in the legacy MATH fitting loader. Correction-only structured routing gives +1.35 points, with nominal interval [0.23, 2.65]; this is important evidence against an unqualified no-gain statement. To remove the legacy/current generation-protocol difference, we then regenerated the two 250-question MATH fitting banks under the evaluation protocol. Homogeneous structured ridge gives +1.08 points [0.25, 1.97], and the structured tree ensemble gives +1.12 [0.08, 2.18]. Their exploratory within-benchmark Holm-adjusted sign-randomization p-values are 0.086 and 0.213 across the six displayed prompt-policy tests. Section 7 and Table 12 report both the nominal positive results and the multiplicity qualification. They do not imply zero signal.

Second, we added 250 GPQA Main questions outside the entire Diamond set, increasing fitting from 66 to 316 questions per model. Deduplication checks record identity and normalized question identity. We retain the original 132 Diamond evaluation questions, extended to 64 outcomes/action. Per-state smoothing uses the actual eight or sixteen fitting observations, not a padded common replication depth. Main-to-Diamond transfer remains explicit.

Expanded-fitting GPQA pooled gains against fit-selected fixed actions are +0.21 points [-1.27, 1.70] for structured ridge, -1.39 [-3.69, 0.76] for signed-hash ridge and -0.52 [-1.36, 0.31] for structured forests. In problem-only model routing, the 316-question fitting intersection gives -4.11 [-8.84, 0.64] for structured ridge and -8.21 [-13.72, -2.96] for signed-hash ridge, while the forest returns the fixed choice. This is evidence about these expanded configurations, not a claim about all policies.

Third, the frozen LLM router sees the problem, aggregate action utilities from the original disjoint fitting data, and, for prompt routing only, unverified prefix responses. It never receives the problem's reference answer or labeled action outcomes. Model routing is problem-only apart from the global fitting summaries. It gives pooled prompt gains of -0.17 points [-0.70, 0.22] on MATH and -0.15 [-1.02, 0.66] on GPQA against its own original fixed comparators. The latter are not the expanded trained-router comparators. Routing costs two additional model calls: up to 2,048 reasoning tokens followed by a constrained one-token route readout. There are no invalid-route fallbacks, but reasoning reaches the cap on 99/771 prompt states and 110/257 model routes, including 76/132 GPQA model routes. This budgeted small-model test is not evidence about unconstrained or frontier LLM routers. We have not measured fresh confidence probes or token log-probability features and will not claim otherwise.

Q4. Detectable gain, learning curves and controls

We agree that nonsignificance alone cannot separate small signal, limited fitting data and unsuitable features. There is no single minimum detectable effect for every router and benchmark. We provide three complementary checks.

  1. Conditional design power. On the original sixteen-outcome banks, we simulate frozen-policy comparisons at feasible gains on a 0, 0.5, 1, 2, 3 and 5-point grid, preserving shared outcomes when policy and comparator select the same action. Many Qwen/R1 GPQA comparisons reach 80% nominal paired-t-test power only at the five-point grid value; some near-fixed comparisons cannot support larger alternatives at their observed action frequencies. This is a model-dependent sensitivity for frozen choices, not a continuous MDE, not the power of the expanded 64-outcome analysis, and not power for the full training algorithm. It does not justify equivalence at sub-point effects.

  2. Training-size curves. Ten nested-subset seeds at 32/66/100/200/316 GPQA fitting questions give nonuniform trends. R1's mean structured gain changes from -0.74 to +1.55 points, Qwen from -2.00 to -1.10 with an intermediate near-zero result, and Phi from +0.72 to +0.18. The curves do not establish saturation or unlearnability. They reuse the same evaluation questions, and fitting size and Main/Diamond composition are not independently randomized.

  3. Positive controls. A nonlinear synthetic decision-time signal with eight Bernoulli observations/action is partially recovered by the forest learner: a ten-point signal yields about 2.22 and 2.75 points of expected-utility gain at fitting sizes fifty and sixty-six, averaged over ten fitting seeds. Recovery is weaker for smaller signals. This checks that the learner can recover some signal, while exposing its limited sensitivity; it does not establish sufficient capability on the real action bank.

The revision also separates tie breadth from maximizing-set changes and removes the unsupported causal link from hard-label instability to utility-trained router performance. Appendices C and F clarify that the executed GPQA scorer can assign a correct constrained option readout after truncated reasoning. A sensitivity marking truncated fresh continuations incorrect, with prefix/stop utilities unchanged and no attempts omitted, gives an extended coverage-minus-V32 gap of 11.69 points [10.31, 13.13], versus 11.60 [10.22, 13.01] under the executed scorer.

Sections 6 and 9 retain the near-ceiling MATH qualification and distinguish exploratory analyses from the generation design. The revision does not claim routing is intrinsically unlearnable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.