acceptodds
Under review as a conference paper at ICLR 2027

Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?

Abstract

Policies trained from the same base model can appear to solve different problems. An oracle that chooses the best policy for each problem may therefore appear much stronger than any single policy. Selecting the largest estimated success rate also selects favorable sampling errors. We study this effect through reliability coverage, the fraction of problems whose success probability reaches a chosen threshold. Our first test redistributes stored correctness outcomes across policies within each problem. A second also preserves each policy's total successes, accounting for overall quality differences under a specified statistical model. For five training seeds of a seven-billion-parameter vision-language model, redistribution reproduces 0.096 of an estimated 0.113 oracle gap at threshold 0.10. Neither test finds significant evidence at this threshold. Small advantages remain unresolved. Mixtures, routers, voting, and weight averaging show no detectable improvement over their corresponding single-policy baselines. Training policies on different datasets shows little detectable specialization under light post-training, and no router gain. A stronger recipe does create it, both tests detect it, and a router gain appears only at the high thresholds where the specialists separate. In a control with predictable specialization, a router recovers about half the oracle gap. Coverage bounds explain why even a genuine oracle advantage need not yield a deployment gain. The tests assess apparent specialization from stored responses before investment in routing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, model routing, model ensembles, stochastic evaluation, coverage

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XMnPOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether apparent oracle improvements across related language-model policies reflect reproducible specialization or finite-response sampling noise. It introduces a within-prompt re-deal test and a margin-preserving conditional test under an additive-logit model. Simple coverage identities distinguish an oracle opportunity from gains achievable by a mixture or router. Experiments examine seed variation, cross-dataset training, stronger training, multiple response budgets, and several deployment rules, with simulations and positive controls characterizing the tests.

Critical strengths
  1. The evaluation perspective is distinctive and valuable. Separating a noisy estimated oracle gap, a population specialization gap, and a deployable routing gain clarifies what policy-pool experiments can actually establish.

  2. The paper supports that perspective with a careful statistical investigation: independent selection/scoring, power simulations, misspecification checks, larger response budgets, and positive controls. These analyses provide substantial insight into when apparent specialization can be misleading. The explicit distinctions among the statistical quantities and the reported limitations help bound the interpretation of the results.

Critical weaknesses
  1. The paper could make its operational guidance more accessible. At k=16, Table 2 detects a population gap near 0.05 in only about half of the simulated alternatives; the paper also documents link-function sensitivity and limits of subtracting the null mean. The paper already provides useful analyses of these limits; a concise explanation of what a non-significant result permits practitioners to conclude would make the contribution easier to apply. Uncertainty in the estimated break-even routing requirement L/D would also help distinguish a statistical diagnostic from a practical routing decision.

  2. The light-versus-strong training comparison changes both the training configuration and the training-time verifier. Appendix G states that the primary light Geometry3K models used a checker with a degree-symbol issue, whereas stronger training used the corrected checker. Correct re-scoring of evaluations does not remove this training-time difference. A matched corrected-checker control would be informative; alternatively, explicitly describing the comparison as two recipes with different verifiers would appropriately limit its interpretation.

Questions for authors
  1. How should a practitioner interpret a non-significant result at k=16 given the measured power, and how does that guidance change at k=64?

  2. How uncertain is the estimated break-even requirement L/D, and does accounting for that uncertainty change the practical routing conclusions?

  3. Could you clarify whether a corrected-verifier light-training control is available? If not, explicitly noting both differences between the recipes would address the attribution concern.

Reviewer_vgN9OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The research question (put simply) addressed in this paper is: Are different AI models actually good at different things, or do they just get lucky on different attempts? To answer that question, the paper shufflles the models’ past right and wrong answers in two ways: one treats the models as equally capable on each question (ie, for any particular question, every model has the same chance of answering correctly), while the other allows some to be stronger overall. It also explains how choosing an unsuitable model hurt performance and what performance can still be guaranteed when a model is chosen at random for each answer. In their experiments, lightly trained models showed no clear routing benefit. Stronger training produced clearer specialization but routing gains remained limited. The main take away from the paper is: check that the advantage is real, then check whether you can predict which model to choose.

Critical strengths
  1. The paper separates several quantities that are easily mixed up. An estimated oracle advantage, a population specialization advantage, and an advantage recoverable by a router are not interchangeable. The Geometry 3K result make the estimation issue concrete. The explicit warning that substracting the null mean doesn't estimate the population gap is important. This distinction can improve how routing and ensemble opportunities are evaluated.
  2. The empirical investigation includes meaningful controls. The authors examine calibration, concentrated and diffuse alternatives, independent selection and scoring, larger response budgets, additional prompts and a smaller backbone. One of the controls shows that the framework can identify a setting in which routing genuinely helps. Voting comparisons and single-policy baselines further strengthen the analysis.
Critical weaknesses
  1. Non-rejection remains not very decisive for the proposed screening use. Table 2 reports only 50% power against a population oracle gap near five percentage points at the primary response budget.
  2. The margin preserving test controls the quality of only within a particular model. Its rejection establishes a departure from aditive log-odds, not a model-independent form of specialization. (Appendix C4 demonstrates the practical importance of this distinction)
Questions for authors
  1. On the primary Geometry 3k pool, how large could the true advantage of a perfect model picker still be? Could you please give an uncertainty range with its assumptions stated?
  2. Does stronger training produces different strengths even on the same dataset? You have two strongly trained seeds for each dataset. What happens when you run both tests on just the two Geometry3K seeds, and then on just the two Mathvista seeds? This would help clarify whether detectable differences also appear without changing the training dataset.
  3. What do the disagreement results say about the main pool? Table 1 shows how often the models disagree about which questions they cover, alongside the shuffled average, but does not report the p-values. Could you please report them?
Reviewer_41jUOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies studies whether seemingly different policies trained from the same base model solve different problems through specialisation or it this behaviour is attributable to finite-sampling noise. To this end, they leverage the concept of reliability coverage, where a problem is considered covered if a policy's probability of success exceeds a threshold τ\tau. It is shown that a plug-in oracle can overstate the benefit of selecting the best policy per prompt due to maximizing noisy estimates induces a "winner’s-curse" effect.

To verify this, the authors propose two null tests based on redistributing stored outcomes across policies: i) re-deal / plain null and ii) margin preserving null and derive conditions under which routing or mixing can outperform the best single policy.

Experiments with various Qwen2.5-VL policies trained under different seeds show that most of the observed oracle gap can be explained by sampling noise without statistically detectable gains from routing, mixing, voting, or weight averaging.

Critical strengths
  • The paper challenges a common assumption model weak model ensembles, that a large oracle gap necessarily reflects meaningful model specialization and therefore represents a deployment opportunity.
  • The paper is careful to establish a statistical diagnostic framework before interpreting an oracle gap as a routing opportunity.
  • The evaluation offers a tightly controlled setting to isolate the root-cause. The authors progressively relax the setting to introduces stronger sources of specialization. This progression give a causal structure in the narrative.
Critical weaknesses
  • The paper has strong presentation issues that prevent it from communicating the results clearly to the reader. A major issue is that the paper introduces too many closely related terms before establishing a simple narrative and properly motivating the problem. The writing itself also feels over-compressed and synthetic/generated at times.
  • The experiments span across one model, LoRA-based GRPO and two datasets. This makes it difficult to draw broader conclusions about model specialization or oracle-gap inflation across heterogeneous model pools.
  • The practical significance of the main seed-variation setting is limited. Routing among lightly post-trained variants of the same model that differ primarily by random seed is a useful controlled experiment, but is less representative of deployments in which candidate models differ deliberately in capability, specialization, latency, or cost. Even under this premise, the router baselines are fairly weak, acting more like diagnostics, rather than true baselines. A stronger selection would include routers closer to SOTA, including preference-based routers, reasoning routers and/or contectual-bandit formulations.
Questions for authors
  • What motivated the choice of k=16k=16 responses per policy and prompt? At this budget, the resolution of reliability coverage is quite coarse. e.g., at τ=0.05\tau=0.05, a single success out of 16 changes the coverage decision. Could the authors clarify why k=16k=16 is the appropriate primary operating point, given that the k=64k=64 replication in Appendix F.2 materially improves statistical resolution?
  • How should τ\tau be chosen in a real deployment?
  • How is stochasticity controlled across evaluation generations? I assume that the response counts across different generations are not isoFLOPS, due to different generation lengths. It would be interesting to see this dimension discussed in the evaluation results.
  • Are the same policy advantages reproduced across independent rollout batches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.