acceptodds
Withdrawn from ICLR 2027

Auditing Oracle Headroom: Decision-Level Cross-Realization Validation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Instance-wise oracle upper bounds are read as headroom for adaptive acquisition, but when unacquired outcomes are estimated by a learned evaluator, the same realization both chooses the oracle's per-case plan and scores it. Gap stability cannot detect this: the gap can be identical under two realizations even when every per-case decision reverses. We therefore validate the decision rather than the gap. Cross-realization transfer holds the case and the source-selected plan fixed, changes the evaluator realization, and scores that plan against the target realization's own fixed baseline. Under an idealized zero-mean independent-error model, the gap is optimistic and transfer is conservative; in a controlled generator with realization noise alone, the gap grows without bound while transfer stays at zero. Across four grounded world-model contracts, transfer depends on the registered perturbation: positive transfer is supported in the three development regimes under same-data retraining, but only in DINO-WM/PushT under post-hoc pipeline re-instantiation. An oracle-headroom claim is only as strong as the evaluator change it has survived.

Primary area:
datasets and benchmarks
Keywords:
adaptive evaluation, oracle upper bounds, cross-realization transfer, decision-level validation, world models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ezcSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Despite multiple careful readings of the paper, it is impossible to understand what the paper is about nor what contributions it claims to make.

Critical strengths

None

Critical weaknesses
  • The paper uses a lot of jargon, without properly defining it, even informally (‘headroom’, ‘per-case plan’, ‘gap-stability’, ‘cross-realization transfer’, ‘world-model contracts').
  • The abstract and introduction fail to present the problem and the contribution.
Questions for authors

None.

Reviewer_NZGTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies an important validity gap in how instance-wise oracle upper bounds are used to motivate adaptive acquisition in learned evaluation settings. When unacquired outcomes are estimated by a learned evaluator, the same realization both selects the oracle's per-case plan and scores it. The paper shows that scalar oracle-gap stability cannot detect realization-specific ranking noise: the gap can be identical across realizations even when every per-case oracle decision is wrong under the target realization (Figure 1). The core methodological contribution is cross-realization transfer (T), a decision-level estimand defined by four choices: (i) the same case, (ii) the source-selected plan identity preserved, (iii) the evaluator realization changed, and (iv) scoring against the target realization's own fixed baseline. Under an idealized zero-mean independent-error model, the same-realization gap O is optimistic (one noise draw selects and scores the winner), while T is conservative (the source-selected plan need not be the stable optimum). In a controlled additive-noise generator with no stable case-specific signal, O grows without bound as noise increases while T stays at zero, demonstrating that realization noise alone can inflate oracle gaps without creating transferable value.

Critical strengths
  1. The paper identifies an important limitation of instance-wise oracle gaps: stable gap magnitudes do not necessarily imply stable per-instance decisions. The motivating example and controlled simulation clearly illustrate this issue.

  2. The proposed cross-realization transfer metric is well defined and provides a useful way to assess whether oracle-selected decisions remain valuable under evaluator changes.

  3. The experiments cover four world-model evaluation settings and distinguish different evaluator perturbations. The authors also transparently report negative results and limitations.

Critical weaknesses
  1. The framework evaluates clairvoyant oracle decisions rather than deployable acquisition policies. Although legal-policy gains are discussed in the supplement, their relationship to cross-realization transfer is not sufficiently demonstrated in the main results.

  2. Transfer under pipeline re-instantiation is fragile, with only PushT showing supported positive transfer. Since evaluator quality can also change across realizations, the contribution of evaluator-quality differences to these results remains unclear.

  3. The empirical evidence for positive transfer is limited. The three supporting narrow-nuisance results come from development analyses, while the only prospectively specified validation, DreamerV3, does not establish positive transfer.

Questions for authors
  1. Could the authors provide a direct comparison between cross-realization transfer and the gains achieved by deployable acquisition policies?

  2. Could the authors report evaluator quality across realizations and assess whether the broad-nuisance results persist after controlling for evaluator-quality differences?

  3. Could the authors provide additional prospective validation of positive transfer and report per-regime oracle-plan agreement across evaluator realizations?

Reviewer_XMmKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

when training a model under a scheme with adaptive feature selection, one can select features based on an "explanation ranked order" (which is in some sense an oracle, in that these explainers rank features according to their importance to the outcome of interest) to check if there is room to grow when adaptively selecting other features (think: SHAP values or other feature importance schemes). one could assume that the best one could do, in other words, is have access to this explanation based ranking. then, one could compare an adaptive selection technique relative to the ordering given by a post-hoc explanation, like SHAP. this paper argues that this upper bound (in terms of the best features selection scheme) can be inflated; the oracle's plan is both chosen and scored under the same random draw of the learned evaluator, so some of the room to improve is just due to the noise of that draw.

Critical strengths

separating selection and evaluation "oracles" under the adaptive feature selection problem seems like the right idea.

Critical weaknesses

the setting for this paper was so unclear that i had to go to one of the main citations of this paper (guney et al. 2025) to understand the problem statement (which i was able to understand within 3 sentences of reading their abstract, as that is a properly written paper). my understanding is now that this paper critiques the "empirical oracle" line of the guney paper, and similar other "empirical oracles" under the adaptive feature selection problem.

as far as i can tell, the paper doesnt audit guney's oracle per se; guney is just a motivating citation. instead, the experiments are on world-model setups (dino-wm, td-mpc2, dreamerv3) where the eval is a learned model that imputes probe outcomes. its more of a critique of Guney et al.'s assumptions (and other papers that make similar assumptions) than a direct improvement / modification to their results.

i am not sure why the experiments the authors chose are the appropriate setting for comparison here.

even under a generous reading, allowing for all the presentation issues, the paper's contribution is clearly below the bar at ICLR.

the paper is poorly written and reads like AI slop: language like "we validate the decision rather than the gap" reads like ai slop. what? please ground everything better, properly set up the problem with sufficient background. reading an ICLR paper should not be such a puzzle.

Questions for authors
  1. what is the main methodological contribution, beyond separating the randomness between feature selection and evaluation, to improve the oracle? is that it?
  2. why are these experiments appropriate here? do they have enough coverage to be compelling, given the motivating examples (i.e. Guney et al. 2025)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.