acceptodds
Withdrawn from ICLR 2027

UBOA: Unpacking Benchmark Outcomes and Assumptions for Properly Scoped Forecasting Claims

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A validation-selected forecasting rule may reject a 5% improvement null without supporting a guarantee conditional on its selection history. UBOA makes this distinction explicit through an executable specification for matching statistical claims to evidence. Each request binds the selected rule, comparator, loss, evaluation law, error event, conditioning information, and coverage unit. Eight rules either license the unchanged conclusion or retain an explicit mismatch alongside separately scoped evidence. Relative soundness requires correct annotations and external premises, which the checker does not authenticate. In a 140-case contract challenge fixed before access to the checker implementation, the revised matcher made no false accepts or false rejects; coordinate equality produced 29/70 false accepts and 10/70 false rejects. We also instantiate existing concentration tools in two synthetic settings, preserving raw mean absolute error (MAE) or mean squared error (MSE), overlapping horizons, and the selected rule while supplying finite-sample history-conditional local control. A preregistered audit retained 80 histories and 327,680 continuations, with one false promotion, zero recorded first-true-null inclusion violations, and no Holm-adjusted evidence of excess error. Supplementary coefficients and counts support deterministic reconstruction of the certificates and aggregate audit quantities. Earlier simulations and six real applications retain their historical interpretations; the real applications do not have these conditional certificates.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
time series forecasting, forecast evaluation, post-selection inference, conditional inference, multiple testing, benchmark auditing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cJBAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents UBOA, an executable specification for checking whether a statistical claim about a validation-selected forecasting rule is supported by the available evidence. Each request records two groups of coordinates. Scientific coordinates are the selected rule, comparator, loss, index set, weights, evaluation law and threshold ladder. Validity coordinates are the error event, conditioning field, law quantifier, coverage unit and support mode. Eight rules (M1–M8) either return the unchanged conclusion as licensed or return the request as unsupported, with any weaker evidence kept as a separate record. The authors prove relative soundness, which holds only if the supplied annotations and external premises are correct. They also show that fixed-sequence composition yields history-conditional false-promotion control when each local test satisfies a conditional premise (Eq. 5). They instantiate that premise with two finite-sample certificates: bounded-innovation MAE with McDiarmid-type bounds, and Gaussian quadratic MSE with Cantelli bounds.

Critical strengths
  1. It separates claim types precisely, and the separation is formal and executable.

The paper distinguishes an observed decision, a fixed-rule marginal result, repeated-procedure control and full-history conditional validity. The ridge example and the two constructions in §4.2 show that fixed-rule marginal levels do not imply Eq. 5. This addresses a real reporting error in forecasting evaluation, where a test that rejects for a selected model is often read as a conditional guarantee. The distinction is encoded as machine-checkable rules rather than given only as advice.

  1. It limits its own claims carefully.

Soundness is stated as relative to the supplied premises. The real applications are explicitly excluded from the certified scope. The one false promotion and the 23.8% detection power of the audit are reported. The matcher labels were fixed before the checker was accessed.

Critical weaknesses
  1. The conditional guarantee does not cover the setting that motivates the paper.

The certificates need a declared future law and numerical structural bounds (contraction constants, Gaussian quadratic form) before evaluation. Neither of the paper's real applications has these. The fresh audit uses one contraction coefficient and two-rule pools with equal conditional risk. So it never tests selection among candidates with unequal risk, which is the source of selection bias. The certificates are also so conservative that terminal 5% reports occur in 0% of MAE-8% continuations and 0.046% of MSE-8% continuations. The paper therefore shows that conditional control exists in narrow synthetic cases, not that it can be used in practice.

  1. Most of the conformance evidence is circular, and the technical novelty is limited.

The 140 cases and their labels were written by an agent to the authors' own contract. A score of 0/0 therefore shows that the implementation matches the specification, not that the specification captures valid inference or helps reviewers. The theory consists of the classical fixed-sequence argument (Goeman & Solari) and existing concentration inequalities, which the paper acknowledges. With the human-review utility left untested, the contribution is mainly a reporting formalism. Its significance for a venue like ICLR is not established.

Questions for authors
  1. Can any real forecasting setting in practice supply the M6 premises (declared conditional law plus valid structural bounds) before evaluation? If so, please give one concrete example. If not, which part of the contribution remains useful when M6 can never fire?
  2. Does the guarantee still hold when candidates have unequal conditional risk, so that selection correlates with the future loss? Please report even one audit cell with that design.
  3. How were the 140 challenge cases generated, and who authored the reference labels? Do the 9 cases the original matcher falsely accepted, and the fixes that removed those errors, come from the same distribution? If they do, the revised matcher's 0/0 result may be an in-sample fit.
  4. The stored critical value is 5.2988, while quadrature gives 5.32268. Does this gap change any historical decision near the threshold, or the null-rejection rates in Table 5?
Reviewer_afx1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to do verification of the model, i.e., predicting whether some model really improves the task performance and decide whether to accept or reject changes. The paper proposes a new method called "UBOA", which uses some statistical claims as the evidence of acceptance/rejection decision of the model. The experiments are conducted on 140 synthesized scenarios.

Critical strengths

None (I did not really understand the paper due to reasons listed in weaknesses. I indeed put a lot of effort trying to understand, but failed because many terminologies are introduced without proper definitions.)

Critical weaknesses

The biggest issue is that the paper is simply human-unreadable. I list the specific issues below.

(1) Problem setting is not really explained. For instance, abstract and intro starts with the issue of "reject a later 5% improvement null", but the background condition is not provided. Is it rejected because the verification method only look at the mean and do not consider the entire distribution? Usually in a paper, these background assumptions and situations should be presented, while this paper do not. The paper has abstract-level discussion in almost entire context, making it hard to follow.

(2) "relative soundness", which appears to be important notion is neither defined nor explained in both abstract and intro. Similarly, in related work, several seemingly important ideas of "pooling-level testing", "inference after explicit selection", "independent calibration testing", etc, are only mentioned by these abstract words and what each paper exactly does were never explained. It's fine to use LLM in writing, but the author(s) need to make them at least digestable for humans by adding sufficient explanation.

(3) In the notation introduction in Section 3, what is "evidence-law kernel" is neither defined or explained. What does the "relative ratio" θ\theta means is not explained.

(4) In the paper, it is not clear what the proposed method "UBOA" refers to. Specifically, it is not clearly explained whether UBOA is some validation criteria (e.g., loss function in Eq. (2)), or some architecture, or some pipeline procedure. It will be useful to make it clearer by which part is done by existing work and from which part is the proposed method.


Aside from the above issues in writing, the paper should explain whether the experiment procedure follows standard process for the validation work, and introduce some reasonable baselines. I'm not very familiar with the literature and thus I cannot make specific recommendation, but if the experiment follows the standard, citation should be included. If the setting is original to this paper, why the setting is reasonable should be explained, and some reasonable baselines should be included.

Questions for authors

Please write rebuttals with author(s)' own words, as LLM-generated sentence may be hard to follow due to lack of contexts.

  • What is "relative soundness"?

  • What are the specific approaches in related work?

  • What is the scope of the proposed approach, "UBOA" (e.g., it is loss function or pipeline, etc)?

  • Please explain what each variable means in the notation.

Reviewer_jkYKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes UBOA, an executable specification for matching claims about validation-selected forecasting rules to the evidence that supports them. The authors also instantiate finite-sample, full-history-conditional local tests (bounded differences for MAE, Cantelli for Gaussian MSE) in synthetic AR(1) settings and compose them by fixed sequence.

Critical strengths
  • The paper separates fixed-rule marginal, repeated-procedure, selected-label, and full-history-conditional guarantees. The sequential counterexamples in Section 4.2 cleanly show that marginal level does not imply conditional level.
  • The overlap coefficients account for shared innovations across overlapping origins and horizons, so the forecasting target is preserved.
Critical weaknesses

Limited contribution and evaluation of the matcher.

  • The relative soundness result largely follows from established principles (theorem instantiation, the tower property, and fixed-sequence testing), assuming correct premises that the matcher does not verify.
  • The AI-authored 140-case challenge demonstrates conformance to the authors' specification rather than practical utility.
  • The baselines are weak, and the matcher is not evaluated on published forecasting claims.

Unclear practical value of the conditional certificates.

  • Conditioning on the full pre-evaluation history fixes the selected rule, so inference reduces to a fixed-rule test under the conditional future law and sidesteps the core difficulty of post-selection inference.
  • The certificates require strong structural knowledge; in both synthetic settings the target conditional risk is directly computable, and no real application satisfies the assumptions.
  • The experiments also use candidates with equal conditional risk, so selection effects are not tested. Power is extremely low at moderate effects: terminal 5% reports occur in 0% (MAE) and 0.05% (MSE) of continuations at a true 8% improvement.
Questions for authors

see above weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.