acceptodds
Withdrawn from ICLR 2027

Falsifying Our Own Model: A Prospectively Frozen Protocol for Spatio-Temporal Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Claims that a model change improves spatio-temporal forecasting often depend on evaluation choices rather than on the change itself. With test predictions fixed, leave-one-comparator-out analyses give six of twelve models at least one new best-in-cell highlight; omitting MetaDG moves Orion from 0/36 to 12/36. Such claims are made one change at a time during development. We apply preregistration principles at this level by binding each change to a matched control, executable decision rule, and auditable evidence bundle. A deterministic adjudicator returns a verdict or withholds one when evidence is incomplete; a non-exhaustive ledger records 71 logical events. The three-seed rule crossed in 10.4% of dependent real-seed null splits, whereas bolding the lower mean would have favored the prespecified treatment in 50% of them. Under a normal model, the rule detects a one-standard-deviation effect in 44% of comparisons. Applied to 48 frozen hypotheses on Orion, the protocol overturned the account we would otherwise have written. One observational probe and three intervention tests motivated by a diagnosed pattern missed their frozen criteria; the probe lay outside this denominator. All three recipe changes tested on fresh three-seed cohorts crossed the same gate. Orion, although second only to MetaDG in every PEMS08 test cell, ranks fifth of twelve overall. Positive controls recovered two known harms, and the protocol was applied unchanged to STID. Eight of the 48 contracts carry Bitcoin anchors whose block times precede their first results or no-run bounds; five of them do so by more than the two-hour tolerance. Contracts, adjudicators, outcomes, and code are in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
empirical evaluation, benchmarking, reproducibility, spatio-temporal forecasting, traffic forecasting, time series forecasting, evaluation protocol

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sH9bOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes whether a claimed advantage of a model, as shown by metrics, is trustworthy. It identifies some serious potential issues, including variability in random seeds, changing evaluation criteria, and missing baselines. It then proposes a way to preregister an experiment tied to a fixed configuration, evaluation metric, and random seed selection. It then provides a case study of developing models in spatial-temporal forecasting to highlight these potential issues and how the method can partially fix the problems.

Critical strengths
  • The paper correctly identifies the potential issues in common practices in the current model and method search. These issues are important to avoid and often lead to false claims about the effectiveness of a proposed method.
Critical weaknesses
  • Preregistering an experiment does not eliminate selection bias. The criterion that the improvement must exceed one standard deviation also seems a bit arbitrary, and it is unclear whether it is too restrictive or too lenient, or if it should depend on the specific domain and problem. In addition, one can still achieve apparent improvements by chance simply by trying many different methods and selecting the best-performing one, so the method does not seem to avoid selection bias.
  • The paper only provides a case study in the spatio-temporal forecasting domain. The discussion of the problem and the proposed solution should not be tied to this specific domain, and the empirical study will be much stronger if extended to more applications.
Questions for authors
  1. The issue studied in the paper is a fundamental one, and should not be limited to just the spatial-temporal domain. Similar biases happen in virtually any other domain that involves neural network training. Is there a particular reason for focusing on spatio-temporal forecasting, and is there anything specific about the proposed method that only applies to this domain?
  2. The proposed protocol evaluates each experiment independently, but one could technically still conduct a large number of preregistered experiments with different modifications and obtain apparent improvements simply by chance. For instance, under some normal hypothesis, using the 1 standard deviation test, the proposed method will incorrectly accept roughly 16% of false positive improvements. In light of this, could the framework also incorporate quantities such as the total number of experiments conducted (e.g., when searching through a hyperparameter space) when determining whether an improvement is statistically significant, thereby accounting for this form of selection bias?
Reviewer_jryHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new protocol to properly compare machine learning models (here specifically spatio-temporal forecasting models) to prevent authors from over reporting method performance and architecture modification gains. Exemplary the paper applies this approach to Orion, their own method, and show that the method is competitive but does not improve on other approaches.

Critical strengths

This paper and especially their proposed protocol,tackles an issue that is affecting the entirety of machine learning research (researchers cherrypick results, runs and baselines to make their method look superior). As the is currently no widely adopted protocol that operates on model change level, I believe any discussion on such protocols are extremely valuable, especially with the development of recent submission numbers.

Demonstrating the protocol on their own method and showing its non-superiority is honest and mixes a negative (the method isn't working) with a positive (the protocol is solid) result which is an interesting read.

Critical weaknesses

Despite my positive attitude towards the general topic of the paper, I have a few major concerns about the current manuscript that are mainly concerned with its presentation:

I am unsure if the paper wants to propose a new protocol, a new method or something between both. For the rest of the review, I am assuming its option a) a new protocol for method and method alteration evaluation.

The abstract, and the introduction need major revisions. A large number of concepts and references (Method names, Numbers (0/36, 71 logical events, intervention, evidence bundle, etc. etc) are only defined much later in the manuscript, making the text almost incomprehensible without prior knowledge of the intention of the authors. Fundamentally, the paper reads like an extension to their method paper (Orion) and is does not really stand on its own. If Orion is published somewhere else already the manuscript bundles a method paper and a protocol paper and should probably be split in a clearer manner. If not, the paper simply attempts a little to much. Notably, in the interest of not breaking the double-blindness I did not look it up. Why does the abstract start with a sentence about spatio-temporal forecasting? Why are we talking about specific methods in the abstract? I assume the reasoning is marked in the abstract: “the protocol overturned the account we would otherwise have written”. This shows in many sections of the paper and makes it often very confusing to read.

While negative results are interesting and the protocol seems very reasonable, I am unsure if it provides enough novelty for this venue as a number of steps (run multiple seeds, rerun and validate baselines) are standard practice and other boil down to preregistration( per change frozen contract, bitcoin anchoring).

Questions for authors

If you want to highlight the effectiveness of your evaluation protocol, would it be possible to test it on a simple other domain? Maybe some toy regression task and a number of standard approaches for forecasting? Demonstrating the protocol on a genuine different task would strengthen the generality claim of the protocol.

Why is it important to use a cryptographic decentralization? Wouldn't it be enough to rely on a signed timestamped registry?

How do you see the usage of this protocol in a larger development cycle where many unsuccessful attempts (often through bugs) are to be expected. Is the overhead worth it?

Reviewer_BNVROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of model improvement claims in machine learning experiments. Using their spatio-temporal forecasting model Orion, the authors demonstrate how baseline selection and evaluation choices can affect reported results, including a change from 0/36 to 12/36 best-in-cell results when MetaDG is excluded.

The authors propose a Prospectively Frozen Protocol that combines pre-result hash-frozen experimental contracts, deterministic adjudication, fail-closed checks, and evidence records. They evaluate the protocol through 48 registered Orion hypotheses, statistical analyses of the three-seed decision rule, fresh-seed replications, positive controls, and additional experiments on STID.

Critical strengths

The paper integrates established evaluation practices into an executable intervention-level auditing framework. The systematic Orion self-audit, transparent reporting of negative results, and additional replication experiments demonstrate its practical applicability and provide useful evidence for improving experimental accountability.

Critical weaknesses
  1. The incremental methodological contribution is limited. The central problems addressed, including baseline selection bias, random-seed variability, selective reporting, and test-set reuse, are well established, as are the main underlying practices of preregistration, multi-seed evaluation, and reproducibility auditing. The novelty primarily lies in integrating these practices into an executable protocol. However, the experiments do not sufficiently demonstrate substantive advantages over rigorous existing evaluation workflows. For example, Orion's change from 12/36 to 0/36 best-in-cell results can be identified through complete baseline comparisons without the proposed protocol. Although Appendices B and C discuss related practices and protocol components, the incremental value of the integrated framework remains insufficiently established.

  2. The empirical validation remains limited. Only eight of the 48 Orion contracts have Bitcoin block-header times preceding their first results or documented no-run bounds, with five exceeding the stated two-hour tolerance. Although the authors acknowledge this limitation and provide additional records, independent verification of prospective freezing remains incomplete. Moreover, the evaluation primarily relies on the authors' own Orion model, with limited additional experiments on STID, leaving the framework's broader applicability insufficiently validated.

Questions for authors
  1. Suppose an existing evaluation workflow already includes complete baseline comparisons, applies appropriate multi-seed statistical analyses, and maintains reproducible records. What additional errors or unreliable conclusions would the proposed protocol identify or prevent? Can the authors provide a direct comparison or concrete evidence establishing the incremental benefits of the integrated framework?

  2. For the 40 out of 48 Orion contracts without external pre-result timestamps, how can third parties independently verify that the evaluation rules were fixed before the results were observed? Without such verification, what evidence supports treating these experiments as prospectively frozen?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.