Falsifying Our Own Model: A Prospectively Frozen Protocol for Spatio-Temporal Forecasting

Claims that a model change improves spatio-temporal forecasting often depend on evaluation choices rather than on the change itself. With test predictions fixed, leave-one-comparator-out analyses give six of twelve models at least one new best-in-cell highlight; omitting MetaDG moves Orion from 0/36 to 12/36. Such claims are made one change at a time during development. We apply preregistration principles at this level by binding each change to a matched control, executable decision rule, and auditable evidence bundle. A deterministic adjudicator returns a verdict or withholds one when evidence is incomplete; a non-exhaustive ledger records 71 logical events. The three-seed rule crossed in 10.4% of dependent real-seed null splits, whereas bolding the lower mean would have favored the prespecified treatment in 50% of them. Under a normal model, the rule detects a one-standard-deviation effect in 44% of comparisons. Applied to 48 frozen hypotheses on Orion, the protocol overturned the account we would otherwise have written. One observational probe and three intervention tests motivated by a diagnosed pattern missed their frozen criteria; the probe lay outside this denominator. All three recipe changes tested on fresh three-seed cohorts crossed the same gate. Orion, although second only to MetaDG in every PEMS08 test cell, ranks fifth of twelve overall. Positive controls recovered two known harms, and the protocol was applied unchanged to STID. Eight of the 48 contracts carry Bitcoin anchors whose block times precede their first results or no-run bounds; five of them do so by more than the two-hour tolerance. Contracts, adjudicators, outcomes, and code are in the supplementary material.