- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks whether static group labels (store, platform, category) help forecasters built on frozen time-series foundation models. It splits the question into access, fitted value and transfer.
- Access. The authors give a sufficient condition (Proposition 1) under which per-channel normalization exactly erases a constant covariate. Tests of five released forecasters on 70 M5 series agree:
- Chronos‑2, TimesFM‑3 and TabPFN‑TS give bit-identical forecasts under relabelling.
- Moirai‑2's response is traced to unobserved padding entries.
- TimesFM‑2.5 responds only negligibly.
- Fitted value. On a proprietary fundraising panel, a count head on frozen Chronos‑2 or Moirai features reads a platform-code embedding. Over ten seeds, the true code beats constant and permuted codes, and also group mean-and-dispersion calibration. This holds only for campaigns seen during development.
- Transfer.
- Gains are unresolved for unseen campaigns and absent for M5 store labels.
- On one Moirai‑2 checkpoint, the true code beats a permuted code but loses to a label-free shared code.
- On MDCC, ordinary history recalibration is already a strong baseline.
The authors conclude that true-versus-permuted label ablations need shared-code and ordinary-calibration references. They propose a three-step check: relabel the input bit for bit, refit with the controls, and report results per population.
Critical strengths
- Clear framing of the question. Splitting it into access, fitted value and transfer is a sensible way to organize label ablations, and the paper is the first to address them systematically for frozen time-series foundation models.
- Exact, reproducible interface census. Testing five released forecasters bit for bit, with time-varying liveness controls, gives a concrete, cheap and public check. Tracing Moirai‑2's response to its padding entries, then removing it by intervention, is careful mechanistic work.
- Useful methodological caution. The paper shows that a favorable true-versus-permuted result can coexist with a loss to a label-free shared code. This is a concrete reason to add shared-code and calibration baselines to label ablations.
- Practically relevant side finding. Passing a covariate channel at all can change forecasts substantially, even when its value is erased. Practitioners comparing runs with and without covariates should know this.
- Transparency. Results are prespecified, every arm and population is reported (including null and adverse results), the claim status table (Table 56) is honest, and saved scores are reconstructed independently.
Critical weaknesses
- Limited novelty of the theory.
- Proposition 1 amounts to (c − c)/s = 0.
- Eq. 1 is the classical conditional-entropy identity, and the authors concede it "neither predicts nor bounds" the fitted gain.
- The partition lemma and Gaussian-smoother examples restate known cautions on refit-based importance and covariance penalties.
- No theory explains the paper's own key findings: the shared code beating true codes, the ordering flip between validation and later windows, and the lack of transfer.
- Weak, fragile empirical support for fitted value.
- The gains are 0.005–0.015 nats per path.
- Their intervals use optimization seeds as the unit of uncertainty and mostly include zero when campaigns are resampled.
- On Moirai, the gain is concentrated in one 72-campaign group.
- The ordering differs on the earlier validation split, which the authors cannot explain.
- Evaluation data reused and many analyses post hoc. The later window was scored under several protocols without adjustment across them, and several decompositions and resampling analyses were added after results were known.
- No transfer, and the main result cannot be verified.
- The positive result is limited to development-seen campaigns in a private dataset.
- It is unresolved for unseen campaigns and absent for M5 store labels.
- The shared-code counterexample comes from one checkpoint with hand-injected codes, and the five-seed follow-up no longer resolves true versus permuted.
- Controls do not isolate information. The calibration baselines do not nest with the head, use a finite search, differ in capacity (588 vs. 840 parameters in MDCC), and lack confirmed convergence. "Beats calibration" therefore mixes information with function class and optimization.
- Limited practical payoff.
- Results are reported only in nats, with no standard accuracy metric or decision-level impact.
- No method, fix or rule for deciding in advance whether to use a label is offered; Table 56 marks this "Not evaluated."
- The proposed protocol is essentially standard good practice.
- Presentation. Heavy hedging, 13 appendices and many separately adjusted experiment families make the core argument hard to reconstruct.
Questions for authors
- Can you report the fundraising results in a standard forecasting metric (e.g., MASE, CRPS or weighted quantile loss) and explain what a gain of 0.005–0.015 nats per path means in practice?
- Do the main fitted-value contrasts remain significant when uncertainty accounts for campaign and seed variation together?
- Why does the ordering of the true, constant and permuted arms differ between the development validation split and the later window?
- What mechanism do you propose for the shared code beating the true code, and does that result hold with more seeds or on other checkpoints?
- Can a matched recalibration control (equal capacity, nested with the head, converged fits) be added, so that "beats calibration" isolates label information?
- Beyond the relabelling check, what decision rule do you recommend for when a practitioner should include a static label?
Thanks for the comments. What is new (W1, Q4). Proposition 2 isolates the cost a shuffle leaves out. It applies to balanced groups, an equiangular codebook, a linear readout and orthogonal nonlabel inputs. With lower risk better, true minus shuffled risk equals Λ−Γ, true minus shared equals Σ−Γ and shuffled minus shared equals Σ−Λ. Γ is label signal, Λ is signal that survives shuffling, and Σ is the cost of estimating group components rather than one common component. A shuffle cancels that cost. Thus true beats shuffled yet loses to shared precisely when Λ<Γ<Σ; the shuffle table bounds leakage before fitting (Appendix H). Centering a constant is Observation 1. Appendix B credits the classical information identity. In the nonlinear M5 encoder, the shared code also beats an erased input, 7.99774 versus 8.01432 nats; a linear model with a free intercept would give equality. Unequal code distances and the count loss also limit the analogy.
Fitted gains, uncertainty and transfer (W2–W4, Q1–Q3). On the fixed private cohort the mean log-loss gains over sequential calibration correspond to 1.55% and 0.48% on the geometric likelihood scale. Varying campaigns and seeds together, true minus calibration gives adjusted intervals [−0.06079,+0.00412] (Chronos-2) and [−0.01592,+0.00449] (Moirai-large). Crossing 27 encoded platforms and seeds gives [−0.07380,+0.00472] and [−0.03978,+0.01045]; even ordinary 95% intervals span zero. True versus constant also spans zero. Matched permutations have no corresponding crossed analysis (Appendix J.1). In the later cohort, every single-platform omission preserves the gain, but taking out one 72-campaign group leaves only 16% and 2% of the Moirai gains over shuffled and constant codes. All adjusted ordinary-calibration comparisons for 607 training-unseen campaigns span zero. Four earlier protocols used the later window, and our post-review analyses reuse it. These are opened cohorts.
For counts, daily CRPS with true labels versus sequential calibration is 0.01072 versus 0.01163 on Chronos-2 and 0.00958 versus 0.00934 on Moirai-large. All fitted arms beat zero (0.01569) in point estimates; fitted-reference intervals span zero. Medians beat zero on daily MAE; means lost (Table 4). On validation, constant beat true, and true beat shuffled. For 1,767 identical campaigns under the same heads, the true-minus-constant contrast decreases by 0.0074 from validation to later scoring (Table 19). Composition alone cannot explain that within-campaign change; its cause remains open.
On these same campaigns the Chronos-2 change stays negative after every platform omission. Moirai-large is near zero and 11 of 29 omissions reverse it (Table 45).
How strong is the public result (W4, W6, Q4)? On one M5 checkpoint, true beats shuffled in the original ten-seed log score and loses to a shared code. With five further seeds the true-minus-shuffled interval is [−0.00824,+0.00508]; a reserved shared code equidistant from all stores retains its advantage. The adjusted true-minus-shared CRPS interval is [+0.00126,+0.00446]; against shuffled it is [−0.00151,+0.00094] (Table 43). Across eight primary retail store transfer contrasts, none supports true. Only one checkpoint was tested.
Jointly resampling paired seeds and series within retail strata reinforces this distinction. All nine percentile shared-code intervals favor sharing. Seed-only Student intervals retain seven, a Student multiplier on crossed uncertainty three, and a small-seed correction two (J.7). Before extending to thirty new seeds we repeated twenty old fits. Their maximum mean discrepancy, 0.00153 nats, exceeded the 0.00050 tolerance, so no new-seed fit was run. All five shuffle intervals cross zero after adjustment across 14 contrasts on one checkpoint (Table 44). Resampling whole stores instead, under the original ten seeds, preserves the advantage over true codes: sharing wins in nine of ten individual stores, and the pooled shuffle contrast spans zero (Appendix J.7).
Controls and use (W5–W7, Q5–Q6). We have no equal-capacity, nested and converged private calibration control. Joint fitting raises gradient convergence from 100/740 to 734/740 groups; the later margins move by under 0.0001 and both intervals still span zero (Appendix D.3). On MDCC, history has 588 coefficients and the true metadata fit 840. The added shuffled comparison holds both arms at 840. True beats five shuffled fits by 0.03757 nats, under the penalty chosen on real labels. Neither meets the strict gradient threshold. The percentile date interval favors true over shuffled; a weighted Student sensitivity spans zero, as does the percentile comparison against history (Table 5, Section 5). The usual shuffle test omits the shared code, which on M5 changes the decision. For W4, public aggregate replays reproduce the contrasts, not the proprietary fits. For W7, Appendix A.1 maps the studies and Table 6 names the sampling units.