- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether factual forecasts are sufficient to reliably use time-series models as action simulators. The authors propose a four-dimensional evaluation protocol covering accuracy, cross-step dependence, trajectories, and interventions. They formalize the limitations of marginal and factual-only evaluation, and conduct experiments using controlled forecasting architectures and pretrained foundation models. The results show that marginal accuracy can disagree with aggregate calibration, and that improvements in factual forecasting do not necessarily translate into better intervention estimates.
Critical strengths
1. The paper studies a practically important evaluation problem. The paper highlights an important distinction between factual forecasting and intervention-aware simulation. This is useful as forecasting foundation models are increasingly used for decision-making. The proposed evaluation dimensions provide a useful framework for diagnosing failures that conventional forecasting metrics may ignore.
2. Informative controlled experiments. The matched BD-JOINT versus BD-DIAG ablation provides evidence that better marginal forecasts can have substantially worse aggregate calibration. The cross-architecture fine-tuning experiments with Chronos-2 and Moirai further demonstrate that improvements in factual accuracy can lead to different changes in interventional error. These are notable empirical findings.
Critical weaknesses
-
Unclear separation between forecasting and interventional errors. Eq. (4) establishes baseline cancellation for log contrasts, but the primary aggregate CATE metric still depends on the baseline trajectory. Therefore, the reported interventional error may confuse baseline forecasting inaccuracies with errors in the learned action-response mechanism.
-
Incomplete evaluation of trajectory-level functionals. Although the proposed protocol includes five trajectory functionals, the quantitative results focus primarily on horizon-total calibration. It remains unclear whether the main conclusions generalize to other functionals.
-
Limited generalizability. The main correlations rely on only six configurations, most sharing the same backbone, and the experiments are restricted to retail datasets. Moreover, all models achieve zero regret on the in-support action menu (Appendix F.2), leaving the practical decision-making implications insufficiently demonstrated.
Questions for authors
-
Can the authors clarify why Eq. (4)'s cancellation of the baseline on the log scale is sufficient to justify the interpretation of aggregate CATE error? How much of the reported CATE error is caused by baseline forecast inaccuracies rather than action-response inaccuracies?
-
Is the ground-truth CATE computed as a conditional expectation given the available history? How are zero-sales observations handled in the log-contrast evaluation?
-
Can the authors provide the complete quantitative results for other evaluation functionals such as volatility, autocorrelation?
-
Could the authors provide more complete specifications of the synthetic logging policy and adaptation experiments, like the confounding signal, model adaptation hyperparameters, and the Chronos-2 fine-tuning results?
6 comments
Response to Reviewer kCE9 (2/2): Pricing regret, generalizability, logging and training details
(2/2) Here we address breadth, the zero-regret concern and experiment specifications.
W3. Generalizability and zero regret. The original unit-volume menu was degenerate. We added eleven synthetic prices and constant-action baselines (App. L.2, Tables 26–28).
Prices span −2.5σ to +2.5σ in .5σ steps at assumed costs .3/.5/.7 across all 28 runs. Selection uses predictions, prices, costs and the known Open calendar, never future outcomes. Comparators include an earlier-validation-selected constant price, an action-ignoring policy and a fixed reference price. The readout was locked before evaluation but reuses the cohort and studied costs; it is exploratory.
All three confounded Chronos adaptations improve RMSE 1174 → 781–801 yet worsen regret at every cost; at .5, 13.3% → 22.4–24.7%. Moirai improves RMSE and full-profile error but raises regret in 5/5 pairs at .5 (.1% → .2–8.0%). At .3, 4/5 pairs improve (2.2% → .6–1.8%); one worsens to 4.3%.
The constant/action-ignoring policy is strong: regret is 2.2%/.1%/0.0%. At .5, the highest price is optimal on 86% of stores and frozen Moirai always chooses it; adaptation departs from this near-optimal constant. Choices vary more at .3; at .7 the highest price wins everywhere. These are decision counterexamples, not evidence of general policy superiority or real profit gains.
- Model breadth: 15 configurations from five families now enter the reference comparison. RMSE/shared-total Spearman is +.05, but +.72 without Chronos: the correlation is family-sensitive, so within-model controlled comparisons carry the argument.
- Non-retail data: electricity passes all three documented adaptation rules: all three runs improve factual RMSE and improve shared-total error under randomized logs, but worsen it under confounding (App. G). The separate dependence ablation fails its marginal-calibration rule (.830 versus .686 per-step coverage). We retain that failure; domain changes do not isolate the mechanism.
Q4. Logging and adaptation specifications. App. A.5–A.6 gives complete settings.
The action is . The signal is the standardized log of the arithmetic mean of open-day demand in the trailing seven days. We set , and . Action scale is capped at .25; log-price at ±1. Elasticities are −1.5/−3/−2/−2.5, carryover .30, decay .60; realized confounding is .001/.152/.621.
Chronos full adaptation uses history only, context 462, 3,000 steps, batch 32 and learning rate 1e-5. Three series-split seeds are used; the script does not forward them to trainer RNG. LoRA uses 5e-4, rank 8, alpha 16, dropout 0. Moirai uses AdamW, 3,000 steps, 1e-5, weight decay .01, context 256, patch 16, with five small and three base runs.
Table 13 gives both aggregate metrics at all three confounding levels; App. J gives every stepwise run. All six confounded Chronos settings improve RMSE (1174 → 778–932) while worsening pathwise total-effect error (.208 → .49–.73). General B covers PA, normalization and the separate origin-only control. Specifications support reimplementation; complete code release requires institutional approval.
If these changes resolve your concerns, we would appreciate reconsideration of your rating.
Thank you for the detailed responses. The authors have addressed several of my main concerns. In particular, the new full-profile control provides stronger evidence for the intervention findings. The clarification of several other aspects such as the CATE metric and additional trajectory-level results also improve the paper substantially.
However, I still have concerns about the limited generalizability of the findings, as the main conclusions rely heavily on semi-synthetic interventions. And the lack of publicly available code or experimental artifacts makes independent verification difficult. I also find the overall novelty somewhat limited. While the paper is now more convincing, I am not yet confident that its contribution is sufficient for acceptance. Therefore, I lean toward maintaining my rating of 2.
Response to Reviewer kCE9 (1/2): Baseline controls, effect target, zero sales and trajectory functionals
(1/2) We corrected the baseline claim and added a stronger control. Here we address it, the target and all five functionals.
W1 / Q1. Forecasting versus interventional error. You are right: Eq. (4) does not justify baseline immunity for a total-demand effect. We renamed the experimental metric pathwise total-effect error. General A gives the corrected identity and Table 12's baseline-only counterfactual. Sharing the true total changes eight of nine means by ≤.012, but BD-AR moves .117 → .201. Signed terms can offset, so baseline contributions are not additive percentages. A shared scalar total also leaves timing error in the predicted lift.
The stronger control applies predicted daily intervention/factual ratios to true factual demand on each day, fixing both magnitude and timing of those weights (App. L.1, Table 25).
Chronos improves RMSE 1174 → 781–801 but worsens full-profile error in all three confounded adaptations; randomized logging improves both. Moirai improves in all five pairs per cell. Predicted factual weights cannot alone explain the Chronos counterexample. Ratios can still contain model error; this is not pure causal-error isolation. Table 27 gives all 28 runs. The readout was locked before evaluation but reuses the cohort and is exploratory.
Q2. Conditional expectation and zero sales. Ground truth replays the mechanism on the realized held-out path, carrying its history state forward. We do not average over random future paths given history. The text after Eq. (12) distinguishes this target from nondegenerate conditional CATE.
Level metrics use 19,121/24,000 open, finite, positive-sales steps (79.7%) in the 500-series Rossmann reference. Zeros have zero multiplicative level effect; zeros and missing outcomes are excluded. Log scoring requires positive finite predictions in both arms: up to 32 extra exclusions (.17%) for the cut, 88 (.46%) across reference arms. Positive levels are then floored at 1, which can break scale invariance. Exclusion counts do not measure floor activation (General A; App. A.1/J).
The new control uses matched masks, retains nonpositive intervention predictions and reports one floor-active step. Sensitivities change full-profile error by <.002, preserving the result (App. L.1); this does not measure all historical cells.
W2 / Q3. Complete functional results. Tables 5–6 give CRPS and coverage for all five functionals and ten configurations, including DeepAR and frozen Moirai. The historical matched comparison is:
Coverage gains extend to volatility and autocorrelation. Peak barely changes; both rows overcover time-to-peak. This is not a uniform gain across functionals or scores. App. K adds the origin-only comparison; General B gives its limits. Table 14 additionally reports peak-effect errors for all 15 configurations, including improved total error alongside worse peak error. Peak uses each model's own factual predictions and has no scalar shared-total decomposition.