acceptodds
Under review as a conference paper at ICLR 2027

Forecast Quality Does Not Certify an Action Simulator

Abstract

Forecasting models are increasingly used to predict demand under hypothetical actions, effectively treating them as action simulators. Yet accurate factual forecasts do not guarantee reliable simulation. We identify two blind spots. First, per-step forecast evaluation can leave cross-step dependence unconstrained, even though it shapes uncertainty in trajectory-level quantities such as total demand. Second, evaluation under observed actions does not constrain how a model responds when those actions change. On two public retail panels, standard multivariate proper scores largely reproduce marginal-accuracy rankings. A matched rank-zero ablation improves weighted quantile loss (WQL) from to , while coverage of total demand falls from to and the detected dependence signal disappears. Fine-tuning improves factual forecasts for Chronos-2 and Moirai at every confounding level, but their intervention estimates differ: Chronos-2 shifts from improvement under randomized actions to degradation under confounding, whereas Moirai improves throughout. Decoder rankings also reverse under a second action-response mechanism, showing that simulator comparisons depend on the response being simulated. These findings motivate separate evaluation of marginal accuracy, cross-step dependence, aggregate calibration, and interventional fidelity rather than treating forecast quality as a certificate of simulation quality.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, evaluation, foundation models, counterfactual prediction, causal inference, proper scoring rules, probabilistic forecasting, world models, benchmarks, decoding strategies

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kCE9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether factual forecasts are sufficient to reliably use time-series models as action simulators. The authors propose a four-dimensional evaluation protocol covering accuracy, cross-step dependence, trajectories, and interventions. They formalize the limitations of marginal and factual-only evaluation, and conduct experiments using controlled forecasting architectures and pretrained foundation models. The results show that marginal accuracy can disagree with aggregate calibration, and that improvements in factual forecasting do not necessarily translate into better intervention estimates.

Critical strengths

1. The paper studies a practically important evaluation problem. The paper highlights an important distinction between factual forecasting and intervention-aware simulation. This is useful as forecasting foundation models are increasingly used for decision-making. The proposed evaluation dimensions provide a useful framework for diagnosing failures that conventional forecasting metrics may ignore.

2. Informative controlled experiments. The matched BD-JOINT versus BD-DIAG ablation provides evidence that better marginal forecasts can have substantially worse aggregate calibration. The cross-architecture fine-tuning experiments with Chronos-2 and Moirai further demonstrate that improvements in factual accuracy can lead to different changes in interventional error. These are notable empirical findings.

Critical weaknesses
  1. Unclear separation between forecasting and interventional errors. Eq. (4) establishes baseline cancellation for log contrasts, but the primary aggregate CATE metric still depends on the baseline trajectory. Therefore, the reported interventional error may confuse baseline forecasting inaccuracies with errors in the learned action-response mechanism.

  2. Incomplete evaluation of trajectory-level functionals. Although the proposed protocol includes five trajectory functionals, the quantitative results focus primarily on horizon-total calibration. It remains unclear whether the main conclusions generalize to other functionals.

  3. Limited generalizability. The main correlations rely on only six configurations, most sharing the same backbone, and the experiments are restricted to retail datasets. Moreover, all models achieve zero regret on the in-support action menu (Appendix F.2), leaving the practical decision-making implications insufficiently demonstrated.

Questions for authors
  1. Can the authors clarify why Eq. (4)'s cancellation of the baseline on the log scale is sufficient to justify the interpretation of aggregate CATE error? How much of the reported CATE error is caused by baseline forecast inaccuracies rather than action-response inaccuracies?

  2. Is the ground-truth CATE computed as a conditional expectation given the available history? How are zero-sales observations handled in the log-contrast evaluation?

  3. Can the authors provide the complete quantitative results for other evaluation functionals such as volatility, autocorrelation?

  4. Could the authors provide more complete specifications of the synthetic logging policy and adaptation experiments, like the confounding signal, model adaptation hyperparameters, and the Chronos-2 fine-tuning results?

6 comments
AuthorsOpenReview ↗

Response to Reviewer kCE9 (1/2): Baseline controls, effect target, zero sales and trajectory functionals

(1/2) We corrected the baseline claim and added a stronger control. Here we address it, the target and all five functionals.

W1 / Q1. Forecasting versus interventional error. You are right: Eq. (4) does not justify baseline immunity for a total-demand effect. We renamed the experimental metric pathwise total-effect error. General A gives the corrected identity and Table 12's baseline-only counterfactual. Sharing the true total changes eight of nine means by ≤.012, but BD-AR moves .117 → .201. Signed terms can offset, so baseline contributions are not additive percentages. A shared scalar total also leaves timing error in the predicted lift.

The stronger control applies predicted daily intervention/factual ratios to true factual demand on each day, fixing both magnitude and timing of those weights (App. L.1, Table 25).

Full-profile error, sustained cutFrozenAdapted mean [run range]
Chronos-2, randomized.159.054 [.045, .061]
Chronos-2, confounded.216.504 [.503, .505]
Moirai-small, randomized.286.237 [.210, .260]
Moirai-small, confounded.257.159 [.095, .245]

Chronos improves RMSE 1174 → 781–801 but worsens full-profile error in all three confounded adaptations; randomized logging improves both. Moirai improves in all five pairs per cell. Predicted factual weights cannot alone explain the Chronos counterexample. Ratios can still contain model error; this is not pure causal-error isolation. Table 27 gives all 28 runs. The readout was locked before evaluation but reuses the cohort and is exploratory.

Q2. Conditional expectation and zero sales. Ground truth replays the mechanism on the realized held-out path, carrying its history state forward. We do not average over random future paths given history. The text after Eq. (12) distinguishes this target from nondegenerate conditional CATE.

Level metrics use 19,121/24,000 open, finite, positive-sales steps (79.7%) in the 500-series Rossmann reference. Zeros have zero multiplicative level effect; zeros and missing outcomes are excluded. Log scoring requires positive finite predictions in both arms: up to 32 extra exclusions (.17%) for the cut, 88 (.46%) across reference arms. Positive levels are then floored at 1, which can break scale invariance. Exclusion counts do not measure floor activation (General A; App. A.1/J).

The new control uses matched masks, retains nonpositive intervention predictions and reports one floor-active step. Sensitivities change full-profile error by <.002, preserving the result (App. L.1); this does not measure all historical cells.

W2 / Q3. Complete functional results. Tables 5–6 give CRPS and coverage for all five functionals and ten configurations, including DeepAR and frozen Moirai. The historical matched comparison is:

Coverage, nominal .80TotalPeakVolatilityTime-to-peakLag-1 ACF
BD-JOINT.669.811.787.995.837
BD-DIAG.283.797.494.999.589

Coverage gains extend to volatility and autocorrelation. Peak barely changes; both rows overcover time-to-peak. This is not a uniform gain across functionals or scores. App. K adds the origin-only comparison; General B gives its limits. Table 14 additionally reports peak-effect errors for all 15 configurations, including improved total error alongside worse peak error. Peak uses each model's own factual predictions and has no scalar shared-total decomposition.

AuthorsOpenReview ↗

Response to Reviewer kCE9 (2/2): Pricing regret, generalizability, logging and training details

(2/2) Here we address breadth, the zero-regret concern and experiment specifications.

W3. Generalizability and zero regret. The original unit-volume menu was degenerate. We added eleven synthetic prices and constant-action baselines (App. L.2, Tables 26–28).

Prices span −2.5σ to +2.5σ in .5σ steps at assumed costs .3/.5/.7 across all 28 runs. Selection uses predictions, prices, costs and the known Open calendar, never future outcomes. Comparators include an earlier-validation-selected constant price, an action-ignoring policy and a fixed reference price. The readout was locked before evaluation but reuses the cohort and studied costs; it is exploratory.

All three confounded Chronos adaptations improve RMSE 1174 → 781–801 yet worsen regret at every cost; at .5, 13.3% → 22.4–24.7%. Moirai improves RMSE and full-profile error but raises regret in 5/5 pairs at .5 (.1% → .2–8.0%). At .3, 4/5 pairs improve (2.2% → .6–1.8%); one worsens to 4.3%.

The constant/action-ignoring policy is strong: regret is 2.2%/.1%/0.0%. At .5, the highest price is optimal on 86% of stores and frozen Moirai always chooses it; adaptation departs from this near-optimal constant. Choices vary more at .3; at .7 the highest price wins everywhere. These are decision counterexamples, not evidence of general policy superiority or real profit gains.

  • Model breadth: 15 configurations from five families now enter the reference comparison. RMSE/shared-total Spearman is +.05, but +.72 without Chronos: the correlation is family-sensitive, so within-model controlled comparisons carry the argument.
  • Non-retail data: electricity passes all three documented adaptation rules: all three runs improve factual RMSE and improve shared-total error under randomized logs, but worsen it under confounding (App. G). The separate dependence ablation fails its marginal-calibration rule (.830 versus .686 per-step coverage). We retain that failure; domain changes do not isolate the mechanism.

Q4. Logging and adaptation specifications. App. A.5–A.6 gives complete settings.

The action is Δpt=−λκist+σξ,iξt\Delta p_t=-\lambda\kappa_i s_t+\sigma_{\xi,i}\xi_t. The signal is the standardized log of the arithmetic mean of open-day demand in the trailing seven days. We set κi=.7σi/∣βi∣\kappa_i=.7\sigma_i/|\beta_i|, σtot,i=κi/.5\sigma_{\mathrm{tot},i}=\kappa_i/\sqrt{.5} and σξ,i2=σtot,i2−(λκi)2\sigma_{\xi,i}^2=\sigma_{\mathrm{tot},i}^2-(\lambda\kappa_i)^2. Action scale is capped at .25; log-price at ±1. Elasticities are −1.5/−3/−2/−2.5, carryover .30, decay .60; realized confounding is .001/.152/.621.

Chronos full adaptation uses history only, context 462, 3,000 steps, batch 32 and learning rate 1e-5. Three series-split seeds are used; the script does not forward them to trainer RNG. LoRA uses 5e-4, rank 8, alpha 16, dropout 0. Moirai uses AdamW, 3,000 steps, 1e-5, weight decay .01, context 256, patch 16, with five small and three base runs.

Table 13 gives both aggregate metrics at all three confounding levels; App. J gives every stepwise run. All six confounded Chronos settings improve RMSE (1174 → 778–932) while worsening pathwise total-effect error (.208 → .49–.73). General B covers PA, normalization and the separate origin-only control. Specifications support reimplementation; complete code release requires institutional approval.

If these changes resolve your concerns, we would appreciate reconsideration of your rating.

Thank you for the detailed responses. The authors have addressed several of my main concerns. In particular, the new full-profile control provides stronger evidence for the intervention findings. The clarification of several other aspects such as the CATE metric and additional trajectory-level results also improve the paper substantially.

However, I still have concerns about the limited generalizability of the findings, as the main conclusions rely heavily on semi-synthetic interventions. And the lack of publicly available code or experimental artifacts makes independent verification difficult. I also find the overall novelty somewhat limited. While the paper is now more convincing, I am not yet confident that its contribution is sufficient for acceptance. Therefore, I lean toward maintaining my rating of 2.

Reviewer_Vuu3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to evaluate time-series forecasting models when they are used as action simulators. The authors propose an evaluation protocol that checks whether sampled trajectories preserve dependence across time and produce reliable uncertainty estimates for total demand. It also tests action responses using semi-synthetic data with known intervention effects. They show that improving forecast accuracy can worsen total-demand calibration. Fine-tuning can also improve factual predictions while degrading intervention estimates. The paper therefore argues that simulation quality must be evaluated directly rather than inferred from forecasting performance.

Critical strengths
  • The question studied is interesting and meaningful. The dependency problem over time showcases the potential weakness of LLMs
  • The fine-tuning and decoding experiments provide useful empirical insights. Especially for the decoding experiments, it shows that changing the decoding strategy can significantly affect aggregate calibration and intervention estimates.
Critical weaknesses
  • The authors should clarify the scope of their claim that the contrast metrics are immune to baseline forecasting errors. Equation (4) establishes that the baseline cancels out in the stepwise log contrast, but it does not directly establish that the aggregate CATE error has the same property.
  • The training details are not given enough.
  • The paper is somewhat difficult to follow due to its writing and organization.
Questions for authors

Could the authors clarify how the AR and prefix-augmented configurations are trained? For example, when predicting a future step during training, does the model condition on the ground-truth values of preceding steps within the forecast horizon?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Vuu3

Your comments led us to correct the cancellation claim, clarify AR training and rewrite the introduction and results. The revised argument is that factual forecast quality does not certify a different query, such as total-demand uncertainty or response to a changed action.

W1. The baseline-immunity claim. You are right: Eq. (4) establishes cancellation for unfloored positive stepwise log contrasts, not total-demand effects. We withdrew the broader immunity claim and renamed the experimental metric pathwise total-effect error.

The stronger control applies predicted daily response ratios to true factual demand on each day. Confounded Chronos improves RMSE 1174 → 781–801 but worsens full-profile error .216 → .503–.505 in all three adaptations (§6.2; App. L.1, Table 25). Randomized Chronos and both Moirai-small cells improve. Predicted factual magnitudes or timing weights cannot alone explain the counterexample; this does not identify a pure causal-error component.

General A gives the shared-total identity, remaining timing dependence and independent stepwise check. Tables 19–21 retain exceptions: strong-confounding stepwise error increases in 3/3 Chronos, 3/5 Moirai-small and 2/3 Moirai-base runs, with floor/mask qualifications.

Q. Do AR/PA configurations condition on preceding ground truth? Yes. BC-AR predicts from true preceding targets; BD-AR predicts a masked terminal target from preceding true values. BD-OS masks its last 48 raw targets. At inference, AR feeds back sampled values and OS samples positions independently. Table 1 separates these behaviors.

Original PA uses 10 OS or 461 AR prefixes, with length bucketing and unequal compute; App. A.5 gives lengths, strides and target coverage. Added recipes remain separate.

Remark 1 excludes teacher-forced conditional objectives from Corollary 1: complete conditionals can learn dependence by the chain rule. Historical masked-label normalization remains disclosed. The separate origin-only matched comparison addresses that preprocessing issue, with mean gates and width/seed limits in General B and App. K.

W2. Training details. App. A.5–A.6 now supplies optimizer, sampling, normalization, adaptation, logging and mechanism settings. General B summarizes the changes. These support reimplementation; the complete pipeline is not yet available and requires institutional release approval.

W3. Readability. The rewritten introduction asks two questions. Section 5 explains what each effect check measures and controls. Sections 6.1 and 6.2 now open with the matched origin-only comparison and true daily factual-weight control, before supporting cases. Tables 1 and 3 separate configuration and metric definitions; detailed run tables stay in the appendix. The withdrawn-claim cross-references are corrected. A short reading path is §1 → §5/Table 3 → §6.1–6.2; General A/B contain the shared technical details.

On the remark about LLMs: the models here are time-series forecasters. The theoretical distinction concerns objectives and evaluation queries; we do not claim a general weakness of language models.

If these changes resolve your concerns, we would appreciate reconsideration of your rating.

Dear Reviewer Vuu3,

Thank you again for your time and thoughtful review of our work. As the discussion period draws to a close, we wanted to gently follow up to ensure you have had a chance to read our response and review the updated manuscript.

We would greatly appreciate it if you could let us know whether our rebuttal has addressed your concerns, and we hope you might consider re-evaluating your score in light of these revisions. Thank you for your consideration.

Best regards,
The Authors

I appreciate the authors’ efforts in making substantial revisions. However, these revisions involve extensive changes to the paper and a narrowing of its original claims. Therefore, I have decided to maintain my original score.

Reviewer_V2kCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether good forecasting performance is enough to make a time-series model a reliable action simulator. The authors propose evaluating models along four dimensions: marginal accuracy, cross-step dependence, trajectory functionals, and interventional consistency. Experiments on Rossmann and Favorita compare different model architectures and decoding strategies, using semi-synthetic data to evaluate intervention effects. The results show that better forecasting accuracy does not always lead to better calibration of total demand or more accurate intervention estimates. The authors also find that the relative performance of different decoding strategies depends on the assumed action-response mechanism.

Critical strengths
  1. The paper addresses an important problem in evaluating forecasting models used as simulators. Good forecasting accuracy does not necessarily mean that a model can correctly predict what happens when actions change. I think the proposed evaluation framework is useful for identifying these differences.
  2. The comparison experiment provides convincing evidence that better marginal accuracy does not guarantee better aggregate calibration.
  3. The paper includes several additional experiments to verify its findings, such as multiple training seeds, different forecast cutoffs, and a second action-response mechanism.
Critical weaknesses
  1. The paper argues that its contrast metrics are not affected by baseline forecast errors, based on Eq. (4). I can see why this holds for the stepwise log contrast, but I am less sure about the aggregate CATE metric. For total demand, the estimated effect still seems to depend on the baseline sales values. It would be helpful to clarify this distinction, especially when interpreting the Moirai fine-tuning results.
  2. I am not sure how Corollary 1 applies to the autoregressive models. If these models are trained to predict each step given the previous observed targets, the stepwise losses may still help them learn temporal dependence. The authors do not clearly explain the conditioning used during autoregressive training, particularly for the prefix-augmented models. More details would help clarify whether the corollary applies to these settings.
  3. Some important details for reproducing the experiments are missing. The appendix provides the main training settings and equations for the semi-synthetic data, but I could not find all the numerical parameters and details of the logging policy. The fine-tuning settings and implementation of prefix-augmented training are also not fully described. Since the paper relies heavily on controlled experiments, more complete implementation details or code would be helpful.
Questions for authors
  1. For the aggregate CATE metric, how much of the error comes from the baseline forecast rather than the estimated action response? Can the authors evaluate the response estimates using the same oracle baseline?
  2. How exactly are the autoregressive models trained? Are they conditioned on the previous ground-truth targets or only on the original forecast context? How does this affect the application of Corollary 1?
2 comments
AuthorsOpenReview ↗

Response to Reviewer V2kC

Thank you for asking us to check the oracle baseline and clarify AR training. We corrected the cancellation claim and added the requested controls.

W1 / Q1. Aggregate error and Moirai. You are right: total-demand effects depend on the factual forecast. Eq. (4) does not remove that dependence. General A explains the corrected identity and experimental metric rename.

Table 12's baseline-only counterfactual uses the true total lift with the model's own factual total: mean error is ≤.019 in seven of nine configurations, .033 for cRMSN and .049 for BD-AR, versus observed total errors .059–.208. These are configuration means. Signed terms can offset, so this is not an additive share of observed error.

Sharing the true total moves eight of nine means by ≤.012; BD-AR moves .117 → .201 because its baseline error offsets lift error. Timing can still affect the lift. For Moirai-small under strong confounding:

MetricFrozenFine-tuned mean
Pathwise total-effect error.256.135
Shared-total error.257.138 ± .061
Mean predicted log contrast (truth .186)−.002.078
Sign accuracy36%70%
Stepwise log-contrast MAE.206.243 ± .060
Peak-effect error, own predictions.311.797 ± .308

Tuned values average five runs; ± is sample SD. The total gain survives sharing the total, although stepwise and peak errors increase. Better mean contrast and signs can coexist with higher MAE, without establishing growing response variance. Tables 19–21 give all runs: the strong-confounding stepwise increase holds in 3/5 small and 2/3 base runs.

The day-by-day factual-weight control retains the confounded Chronos reversal in all three adaptations; Moirai-small improves in 5/5 pairs per cell (Table 25; General A). Estimated ratios can still contain model error.

W2 / Q2. AR conditioning and Corollary 1. Yes. BC-AR trains on preceding ground-truth targets; BD-AR masks a window's final target and conditions on true preceding values. Both feed back their own draws at inference. BD-OS masks the last 48 raw targets.

Original PA uses 10 OS prefixes (minimum 30, stride 48) or 461 AR prefixes (minimum 2, stride 1), with length bucketing. Both cover 461 distinct next-target positions; compute differs (App. A.5).

We narrowed Corollary 1 to fixed inputs available at the forecast origin (Remark 1). Complete teacher-forced conditionals determine a joint by the chain rule; fitting nine conditional quantiles does not guarantee joint calibration.

BC-AR has the best WQL (.0801), yet total coverage is .288 at nominal .80; historical BD-JOINT covers .669. These rows do not isolate the training objective. The separate origin-only BD-JOINT rerun reaches .810 (Table 22).

Historical normalization includes masked training labels, without held-out evaluation labels. The new origin-only matched comparison passes input checks and locked mean gates. General B/App. K report its wider intervals, seed 0 exception and exploratory shuffle. Teacher-forced AR remains outside the corollary.

W3. Reproducibility. App. A.5–A.6 now supplies the missing training, logging, adaptation and mechanism settings. Changed PA recipes remain separate (General B). The specifications support reimplementation; complete code release requires institutional approval and is not yet available, with no committed date.

If these changes resolve your concerns, we would appreciate reconsideration of your rating.

Thank you for the response. I appreciate the additional clarification and experiments that the authors provided. I have read the other reviewers' comments and the follow-up discussion. Overall, I still see this as a borderline paper and will keep my original rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment A: Aggregate effect error and a shared factual total

The reviewers identified a gap in our original argument: Eq. (4) cancels the base level in a stepwise log contrast, but we had extended that claim to an effect on total demand. We corrected the claim. We also renamed the experimental “aggregate CATE” metric pathwise total-effect error: it evaluates a realized held-out path, not a conditional expectation over possible future paths (§5; App. A.1).

1. Scope of cancellation. Eq. (4) is level-free for unfloored positive log contrasts. The implemented scorer floors positive levels at 1 and requires positive finite predictions in both arms. Floor-active comparisons need not be scale-free. Floor activation is unmeasured across historical cells in App. J; App. L measures it in the new exports. Pathwise total-effect error retains baseline forecast error and is normalized by the historical mean and eligible step count.

We retain both metrics because they answer different questions: the stepwise contrast checks proportional response, with the implementation limits above; the total-demand effect answers volume questions such as how an action changes stocking needs.

2. What the diagnostic controls. Write B,B^B,\hat B for true and predicted factual totals and L,L^L,\hat L for true and predicted intervention-to-factual lifts. Then

τ^−τpath=B(L^−L)+(B^−B)(L^−1).\hat\tau-\tau^{\mathrm{path}}=B(\hat L-L)+(\hat B-B)(\hat L-1).

Sharing the true total gives B(L^−1)B(\hat L-1) and error B∣L^−L∣B|\hat L-L|. This removes error in total magnitude. Timing still matters: even correct daily response multipliers can give the wrong lift when the forecast puts demand on the wrong days. Signed terms can offset, so their absolute values do not give an additive percentage attribution.

3. How much the baseline matters. Table 12 compares three original-recipe PA seeds, five runs for other trained rows and one frozen Chronos-2 run.

ConfigurationPathwise total-effect errorShared-total error
BC-AR.079.084
BD-AR.117.201
BD-AR+PA.116.108
BD-OS.121.132
BD-OS+PA.059.058
BD-JOINT.069.074
Chronos-2 frozen.208.213
DeepAR.099.093
cRMSN.061.058

Eight of nine means move by ≤.012. BD-AR's low factual total offsets an excessive lift, making its own-total error 42% smaller than its shared-total error.

Table 12 also asks a counterfactual question: with the true total lift but its own factual total, a model's mean error would be ≤.019 in seven of nine configurations; cRMSN gives .033 and BD-AR .049. These are configuration means, not bounds on every run or store, and are not additive components of observed error.

4. What remains of the fine-tuning finding. All nine mean adaptation directions agree under own-total and shared-total scoring (Table 13). Under strong confounding, Chronos-2 worsens (.213 → .505 shared), while both Moirai sizes improve.

The stronger full-profile control applies predicted daily intervention/factual ratios to true factual demand on each day (App. L.1, Table 25). Chronos improves RMSE 1174 → 781–801 but worsens full-profile error .216 → .503–.505 in all three confounded adaptations. Randomized Chronos and both Moirai-small cells improve. Predicted factual magnitudes or timing weights cannot alone explain this counterexample; estimated ratios can still contain model error. Table 27 gives all 28 runs. This locked-before-evaluation readout reuses the cohort and is exploratory.

5. Independent stepwise check. Tables 19–21 give every run. All improve factual RMSE. Below, ↓/↑ count lower/higher stepwise log-contrast MAE than the matching frozen reference; remaining runs move oppositely.

ModelRandomized λ=0Moderate λ=.5Strong λ=1
Chronos-2 full3/3 ↓2/3 ↑3/3 ↑
Moirai-small5/5 ↓4/5 ↓3/5 ↑
Moirai-base3/3 ↓3/3 ↓2/3 ↑

Confounded Chronos improves RMSE 1174 → 790 while mean stepwise log-contrast MAE worsens .1565 → .2810. This corroborates the finding with the floor/mask limits above; it does not remove aggregate timing dependence. Chronos runs vary series splits, not independently specified optimizer seeds.

6. What the target means. The mechanism is replayed on the held-out realized base path. We do not evaluate nondegenerate conditional CATE. Unfloored log contrasts coincide wherever defined because the base cancels; sharing a total does not change the estimand.

AuthorsOpenReview ↗

General comment B: Training clarification, corrections and additional evidence

The training questions prompted an implementation check and a narrower theoretical statement. The revised PDF marks changes in blue and includes a Summary of Revisions.

1. Corollary 1 and training conditioning. The corollary concerns per-position objectives with fixed inputs available at the forecast origin. Teacher-forced conditional losses are different: complete conditionals determine a joint by the chain rule, although fitting nine conditional quantiles does not guarantee joint calibration. Remark 1 and Table 1 now make this distinction.

BC-AR predicts the next value from preceding true targets. BD-AR masks a window's final target; BD-OS masks its final 48. Historical HSTU standardization includes those masked training labels, without using held-out evaluation labels. Table 1 and App. A.5 retain that disclosure.

We reran BD-JOINT/BD-DIAG at three paired seeds with origin-only statistics. Changing masked targets leaves the trainer's inputs unchanged in all 30 tested windows. WQL improves .1187 → .1021 while total coverage falls .810 → .244, with both orderings in every pair. All locked mean gates pass: total separation .566 ≥ .15; joint relative VS shuffle gap .0339 > three SE .0215, diagonal −.0003; per-step coverage gap .0407 ≤ .05. This fixes masked-target preprocessing in the new comparison, while leaving teacher-forced AR outside the corollary.

Tables 22–24 give every seed and functional. Joint intervals are about 40% wider; seed 0's per-step gap is .081, exceeding the individual .05 tolerance. Exploratory shuffling preserves empirical marginals: joint total coverage falls .810 → .322, diagonal .244 → .235. This supports dependence without an exclusive dependence/width percentage. Native outputs reproduce exactly; old and new runs remain separate.

2. Reproducibility. App. A.5–A.6 supplies the missing masks, prefix schedules, normalization, optimizer, sampling, adaptation, logging and calibration settings. These support reimplementation. An internal package reconstructs cached tables and the 28-run readout on CPU. Reviewer access requires institutional approval and anonymization; the complete pipeline is not yet available, and no release date is committed.

3. What the additional results show.

  • All five predictive functionals are reported for ten configurations (Tables 5–6). In the historical five-seed BD-JOINT/BD-DIAG pair, coverage gains extend to volatility (.787 versus .494) and lag-one autocorrelation (.837 versus .589); peak coverage barely changes. App. K adds the new comparison. Neither pattern implies that extremes are generally copula-insensitive.
  • Stored peak-effect errors broaden the interventional check. Under strong confounding, Moirai-small's total error improves while peak-effect error worsens (.311 → .797); Moirai-base shows the same split (.318 → .957). Peak is a realized-path diagnostic with no scalar shared-total decomposition.
  • On electricity, all three documented fine-tuning rules pass: every run improves factual RMSE, while shared-total error improves under randomized logs and worsens under confounding. The separate dependence ablation agrees in direction but fails its marginal-calibration rule (.830 versus .686 per-step coverage). We report that failed check in Appendix G.
  • App. L adds a 28-run readout with true daily factual weights and eleven prices. General A §4 gives the profile control; Tables 26–28 give pricing, constant-action baselines and every cost. Chronos has decision counterexamples; Moirai's directions vary by cost. The reused cohort, studied costs and strong constant baseline limit general utility claims.

4. PA recipe correction. Original PA uses 10 OS or 461 AR prefixes with length bucketing; added runs use 52 prefixes and different batching. We withdrew the pooled five-seed decoder grouping and retained the three matching PA seeds. The unaugmented five-seed BC-AR/BD-AR ordering still reverses between elasticity and bounded response at matched confounding and under shared-total scoring (Table 16). Both rows are autoregressive; this shows configuration rankings depend on the mechanism.

5. Readability. The rewritten introduction asks two questions; §5 explains each effect check. Sections 6.1–6.2 lead with the matched dependence and daily factual-weight controls. Tables 1 and 3 define configurations and metrics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.