- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that the forecast start, i.e., the transition from history to the first predicted step, plays a larger role in long-term forecasting accuracy than its position suggests. The motivation comes from an experiment in which the ground-truth first future value is appended to the input of a frozen model and the subsequent horizon is re-forecast (Fig. 1, App. C). The authors propose STEP, which builds an initial forecast from multi-order historical differences, learned circular shifts on feature segments, cross-variable attention, and cross-head ("subspace") attention, and then refines it with a shared linear map over the concatenated history and initial forecast. Training uses a transition-aware loss with exponentially decaying horizon weights and value and adjacent-change terms. STEP is evaluated as a standalone forecaster, as a module integrated into seven backbones, and as an inference-time provider of an improved forecast start for frozen backbones.
STEP achieves the best MSE at all horizons on Electricity, Traffic, and Weather in Table 1, and the 3-seed results show low variance. The appendix is unusually transparent: it includes a capacity-matched FFN control, paired block-bootstrap confidence intervals, a dose-response analysis of the start correction (Fig. 8), and a comparison against the untouched one-shot forecast that also reports negative results (Tables 16–17).
However, the forecast-start narrative that defines the paper is only weakly supported by the paper's own controlled results, and it does not explain where the accuracy gains come from. The inference-time enhancement has a very small effect against an honest baseline, and the backbone-integration results mix architecture, capacity, and loss changes. The method is largely a combination of known components. Several reported numbers also need clarification.
Critical strengths
- The history-to-future boundary as an object of study, with a clean intervention protocol: Instead of asking only how to encode the history, the paper asks how much a frozen forecaster depends on where its forecast begins, and tests this by supplying a corrected start a(q) = a + q(Y1 − a) and varying q, including the opposite direction (Fig. 8, App. C). This turns a vague intuition into a measurable dose-response question that can be asked of any direct forecaster. The protocol is useful beyond STEP, and it is also what lets a reader check how large the effect really is.
- Treating the initial forecast as a continuation of the history: STEP places the initial forecast next to the observed window and refines the joint sequence [X; Ỹ] (Eq. 12), so each forecast position can see the recent history levels and changes rather than only a latent summary. This is a simple, architecture-agnostic principle, and it is what allows the same block to be attached to seven different backbones (Table 2) and to supply a start to frozen models (Sec. 4.3).
Critical weaknesses
- The central claim is contradicted by the paper's own full-test evidence: The paper is framed around the claim that "long-range forecasting performance is closely related to how prediction begins" (Sec. 1), but its controlled measurements put this effect at a small fraction of the reported gains.
- The 44%–98% reductions in Fig. 1 are selected examples from frozen STEP variants ("in these examples," p. 1–2). On the full test sets, even a perfect ground-truth start reduces later MSE by only 1.30% (Electricity), 0.88% (Traffic), 1.57% (Weather), and 3.70% (Solar) (Fig. 8, App. C).
- At the same H = 96, STEP's margin over the best baseline is larger than this upper bound (Table 1: Electricity .127 vs. .134, Weather .148 vs. .156, about 5% each). So most of the improvement must come from something other than a better forecast start.
- What the method does differs from the mechanism the narrative describes: The narrative is that first-step error propagates over the horizon, but neither the diagnostic nor the model works through such propagation.
- The intervention appends a value to the input window (X⁺(a) = [x2, …, xL, a], Eq. 13). For a direct multi-output model, this shows that one more recent observation helps; it does not show that the model's own first-step output error drives later errors, since later outputs do not depend on that output.
- Inside STEP, the start is singled out only by the loss weights w_h = exp[−λ(h−1)] (Eq. 7); the refinement treats all of Ỹ alike. The evidence that the transition design specifically fixes the start (16.9% step-1 gain, Fig. 6(b)) comes from a control that also switches the loss to MSE (App. G.2), so refinement and objective are not separated. The components themselves (residual linear refinement, horizon-weighted MSE+MAE+change loss similar to TDAlign, difference embeddings, cross-variable attention) are largely known, so the forecast-start framing is what would make the contribution distinct.
- The two settings meant to show the idea's generality do not isolate the forecast-start effect:
- Inference-time enhancement: the main "Control" is worse than the untouched one-shot forecast (Table 15 vs. Table 17), and the Q4 subgroup is selected with the ground truth (s_n = MSE^Control − MSE^GT, Eq. 14; acknowledged in App. K). Against the one-shot forecast on the full test set, the gains range from −0.36% to +0.99%, with 4/28 regressions (Table 17).
- Backbone integration ("96.4% of settings"): integrated models are trained with the Eq. 7 objective (App. E.1), with no "backbone + Eq. 7 loss only" or capacity-matched control, although Table 4 suggests the loss effect is large (Solar MAE .199 → .271 without the STEP objective). For iTransformer and Client, the native encoder is replaced by STEP (Table 12).
Questions for authors
- Contribution of the forecast start: Given that a perfect ground-truth start reduces later MSE by only 1–4% (Fig. 8), what share of STEP's margin in Table 1 can be attributed to forecast-start improvement? Can you show, e.g., the correlation (controlling for window difficulty) between step-1 error and later-step error in trained models?
- Integration controls: Can you report (a) each backbone trained with the Eq. 7 loss only and (b) each backbone with a parameter-matched FFN block, plus seed variance and ETT results for +STEP?
- Inference-time enhancement without ground truth: Does the effect remain when windows are selected without ground truth, and is it better than simply using STEP's own forecast or a simple average of STEP and the backbone?
- Table 1 numbers and baselines: Which run is reported in Tables 1/10 (best seed, a separate run, or a mean)? Were baselines re-run under the same pipeline and test windows, and were they tuned comparably?
- Refinement vs. loss in Fig. 6(b): What is the step-1 reduction when the control keeps the Eq. 7 loss and only replaces refinement with an FFN? How do per-horizon error curves at H = 720 compare between the chosen λ and λ = 0?
3 comments
Response to Reviewer We15 (Part 2)
Q3. Without ground-truth-based selection, how does start replacement compare with full STEP and forecast averaging?
Inference-time enhancement does not use future ground truth to select samples. Q4 is only a post-hoc sensitivity group. Appendix Table 17 compares start replacement directly with the original one-shot forecast over the full test set: subsequent MSE improves in 24 of 28 settings (range: −0.36% to +0.99%).
We also compared four strategies over the full Electricity test set (L = H = 96), evaluating the original forecast positions 2–96 in every case:
Electricity: MSE at steps 2–96 (lower is better)
| Backbone | Original | Full STEP | 50/50 average | Start replacement |
|---|---|---|---|---|
| DLinear | 0.1960 | 0.1288 | 0.1429 | 0.1953 |
| iTransformer | 0.1485 | 0.1288 | 0.1323 | 0.1484 |
| Client | 0.1428 | 0.1288 | 0.1306 | 0.1426 |
| TQNet | 0.1352 | 0.1288 | 0.1277 | 0.1351 |
| GTR | 0.1353 | 0.1288 | 0.1267 | 0.1351 |
Full STEP or forecast averaging generally gives lower error than replacing only the first step. We use start replacement to test a frozen backbone's response to a local intervention and the feasibility of a targeted correction without changing its weights, not as a claim of optimal inference accuracy.
Q4. Which runs appear in Tables 1/10? Were baselines retrained under the same pipeline?
Tables 1/10 report the single-run test results used in the submission, not three-seed means, and they were not systematically selected as the best seed. For example, Traffic (H = 192) has MSE 0.409 in Table 1, while the three-run mean in Appendix Table 11 is 0.4083 ± 0.0006. Thus, at least one run was better than the main-table result. Appendix Table 11 reports the full three-run statistics separately.
For baselines, we mainly follow published benchmark results rather than rerunning and retuning every method in a single pipeline. The comparisons use the standard datasets, input lengths, prediction horizons, and evaluation metrics.
Q5. What are the separate effects of Refinement and time-weighted Loss at H = 720?
The Transition Design comparison in Figure 6(b) changes both Refinement and the training objective. Its 16.9% step-1 and 6.2% later-step MSE reductions therefore cannot be attributed to Refinement alone. We added controlled comparisons on Electricity (L = 96, H = 720):
H = 720: Relative MSE reductions under separate controls
| Controlled comparison | Step 1 | Steps 2–720 |
|---|---|---|
| Refinement vs. matched FFN (same Eq. (7) Loss) | 2.93% [1.88%, 3.71%] | 0.92% [−0.22%, 1.93%] |
| λ = 0.01 vs. λ = 0 (same STEP architecture) | 9.29% | −0.14% |
The brackets in the first row are 95% confidence intervals; the second row gives verified point estimates from a matched-seed comparison. λ = 0.01 is the Electricity setting in Table 9. With the loss fixed, Refinement clearly improves step 1, while the later-step confidence interval includes zero. With the architecture fixed, time weighting primarily affects early predictions.
We further divided the controlled λ = 0.01 versus λ = 0 comparison into forecast ranges to examine how its effect changes across the horizon:
Horizon-wise MSE reductions: λ = 0.01 vs. λ = 0
| Forecast positions | MSE reduction | 95% CI |
|---|---|---|
| Steps 1–96 | +3.30% | [2.51%, 4.08%] |
| Steps 97–360 | −1.86% | [−3.78%, 0.22%] |
| Steps 361–720 | +0.65% | [−0.74%, 1.99%] |
The benefit is concentrated in early predictions, while changes at later horizons remain uncertain. This is consistent with the design's emphasis on the forecast start and early forecast positions.
Reponse to Authors
Thank you for the response and the new controls. The 3-seed ETT integration results with seven backbones, and the clarifications on Tables 1/10 and the Q4 subgroup, address my questions on reporting and generality. I also agree that the frozen-model intervention (Fig. 8) is not a strict upper bound for a model trained end to end, so my "upper bound" wording was too strong.
However, the new controlled results make the forecast-start narrative harder to support:
- Q5 (H=720): With the loss fixed, refinement improves step 1 by 2.93% but steps 2-720 by only 0.92%, with a CI that includes zero. Time weighting improves step 1 by 9.29% and changes steps 2-720 by -0.14%. In the trained model, a better start does not carry over to later steps, which is the opposite of the claim that "long-range forecasting performance is closely related to how prediction begins".
- Q1: The grouped analysis is correlational and only uses samples whose step-1 error improved. Even the group with almost no step-1 change (median 0.0001) has a 2.84% later reduction, so a large part of the later gain does not depend on the start.
- Q3: Start replacement improves steps 2-96 by at most about 0.4% (e.g., DLinear 0.1960 -> 0.1953), while full STEP or a simple 50/50 average gives much larger gains. This supports reading the inference-time results as a feasibility check rather than evidence for the start mechanism.
- Q2: On GTR, +STEP improves over loss-only by 0.95% and over the matched FFN by 0.79%. On Electricity, where strong baselines are very close (TQNet 0.1352 vs. GTR 0.1353 in your Q3 table), this is a meaningful gain, so the module itself does contribute. However, these controls cover one dataset and two backbones, the Client matched FFN (0.1543) is worse than loss-only (0.1422), which suggests this control is not well tuned, and the ETT runs have no loss-only control.
- The current PDF does not include any of these results, and the baselines are taken from published results rather than re-run.
STEP looks like a useful plug-in module, but the evidence points to the loss and the full refinement, not the forecast start, as the source of the gains. I will keep my score and suggest reframing the paper around what these controls show.
Response to Reviewer We15 (Part 1)
We thank the reviewer for the detailed and constructive feedback, particularly for recognizing the value of studying the history-to-future boundary through controlled interventions.
Q1. How much of STEP's gain comes from the forecast start? Are first-step and later improvements related?
The 1%–4% gains in Figure 8 come from changing only the supplied start of a frozen predictor. In contrast, Table 1 evaluates the full STEP model, which is designed around the forecast start and trained end to end. Training changes the model parameters and improves its overall forecasting capability. Therefore, the input-intervention gain is not a ceiling on STEP's gain, and subtracting the two cannot identify the causal contribution of the first step.
We also performed a paired window–variable analysis on Electricity (L = H = 96), comparing STEP with w/o all modules in Table 4. Among samples whose step-1 MSE improved, we formed ten groups by the magnitude of the step-1 improvement. To reduce differences in sample scale and improvement magnitude, we aggregated the logarithms of per-sample relative RMSE improvement factors. The reduction in RMSE at steps 2–96 increased from 2.84% to 7.92% across the groups. This conditional analysis shows that samples with larger first-step improvements generally also have larger later-error reductions.
Electricity: Conditional groups by step-1 improvement
Q2. What do loss-only, parameter-matched FFN, multi-seed, and ETT integration controls show?
We added Eq. (7) loss-only and parameter-matched FFN controls. The following are two completed, strictly matched comparisons on Electricity (L = H = 96):
Electricity: MSE with loss-only, matched FFN, and +STEP
For Client and GTR, +STEP reduces MSE relative to loss-only by 5.43% and 0.95%, respectively, and relative to the matched FFN by 12.85% and 0.79% (calculated from unrounded values). These matched comparisons show that neither the loss change nor extra parameters alone explain the gains.
We further evaluated +STEP integration with seven backbones on ETTh1 and ETTh2 using three paired random seeds for each backbone (L = H = 96). The table reports mean MSE ± SD, from the original backbone to +STEP.
ETT integration: Three-seed MSE (original → +STEP; mean ± SD)
Using unrounded values, MSE and MAE improve in 20/21 and 21/21 paired ETTh1 runs, respectively, and in 21/21 paired ETTh2 runs for both metrics.