- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes EditCast, a post-training method that revises the outputs of a frozen time-series forecaster without updating its parameters. Given a history and the backbone's reference forecast, a lightweight MLP policy (~33K parameters, shared across channels) selects, per channel, one of 51 predefined "semantic" edits: identity, phase shift, time warp, global amplitude scaling, linear trend, level shift, and Gaussian-windowed local amplitude/phase changes. The authors frame this as one-step, full-information contextual RL. During training, the realized future scores every candidate edit with a reward combining full-horizon MSE reduction and a weighted structural term built from four components (empty-channel false alarms, a first-order surrogate of micro event-F1, dominant-frequency amplitude error, and peak-timing error). Because the action set is finite and all rewards are observed, the policy is trained by minimizing the exact negative expected reward via enumeration rather than sampled policy gradients. Five structural weights λ ∈ {0, 0.05, 0.1, 0.25, 0.5} are trained. Validation selects among them using MSE/MAE non-degradation gates and the count of improved structural metrics, falling back to the unedited forecast when no candidate qualifies.
Experiments cover 308 checkpoints across eight standard long-horizon benchmarks (ETTh1/h2/m1/m2, ECL, Traffic, Weather, Solar), four backbones (PatchTST, iTransformer, MixLinear, PaiFilter), and horizons H ∈ {96, 192, 336, 720}. Validation-selected EditCast reduces aggregate MSE by 1.13% and MAE by 0.80% relative to the frozen checkpoints. The number of "joint wins" (no accuracy loss, no degradation on any of eight structural metrics, at least one improvement) rises from 31 with accuracy-only rewards to 59 with a fixed λ = 0.1, and to 64 with validation selection. On a 12-checkpoint subset (ETTh1/ETTm1, H = 96), the paper reports controls against norm-matched random edit banks, edit-family ablations, sampled policy gradient, affine calibration, a continuous MLP correction head, and backbone fine-tuning. It also compares learning objectives (expected reward, best-action cross-entropy, Q regression, soft targets) and reports an oracle analysis: a future-aware oracle over the same edit set reduces MSE by 13.89%, captures 8.27% of that gain, i.e. ≈1.15% absolute MSE reduction. The authors conclude that outcome-guided semantic editing is a viable post-training stage for frozen forecasters.
Critical strengths
-
Practically important problem and a natural interface for it. Revising the output of a frozen, already-deployed forecaster is a real need in production forecasting pipelines, where retraining is costly and audited models must stay fixed. A finite bank of named, interpretable edits (level shift, amplitude scaling, peak retiming, local adjustments) closely mirrors how practitioners already apply manual overrides, e.g., in demand planning. Framing correction as choosing among such edits, rather than learning an opaque additive residual, is a sensible design and could support human-in-the-loop or auditable override workflows. The paper does not evaluate this angle itself.
-
Clean and correct formulation of the learning problem. The well-known decomposition of each edit's MSE gain into an alignment term and a cost term (Eq. 3) is applied correctly here. It sums exactly to the total forecast MSE reduction across channels and makes clear why a locally plausible edit can increase full-horizon error. The factorization argument (App. A.5), which reduces the joint expectation from A^C to C·A terms, is also correct. Exact enumeration over a small finite action set is a reasonable choice when every action's outcome is observable during training.
-
Lightweight and backbone-agnostic. The method adds only ~33K parameters and never backpropagates through the backbone. Its reported training cost is lower than backbone fine-tuning (212 s vs. 460 s on the control cohort), though that timing excludes candidate caching, while the fine-tuning time includes data loading. It applies unchanged to transformer, linear, and filter-based forecasters, which makes it plausible as a drop-in post-processing stage.
-
Informative controls and diagnostics. On the control cohort, several experiments are well designed and useful beyond the paper's own claims:
- The norm-matched random-direction bank indicates that the direction of semantic edits, not just their magnitude, drives the gain.
- The oracle analysis separates the edit bank's capacity from the policy's ability to predict which edit helps. The deployable accuracy-only policy captures only 8.27% of the oracle's 13.89% MSE reduction (≈1.15 points), which identifies predictability as the bottleneck.
- The per-family harmful-edit rates (Fig. 5) and the qualitative cases (Figs. 4 and 10) make the method's behavior inspectable.
-
Transparent reporting. The paper uses paired comparisons on identical checkpoints and test windows and reports pooled within-setting seed dispersion. It includes per-setting results for all 65 configurations (Table 14) and discloses outcomes unfavorable to the main method: Q regression's larger MSE reductions, the stronger no-timing-edits ablation, and settings that degrade. Implementation details, metric definitions, and selection rules are specified in enough detail to support reimplementation.
Critical weaknesses
-
The "RL" framing is mostly relabeling. The paper acknowledges that its setting is the "full-information specialization" of one-step contextual RL (citing Langford & Zhang, 2008). In that regime, each input gets one decision, every action's outcome is observed, and the expectation is computed exactly. That is full-information cost-sensitive multiclass classification. There is no exploration, no sequential decision, and no credit assignment over time. The analogy to LLM post-training (RLHF, DPO, DeepSeek-R1) does no technical work.
Two bodies of relevant literature are not discussed:
- Cost-sensitive classification and its reductions (e.g., Beygelzimer, Langford et al.).
- Judgmental forecast adjustment, which has long treated forecast generation and forecast revision as separate decisions (e.g., Blattberg & Hoch, 1990; Fildes et al., 2009).
The claimed advantage of exact enumeration is also overstated. Sampled policy gradient with K = 16 matches enumeration (−0.85% vs. −0.83% MSE, Table 3), yet the text describes this as "preserving the exact-enumeration advantage at the largest sampling budget." Enumerating 51 actions is cheap, so this is a reasonable implementation choice, not a contribution.
-
The claimed distinction between expected-reward optimization and Q regression is not principled. With a sufficiently expressive policy class, both objectives yield the same deterministic decision in the population limit: the argmax of E[r | s, a]. Any difference between them is therefore a finite-sample or optimization effect, not a conceptual one. Meanwhile, Q regression gives substantially larger MSE reductions: 1.46% vs. 1.15% at λ = 0, and 1.42% vs. 0.62% at λ = 0.1 (Table 12). The paper prefers expected-reward optimization because it degrades fewer structural endpoints (2.11 vs. 3.50 at λ = 0.1). That trade-off is measured on 12 checkpoints without statistical testing and is not theoretically motivated, so the choice of main objective is weakly justified.
-
No analysis of when correction is possible. An edit helps only if the history and the reference forecast predict the sign and form of the backbone's error. The paper's own diagnostics show this predictability is the bottleneck:
- The deployable policy captures only 8.27% of the oracle's 13.89% MSE reduction (≈1.15 points).
- The pooled harmful rate of selected edits is 0.46 (Fig. 5): 46% of selected edits increase channel MSE on their own window, although some of these may be intended structural trade-offs under λ > 0.
Yet the paper does not analyze predictability. It gives no conditions under which a frozen model's errors are correctable from its own inputs and no relation to backbone underfitting. It also offers no guarantee against degradation beyond the validation gate, which with ε = 10⁻⁹ only ensures the edited forecast is no worse on validation.
-
The edit bank is hand-designed and inconsistently scaled across horizons. Some edits scale with the horizon: the local envelope width (H/8) and the time-warp magnitude (a·H). Others are fixed in absolute steps: global and local phase shifts of ±{1, 2, 4} and ±{1, 2} steps for all H ∈ {96, …, 720}. At H = 720, phase edits therefore become nearly negligible while warps remain large, so the meaning of an action family changes with H. This undermines the interpretability claim. The amplitude, trend, and level grids are chosen without justification or sensitivity analysis, and there is no study of bank size or granularity.
-
The structural reward is heuristic and partly circular. The reward combines three ad-hoc elements:
- A first-order linearization of micro-F1, which the paper acknowledges does not equal the actual change.
- Fixed component weights w = (1, ½, ¼, ¼).
- A scale factor R, chosen to equalize the RMS of the accuracy and structural rewards on each setting's training split.
Because of R, λ has no consistent meaning across settings. The evaluation also overlaps heavily with the reward:
- Four of the eight evaluated structural endpoints are reward components.
- Two more (macro and positive-channel event F1) are closely related to the optimized micro-F1.
- λ is itself selected by counting improved validation structural endpoints.
The reported structural gains are therefore largely gains on the optimized quantities. The "joint win" criterion is introduced in this paper, and what counts as a tie depends on both the 10⁻⁶ tolerance and the six-decimal rounding of metrics before comparison (Appendix A.6).
-
Small effect sizes without statistical testing. The headline gains are 1.13% MSE and 0.80% MAE (≈3.6 × 10⁻³ absolute MSE). No significance tests or confidence intervals are reported. The only dispersion measure (s∆) captures variation across checkpoint seeds, not test-window sampling variability. Because test windows are generated with stride one and overlap heavily, the effective sample size is far smaller than the window count; a block bootstrap over test windows would be appropriate. At the setting level, the seed SD is often comparable to or larger than the mean change (Table 14; e.g., ETTh1/PaiFilter/H=96: +0.0025 ± 0.0049), so many per-setting gains cannot be distinguished from zero.
Seven settings degrade in test MSE: ETTh1/PatchTST/H=336 (+0.0130), ETTh1/PatchTST/H=96, ETTh1/iTransformer/H=96, ETTh1/PaiFilter/H=96, Solar/PaiFilter/H=192, Solar/PaiFilter/H=336, and Solar/PatchTST/H=720. This happens despite the validation gate guaranteeing no MSE/MAE degradation on validation, which suggests the selection procedure overfits to the validation split. Others degrade in MAE while MSE improves (e.g., two Solar/iTransformer settings, Solar/PaiFilter/H=96, ECL/iTransformer/H=96). One setting (Solar/PaiFilter/H=720, 5 seeds) shows exactly zero change because selection falls back to the frozen model.
-
The motivating mechanism is barely used. The paper is motivated by mistimed peaks (Fig. 1). However, on the 12-checkpoint control cohort:
- Global phase and warp edits account for only 293 of 596,820 selected actions (Fig. 5). They have negative mean reward and harmful-edit rates of 86% and 73%. Including local phase, timing edits make up about 0.6% of selections.
- Removing all timing edits does not hurt and nominally improves MSE/MAE (−1.06%/−0.49% vs. −0.83%/−0.43%, Table 3), as well as structural endpoint counts (5.50 vs. 4.92).
- Policies predominantly select global level and amplitude edits, ~89% of actions in Fig. 5. Local amplitude, the third most-used family (43,635 selections), also has negative mean MSE reward and a harmful-edit rate of 0.57; positive mean rewards come only from global level, amplitude, and trend (plus rarely used local phase). In effect, the method performs per-window, affine-style recalibration.
What the method learns in practice therefore differs substantially from the motivating narrative. The "coverage" argument for keeping timing edits (Sec. 4.3) cites peak-timing improvements in 166/308 cells, but these are not attributed to timing edits and could come from level or amplitude changes. A natural baseline is missing: the global affine calibration baseline is not context-dependent, and the continuous MLP head is unconstrained. A per-window regressor restricted to level and scale (two outputs per channel) on the same features would test whether the semantic bank adds anything beyond context-dependent affine recalibration.
Minor inconsistency: the text says the no-timing ablation removes "the 14 phase, warp, and local-phase edits," but Table 5 gives 6 + 4 + 12 = 22. The parameter counts in Table 11 confirm 22 (33,075 − 30,237 = 22 × 129).
-
Missing key baselines.
- Closest post-hoc correction methods: δ-Adapter (Liang et al., 2026) is cited as the closest related work but not compared against. TAFAS (Kim et al., 2025) is also omitted, although it assumes partially observed test targets, so a comparison would need care.
- Simple residual correctors: e.g., gradient-boosted trees or a linear model on residuals using the same features.
- Foundation models: none are evaluated (Chronos, TimesFM, Moirai), although frozen pretrained models are the most natural target for output-side post-training.
- Structure-aware training baselines: Soft-DTW (+35.1% MSE) and FreDF-style training (+145.5% MSE, Table 10) use a single hyperparameter (γ = 1, α = 1) on only 8 cells. The frequency loss is implemented as "MSE + α·MSE(F)" rather than FreDF's convex combination of time- and frequency-domain losses. Without FFT normalization, the frequency term's magnitude scales with H by Parseval's theorem, so at α = 1 it likely dominates the time-domain loss, which may explain the large degradation. These results are inconsistent with the original papers and should not serve as evidence that training-time structural objectives are inferior.
-
Selective and unexplained coverage. The 308 checkpoints cover only 65 of the 128 possible dataset × backbone × horizon combinations, with 1–10 seeds per setting. For example, PaiFilter is absent on ECL, Weather, and Traffic, and Traffic is evaluated only at H = 96. Selection of the main cohort is not explained beyond "available seeds vary by setting" (Sec. 4.1); Appendix B.6 attributes the choice of its long-horizon cells to "forecast availability." All mechanism and objective controls use 12 checkpoints from two ETT datasets at H = 96; Appendix B.6 adds only two H = 720 cells. Conclusions about edit families, learning rules, and enumeration may therefore not generalize to longer horizons or other domains.
-
Limited evidence for the claim to be a general post-training stage. Evaluation uses only standard long-horizon benchmarks, with errors computed in standardized space. There are no demand or retail datasets (e.g., M5, Favorita), no intermittent or hierarchical series, no asymmetric or business-cost metrics, and no probabilistic forecasts. These are the settings where output-side correction and overrides matter most in practice. The interpretability of named edits is asserted but never evaluated, e.g., through agreement with human overrides, override logs, or a user study.
Questions for authors
I am willing to raise my score if the authors adequately address the questions below, particularly Questions 1, 3, 7, and 8.
-
Can you report confidence intervals or significance tests for the headline MSE/MAE changes that account for test-window variability? Because test windows overlap with stride one, a block bootstrap over windows, paired with the frozen baseline, would be appropriate. How many of the 65 settings show a significant improvement, and how many a significant degradation?
-
The validation gate ensures no MSE/MAE degradation on validation for each checkpoint. Yet seven settings show a mean degradation in test MSE (Table 14; e.g., ETTh1/PatchTST/H=336, +0.0130). Selection searches over up to 120 candidates (24 epochs × 5 λ values), taking the minimum validation MSE within each λ and then ranking λ values by the number of improved validation structural endpoints, with a gate tolerance of only ε = 10⁻⁹. Is validation overfitting the explanation? Does performance change if selection uses a separate held-out portion of the training split, or a smaller candidate set? How many individual checkpoints degrade on test, including those within settings that improve on average?
-
On the control cohort, global level and amplitude edits account for ~89% of selections and timing edits for ~0.6% (Fig. 5). How does EditCast compare with a context-dependent regressor restricted to per-channel level and scale (two outputs per channel) using the same features? If the gap is small, what does the semantic bank contribute beyond per-window affine recalibration?
-
On the control cohort, removing timing edits does not hurt and nominally improves MSE, MAE, and structural counts (Table 3). Can you attribute the reported peak-timing improvements (166/308 cells) to specific edit families? How often are timing edits selected across the full 308 cells, and what is their realized effect there? Please also clarify the number of timing edits: 14 in the text vs. 22 implied by Table 5 and the parameter counts in Table 11.
-
With a sufficiently expressive policy class, expected-reward optimization and Q regression yield the same greedy decision in the population limit. Why is expected reward the main objective when Q regression gives larger MSE reductions (1.42% vs. 0.62% at λ = 0.1, Table 12)? If Q regression is run on all 308 cells with the same λ-selection procedure, does its disadvantage on structural endpoints persist?
-
The deployable policy captures only 8.27% of the oracle's MSE reduction, and 46% of selected edits increase channel MSE (Fig. 5). Can you characterize when a frozen model's errors are correctable from its own inputs? For example, does the gain correlate with backbone fit (e.g., training error or the train–validation gap), residual autocorrelation, or the oracle's per-setting headroom? The identity action already allows the policy to abstain: how often is it selected, and would a calibrated abstention rule (e.g., editing only when the predicted gain from Q regression exceeds a threshold) reduce the harmful-edit rate?
-
How does EditCast compare with δ-Adapter (Liang et al., 2026) under the same frozen checkpoints and splits? How does it compare with simple residual correctors, e.g., gradient-boosted trees or a linear model predicting residuals from the same features? Where feasible, a comparison with TAFAS under an adapted protocol would also help.
-
Does the method help frozen pretrained forecasters such as Chronos, TimesFM, or Moirai? These are the most natural targets for output-side post-training, since users typically do not update their parameters.
-
The Soft-DTW and frequency-domain baselines (Table 10) degrade MSE by 35% and 145%. Were their hyperparameters (γ, α) tuned, and is the frequency loss normalized by H? Would FreDF's original convex combination of time- and frequency-domain losses, with tuned α, change the conclusion?
-
Global and local phase shifts are fixed in absolute steps (±{1, 2, 4} and ±{1, 2}) for all horizons, while warp magnitude and envelope width scale with H. How was the bank chosen? How sensitive are the results to its size, its granularity, and how it scales with horizon (e.g., phase shifts proportional to H)?
-
Six of the eight evaluated structural endpoints are reward components or closely related to them, and λ is selected by counting improved structural endpoints. Per-endpoint counts are reported (Table 7, Fig. 12), but can you report joint-win counts for the fixed λ = 0.1 and validation-selected variants restricted to endpoints that are not optimized, such as dominant-frequency phase and cross-channel discrepancy? How sensitive are the joint-win counts to the 10⁻⁶ tolerance and to six-decimal rounding?
-
How were the 65 settings and their seed counts (1–10) chosen from the 128 possible combinations? Were any settings run and then excluded? Can the mechanism and objective controls (Tables 3 and 12) be extended beyond ETTh1/ETTm1 at H = 96?
-
The named edits closely resemble the manual overrides used in operational forecasting, e.g., in demand planning. Have you evaluated the method on demand or retail data (e.g., M5, Favorita), with intermittent or hierarchical series, or under asymmetric cost metrics? Is there any evidence that practitioners find the selected edits interpretable, for example agreement with logged human overrides?
Dear Reviewers,
Please read authors' rebuttals and confirm if your concerns are addressed well (if you have not done so). You can double check if all concern in your reviews are addressed or not, which will be the key things we need to discuss later.
Best regards,
Your AC