- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies delayed credit assignment in long-horizon autoregressive forecasting. The main observation is that distant BPTT gradients can become large even when the corresponding learning signal is unreliable.
The authors propose Internal-DW, a backward-only modification for residual blocks. It assigns separate Wiener-style weights to the identity and nonlinear gradient routes while keeping the forward rollout and all horizon losses unchanged. The weights are estimated from route-level second moments together with a noise model.
The paper first studies the mechanism in a controlled system where gradient signal and innovation can be separated, and then evaluates the method on a collection of sequence, neural, fluid, and weather forecasting tasks.
Critical strengths
- The controlled known-SNR study is a strong part of the paper.
The authors directly construct a setting where predictable gradient signal and innovation can be separated. This allows them to show that distant gradient magnitude can increase while gradient SNR decreases, rather than inferring this indirectly from forecasting error. The misplaced-gain and oracle controls are also useful for showing that the location of the weighting matters.
- The backward-only formulation is an interesting design choice.
Internal-DW leaves the forward model and rollout unchanged and modifies only internal backward routes. This separates it conceptually from gradient clipping, rollout truncation, and forward Jacobian regularization. The paper also reports several failure cases rather than claiming that the method is universally beneficial.
Critical weaknesses
- The claimed applicability regime is not independently validated.
A central empirical conclusion is that Internal-DW works in "history-dominated, weak-drive" regimes. However, the predictive-regime scores used to define this grouping are themselves evaluated on the test split. The probe hyperparameters are selected on validation, but the final probes are refit on train+validation and then evaluated on test, after which the resulting testbed grouping is used to interpret the forecasting results on those same testbeds.
This makes the regime analysis descriptive rather than predictive. There is no fixed threshold or independently held-out set of datasets showing that the regime score can predict whether Internal-DW will help before observing the test result.
This matters because the overall results are mixed: the method improves some testbeds, is roughly neutral on others, and clearly hurts NARMA and movie fMRI. I do not think regrouping these outcomes using a test-derived diagnostic is sufficient evidence that the paper has identified when the method should be used.
- The practical estimator is much less principled than the oracle formulation suggests, and the experimental comparison does not sufficiently isolate this issue.
The Wiener result assumes access to a decomposition of route variation into predictable signal and innovation, but the paper itself shows that this decomposition is not identifiable from the observed route law. In practice, the key quantity is supplied by a noise sampler whose assumptions differ substantially across datasets: centered residual templates, factor models, subject-specific AR models, and spatial spectral models.
Therefore, much of the practical behavior of Internal-DW is determined by a dataset-specific modeling choice that is outside the Wiener derivation. The paper already shows that sampler mismatch can reverse the benefit of the method.
There is also an additional mismatch between theory and implementation: the route moments are calibrated using a quadratic probe even when the actual training objective is different, and the paper explicitly notes that transfer from the probe loss to the task loss is only an approximation.
I would like to see a much more systematic sampler ablation and stronger direct baselines. The current baseline suite mainly contains full BPTT, clipping, Jacobian regularization, TBPTT, and static damping, while several more directly related recent long-horizon credit-assignment or temporal-supervision methods discussed in Related Work are not evaluated.
Questions for authors
- Can the predictive-regime classification be constructed using training and validation data only, with the grouping rule fixed before any test evaluation?
This would materially affect my assessment. In the current version, the main "history-dominated, weak-drive" explanation is based on regime scores evaluated on the test split.
- How sensitive are the results to the noise sampler itself?
For datasets where both are feasible, I would like to see Generic and Prior samplers evaluated side by side, as well as deliberately simplified or misspecified samplers. Since the sampler determines the estimated noise moment, this seems necessary to separate the contribution of the routing rule from the domain-specific noise model.
- The calibration moments are computed from a quadratic probe, while the actual parameter update can use a different task loss. How sensitive are the learned gains to the choice of calibration probe?
Could the authors calibrate using the actual task loss, or otherwise show that the same gains are obtained under different reasonable probe losses?
-
Why are more direct recent long-horizon credit-assignment / temporal-supervision methods discussed in the related work, such as STLW, HERO, or COLA, not included as experimental baselines?
-
The main positive application results use only three matched seeds, and the reported paired 95% intervals for ETTm1, ETTm2, and shear flow include zero. Are these improvements stable with additional seeds?
-
The "extended optimal training horizon" result is based on quadratic fits over a small grid of training horizons. Does the same conclusion hold if one compares the best actually observed training horizon, without fitting a quadratic curve?
5 comments
Response to Weakness 2 / Questions 2–3: Sampler and calibration sensitivity
Weakness 2 / Question 2 — Sampler assumptions
Reviewer concern (summarized): The noise model is not identified by the Wiener derivation. How sensitive is forecasting to Generic/Prior choices and deliberately misspecified samplers?
Response. The practical estimator depends on sampler assumptions. The oracle formula gives gains conditional on the route moments. Observed moments alone do not identify the innovation law; the practical sampler supplies that assumption. We therefore test both sampler choice and a controlled mismatch.
1. Sampler choice. We compare five matched seeds (0–4), with training/evaluation horizons , on MG/shear and , on ETTm1/2. Full BPTT and original DW runs are reused. Test relative- is mean ± sample SD; lower is better. Displayed values are rounded; percentages use unrounded results.
| Method | MG | ETTm1 | ETTm2 | Shear |
|---|---|---|---|---|
| Full BPTT | 1.053 ± 0.003 | 0.944 ± 0.019 | 0.859 ± 0.070 | 0.151 ± 0.031 |
| DW-Generic | 0.999 ± 0.007 | 0.881 ± 0.018 | 0.770 ± 0.048 | 0.130 ± 0.005 |
| DW-Prior | — | 0.920 ± 0.039 | 0.827 ± 0.049 | 0.137 ± 0.020 |
| Zero gains | 1.005 ± 0.007 | 0.889 ± 0.023 | 0.768 ± 0.064 | 0.293 ± 0.009 |
No Prior was evaluated on MG. On ETT, Prior uses a calendar-VARX model fitted on training data, with calibration frequency and minimum probe count held fixed. Generic wins all five pairs on both ETT datasets. On shear, Generic uses a residual bootstrap and Prior a spatial-spectrum model: Generic has 5.40% lower mean error, mainly driven by seed 1, although Prior wins 3/5 pairs. Domain structure does not consistently improve forecasting.
2. Simulated sampler failure: zero-gain control. We manually fix throughout training to mimic extreme attenuation from severe noise overestimation; an actual sampler did not produce these zeros. Compared with Generic, zero gains are worse in all five MG and shear pairs. On ETTm2, they reduce mean error by 0.27% (3/5 wins); on ETTm1, they increase it by 0.85% (2/5 wins). Adaptive routing does not universally beat closed routes.
3. Controlled sampler misspecification. In the Known-SNR task
all models observe the same future inputs. Conditional sampling holds them fixed and resamples innovations; marginal sampling incorrectly resamples the inputs as noise. At , mean test relative- over horizons 1–48 is 0.539 for conditional DW, 0.549 for marginal DW, and 0.559 for Full BPTT. Conditional DW wins all four seed pairs, with a mean paired reduction of 1.81%. This isolates the effect of one incorrect sampler assumption. Appendix E.4, Table 6 gives sampler protocols and seed-level results; E.4 also details the controlled mismatch.
Weakness 2 / Question 3 — Calibration-probe choice
Reviewer concern (summarized): Do task-loss and quadratic calibration give similar gains?
Response. We agree that transfer between probe losses should be checked. Relative- noise calibration additionally requires the conditional target mean; the quadratic construction cancels this mean between paired futures (Appendix C.5, Table 2). We compare probes in Known-SNR, where the mean is exact, using the same frozen seed-0 checkpoint, sampled futures, and bounded solver across four layers and 32 steps.
Mean absolute gain differences are 0.065 for and 0.066 for . Within-layer Spearman correlations are 0.980–0.993 for gains and 0.986–0.992 for trace reliability. Relative- generally gives lower gains, but the main rankings agree (Fig. 10).
From the same checkpoint, two copies train for three additional epochs with their respective fixed gain schedules, the same relative- task loss, batch order, learning rate, and fresh optimizer state. Calibration samples are not reused for validation.
| Epoch | Validation: quadratic probe | Validation: relative- probe | Median gradient angle | Median norm ratio |
|---|---|---|---|---|
| 1 | 1.030564 | 1.029395 | 9.29° | 1.095 |
| 2 | 1.057403 | 1.055297 | 15.46° | 1.101 |
| 3 | 1.032472 | 1.034275 | 15.11° | 1.059 |
Norm ratios are relative--probe divided by quadratic-probe gradient norms. Validation losses differ by at most 0.20% at these endpoints (Table 3). Later gradient comparisons also reflect divergence between trained models. This supports similar short-run behavior in this controlled case, not application probe equivalence or task-gradient optimality.
On MG, quadratic-calibrated DW also reduces mean relative- validation loss by 6.56% and wins all three matched runs (C.5, Table 4). This is complementary application-level evidence, not a direct comparison of probes.
Response to Weakness 2 / Questions 4–6: Baselines, robustness, and training horizons
Weakness 2 / Question 4 — Related baselines
Reviewer concern (summarized): Why are STLW, HERO, and COLA discussed but not evaluated as baselines?
Response. We appreciate the suggestion to compare with these approaches. Direct comparisons would help clarify their relative strengths. Our main comparison holds the forward horizon and training objective fixed to isolate the effect of changing internal backward routing. STLW gradually increases later-horizon loss weights (Coker et al., 2026); HERO adds relative supervision against a detached failed-rollout reference (Zhang et al., 2026). They change how forecast errors enter the objective, whereas DW changes internal gradient transport. On a residual backbone, either objective could potentially be retained while applying DW, with calibration adapted to its learning signals.
COLA uses dynamical criticality for online local credit assignment approximating global error propagation (Wang et al., 2026). It changes the learning rule itself; extending reliability weighting would require defining and calibrating the corresponding local update contributions.
These methods address different aspects of training and are potentially complementary to Internal-DW. Our paper focuses on reliability-weighted internal gradient transport and does not claim superiority over alternative training objectives or credit-assignment methods. Head-to-head comparisons with these methods are therefore not required to establish the paper’s specific claims.
Nevertheless, to address the reviewer’s question, we conducted an additional practical comparison with STLW on MG, ETTm1, ETTm2, and shear flow using three seeds per dataset. The results are shown below.
Test relative- error (mean ± sample SD across seeds 0–2; lower is better).
| Dataset | Full BPTT | STLW | Internal-DW |
|---|---|---|---|
| MG | 1.0541 ± 0.0044 | 1.0589 ± 0.0287 | 0.9997 ± 0.0096 |
| ETTm1 | 0.9422 ± 0.0252 | 0.9275 ± 0.0122 | 0.8786 ± 0.0176 |
| ETTm2 | 0.8684 ± 0.0913 | 0.8129 ± 0.0372 | 0.7489 ± 0.0489 |
| Shear flow | 0.1639 ± 0.0354 | 0.1359 ± 0.0029 | 0.1452 ± 0.0247 |
Internal-DW achieves lower test error than STLW on MG, ETTm1, and ETTm2. On shear flow, both methods outperform Full BPTT, with STLW slightly outperforming Internal-DW. These additional results provide a practical comparison without implying general superiority over alternative methods.
Question 5 — Additional training seeds
Reviewer concern (summarized): Do the gains persist with more seeds, particularly where the original paired intervals included zero?
Response. To assess stability across training runs, we expanded the standard-training comparison to seven paired seeds. DW has lower test relative- in all seven pairs on each of the four datasets. All paired 95% confidence intervals for the mean difference, DW minus Full BPTT, are below zero:
| Dataset | Mean DW–Full difference | Paired 95% CI |
|---|---|---|
| MG | -0.0537 | [-0.0597, -0.0476] |
| ETTm1 | -0.0679 | [-0.1010, -0.0349] |
| ETTm2 | -0.0745 | [-0.1378, -0.0112] |
| Shear | -0.0120 | [-0.0202, -0.0037] |
Each dataset uses the same test split across training seeds. Appendix L.2, Tables 15–16 gives the paired analysis and seed-level scores. This supports robustness to training seeds under this protocol, not across independent dataset samples.
We retain the unfavorable comparisons: TBPTT8 has lower mean error on shear in the three-seed control study (L.2, Table 18). Equal-time benefits are also less consistent: at 90 minutes DW has lower mean shear error but wins only 2/7 pairs, with worse mean error at 60/75 minutes. Appendix M.1–M.2 reports the complete matched-budget results, including calibration cost. Standard-training wins do not imply uniform equal-time superiority.
Question 6 — Observed rather than fitted best horizons
Reviewer concern (summarized): Does the training-horizon result hold when using the best observed grid point instead of a quadratic fit?
Response. The observed grid points support a later optimum on three datasets, with no shift in the mean optimum on shear. We take the minimum mean test relative- over the original seven training-horizon grid points, holding evaluation horizon fixed and averaging over three matched seeds:
| Dataset | Evaluation | Full BPTT: best | DW: best |
|---|---|---|---|
| MG | 64 | 16 | 32 |
| ETTm1 | 128 | 16 | 32 |
| ETTm2 | 128 | 32 | 64 |
| Shear | 48 | 32 | 32 |
Across the 12 dataset–seed pairs, the measured optimum shifts later in nine and ties in three. Thus, the shift is not solely an artifact of fitting a quadratic, but the mean-optimal horizon is unchanged on shear. These are observed optima on the tested grid, not globally optimal horizons. Appendix N.1, Table 25 reports the selected grid points and their forecasting errors.
Response to Weakness 1 / Question 1: Regime diagnostics without test-split scores
Thank you for the constructive feedback. We have revised the manuscript to address your concerns and respond point by point below, with references to the relevant sections, tables, and figures.
Response. We agree that the original map was descriptive rather than an independently validated rule for predicting when DW helps. We have replaced the test-split diagnostics with a training/validation-only analysis in the revised paper.
1. What information enters the diagnostic. We select history width W on an internal training holdout, fit the linear and nonlinear probes using official training examples only, and evaluate them on the official validation split. Feature scaling uses fitting data only; neighbor counts and the ridge coefficient are fixed. Thus, test-split scores do not enter the revised diagnostic. Appendix H.1 gives the construction and probe settings (Table 10).
2. Does the pattern persist? Yes. MG, shear, ETTm1, and ETTm2 still show substantial additional history gain and little detected drive gain. Below, ΨD measures drive-only predictive gain and ΔΨH measures additional history gain above the misaligned-history null. Scores are averaged over linear/nonlinear probes and horizons k≥8, before display clipping.
The revised main-text Fig. 3 uses validation scores; Appendix H.1, Table 11 reports the unclipped values. Negative scores mean no detected predictive value under these probes, not negative information (H.2).
3. What this establishes. The predictive-regime classification is constructed independently of DW performance, based on the relative predictive contributions of history and external drive. The regime criteria were established prior to evaluating DW’s forecasting outcomes, rather than retrospectively defined to explain where DW performed well. The revised analysis further confirms that the classification does not depend on test-split scores: the same regime patterns persist using training/validation data only.
These results support the use of predictive-regime diagnostics to characterize the conditions under which DW is expected to be beneficial. While the observed correspondence between regime characteristics and DW performance supports this hypothesis, generalization to entirely new datasets remains to be established.