acceptodds
Under review as a conference paper at ICLR 2027

Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting

Abstract

In autoregressive forecasting, long prediction rollouts provide distant supervision, but backpropagation through time (BPTT) carries gradients from those losses through many autoregressive steps. Repeated Jacobian products can make distant gradients dominate the update while amplifying predictable signal and unpredictable innovation together; a large distant gradient therefore need not carry reliable learning signal. Motivated by this observation, we introduce Internal Dual-Wiener routing (Internal-DW), a principled backward-only intervention that preserves the full forward rollout and all horizon losses while reliability-weighting internal gradient routes. At each residual block, we derive bounded Wiener gains for the identity and nonlinear routes that balance preserving predictable learning signal against suppressing unpredictable variation, and estimate them from route-level gradient statistics and an explicit noise model. In a controlled system with known gradient signal-to-noise ratio (SNR), we show that distant gradients can grow even as their SNR falls, and that Internal-DW reduces error in recovering predictable gradient signals and improves forecasting. On four history-dominated, weak-drive testbeds, Internal-DW reduces forecast error by 5.2%–13.8% relative to full BPTT, outperforms gradient clipping and Jacobian regularization on three testbeds, with similar performance on shear flow, and outperforms validation-selected truncated BPTT (TBPTT) on three. It also extends or preserves the fitted optimal training-horizon range across these four testbeds. Across the full benchmark suite, the current Internal-DW estimator has a clear applicability boundary: its benefit diminishes or reverses when usable history is limited or when the selected sampler fails to represent dominant drive-dependent variation. The results show that retaining long-horizon supervision does not require trusting every backward contribution equally.

Primary area:
learning on time series and dynamical systems
Keywords:
Temporal Credit Assignment, Long-Horizon Autoregressive Forecasting, Backpropagation Through Time

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kffAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies delayed credit assignment in long-horizon autoregressive forecasting. The main observation is that distant BPTT gradients can become large even when the corresponding learning signal is unreliable.

The authors propose Internal-DW, a backward-only modification for residual blocks. It assigns separate Wiener-style weights to the identity and nonlinear gradient routes while keeping the forward rollout and all horizon losses unchanged. The weights are estimated from route-level second moments together with a noise model.

The paper first studies the mechanism in a controlled system where gradient signal and innovation can be separated, and then evaluates the method on a collection of sequence, neural, fluid, and weather forecasting tasks.

Critical strengths
  1. The controlled known-SNR study is a strong part of the paper.

The authors directly construct a setting where predictable gradient signal and innovation can be separated. This allows them to show that distant gradient magnitude can increase while gradient SNR decreases, rather than inferring this indirectly from forecasting error. The misplaced-gain and oracle controls are also useful for showing that the location of the weighting matters.

  1. The backward-only formulation is an interesting design choice.

Internal-DW leaves the forward model and rollout unchanged and modifies only internal backward routes. This separates it conceptually from gradient clipping, rollout truncation, and forward Jacobian regularization. The paper also reports several failure cases rather than claiming that the method is universally beneficial.

Critical weaknesses
  1. The claimed applicability regime is not independently validated.

A central empirical conclusion is that Internal-DW works in "history-dominated, weak-drive" regimes. However, the predictive-regime scores used to define this grouping are themselves evaluated on the test split. The probe hyperparameters are selected on validation, but the final probes are refit on train+validation and then evaluated on test, after which the resulting testbed grouping is used to interpret the forecasting results on those same testbeds.

This makes the regime analysis descriptive rather than predictive. There is no fixed threshold or independently held-out set of datasets showing that the regime score can predict whether Internal-DW will help before observing the test result.

This matters because the overall results are mixed: the method improves some testbeds, is roughly neutral on others, and clearly hurts NARMA and movie fMRI. I do not think regrouping these outcomes using a test-derived diagnostic is sufficient evidence that the paper has identified when the method should be used.

  1. The practical estimator is much less principled than the oracle formulation suggests, and the experimental comparison does not sufficiently isolate this issue.

The Wiener result assumes access to a decomposition of route variation into predictable signal and innovation, but the paper itself shows that this decomposition is not identifiable from the observed route law. In practice, the key quantity is supplied by a noise sampler whose assumptions differ substantially across datasets: centered residual templates, factor models, subject-specific AR models, and spatial spectral models.

Therefore, much of the practical behavior of Internal-DW is determined by a dataset-specific modeling choice that is outside the Wiener derivation. The paper already shows that sampler mismatch can reverse the benefit of the method.

There is also an additional mismatch between theory and implementation: the route moments are calibrated using a quadratic probe even when the actual training objective is different, and the paper explicitly notes that transfer from the probe loss to the task loss is only an approximation.

I would like to see a much more systematic sampler ablation and stronger direct baselines. The current baseline suite mainly contains full BPTT, clipping, Jacobian regularization, TBPTT, and static damping, while several more directly related recent long-horizon credit-assignment or temporal-supervision methods discussed in Related Work are not evaluated.

Questions for authors
  1. Can the predictive-regime classification be constructed using training and validation data only, with the grouping rule fixed before any test evaluation?

This would materially affect my assessment. In the current version, the main "history-dominated, weak-drive" explanation is based on regime scores evaluated on the test split.

  1. How sensitive are the results to the noise sampler itself?

For datasets where both are feasible, I would like to see Generic and Prior samplers evaluated side by side, as well as deliberately simplified or misspecified samplers. Since the sampler determines the estimated noise moment, this seems necessary to separate the contribution of the routing rule from the domain-specific noise model.

  1. The calibration moments are computed from a quadratic probe, while the actual parameter update can use a different task loss. How sensitive are the learned gains to the choice of calibration probe?

Could the authors calibrate using the actual task loss, or otherwise show that the same gains are obtained under different reasonable probe losses?

  1. Why are more direct recent long-horizon credit-assignment / temporal-supervision methods discussed in the related work, such as STLW, HERO, or COLA, not included as experimental baselines?

  2. The main positive application results use only three matched seeds, and the reported paired 95% intervals for ETTm1, ETTm2, and shear flow include zero. Are these improvements stable with additional seeds?

  3. The "extended optimal training horizon" result is based on quadratic fits over a small grid of training horizons. Does the same conclusion hold if one compares the best actually observed training horizon, without fitting a quadratic curve?

5 comments
AuthorsOpenReview ↗

Response to Weakness 1 / Question 1: Regime diagnostics without test-split scores

Thank you for the constructive feedback. We have revised the manuscript to address your concerns and respond point by point below, with references to the relevant sections, tables, and figures.

Reviewer concern (summarized): The regime grouping uses test-split scores to interpret outcomes on the same benchmarks. Can it be constructed from training/validation data, with a rule fixed before test evaluation?

Response. We agree that the original map was descriptive rather than an independently validated rule for predicting when DW helps. We have replaced the test-split diagnostics with a training/validation-only analysis in the revised paper.

1. What information enters the diagnostic. We select history width WW on an internal training holdout, fit the linear and nonlinear probes using official training examples only, and evaluate them on the official validation split. Feature scaling uses fitting data only; neighbor counts and the ridge coefficient are fixed. Thus, test-split scores do not enter the revised diagnostic. Appendix H.1 gives the construction and probe settings (Table 10).

2. Does the pattern persist? Yes. MG, shear, ETTm1, and ETTm2 still show substantial additional history gain and little detected drive gain. Below, ΨD\Psi_D measures drive-only predictive gain and ΔΨH\Delta\Psi_H measures additional history gain above the misaligned-history null. Scores are averaged over linear/nonlinear probes and horizons k≥8k\ge8, before display clipping.

TestbedOriginal test: ΨD\Psi_DOriginal test: ΔΨH\Delta\Psi_HValidation: ΨD\Psi_DValidation: ΔΨH\Delta\Psi_H
MG0.00000.40600.00000.4046
MG+drive0.31110.57170.30090.5671
NARMA0.5701-0.00050.57510.0007
iEEG-1.70440.0401-1.78830.0405
Shear0.00000.70130.00000.7482
WeatherBench-20.2250-0.08750.1875-0.1150
Movie fMRI0.22950.02750.22320.0199
ETTm1-0.34750.4032-0.24740.4408
ETTm2-0.08150.61720.01720.6956

The revised main-text Fig. 3 uses validation scores; Appendix H.1, Table 11 reports the unclipped values. Negative scores mean no detected predictive value under these probes, not negative information (H.2).

3. What this establishes. The predictive-regime classification is constructed independently of DW performance, based on the relative predictive contributions of history and external drive. The regime criteria were established prior to evaluating DW’s forecasting outcomes, rather than retrospectively defined to explain where DW performed well. The revised analysis further confirms that the classification does not depend on test-split scores: the same regime patterns persist using training/validation data only.

These results support the use of predictive-regime diagnostics to characterize the conditions under which DW is expected to be beneficial. While the observed correspondence between regime characteristics and DW performance supports this hypothesis, generalization to entirely new datasets remains to be established.

AuthorsOpenReview ↗

Response to Weakness 2 / Questions 2–3: Sampler and calibration sensitivity

Weakness 2 / Question 2 — Sampler assumptions

Reviewer concern (summarized): The noise model is not identified by the Wiener derivation. How sensitive is forecasting to Generic/Prior choices and deliberately misspecified samplers?

Response. The practical estimator depends on sampler assumptions. The oracle formula gives gains conditional on the route moments. Observed moments alone do not identify the innovation law; the practical sampler supplies that assumption. We therefore test both sampler choice and a controlled mismatch.

1. Sampler choice. We compare five matched seeds (0–4), with training/evaluation horizons K=32K=32, H=48H=48 on MG/shear and K=64K=64, H=96H=96 on ETTm1/2. Full BPTT and original DW runs are reused. Test relative-L2L_2 is mean ± sample SD; lower is better. Displayed values are rounded; percentages use unrounded results.

MethodMGETTm1ETTm2Shear
Full BPTT1.053 ± 0.0030.944 ± 0.0190.859 ± 0.0700.151 ± 0.031
DW-Generic0.999 ± 0.0070.881 ± 0.0180.770 ± 0.0480.130 ± 0.005
DW-Prior—0.920 ± 0.0390.827 ± 0.0490.137 ± 0.020
Zero gains1.005 ± 0.0070.889 ± 0.0230.768 ± 0.0640.293 ± 0.009

No Prior was evaluated on MG. On ETT, Prior uses a calendar-VARX model fitted on training data, with calibration frequency and minimum probe count held fixed. Generic wins all five pairs on both ETT datasets. On shear, Generic uses a residual bootstrap and Prior a spatial-spectrum model: Generic has 5.40% lower mean error, mainly driven by seed 1, although Prior wins 3/5 pairs. Domain structure does not consistently improve forecasting.

2. Simulated sampler failure: zero-gain control. We manually fix α=m=0\alpha=m=0 throughout training to mimic extreme attenuation from severe noise overestimation; an actual sampler did not produce these zeros. Compared with Generic, zero gains are worse in all five MG and shear pairs. On ETTm2, they reduce mean error by 0.27% (3/5 wins); on ETTm1, they increase it by 0.85% (2/5 wins). Adaptive routing does not universally beat closed routes.

3. Controlled sampler misspecification. In the Known-SNR task

xt+1=Axt+βbut+ϵt+1,x_{t+1}=Ax_t+\beta b u_t+\epsilon_{t+1},

all models observe the same future inputs. Conditional sampling holds them fixed and resamples innovations; marginal sampling incorrectly resamples the inputs as noise. At K=32K=32, mean test relative-L2L_2 over horizons 1–48 is 0.539 for conditional DW, 0.549 for marginal DW, and 0.559 for Full BPTT. Conditional DW wins all four seed pairs, with a mean paired reduction of 1.81%. This isolates the effect of one incorrect sampler assumption. Appendix E.4, Table 6 gives sampler protocols and seed-level results; E.4 also details the controlled mismatch.

Weakness 2 / Question 3 — Calibration-probe choice

Reviewer concern (summarized): Do task-loss and quadratic calibration give similar gains?

Response. We agree that transfer between probe losses should be checked. Relative-L2L_2 noise calibration additionally requires the conditional target mean; the quadratic construction cancels this mean between paired futures (Appendix C.5, Table 2). We compare probes in Known-SNR, where the mean is exact, using the same frozen seed-0 checkpoint, sampled futures, and bounded solver across four layers and 32 steps.

Mean absolute gain differences are 0.065 for α\alpha and 0.066 for mm. Within-layer Spearman correlations are 0.980–0.993 for gains and 0.986–0.992 for trace reliability. Relative-L2L_2 generally gives lower gains, but the main rankings agree (Fig. 10).

From the same checkpoint, two copies train for three additional epochs with their respective fixed gain schedules, the same relative-L2L_2 task loss, batch order, learning rate, and fresh optimizer state. Calibration samples are not reused for validation.

EpochValidation: quadratic probeValidation: relative-L2L_2 probeMedian gradient angleMedian norm ratio
11.0305641.0293959.29°1.095
21.0574031.05529715.46°1.101
31.0324721.03427515.11°1.059

Norm ratios are relative-L2L_2-probe divided by quadratic-probe gradient norms. Validation losses differ by at most 0.20% at these endpoints (Table 3). Later gradient comparisons also reflect divergence between trained models. This supports similar short-run behavior in this controlled case, not application probe equivalence or task-gradient optimality.

On MG, quadratic-calibrated DW also reduces mean relative-L2L_2 validation loss by 6.56% and wins all three matched runs (C.5, Table 4). This is complementary application-level evidence, not a direct comparison of probes.

AuthorsOpenReview ↗

Response to Weakness 2 / Questions 4–6: Baselines, robustness, and training horizons

Weakness 2 / Question 4 — Related baselines

Reviewer concern (summarized): Why are STLW, HERO, and COLA discussed but not evaluated as baselines?

Response. We appreciate the suggestion to compare with these approaches. Direct comparisons would help clarify their relative strengths. Our main comparison holds the forward horizon and training objective fixed to isolate the effect of changing internal backward routing. STLW gradually increases later-horizon loss weights (Coker et al., 2026); HERO adds relative supervision against a detached failed-rollout reference (Zhang et al., 2026). They change how forecast errors enter the objective, whereas DW changes internal gradient transport. On a residual backbone, either objective could potentially be retained while applying DW, with calibration adapted to its learning signals.

COLA uses dynamical criticality for online local credit assignment approximating global error propagation (Wang et al., 2026). It changes the learning rule itself; extending reliability weighting would require defining and calibrating the corresponding local update contributions.

These methods address different aspects of training and are potentially complementary to Internal-DW. Our paper focuses on reliability-weighted internal gradient transport and does not claim superiority over alternative training objectives or credit-assignment methods. Head-to-head comparisons with these methods are therefore not required to establish the paper’s specific claims.

Nevertheless, to address the reviewer’s question, we conducted an additional practical comparison with STLW on MG, ETTm1, ETTm2, and shear flow using three seeds per dataset. The results are shown below.

Test relative-L2L_2 error (mean ± sample SD across seeds 0–2; lower is better).

DatasetFull BPTTSTLWInternal-DW
MG1.0541 ± 0.00441.0589 ± 0.02870.9997 ± 0.0096
ETTm10.9422 ± 0.02520.9275 ± 0.01220.8786 ± 0.0176
ETTm20.8684 ± 0.09130.8129 ± 0.03720.7489 ± 0.0489
Shear flow0.1639 ± 0.03540.1359 ± 0.00290.1452 ± 0.0247

Internal-DW achieves lower test error than STLW on MG, ETTm1, and ETTm2. On shear flow, both methods outperform Full BPTT, with STLW slightly outperforming Internal-DW. These additional results provide a practical comparison without implying general superiority over alternative methods.

Question 5 — Additional training seeds

Reviewer concern (summarized): Do the gains persist with more seeds, particularly where the original paired intervals included zero?

Response. To assess stability across training runs, we expanded the standard-training comparison to seven paired seeds. DW has lower test relative-L2L_2 in all seven pairs on each of the four datasets. All paired 95% tt confidence intervals for the mean difference, DW minus Full BPTT, are below zero:

DatasetMean DW–Full differencePaired 95% CI
MG-0.0537[-0.0597, -0.0476]
ETTm1-0.0679[-0.1010, -0.0349]
ETTm2-0.0745[-0.1378, -0.0112]
Shear-0.0120[-0.0202, -0.0037]

Each dataset uses the same test split across training seeds. Appendix L.2, Tables 15–16 gives the paired analysis and seed-level scores. This supports robustness to training seeds under this protocol, not across independent dataset samples.

We retain the unfavorable comparisons: TBPTT8 has lower mean error on shear in the three-seed control study (L.2, Table 18). Equal-time benefits are also less consistent: at 90 minutes DW has lower mean shear error but wins only 2/7 pairs, with worse mean error at 60/75 minutes. Appendix M.1–M.2 reports the complete matched-budget results, including calibration cost. Standard-training wins do not imply uniform equal-time superiority.

Question 6 — Observed rather than fitted best horizons

Reviewer concern (summarized): Does the training-horizon result hold when using the best observed grid point instead of a quadratic fit?

Response. The observed grid points support a later optimum on three datasets, with no shift in the mean optimum on shear. We take the minimum mean test relative-L2L_2 over the original seven training-horizon grid points, holding evaluation horizon HH fixed and averaging over three matched seeds:

DatasetEvaluation HHFull BPTT: best KKDW: best KK
MG641632
ETTm11281632
ETTm21283264
Shear483232

Across the 12 dataset–seed pairs, the measured optimum shifts later in nine and ties in three. Thus, the shift is not solely an artifact of fitting a quadratic, but the mean-optimal horizon is unchanged on shear. These are observed optima on the tested grid, not globally optimal horizons. Appendix N.1, Table 25 reports the selected grid points and their forecasting errors.

Reviewer_huiJOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Internal Dual-Wiener routing (Internal-DW) to improve gradient reliability in long-horizon autoregressive forecasting. The method addresses the issue that gradients propagated through long BPTT chains may amplify unpredictable noise alongside useful learning signals. Internal-DW preserves the full forward rollout and horizon losses while applying reliability-based gradient weights to the identity and nonlinear branches of residual blocks, derived from a Wiener optimization framework. Experiments on controlled systems and forecasting benchmarks show that the method improves gradient signal recovery and reduces forecasting errors compared with full BPTT and several gradient control baselines. The authors also identify limitations when historical information is insufficient or the noise sampler fails to capture dominant variations.

Critical strengths
  1. This paper shifts the focus from gradient magnitude to gradient reliability, noting that amplified gradients in long-horizon BPTT may contain substantial unpredictable noise. This provides useful insights into credit assignment in long-sequence training and motivates more refined gradient propagation methods.

  2. The proposed method is supported by a clear mathematical formulation. Specifically, the authors derive separate gradient weights for the identity and nonlinear branches of residual blocks through a two-dimensional Wiener optimization problem, rather than applying a uniform gradient attenuation scheme.

  3. Beyond reporting prediction errors, the authors conduct controlled experiments with known signal-to-noise ratios (SNRs) to examine the relationships among gradient magnitude, reliability, local risk, and prediction performance.

Critical weaknesses
  1. Theorem 1 establishes the optimality of local gradient routing in terms of mean squared error under given signal and noise matrices. However, lower local risk does not necessarily lead to better parameter updates or forecasting performance, especially across multiple prediction steps, residual blocks, and shared parameters. The theoretical connection to end-to-end improvements remains unclear.

  2. Internal-DW relies on separating predictable gradient signals from unpredictable noise, while DW-Generic and DW-Prior introduce additional assumptions. The paper lacks systematic experiments evaluating how sampler mismatch affects the method's performance.

  3. Although runtime results are provided, a more direct comparison between performance gains and computational overhead is needed to justify the additional complexity of Internal-DW.

Questions for authors

See the weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Weakness 1: From local routing to prediction risk after one update

Thank you for the constructive feedback. We have revised the manuscript to address your concerns and respond point by point below, with references to the relevant sections, tables, and figures.

Reviewer concern (summarized): Lower local route risk need not improve parameter updates or forecasting across multiple steps, residual blocks, and shared parameters. What connects Theorem 1 to end-to-end prediction?

Response. We agree that Theorem 1 alone does not establish this connection. Appendix C.2 adds a one-update analysis of a linear residual rollout with shared parameters and exact oracle moments. It follows the complete parameter update, rather than summing local risk bounds.

1. Model and shared-parameter update. At each of KK prediction steps, the same parameter θ\theta enters N≥2N\ge2 blocks:

zt,0=x^t−1,zt,l=zt,l−1+(A−I)zt,l−1+Bθ,x^t=βzt,N.z_{t,0}=\hat x_{t-1},\quad z_{t,l}=z_{t,l-1}+(A-I)z_{t,l-1}+B\theta,\quad \hat x_t=\beta z_{t,N}.

Here AA is fixed and symmetric, A⪰IA\succeq I with at least two distinct eigenvalues, BB has full column rank, and β>0\beta>0; only the shared affine term is learned. Thus f^θ(x)=Dx+Cθ\hat f_\theta(x)=Dx+C\theta, where D=βAND=\beta A^N and C=β(∑r=0N−1Ar)BC=\beta(\sum_{r=0}^{N-1}A^r)B. Assume f=f^θ∗f=\hat f_{\theta^*} and observations xt=f(xt−1)+σξtx_t=f(x_{t-1})+\sigma\xi_t, with independent zero-mean innovations of covariance II. The prediction risk is

Rf(θ)=12∥C(θ−θ∗)∥2.R_f(\theta)=\tfrac12\|C(\theta-\theta^*)\|^2.

Both methods start at the same θ\theta, use L=(2K)−1∑t=1K∥x^t−xt∥2L=(2K)^{-1}\sum_{t=1}^K\|\hat x_t-x_t\|^2 with x^0=x0\hat x_0=x_0, and take θ+=θ−ηg\theta^+=\theta-\eta g. Under DW's backward rule,

Z_0^{\rm out}=0,\quad Z_{t,0}=Z_{t-1}^{\rm out},\\ Z_{t,l}=[\alpha_{t,l}I+m_{t,l}(A-I)]Z_{t,l-1}+B,\quad Z_t^{\rm out}=\beta Z_{t,N},\\ g_{\rm DW}=\frac1K\sum_{t=1}^K(Z_t^{\rm out})^\top(\hat x_t-x_t). \end{gathered}$$ $Z$ is the parameter sensitivity under the modified backward rule. The $+B$ term preserves each block's direct parameter contribution; temporal carry and the final sum account for all steps in the same parameter space. Theorem 1 supplies gains from fully open oracle moments; setting all gains to one recovers Full BPTT. **2. At and near the optimum.** At $\theta=\theta^*$, the route signal is zero and the route-noise matrix is positive definite, so the unique oracle gains are $\alpha=m=0$. Expanding the complete gradients and grouping terms by innovation accounts for cross-step noise correlations. Comparing the resulting prediction risks gives, for $\eta,\sigma>0$, $$\mathbb E R_f(\theta_{\rm DW}^+)\le N^{-2}\mathbb E R_f(\theta_{\rm Full}^+)<\mathbb E R_f(\theta_{\rm Full}^+).$$ The expectation is over training noise. This means less noise-induced deterioration from the optimum, not improvement over its zero initial risk. Continuity extends the strict advantage to a neighborhood; the $N^{-2}$ factor is established at the optimum. The same argument applies to $K$-step conditional-mean forecast MSE. **3. Away from the optimum.** Writing $e=\theta-\theta^*$ and $s\in\{\mathrm D,\mathrm F\}$ for DW and Full BPTT, respectively, the exact comparison is $$\mathbb E R_f(\theta_s^+)=R_f(\theta)-a_s\eta+b_s\eta^2,\quad a_s=\langle Ce,C\mathbb E g_s\rangle,\quad b_s=\tfrac12\mathbb E\|Cg_s\|^2.$$ If $a_{\rm F}>a_{\rm D}>0$ and $b_{\rm F}>b_{\rm D}>0$, both methods improve and DW improves more whenever the following interval is nonempty and $\eta$ lies in it: $$\frac{a_{\mathrm{F}}-a_{\mathrm{D}}}{b_{\mathrm{F}}-b_{\mathrm{D}}} < \eta < \min\left\lbrace \frac{a_{\mathrm{F}}}{b_{\mathrm{F}}}, \frac{a_{\mathrm{D}}}{b_{\mathrm{D}}} \right\rbrace.$$ Table 1 gives a stable two-dimensional example with $N=K=3$, where this interval is $0.57164<\eta<0.66115$. At the same $\eta=0.6$, exact expected risks are (rounded): | Prediction risk | Before update | Full BPTT | Oracle DW | |---|---:|---:|---:| | One-step conditional mean | 0.446 | 0.301 | 0.282 | | Three-step conditional-mean rollout | 0.804 | 0.701 | 0.513 | The advantage also holds when each method's learning rate is optimized separately for each metric: minima are approximately 0.192 versus 0.251 for one step, and 0.387 versus 0.497 for three steps (DW versus Full; Fig. 9). Appendix C.2 contains the full derivation, model schematic (Fig. 8), example (Table 1), and learning-rate curves (Fig. 9). This establishes a one-update connection in the stated linear oracle model, not a convergence or forecasting guarantee for the practical plug-in estimator in nonlinear networks.
AuthorsOpenReview ↗

Response to Weaknesses 2–3: Sampler sensitivity and computational cost

Weakness 2 — Sensitivity to sampler assumptions

Reviewer concern (summarized): How sensitive is Internal-DW to noise-sampler choice and misspecification? A systematic comparison is needed to assess their effect on forecasting.

Response. We agree that sampler assumptions need direct evaluation. We added sampler comparisons and a controlled misspecification test.

1. Sampler choice and extreme attenuation.

We compare Full BPTT, DW-Generic, DW-Prior where available, and zero gains using five matched seeds (0–4). Training/evaluation horizons are K=32K=32, H=48H=48 on MG and shear, and K=64K=64, H=96H=96 on ETTm1/2. Full BPTT and original DW runs are reused. Entries are test relative-L2L_2 mean ± sample SD; lower is better. Displayed values are rounded; percentages use unrounded results.

MethodMGETTm1ETTm2Shear
Full BPTT1.053 ± 0.0030.944 ± 0.0190.859 ± 0.0700.151 ± 0.031
DW-Generic0.999 ± 0.0070.881 ± 0.0180.770 ± 0.0480.130 ± 0.005
DW-Prior—0.920 ± 0.0390.827 ± 0.0490.137 ± 0.020
Zero gains1.005 ± 0.0070.889 ± 0.0230.768 ± 0.0640.293 ± 0.009

No Prior sampler was evaluated on MG. On ETT, the Prior is a calendar-VARX model fitted on the training split; calibration frequency and minimum probe count are held fixed. Generic has lower error in all five pairs on both ETT datasets. On shear, Generic uses a residual bootstrap and Prior a spatial-spectrum model. Generic has 5.40% lower mean error, driven mainly by seed 1, although Prior wins three of five pairs. Adding domain structure therefore does not consistently improve forecasting.

We manually fixed α=m=0\alpha=m=0 throughout training to test permanently closed routes. It is worse than Generic in all five MG pairs and worse than Full, Generic, and Prior in all five shear pairs. Against Generic on ETT, zero gains reduce mean error by 0.27% on ETTm2 (three wins), but increase it by 0.85% on ETTm1 (two wins). Thus, adaptive routing improves on closed routes in some settings, not universally. Appendix E.4, Table 6 provides the protocols and seed-level results.

2. Controlled sampler misspecification.

To isolate one incorrect noise assumption, we use a Known-SNR task,

xt+1=Axt+βbut+ϵt+1,x_{t+1}=Ax_t+\beta b u_t+\epsilon_{t+1},

where all models observe the same future input sequence utu_t. The conditional sampler holds this sequence fixed and resamples only innovations; the marginal sampler also resamples the known inputs, incorrectly treating their variation as noise. At K=32K=32, mean test relative-L2L_2 over horizons 1–48 is 0.539 for conditional DW, 0.549 for marginal DW, and 0.559 for Full BPTT across four seeds. Conditional DW beats marginal DW in all four pairs, with a mean paired reduction of 1.81% (Appendix E.4). These experiments show sensitivity to sampler assumptions, not a general guarantee that a more structured sampler improves forecasting.

Weakness 3 — Computational cost and equal-time utility

Reviewer concern (summarized): Do the forecasting improvements justify the additional calibration and training cost?

Response. A practical comparison should include calibration overhead. We therefore measured this overhead and compared forecasting under equal wall-clock budgets.

On one NVIDIA H200 NVL, training-loop overhead is 26.6–39.5%, including online calibration. Epoch timings below are means of three paired repeats and exclude validation, checkpoint I/O, and one-time prior preprocessing (Appendix M, Table 21).

We also compare DW with Full BPTT at 60/75/90-minute budgets using seven paired seeds. The table pairs epoch cost with the 90-minute test results:

DatasetEpoch time: Full → DW (s)Mean test-error reduction at 90 minDW wins at 90 min
MG21.28 → 27.431.91%7/7
ETTm127.78 → 35.322.19%5/7
ETTm228.68 → 36.318.21%6/7
Shear29.79 → 41.551.61%2/7

Benefits depend on the dataset and budget. ETTm2 has lower mean error at all three budgets; MG first does so at 75 minutes. Shear is worse at 60/75 minutes and wins only 2/7 pairs at 90 minutes despite a lower mean. Standard-training wins do not establish consistent equal-time gains.

Each pair uses the same GPU type, with activation checkpointing disabled and early stopping retained. We select the best validation checkpoint within each budget. Timing includes initialization, training, calibration, validation, and checkpoint I/O, but excludes test evaluation. The 75/90-minute runs resume checkpoints and count cumulative computation, excluding waiting; checkpoint RNG state is not preserved. Appendix M.1–M.2, Tables 22–24 give all budgets, protocols, and seed-level results.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer huiJ,

Thank you again for your detailed feedback. We wanted to briefly follow up on our responses, particularly the added analysis connecting local routing to prediction risk and the additional sampler and computational-cost experiments.

We would appreciate any feedback on whether these additions address your concerns. If any questions remain, we would be happy to clarify them during the discussion period.

Thank you for your time and consideration.

Best regards,
The Authors of Submission 14371

Reviewer_zERWOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why large gradients from distant autoregressive forecast losses can carry unreliable learning signals. Internal Dual-Wiener routing leaves forward computation unchanged and reweights the identity and nonlinear backward routes of residual blocks using estimated signal and noise moments. A local minimum-risk derivation, a controlled known-SNR experiment, and eight application testbeds connect this mechanism to forecasting gains and identify regimes where the estimator fails.

Critical strengths
  1. The paper gives reliability weighting a precise target. Theorem 1 derives the optimal bounded two-route weights for a stated local risk, and Section 3.3 explains why estimating these weights requires additional noise-model assumptions. This distinction makes the contribution more informative than an empirically chosen gradient-damping rule.

  2. The experiments examine both the mechanism and its boundaries. Figure 4 measures gradient signal and noise in a controlled system, including a control that misplaces the learned gains. Figures 5-6 and Appendix L then show benefits in history-dominated settings alongside neutral or harmful outcomes elsewhere. This supports a useful, appropriately qualified empirical contribution.

Critical weaknesses
  1. The reliability criterion is not validated for the gradients used in application training. Equation 15 calibrates gains with a quadratic output probe, whereas Table 3 uses relative-L2 task losses. Changing the loss changes the route moments, so Theorem 1 does not establish optimality for the deployed task gradients. The authors acknowledge this approximation, but do not measure whether the probe and task gradients imply similar gains or reliability rankings. This leaves an important gap between the theoretical motivation and the application mechanism.

  2. The application controls do not isolate the benefit of the two-route reliability estimator. The Static baseline uses one global tied coefficient, while Internal-DW varies two gains across horizons and layers. Consequently, this comparison also changes the flexibility of the damping schedule. The misplaced-gain control in Figure 4 measures local risk in the controlled system, not application forecasting. A tied but horizon/layer-dependent control or a misplaced-gain training control would help distinguish noise-informed routing from more flexible attenuation.

  3. The comparative benefit remains uncertain relative to its training cost. Appendix L reports only three matched seeds, with DW-versus-BPTT intervals spanning zero on three of the four favorable testbeds; consistent improvement signs are encouraging but do not resolve effect-size uncertainty. TBPTT also has lower mean error on shear flow. Appendix M reports 20.3%-41.7% more time per epoch than full BPTT, without a comparison at equal wall-clock budgets. Thus, the practical gain over simpler training controls is less established than the headline reductions suggest.

Questions for authors
  1. How sensitive are the gains and held-out utility of the reweighted task gradients to using task-loss route messages rather than the quadratic probe, with a justified corresponding noise construction? A focused comparison on a favorable testbed would clarify whether calibration transfers to the actual updates.

  2. Do you have application results for gains tied between routes at each horizon/layer, or for a permutation of the learned horizon gains? Please distinguish the evidence for adaptive damping from evidence for the specific dual-route reliability estimate. One representative controlled comparison would be informative.

  3. Can you report paired seed-level differences against the strongest existing controls, and compare validation or test performance at matched training time using available learning curves? This would clarify whether the improvement survives both seed variation and calibration overhead.

5 comments
AuthorsOpenReview ↗

Response to Weakness 1 / Question 1: Quadratic-probe calibration and task-loss gradients

Thank you for the constructive feedback. We have revised the manuscript to address your concerns and respond point by point below, with references to the relevant sections, tables, and figures.

Reviewer concern (summarized): Calibration uses a quadratic probe, whereas application training uses relative-L2 loss. Do the probes imply similar gains and reliability rankings, and does calibration transfer to the actual task gradients?

Response. We agree that Theorem 1 does not establish optimality for relative-L2L_2 task gradients under quadratic calibration. We have clarified this approximation in Appendix C.4 ("How the Route Moments Are Calibrated"). Appendix C.5 adds a direct probe comparison in Known-SNR and application-level evidence on Mackey–Glass (MG).

1. Loss-dependent moments and noise construction. Let CC denote the observed history and known inputs, e=Y^−Ye=\hat Y-Y, and D(g)=[AIg,AFg]D(g)=[A_Ig,A_Fg] collect the two open-route VJPs. The quadratic output gradient is ee; the relative-L2L_2 gradient is c(Y)ec(Y)e, with c(Y)=1/[max⁡(∥Y∥2,ϵ)∥e∥2]c(Y)=1/[\max(\|Y\|_2,\epsilon)\|e\|_2]. This target-dependent scaling changes both signal and noise moments. Writing μ(C)=E[Y∣C]\mu(C)=\mathbb E[Y\mid C] and Y=μ(C)+εY=\mu(C)+\varepsilon gives

=\frac{1}{\max(\|\mu(C)+\varepsilon\|_2,\epsilon)\,\|\hat Y-\mu(C)-\varepsilon\|_2}.$$ For two independent futures under the same $C$, the quadratic route-message difference is $-D(\varepsilon_1-\varepsilon_2)$: the common mean cancels, allowing noise estimation from innovations alone. Relative-$L_2$ assigns a different $c(Y)$ to each future, so this cancellation generally fails. Our relative-$L_2$ noise construction therefore also needs $\mu(C)$. In applications, estimating this mean requires an additional predictor; its errors affect both norms above. This motivates the quadratic approximation. Table 2 gives the full moment comparison. **2. Direct probe comparison in Known-SNR.** Here, $$Y_{t+1}=a\odot Y_t+\sqrt{1-a^2}\odot Z_{t+1},\qquad \mu_k(C)=a^k\odot Y_0,$$ with independent standard Gaussian $Z$ and elementwise multiplication $\odot$. The conditional mean is thus known exactly. At one frozen Full-BPTT checkpoint (seed 0, four residual layers), we compare both probes over 32 steps, using eight validation trajectories, two origins, and 64 independent future pairs per origin. Both probes use the same sampled futures and bounded solver. Mean absolute gain differences are 0.0647 for $\alpha$ and 0.0660 for $m$. Within-layer Spearman correlations are 0.985–0.990 for $\alpha$ and 0.980–0.993 for $m$; trace-reliability rankings, $\operatorname{tr}(T-R)/\operatorname{tr}(T)$, correlate at 0.986–0.992. Relative-$L_2$ generally gives lower gains, but the main ranking trends agree (Fig. 10). **3. Do similar gains yield similar task gradients and losses?** From that checkpoint, we train two copies for three additional epochs using their respective fixed gain schedules, with the same relative-$L_2$ training loss, batch order, learning rate, and fresh optimizer state. The calibration samples were not reused for validation. | Epoch | Validation loss: quadratic probe | Validation loss: relative-$L_2$ probe | Median gradient angle | Median norm ratio | |---|---:|---:|---:|---:| | 1 | 1.030564 | 1.029395 | $9.29^\circ$ | 1.095 | | 2 | 1.057403 | 1.055297 | $15.46^\circ$ | 1.101 | | 3 | 1.032472 | 1.034275 | $15.11^\circ$ | 1.059 | Angles and norm ratios compare complete parameter gradients across matched batches; the norm ratio is the relative-$L_2$-probe gradient norm divided by the quadratic-probe gradient norm. Validation losses differ by at most 0.20% at these epoch endpoints. This supports similar short-run behavior in this controlled case; later gradient comparisons also reflect divergence between the trained models (Table 3). **4. Application-level utility on MG.** With relative-$L_2$ training unchanged, quadratic-calibrated DW improves validation loss over Full BPTT in all three matched seeds (0–2). At each run's validation-selected checkpoint, evaluated over $K=32$ steps: | Method | Validation loss, mean $\pm$ SD | |---|---:| | Full BPTT | 1.0036 $\pm$ 0.0176 | | DW | 0.9378 $\pm$ 0.0033 | The mean reduction is 6.56% (Table 4). This demonstrates application-level utility. Separately, a seed-0 MG training-loss check keeps quadratic calibration fixed and selects each checkpoint using its corresponding validation loss. Relative-$L_2$ test error over the same 384 origins at $H=48$ is 0.993871 with relative-$L_2$ training versus 1.002394 with MSE training, a 0.86% increase. Together, these checks support the practical approximation, without establishing application task-gradient optimality. The direct probe comparison remains limited to one Known-SNR checkpoint and fixed gain schedules.
AuthorsOpenReview ↗

Response to Weakness 2 / Question 2: Adaptive damping and gain placement

Reviewer concern (summarized): Internal-DW is more flexible than the global Static baseline, so their comparison does not isolate the contribution of the two-route reliability estimator. Can a horizon/layer-dependent tied-gain control or a misplaced-gain training control distinguish adaptive damping from the benefit of the calibrated routing weights?

Response. We agree that a matched control helps isolate the role of gain placement. We added a misplaced-gain training comparison on Mackey–Glass (MG), directly testing forecasting performance rather than only the local route-risk objective.

1. What the control changes. We compare original DW with misplaced DW across three matched seeds (0–2). At each optimizer update, the misplaced control jointly permutes the calibrated (α,m)(\alpha,m) pairs across all 32×432\times4 horizon–layer positions, keeping each pair together. This preserves the calibrated gain values within that update while changing where they are applied. We use training horizon K=32K=32 and report test relative-L2L_2 at H=48H=48, with Full BPTT as a reference.

MethodTest error, mean ± sample SD (n=3)
Full BPTT1.0541 ± 0.0044
Original DW0.9997 ± 0.0096
Misplaced DW1.0032 ± 0.0089

2. What the results establish. Original DW has lower error in all three seeds; shuffling increases mean error by 0.35%. The placement benefit is therefore consistent across these seeds but modest. Misplaced DW still improves over Full BPTT by 4.83%, versus 5.17% for original DW. Thus, calibrated placement adds a small benefit on MG, while most of the improvement over Full BPTT persists after shuffling.

3. Scope of our claim. Internal-DW constructs an adaptive damping schedule from estimated signal and noise moments by minimizing a local gradient-reconstruction objective. These results support its practical utility, with a modest additional benefit from gain placement in this comparison. They do not establish that the estimator finds the best forecasting weights or that separate weights for the two routes are necessary.

We clarify this scope in the final paragraph of Appendix L.3, following the misplaced-gain results in Table 20. Section 5.2, under “Forecasting across predictive regimes,” points readers to these additional routing controls.

AuthorsOpenReview ↗

Response to Weakness 3 / Question 3: Seed robustness, stronger controls, and training cost

Reviewer concern (summarized): Three seeds and confidence intervals spanning zero leave the forecasting gains uncertain; TBPTT performs better on shear, and DW costs more per epoch. Do paired comparisons against stronger controls and equal-time evaluations support a practical benefit?

Response. To assess robustness and practical cost, we expanded the evaluation to seven paired seeds and added stronger-control and equal-time comparisons, including calibration overhead.

Displayed values are rounded; paired differences and percentages use unrounded results.

1. Seven-seed robustness under standard training. DW has lower test relative-L2L_2 in all seven pairs on each dataset. All paired 95% tt confidence intervals for the mean difference (DW minus Full) lie below zero:

DatasetFull, mean ± SDDW, mean ± SDPaired 95% CI
MG1.054 ± 0.0031.000 ± 0.006[-0.0597, -0.0476]
ETTm10.950 ± 0.0190.882 ± 0.029[-0.1010, -0.0349]
ETTm20.841 ± 0.0660.767 ± 0.040[-0.1378, -0.0112]
Shear0.145 ± 0.0270.133 ± 0.018[-0.0202, -0.0037]

These are standard-training results, not equal-time results. Appendix L.2, Tables 15–16 provide the paired effects and seed-level scores.

2. Paired differences against stronger controls. On the original seeds 0–2, the strongest non-Full baseline by mean test error in the original comparison is Static on MG/ETT and TBPTT8 on shear. Static's cc is selected on seed-0 validation and fixed across seeds. Below, negative differences favor DW:

DatasetControlDW minus control: seed 0Seed 1Seed 2
MGStatic, c=0.6-0.0039-0.0015-0.0096
ETTm1Static, c=0.3-0.0250-0.0533-0.0607
ETTm2Static, c=0.6-0.0106-0.0968-0.0586
ShearTBPTT8+0.0013+0.0436-0.0058

DW beats Static in all three pairs on MG and both ETT datasets. On shear, TBPTT8 wins two of three pairs and has lower mean error: 0.132 versus 0.145 for DW. DW has lower mean error than TBPTT on MG and both ETT datasets. Full values are in Appendix L.2, Tables 17–18.

Theorem 1 provides a local risk-minimization justification for improving gradient transport in Full BPTT, not a guarantee of forecasting superiority over TBPTT. We include TBPTT as a practical baseline. A truncation-based DW variant, calibrated within the truncated graph, remains future work.

3. Training cost and equal wall-clock budgets. On one NVIDIA H200 NVL, training-loop time per epoch is:

DatasetFull (s)DW (s)Increase
MG21.28 ± 0.0927.43 ± 0.4228.9%
ETTm127.78 ± 0.0635.32 ± 0.2327.1%
ETTm228.68 ± 0.0836.31 ± 0.3126.6%
Shear29.79 ± 0.4341.55 ± 0.8039.5%

These timings include online calibration but exclude validation, checkpoint I/O, and one-time prior preprocessing (Appendix M, Table 21; three paired repeats).

Equal-time comparison, against Full BPTT only. We compare seven paired seeds at 60/75/90 minutes, matching GPU type and retaining early stopping. Each score uses the best validation checkpoint available within the budget. Timing includes initialization, training, calibration, validation, and checkpoint I/O, but excludes test evaluation. Entries are test relative-L2L_2 mean ± sample SD; wins count lower DW error.

MinutesMethodMGETTm1ETTm2Shear
60Full1.052 ± 0.0040.970 ± 0.0180.843 ± 0.0640.281 ± 0.010
DW1.087 ± 0.0480.969 ± 0.0380.771 ± 0.0410.293 ± 0.015
DW wins2/73/76/71/7
75Full1.052 ± 0.0040.971 ± 0.0230.835 ± 0.0740.277 ± 0.012
DW1.048 ± 0.0140.956 ± 0.0410.767 ± 0.0400.304 ± 0.035
DW wins5/75/76/72/7
90Full1.052 ± 0.0040.963 ± 0.0120.835 ± 0.0740.276 ± 0.030
DW1.032 ± 0.0060.942 ± 0.0250.767 ± 0.0400.271 ± 0.008
DW wins7/75/76/72/7

At 90 minutes, DW reduces mean error by 1.91%, 2.19%, 8.21%, and 1.61% on MG, ETTm1, ETTm2, and shear, respectively. Benefits depend on the dataset and budget: ETTm2 has lower mean error at all three budgets, MG first does so at 75 minutes, and shear is worse at 60/75 minutes. Even at 90 minutes, shear wins only 2/7 pairs despite its lower mean. Standard-training wins do not establish consistent equal-time gains.

The 75/90-minute runs resume saved checkpoints and count cumulative computation, excluding waiting intervals; checkpoint random-number-generator state is not preserved. Appendix M.1–M.2, Tables 22–24 provide the timing protocol and complete seed-level results.

Section 5.2 points to Appendices L.2–L.3 for additional seed comparisons and routing controls, and to Appendix M for runtime measurements.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to AC

Dear Area Chair,

Thank you for overseeing the review of our submission. We have revised the manuscript and provided detailed point-by-point responses to all three reviewers, including substantial new theoretical analyses and experiments.

We would like to briefly highlight the key additions:

Stronger theoretical connection to forecasting performance (Reviewer huiJ). We added a one-update analysis of a shared-parameter linear residual rollout, establishing conditions under which oracle DW improves prediction risk relative to Full BPTT, beyond minimizing local gradient-reconstruction risk. Improved empirical robustness and computational comparisons (Reviewers zERW, huiJ, kffA). We expanded the main comparisons to seven paired seeds, with DW outperforming Full BPTT in all seven runs across each of the four favorable testbeds and paired 95% confidence intervals excluding zero. We also added equal-wall-clock comparisons that account for calibration overhead and explicitly report mixed results where applicable. Additional controls and calibration validation (Reviewers zERW, kffA). We added a misplaced-gain training control, direct comparisons between quadratic and task-loss calibration probes in Known-SNR, and additional application-level evidence. These experiments clarify the contributions and limitations of reliability-weighted routing. Sampler sensitivity and applicability (Reviewers huiJ, kffA). We added Generic/Prior sampler comparisons, zero-gain controls, and a controlled sampler-misspecification experiment. We also reconstructed the predictive-regime diagnostics using training/validation data only, independently of DW test performance. Training-horizon analysis (Reviewer kffA). We verified the optimal-horizon findings using observed grid points rather than quadratic fits, confirming a later optimum on three testbeds and no shift on shear flow.

We have also clarified the relationship to recent long-horizon training methods and explicitly acknowledged the limitations of the current comparisons.

We hope these additions help clarify the contribution and address the main concerns raised during review. We would greatly appreciate any further discussion while there is still time to address remaining questions before November 18.

Thank you again for your time and consideration.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.