acceptodds
Under review as a conference paper at ICLR 2027

Reconstruction Advance Prediction

Abstract

The core objective of diverse time-series prediction methods is to learn a function that maps historical inputs to future outputs. However, the overlap between the input and output segments is often overlooked, which can be deemed the reconstruction of an overlap window. This shared portion naturally carries information about the underlying dynamics but is not explicitly exploited by many methods. Here, we propose **Re**construction **A**dvance **P**rediction (**ReAP**) as a plug-in training paradigm to explicitly leverage this overlap information. Theoretically, we show that reconstructing the overlap window can enhance prediction performance from the perspective of Dynamic Mode Decomposition (DMD). Building on this insight, we introduce a consistent loss to ensure that the learned function aligns with the real dynamics underlying the temporal system. We first demonstrate ReAP’s effectiveness with a single linear layer, then generalize the paradigm to several state-of-the-art (SOTA) models. Extensive experiments show that ReAP consistently improves prediction performance across various deep learning backbones on 7 real-world datasets. An ablation study further confirms that a suitable ReAP configuration enables the learned function to follow the scaling law of Time Series Forecasting (TSF). Code is available at this repository: https://anonymous.4open.science/r/ReAP-BB71/.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series prediction, Dynamic Mode Decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GgZGOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

I appreciate the authors' effort on this work. I have read the paper carefully, and below is my summary of the paper and its contributions: The paper introduces Reconstruction Advance Prediction (ReAP), a plug-in-and-play training paradigm for LTSF, inspired by Dynamic Mode Decomposition (DMD). Rather than training a model solely to map a historical window [1,…,n][1, \dots, n] to a disjoint future window [n+1,…,m][n+1, \dots, m], ReAP modifies the model's output during training to jointly reconstruct an overlapping tail of the historical window [τ,…,n][\tau, \dots, n] alongside the future forecast. To encourage the neural network predictor fθf_\theta to align with the underlying dynamical transition operator while avoiding the unstable computation of the Moore-Penrose pseudoinverse X†X^\dagger, the authors replace the theoretical operator error with a transpose surrogate consistency loss. For the experimental part, they evaluate the mechanism ReAP on a single linear layer to form (ReLinear), and integrate into 5 different backbones, reporting consistent error reductions, improved performance and also report ReLinear's competitiveness with SOTA Models. Lastly, the authors conduct experiments on the last section for scaling-law where they claim that baseline degrades as the history length goes from 96 to 720 while ReLinear+ (trained with ReAP) improves monotonically.

Critical strengths

Here I listed some bullet points that I believe are the strengths of the paper:

  • (S1) Promising Direction/Importance of the paper: LTSF is an impactful research field. The core direction of this paper of moving beyond simple disjoint past-to-future mappings by explicitly leveraging the overlap window to capture underlying system dynamics through Dynamic Mode Decomposition (DMD) is well-motivated and conceptually appealing.
  • (S2) Flexibility: Formulating ReAP as a training-time paradigm makes it flexible to integrate into existing deep learning backbones.
  • (S3) Substantial Experimental Effort and Reproducibility: I appreciate the authors' extensive effort in conducting experiments across seven standard real-world benchmarks, multiple prediction horizons, and different baseline families. Furthermore, providing detailed sets of hyperparameter tables (α\alpha and LoverlapL_{overlap}) configurations across datasets and horizons in Appendix C demonstrates transparency for reproducibility.
Critical weaknesses

I really appreciate the authors' effort on the paper. Here are some bullet points on the weaknesses of the paper:

Theoretically:

  • (W1): I am not convinced the gain comes from reconstructing the overlap. In the released code, ReLinear is a single nn.Linear(seq_len, pred_len + label_len) applied along time, with no normalization layer and no parameters shared across output steps. So Eqs. 27 and 28 are sums over output steps, and the overlap columns share no weights with the future columns. They cannot change the evaluated horizon, and with Adam, loss normalization only rescales α\alpha. What is left, up to normalization, is αd∥ΔX⊤X∥F2=αd∑jΔj⊤XX⊤Δj\frac{\alpha}{d}\Vert \Delta X^\top X\Vert_F^2 = \frac{\alpha}{d}\sum_j \Delta_j^\top XX^\top \Delta_j. This is a Gram-weighted squared error, not a DMD operator. In the experimental part, the tables point the same way: in Tables 9-12, ReLinear without the consistency loss is about on par with DLinear, and the pattern of the gain from the consistency loss seems to match what a regularizer would do.

  • (W2) I could not find a theorem that supports the central claim. The abstract says reconstructing the overlap "can enhance prediction performance", but nothing compares training with and without the overlap term, and there is no risk or generalization statement. Prop. 1 is the standard consistency of least squares, and it assumes stationarity, while Sec. 5 itself points to non-stationary dynamics. Lemma 1 is an exact SVD identity, and its interpretations are informal remarks. Prop. 2 bounds the surrogate against ∥ΔX⊤(Xk⊤)†∥F2\Vert \Delta X^\top (X_k^\top)^\dagger \Vert_F^2, but Eq. 17 is ∥ΔX X†∥F2\Vert \Delta X\, X^\dagger \Vert_F^2, which is a different quantity, and the paper never connects the two. The bound's constant is also (σ1/σk)4(\sigma_1/\sigma_k)^4, so it is weak for ill-conditioned windows.

Empirically:

  • (W3): The paper reports no seeds, standard deviations, or repeated runs, and many gains are small so they could be noise. ReAP is grid-searched over LoverlapL_{\text{overlap}} and α\alpha for each model, dataset and horizon, with α\alpha ranging from 10−810^{-8} to 1.5×1051.5\times10^{5} across Tables 3, 4 and 7 (about 13 orders of magnitude and about 7 for ReLinear alone), while the baselines use their official settings. The baselines also miss recent strong models, such as OLinear and TimeMixer++, and several works the paper cites are not compared, including Koopa, PatchTST, FITS(Citations are included below).

References

  • Yue, W., Liu, Y., Wang, H., Li, H., Ying, X., Guo, R., Xing, B., Shi, J. OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domain. NeurIPS 2025. arXiv:2505.08550.
  • Wang, S., Li, J., Shi, X., Ye, Z., Mo, B., Lin, W., Ju, S., Chu, Z., Jin, M. TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis. ICLR 2025. arXiv:2410.16032.
  • Liu, Y., Li, C., Wang, J., Long, M. Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors. NeurIPS 2023.
  • Nie, Y., Nguyen, N. H., Sinthong, P., Kalagnanam, J. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. ICLR 2023.
  • Xu, Z., Zeng, A., Xu, Q. FITS: Modeling Time Series with 10k Parameters. ICLR 2024.
Questions for authors

Questions for Authors

  • (Q1) In ReLinear, the overlap columns share no weights with the future columns. If you train with only the future outputs plus αd∥ΔX⊤X∥F2\frac{\alpha}{d}\Vert \Delta X^\top X\Vert_F^2 (no overlap reconstruction), do you get the same gain?
  • (Q2) Does ReAP beat ordinary regularization of the same linear layer (weight decay, ridge) when each is tuned with the same budget as α\alpha? Can DLinear get the same search over learning rate and weight decay?
  • (Q3) Can you report mean and standard deviation over several seeds for ReLinear and ReLinear+, since many gains are small?
  • (Q4) Is there any result, even for a linear model, showing that adding the overlap term lowers the forecasting risk? Prop. 2 bounds ∥ΔX⊤(Xk⊤)†∥F2\Vert \Delta X^\top (X_k^\top)^\dagger\Vert_F^2, but Eq. 17 is ∥ΔX X†∥F2\Vert \Delta X\, X^\dagger\Vert_F^2. How do you connect the two?
  • (Q5) Were α\alpha and LoverlapL_{\text{overlap}} selected on validation or on test? Can you also compare against OLinear, TimeMixer++, Koopa and PatchTST under the same tuning?
3 comments

We thank Reviewer 3 for your insightful comments and careful code inspection. Due to the 5,000-character limit per comment, we split our response into Part 1 (W1–W2), Part 2 (W3–Q2), and Part 3 (Q3–Q5).

W1. Overlap Coupling, Surrogate Role, & Ablated ReLinear

  • How Overlap Coupling Works: Implementing ReLinear as nn.Linear(seq_len, pred_len + label_len) forces a single linear map from seq_len to jointly predict the label_len overlap window (which reconstructs the tail of seq_len) together with pred_len. Because L^sur\hat{\mathcal{L}}_{\text{sur}} is computed over the concatenated trajectory label_len + pred_len, the pred_len output weights are directly coupled to the label_len reconstruction weights rather than being learned independently (Lines 287–288; Lemma 1).
  • Role of L^sur\hat{\mathcal{L}}_{\text{sur}} as a Dynamical Consistency Surrogate: We agree that L^sur=1n∥ΔXτ⊤X∥F2\hat{\mathcal{L}}_{\text{sur}} = \frac{1}{n}\Vert \Delta X_\tau^\top X\Vert_F^2 is not itself the DMD operator. Rather, as proven in Proposition 2, it is a pseudoinverse-free surrogate for the consistency loss L^consistent=1n∥ΔXτ⊤(X⊤)†∥F2\hat{\mathcal{L}}_{\text{consistent}} = \frac{1}{n}\Vert \Delta X_\tau^\top (X^\top)^\dagger\Vert_F^2, penalizing discrepancies between observed and model-predicted dynamics across deep backbones without unstable matrix inversion.
  • Why Ablated ReLinear Matches DLinear (Tables 9–12): ReLinear in Tables 9–12 is specifically an ablated baseline trained without L^sur\hat{\mathcal{L}}_{\text{sur}} (α=0\alpha = 0) to isolate the contribution of the consistency loss. Because α=0\alpha=0 reduces ReLinear to a standard linear projection, matching DLinear is expected, while ReLinear+ (α>0\alpha > 0) outperforming both directly confirms the effectiveness of dynamical consistency.

W2. Theoretical Chain, Eq. 17 vs. Prop. 2, & Condition Ratio (λ1/λk)2(\lambda_1/\lambda_k)^2

  • Connecting Corollary (Section 4): We will add an explicit connecting corollary in Section 4: Proposition 1 establishes local DMD operator consistency over short windows; Lemma 1 factorizes the reduced-order DMD transition operator as A^τ,k=(Uk⊤Uτ,k)Στ,k(Vτ,k⊤Vk)Σk−1\hat{A}_{\tau,k} = (U_k^\top U_{\tau,k})\Sigma_{\tau,k}(V_{\tau,k}^\top V_k)\Sigma_k^{-1}, showing that Loverlap>0L_{\text{overlap}} > 0 shortens the lag τ=Lhistory−Loverlap\tau = L_{\text{history}} - L_{\text{overlap}} and keeps temporal subspace alignment Vτ,k⊤VkV_{\tau,k}^\top V_k well-conditioned (lowering operator estimation variance); and Proposition 2 upper-bounds the true DMD consistency error via our practical training surrogate L^sur,k\hat{\mathcal{L}}_{\text{sur},k}.
  • Eq. 17 vs. Proposition 2 (Transpose Equivalence): In Eq. 17, Lconsistent=1d∥ΔXτX†∥F2\mathcal{L}_{\text{consistent}} = \frac{1}{d}\Vert \Delta X_\tau X^\dagger\Vert_F^2 is written in feature space, whereas in multi-horizon forecasting (Loverlap+Lfuture≠LhistoryL_{\text{overlap}} + L_{\text{future}} \neq L_{\text{history}}), XX and XτX_\tau have different temporal dimensions. Contracting across the shared variate dimension dd in the transpose space yields L^consistent,k=1n∥ΔXτ,k⊤(Xk⊤)†∥F2\hat{\mathcal{L}}_{\text{consistent},k} = \frac{1}{n}\Vert \Delta X_{\tau,k}^\top (X_k^\top)^\dagger\Vert_F^2 (Eq. 22). By unitary invariance of the Frobenius norm under SVD (Xk=UkΣkVk⊤X_k = U_k \Sigma_k V_k^\top), (Xk⊤)†(X_k^\top)^\dagger shares the exact spectrum Σk−1\Sigma_k^{-1} with Xk†X_k^\dagger, preserving spectral equivalence while resolving the horizon mismatch.
  • Empirical Condition Ratio (λ1/λk)2(\lambda_1/\lambda_k)^2: Across all 7 benchmarks (Lin=96,Lout=96,Lov=48L_{\text{in}}=96, L_{\text{out}}=96, L_{\text{ov}}=48), retaining dominant modes capturing >99.7%>99.7\% variance on ETT/Weather/ECL (k=2k=2) and 93.6%93.6\% on Traffic (k=3k=3) keeps (λ1/λk)2=(σ1/σk)4(\lambda_1/\lambda_k)^2 = (\sigma_1/\sigma_k)^4 tightly bounded within 1.8×1031.8\times 10^3 to 2.0×1042.0\times 10^4, whereas trailing noise modes (i>ki>k) diverge (λi→0\lambda_i \to 0). Weighting modes by λi2\lambda_i^2 in L^sur\hat{\mathcal{L}}_{\text{sur}} automatically damps this noisy tail without explicit rank truncation:
Metric / DatasetETTh1ETTh2ETTm1ETTm2WeatherECLTraffic
λ1\lambda_1 (Variance Ratio)99.1%99.8%99.4%99.7%99.8%97.9%83.9%
λ2\lambda_2 (Variance Ratio)0.7%0.1%0.5%0.1%0.1%1.8%7.8%
λ3\lambda_3 (Variance Ratio)0.06%0.04%0.03%0.04%0.04%0.1%1.9%
Retained Rank kk (Cum. Energy)22 (99.8%)22 (99.9%)22 (99.9%)22 (99.8%)22 (99.9%)22 (99.7%)33 (93.6%)
Bound Ratio (λ1/λk)2(\lambda_1/\lambda_k)^23.7×1033.7\times 10^32.0×1042.0\times 10^49.5×1039.5\times 10^31.5×1041.5\times 10^48.6×1038.6\times 10^35.4×1035.4\times 10^31.8×1031.8\times 10^3

###Part 2 (W3–Q2)

W3. Multi-Seed Evaluation, TimeMixer++, & Tuning Protocol

  • TimeMixer++ Clarification: Our experiments use Time-Series-Library/models/TimeMixer.py, which implements the TimeMixer++ architecture (identical to the official TimeMixer codebase); thus, the results reported as "TimeMixer" in our paper are actually TimeMixer++. We will update the model name and clarify this explicitly in the revised manuscript.
  • 3-Seed Runs Across 5 Backbones: Across 3 random seeds and all 7 datasets (35 pairs; full results in Appendix H), ReAP achieves mean reductions of 2.99% (MSE) and 2.37% (MAE), substantially exceeding run-to-run standard deviations (0.27 pp and 0.16 pp):
MetricDLinearPatchTSTKoopaOLinearFITS
MSE Improve (%)3.38±0.29 pp3.38 \pm 0.29\text{ pp}3.63±0.48 pp3.63 \pm 0.48\text{ pp}1.55±0.09 pp1.55 \pm 0.09\text{ pp}0.75±0.51 pp0.75 \pm 0.51\text{ pp}0.49±0.00 pp0.49 \pm 0.00\text{ pp}
MAE Improve (%)3.00±0.24 pp3.00 \pm 0.24\text{ pp}2.23±0.25 pp2.23 \pm 0.25\text{ pp}0.96±0.03 pp0.96 \pm 0.03\text{ pp}0.34±0.27 pp0.34 \pm 0.27\text{ pp}1.63±0.00 pp1.63 \pm 0.00\text{ pp}

Q1. Future-Only L^sur\hat{\mathcal{L}}_{\text{sur}} (Loverlap=0L_{\text{overlap}}=0) vs. Overlap Reconstruction (Loverlap>0L_{\text{overlap}}>0)

No—setting Loverlap=0L_{\text{overlap}}=0 (Linear+) does not reproduce ReAP's gains and degrades long-horizon scaling. Averaged across all 7 datasets and prediction horizons {96,192,336,720}\{96, 192, 336, 720\} (Appendix I), enforcing L^sur\hat{\mathcal{L}}_{\text{sur}} without overlap (Linear+) worsens MSE by -2.50% (Lin=336L_{\text{in}}=336) and -5.51% (Lin=720L_{\text{in}}=720) vs. vanilla Linear, whereas ReLinear+ (Loverlap>0L_{\text{overlap}}>0) outperforms Linear+ by 8.54% MSE (11.70% at Lin=720L_{\text{in}}=720):

ComparisonMetricLin=96L_{\text{in}}=96Lin=192L_{\text{in}}=192Lin=336L_{\text{in}}=336Lin=720L_{\text{in}}=720Across Input Lengths
ReLinear+ vs. Linear+MSE / MAE6.45 / 5.955.38 / 4.7810.62 / 9.6111.70 / 9.618.54 / 7.49
ReLinear+ vs. LinearMSE / MAE6.49 / 5.056.29 / 5.158.44 / 6.977.32 / 5.857.14 / 5.76
Linear+ vs. LinearMSE / MAE0.05 / -0.981.07 / 0.38-2.50 / -3.11-5.51 / -4.32-1.72 / -2.01

Consistent with Lemma 1, Loverlap=0L_{\text{overlap}}=0 maximizes the shift lag τ=Lhistory\tau = L_{\text{history}}; at large LhistoryL_{\text{history}}, temporal decorrelation causes Vτ,k⊤VkV_{\tau,k}^\top V_k to become ill-conditioned, making the unanchored consistency penalty noisy. Reconstructing the overlap (Loverlap>0L_{\text{overlap}}>0) shortens τ=Lhistory−Loverlap\tau = L_{\text{history}} - L_{\text{overlap}} and anchors the trajectory to the historical boundary.

Q2. Matched-Budget Ridge, Weight Decay, & Learning Rate Sweeps (ETTh1, 96→9696 \to 96)

Under a fixed random seed, batch size =64= 64, and official lr=1×10−3\text{lr} = 1\times 10^{-3} (or wd=0\text{wd}=0 for LR sweeps), ReLinear+ (MSE = 0.384, MAE = 0.398) and DLinear+ (MSE = 0.382, MAE = 0.394) outperform all 17 baseline configurations:

1. Explicit Ridge Sweep (LMSE+α∥W∥F2\mathcal{L}_{\text{MSE}} + \alpha \Vert W \Vert_F^2, lr=10−3,wd=0\text{lr}=10^{-3}, \text{wd}=0):

ModelMetric10−610^{-6}10−510^{-5}10−410^{-4}5×10−45\times 10^{-4}10−310^{-3}5×10−35\times 10^{-3}
Linear+RidgeMSE / MAE.413 / .424.412 / .423.411 / .423.410 / .423.412 / .424.446 / .445

2. Adam Weight Decay Sweep (lr=10−3\text{lr}=10^{-3}):

ModelMetric0010−510^{-5}10−410^{-4}5×10−45\times 10^{-4}10−310^{-3}10−210^{-2}
LinearMSE / MAE.400 / .415.400 / .415.400 / .415.400 / .415.400 / .415.410 / .422
DLinearMSE / MAE.392 / .408.392 / .408.392 / .408.392 / .408.392 / .408.397 / .415

3. Learning Rate Sweep (wd=0\text{wd}=0):

ModelMetric10−410^{-4}5×10−45\times 10^{-4}10−310^{-3}5×10−35\times 10^{-3}10−210^{-2}
LinearMSE / MAE.603 / .517.433 / .437.400 / .415.396 / .401.395 / .398
DLinearMSE / MAE.411 / .422.397 / .400.392 / .408.387 / .399.389 / .402

Unlike isotropic L2L_2 penalties (α∥W∥F2\alpha\Vert W\Vert_F^2) that shrink weights uniformly toward zero (degrading at α=5×10−3\alpha=5\times 10^{-3} or wd=10−2\text{wd}=10^{-2}), L^sur\hat{\mathcal{L}}_{\text{sur}} anchors transitions to empirical cross-covariances along dominant singular modes while preserving forecast amplitude.

Part 3 (Q3–Q5)

Q3. Multi-Seed Evaluation of ReLinear+ vs. ReLinear (Lin=96L_{\text{in}}=96, 3 Seeds)

Averaged across Lfuture∈{96,192,336,720}L_{\text{future}} \in \{96, 192, 336, 720\} (Appendix J), ReLinear+ reduces dataset-averaged MSE by 4.32% (0.393→0.3760.393 \to 0.376) and MAE by 3.32% (0.385→0.3720.385 \to 0.372) with lower variance (mean std 0.0029 vs. 0.0049 for MSE):

ModelMetricETTh1ETTh2ETTm1ETTm2WeatherECLTraffic
ReLinear+MSE.448±.002\mathbf{.448}_{\pm.002}.395±.007\mathbf{.395}_{\pm.007}.400±.003\mathbf{.400}_{\pm.003}.289±.005\mathbf{.289}_{\pm.005}.264±.003\mathbf{.264}_{\pm.003}.209±.000\mathbf{.209}_{\pm.000}.624±.000\mathbf{.624}_{\pm.000}
ReLinear+MAE.442±.001\mathbf{.442}_{\pm.001}.420±.005\mathbf{.420}_{\pm.005}.412±.003\mathbf{.412}_{\pm.003}.337±.006\mathbf{.337}_{\pm.006}.317±.003\mathbf{.317}_{\pm.003}.291±.000\mathbf{.291}_{\pm.000}.384±.000\mathbf{.384}_{\pm.000}
ReLinearMSE.456±.004.456_{\pm.004}.452±.005.452_{\pm.005}.410±.002.410_{\pm.002}.308±.017.308_{\pm.017}.267±.003.267_{\pm.003}.216±.001.216_{\pm.001}.640±.002.640_{\pm.002}
ReLinearMAE.450±.004.450_{\pm.004}.440±.002.440_{\pm.002}.416±.004.416_{\pm.004}.360±.012.360_{\pm.012}.318±.006.318_{\pm.006}.305±.002.305_{\pm.002}.403±.002.403_{\pm.002}

Q4. Lowering Linear Risk & Connecting Eq. 17 to Proposition 2

  1. Lowering Linear Forecasting Risk: Across all 7 datasets and horizons {96,192,336,720}\{96, 192, 336, 720\}, ReLinear+ (Lov>0L_{\text{ov}}>0) reduces error over vanilla Linear by 7.14% MSE (5.76% MAE) and over Linear+ (Lov=0L_{\text{ov}}=0) by 8.54% MSE. Theoretically, Lemma 1 proves Loverlap>0L_{\text{overlap}}>0 shortens τ=Lhistory−Loverlap\tau = L_{\text{history}} - L_{\text{overlap}}, keeping Vτ,k⊤VkV_{\tau,k}^\top V_k well-conditioned and lowering operator estimation variance.
  2. Connecting Eq. 17 (Lconsistent\mathcal{L}_{\text{consistent}}) and Proposition 2 (L^consistent,k\hat{\mathcal{L}}_{\text{consistent},k}): Contracting across shared variates dd in the transpose space resolves the temporal dimension mismatch when Lhistory≠Loverlap+LfutureL_{\text{history}} \neq L_{\text{overlap}} + L_{\text{future}}. On the rank-kk SVD subspace (Xk=UkΣkVk⊤X_k = U_k \Sigma_k V_k^\top), both losses bound the identical projected residual C:=Uk⊤ΔXτVkC := U_k^\top \Delta X_\tau V_k via σ1−2∥C∥F2≤dLconsistent,k\sigma_1^{-2}\Vert C\Vert_F^2 \le d\mathcal{L}_{\text{consistent},k} and nL^consistent,k≤σk−2∥C∥F2n\hat{\mathcal{L}}_{\text{consistent},k} \le \sigma_k^{-2}\Vert C\Vert_F^2 with shared zero-error optimum (C=0C=0), while L^sur\hat{\mathcal{L}}_{\text{sur}} weights modes by λi2\lambda_i^2 to damp noisy tail modes.

Q5. Validation-Only Selection & Backbone Comparisons

All (α,Loverlap)(\alpha, L_{\text{overlap}}) hyperparameters are selected strictly on validation set MSE, with the test set held out for a single final evaluation under identical chronological splits and history–future boundaries. Under the exact same validation-only tuning protocol, ReAP improves OLinear, Koopa, PatchTST (Part 1, W3), and TimeMixer++ (Table 7).

Reviewer_WZXLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReAP, a training scheme for long-term time-series forecasting. The model's output window starts inside its input window, so the model also reconstructs the last part of its input (the "overlap") before forecasting the horizon. Training adds a "consistent loss" motivated by Dynamic Mode Decomposition (DMD). A single linear layer trained this way ("ReLinear") is reported to be competitive with recent baselines, and adding ReAP to five deep backbones is reported to improve them consistently.

Critical strengths
  • The overall method is simple and architecture-agnostic.
  • Experimental results are reported in enough detail.
Critical weaknesses
  • It is unclear how the theory shows that reconstruction helps, and the bound does not match the trained loss.
  • From the paper, hyperparameters are tuned unequally on an unstated validate split, and the overlap is not properly ablated.
Questions for authors
  • Given the 8 of 84 Table 2 cells without improvement, would you restate "consistently outperform their default counterparts across all benchmark datasets and prediction horizons" (L371-373)?

  • Which step of the theory links the overlap length to forecast error?

  • On which split were the overlap and α selected? From the Figure 3, the minima equal the Table 11 test numbers.

1 comment

We thank Reviewer 2 for your constructive comments.

W1. How Overlap Reconstruction Helps & How the Bound Matches the Trained Loss

  • How Reconstruction Helps (Lemma 1): For snapshot matrices XnX_n and Xτ,nX_{\tau,n} with lag τ=Lhistory−Loverlap\tau = L_{\text{history}} - L_{\text{overlap}}, Lemma 1 factorizes the reduced-order DMD operator as A^τ,k=(Uk⊤Uτ,k)Στ,k(Vτ,k⊤Vk)Σk−1\hat{A}_{\tau,k} = (U_k^\top U_{\tau,k})\Sigma_{\tau,k}(V_{\tau,k}^\top V_k)\Sigma_k^{-1}. When Loverlap=0L_{\text{overlap}} = 0, temporal decorrelation drives canonical angle cosines in Vτ,k⊤VkV_{\tau,k}^\top V_k toward zero, inflating operator estimation variance. Supervising overlap reconstruction (Loverlap>0L_{\text{overlap}} > 0) preserves temporal coherence and spatial alignment, stabilizing transitions across the shared subspace.
  • Bound-to-Loss Equivalence (Eq. 17, Eq. 18, & Prop. 2): Classical DMD (Eq. 17, Lconsistent=1d∥ΔXτX†∥F2\mathcal{L}_{\text{consistent}} = \frac{1}{d}\Vert \Delta X_\tau X^\dagger\Vert_F^2) requires identical snapshot counts nn, whereas multi-horizon forecasting (Lhistory≠Loverlap+LfutureL_{\text{history}} \neq L_{\text{overlap}} + L_{\text{future}}, Eq. 25–28) makes ΔXτX†\Delta X_\tau X^\dagger dimensionally incompatible. Contracting across shared variates dd in the transpose space yields L^consistent,k=1n∥ΔXτ⊤(Xk⊤)†∥F2\hat{\mathcal{L}}_{\text{consistent},k} = \frac{1}{n}\Vert \Delta X_\tau^\top (X_k^\top)^\dagger\Vert_F^2 (Eq. 22) and our trained surrogate L^sur=1n∥ΔXτ⊤X∥F2\hat{\mathcal{L}}_{\text{sur}} = \frac{1}{n}\Vert \Delta X_\tau^\top X\Vert_F^2 (Eq. 18). On the rank-kk SVD subspace (Xk=UkΣkVk⊤X_k = U_k \Sigma_k V_k^\top), Eq. 17 and Eq. 22 bound the identical projected residual C:=Uk⊤ΔXτVkC := U_k^\top \Delta X_\tau V_k with shared zero-error optimum (C=0C = 0), while Proposition 2 (Eq. 24) bounds L^consistent,k\hat{\mathcal{L}}_{\text{consistent},k} from above and below via L^sur,k\hat{\mathcal{L}}_{\text{sur},k}. Empirically (Global Item 2), retaining dominant modes (>99.7%>99.7\% variance on ETT/Weather/ECL, k=2k=2; 93.6%93.6\% on Traffic, k=3k=3) keeps (λ1/λk)2∈[1.8×103,2.0×104](\lambda_1/\lambda_k)^2 \in [1.8\times 10^3, 2.0\times 10^4], whereas trailing noise modes (i>ki>k) diverge (λi→0\lambda_i \to 0)—showing why L^sur\hat{\mathcal{L}}_{\text{sur}} (weighting modes by λi2\lambda_i^2) damps noise without explicit rank truncation.

W2. Identical Splits, Fair Tuning, & LoverlapL_{\text{overlap}} Ablation

  • Splits & Fair Tuning: ReAP preserves standard chronological splits, LhistoryL_{\text{history}}, and LfutureL_{\text{future}}; it only extends the predicted output window backward by LoverlapL_{\text{overlap}} during training (discarded at inference), keeping the history–future boundary identical across baselines. All architectural and optimization hyperparameters stay at official defaults, tuning only (α,Loverlap)(\alpha, L_{\text{overlap}}) on validation.
  • Selection & Ablation of LoverlapL_{\text{overlap}}: Selected LoverlapL_{\text{overlap}} values appear in Tables 5–6, Table 8 (Appendix C), and Table 17 (Appendix I), and we ablate Loverlap=0L_{\text{overlap}} = 0 vs. Loverlap>0L_{\text{overlap}} > 0 in Global Item 4. Treating the overlap as a structured state-space representation is a promising future direction (Line 485).

Q1. Qualifying "All" to "Vast Majority" (Lines 371–373) Thank you for catching this overstatement. We will revise Lines 371–373 to "consistently outperform their default counterparts across the vast majority of benchmark datasets and prediction horizons." In Table 13 (Appendix D, aggregated in Table 2), ReAP underperforms the baseline in both MSE and MAE in only 11 of 175 model–dataset–horizon settings, primarily where official settings are tightly tuned and joint (α,Loverlap)(\alpha, L_{\text{overlap}}) optimization can over-regularize (Remark 5).

Q2. Theoretical Chain Linking LoverlapL_{\text{overlap}} to Forecast Error

  1. Proposition 1: Extending predictions backward by LoverlapL_{\text{overlap}} shifts the lag to τ=Lhistory−Loverlap\tau = L_{\text{history}} - L_{\text{overlap}}, coupling historical reconstruction and future prediction under one operator.
  2. Lemma 1: Shortening τ\tau via Loverlap>0L_{\text{overlap}} > 0 prevents temporal decorrelation (σmin⁡(Vτ,k⊤Vk)→0\sigma_{\min}(V_{\tau,k}^\top V_k) \to 0), lowering finite-sample operator estimation variance.
  3. Proposition 2: Minimizing L^sur,k\hat{\mathcal{L}}_{\text{sur},k} upper-bounds the true DMD operator consistency error L^consistent,k\hat{\mathcal{L}}_{\text{consistent},k} (up to (λ1/λk)2(\lambda_1/\lambda_k)^2), tightening out-of-sample forecast error.

Q3. Validation-Only Selection & Figure 3 Correspondence

  • Validation Selection: Both α\alpha and LoverlapL_{\text{overlap}} were selected strictly on the validation set and never tuned on test data; test metrics are computed solely over LfutureL_{\text{future}}.
  • Figure 3: Subplots 1–4 of Figure 3 directly plot Tables 9–12, respectively. We will state this explicitly in Figure 3's caption and Section 6.3.
Reviewer_SUQnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReAP, a plug-in training scheme for time-series forecasting in which the model also reconstructs the part of the history that overlaps with the forecast window, with an extra consistency loss motivated by Dynamic Mode Decomposition. It evaluates ReAP with a single linear layer and as an add-on on standard long-term forecasting benchmarks. ReAP gives small improvements for most backbones and datasets, and the linear model keeps improving as the input history grows.

Critical strengths
  1. The work explores a useful connection between dynamical modeling and training objectives for forecasting.
  2. The proposed scheme is simple and broadly applicable. It adapts the output windows and training objectives of existing forecasters without requiring a redesign of their core backbones.
  3. The evaluation is broad, covering the standard long-term forecasting benchmarks, diverse backbones, and several input lengths.
  4. A single linear layer trained with ReAP is competitive with more complex baselines on several datasets, particularly with longer input histories. This is a useful practical result.
Critical weaknesses
  1. The theory analyzes DMD estimation but does not show that overlap reconstruction lowers forecast error. It would help to explain this connection. The loss derivation also needs clarification. Eq. 17 uses ∣ΔXτX†∣F2\\|\Delta X_\tau X^\dagger\\|_F^2, whereas Proposition 2 bounds ∣ΔXτ⊤(Xk⊤)†∣F2\\|\Delta X_\tau^\top (X_k^\top)^\dagger\\|_F^2 (Eq. 22). These norms generally differ even when the full matrix is used (Xk=XX_k=X): transposing the product in Eq. 17 gives (X†)⊤ΔXτ⊤(X^\dagger)^\top \Delta X_\tau^\top, with the factors in the opposite order from Eq. 22. Please clarify how Proposition 2 justifies replacing the loss in Eq. 17 with the surrogate in Eq. 18.

  2. Results over several seeds and comparable tuning budgets would make the small gains more convincing. Section 6.3 mentions validation selection, but please specify how overlap and α were tuned. The improvement claims could also acknowledge exceptions, such as TimeMixer's worse results on ECL (Table 2).

  3. A comparison with Informer and Autoformer's use of historical decoder context, and with N-BEATS backcasting, would help clarify what ReAP adds. These mechanisms differ from ReAP's explicit reconstruction loss, so explaining the distinction would strengthen the novelty discussion.

  4. The scaling claims could be more precise. DLinear also benefits from longer histories (Table 1), and ReLinear sometimes worsens as history grows (Tables 11 and 12). Please qualify the claims of monotonic improvement and of this behavior being specific to ReAP.

  5. The related-work discussion would benefit from including Attraos: Attractor Memory for LTSF (NeurIPS 2024), which uses phase-space reconstruction, and LETS Forecast: Learning Embedology for TSF (ICML 2025), which uses time-delay embeddings. Both provide relevant comparisons for ReAP's approach to modeling dynamics.

Questions for authors

See weaknesses above.

1 comment

We thank Reviewer 1 for your expertise and insightful comments.

W1. Dimensional Compatibility & How Overlap Lowers Forecast Error

  • Dimensional Compatibility (Eq. 17 →\to Eq. 18 / Prop. 2): Classical DMD (Eq. 17, Lconsistent=1d∥ΔXτX†∥F2\mathcal{L}_{\text{consistent}} = \frac{1}{d}\|\Delta X_\tau X^\dagger\|_F^2) requires identical snapshot counts nn, causing a mismatch when Lhistory≠Loverlap+LfutureL_{\text{history}} \neq L_{\text{overlap}} + L_{\text{future}}. Contracting across the shared variate dimension dd in the transpose space yields L^consistent,k=1n∥ΔXτ⊤(Xk⊤)†∥F2\hat{\mathcal{L}}_{\text{consistent},k} = \frac{1}{n}\|\Delta X_\tau^\top (X_k^\top)^\dagger\|_F^2 (Eq. 22) and our surrogate L^sur=1n∥ΔXτ⊤X∥F2\hat{\mathcal{L}}_{\text{sur}} = \frac{1}{n}\|\Delta X_\tau^\top X\|_F^2 (Eq. 18), preserving spectral equivalence while enabling plug-in use across arbitrary horizons. We will add a Remark to Proposition 2.
  • Why Overlap Reduces Error (Lemma 1): Lemma 1 factorizes the reduced-order DMD operator as A^τ,k=(Uk⊤Uτ,k)Στ,k(Vτ,k⊤Vk)Σk−1\hat{A}_{\tau,k} = (U_k^\top U_{\tau,k})\Sigma_{\tau,k}(V_{\tau,k}^\top V_k)\Sigma_k^{-1} with effective lag τ=Lhistory−Loverlap\tau = L_{\text{history}} - L_{\text{overlap}}. When Loverlap=0L_{\text{overlap}} = 0, Xτ,nX_{\tau,n} shares no snapshots with XnX_n; temporal decorrelation drives σmin⁡(Vτ,k⊤Vk)→0\sigma_{\min}(V_{\tau,k}^\top V_k) \to 0, inflating operator estimation variance. Supervising overlap reconstruction (Loverlap>0L_{\text{overlap}} > 0) preserves temporal subspace coherence (σi(Vτ,k⊤Vk)≈1\sigma_i(V_{\tau,k}^\top V_k) \approx 1) and spatial alignment (Uk⊤Uτ,kU_k^\top U_{\tau,k}), stabilizing transitions across the shared subspace.

W2. Multi-Seed Significance, Tuning Protocol, & TimeMixer on ECL

  • Statistical Significance: Across 3 seeds and 35 model–dataset pairs (Global Item 3; Appendix H), ReAP achieves mean reductions of 2.99% (MSE) and 2.37% (MAE), far exceeding mean stds (0.27 pp and 0.16 pp).
  • Hyperparameter Protocol: α\alpha (training loss weight) and LoverlapL_{\text{overlap}} (fixed output window extension, discarded at inference) are selected via grid search strictly on the validation set under official early-stopping/optimization protocols. Exploring adaptive α\alpha scheduling is noted as future work (Section 7).
  • TimeMixer on ECL: We will explicitly note this exception in the text. As shown in Table 13 (Appendix D), TimeMixer+ outperforms TimeMixer at longer horizons (Lfuture∈{336,720}L_{\text{future}} \in \{336, 720\}) but degrades at short horizons (96,19296, 192) where official hyperparameters are tightly tuned and joint optimization with (α,Loverlap)(\alpha, L_{\text{overlap}}) can over-regularize (Remark 5).

W3. Comparison with N-BEATS, Autoformer, and Informer We will add a dedicated comparison in the Related Work / Discussion section:

  • N-BEATS: Uses dual branches (backcast and forecast heads) to subtract explained input signals (xℓ=xℓ−1−x^ℓ−1x_\ell = x_{\ell-1} - \hat{x}_{\ell-1}). ReAP uses a single unified prediction head and couples historical and future spectral dynamics via an explicit overlap consistency loss.
  • Autoformer: Embeds period-based autocorrelation RXX(τ)\mathcal{R}_{\mathcal{X}\mathcal{X}}(\tau) and top-kk lag aggregation inside its attention blocks, whereas ReAP imposes an architecture-agnostic dynamical consistency loss across the overlap boundary.
  • Informer: Samples an earlier LtokenL_{\text{token}} start-token slice before the target sequence during generative decoding to anchor long-range dependencies, conceptually paralleling ReAP's LoverlapL_{\text{overlap}} window.

W4. Linear Baselines & "Nearly Monotonic" Scaling

  • Linear Baselines: We evaluate a naive single linear layer across Lhistory∈{96,192,336,720}L_{\text{history}} \in \{96, 192, 336, 720\} in Appendix I; while both Linear and DLinear improve with longer look-backs, ReLinear+ consistently outperforms both (by 7.14% MSE on average over Linear). Note that ReLinear (Tables 11–12) is the ablated variant without L^sur\hat{\mathcal{L}}_{\text{sur}} (α=0\alpha=0).
  • "Nearly Monotonic": We agree and will revise "monotonic" to "nearly monotonic" throughout the manuscript, explicitly noting the two minor fluctuations in Fig. 2 / Appendix E (ETTm1 at Lfuture=720L_{\text{future}}=720 from Lin=192→336L_{\text{in}}=192 \to 336, and ETTm2 at Lfuture=96L_{\text{future}}=96 from Lin=336→720L_{\text{in}}=336 \to 720).

W5. Related Work on Dynamical Systems (Attraos & LETS) We will incorporate both works into our DMD discussion in Related Work: Attraos leverages Takens's delay-coordinate embedding to reconstruct hidden phase-space attractor dynamics (conceptually related to Koopman/DMD theory), while LETS maps delay-embedded state vectors into a latent space to kernel-weight historical successors before decoding.

Comments on the paper

6 comments

We sincerely thank the Area Chair and all three reviewers for their constructive feedback and for recognizing the core strengths of our work.

1. Connecting Spatial DMD (Eq. 17) to Transpose-Space Surrogate (Eq. 18 & Prop. 2) Classical DMD (Eq. 17, Lconsistent=1d∥ΔXτX†∥F2\mathcal{L}_{\text{consistent}} = \frac{1}{d}\Vert \Delta X_\tau X^\dagger\Vert_F^2) requires input and output snapshots of identical length nn (X,Xτ∈Rd×nX, X_\tau \in \mathbb{R}^{d \times n}), causing a horizon mismatch when Lhistory≠Loverlap+LfutureL_{\text{history}} \neq L_{\text{overlap}} + L_{\text{future}}. Contracting across the shared variate dimension dd in the transpose space yields L^consistent,k=1n∥ΔXτ⊤(Xk⊤)†∥F2\hat{\mathcal{L}}_{\text{consistent},k} = \frac{1}{n}\Vert \Delta X_\tau^\top (X_k^\top)^\dagger\Vert_F^2 (Eq. 22) and our pseudoinverse-free surrogate L^sur=1n∥ΔXτ⊤X∥F2\hat{\mathcal{L}}_{\text{sur}} = \frac{1}{n}\Vert \Delta X_\tau^\top X\Vert_F^2 (Eq. 18). Under SVD (Xk=UkΣkVk⊤X_k = U_k \Sigma_k V_k^\top), (Xk⊤)†(X_k^\top)^\dagger shares the exact spectrum Σk−1\Sigma_k^{-1} with Xk†X_k^\dagger, preserving spectral equivalence while supporting arbitrary input/output horizons across deep backbones. We will add a Remark to Proposition 2.

2. Empirical Spectral Conditioning of Proposition 2 Retaining dominant modes capturing >99.7%>99.7\% variance on ETT, Weather, and ECL (k=2k=2) and 93.6%93.6\% on Traffic (k=3k=3) keeps the relative bound ratio (λ1/λk)2(\lambda_1/\lambda_k)^2 tightly bounded within 1.8×1031.8\times 10^3 to 2.0×1042.0\times 10^4 (Table 1; full horizons in Appendix G). Conversely, trailing noise modes (i>ki>k) have λi→0\lambda_i \to 0 and (λ1/λi)2→∞(\lambda_1/\lambda_i)^2 \to \infty. Our surrogate L^sur\hat{\mathcal{L}}_{\text{sur}} weights modes by λi2\lambda_i^2 instead of λi−2\lambda_i^{-2}, automatically damping the noisy tail without explicit rank truncation.

Metric (Lin=96,Lout=96,Lov=48L_{\text{in}}=96, L_{\text{out}}=96, L_{\text{ov}}=48)ETTh1ETTh2ETTm1ETTm2WeatherECLTraffic
λ1\lambda_1 Variance Ratio99.1%99.8%99.4%99.7%99.8%97.9%83.9%
λ2\lambda_2 Variance Ratio0.7%0.1%0.5%0.1%0.1%1.8%7.8%
λ3\lambda_3 Variance Ratio0.06%0.04%0.03%0.04%0.04%0.1%1.9%
Retained Rank kk (Cum. Energy)22 (99.8%)22 (99.9%)22 (99.9%)22 (99.8%)22 (99.9%)22 (99.7%)33 (93.6%)
Bound Ratio (λ1/λk)2(\lambda_1/\lambda_k)^23.7×1033.7\times 10^32.0×1042.0\times 10^49.5×1039.5\times 10^31.5×1041.5\times 10^48.6×1038.6\times 10^35.4×1035.4\times 10^31.8×1031.8\times 10^3

3. Multi-Seed Evaluation Across Backbones (3 Seeds, Full Results in Appendix H) Across 35 model–dataset pairs (7 datasets), ReAP yields a mean reduction of 2.99% (MSE) and 2.37% (MAE), well above the mean run-to-run stds of 0.27 pp and 0.16 pp:

MetricDLinearPatchTSTKoopaOLinearFITS
MSE Improve (%)3.38±0.29 pp3.38 \pm 0.29\text{ pp}3.63±0.48 pp3.63 \pm 0.48\text{ pp}1.55±0.09 pp1.55 \pm 0.09\text{ pp}0.75±0.51 pp0.75 \pm 0.51\text{ pp}0.49±0.00 pp0.49 \pm 0.00\text{ pp}
MAE Improve (%)3.00±0.24 pp3.00 \pm 0.24\text{ pp}2.23±0.25 pp2.23 \pm 0.25\text{ pp}0.96±0.03 pp0.96 \pm 0.03\text{ pp}0.34±0.27 pp0.34 \pm 0.27\text{ pp}1.63±0.00 pp1.63 \pm 0.00\text{ pp}

4. Overlap Ablation: ReLinear+ (Lov>0L_{\text{ov}}>0) vs. Linear+ (Lov=0L_{\text{ov}}=0 with L^sur\hat{\mathcal{L}}_{\text{sur}}) vs. Linear Averaged across all 7 datasets and horizons {96,192,336,720}\{96, 192, 336, 720\} (Appendix I), enforcing L^sur\hat{\mathcal{L}}_{\text{sur}} without overlap (Linear+) hurts long-horizon scaling, whereas ReLinear+ reduces MSE by 8.54% over Linear+ and 7.14% over Linear:

ComparisonMetricLin=96L_{\text{in}}=96192192336336720720Avg
ReLinear+ vs. Linear+MSE / MAE6.45 / 5.955.38 / 4.7810.62 / 9.6111.70 / 9.618.54 / 7.49
ReLinear+ vs. LinearMSE / MAE6.49 / 5.056.29 / 5.158.44 / 6.977.32 / 5.857.14 / 5.76
Linear+ vs. LinearMSE / MAE0.05 / -0.981.07 / 0.38-2.50 / -3.11-5.51 / -4.32-1.72 / -2.01

5. Multi-Seed ReLinear+ vs. ReLinear (Lin=96L_{\text{in}}=96, 3 Seeds, Appendix J) ReLinear+ reduces dataset-averaged MSE by 4.32% (0.393→0.3760.393 \to 0.376) and MAE by 3.32% (0.385→0.3720.385 \to 0.372) with lower variance (mean std 0.0029 vs. 0.0049 for MSE).

We sincerely thank Reviewer 3 for this thorough, incisive critique. You raise valid mathematical points regarding our previous explanation, and we appreciate the opportunity to align our response with the exact structure of Sections 4 and 5 in the manuscript.

1. Equal-Window Theory in Section 4 (n=mn = m) vs. Unequal-Horizon Training in Section 5 (n≠mn \neq m)

Let dd denote the variate dimension, nn the input history length (X∈Rd×nX \in \mathbb{R}^{d \times n}), and mm the output window length (m=Loverlap+Lfuturem = L_{\mathrm{overlap}} + L_{\mathrm{future}}, with shifted target trajectory Yτ∈Rd×mY_\tau \in \mathbb{R}^{d \times m}). Throughout Section 4, Lconsistent\mathcal{L}_{\mathrm{consistent}} is formulated strictly in the equal-window DMD setting (n=mn = m, using X,Xτ∈Rd×nX, X_\tau \in \mathbb{R}^{d \times n}) without introducing YτY_\tau (n≠mn \neq m). Rather than claiming an algebraic reformulation of Eq. 17, L^consistent,k\widehat{\mathcal{L}}_{\mathrm{consistent},k} and L^sur\widehat{\mathcal{L}}_{\mathrm{sur}} resolve the temporal dimension mismatch by formulating least-squares operator consistency in the temporal (row) space rather than the variate (column) space:

  • Variate-Space Operator Consistency (Eq. 17, defined only for n=mn = m): Standard DMD estimates a left-acting variate transition matrix Aτ∈Rd×dA_\tau \in \mathbb{R}^{d \times d} satisfying Xτ≈AτXX_\tau \approx A_\tau X, giving the empirical operator Aτ=XτX†A_\tau = X_\tau X^\dagger and model-estimated operator A^τ=X^τX†\widehat{A}_\tau = \widehat{X}_\tau X^\dagger. Writing ΔXτ:=Xτ−X^τ∈Rd×n\Delta X_\tau := X_\tau - \widehat{X}_\tau \in \mathbb{R}^{d \times n}, Eq. 17 measures their discrepancy in variate space:

Lconsistent:=1n∥Aτ−A^τ∥F2=1n∥ΔXτX†∥F2,ΔXτX†∈Rd×n⋅Rn×d=Rd×d,\mathcal{L}_{\mathrm{consistent}} := \frac{1}{n}\lVert A_\tau - \widehat{A}_\tau \rVert_F^2 = \frac{1}{n}\lVert \Delta X_\tau X^\dagger \rVert_F^2, \qquad \Delta X_\tau X^\dagger \in \mathbb{R}^{d \times n} \cdot \mathbb{R}^{n \times d} = \mathbb{R}^{d \times d},

which contracts across time and is defined only when n=mn = m (for n≠mn \neq m, ΔYτX†∈Rd×m⋅Rn×d\Delta Y_\tau X^\dagger \in \mathbb{R}^{d \times m} \cdot \mathbb{R}^{n \times d} is dimensionally undefined).

  • Temporal-Space Operator Consistency (L^consistent,k\widehat{\mathcal{L}}_{\mathrm{consistent},k} in Section 4, and Extension to YτY_\tau in Section 5): Transposing the matrices reverses the contraction order so that the inner product is taken over the shared variate dimension dd rather than time. In Section 4 (n=mn = m), this yields L^consistent,k=1n∥ΔXτ⊤(Xk⊤)†∥F2\widehat{\mathcal{L}}_{\mathrm{consistent},k} = \frac{1}{n}\lVert \Delta X_\tau^\top (X_k^\top)^\dagger \rVert_F^2. When extending to unequal horizons (n≠mn \neq m) in Section 5 with ΔYτ:=Yτ−Y^τ∈Rd×m\Delta Y_\tau := Y_\tau - \widehat{Y}_\tau \in \mathbb{R}^{d \times m}, both L^consistent,k\widehat{\mathcal{L}}_{\mathrm{consistent},k} and our pseudoinverse-free surrogate L^sur\widehat{\mathcal{L}}_{\mathrm{sur}} compute products of shape Rm×d⋅Rd×n=Rm×n\mathbb{R}^{m \times d} \cdot \mathbb{R}^{d \times n} = \mathbb{R}^{m \times n}:

L^consistent,k=1n∥ΔYτ⊤(Xk⊤)†∥F2,L^sur=1n∥ΔYτ⊤X∥F2,ΔYτ⊤(Xk⊤)†,  ΔYτ⊤X∈Rm×d⋅Rd×n=Rm×n,\widehat{\mathcal{L}}_{\mathrm{consistent},k} = \frac{1}{n}\lVert \Delta Y_\tau^\top (X_k^\top)^{\dagger} \rVert_F^2, \qquad \widehat{\mathcal{L}}_{\mathrm{sur}} = \frac{1}{n}\lVert \Delta Y_\tau^\top X \rVert_F^2, \qquad \Delta Y_\tau^\top (X_k^\top)^{\dagger},\; \Delta Y_\tau^\top X \in \mathbb{R}^{m \times d} \cdot \mathbb{R}^{d \times n} = \mathbb{R}^{m \times n},

resolving the dimensional mismatch across different input–output length pairs (n,m)(n, m) in neural network training.

We will add a dedicated Remark following Proposition 2 in Section 4 to state this distinction explicitly and clarify that L^consistent,k\widehat{\mathcal{L}}_{\mathrm{consistent},k} is not an algebraic reformulation of Lconsistent\mathcal{L}_{\mathrm{consistent}}.

2. Clarification on Proposition 2, Covariance Projector Πk\Pi_k, & Low-Rank Concentration

Thank you for catching the algebraic error in our previous comment. We explicitly withdraw the erroneous two-sided inequality via Uk⊤ΔXτVkU_k^\top \Delta X_\tau V_k and agree that Lconsistent\mathcal{L}_{\mathrm{consistent}} (Eq. 17) and L^consistent,k\widehat{\mathcal{L}}_{\mathrm{consistent},k} act on different spaces without sharing a common null space. Importantly, Proposition 2 in the manuscript does not rely on that inequality or equate Eq. 17 with L^consistent,k\widehat{\mathcal{L}}_{\mathrm{consistent},k}. Instead, Proposition 2 and its proof establish:

  • Shared Left-Singular Projection MM: From the rank-kk SVD Xk=UkΣkVk⊤X_k = U_k \Sigma_k V_k^\top (where Uk∈Rd×kU_k \in \mathbb{R}^{d \times k} spans the top-kk variate modes) and Ak:=Xk⊤Xk=VkΣk2Vk⊤A_k := X_k^\top X_k = V_k \Sigma_k^2 V_k^\top, Step 1 gives ΔXτ⊤(Xk⊤)†=BAk†\Delta X_\tau^\top (X_k^\top)^\dagger = B A_k^\dagger with B:=ΔXτ⊤XkB := \Delta X_\tau^\top X_k. Defining M=[m1,…,mk]:=ΔXτ⊤UkΣk∈Rn×kM = [m_1,\ldots,m_k] := \Delta X_\tau^\top U_k \Sigma_k \in \mathbb{R}^{n \times k}, unitary invariance of the Frobenius norm under VkV_k (Step 2) yields

L^sur,k=1n∥M∥F2=1n∑i=1k∥mi∥22,L^consistent,k=1n∥MΣk−2∥F2=1n∑i=1kσi−4∥mi∥22.\widehat{\mathcal{L}}_{\mathrm{sur},k} = \frac{1}{n}\lVert M \rVert_F^2 = \frac{1}{n}\sum_{i=1}^k \lVert m_i \rVert_2^2, \qquad \widehat{\mathcal{L}}_{\mathrm{consistent},k} = \frac{1}{n}\lVert M\Sigma_k^{-2} \rVert_F^2 = \frac{1}{n}\sum_{i=1}^k \sigma_i^{-4}\lVert m_i \rVert_2^2.

Both losses depend solely on the same projected residual M=ΔXτ⊤UkΣkM = \Delta X_\tau^\top U_k \Sigma_k and vanish simultaneously iff ΔXτ⊤Uk=0\Delta X_\tau^\top U_k = 0 (no mismatch between left and right singular subspaces occurs).

  • Asymptotic Covariance Conditioning on range⁡(Πk)\operatorname{range}(\Pi_k): Let Πk=Uk∗(Uk∗)⊤∈Rd×d\Pi_k = U_k^\ast (U_k^\ast)^\top \in \mathbb{R}^{d \times d} denote the orthogonal projector onto the top-kk eigenspace of the stationary variate covariance C0=E[xnxn⊤]≻0C_0 = \mathbb{E}[x_n x_n^\top] \succ 0 (the asymptotic limit of the empirical rank-kk left-singular projector UkUk⊤U_k U_k^\top), so ΠkC0Πk∈Rd×d\Pi_k C_0 \Pi_k \in \mathbb{R}^{d \times d} is the rank-kk truncated variate covariance. Because Ak=Xk⊤XkA_k = X_k^\top X_k and XkXk⊤=UkΣk2Uk⊤X_k X_k^\top = U_k \Sigma_k^2 U_k^\top share nonzero eigenvalues {αi=σi2}i=1k\{\alpha_i = \sigma_i^2\}_{i=1}^k, ergodic convergence 1nXkXk⊤→a.s.ΠkC0Πk\frac{1}{n}X_k X_k^\top \xrightarrow{\mathrm{a.s.}} \Pi_k C_0 \Pi_k gives αmax⁡/n→λkmax⁡\alpha_{\max}/n \to \lambda_k^{\max} and αmin⁡/n→λkmin⁡\alpha_{\min}/n \to \lambda_k^{\min} almost surely (Steps 3–4), bounding the asymptotic ratio via the extreme nonzero eigenvalues of ΠkC0Πk\Pi_k C_0 \Pi_k:

1(λkmax⁡)2≤lim inf⁡n→∞n2L^consistent,kL^sur,k≤lim sup⁡n→∞n2L^consistent,kL^sur,k≤1(λkmin⁡)2a.s.\frac{1}{(\lambda_k^{\max})^2} \le \liminf_{n\to\infty} n^2\frac{\widehat{\mathcal{L}}_{\mathrm{consistent},k}}{\widehat{\mathcal{L}}_{\mathrm{sur},k}} \le \limsup_{n\to\infty} n^2\frac{\widehat{\mathcal{L}}_{\mathrm{consistent},k}}{\widehat{\mathcal{L}}_{\mathrm{sur},k}} \le \frac{1}{(\lambda_k^{\min})^2} \quad \mathrm{a.s.}

While residuals in ker⁡(Πk)\ker(\Pi_k) lie in the null space of both rank-kk terms, our spectral analysis across all seven datasets and prediction horizons {96,192,336,720}\{96, 192, 336, 720\} (Appendix G) shows that ΠkC0Πk\Pi_k C_0 \Pi_k captures >99%>99\% of total variance at k=2k=2 on ETT, Weather, and ECL (>90%>90\% at k=3k=3 on Traffic), with finite condition ratios (λkmax⁡/λkmin⁡)2∈[1.8×103,2.0×104](\lambda_k^{\max}/\lambda_k^{\min})^2 \in [1.8\times 10^3, 2.0\times 10^4]. Thus, Proposition 2 captures virtually all dynamical energy and remains non-vacuous. We will state this exact relationship explicitly in Section 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.