acceptodds
Withdrawn from ICLR 2027

The Forecast–Reconstruction Tradeoff Is an Artifact of Coupling a Single Latent to Both Objectives

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent dynamical models that share a single latent space between reconstruction and forecasting face a tradeoff: improving one objective degrades the other. We show this tradeoff is architectural, not inherent. For linear systems, we prove that reconstruction-optimal and prediction-optimal projections generically occupy different subspaces; empirically, we observe persistent gradient conflict on the shared encoder. We propose carrier-residual decoupling: a wide carrier handles reconstruction, a narrow residual code handles forecasting, and no parameters are shared between the two. On five dynamical systems (6-20D), a 64D spatiotemporal benchmark, three forecast heads (DMD, MLP, neural ODE), and four multi-task gradient methods, no param-matched coupled model Pareto-dominates the decoupled one.

Primary area:
learning on time series and dynamical systems
Keywords:
latent dynamics, Koopman autoencoders, reconstruction-forecasting tradeoff, decoupling, dynamic mode decomposition, gradient conflict, multi-task learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tzWBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the reconstruction-forecasting tradeoff in latent dynamical models and argues that it mainly arises from forcing a single latent representation to serve both objectives. The authors propose a carrier–residual decoupling framework, where a wide carrier is optimized for reconstruction and a narrow residual code is used for forecasting, and evaluate it on several synthetic dynamical systems with different forecasting heads and multi-task optimization baselines.

Critical strengths
  1. The paper identifies a relevant issue in latent dynamical modeling and gives a simple intuition for why reconstruction and forecasting objectives may interfere.
  2. The authors evaluate several forecasting heads, multi-task optimization methods, noise settings, and multiple dynamical systems.
  3. Parameter counts and training details are reported relatively clearly.
Critical weaknesses
  1. Major concerns about the theoretical analysis. Proposition 1 claims that the reconstruction-optimal projection is not even a stationary point of the prediction objective under a non-commutativity condition, but the provided proof mainly compares a small set of alternative projections and does not establish the claimed first-order result. The nonlinear corollary further makes a strong leap from a varying Jacobian to generic incompatibility of the optimal subspaces.
  2. Latent capacity is not really controlled by parameter matching. The proposed approach explicitly gives reconstruction a larger j-dimensional carrier while coupled baselines operate through a smaller k-dimensional bottleneck. Increasing hidden width to match parameter count does not remove this difference in representational bottleneck.
  3. The proposed method is not convincingly separated from a much simpler baseline. The recon-only wide-latent Frozen_j baseline achieves nearly identical reconstruction and forecasting performance to the full decoupled method in many settings. The multi-horizon results make this especially apparent, where Frozen_j and Decoupled are almost indistinguishable across several systems.
Questions for authors
  1. Could the authors provide a complete derivation showing that the reconstruction-optimal projection is not stationary, rather than only showing that another projection can achieve lower prediction loss?
  2. Why is the carrier–residual pipeline necessary when Frozen_j already obtains almost identical results in the ablation and multi-horizon experiments?
Reviewer_k3cxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a new architectural idea for latent dynamical models that decouples reconstruction from forecasting. The method is in part motivated by provided theoretical and empirical evidence of gradient conflict between the reconstruction and the forecasting losses.

Critical strengths
  • The paper introduces an interesting idea, namely to architecturally decouple the reconstruction and forecasting parts.
  • Theoretical and empirical evidence support the claims of gradient conflict.
  • The paper clearly mentions the experimental protocol for reproducibility, including an anonymous github repo.
Critical weaknesses
  • the experiments are only performed on synthetic toy-ish tasks. The dimensions are small 5D, 6D, 10D, 10D, 20D and 64D. It would be highly relevant to test this on higher dimensions (e.g. neural surrogates) and see if the results hold there too.
  • The paper shows mixed empirical results on the reconstruction and single-step forecasting (F1).
  • The multi-horizon results are also mixed (Table 10 in Appendix), indicating that error accumulation might be a problem. This is not further analyzed in the paper and feels a bit "swept under the rug".
Questions for authors
  • In Table 9, I don't understand the bold: best per cell. why is .997 in bold (typo?)
  • Fig.3, row 1 just analyzes F1 (forecast for 1 timestep), correct?
Reviewer_rBBNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors study how latent representations used for both reconstruction and prediction can interfere with one another. They argue that in generic cases, these two criteria compete with each other. Empirically, in several cases, the gradients of each objective are negatively correlated. Finally, they introduce a decoupled setup, where a high-D latent is used for reconstruction, and a separate, low-D, latent is used for prediction. They then show that this achieves much better results on both objectives compared to the baselines used.

Critical strengths

This is an elegant dissection of two commonly used objectives (prediction and reconstruction) of the same object (latent). The empirical findings of negative correlations and performance of the decoupled setup are intriguing.

Critical weaknesses

If I’m not mistaken, proposition 1 is incorrect. First, the statement is that if the matrices do not commute, then there is no common minimizer. But the proof says this is true only if the off-diagonal is large enough. Second, I think there is a missing Lambda1*F_21^2 term in the P=e2 case (line 103-105).

Consider this counterexample. Sigma_x = diag(3,2,1). F = e1*e2^T. Q = diag(1,2,1). K = 2. The dynamics copy coordinate 2 into coordinate 1. And this happens in the subspace which has most variance and thus important for reconstruction.

Negative correlation in the gradients does not mean there is no direction in which both objectives improve. Only that the maximal improvement directions are negatively correlated. For instance, if one gradient is (1,0) and the other is (-1,2). Inner product is negative (-1). But moving along (-1,-1) improves both. This should probably be stressed.

The improvement of the decoupled model – could it be that the main effect is the larger latent? Table 3 shows that coupled_j is the main jump in performance.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.