- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
EchoKoop investigates whether neural Koopman forecasters need trainable latent transition matrices. The method learns the encoder and decoder while using fixed multiscale transition operators built from damped rotations, together with a phase-aligned initialization scheme. The paper evaluates this design on forecasting benchmarks, chaotic systems, and physical systems, with ablations examining each component's contribution.
Critical strengths
-
A clear and worthwhile research question. The paper asks if learning the latent transition is needed when the surrounding representation is learned. Exploring fixed dynamics is a bold and interesting direction, with a natural connection to reservoir computing. The controlled comparisons make this question scientifically useful, even without consistent improvements across every benchmark.
-
A simple and effective initialization scheme. The phase-aligned initializer is a particularly appealing component. It provides a concrete way to connect the observed history to the prescribed dynamics, and the ablations help establish its contribution. The detailed description in the appendix is also valuable.
Critical weaknesses
-
The central claim needs stronger support, and the paper's breadth makes it harder to assess. The paper asks a clear and worthwhile question: does the latent transition operator need to be learned? Table 13 shows near-parity: training the operators improves average MSE. This supports the competitiveness of the fixed bank in the tested configuration, but leaves uncertainty about when learning the operator is beneficial. A nonsignificant difference also does not establish equivalence. Since the trainable variant starts from the frozen bank, reporting how much the operators actually change and testing sensitivity to the learning rate would make this comparison more informative.
The additional chaotic and physical-system experiments do not resolve this uncertainty or even help much with the main thesis. Their connection to the central question needs to be clearer, particularly given ConvLSTM's consistent advantage across Table 19. Also FlowMixer's weak results, including the failed cylinder runs, warrant clarification. The original study reports better performance than ConvLSTM on fluid forecasting. Although the experimental settings differ, the discrepancy raises questions about the spatial representation, model capacity, and training procedure used here.
In essence, a more focused study of long-horizon forecasting, with stronger controlled comparisons of frozen and learned dynamics, would give the main idea more convincing support. The Koopman interpretation also deserves clarification, since forecast-only training does not explicitly enforce consistency between encoded observations and the latent rollout.
-
The baseline protocol and released code leave important questions about fairness and reproducibility. EchoKoop's strongest gains are concentrated on PEMS, evaluated with only 12 input steps (L=12). Elsewhere, the input length is fixed at L=96. These short windows make the baseline adaptations unclear. In particular, how were FlowMixer and Koopa configured for horizons of H =192 to 720? Were inputs padded, were forecasts generated autoregressively in blocks of 96, or were the methods adapted in some other way? Padding changes the input presented to the model, while blockwise forecasting can compound prediction errors. The exact procedure is therefore necessary to interpret the reported differences. For further clarity: FlowMixer needs an input at least of the same length as the horizon or larger (L>=H), while Koopa needs twice the input to be twice the horizon (L=2H).
Overall the comparison would benefit from explicit adaptation details, configuration files, and comparable tuning budgets. It remains also unclear whether the baselines were tuned for these short windows or retained their default settings. Comparisons using longer contexts (typically L=512, or 1440 like Koopa) would help establish whether the rankings are robust. The supplied code appears to cover EchoKoop mainly. Providing these materials and reconciling differences between the released configurations and the paper would make the results reproducible and strengthen confidence in the conclusions.
Questions for authors
-
Baseline implementation: How were FlowMixer and Koopa configured when the prediction horizon exceeded the input length (H>L)? Did you use padding, autoregressive forecasting in shorter blocks, or architectural changes?
-
Frozen versus learned transitions: How much did the trainable operators change from their frozen initialization? How sensitive was this comparison to the learning rate, and how do the differences in Table 13 compare with variability across runs?
-
Koopman interpretation: With forecasting loss alone, what evidence shows that applying the latent transition produces states consistent with encoding the corresponding future observations?
-
Chaotic and physical systems: Which results specifically demonstrate the benefit of freezing the transition operator? Could you also plot predicted and reference trajectories for the Lorenz attractor and report EchoKoop's valid prediction horizon in Lyapunov times?
7 comments
Response to Reviewer qn2y [2/4]
Q4: Baseline implementation: How were FlowMixer and Koopa configured when the prediction horizon exceeded the input length (H>L)? Did you use padding, autoregressive forecasting in shorter blocks, or architectural changes?
A4: We retained the core architectures of Koopa and FlowMixer.
Koopa. The relation L = 2H is an experimental setting used in Table 1 of the original paper. Koopa’s original code allows the segment length S to be set independently of the forecast horizon H. In the Time-Series-Library implementation we used, S defaults to H. We therefore restored S as an independent parameter, using L = 96 and S = 24 for the standard benchmarks and L = 12 and S = 4 for PEMS. The ratio S = L/4 used on the standard benchmarks was also evaluated in the original paper (Figure 8, p. 16).
FlowMixer. To give all methods the same length of observed history, we retained FlowMixer’s core temporal and feature mixing operations and used a blockwise rollout with shared parameters when H > L. We compute the normalization statistics using only the observed history and keep them fixed throughout the forecast. We repeatedly apply the same mixing block in normalized space to generate the next L steps, then concatenate the predicted blocks and retain the first H steps. For example, with L = 96 and H = 720, eight applications yield seven full 96-step blocks and the first 48 steps of the final block.
The anonymous baseline supplement includes the adapted Koopa and FlowMixer implementations, configuration files, and available run commands.
Q5: Frozen versus learned transitions: How much did the trainable operators change from their frozen initialization? How sensitive was this comparison to the learning rate, and how do the differences in Table 13 compare with variability across runs?
A5: In the extended experiments, both trainable variants moved away from their frozen initialization. The dense variant (which learns every entry of each transition matrix) became non-normal, while the structured variant preserved normality and contractivity as it learned the frequencies and half-lives. In the original Table 13 ablation, the mean paired test MSE difference across the 24 settings was −0.00029, small relative to the across-seed standard deviation of 0.00190, with only one of three seeds showing an overall improvement. The original ablation did not retain the trained transition matrices, so we report operator diagnostics from the extended experiments, which started from the same frozen modal bank. At the learning rates selected by validation MSE, the mean relative Frobenius distance from initialization ranged from 0.0969 to 0.8912 for the dense variant and from 0.0018 to 0.0950 for the structured variant. Averaging the maximum sampled power gain in each of the six settings, including step zero, gives 10.63 and 1.00, respectively. The corresponding mean normalized Henrici departures from normality were 0.1469 and 0.0000 (Table 18).
| Dataset (H) | Dense 5 × 10⁻⁵ | Dense 2 × 10⁻⁴ | Dense 10⁻³ | Structured 10⁻⁴ | Structured 10⁻³ |
|---|---|---|---|---|---|
| ETTm1 (96) | 0.1192 / 0.0740† | 0.2845 / 0.1286 | 0.7887 / 0.3044 | 0.0032 / 0.0000† | 0.0140 / 0.0000 |
| ETTm1 (720) | 0.0969 / 0.0478† | 0.3822 / 0.0903 | 0/3 completed | 0.0018 / 0.0000† | 0.0049 / 0.0000 |
| Weather (96) | 0.3849 / 0.1064 | 0.8912 / 0.2329† | 0.9595 / 0.4363 | 0.0099 / 0.0000 | 0.0216 / 0.0000† |
| Weather (720) | 0.1373 / 0.0531† | 0.3912 / 0.1062 | 0/3 completed | 0.0042 / 0.0000 | 0.0086 / 0.0000† |
| PEMS08 (12) | 0.5679 / 0.2120 | 0.7992 / 0.4326† | 0.9516 / 0.6301 | 0.0443 / 0.0000 | 0.0950 / 0.0000† |
| PEMS08 (96) | 0.3935 / 0.1677† | 0.6434 / 0.2957 | 0.9483 / 0.5440 (1/3 completed) | 0.0201 / 0.0000 | 0.0378 / 0.0000† |
Each cell reports the mean relative Frobenius distance from initialization / mean normalized Henrici departure from normality. Both metrics are averaged over operators and completed seeds at their best validation checkpoints. † marks the learning rate selected by validation MSE.
| Dataset (H) | Frozen | Dense 5 × 10⁻⁵ | Dense 2 × 10⁻⁴ | Dense 10⁻³ | Structured 10⁻⁴ | Structured 10⁻³ |
|---|---|---|---|---|---|---|
| ETTm1 (96) | 0.324 | 0.321† | 0.326 | 0.331 | 0.323† | 0.324 |
| ETTm1 (720) | 0.431 | 0.430† | 0.437 | 0/3 completed | 0.431† | 0.432 |
| Weather (96) | 0.180 | 0.175 | 0.177† | 0.178 | 0.180 | 0.180† |
| Weather (720) | 0.373 | 0.373† | 0.361 | 0/3 completed | 0.382 | 0.381† |
| PEMS08 (12) | 0.078 | 0.077 | 0.077† | 0.078 | 0.078 | 0.078† |
| PEMS08 (96) | 0.387 | 0.369† | 0.376 | 1/3 completed | 0.378 | 0.372† |
Numerical entries report mean test MSE over three paired seeds. † marks the learning rate selected by validation MSE for each dataset, horizon, and trainable variant.
Response to Reviewer qn2y [3/4]
The overall gain in the original ablation was small relative to seed-to-seed variation. In Table 13 of the original manuscript (Table 15 in the revised manuscript), the dense trainable variant achieved lower mean test MSE in 14 of 24 settings and reduced macro-averaged test MSE by 0.11%. We paired 72 runs by dataset, forecast horizon, and seed. For the overall comparison, we first averaged test MSE over the 24 settings within each seed, then computed the difference between the dense and frozen variants.
| Training budget | Setting | Mean paired MSE difference | Sample SD of paired differences | Seeds with lower MSE |
|---|---|---|---|---|
| Original, up to 25 epochs | Macro-average over 24 settings | −0.00029 | 0.00190 | 1/3 |
| Original, up to 25 epochs | PEMS08 (H = 96) | −0.02414 | 0.00605 | 3/3 |
| Extended, up to 50 epochs | ETTm1 (H = 720) | −0.00140 | 0.00684 | 1/3 |
| Extended, up to 50 epochs | Weather (H = 720) | −0.00026 | 0.02156 | 1/3 |
| Extended, up to 50 epochs | PEMS08 (H = 96) | −0.01774 | 0.00654 | 3/3 |
Each difference is the dense trainable variant’s test MSE minus the frozen variant’s test MSE for the same seed. Negative values indicate improvement. The extended experiments use the learning rate selected by validation MSE for each setting.
Consistent gains depended on the learning rate. A larger training budget did not increase the average gain on the six shared settings. We raised the training cap from 25 to 50 epochs and increased early-stopping patience from 3 to 10. At 5 × 10⁻⁵, the dense variant improved all six shared settings under both budgets. Its reduction in macro-averaged MSE relative to the corresponding frozen variant went from 1.84% to 1.60%. Under the extended budget, 8 of 18 runs encountered non-finite values when the dense transition learning rate was raised to 10⁻³. The structured variant completed all runs at both learning rates.
| Transition type | Transition learning rate | Settings with three completed seeds | Settings with lower MSE | Macro-averaged MSE change vs. frozen |
|---|---|---|---|---|
| Dense | 5 × 10⁻⁵ | 6/6 | 6/6 | −1.60% |
| Dense | 2 × 10⁻⁴ | 6/6 | 4/6 | −1.07% |
| Dense | 10⁻³ | 3/6 | 1/3 | +0.81% |
| Structured | 10⁻⁴ | 6/6 | 5/6 | −0.07% |
| Structured | 10⁻³ | 6/6 | 2/6 | −0.35% |
For each learning rate, we compute the macro-averaged MSE change against the frozen variant on the settings for which all three seeds completed. The full learning-rate sweep and operator diagnostics are reported in Appendix G.3, Figure 9, and Tables 17–18.
Response to Reviewer qn2y [1/4]
We sincerely thank Reviewer qn2y for the detailed questions and constructive suggestions. Below, we address the comparison between frozen and trainable transitions, the adaptation and reproducibility of baselines under short-history settings, and what the chaotic-system and physical-field experiments establish.
Q1: … , but leaves uncertainty about when learning the operator is beneficial. A nonsignificant difference also does not establish equivalence. … , reporting how much the operators actually change and testing sensitivity to the learning rate would make this comparison more informative.
A1: We agree that Table 13 shows the frozen bank is competitive under the original training protocol, and that the nonsignificant result does not establish equivalence. The average gain from learning the transition was small relative to seed-to-seed variation, although all three paired seeds improved on PEMS08 at H = 96. We report the operator diagnostics and learning-rate sensitivity in our response to Frozen versus learned transitions below.
Q2: The additional chaotic and physical-system experiments do not resolve this uncertainty or even help much with the main thesis. … Also FlowMixer's weak results, including the failed cylinder runs, warrant clarification. … Although the experimental settings differ, the discrepancy raises questions about the spatial representation, model capacity, and training procedure used here.
A2: The purpose of these experiments is to evaluate frozen structured flows beyond standard forecasting benchmarks by testing finite-horizon accuracy and preservation of selected long-run statistics in low-dimensional dynamical systems.
Due to compatibility issues between TensorFlow and our experimental environment, we ported the core operations from the official TensorFlow/CuPy Cylinder notebook, using the official PyTorch demo as a reference.The adapted implementation retains mixing along the time and feature dimensions, reversible instance normalization, and spatial downsampling with a stride of 3.
The official Cylinder notebook uses 64 input frames and 64 output frames. In our evaluation, the model uses 16 input frames, is trained on 20-step free rollouts, and is evaluated at forecast steps 5, 20, and 50. The training loss is the field MSE plus 0.05 times the vorticity MSE. The three runs stopped at epochs 10, 6, and 2, respectively, after encountering non-finite gradient norms. Table 24 marks these results as missing. We documented the differences between the two protocols in Appendix J.2.
Q3: Comparisons using longer contexts (typically L=512, or 1440 like Koopa) would help establish whether the rankings are robust. ... Providing these materials and reconciling differences between the released configurations and the paper would make the results reproducible and strengthen confidence in the conclusions.
A3: We repeated the comparison with all three methods using the same longer input history. We kept the original forecast horizons and ran each configuration with three random seeds. Koopa used
seg_len = H. FlowMixer generated the forecast in a single pass without blockwise recursion and used its original supervision over the full shifted window.With the longer shared input history, FlowMixer achieves the lowest average MSE and MAE on ETTm1 and Weather. EchoKoop still outperforms both baselines on PEMS04 and PEMS08.
All three methods use the same input history length (L = 192). Each cell reports MSE / MAE. “Average” denotes the mean across the four forecast horizons. Bold indicates the lowest value for each metric.