- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates overfitting in deep Koopman learning, particularly when dictionary learning and operator fitting reuse the same stochastic transitions. The authors propose a model-assisted approach that learns the dictionary from forecasts of an approximate dynamical core and fits the Koopman operator using observed transitions. Experiments on ERA5 and stochastic shallow-water equations show improved held-out VAMP-E scores, especially at longer lags and under stronger stochastic forcing.
Critical strengths
-
The paper identifies a meaningful generalization issue in deep Koopman learning and provides a clear analysis of how reusing stochastic transitions can lead to overfitting.
-
The proposed separation of dictionary learning and operator fitting is intuitive and effective. The experiments are thorough, especially the controlled stochastic forcing experiments and independent operator refitting analysis, which help explain where the improvements come from.
Critical weaknesses
The unique benefit of using a dynamical core is not fully established. Independent operator refitting already recovers most of the performance gap between data-trained and core-trained dictionaries. It remains unclear whether the additional improvement comes from physically meaningful dynamics or simply from using a deterministic, noise-free training target. More comparisons with alternative approaches would help clarify the necessity of the dynamical core.
Questions for authors
-
How does the proposed method compare with standard data splitting or cross-fitting, where dictionary learning and operator fitting use independent subsets of the observed transitions under the same total data budget?
-
Could a deterministic learned forecasting model replace the dynamical core for dictionary learning? Such a comparison would help clarify whether the improvement specifically comes from physical dynamics.
-
How sensitive is the advantage to the choice of operator rank and shrinkage? In Appendix D.3.2, data-trained dictionaries outperform core-trained ones at 5- and 7-day lags when both hyperparameters are independently selected. Could the authors further discuss this result?
4 comments
Response to Reviewer W6Up [2/2]
Q1. Splitting the data at the same budget. The refit closes three quarters of the gap (L438–443) only with twice the data. At the same budget, splitting halves the data and closes 9–15% of the gap (shallow-water benchmark, fixed setting).
In the split, the data-trained dictionary trains on one half of the data and its operator is fitted on the other, then the reverse, and the two scores are averaged. In ERA5, with alternate years of 1979–2010 (the split on half of the pairs), its dictionary scores 2.0–2.3 at the fixed setting, below the data-trained dictionary (3.8–4.7), while ours scores 16.8–20.2. Tuned on the training years, it scores 8.6–11.9 below ours.
The refit shows that the reuse causes the gap. It keeps the data-trained dictionary and fits its operator again on a regenerated set of trajectories, which a simulation can supply and the atmosphere's single record cannot. The core target needs no such set, and its operator is fitted on every observed transition. The revision will call the refit a diagnostic, not sample splitting, and will add the comparison above.
Q3. Sensitivity to rank and shrinkage, and the 5- and 7-day reversal. At 10 and 14 days ours scores above the data-trained dictionary however rank and shrinkage are chosen. At 5 and 7 days it depends on the rank: the reversal in App. D.3.2 comes from choosing the rank on halves of the held-out years (L616–617).
There each dictionary picks rank and shrinkage on one half of the held-out years and is scored on the other. The core-trained dictionary scores 4.1–9.7 higher on one half than on the other, so a pick made on one half can miss on the other. At 5 and 7 days the data-trained dictionary keeps all 200 functions at the strongest shrinkage in App. D.3.2's search, rank 200 at shrinkage 1, on both halves (L652). At short lags it overfits less, so its later functions still add held-out score once shrunk. This raises its score by 7.0–8.0 at 5 days and 3.4–4.1 at 7 days, enough to put it above ours. With the rank held at 60, ours scores 2.5–2.8 above it at 7 days in every seed, and −0.5 to +0.7 at 5 days.
Ours scores above the data-trained dictionary at every lag in every seed when rank and shrinkage are chosen on years before the held-out years, with every dictionary retrained without those years. It still scores above with the shrinkage searched up to 100, a range that contains the picks of App. D.3.2. Ours minus the data-trained dictionary:
| tuned on | 5 days | 7 days | 10 days | 14 days |
|---|---|---|---|---|
| half of the held-out years 2012–2020 (App. D.3.2) | −6.8 to −3.1 | −3.9 to −0.8 | +4.2 to +5.2 | +4.4 to +6.4 |
| 2008–2011, every dictionary retrained without them, shrinkage up to 0.1 | +5.4 to +7.4 | +4.9 to +5.5 | +5.8 to +8.6 | +8.1 to +10.3 |
| the same, shrinkage up to 100 | +3.9 to +4.1 | +6.1 to +9.4 | +9.4 to +11.4 | +9.3 to +10.8 |
If these answers address your questions, we would be grateful if you would consider raising your rating.
Thank you for the addtional experiments and the detailed and thoughtful responses. I appreciate the experiments on the noise-free target, learned forecaster, and equal-budget data splitting. These comparisons help clarify the specific benefit of using a dynamical core, and provide more convincing evidence that the improvement cannot be simply explained by denoising or data splitting alone. Also, the additional analysis of rank and shrinkage gives a reasonable explanation for the results at shorter lags.
Generally speaking, I think these new experiments have addressed most of my concerns, and I encourage the authors to include these results and discussions in the revised paper. Overall, I remain positive about this work and will keep my original positive score.
Thank you
Thank you for evaluating our responses and for your positive assessment. We will definitely add the new controlled experiments and their discussion to the paper.
Response to Reviewer W6Up [1/2]
We thank the reviewer for recognizing the diagnosis and finding the method intuitive and effective. Each alternative the reviewer raised trains a dictionary that scores below the core-trained one: a noise-free target without dynamics scores at the data-trained dictionary's level, a learned forecaster scores 12.7–14.6 lower in ERA5, and an equal-budget split closes 9–15% of the gap. The conventions are at the end of the general response.
W1 and Q2. A noise-free target, or a learned forecaster, in place of the core. A learned forecaster trains a dictionary within 1.4–3.2 of the core-trained one only where separate trajectories can be simulated. On ERA5's single record it must be fitted either on the training years, which reuses their transitions, or on four held-back years. Either way its dictionary scores below the core-trained one.
A noise-free target without dynamics fails (W1). It applies only the core's diffusion and step filter, which multiply each spherical-harmonic coefficient by a fixed factor of at most one, so it only damps the state. In the shallow-water benchmark it scores 0.6–2.7 at the fixed setting, about the data-trained dictionary's level, against 19.8–21.3 for the core.
A more accurate linear forecaster is not a better target (Q2). We tested the linear inverse model (L72), a least-squares linear map of the state's leading empirical orthogonal function (EOF) coefficients to their values one lag later. Unlike the dissipation, it mixes the coefficients: it is a linear approximation of the dynamics, fitted to data.
In ERA5, fitted on 2008–2011, years no dictionary trains on, the forecaster is more accurate than the core, yet trains a dictionary 12.7–14.6 below the core-trained one in every seed, at either setting. Its held-out error is 7% above persistence's, against 48% for the core. With 5,728 states for 1,540 EOF coefficients, it nearly memorizes those four years (0.18 times persistence's error there).
With rank and shrinkage tuned on the training years, the dictionaries whose targets contain the observed futures collapse, because that tuning rewards features fitted to their noise.
We also fitted the forecaster on the training years 1979–2011, the transitions the operator also uses, so its target contains their observed futures. Its dictionary then scores 13.2–14.5 at the fixed setting, against 20.3–21.9 for the core, and −11.3 to 7.9 with rank and shrinkage tuned on those years.
In the shallow-water benchmark, fitted on separate trajectories, the forecaster meets both conditions of our argument and trains a good dictionary, 1.4–3.2 below the core. Fitted on the operator's own transitions, it scores below the forecaster fitted on separate trajectories in every seed, since its target then contains their noise. Seeds 0 / 1 / 2:
The dynamical core needs no data beyond the present states it starts from, so a single record such as ERA5 does not limit it.