- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a method they name "off-attractor supervision" for neural network surrogates of turbulence. The basic idea is to use model rollouts, pass those through a normal scientific numerical solver, roll out the trajectory, and then use those as fine-tuning data for the surrogate model. The advantage being that this helps the model become robust to "off-attractor" states which it might encounter during normal rollouts.
Critical strengths
1/ I think this is an interesting idea and a very useful direction for the PDE emulator community to explore in more detail. There are many questions to investigate here.
Critical weaknesses
1/ I have various concerns about the contributions of this paper over previous methods.
The core issue I have is that APEBench (2024) had already largely introduced this technique with own-future targets, in a very similar PDE surrogate/numerical solver context. It seems the authors of this submitted paper overlooked that APEBench's proposed approach even explicitly described stop-gradients, which was one of the claimed distinguishing ideas of this submitted paper. In Appendix A, after describing APEBench, the authors state "the setup requires that simulator to be differentiable, and the benchmark reports it up to 32^3." In other words, one of the key differentiating factors made against APEBench is not entirely true, because that prior work actually already documented a non-differentiable variant of this idea.
So, I think that once this technique is compared against the methods explored in the APEBench paper, unfortunately, there are fewer new ideas to present. This would be okay, however, provided the ideas that are left had sufficient controlled experiments to disentangle the actual impact against the previous baselines. Unfortunately, I think the remaining claimed contribution is the offline training recipe/caching and the particular state selection rules, and these are not explored in enough detail to merit publication. I am not sure if this is enough to justify a paper, and certainly in its present form, there is not enough experimentation to really explore this or give it enough exposition to be useful for the community. For example, I think that there should really be a fixed-compute comparison against APEBench's diverted-chain training.
2/ More of an issue with the experimentation: I did not feel convinced that the survival times establish a useful improvement in the physical simulation. The main metric used here is that the overall kinetic energy leaves a very broad interval. Or that the model freezes. But, keep in mind that a rollout can produce completely incorrect turbulence while still satisfying this survival metric. And indeed the proposed method actually was at 7.4 times (on average) the reference enstrophy from the simulation, while still being claimed to have "survived." So I would say that this is far too permissive. In addition, I would say that these baselines seem (by eye) weak overall, so it's not clear whether this technique is simply smoothing out a very noisy baseline, or actually rescuing any physical validity.
Questions for authors
[discussed in weaknesses]
Thank you for the detailed critique. APEBench. We mischaracterized APEBench. Own-future targets, including a stop-gradient variant that does not require a differentiable solver, are already described there, so we withdraw that distinction. What remains is the following: (i) A measurement of why surrogates fail, using a bit-exact solver twin. It compares the true dynamics' amplification of a deviation with the surrogate's. (ii) Solver-free, failure-mode-targeted state selection. In our ablations, rollout states alone (the diverted-chain analogue) trade blow-up for freezing, and the targeted states are what change this. We agree that a fixed-compute comparison against diverted-chain training is the control that would establish (ii). We do not have it yet. Survival metric. We agree. The energy band and the freeze criterion are necessary conditions for a physically valid rollout, not sufficient ones. The elevated enstrophy shows that our surviving rollouts are not yet physically faithful. We will report a statistics-based validity horizon alongside the energy band and the freeze criterion, and narrow the claim to delaying failure and reducing statistical drift rather than rescuing physical validity.