acceptodds
Withdrawn from ICLR 2027

Staying on the Attractor: Supervising Neural Surrogates of 3D Turbulence Where They Leave It

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural surrogates are trained to predict 3D turbulent flows in place of direct numerical simulation (DNS). For chaotic flows, the goal is short-term pointwise accuracy followed by long-term physical and statistical fidelity. However, small prediction errors can carry a surrogate away from the flow's attractor. Off-attractor states are poorly represented in training data, leaving their evolution weakly constrained. The learned dynamics can then amplify deviations and lead to blow-up, freezing, or statistical drift. In this paper, we propose off-attractor supervision (OAS) to supervise neural surrogates where they leave the attractor. OAS teaches the model how the true Navier–Stokes dynamics would evolve from these states. Each selected state is paired with its own future computed by DNS. Three generators select a few hundred states for relabeling. The first collects states from the surrogate's own rollouts. The second uses surrogate attacks to target freezing, excessive amplification, and violations of incompressibility and energy balance. The third perturbs training states along an amplified direction and a strongly damped random direction of the dynamics. All attacks run on the surrogate alone, and DNS relabeling is performed offline once per selected state. Experiments on turbulence show that OAS increases the median time to failure from 21 to 721 steps. The compared baselines achieve medians of at most 110 steps, and the advantage holds across training seeds. OAS also achieves the lowest pointwise error at step 15 and the best long-horizon statistics among the compared methods. OAS integrates physical models into neural simulation by extending supervision from fixed reference trajectories to states where the surrogate is likely to fail. This principle can guide the development of more reliable scientific surrogates when deployment takes models beyond the coverage of their training data.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
scientific machine learning, representation learning, turbulence data mining, adversarial machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_p7i8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method they name "off-attractor supervision" for neural network surrogates of turbulence. The basic idea is to use model rollouts, pass those through a normal scientific numerical solver, roll out the trajectory, and then use those as fine-tuning data for the surrogate model. The advantage being that this helps the model become robust to "off-attractor" states which it might encounter during normal rollouts.

Critical strengths

1/ I think this is an interesting idea and a very useful direction for the PDE emulator community to explore in more detail. There are many questions to investigate here.

Critical weaknesses

1/ I have various concerns about the contributions of this paper over previous methods.

The core issue I have is that APEBench (2024) had already largely introduced this technique with own-future targets, in a very similar PDE surrogate/numerical solver context. It seems the authors of this submitted paper overlooked that APEBench's proposed approach even explicitly described stop-gradients, which was one of the claimed distinguishing ideas of this submitted paper. In Appendix A, after describing APEBench, the authors state "the setup requires that simulator to be differentiable, and the benchmark reports it up to 32^3." In other words, one of the key differentiating factors made against APEBench is not entirely true, because that prior work actually already documented a non-differentiable variant of this idea.

So, I think that once this technique is compared against the methods explored in the APEBench paper, unfortunately, there are fewer new ideas to present. This would be okay, however, provided the ideas that are left had sufficient controlled experiments to disentangle the actual impact against the previous baselines. Unfortunately, I think the remaining claimed contribution is the offline training recipe/caching and the particular state selection rules, and these are not explored in enough detail to merit publication. I am not sure if this is enough to justify a paper, and certainly in its present form, there is not enough experimentation to really explore this or give it enough exposition to be useful for the community. For example, I think that there should really be a fixed-compute comparison against APEBench's diverted-chain training.

2/ More of an issue with the experimentation: I did not feel convinced that the survival times establish a useful improvement in the physical simulation. The main metric used here is that the overall kinetic energy leaves a very broad interval. Or that the model freezes. But, keep in mind that a rollout can produce completely incorrect turbulence while still satisfying this survival metric. And indeed the proposed method actually was at 7.4 times (on average) the reference enstrophy from the simulation, while still being claimed to have "survived." So I would say that this is far too permissive. In addition, I would say that these baselines seem (by eye) weak overall, so it's not clear whether this technique is simply smoothing out a very noisy baseline, or actually rescuing any physical validity.

Questions for authors

[discussed in weaknesses]

1 comment

Thank you for the detailed critique. APEBench. We mischaracterized APEBench. Own-future targets, including a stop-gradient variant that does not require a differentiable solver, are already described there, so we withdraw that distinction. What remains is the following: (i) A measurement of why surrogates fail, using a bit-exact solver twin. It compares the true dynamics' amplification of a deviation with the surrogate's. (ii) Solver-free, failure-mode-targeted state selection. In our ablations, rollout states alone (the diverted-chain analogue) trade blow-up for freezing, and the targeted states are what change this. We agree that a fixed-compute comparison against diverted-chain training is the control that would establish (ii). We do not have it yet. Survival metric. We agree. The energy band and the freeze criterion are necessary conditions for a physically valid rollout, not sufficient ones. The elevated enstrophy shows that our surviving rollouts are not yet physically faithful. We will report a statistics-based validity horizon alongside the energy band and the freeze criterion, and narrow the claim to delaying failure and reducing statistical drift rather than rescuing physical validity.

Reviewer_UWr9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Off-attractor supervision (OAS), a method that uses a DNS simulator offline to improve the rollout error and long term statistics of a neural surrogate on a 3D Navier-Stokes turbulent system. They show OAS outperforms a variety of neural surrogate stability improvement methods including the pushforward trick and more. They also provide a criterion beyond MSE for judging the rollout performance of the surrogates.

Critical strengths
  • The OAS method makes significant improvements in long term rollout stability (21 steps to 721) while also improving short term accuracy and statistical metrics. Also the ablation experiments in section 4.3 show some evidence that the their specific generation strategy for next states is important and more arbitrary DNS data would not have the same effect.
  • The explanation of why and how neural surrogates fail when only trained on clean simulator data by amplifying signals incorrectly as well as freezing or damping to an unrealistic steady state is a good way to motivate the problem.
Critical weaknesses
  • The main idea of using DNS to augment the training/finetuning data of a surrogate has been done before (Koehler et al. (2024) diverted-chain supervision, Sun (2026) solver-integrated adversarial training, Roy et al. (2026) active learning with solver). While OAS adds its own generators and criterion, it is missing a comparison to one of these DNS based methods with a matched budget (same number of DNS calls and NN finetuning). Another important reference or comparison method is generative/probabilistic surrogate models (diffusion, stochastic interpolants) as they have been shown to reproduce attractor statistics better than standard surrogates (https://arxiv.org/pdf/2607.00196, https://arxiv.org/abs/2610.06952).
  • The OAS method is only tested on homogenous isotropic turbulent NS. Additionally, the method seems to work well for the UNet but the improvement on the FNO is much smaller. This suggests that OAS is specific to this PDE system and not generalizable. The generation strategy for finetuning may have to be PDE system specific.
Questions for authors

Have you compared OAS to another method using DNS based training with a matched budget?

1 comment

Thank you for the careful reading and for the constructive questions. Prior DNS-based augmentation. We agree that relabeling a surrogate's states with the solver is not new (diverted-chain supervision in APEBench, Sun et al., Roy et al.), and we will state this plainly. Two internal controls are the closest analogues we have:

Rollout states relabeled with their own futures alone, i.e., diverted-chain-style supervision: these remove blow-up but cause the model to collapse to a fixed point. An error-maximizing attack with the solver inside the loop: it costs far more solver calls and gives a shorter lifetime than our solver-free objectives at the same pool size. These are not budget-matched reproductions of the original methods. We agree that such a comparison is required. Generative surrogates. We agree that they are the relevant comparison for attractor statistics and will include them. Generality. The reviewer is right that we test only forced HIT. In our setup the spectral and attention backbones already fail under one-step training. The backbone comparison therefore does not establish generality in either direction. We will scope the claims accordingly.

Reviewer_WbiTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes off-attractor supervision (OAS), which supplements the training dataset with additional state pairs for supervision and which are based on three selection rules: on-policy states, surrogate attacks and DNS probes. All these improve the long-horizon statistics and median time to failure substantially. The authors evaluate off-attractor supervision on forced homogeneous isotropic turbulence on a 128^3 sized grid against several established strategies to improve the rollout performance for neural surrogates.

Critical strengths
  • OAS shows very promising results against a comprehensive set of standard techniques to improve rollout stability
  • The detailed ablations about different attack strategies are insightful
  • I enjoyed reading the sections about attractors in turbulence and how dynamics respond to a deviation
  • Stabilizing neural surrogates for turbulent flows is a challenging task and the proposed method shows strong advancements in the field.
Critical weaknesses
  • The comparison between OAS and the baseline techniques does not seem fair. OAS requires rerunning the solver, which none of the baseline techniques do. Rerunning the solver is a significant downside to this approach, since it complicates the training process and requires additional compute beyond the training dataset generation. Please include some analysis here.
  • A fair comparison should take the solver compute costs into account. The used dataset contains 1800 training samples, which seems low. It is not clear to me how large the fraction of solver compute added to the dataset generation cost for OAS is. A better comparison should step-by-step scale the dataset solver compute budget and compare the baseline methods trained on a larger dataset dataset whose extra size matches that of the additional compute required by OAS.
  • There are networks that show better long-horizon stability when trained as a baseline than the U-Net considered in this work (assuming sufficient amounts of data). See e.g. Holzschuh et al. (2026) https://openreview.net/forum?id=8UdCE5nhFl The results would be stronger using a better baseline network.
  • A comparison to diffusion methods that achieve very good performance on long-horizon prediction would be insightful. Specifically Thermalizer (Pedersen et al. 2025) and diffusion forcing would be interesting (Chen et al. 2024 https://openreview.net/forum?id=yDo1ynArjj)
  • The paper is hard to read at times and the language seems unnatural in many places. I often had the feeling of reading an experiment report rather than a polished paper with a well structured narrative. For me, this seems like an artifact of LLM writing.
Questions for authors

My main questions are already listed in the critical weaknesses:

  • How do other techniques compare in a solver compute matched setting and when more training data is available?
  • How sensitive are results to the network used? Do we obtain the same improvements with sota networks for neural surrogates?
  • How do these methods perform compared to diffusion-based approaches?
  • Did you face any difficulties projecting training states back to a divergence-free space that can be used as input for the solver?
  • Why do you think OAS has the worst one-step validation loss, but is much better than other methods at step 20?
  • Can we reuse the on-policy states to train another model from scratch?
1 comment

Thank you for the careful reading and for the constructive questions. Compute fairness. We agree that this is the most important missing control. OAS adds solver work in the form of a few hundred short relabels, each a few frames long. Measured in labeled frames, that is not negligible next to our training corpus. The right control is therefore a baseline trained on additional clean DNS data of equal solver cost, plus matched fine-tuning epochs. Our current controls match fine-tuning schedule and gradient count but not solver cost, so we will add a data- and compute-matched comparison. Backbones and generative methods. We agree that the claim is currently limited to the convolutional family. Comparisons with stronger backbones (e.g., Holzschuh et al.) and with diffusion-based methods (Thermalizer, Diffusion Forcing) are needed before any claim of generality. Q: divergence-free projection. It is exact and cheap in the spectral basis: a Leray projection plus dealiasing. The only practical issue was large-velocity on-policy states. We relabeled those with a reduced time step to satisfy CFL, and discarded the few beyond that limit. Q: worse one-step loss, better rollout. One-step loss is measured on clean, on-attractor states, and OAS spends part of its capacity on off-attractor states. Over a rollout, error is governed by how the learned map amplifies its own deviations rather than by one-step accuracy. We measure this amplification directly, and it is what separates the arms. Q: reusing on-policy states for a new model. We have not tested training from scratch. Our evidence suggests the states are model-specific: attack pairs mined against one model helped a differently trained base much less. Writing. We accept the criticism and are restructuring the narrative.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.