acceptodds
Under review as a conference paper at ICLR 2027

Mechanisms Off the Data Manifold: Distribution-Relative Identifiability in Neural Operators

Abstract

The standard Navier-Stokes benchmark for operator learning contains a blind spot introduced by its own data pipeline: the generator enforces zero spatial-mean vorticity across all examples, whereas the true solution operator acts as the identity along this constant mode. Operators trained to benchmark accuracy recover that response at roughly one-eighth of its true strength, incurring a 28% error under a mean offset that the true dynamics handle trivially. This failure is invisible to standard in-distribution evaluations, is predictable from the training covariance alone, and admits an exact, zero-cost structural repair once diagnosed. This reflects a broader phenomenon: function-value empirical training identifies local mechanisms only along directions excited by data leaving the remaining modes governed entirely by architectural inductive bias. For translation-equivariant model families, this completion is predictable from a continuity dichotomy between localized spatial kernels and per-mode spectral weights. Controlled experiments show that failure follows excitation rather than frequency, that common canonicalization removes the evidence required to resolve a symmetry's mechanism, and this vulnerability amplifies as physical dissipation weakens. The diagnosis is actionable: training covariance flags which data to acquire, while a known symmetry enables exact structural repairs that degrades gracefully under misspecification.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Neural Operators, Operator Learning, OOD Generalization, Inductive Bias, AI for Science

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_iSnaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The idea of the paper is that if a model’s training data never varies along some direction (e.g., every training field has zero mean), the data can’t tell the model how to respond when inputs move that way, so its behavior there is set by the architecture rather than learned. The paper shows this on FNOs and argues you can predict these blind spots from the training data’s variance and fix them by hard-coding known symmetries.

Critical strengths

None

Critical weaknesses

The paper reads end-to-end generated, with overly verbose statements, with complicated words (likely generated by Claude); the abstract doesn't actually abstract -- the headline Navier–Stokes result from the abstract relies on a physically inadmissible perturbation. Things are not carefully defined. The figures clearly have text overlapping with axes -- common issue when having LLMs generate plots.

Theorem 1 is a trivial statement.

Truthfully, when the paper summary is put in the simple language as I did above, one realizes that the paper itself carries no novelty or significance.

Questions for authors

See weaknesses.

Reviewer_tMN3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates distribution-relative identifiability in neural operators, arguing that empirical function-value training only identifies directional derivatives along input modes actively excited by training data. The authors argue that normal directions are left unexcited (guided by architecture inductive bias). Using benchmarks like 2D periodic Navier-Stokes, Burgers' equation, and linear diffusion, the authors demonstrate that preprocessing steps like enforcing zero spatial mean blinds the model to constant offsets and scaling. Through controlled experiments, they show that derivative error tracks low training excitation rather than spatial frequency. Additionally, they explain FNOs' near-zero response on unexcited modes by the fact that their spectral layers parameterize Fourier modes independently rather than coupling them via localized spatial kernels. To address this, they propose "structural completions" a post-hoc wrapper that canonicalize the input before passing them to the model and analytically apply the known symmetry group action at the output.

Critical strengths
  • Transparent claims about the result: seems like higher viscosity systems don't closely align with their result
Critical weaknesses
  • The paper is difficult to follow and would benefit from a thorough reorganization to promote clarity. Currently, the message isn't well organized and doesn't communicate clearly
  • I'm not convinced by the Theorem 1: It doesn't seem like there's anything new here. As I understand it: if two functions agree on a submanifold M\mathcal{M}, their tangential derivatives along M\mathcal{M} are identical while their normal derivatives perpendicular to M\mathcal{M} can be anything. In plain English, if a variable never changes in your training set, your training data cannot tell you the derivative with respect to that variable. I don't see why this exposes anything new. Can you please clarify if I'm missing something here?
  • The NS blindspot of mean vorticity k=0k = 0 would need to be zero on a periodic domain wouldn't it? Wouldn't shifting that make your test of the model unphysical?
  • Rather toy experiments: one NS case on a low-resolution grid and a 1D problem. I would think the paper would benefit from application to more complex higher problems under other transforms.
  • The structured completion seems like plain de-normalization (isn't this already standard operations of canonicalization in ML). How's this any different? I'm unable to understand (so please clarify)
Questions for authors

Please see weaknesses and address the points raised

Reviewer_e8fYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an interesting finding in terms of identifying which responses lack training evidence (the missing directions), which is mostly diagnostic for decomposing and analyzing OOD failure when we have discretized fields as our data when training a neural operator. The contributions are mostly analytical in the sense that it provides concrete missing directions in the training data manifold that caused the failure in response, including the interesting result showing that removing the average (a seemingly simple direction) could still hide a physically meaningful response that can cause failure in the response. The paper provides concrete experiments in which different transformations of the data and the corresponding response failures are shown. The claims in the paper are well supported by the diagnosis.

Critical strengths

It's concretely formulating the OOD of the data manifold using apt mathematical language and concepts, especially demonstrated in its consistent analysis of transformation on fields. In another word, it concretely points out some of the transformations that are missing (i.e. the OOD directions) would result in response failure. Its analytical purpose is clear and well-posed. The experiments also support the claims decently. The findings are interesting in the sense that they point out that even simple transformations (such as translation) can be blind spots for the operator.

Critical weaknesses
  1. The limitation of its analytical value. The exact blind spots the paper identifies are indeed interesting, but they still seem to be concrete examples, i.e. the paper doesn't propose a general categorization or analyses of how to identify these blind spots when any arbitrary field data and a trained neural operator are given. This, in my opinion, is a key limitation because knowing some special blind spots do not formally or structurally provide analyses of the PDE operator/neural operator themselves - knowing how these blind spots are organized or how do we discover the structures behind these blind spots would then actually give strong analytical values to understanding the data manifold.

  2. No or little applications on real field data that are not synthesized using some known PDE. The experiments in the paper are mostly synthesized field data under known PDEs such as NS, Burgers, and diffusion etc, while no real field data are used to discover their blind spots. It's understandable that analytic PDEs are needed to better test the blind-spot transformations, but data synthesis still limits its applicability to real field data.

  3. It's mostly analytical without engineering values. The paper neither 1) presents downstream use cases after knowing these special blind spots (except for the analytic value provided by them that potentially provides guidance on "which direction/kinds of data should we obtain") nor 2) provide a consistent method of fixing the blind spots. The ways that are used to fix these blind spots proposed in the paper are mostly generic and well-established to practitioners, such as adding more data with variation in the missing directions and enforce known physical symmetries. The paper doesn't propose a method to consistently fix these blind-spots, thus missing some relevant engineering contributions.

  4. Restrictiveness to field data. This might be an "OOD" question for this paper, but field data doesn't seem to be a relatively mainstream choice for data representation, since mostly data manifolds consist of points. This doesn't count as a weakness, but some relevant blind-spot analysis on point-based data manifolds (besides the tabular data experiment, which is already non-field data) would be interesting.

Questions for authors
  1. Is there a consistent way that we can discover these blind spots instead of manual trials-and-errors or requiring knowledge of the PDE operator itself?
  2. What would be a way to utilize the blind spots in some relevant downstream engineering tasks? Is there a consistent way to fix these blind spots other than the basic ones such as generating more data in the missing directions?
  3. Can the pipeline be applied on field data that are not generated from an analytic PDE? I.e. we train a neural operator on a set of real field data, and then identify the blind spots?
  4. Is there a way to observe these blind spots on point-based data manifolds other than the tabular data experiment?
Reviewer_JTXtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that training only pins down a model’s derivatives along directions the data actually vary in, and everything else is down to architecture. The main example is the Navier–Stokes benchmark: it zeroes out mean vorticity, so FNOs never learn the (trivial) response to a mean shift. Further experiments cover excitation vs. frequency, canonicalization, and how different architectures fill the gaps. The proposed fixes are covariance-guided data collection and symmetry wrappers. The paper frames all of this as a lesson about causal interventions in interpretability.

Critical strengths

None

Critical weaknesses
  • The paper uses heavy causality language early on, but in my opinion the formulation is ill-posed. The paper uses causal language such as interventions and identifiability throughout, but there’s no rigorously defined causal model behind it. Their intervention is just an input perturbation, mechanism is just a Jacobian, and the theorem is about off-support derivatives of a fixed function. The obvious formal link to causal inference (positivity/overlap) is never made. The paper also doesn’t explain why input-space results would transfer to the hidden-activation interventions that interpretability actually uses.

  • The official-data replication doesn’t really replicate. Table 18 vs. Table 9: 11.4% vs. 0.36% test error, and 0.48 vs. 0.13 mean gain. The fix barely helps at small offsets (0.113 vs. 0.129), and the gap isn’t explained. The high error makes the model look undertrained.

  • Extremely difficult to read. The biggest problem of this paper is the readability. Many LLM-like languages such as "Let’s raise theorem 1", "famzeily-dependent", "audit", and the constant use of "I" instead "we" throughout make the paper particularly difficult to follow. Additionally, terms are used before they’re defined and results are crammed together.

  • Overclaims. A few theorems were just standard differential geometry theorems. This is not the problem. The problem is that these theorems are poorly connected with the main claims of the paper.

Questions for authors
  • In what precise sense are these “causal” interventions or “identifiability” claims? Can the framework be stated in standard causal terms (e.g. as a positivity violation)? Why should conclusions about input-space derivatives carry over to interventions on internal activations?

  • Is there a realistic case where an unidentified direction actually causes a downstream failure, not just a constructed offset?

  • Why do the official and regenerated results differ so much? Was the official-data model trained to convergence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.