acceptodds
Under review as a conference paper at ICLR 2027

Protecting Small-Coefficient Terms in PDE Discovery with a Physical-Category Magnitude-Disentangled Transformer

Abstract

PDE discovery recovers the governing equation from observed data. An equation has terms and coefficients. Recovering it has two steps: choosing terms, and fitting coefficients. Terms often have coefficients of very different sizes. Existing methods compare all terms together in one step. Large-coefficient terms then dominate, and small-coefficient terms are dropped when choosing terms. When a small term is dropped, coefficient fitting gives biased values to compensate, so the recovered equation is incomplete. We introduce the Physical-Category Magnitude-Disentangled Transformer (PC-MDT) with two mechanisms. Masked local category competition compares each term only with terms of the same physical category, time, space, or nonlinear, and a learned gate exchanges information across categories only when the equation needs it. Soft-gated global assembly elects at most one term per category or none and fits coefficients only after the structure is fixed, so size plays no role in the choice. On four benchmark equations, PC-MDT recovers all exactly, while every baseline fails on at least one. A model trained on heat transfers to Burgers with two of three true terms already in place.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
PDE Discovery, Symbolic Regression, Scientific Machine Learning, Transformer, Sparse Regression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SnTtOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PC-MDT for PDE discovery, aiming to retain physically relevant terms with small coefficients. It partitions candidate terms into temporal, spatial, and nonlinear categories, combines category-restricted attention with gated cross-category interactions, and selects at most one term per category before fitting coefficients. Residual-based feedback guides iterative search. Validation includes four synthetic PDE benchmarks, coefficient sweeps, transfer experiments, and ablations. The method achieves exact recovery on the noise-free benchmarks, but complete recovery fails at all tested nonzero noise levels.

Critical strengths
  1. The paper addresses an important failure mode in PDE discovery: omitting small-coefficient terms can remove genuine physical mechanisms. Category-based selection provides an interpretable inductive bias for tackling this problem.
  2. Coefficient sweeps, multiple seeds, and component ablations provide useful evidence beyond isolated benchmark successes. The reported noise failures also make the method’s current limitations explicit.
Critical weaknesses

1.The proposed mechanism is insufficiently substantiated. The softmax bound does not establish that small coefficients produce low attention scores, and removing the category mask preserves perfect recovery on all four main benchmarks, weakening attribution to a central component. 2.The experiments do not adequately separate architectural gains from restrictive search assumptions and baseline thresholds. The small candidate library, one-term-per-category constraint, and coordinate-descent refinement warrant simple search controls, while fixed baseline thresholds disadvantage recovery of coefficients below those thresholds. Failure at every tested nonzero noise level further limits practical relevance.

Questions for authors

same to weaknes

Reviewer_emxeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

PDE discovery aims to recover governing PDEs from observed data. Popular methods choose a basis library and a set of parameters which are fitted by sparse regression to minimize the equation residual. However, the terms in a candidate library have coefficients that differ by order of magnitude. Large coefficient terms dominate and small coefficient terms are suppressed. When the small coefficient terms are removed the remaining terms become biased to compensate for the missing terms. This imbalance is said to occur due to a single shared normalization shared by the candidate terms.

Critical strengths

The method targets a limitation of existing discovery methods which makes them limited for handling small coefficients due to a similar normalization applied across terms. By using a divide and conquer method across categories of terms the method can preserve small coefficients by considering terms of each category with others of the same category and going across categories when required by the equation.

Critical weaknesses

The paper does not do a good job of situating the method within prior work. There is discussion of the SINDy-like methods and one of those methods is said to be closest (BG-SINDy). But that closest method is not a transformer method.

The main problem I see is one of scope: is the method correcting/improving small coefficient retention over all these methods. Or just in the transformer methods listed. If the latter then most of the discussion in the introduction (regarding the SINDy methods) is not relevant. In that case there needs to be a more thorough discussion of the transformer models and their limitations and how the current method addresses those. If the former, then there needs to be a more thorough analysis of the sparse regression methods that causes them to lose small coefficient terms.

The prior transformer models are not described well. There should at least be a clear description of the methodology or architecture used by the transformer models for discovery. Only the transformer attention mechanism is described in section 3. For instance, how is sparse regression replaced by attention as mentioned in lines 124-125. If the goal of the paper is a transformer model then the SINDy description is really not relevant and can be replaced by a thorough and clear description of discovery with transformers.

It is also not clear to me which of the components of the model are new and which ones come from prior work. The actual method for preserving small coefficients is not clearly described. There should be clear intuition given for the method before the details. The current overview in section 3 is not very enlightening for the PDE discovery task.

Questions for authors

What exactly is the shared normalization in the prior methods (such as BG-SINDy, PDE-Learn) that hurts coefficient magnitude balance? From my understanding BG-SINDy is not a transformer method and proposes only a new regularization objective.

The paper says that the closest work is BG-SINDy. is there an analysis of the limitation of the method that does not protect small coefficient terms? and how does the method here correct that limitation?

Is the main problem that this paper targets a matter of scale dependence in the parameters (coefficients)? In that case how does the method compare with classical methods for dealing with differences of scale. For instance, natural gradient is one approach that comes to mind. I think classical approaches to such problems of scale should at least be discussed in the paper.

Can you explain the tokenization into temporal, spatial and nonlinear tokens?

Would this method work with cross category terms: such as spatial and temporal?

What is the size of the candidate terms for each equation? How much sparsity does the model handle?

Reviewer_4c8nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper makes five contributions:

  1. Per-category selection head with an empty option. (2) Masked attention within each category. (3) Gates between categories , computed using residual statistics. (4) Fitting coefficients only after the structure is fixed, scored by the least-squares residual. (5) Transfer between equation. The paper validates a structural prior together with a gate on clean benchmarks that fit that prior. It does not validate the claim in its title: that separating categories during normalization is what protects small-coefficient terms. The one experiment that tests the masking directly, no_intra, shows no effect.
Critical strengths

The per-category selection head with an empty option is a clean, interpretable structural prior, and it appears to drive the main results. The reviewer encourages the authors to make it the center of the paper and to isolate its effect with a direct ablation.

Critical weaknesses

The title, abstract, and method section all rest on one claim that a softmax normalization shared across all candidates suppresses small-coefficient terms, and that normalizing within each category removes this suppression. The paper’s own results point elsewhere:

  1. The direct test shows no effect. The ablation that removes the within-category masking, no_intra, matches the full model in every cell of Table 2.
  2. The baseline failures have other causes, and the paper names them.None of these is the softmax mechanism.
  3. The mechanism has no visible path to the inputs. The tokens are a term embedding plus a category embedding (Eq. 3), so fitted coefficient values never enter the attention scores.
Questions for authors
  1. What prevents the model from selecting extra terms?
  2. How is the target chosen: u_t or u_tt?
  3. What exactly does each variant contain? Does “identical post-processing” in Table 2 include the 0.15 STRidge threshold that Vanilla uses.
  4. Can you ablate the per-category readout alone? Replace it with a single readout over all candidates, and keep the masks and the gate unchanged.
  5. What are the results without the coordinate-descent revisit (Appendix E), on all four equations and the full sweep? The authors state that the revisit is what lets KdV converge at \beta \leq 0.1.
  6. Were the baseline thresholds tuned?
  7. How would the method handle equations with two terms in the same category? Examples are advection–diffusion (u_x and u_xx), Kuramoto–Sivashinsky (u_xx and u_xxxx), and KdV–Burgers. Can the at-most-one-term rule be relaxed, and what would that do to the search?
  8. Why does every method fail at 1% noise?
  9. Fig. 5 shows 300 iterations on its x-axis, but the caption says 500.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.