acceptodds
Under review as a conference paper at ICLR 2027

From Objectives to What Models Learn: A Landau Theory of Invariant Learning

Abstract

Invariant-learning objectives pursue similar goals yet produce qualitatively different regularization paths, leaving unclear when shortcuts can be suppressed without damaging stable modes. Our starting point is simple: in the desired shortcut-suppressed regime, shortcut loading is small, so the objective's low-order expansion governs local stability and residual amplitude. This brings the problem into the domain of Landau phase-transition theory. We establish a mathematical isomorphism between the near-critical normal form of predictive-mode learning and Landau theory, identifying the learning objective as an effective free energy and critical-mode amplitude as an order parameter. The resulting low-order objective signatures predict distinct regularization phenotypes: quadratic terms shift phase boundaries and enable finite-strength elimination, whereas quartic terms continuously attenuate acquired modes while leaving nonzero residual loading at finite strengths. Higher-order terms may further shape nonlinear tails. In a canonical bilinear model, we derive exact phase boundaries and equilibrium loadings, identifying conditions for a selective-retention window in which shortcut suppression preserves stable structure. Controlled bilinear and ReLU experiments, including an MNIST construction, support the predicted signature–phenotype relation, while coupled-feature experiments validate its extension to collective modes. The framework connects the mathematical structure of invariant-learning objectives to what models learn as regularization varies.

Primary area:
learning theory
Keywords:
invariant learning, Landau theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Xf9mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies why invariant-learning objectives produce different regularization paths, particularly whether they suppress shortcut features while preserving stable predictive structure. The main idea is that near a continuous transition, the objective can be reduced to a Landau expansion: quadratic terms R_2 change the stability of the zero-loading state, whereas quartic terms R_4 control the amplitude of an acquired mode. For a canonical bilinear model, the paper derives exact critical strengths and equilibrium loadings (Theorem 4.1 and Proposition 4.3), and characterizes a selective-retention window in which the shortcut disappears while the stable mode remains (Corollary 4.2). The authors then extend the analysis to coupled features through a spectral phase-boundary condition (Theorem 5.1), and derives local signatures for existing invariant-learning objectives (Table 1). Bilinear, ReLU, and controlled MNIST experiments test the predicted onset shifts, loading laws, and qualitative regularization paths.

Critical strengths

A clear distinction between changing mode stability and reducing mode amplitude

The paper identifies a concrete question that is not apparent from the usual formulations of invariant-learning objectives: does increasing regularization actually make the shortcut-free state stable, or merely reduce the strength of a shortcut that remains acquired? The canonical analysis gives an explicit answer. In Eq. (13), R_2 changes the numerator of the equilibrium loading and can eliminate the mode at a finite strength, whereas R_4 increases the denominator and continuously attenuates it. Figures 2–4 test onset shifts, shortcut loadings, and critical strengths separately, providing more direct evidence for this mechanism than OOD accuracy alone. Appendix E also examines susceptibility and critical slowing down.

The collective-mode analysis explains why feature-wise suppression can destroy stable information

Section 5 addresses the case in which individually environment-sensitive features form a stable combination. Theorem 5.1 characterizes local onset through the joint task and environmental-sensitivity matrices, while Proposition 5.2 distinguishes penalizing representation gates from penalizing the final predictor.

Figure 7 provides a controlled illustration: diagonal penalties damage the compensatory stable pair, whereas full-matrix oracle penalties preserve the combination while suppressing the shortcut. This is a consequential extension because it shows that identifying which individual features vary across environments is insufficient for selective retention. The paper also acknowledges that recovering the oracle sensitivity matrix from finitely many observational environments requires additional assumptions.

Critical weaknesses

The connection between the local signatures and selective retention in nonlinear networks remains insufficiently established

The exact results concern the canonical bilinear model. The general-network reduction in Theorem B.2 requires a stationary reference branch, a single critical Hessian direction with positive transverse curvature, a local sign-reversal symmetry, and a positive quartic coefficient. Appendix B.3 further limits the ReLU argument to the interior of a fixed activation region. The smooth-network controls in Appendix E support this reduction, but the ReLU and MNIST experiments mainly compare qualitative loading curves without establishing the relevant local coefficients or assumptions.

In particular, selective retention concerns a shortcut-free state in which stable features remain active. The signatures derived at the zero reference in the bilinear probe do not establish the effective curvature around that partially learned state in a network with shared parameters. These caveats are acknowledged, but the paper does not yet demonstrate how to determine the applicable signature in the nonlinear experiments before inspecting their paths. This limits the broader claim of predicting behavior directly from objective structure.

Finite-budget training leaves uncertainty about the equilibrium interpretation

The paper studies equilibrium loading, but the principal path experiments use fixed optimization budgets: 2,500 Adam steps for Figure 4, 3,000 for Figures 5–7, and 2,000 mini-batch updates for MNIST. Although those experiments carefully validate local relaxation predictions, they do not establish convergence of the Adam runs used for the main regularization paths. Without convergence diagnostics or training-budget comparisons, small residual loading, apparent elimination, or collateral feature loss can partly reflect optimization transients.

The paper writing assumes familiarity with Landau theory and physics terminology and is difficult to follow for a machine learning audience

The paper relies heavily on physics terminology, including “order parameter,” “soft at criticality,” “effective mass,” and “quartic stiffness,” without providing a sufficiently accessible introduction for readers with a machine learning background. For example, Section 3 introduces (q) as a parameter combination that becomes “soft at criticality,” but this description requires understanding the physical terminology before the reader can understand the mathematical object being defined.

Although the paper provides formal derivations, the presentation would benefit from first explaining the argument using familiar optimization concepts: Hessian curvature, stability of stationary points, and Taylor expansion along a critical direction. A worked example showing how the original parameters are reduced to (q), followed by a glossary connecting the physics and optimization terminology, would make the contribution substantially easier to assess. Because the Landau correspondence is the paper’s central contribution, this accessibility issue materially limits how effectively the paper communicates its results to the intended machine learning audience.

Questions for authors
  1. Local signatures around the selective-retention state. Can the authors identify the shortcut-free reference branch with stable features retained in a nonlinear experiment, measure its critical Hessian direction and reduced coefficients, and predict the shortcut transition without fitting the observed loading path? This would directly test whether the bilinear signature classification transfers to the state of interest.

  2. Equilibrium and calibration coverage. How do the estimated critical strengths and residual loadings change with longer training or a stationarity-based stopping criterion?

Reviewer_NWxNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether penalties that encourage similar behavior across training environments can remove shortcuts while preserving useful features. In a simple two-layer linear model, the authors show that some penalties can eliminate a feature at a finite penalty strength, while others only shrink its contribution. They use a local approximation of the loss to explain these differences and derive conditions for retaining useful features after removing a shortcut. The authors then perform controlled experiments which support the predictions, and experiments with nonlinear networks which show similar qualitative behavior.

Critical strengths
  • (S1) The analysis explains why some invariance penalties can remove a feature while others only weaken it. Controlled experiments support the resulting quantitative predictions.
Critical weaknesses
  • (W1) The paper does not do a good job at exposition and at contextualizing the contributions. This makes the central argument unnecessarily difficult to follow. The paper introduces a physics analogy and terminology before a concrete example. It is therefore hard to see how the abstract penalties arise from comparisons across environments. Introducing the mean matching and covariance matching examples from the appendix earlier would make things clearer.

  • (W2) The connection to nonlinear networks remains incomplete. The theory makes assumptions about the loss near the point where a feature stops being used, but the ReLU experiments do not check these assumptions. A penalty's behavior can also depend on which other features are already learned.

Questions for authors
  • (Q1) Can you identify the critical branch and measure the reduced quadratic and quartic coefficients in a nonlinear experiment?
Reviewer_4Ao4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how invariant-learning objectives suppress features that are predictive only in some environments while retaining stable features. Through a local expansion of the objective, it distinguishes regularizers that eliminate a feature at a finite strength from those that continuously shrink its contribution. The paper derives explicit results for a bilinear model and uses controlled experiments to investigate their extension to nonlinear networks.

Critical strengths
  • Understanding why different invariant-learning objectives produce different outcomes is an interesting and relevant question.
  • The explicit calculations in the bilinear model provide an interpretable comparison between different regularizers. The controlled experiments help illustrate these differences.
Critical weaknesses
  • Unclear motivation and setting: The paper introduces substantial terminology before providing a clear account of the model and the quantities being studied. For instance, “we denote by q the effective order parameter” assumes familiarity with the physical interpretation before explaining what q represents in the learning problem. Its construction is given later, but switching between q, the parameter w, and the predictive coefficient makes the main discussion difficult to follow.
  • Unclear novelty: It is unclear what additional insight the Landau interpretation provides beyond directly analyzing the bilinear objective. In particular, the elimination and shrinkage results follow from relatively simple calculations in the canonical model. The paper would benefit from distinguishing the new conclusions about invariant learning from a reformulation of these calculations in the language of phase transitions.
  • Limited scope of the conclusions: The general reduction is local and requires specific smoothness, symmetry, and nondegeneracy assumptions. It is not sufficiently clear which conclusions extend to general neural networks or entire regularization paths. The experiments provide evidence for such extensions, but do not establish the assumptions needed by the theory.
  • Insufficient justification of the dynamical interpretation: Proposition E.1 appears to identify coefficients obtained by eliminating auxiliary parameters at equilibrium with those governing the reduced training dynamics. These reductions need not agree away from the critical point. This step needs further justification, particularly since the paper also makes predictions about learning dynamics.
Questions for authors
  • What does the order parameter represent in a concrete learning example, and how would it be identified from the network parameters?
  • Which prediction about invariant learning requires the Landau formulation and would not follow from a direct analysis of the canonical objective?
  • Under what conditions do the equilibrium calculations also describe training dynamics? How is the identification in Proposition E.1 justified?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.