acceptodds
Under review as a conference paper at ICLR 2027

Efficient and Expressive Equivariance Discovery and Enforcement using Vector Fields

Abstract

Symmetry plays an increasingly important role in the design and analysis of machine learning models, offering improvements in both generalizability and interpretability. Recent work has demonstrated the discovery and enforcement of continuous symmetries using the Lie derivative along vector fields; however, such approaches either do not explore expressive symmetries, are limited to the topic of invariance, or rely on heuristic elbow curve criteria to detect the number of symmetries, leaving nonlinear equivariance and practical scalability underexplored. In this work, we build upon the vector field approach in two significant ways. First, we introduce a PCA-enhanced formulation for discovery that improves numerical stability and computational efficiency, while replacing heuristic elbow curve selection with principled explained variance criteria. Second, we generalize the approach from invariance to equivariance, enabling nonlinear continuous discovery and enforcement in the context of equivariance. Our empirical results demonstrate a distinct computational advantage, being at least an order of magnitude faster than competing methods. We further demonstrate that vector field regularization can improve generalizability under out of sample test distributions, providing a flexible alternative to approaches that enforce equivariance architecturally. Our results thus establish the vector field approach as a scalable, practical, and flexible tool for symmetry-based learning beyond invariance.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
machine learning, equivariance, Lie derivative, group actions, turbulence closures, projective symmetry, conformal symmetry

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V3zZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a PCA-based method to discover symmetries. As opposed to the method it builds upon, it solves the problem without the need for elbow curve heuristics and extends it to equivariance. As opposed to other methods in the literature, it is able to solve symmetry discovery problems effectively and in dramatically less time than the baselines, under the evaluated scenarios.

Critical strengths
  • I believe that the motivation that current methods do not explore expressive symmetries and are limited regarding nonlinear equivariance is sharp.
  • The writing of the paper is of high quality.
  • The method is simple and improves over several baselines on the selected benchmarks.
Critical weaknesses
  • The novelty is minimal. The paper builds heavily on the computation of the nontrivial nullspace for discovering symmetries, which is at the core of the method, but this is a construction from another paper as the authors correctly acknowledge throughout. Therefore, the authors' contributions go along the lines of (i) proposing the nullspace-based discovery extension to equivariance, and the enforcement of both invariance and equivariance through regularization (which in my opinion is a fairly direct extension) and (ii) avoiding elbow curve heuristics and resorting to using "least Principal Components" with a threshold for solving the symmetry nullspace computation problem. I believe these contributions to be correct and real, but I am not convinced this is enough to constitute a conference research paper in a top venue, at least by itself. The empirical validation and limitations also do not contribute to compensating for this, as I describe in the next point.
  • I believe the practical usefulness of the method is limited. The method performs well in synthetic settings with exact symmetries, and in the turbulence setting where the symmetry is known a priori. Several methods cited in the related work have been applied to more "realistic" scenarios e.g. on image data (e.g. LieGAN and Augerino on rotated MNIST) but the proposed method does not evaluate this or similar scenarios, and I believe it cannot be applied to those (at least in its current formulation). Taking MNIST as an example and following the use of the Euclidean Killing basis used in the paper, this yields a search space of q=d+(d2)=307,720q = d + \binom{d}{2} = 307{,}720 candidate fields for d=784d = 784 (dd translations ∂xi\partial_{x_i} and (d2)\binom{d}{2} rotations xi∂xj−xj∂xix_i\partial_{x_j} - x_j\partial_{x_i}, one per pair of axes). The resulting matrix has NmNm rows, one per sample and output component of FF. It is therefore underdetermined for N=60,000N = 60{,}000 and small mm. In this setting, the method basically reports hundreds of thousands of "discovered symmetries", independently of the data, since there are many more unknowns than equations in the system. I believe this represents a failure mode of the method. On another note, even for m=1m = 1, a matrix of size 60,000×307,72060,000 \times 307,720 also requires significant memory. All this, despite MNIST being close to a toy dataset by current standards. These are in my opinion very strong limitations that should be discussed in the paper.
Questions for authors
  • Could the authors clarify whether the method can be applied to image data or other datasets with more relevant practical use?
11 comments

We thank the reviewer for the time given to review our work. As we wish to address the reviewer's concerns thoughtfully, our response will span two comments, the first addressing the perceived weaknesses and the second addressing the question.

Weakness 1: We appreciate the acknowledgement of the novelty of our approach and wish to address concerns about the "sufficiency" of the novelty. Primarily, we are aware of no other method which has successfully performed nonlinear equivariance discovery and enforcement, so that our method is filling a cruicial gap in the literature. We also believe there is some novelty present in connection with section 4.3.1. Our method can also be used for nonlinear invariance discovery and enforcement, as well as linear equivariance enforcement, as can many existing methods, perhaps appearing to duplicate previous work. But crucially, we have empirically shown that our method consistently outperforms a representative sample of existing methods, most especially in terms of computation time, with our method commonly being multiple orders of magnitude faster when compared with other methods. Superiority over state of the art methods should be taken into account when assessing the "goodness of fit" for top-tier publication venues.

Weakness 2: We wish to address the issue of practical utility while kindly offering corrections to some assertions and assumptions made by the reviewer. The perceived scaling problem with applying our method to MNIST appears to begin with the assertion that the group action for our method is different than that for existing methods such as LieGAN: where LieGAN need only consider group actions in the 22-dimensional plane, our method is said to need to consider the affine group acting on R784\mathbb{R}^{784}. In practicality, the task of symmetry discovery for image data would begin with the careful consideration of the precise group action on the images, and the group itself--in particular, the dimension thereof--would be no different for our method than that of existing methods: a 6-parameter symmetry discovery task for LieGAN is a 6-parameter, not 307,720307,720-parameter, task for our method.

The confusion appears to lie in the assumption that the affine action on images is free, uniquely when applying our method, to translate a single pixel but not all others simultaneously. Indeed, the transformation induced by the proposed generator of ∂xj\partial_{x_j} would translate only the pixel associated with dimension jj. The simultaneous action of the affine group on each pixel can be captured using the concept of diagonal group actions, and the construction of infinitesimal generators for diagonal group actions is given explicitly in appendix C3. The six generators for the diagonal action of the affine group on 784 copies of R2\mathbb{R}^2 in coordinates (xi,yi)(x_i,y_i) are as follows:

X1=∑i=1784∂xi,X2=∑i=1784∂yi,X3=∑i=1784−yi∂xi+xi∂yi,X_1 = \sum_{i=1}^{784} \partial_{x_i}, \qquad X_2 = \sum_{i=1}^{784} \partial_{y_i}, \qquad X_3 = \sum_{i=1}^{784} -y_i \partial_{x_i} + x_i \partial_{y_i}, X4=∑i=1784xi∂xi+yi∂yi,X5=∑i=1784yi∂xi+xi∂yi,X6=∑i=1784xi∂xi−yi∂yi.X_4 = \sum_{i=1}^{784} x_i \partial_{x_i} + y_i \partial_{y_i}, \qquad X_5 = \sum_{i=1}^{784} y_i \partial_{x_i} + x_i \partial_{y_i}, \qquad X_6 = \sum_{i=1}^{784} x_i \partial_{x_i} - y_i \partial_{y_i}.

Moreover, the task of symmetry enforcement requires advance knowledge of the symmetries, be they pre-discovered or else prescribed by the nature of the problem. Thus, the applicability of the turbulence example is not diminished in the slightest by the fact that the symmetry group is specified in advance: it is a consequence of the Navier-Stokes equations.

Continuing our initial response, we address the reviewer's question.

The subject of Reynolds stress prediction is highly applicable in the field of ML-enhanced computational fluid dynamics. More generally, there is no shortage of applicability when it comes to symmetry enforcement: missing from existing literature is a principled, expressive, and flexible way to enforce a wide variety of symmetries without explicit constraints on model architecture. Nonlinear symmetry enforcement via vector field regularization helps to fill this gap. The turbulence closure case is linear, however one need look no further than the Hamiltonian vector field for insight into applicability for nonlinear symmetry enforcement in the context of physical systems.

On the subject of symmetry discovery, vector field discovery most certainly applies to image data, though not in the way assumed by the reviewer, as discussed in the previous comment. Herein, we avoided application to images in part to avoid potential duplication with potentially-ongoing efforts (see https://openreview.net/forum?id=RconNmjCoW).

But the question itself implies an underappreciation for the broad applicability of vector field symmetry discovery: the method is not limited to discovering symmetry apparent in a particular dataset. In appendix A2, we explicitly compare the task of model symmetry discovery to that of LieNLSD. Model symmetry discovery is a particularly applicable task, allowing one to probe a blackbox (and smooth) model for expected generalization behavior.

We have evaluated symmetry discovery in many elementary settings in order for the ground truth to be well-understood, but also to reveal the prohibitive scalability and performance-related issues present in competing methods. The fact of competing methods taking so much longer while also underperforming when compared with ours even on the simplest of symmetry discovery tasks demonstrates the acute need of more efficient--and practical--methods.

Reviewer_V3zZOpenReview ↗

Rebuttal reply

I thank the authors for the detailed clarifications.

Regarding Weakness 2: I understand that the correction is that, for image data, one should not search over the Euclidean Killing bases of the data space R784\mathbb{R}^{784}, but over the 6-dimensional planar affine algebra acting on the pixel positions in R2\mathbb{R}^{2}. I agree this is the right formulation to attack the proposed problem, and therefore my previous statement that the method cannot be applied to images is overly strong. I withdraw that statement.

However, I should point out that my application to images follows directly from applying the construction in the paper. Section 4.2.1 defines the vector field bases on the input space of F:Rd→RmF:\mathbb{R}^d\to\mathbb{R}^m, and the discovery experiments take this to be the Killing basis of the data space itself. For MNIST, the data space is the space of pixel intensities, which indeed gives the d=784d=784 case I described in my review. Indeed, one would not choose this way in the case of MNIST, but the formulation in the paper leads to that. I would argue that the most sensible approach is to make such constructions explicit in the paper, as to show how the method applies to such cases which are of broad interest. This change would correctly address the objection of image applicability I made in my review, as well as guide the reader into how or whether the method can be applied in different cases. I would therefore ask the authors to (i) add a formulation to the method section of the paper that contemplates these cases (in which the generators act on the pixel intensities that FF takes as input, which might need some extra formalizations), (ii) discuss the scaling of the generic basis and corresponding memory, and (iii) empirically validate the image case against other baselines. Method-wise, this would be an accurate description of the intended application of the method on different cases. Experimentally, a reassuring result would for example discover a rotation generator on rotated MNIST, as e.g. LieGAN does, and would put this method in equal grounds to several other symmetry discovery methods, which are validated on images and therefore show stronger evidence of their applicability.

Regarding the broad applicability claim through the lens of model symmetry discovery made by the authors in the rebuttal: I understand this is the case, but the evidence presented in the paper does not show that this method is strong for this particular application, neither the "probing a model for expected generalization behaviour" claim made in the rebuttal is supported in the experiments of the paper. In effect, Appendix A.2 shows (and acknowledges) that the method does not show a clear accuracy advantage over the baseline LieNLSD. It is indeed faster though, about 3x times, but I am not convinced this is such a good tradeoff. After all, discovery is a one-off analysis, so in practice solving it in 77 seconds vs 232 seconds as for the N=40,000N=40,000 case does not change a lot. Speed is interesting as a proxy for scaling, but I understand that both methods here scale linearly in NN, so in conclusion, I am not convinced that the applicability to the model symmetry discovery is a strong point of the proposed method. In addition, these are both toy experiments on exp(x2+y2)exp(x^2 + y^2) and 2xy/(x2+y2)2xy/(x^2 + y^2) with known symmetries. It is no closer to practical use than the surfaces of revolution.

However, the "probe a model for expected generalization" experiment would be very interesting, and I believe this angle would strengthen the paper a lot, but as I pointed out, it is asserted on the rebuttal response but not supported by the evidence in this paper, therefore all we can claim is that in principle, it could be an application of this method. Overall, I am pushing back a bit on the authors' argument for applicability through the lens of model symmetry discovery. Additionally, in that regard, model symmetry discovery is also a capability available to other methods (such as the LieNLSD and apparently, with a slight accuracy edge) or such as LieGG [1] (which is not discussed in the related work section).

Reviewer_HzbjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper efficiently discovers continuous equivariance present in data by expressing an infinitesimal generator of the Lie group as a vector field. Given candidate vector field bases, the method determines a linear combination of bases representing equivariance present in data. If we use a nonlinear basis, we can also find nonlinear group equivariance.

Critical strengths

The paper proposes a computationally efficient symmetry discovery method using PCA and SVD, achieving more than an order-of-magnitude speedup over existing methods while maintaining competitive accuracy.

Critical weaknesses
  1. To discover nonlinear equivariance, the method requires a predefined nonlinear vector field basis and corresponding input-output generators, which require strong prior knowledge of the underlying symmetry. Therefore, I am not sure whether the method can truly discover nonlinear equivariance rather than identify it within a predefined search space.
  2. The experiments do not include nonlinear equivariance discovery. If the paper claims that the method can discover nonlinear continuous equivariance, this should be demonstrated experimentally. However, the equivariance discovery experiment only considers linear group actions.
Questions for authors

None

5 comments

We thank the reviewer and have addressed the points raised below.

Weakness 1: Our method is parametric, and all parametric machine learning methods require the specification of a parametric family. The task of discovering nonlinear equivariance is thus identical to the task of identifying nonlinear symmetries within a predefined search space. This is not unique to our method. For example, LieGAN requires the specification of a Lie algebra of (matrix) generators: the only difference is that our generators are represented as vector fields. LieNLSD also requires the specification of a parametric family of vector fields.

Weakness 2: We see the reviewer's point and have conducted two experiments to explicitly demonstrate nonlinear equivariance discovery. The first experiment serves as a simple baseline, while the second is intended to point in the direction of potential applications. The experiments are summarized below and will be included in the revision.

(1) We consider the task of discovering SL$(2)$ equivariance of a function F:R2→RF: \mathbb{R}^2 \to \mathbb{R} defined by F(x,y)=xx+yF(x,y) = \dfrac{x}{x+y} on the region (0,4)×(0,4)(0,4) \times (0,4). Bases for the vector field infinitesimal generators for the input and output spaces can be given as

X1=y∂x,X2=x∂y,X3=x∂x−y∂y,X_1 = y\partial_x, \qquad X_2 = x\partial_y, \qquad X_3 = x\partial_x - y \partial_y, Y1=∂u,Y2=−u2∂u,Y3=2u∂u,Y_1 = \partial_u, \qquad Y_2 = -u^2 \partial_u, \qquad Y_3 = 2u \partial_u,

where uu is used as the coordinate for the output space. The ground truth equivariance is given by the single pair (X2,Y2)(X_2,Y_2), so that the equivariance discovery task is nonlinear. Generating 1,0001,000 points uniformly on the grid [0.001,4]×[0.001,4][0.001,4] \times [0.001,4], we train a simple neural network to learn the ground truth labels of x/(x+y)x/(x+y). Subsequently, we conduct equivariance discovery using automatic differentiation of the trained model, as with other model symmetry tasks. The discovered vector field pair is given as

X^=(−0.0311x+0.0154y)∂x+(0.9994x+0.0311)∂y,Y^=(−0.9994u2−0.0622u+0.0154)∂u,\hat{X} = (-0.0311 x + 0.0154y)\partial_x + (0.9994 x + 0.0311) \partial_y, \quad \hat{Y} = (-0.9994 u^2 - 0.0622 u + 0.0154) \partial_u,

which is numerically close to the ground truth pair of X2X_2, Y2Y_2.

(2) In the second experiment, we consider the task of time series forecasting from trajectories in 2-dimensions. The trajectories are assumed to be generated from sequential image data in which an object is tracked using a segmentation model, and the trajectory data is obtained from centroids of bounding boxes. Thus, the prediction task should be symmetry aware, owing to potential camera misalignment: a model which is equivariant under projective transformations is thus warranted. Symmetry discovery post-training in this task serves the purpose of diagnosing model symmetry, as in the source handling affine transformations pointed out by reviewer AgnD.

The machine learning task is to use 10 historical points to forecast 5 subsequent points. The action on the input sequences is thus the diagonal action of the projective transformations on each of the 10 copies of points in R2\mathbb{R}^2. The action on the model output is similar, though with only 5 simultaneous planar points.

We generate 8,0008,000 trajectories using a quadratic function of time tt: b+vt+a2t2b + v t + \frac{a}{2} t^2. Having control of the simulated data, we augment the dataset with transformed copies of baseline trajectories using projective transformations, encouraging the trained model to learn to exhibit projective equivariance. Post-training, we evaluate the trained model for equivariance. To uncover a subset of a larger set, we consider the set of all vector fields on R2\mathbb{R}^2 with quadratic components, which is a 12-dimensional search space (there are eight independent projective transformations in the plane). All 12 singular value proportions in the discovery task are given as

σ1=0.47829568,σ2=0.19100548,σ3=0.10995245,σ4=0.08552236,\sigma_1 = 0.47829568, \quad \sigma_2 = 0.19100548, \quad \sigma_3 = 0.10995245, \quad \sigma_4 = 0.08552236, σ5=0.06910881,σ6=0.02981168,σ7=0.01077973,σ8=0.00882984,\sigma_5 = 0.06910881, \quad \sigma_6 = 0.02981168, \quad \sigma_7 = 0.01077973, \quad \sigma_8 = 0.00882984, σ9=0.00586524,σ10=0.00521935,σ11=0.00321494,σ12=0.00239443.\sigma_9 = 0.00586524, \quad \sigma_{10} = 0.00521935, \quad \sigma_{11} = 0.00321494, \quad \sigma_{12} = 0.00239443.

A cutoff proportion value of $ \sigma < 0.1$ yields a 9-dimensional space, though there is arguably a larger (relative) gap between σ5\sigma_5 and σ6\sigma_6, which if used to define the cutoff would yield a 7-dimensional space. Either case includes quadratic vector fields, since the affine space is 6-dimensional.

Thank you for the detailed clarification. However, I still have a concern regarding your rebuttal.

Existing methods sharing the same limitation does not establish the practicality of the proposed approach for discovering previously unknown nonlinear symmetries. Unlike linear symmetries, nonlinear symmetries have a much broader space of possible generators, making it difficult to construct an appropriate basis without substantial prior knowledge. For example, in the added SL(2) experiment, the true generator (X2X_2) was already included among the three candidate generators, allowing the method to identify the correct nonlinear equivariance. I acknowledge that the second experiment demonstrates the discovery of candidate nonlinear symmetry directions from a generic quadratic basis. However, the recovered subspace has either 7 or 9 dimensions, whereas the projective Lie algebra is 8-dimensional. Without quantitatively comparing the recovered generators with the ground-truth projective generators, it remains unclear whether the method successfully recovers the intended nonlinear equivariance.

Your baseline LieNLSD also shares this problem. Therefore, the main contribution over LieNLSD appears to be improved computational efficiency when a suitable basis is predefined. While this is a useful improvement, I am not convinced that it substantially advances the practical discovery of previously unknown nonlinear symmetries.

In addressing lingering concerns, we first offer additional clarification regarding the problem definition specifically as it relates to invariance discovery, since LieNLSD was invoked more especially. The task of symmetry discovery is not to describe the set of all vector fields under which a given function is symmetric. In fact, such an unconditional task is mathematically trivial: for a smooth, real-valued function, this set of vector fields can be described using a basis of the nullspace of the Jacobian matrix--a basis, that is, for the module of vector fields over the ring of smooth (locally, at least) functions. The vector-valued case is no more difficult conceptually: one instead uses the nullspace of the matrix constructed by vertically stacking the Jacobian matrices for each component. However, this vector field basis does not have an appealing structure, as the basis may not define a Lie algebra, so that the transformations associated with the vector field basis may not define a group. In particular, the composition of two transformations may not belong in the set. For example, the continuous symmetries of the 2-sphere are the rotations about the origin: however, the nullspace of the Jacobian of x2+y2x^2+y^2 can be described by only two vector fields, with the third being a linear combination (over the ring of functions) of the other two. And yet, a Lie algebra basis for the symmetries is 3-dimensional (e.g. rotations about each coordinate axis).

The task at hand, as with all other recent continuous symmetry discovery literature to our knowledge, is to identify a symmetry group, and our nonlinear extension merely allows this process to take place even when the group action is nonlinear in the given coordinates/representation. The group actions our method can perform discovery with are those of continuously-parameterized groups. Therefore, the appearance of parametric families of vector fields is not an artifact of any particular method: rather, it is an inescapable fact of conducting symmetry discovery in the context of multi-parameter group actions. The parametric family assumption is not a weakness of our approach or of previous approaches, and there is no reason to suspect that a future method will characterize symmetry in the context of multi-parameter groups without invoking either the group parameters or else the basis of infinitesimal generators associated with the group parameters.

Without restriction, symmetry identification also becomes problematic. One may claim that the "ground truth" symmetry is given by XX, but owing to the freedom explained above, the ground truth symmetry is justifiably fXfX for some nonzero function ff. For this reason, there will always be vector fields that are not expressible as linear combinations (over real numbers) of discovered symmetries. In summary, the only way to accurately characterize all possible symmetries (again, in the invariant case) is to provide a module basis over the ring of functions, which does not afford desirable properties: we therefore restrict the search so as to recover a perfectly identifiable (and implicit) symmetry (sub)group action. In the end, we claim to find all GG-symmetries of a given function (e.g. distance- or angle-preserving transformations), not all possible and unconstrained quantity-preserving transformations.

The perceived dimensionality recovery problem for the new "projective discovery" experiment also deserves further discussion. Simply stated, there is no ground truth symmetry to compare with. With analytical examples, it can readily be shown that the function in question is, in fact, equivariant: for this reason, our paper offers several analytical examples. However, the model trained on augmented trajectory data is not necessarily equivariant. The failure to recover the intended 8-dimensional space is at least as attributable to the potential failure of the learned model to be equivariant as it is to the symmetry discovery task. (In fact, our symmetry discovery method is the evaluative tool for assessing nonlinear equivariance.) For this reason, we offered the first experiment to demonstrate recovery of ground truth, with the second method intending to contain aspects of interest for all three reviewers: for reviewer AgnD, it mirrors the "symmetry diagnosis" tool that exists in the linear case; for reviewer V3zZ, the application was image-adjacent, and for the present reviewer it dealt with equivariance discovery.

The comparison to LieNLSD in light of the new experiments is not quite correct, as LieNLSD is restricted to the task of invariance discovery, not equivariance. Indeed, the method of LIeNLSD is similar to Shaw et al., centering around the equation LX(f)=0\mathcal{L}_X(f)=0. Therefore, we are not merely introducing a more computationally-efficient method: we are introducing a method which can discover nonlinear equivariance, which no other method has done, to our knowledge.

Reviewer_AgnDOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a vector-based approach to learning an enforcing non-linear symmetry in a few different settings:

  • learning invariance of a data manifold
  • learning equivariance of a known map
  • enforcing or regularizing toward equivariance under known symmetries. The key tool in the work is deriving differential constraints that an invariant/equivariant function must obey, in terms of infinitesimal generators, written as a matrix-vector product equalling zero. The vector coefficients represent linear combinations of generators, and the equation is solved approximately using PCA. For invariance of a data manifold, the function in question is itself fit similarly, to represent the data manifold as the zero of a linear combination of basis functions.

All of the above settings are tested experimentally, with substantial improvement in runtime over baselines, and comparable performance in recovering the right symmetries.

Critical strengths

The framework for non-linear infinitesimal symmetries, their use in characterizing invariance/equivariance, and their estimation, is very clearly presented.

The runtime improvement in invariance discovery, and the equivariance discovery experiments are especially compelling.

Critical weaknesses

There are a few disadvantages relative to baselines which do not seem to be explored. First, one needs to choose a basis of vectors (and of functions for level-set discovery). Second, for equivariance, the bases for the input and output apparently need to be ordered correctly in order to match. It is unclear how one one achieve this without prior knowledge of the task, and is also a disadvantage not apparently share by other techniques. (Note the question below about whether existing techniques really can't treat equivariance.)

Additionally, what happens under less clean examples or real-world data, for example for invariance discovery?

Questions for authors

In what way does existing work treat invariance but not equivariance? Does the classical fact that equivariance gf(x)=f(gx)gf(x)=f(gx) is equivalent to invariance of ff under the action f↦g∘f∘g−1f\mapsto g\circ f\circ g^{-1} not make these equivalent (for example for generative-adversarial type discovery like Desai et al. or LieGAN)?

How does vector equivariance regularization compare to work such as Gruver et al.'s "Lie Derivative [...]" paper? Or to regularization by sampling group elements?

Do the authors believe a comparison to Finzi et al.'s EMLP is valuable? It appears one may be justified both in the theory and perhaps in experiments.

On line 103, is G⊆RsG\subseteq \mathbb R^s meant literally? Or do the authors mean GG is an ss-dimensional Lie group? (Note also line 113, where the group is naturally R2×SO(2)\mathbb R^2 \times SO(2).)

6 comments

We thank the reviewer for their thoughtful questions and have provided responses below.

Weaknesses: The need to select a basis of vector fields is actually not a relative disadvantage. The machine learning task is to determine the symmetry subgroup of a larger, prespecified group or search space, which is common to other methods: in the case of LieGAN, for example, a basis of (matrix) infinitesimal generators is pre-specified. The need to specify a basis of functions is a potential weakness of level set estimation, but not one introduced by our work: herein, we merely improve the scalability of level set estimation, where the task itself was introduced previously.

The need to align the symmetry generators does not require prior knowledge of the task. The search space (essentially, the basis of vector fields) must be specified in advance, and the input/output generators of that search space must be put into correspondence, but this is not especially prohibitive, requiring only knowledge of how the larger search space acts on both the input and output spaces. The machine learning task is to discover the subset of pairs (Xi,Yi)(X_i,Y_i) of basis elements under which the underlying function is equivariant, which subset does not need to be known to align input/output generators of the larger (and pre-specified, as noted above) search space.

Question 1: Much of the existing work dealing with symmetry discovery does not directly deal with symmetry enforcement, so that a distinction between invariance and equivariance may seem trivial. But invariant function construction is fundamentally different than equivariant function construction, which notion is elaborated upon in section 3.2 of the paper. Additionally, some existing methods apply only to the task of invariance, which does not justify the more general term of equivariance.

Question 2: In Gruver et al., the authors quantify the extent to which models are symmetric with respect to a given symmetry group. This is similar in spirit to the task of model symmetry discovery, explored by the method of LieNLSD as well as in the present paper, and may be seen as a special case/application of the present paper. We will include this source in the revised paper in the section on relevant work.

Regularization by sampling group elements is precisely the scenario in which vector field-based methods seek to avoid. A continuous, one-parameter group has infinitely many group elements: scaling to higher group dimensions amplifies the issue of obtaining a representative sample efficiently. Regularizing with transformed copies (using sampled group elements) also introduces the overhead of explicitly transforming the data. But most importantly, the group itself must be given explicitly in order for this to take place. In many examples, recovering the group from the vector field infinitesimal generators is trivial. But as pointed out in Shaw et al., the general case requires computing flows of vector fields, which in turn requires solving first order ordinary differential equations, which is difficult in the nonlinear case.

Question 3: We will add EMLP to the list of methods for the experiment in section 5.3, though the results will not change the narrative. Exact equivariance models such as EMLP are expected to outperform vector field regularization in exact equivariance tasks with affine/linear group actions, as in the case of the Reynolds stress prediction task. The EMLP method is not suited for the "projective" dataset, however, highlighting the methodological gap filled by nonlinear equivariance enforcement.

Question 4: The expression G⊆RsG \subseteq \mathbb{R}^s is meant literally. The group action defining rotations in the plane is commonly referred to as the action of SO(2)SO(2) in the plane, where the action itself is commonly understood. But an alternate representation is the action of the group [0,2π)[0,2\pi) (modular addition) via the rotation of (x,y)(x,y) by the given angle. Thus, line 113 could be made more precise by specifying that the angular parameter θ\theta is better described as [0,2π)∈R[0,2\pi) \in \mathbb{R}, which we will add to the revision.

Question 5: Our symmetry discovery method relies on pre-learned or pre-specified functions (gradients of a smooth function, that is): if the function being examined for symmetry is misspecified, the symmetry discovery task may fail. In particular, level set estimation using polynomial basis functions is susceptible to noise, and both level set estimation and symmetry discovery using high degree polynomials may amplify the effect of far-away points. In the case of noise, thresholds for singular values should be increased. Additional and standard pre-processing tasks can be employed, such as standardization, which can mitigate these effects. (zz-scoring is also a bijective transformation, allowing symmetries in the original data coordinates to be recovered from the symmetries in the standardized coordinates.)

As promised, we have applied EMLP to the task of Reynolds stress prediction. The obtained R2R^2 score on the test data is 0.8978±0.00110.8978 \pm 0.0011, which we will add to the revised table. This outperforms regularization, as expected (though only marginally, it seems), but is below the performance of TBNN. The narrative remains the same: for problems exhibiting exact and linear/affine equivariance, existing methods of symmetry enforcement tend to outperform regularization. Regularization exhibits gains over the non-symmetric baseline, and unlike existing methods, can be applied in the nonlinear case. Regularization is also model/architecture agnostic (among differentiable models), allowing existing architectures to be trained with equivariance.

We are eager to address remaining concerns, should there be any, which might prevent the reviewer from fully endorsing our work. Of potential interest are experimental results conducted as requested by the other reviewers (which results will continue to unfold).

Thank you for the thorough replies. The work to update and add experiments is appreciated, and I look forward to seeing more detailed updates in the revision. I have some remaining concerns about the scope of the contribution, which I hope the authors can help clarify


Weakness:

As I understand, the authors scope their work to the following setting. There is an ambient Lie group GG, with a specified joint action (μX,μY)(\mu_X,\mu_Y). A set of candidates is pre-specified, and then a subalgebra of g\mathfrak g is selected using the proposed method.

I agree that selecting some restricted set over which to discover symmetries is natural (and arguably necessary). I think it's worth acknowledging and clarifying the tradeoff between efficiency and flexibility that has to be made here, and how it is reflected in experiments. In particular:

  1. LieGAN/LaLiGan allow the generators themselves to be learned by gradient descent, rather than fixing a set of choices. Of course, learning a matrix is equivalent to learning the coefficients in a linear combination of the basis EijE_{ij} of matrices with a single non-zero entry, one in row ii and column jj. But for SymmetryML this could come at an increased computational cost, relative to the small candidate set used in the current experiments. (As I understand, LieGAN is not restricted to learn linear combinations of the same candidates).
  2. LieNLSD generally has a more flexible search space. The authors seem to have updated experiments here to make the comparison equitable, I didn't see the updated runtimes (https://openreview.net/forum?id=vBEHc2BwYa&noteId=BKrdxyZLQK).

I appreciate the authors' clarification around my concern about the relative "ordering" of XiX_i and YiY_i, which I agree does not apply to the setting described above. My intuition here was getting at a more general point, which I think is also one Reviewer Hzbj is pointing out (https://openreview.net/forum?id=vBEHc2BwYa&noteId=BO0ItiWFUi): the joint action needs to be known for the proposed method to work. Perhaps clarifying this even further in the paper could be helpful.


Question 1:

I am happy to accept that equivariance may be more difficult for "symmetry enforcement" than for discovery. I am, however, still unclear about the work's framing as explicitly improving symmetry discovery (see lines 37-39):

In particular, many methods of symmetry discovery appear limited to the topic of invariance, despite broader claims of applicability to equivariance.

The authors seem to emphasize this in another rebuttal (https://openreview.net/forum?id=vBEHc2BwYa&noteId=9blD9GtLXy):

The comparison to LieNLSD in light of the new experiments is not quite correct, as LieNLSD is restricted to the task of invariance discovery, not equivariance. Indeed, the method of LieNLSD is similar to Shaw et al., centering around the equation.

But (as pointed out in the original review), equivariance in fact is invariance, under the action f↦g∘f∘g−1f \mapsto g \circ f \circ g^{-1}. This suggests to me that existing methods should in principle be extendible to discovering equivariance. Further, LieGAN/LaLiGAN actually do aim at equivariance (e.g. see the former's Theorem 1).

It could be helpful for the authors to clarify the contribution here.

Comments on the paper

2 comments
Area_Chair_zaeMOpenReview ↗

Reminder: Author-reviewer discussion period ending soon

Dear Reviewers,

Thank you again for your time and effort in reviewing this paper. This is a friendly reminder that the author-reviewer discussion period is coming to a close.

If you haven't already, please check out the authors' rebuttal and engage in the discussion by leaving comments. Even a brief acknowledgment confirming whether your questions were addressed would be very helpful for our discussion.

At the close of the discussion period, we wish to thank the reviewers for their thoughtful engagement and feedback, which has resulted in meaningful engagement and the exchange of interesting scientific ideas. Our impression is that the discussion has led to a place where required edits are mutually understood, and that the implementation of these edits would resolve the reviewers' concerns and increase their estimation of the significance of our work. And so, we appreciate any grace that can be accorded on the subject of the timing of implementing the promised edits explicitly.

Additional experiments conducted at the reviewers' requests have further demonstrated the broad applicability and significance of our work. The discussion below has also helped to clarify the contribution of our work relative to existing literature. We believe that our work will be of great interest to the broader scientific community and, in particular, to conference attendees.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.