- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides a PCA-based method to discover symmetries. As opposed to the method it builds upon, it solves the problem without the need for elbow curve heuristics and extends it to equivariance. As opposed to other methods in the literature, it is able to solve symmetry discovery problems effectively and in dramatically less time than the baselines, under the evaluated scenarios.
Critical strengths
- I believe that the motivation that current methods do not explore expressive symmetries and are limited regarding nonlinear equivariance is sharp.
- The writing of the paper is of high quality.
- The method is simple and improves over several baselines on the selected benchmarks.
Critical weaknesses
- The novelty is minimal. The paper builds heavily on the computation of the nontrivial nullspace for discovering symmetries, which is at the core of the method, but this is a construction from another paper as the authors correctly acknowledge throughout. Therefore, the authors' contributions go along the lines of (i) proposing the nullspace-based discovery extension to equivariance, and the enforcement of both invariance and equivariance through regularization (which in my opinion is a fairly direct extension) and (ii) avoiding elbow curve heuristics and resorting to using "least Principal Components" with a threshold for solving the symmetry nullspace computation problem. I believe these contributions to be correct and real, but I am not convinced this is enough to constitute a conference research paper in a top venue, at least by itself. The empirical validation and limitations also do not contribute to compensating for this, as I describe in the next point.
- I believe the practical usefulness of the method is limited. The method performs well in synthetic settings with exact symmetries, and in the turbulence setting where the symmetry is known a priori. Several methods cited in the related work have been applied to more "realistic" scenarios e.g. on image data (e.g. LieGAN and Augerino on rotated MNIST) but the proposed method does not evaluate this or similar scenarios, and I believe it cannot be applied to those (at least in its current formulation). Taking MNIST as an example and following the use of the Euclidean Killing basis used in the paper, this yields a search space of candidate fields for ( translations and rotations , one per pair of axes). The resulting matrix has rows, one per sample and output component of . It is therefore underdetermined for and small . In this setting, the method basically reports hundreds of thousands of "discovered symmetries", independently of the data, since there are many more unknowns than equations in the system. I believe this represents a failure mode of the method. On another note, even for , a matrix of size also requires significant memory. All this, despite MNIST being close to a toy dataset by current standards. These are in my opinion very strong limitations that should be discussed in the paper.
Questions for authors
- Could the authors clarify whether the method can be applied to image data or other datasets with more relevant practical use?
11 comments
Continuing our initial response, we address the reviewer's question.
The subject of Reynolds stress prediction is highly applicable in the field of ML-enhanced computational fluid dynamics. More generally, there is no shortage of applicability when it comes to symmetry enforcement: missing from existing literature is a principled, expressive, and flexible way to enforce a wide variety of symmetries without explicit constraints on model architecture. Nonlinear symmetry enforcement via vector field regularization helps to fill this gap. The turbulence closure case is linear, however one need look no further than the Hamiltonian vector field for insight into applicability for nonlinear symmetry enforcement in the context of physical systems.
On the subject of symmetry discovery, vector field discovery most certainly applies to image data, though not in the way assumed by the reviewer, as discussed in the previous comment. Herein, we avoided application to images in part to avoid potential duplication with potentially-ongoing efforts (see https://openreview.net/forum?id=RconNmjCoW).
But the question itself implies an underappreciation for the broad applicability of vector field symmetry discovery: the method is not limited to discovering symmetry apparent in a particular dataset. In appendix A2, we explicitly compare the task of model symmetry discovery to that of LieNLSD. Model symmetry discovery is a particularly applicable task, allowing one to probe a blackbox (and smooth) model for expected generalization behavior.
We have evaluated symmetry discovery in many elementary settings in order for the ground truth to be well-understood, but also to reveal the prohibitive scalability and performance-related issues present in competing methods. The fact of competing methods taking so much longer while also underperforming when compared with ours even on the simplest of symmetry discovery tasks demonstrates the acute need of more efficient--and practical--methods.
Rebuttal reply
I thank the authors for the detailed clarifications.
Regarding Weakness 2: I understand that the correction is that, for image data, one should not search over the Euclidean Killing bases of the data space , but over the 6-dimensional planar affine algebra acting on the pixel positions in . I agree this is the right formulation to attack the proposed problem, and therefore my previous statement that the method cannot be applied to images is overly strong. I withdraw that statement.
However, I should point out that my application to images follows directly from applying the construction in the paper. Section 4.2.1 defines the vector field bases on the input space of , and the discovery experiments take this to be the Killing basis of the data space itself. For MNIST, the data space is the space of pixel intensities, which indeed gives the case I described in my review. Indeed, one would not choose this way in the case of MNIST, but the formulation in the paper leads to that. I would argue that the most sensible approach is to make such constructions explicit in the paper, as to show how the method applies to such cases which are of broad interest. This change would correctly address the objection of image applicability I made in my review, as well as guide the reader into how or whether the method can be applied in different cases. I would therefore ask the authors to (i) add a formulation to the method section of the paper that contemplates these cases (in which the generators act on the pixel intensities that takes as input, which might need some extra formalizations), (ii) discuss the scaling of the generic basis and corresponding memory, and (iii) empirically validate the image case against other baselines. Method-wise, this would be an accurate description of the intended application of the method on different cases. Experimentally, a reassuring result would for example discover a rotation generator on rotated MNIST, as e.g. LieGAN does, and would put this method in equal grounds to several other symmetry discovery methods, which are validated on images and therefore show stronger evidence of their applicability.
Regarding the broad applicability claim through the lens of model symmetry discovery made by the authors in the rebuttal: I understand this is the case, but the evidence presented in the paper does not show that this method is strong for this particular application, neither the "probing a model for expected generalization behaviour" claim made in the rebuttal is supported in the experiments of the paper. In effect, Appendix A.2 shows (and acknowledges) that the method does not show a clear accuracy advantage over the baseline LieNLSD. It is indeed faster though, about 3x times, but I am not convinced this is such a good tradeoff. After all, discovery is a one-off analysis, so in practice solving it in 77 seconds vs 232 seconds as for the case does not change a lot. Speed is interesting as a proxy for scaling, but I understand that both methods here scale linearly in , so in conclusion, I am not convinced that the applicability to the model symmetry discovery is a strong point of the proposed method. In addition, these are both toy experiments on and with known symmetries. It is no closer to practical use than the surfaces of revolution.
However, the "probe a model for expected generalization" experiment would be very interesting, and I believe this angle would strengthen the paper a lot, but as I pointed out, it is asserted on the rebuttal response but not supported by the evidence in this paper, therefore all we can claim is that in principle, it could be an application of this method. Overall, I am pushing back a bit on the authors' argument for applicability through the lens of model symmetry discovery. Additionally, in that regard, model symmetry discovery is also a capability available to other methods (such as the LieNLSD and apparently, with a slight accuracy edge) or such as LieGG [1] (which is not discussed in the related work section).
We thank the reviewer for the time given to review our work. As we wish to address the reviewer's concerns thoughtfully, our response will span two comments, the first addressing the perceived weaknesses and the second addressing the question.
Weakness 1: We appreciate the acknowledgement of the novelty of our approach and wish to address concerns about the "sufficiency" of the novelty. Primarily, we are aware of no other method which has successfully performed nonlinear equivariance discovery and enforcement, so that our method is filling a cruicial gap in the literature. We also believe there is some novelty present in connection with section 4.3.1. Our method can also be used for nonlinear invariance discovery and enforcement, as well as linear equivariance enforcement, as can many existing methods, perhaps appearing to duplicate previous work. But crucially, we have empirically shown that our method consistently outperforms a representative sample of existing methods, most especially in terms of computation time, with our method commonly being multiple orders of magnitude faster when compared with other methods. Superiority over state of the art methods should be taken into account when assessing the "goodness of fit" for top-tier publication venues.
Weakness 2: We wish to address the issue of practical utility while kindly offering corrections to some assertions and assumptions made by the reviewer. The perceived scaling problem with applying our method to MNIST appears to begin with the assertion that the group action for our method is different than that for existing methods such as LieGAN: where LieGAN need only consider group actions in the 2-dimensional plane, our method is said to need to consider the affine group acting on R784. In practicality, the task of symmetry discovery for image data would begin with the careful consideration of the precise group action on the images, and the group itself--in particular, the dimension thereof--would be no different for our method than that of existing methods: a 6-parameter symmetry discovery task for LieGAN is a 6-parameter, not 307,720-parameter, task for our method.
The confusion appears to lie in the assumption that the affine action on images is free, uniquely when applying our method, to translate a single pixel but not all others simultaneously. Indeed, the transformation induced by the proposed generator of ∂xj would translate only the pixel associated with dimension j. The simultaneous action of the affine group on each pixel can be captured using the concept of diagonal group actions, and the construction of infinitesimal generators for diagonal group actions is given explicitly in appendix C3. The six generators for the diagonal action of the affine group on 784 copies of R2 in coordinates (xi,yi) are as follows:
X1=∑i=1784∂xi,X2=∑i=1784∂yi,X3=∑i=1784−yi∂xi+xi∂yi, X4=∑i=1784xi∂xi+yi∂yi,X5=∑i=1784yi∂xi+xi∂yi,X6=∑i=1784xi∂xi−yi∂yi.
Moreover, the task of symmetry enforcement requires advance knowledge of the symmetries, be they pre-discovered or else prescribed by the nature of the problem. Thus, the applicability of the turbulence example is not diminished in the slightest by the fact that the symmetry group is specified in advance: it is a consequence of the Navier-Stokes equations.