-
How are the symmetry transformations sampled? For positive rescaling, what distribution over log-scales α is used within the ball ∥α∥2≤R, for both training augmentation and random evaluation? For the ViT experiment, how are orthogonal and GL matrices with a prescribed condition number generated?
-
What do the optimized (attacked) rescalings look like? For example, do they concentrate on a few hidden units or a particular layer? Since the augmentation baseline is trained only on random rescalings, have the authors tried adversarial augmentation during training, which would be a fairer test of whether invariance can be learned?
-
The paper focuses on positive scaling and GL symmetries but compares against no existing method that handles them, although several are cited: ScaleGMN [1] and Monomial-NFN [2] for scaling, and Transformer-NFN [3] and GL-equivariant LoRA processing [4] for GL symmetries. Why were these excluded?
-
Why do the authors only use sign + log|w| to populate the graph edges? how does the raw baseline behave when given the weights directly, as in standard GMN [7] and neural-graph encoders?
-
Standard weight-space baselines are missing. For the MLP prediction task, DWS [5], NFN [6], a standard GMN [7] on raw weights, and the statistics-based StatNN [8] would situate the results within the weight-space literature. Could the authors include them, or explain why not?
-
In addition to the above, if the authors wish to keep only the invariant methods, they should try canonicalizing the input NNs and then using a metamodel like GMN [7] for the processing. To my understading what they currently use is the log-magnitudes of the canonicalized weights.
-
To my understanding, the head-matching task recovers a within-layer permutation between exactly equivalent copies, where the invariant costs are near zero for the correct pair, so success follows directly from Theorem 5.1. How does the operator representation perform in harder, practically relevant settings, e.g. matching heads across independently trained models?
[1] Kalogeropoulos, Ioannis, Giorgos Bouritsas, and Yannis Panagakis. "Scale equivariant graph metanetworks." Advances in neural information processing systems 37 (2024): 106800-106840.
[2] Tran, Viet-Hoang, et al. "Monomial matrix group equivariant neural functional networks." Advances in Neural Information Processing Systems 37 (2024): 48628-48665.
[3] Tran, Viet-Hoang, et al. "Equivariant Neural Functional Networks for Transformers." International Conference on Learning Representations. Vol. 2025.
[4] Putterman, Theo, et al. "GL Equivariant Metanetworks for Learning on Low Rank Weight Spaces." The Fourth Learning on Graphs Conference. 2025.
[5] Navon, Aviv, et al. "Equivariant architectures for learning in deep weight spaces." International Conference on Machine Learning. PMLR, 2023.
[6] Zhou, Allan, et al. "Permutation equivariant neural functionals." Advances in neural information processing systems 36 (2023): 24966-24992.
[7] Lim, Derek, et al. "Graph metanetworks for processing diverse neural architectures." International Conference on Learning Representations, 2024.
[8] Unterthiner, Thomas, et al. "Predicting neural network accuracy from weights." arXiv preprint arXiv:2002.11448 (2020).
We thank the reviewer for the review. All changes in the revised PDF are marked in blue; the new experiments are in Appendix H.
W1 (readability and clarity of the contributions). Sections 2 to 7 were revised for clarity. Section 3 defines every object used later where it first appears, and paragraphs open with a sentence stating their point. Section 4.1 and Table 2 in Appendix A separate what was known from what each theorem adds, and Section 4.2 ends with the procedure that follows from the zero-weight theorem. In plain terms, the paper contributes a graph representation that does not depend on the parameterization (Section 3); for rescaling, a one-solve procedure that computes the unique balanced representative and provably loses no information about the orbit (Section 4); for changes of basis, operator attributes that stay complete when ranks drop (Section 5); a bound relating a prediction's variation along an orbit to its error (Section 6); and experiments showing that common analyses change their answers under exact reparameterization while the invariant versions do not (Section 7).
W2 and Q1 (practical results and comparison with ScaleGMN and Monomial-NFN). Table 1 and Appendix H compare against released methods on three collections. ScaleGMN is a learned predictor that is invariant by construction; our method is a preprocessing step that makes an existing analysis invariant to positive rescaling without changing it, so the comparison that tests it is raw versus canonicalized for each method. On the MLPs, StatNN's RMSE rises from 0.0285 to 0.4222 under random rescaling and to 0.0528 under a deterministic equalization shift, while Can→StatNN stays at 0.0317 and ScaleGMN at 0.0260. On CIFAR-10, GMN rises from 0.0152 to 0.0806, Can→GMN stays at 0.0798, and ScaleGMN has the lowest error at 0.0129. Canonicalization therefore does not improve on ScaleGMN's accuracy; it gives exact invariance to methods that have none, including hand-designed descriptors (Appendix H.6) and classification pipelines (Appendix H.7), at the cost of one sparse solve. Monomial-NFN was not run; it would be a second exactly invariant baseline for the same symmetries. The experiments test the consequences the theory predicts for existing analyses.
Q2 (an optimization algorithm in the spirit of G-SGD). The fixed-support results give such an algorithm directly. On a fixed support and sign pattern, write w = σ ⊙ e^y. The rescaling orbit through y is the affine space y + im B_Sᵀ, and an invariant loss depends on y only through Π_S y (Theorem 4.1), so its gradient with respect to y, which equals w ⊙ ∇_w L, lies in ker B_S, and the update y ← y − η ∇_y L maps equivalent iterates to equivalent iterates. Gradient descent in w lacks this property, which G-SGD addresses by moving in basis-path coordinates and mapping back. Applying Π_S after each step keeps the iterate canonical and makes the trajectory, including weight decay and optimizer state, depend only on the orbit of the initial point. These coordinates exist only on a fixed support (Theorem 4.4); a support change comes from an explicit pruning step, after which the projector is rebuilt. We have not implemented or benchmarked this optimizer.