acceptodds
Under review as a conference paper at ICLR 2027

Invariant Graph Properties for Neural Computation Graphs

Abstract

Representing neural networks as graphs allows their architecture and learned structure to be studied using tools from network science, topological data analysis, discrete geometry, and graph representation learning. Yet the graph associated with a trained network is not uniquely determined by the function it computes: exact parameter symmetries can alter edge weights and graph structure without changing the network’s predictions. We study invariance under common symmetries, including hidden permutations, positive rescaling, and general linear changes of basis. For positive rescaling, we identify minimal coordinates and a canonical representative on each fixed support, and show why zero weights preclude a single continuous exact representation across supports. For general linear symmetries, we derive operator representations that remain complete when ranks drop, including an operator graph for multi-head attention. We then connect this symmetry dependence to the reliability of predictions and interventions. Experiments on collections of MLP and ViT checkpoints show that raw graph analyses can differ substantially across equivalent parameterizations, whereas symmetry-aware methods remain stable and preserve predictions and transported interventions. The appropriate graph representation therefore depends on both the parameter symmetries and the target of the analysis.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Computation graphs, symmetry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3PLeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the graph associated with a trained network. They work in particular on the symmetries of such networks including hidden permutations, positive rescaling, and general linear changes of basis. The main contribution of the paper is to formalize invariance, equivariance on these graphs. In particular, on for positive rescaling, they identify the problems introduced by zero weights, and they study several widely used norms (LayerNorm, RMSNorm, and BatchNorm).

Overall i think that this paper is not ready for publication. The theoretical results need to be verified experimentally and the paper should be rewrote to be more understandable. This paper did not convince me of its usefulness.

Critical strengths

The different modules that this paper analyzes are widely used and present in nearly all architectures. The symmetries that the paper work on can are the building blocks of Neural Networks.

Critical weaknesses

1- This paper is very hard to follow and it is difficult to understand what the authors contributions are.

2- The main weakness is the lack of experimental proofs showing how their results can be used in practice. They do not compare their approach to any method in the litterature.

Questions for authors

1- In the paper you cite ScaleGMN and Monomial-NFN, could you add experimental results showing how the theoretical results that you proved improve on their method.

2- Could the results that you have on positive rescaling be adapted to produce an algorithm such as G-SGD (Meng et al, 2019) ?

1 comment

We thank the reviewer for the review. All changes in the revised PDF are marked in blue; the new experiments are in Appendix H.

W1 (readability and clarity of the contributions). Sections 2 to 7 were revised for clarity. Section 3 defines every object used later where it first appears, and paragraphs open with a sentence stating their point. Section 4.1 and Table 2 in Appendix A separate what was known from what each theorem adds, and Section 4.2 ends with the procedure that follows from the zero-weight theorem. In plain terms, the paper contributes a graph representation that does not depend on the parameterization (Section 3); for rescaling, a one-solve procedure that computes the unique balanced representative and provably loses no information about the orbit (Section 4); for changes of basis, operator attributes that stay complete when ranks drop (Section 5); a bound relating a prediction's variation along an orbit to its error (Section 6); and experiments showing that common analyses change their answers under exact reparameterization while the invariant versions do not (Section 7).

W2 and Q1 (practical results and comparison with ScaleGMN and Monomial-NFN). Table 1 and Appendix H compare against released methods on three collections. ScaleGMN is a learned predictor that is invariant by construction; our method is a preprocessing step that makes an existing analysis invariant to positive rescaling without changing it, so the comparison that tests it is raw versus canonicalized for each method. On the MLPs, StatNN's RMSE rises from 0.0285 to 0.4222 under random rescaling and to 0.0528 under a deterministic equalization shift, while Can→StatNN stays at 0.0317 and ScaleGMN at 0.0260. On CIFAR-10, GMN rises from 0.0152 to 0.0806, Can→GMN stays at 0.0798, and ScaleGMN has the lowest error at 0.0129. Canonicalization therefore does not improve on ScaleGMN's accuracy; it gives exact invariance to methods that have none, including hand-designed descriptors (Appendix H.6) and classification pipelines (Appendix H.7), at the cost of one sparse solve. Monomial-NFN was not run; it would be a second exactly invariant baseline for the same symmetries. The experiments test the consequences the theory predicts for existing analyses.

Q2 (an optimization algorithm in the spirit of G-SGD). The fixed-support results give such an algorithm directly. On a fixed support and sign pattern, write w = σ ⊙ e^y. The rescaling orbit through y is the affine space y + im B_Sᵀ, and an invariant loss depends on y only through Π_S y (Theorem 4.1), so its gradient with respect to y, which equals w ⊙ ∇_w L, lies in ker B_S, and the update y ← y − η ∇_y L maps equivalent iterates to equivalent iterates. Gradient descent in w lacks this property, which G-SGD addresses by moving in basis-path coordinates and mapping back. Applying Π_S after each step keeps the iterate canonical and makes the trajectory, including weight decay and optimizer state, depend only on the orbit of the initial point. These coordinates exist only on a fixed support (Theorem 4.4); a support change comes from an explicit pruning step, after which the projector is rebuilt. We have not implemented or benchmarked this optimizer.

Reviewer_1r64OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies neural computation graphs, i.e. the computatational graphs of neural networks, and the symmetries to which they are invariant. The authors study three common symmetries, namely neuron permutations, positive rescaling, and linear change of basis. They formalize a canonical representation for positive rescaling and derive operator representations for general linear symmetries. Experiments on toy datasets demonstrate the equivalence shift on non-symmetry-aware methods.

Critical strengths

The field of neural network representations and neural computational graphs is a recent and exciting sub-field in machine learning, that lacks thorough theoretical underpinnings. Thus, this work offers a needed theoretical contribution and a useful tool for network analysis through equivalence shift tests.

Critical weaknesses

The experimental section is fairly limited to small datasets. It is thus, unclear, if the results hold in settings with large neural network datasets.

Questions for authors
  • Is the output function h always a regressor? Do the results hold if the output is classification, or if the output is dense prediction?
  • Would the results (Figure 4, 5) hold in bigger data regimes, e.g. assuming thousands or millions of checkpoints? Is there an estimate on how this scales with dataset size?
2 comments

We thank the reviewer for the review. All changes in the revised PDF are marked in blue; the new experiments are in Appendix H.

W1 and Q2 (size of the collections and scaling with dataset size). Appendix H.5 adds the public SmallCNN collections of Unterthiner et al. (2020): 15,126 CIFAR-10 and 15,122 SVHN networks, with 7,567 and 7,505 held out. The conclusions hold there: on CIFAR-10, raw GMN's RMSE rises from 0.0152 to 0.0806 under random rescaling and to 0.3669 under an error-seeking search, while Can→GMN, Can→StatNN, and ScaleGMN do not change (Tables 12 and 13). Appendix H.4 refits the MLP predictors on 8, 16, 32, and 48 training runs; ScaleGMN improves from 0.0366 to 0.0260 and Can→GMN from 0.0441 to 0.0398. Canonicalization has no learned parameters and costs one sparse Laplacian solve per checkpoint, 0.4 to 0.5 ms on these collections (Appendix H.9) and 0.03 s for an MLP with 2.2 million rescaling-affected edges (Table 8), so preprocessing scales linearly in the number of checkpoints. The transformer experiments of Figure 5 still use 12 held-out models, and we have no evidence at the scale of millions of checkpoints.

Q1 (classification and dense outputs). The output function is not restricted to regression. Proposition 6.1 is stated for an arbitrary metric output space, and Section 6 now says that h may be a regressor or a classifier with scalar, probability, or dense outputs. Appendix H.7 adds a classification experiment in which checkpoints are labeled by whether their test loss is below the training median and decision thresholds are fixed before any transformation. Raw StatNN's AUROC drops from 0.9101 to 0.5671 under random exact shifts and its Brier score rises from 0.1233 to 0.5125, while Can→StatNN keeps AUROC 0.9014 and ScaleGMN 0.8654 (Table 15). Dense prediction is covered by the proposition with the metric on the output grid; we have not run a dense-prediction experiment.

I would like to thank the authors for their responses.

If I may follow up, my questions on large neural network datasets (with millions of checkpoints) is on whether the raw or augment methods from figures 4 and 5 are expected to reach a performance similar to the canonical method, or whether the results would hold in very big data regimes.

Overall, I am satisfied with the responses and I maintain my score.

Reviewer_6CznOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper studies how parameter symmetries such as hidden permutations, positive rescaling, and general linear changes of basis make graph representations of neural networks depend on the parameterization rather than on the computed function. It derives complete invariant representations for these symmetries, characterizes their limits at zero weights and rank loss, and shows experimentally that raw graph-based predictions and interventions break under exact function-preserving transformations while the proposed invariant representations remain stable.

Critical strengths
  1. Rigorous theory that also clarifies the limits. The paper characterizes complete, minimal invariant representations for rescaling and general linear symmetries, including the rank-deficient case.

  2. The canonicalization is a cheap preprocessing step that makes any downstream analysis invariant, including hand-crafted descriptors where equivariant architectures can’t be used.

Critical weaknesses
  1. The positioning of the contribution is unclear. The paper combines a theoretical characterization of symmetry quotients, a critique of existing graph descriptors, and new representations, but it does not clearly argue why invariant representations are preferable to the equivariant weight-space models that already address these symmetries.

  2. The symmetries studied (permutation, positive/sign scaling, GL) have been addressed by prior weight-space models, e.g. ScaleGMN [1], Monomial-NFN [2], Transformer-NFN [3] and GL-equivariant LoRA processing [4]. The paper positions canonicalization as complementary to these, and its novelty lies mainly in completeness/minimality results and the zero-weight analysis. The practical benefit of these properties over existing equivariant models is not demonstrated.

  3. The experimental results are largely expected. The proposed representations are exactly invariant by construction, so their zero degradation follows from the theorems. Conversely, a model without built-in scale invariance (MLP experiment) or a coordinate-based matching rule (ViT experiment) is expected to fail under symmetry transformations, particularly with collections of this size (80 MLP runs, 40 ViT runs), where learning the invariance from data or augmentation is difficult. The experiments therefore, confirm that the problem exists, but provide little evidence beyond what the theory already guarantees.

  4. The headline MLP results rely on rescalings optimized at test time to maximize each predictor’s error, while the augmentation baseline is trained only on random rescalings. This mismatch makes the augmentation result unsurprising. The main text should emphasize the random-rescaling results, which show a much smaller degradation or compare against adversarially trained augmentation.

  5. The experiments omit established baselines. No standard GMN [7] or neural-graph encoder on raw weights is evaluated: the MLP “raw” baseline uses sign and log-magnitude features, which is not motivated and hinders comparison with the weight-space literature. More importantly, symmetry-aware models such as ScaleGMN [1] (MLPs) and Transformer-NFN [2] (attention) are absent. These would also be exactly invariant to the tested transformations, so the meaningful comparison would be accuracy on untransformed models, discriminative power, and cost.

  6. The experimental scale is limited: each experiment uses a single architecture, a single dataset, and a few independent runs (80 MLP runs, 40 ViT runs), far smaller than standard model zoos. Have the authors concluded that their methods need only such a small dataset?

[1] Kalogeropoulos, Ioannis, Giorgos Bouritsas, and Yannis Panagakis. "Scale equivariant graph metanetworks." Advances in neural information processing systems 37 (2024): 106800-106840.

[2] Tran, Viet-Hoang, et al. "Monomial matrix group equivariant neural functional networks." Advances in Neural Information Processing Systems 37 (2024): 48628-48665.

[3] Tran, Viet-Hoang, et al. "Equivariant Neural Functional Networks for Transformers." International Conference on Learning Representations. Vol. 2025.

[4] Putterman, Theo, et al. "GL Equivariant Metanetworks for Learning on Low Rank Weight Spaces." The Fourth Learning on Graphs Conference. 2025.

[5] Lim, Derek, et al. "Graph metanetworks for processing diverse neural architectures." International Conference on Learning Representations, 2024.

Questions for authors
  1. How are the symmetry transformations sampled? For positive rescaling, what distribution over log-scales α\alpha is used within the ball ∥α∥2≤R\|\alpha\|_2 ≤ R, for both training augmentation and random evaluation? For the ViT experiment, how are orthogonal and GL matrices with a prescribed condition number generated?

  2. What do the optimized (attacked) rescalings look like? For example, do they concentrate on a few hidden units or a particular layer? Since the augmentation baseline is trained only on random rescalings, have the authors tried adversarial augmentation during training, which would be a fairer test of whether invariance can be learned?

  3. The paper focuses on positive scaling and GL symmetries but compares against no existing method that handles them, although several are cited: ScaleGMN [1] and Monomial-NFN [2] for scaling, and Transformer-NFN [3] and GL-equivariant LoRA processing [4] for GL symmetries. Why were these excluded?

  4. Why do the authors only use sign + log|w| to populate the graph edges? how does the raw baseline behave when given the weights directly, as in standard GMN [7] and neural-graph encoders?

  5. Standard weight-space baselines are missing. For the MLP prediction task, DWS [5], NFN [6], a standard GMN [7] on raw weights, and the statistics-based StatNN [8] would situate the results within the weight-space literature. Could the authors include them, or explain why not?

  6. In addition to the above, if the authors wish to keep only the invariant methods, they should try canonicalizing the input NNs and then using a metamodel like GMN [7] for the processing. To my understading what they currently use is the log-magnitudes of the canonicalized weights.

  7. To my understanding, the head-matching task recovers a within-layer permutation between exactly equivalent copies, where the invariant costs are near zero for the correct pair, so success follows directly from Theorem 5.1. How does the operator representation perform in harder, practically relevant settings, e.g. matching heads across independently trained models?

[1] Kalogeropoulos, Ioannis, Giorgos Bouritsas, and Yannis Panagakis. "Scale equivariant graph metanetworks." Advances in neural information processing systems 37 (2024): 106800-106840.

[2] Tran, Viet-Hoang, et al. "Monomial matrix group equivariant neural functional networks." Advances in Neural Information Processing Systems 37 (2024): 48628-48665.

[3] Tran, Viet-Hoang, et al. "Equivariant Neural Functional Networks for Transformers." International Conference on Learning Representations. Vol. 2025.

[4] Putterman, Theo, et al. "GL Equivariant Metanetworks for Learning on Low Rank Weight Spaces." The Fourth Learning on Graphs Conference. 2025.

[5] Navon, Aviv, et al. "Equivariant architectures for learning in deep weight spaces." International Conference on Machine Learning. PMLR, 2023.

[6] Zhou, Allan, et al. "Permutation equivariant neural functionals." Advances in neural information processing systems 36 (2023): 24966-24992.

[7] Lim, Derek, et al. "Graph metanetworks for processing diverse neural architectures." International Conference on Learning Representations, 2024.

[8] Unterthiner, Thomas, et al. "Predicting neural network accuracy from weights." arXiv preprint arXiv:2002.11448 (2020).

3 comments

Part 1: weaknesses

We thank the reviewer for the detailed review. All changes in the revised PDF are marked in blue; the new experiments are in Appendix H.

W1 (positioning relative to equivariant weight-space models). An equivariant weight-space model is one learned architecture with its symmetry fixed at design time. Canonical preprocessing is applied to the input and makes any downstream analysis invariant to positive rescaling without changing it, including hand-designed descriptors and released encoders; Section 2 now states this distinction. The experiments show what this buys: StatNN's RMSE rises from 0.0285 to 0.4222 under random rescaling while Can→StatNN stays at 0.0317 (Table 10), and neural persistence varies along the orbit by 9.1 times its spread across networks while its canonical version does not (Table 14). We do not claim higher accuracy than ScaleGMN, which has the lowest original-coordinate error on CIFAR-10 and the lowest searched error on both CNN collections.

W2 (novelty; practical value of completeness, minimality, and the zero-weight analysis). Table 2 in Appendix A separates, for each theorem, the established ingredient from the statement proved here. The new statements are the explicit quotient, projector, and one-solve canonical representative on an arbitrary support, the minimal number of continuous coordinates, the zero-weight obstruction, and all-rank completeness of the operator representations. These results guarantee properties rather than improve accuracy. Completeness means that canonicalization discards nothing about the orbit, so accuracy is limited only by the predictor (Can→StatNN reaches 0.0317 against 0.0285 for StatNN on original MLP checkpoints); minimality gives the number of coordinates an invariant representation needs (27,466 for the MLPs of Section 7.1, against more than 2.5 × 10^8 boundary paths); and the zero-weight result is why the thresholding experiment canonicalizes before and after thresholding, which makes the selected support independent of the representative (Appendix G.4).

W3 (the results are expected). Exact invariance of the canonical pipelines follows from Theorem 4.1; three results do not. First, the experiments measure how far raw and augmented predictors are from invariance, and the removed-scale control shows that the information rescaling alters is itself predictive of loss: a predictor that sees only the component removed by canonicalization reaches R² = 0.754. Second, invariance says nothing about accuracy, which the experiments measure and which differs by collection. Third, Proposition 6.2 does not say how much a non-equivariant selector changes an intervention; the independent fine-tune study measures a transferred effect of 12.2 accuracy points through raw matching against 46.3 through operator matching. The same degradation appears with thousands of held-out CNNs (Tables 12 and 13), so it is not an artifact of small collections.

W4 (optimized attack versus random augmentation). Section 7.1 now reports random rescaling before the search, and Appendix H.2 adds GMN trained with adversarial augmentation on the same radius-two box used for evaluation. On CIFAR-10 the searched RMSE is 0.3669 for raw GMN, 0.0968 with random augmentation, and 0.0440 with adversarial augmentation; on SVHN it is 0.7386, 0.1214, and 0.0576. Adversarial training removes most of the searched degradation at five to ten times the fit time of unaugmented GMN, and its remaining error is three to four times that of ScaleGMN.

W5 (baselines; sign and log-magnitude features). Table 1 and Appendix H now include GMN on raw signed weights, StatNN, ScaleGMN, Can→GMN, and Can→StatNN on the MLP collection and both public CNN collections; Q3 addresses Transformer-NFN. For cost, Can→StatNN reaches RMSE 0.0317 on the MLPs with 23,937 parameters and 1.95 ms per checkpoint, against 0.0260 with 392,129 parameters and 3.96 ms for ScaleGMN (Appendix H.9). The sign and log-magnitude features are the coordinates in which positive rescaling acts additively (Equation 1), so raw and canonical inputs differ only by the projection Π_S and the submitted comparison varies only the symmetry.

W6 (scale of the collections). The public collections contain 15,126 CIFAR-10 and 15,122 SVHN networks, with 7,567 and 7,505 held out, and the conclusions hold there (Tables 12 and 13). Canonicalization has no learned parameters and needs no training data; the number of runs matters only for the downstream predictor, and Appendix H.4 reports each predictor's error from 8 to 48 training runs.

Part 2: questions

Q1 (sampling distributions). In the submitted MLP experiment, each of the 96 hidden log scales is drawn independently from Uniform[−R, R] and the search clips each coordinate to [−R, R], so the set is the box ‖α‖∞ ≤ R; the revised Section 7.1 states this. Augmentation drew training scales at R = 1, and the search used 60 projected Adam steps from one start at the identity and three random starts. The new experiments use the box with R = 2 for evaluation, augmentation, and searches; Table 9 adds a control on the sphere ‖α‖₂ = 2, under which StatNN still changes, from 0.0285 to 0.0341. In the submitted ViT experiment, orthogonal matrices are Haar-distributed (QR of a Gaussian matrix with diagonal sign correction), and for condition number κ the change of basis is G = U diag(s) Vᵀ with U and V independent orthogonal and log-singular values equally spaced from −½ log κ to ½ log κ in random order, so the condition number is exactly κ. Each condition also applies an independent head permutation per layer. Appendix H.8 instead draws log-singular values uniformly from [−1, 1], so its condition numbers are at most e².

Q2 (structure of the optimized rescalings; adversarial augmentation). Averaged over checkpoints, runs, and fits, the three hidden layers receive 33.6%, 35.2%, and 31.2% of the squared log scale for the sign/log encoder and 37.0%, 41.6%, and 21.4% for StatNN, so the searched rescalings are spread across layers on average; individual attacks differ in which units they emphasize. Adversarial augmentation is now included (W4).

Q3 (ScaleGMN, Monomial-NFN, Transformer-NFN, and GL-equivariant LoRA processing). ScaleGMN is now in every prediction experiment. Monomial-NFN would be a second exactly invariant baseline for the symmetries ScaleGMN already covers; Transformer-NFN and the GL-equivariant LoRA model output predictions rather than head assignments, whereas our transformer experiment is a transport task, so none of the three was run. We will add any of them that the reviewer considers informative.

Q4 (sign and log-magnitude features; raw weights as input). The input choice is explained under W5. Raw weights are the GMN rows: on the MLPs GMN predicts a near-constant value with or without augmentation (RMSE 0.0685, the error of the training-mean predictor), so its MLP rows are uninformative; on CIFAR-10 it fits well (0.0152) and degrades to 0.0806 under random shifts and 0.3669 under the search.

Q5 (DWS, NFN, GMN, and StatNN). GMN and StatNN are included on all three collections; DWS and NFN are not, because we prioritized the released graph-based predictors, which read the same computation graph as our encoder, and StatNN. Accuracy, discriminative power, and cost are reported for every predictor we ran (Table 1, Appendices H.7 and H.9).

Q6 (canonicalization followed by GMN). This pipeline is now Can→GMN. It is consistent in every condition, with RMSE 0.0398 on the MLPs, where it fits although raw GMN does not, 0.0798 on CIFAR-10, and 0.0669 on SVHN, so on the CNN collections it is less accurate than raw GMN and ScaleGMN. The submitted canonical encoder is a different pipeline, our own message-passing encoder on canonical log magnitudes and signs (0.0256 on the MLPs under the new protocol); the reviewer's reading of it is correct.

Q7 (matching across independently trained models). Appendix H.8 matches heads between different fine-tuned models: all 66 pairs of the 12 held-out ViT-B/16 runs, which share a pretrained initialization and compute different functions. A matching is scored by the held-out disagreement between matched heads' outputs and by transferring the most damaging head of each source layer through the matching and removing the 12 heads jointly from the target, so a larger accuracy drop means a more successful transfer. In native coordinates, raw, operator, Procrustes, and head-index matching tie at cost 0.199. After independent invertible changes of the query–key and value–output coordinates, raw matching rises to 0.784 and its transferred ablation lowers accuracy by 12.2 points instead of 46.3, while operator and Procrustes matching keep the original assignment and the full effect. A learned SpinQuant-style value–output rotation has the same effect on raw value–output matching, with cost 0.711 and a drop of 15.5 points.

Soundness rating. We would welcome a pointer to the theorem or proof step behind the soundness rating and will address it directly; the proofs are in Appendices B to D.

I thank the authors for the detailed rebuttal and the additional experiments. I acknowledge the paper’s theoretical contribution: the explicit quotient, projector, and one-solve canonical representative for positive rescaling on an arbitrary support, the minimal number of continuous coordinates, the zero-weight obstruction, and the all-rank completeness of the operator representations. However, canonicalizing the input before processing it is not a new idea. The new results are also expected: a model that is equivariant to the full symmetry group (e.g., permutations and scaling) outperforms canonicalizing with respect to the additional symmetry and then applying the corresponding model that accounts only for the remaining symmetries (e.g., permutations alone). The contribution, therefore, lies solely in how the canonicalization is achieved, and its practical value over existing equivariant weight-space models is still not demonstrated, so I keep my rating. Since the rebuttal addresses my concerns about baselines, adversarial augmentation, and the scale of the experiments, I have increased my soundness score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.