- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates training-free metrics for evaluating neural network classifiers without training (training-free NAS proxies) based on spectral characteristics of the Neural Tangent Kernel (NTK). The authors show that these characteristics, in particular effective rank and condition number, are not uniquely determined properties of an architecture or of the function it performs. In particular, two functionally equivalent networks can have substantially different values of NTK-based proxy metrics.
The authors focus on the invariance of the softmax function to adding the same shift to all logits and show that such a shift maintains the network outputs, loss, gradients, and training trajectory unchanged, while potentially changing the spectral characteristics of the NTK and the resulting proxy metrics substantially. Thus, NTK-based proxy values can depend on a functionally irrelevant degree of freedom associated with the softmax (a gauge degree of freedom).
The practical consequences of this effect are demonstrated on the full TransNAS-Bench-201 architecture space. The authors show that a logit shift can substantially alter the ranking of architectures according to proxy metrics while preserving their classification accuracy. In particular, targeted manipulation of these metrics can result in the selection of an architecture with substantially worse actual accuracy. The work therefore demonstrates both the theoretical non-identifiability of NTK-based proxies and its potential impact on the NAS procedure.
As a way to eliminate this dependence, the authors consider the Quotient NTK, which removes the contribution of the considered softmax gauge freedom. The idea of class projection that Quotient NTK uses has already been considered in Plenk et al. (2026); the contribution of the given paper is the proof of the non-identifiability of trace-NTK spectral characteristics, the construction and theoretical analysis of a gradient-preserving modification of the neural network's last-layer parameters, and an audit of the full NAS architecture space on benchmark datasets. The authors also propose a simple test for detecting vulnerability of a training-free proxy metrics: if a small function-preserving transformation results in a substantial change in the proxy metric value, this indicates that the metric depends on an unidentifiable gauge degree of freedom.
The main contribution of the paper is the theoretical proof that NTK-based proxy metrics are not intrinsic properties of a neural network architecture or of the function it performs. However, the paper does not demonstrate that the proposed Quotient NTK is a stronger predictor of architectural quality. In particular, the paper does not answer the important practical question of how well the Quotient NTK predicts the actual performance or potential of architecture prior to training. The authors demonstrate that raw NTK-based proxies can produce different, and potentially misleading, rankings under function-preserving gauge transformations, but they do not show that removing the softmax gauge degree of freedom makes the Quotient NTK a more reliable criterion for architecture selection.
Critical strengths
-
The paper challenges a fundamental assumption of training-free NAS that the value of a proxy metric is an intrinsic property of an architecture and can therefore be used to compare architectures prior to training. The authors show that this assumption is, in general, invalid for NTK-based proxies.
-
The paper provides a strong theoretical result: it shows that a function- and training-preserving gauge transformation can introduce a PSD component into the NTK and thereby alter NTK-based proxy metrics without changing the network's predictions, loss, gradients, or training trajectory.
-
A full audit of the TransNAS-Bench-201 search space is conducted, covering all 15,625 architectures. The authors demonstrate that gauge transformations can substantially change the values and rankings of NTK-based proxy metrics, potentially causing the NAS procedure to select architecture with substantially worse final accuracy.
-
The paper shows that sample centering is not a universal solution to the non-identifiability problem of NTK-based proxies. In particular, the authors provide a criterion for when sample centering removes the gauge component.
-
The softmax-free/regression-head experiment provides an important negative control. It shows that the observed effect is specifically related to the redundancy of softmax classification: the analogous modification of a regression head does not preserve the model outputs and does not produce the same proxy instability.
Critical weaknesses
-
The novelty is partially based on a previously known idea. The projection underlying the Quotient NTK was already considered in Plenk et al. (2026). However, the authors do not claim novelty for the projection itself; rather, the main contribution is showing that this projection removes the softmax gauge degree of freedom from the considered NTK-based proxies.
-
The theoretical results are specific to the additive gauge symmetry of softmax classifiers and to particular training dynamics for which the gauge transformation preserves the relevant gradients and parameter updates. In particular, the equivalence of training trajectories does not generally hold for optimization schemes that depend explicitly on parameter values, e.g. those involving weight decay or other parameter-dependent regularization. It is also unclear whether the claimed equivalence extends without modification to architectures involving batch normalization, layer normalization, or other components that may interact with the modified final-layer parameters.
-
The claim that sample centering is “not a repair” seems too categorical. The authors show that sample centering does not remove the gauge component in general and provide an exact criterion for when it does. However, this does not imply that sample centering cannot serve as an effective method for eliminating gauge dependence in general.
-
The statistical robustness of the experimental results is insufficiently investigated. In particular, there is no systematic analysis of sensitivity to random initialization or other sources of experimental variability. The experiments use a single seed, which limits the assessment of the reproducibility of the quantitative results and the robustness of the observed effects.
Questions for authors
The paper does not answer the important practical question of how well the Quotient NTK predicts the actual performance or potential of architecture prior to training. It is also unclear how the proposed gauge-sensitivity test can be used to detect that a benchmark has actually been compromised by an adversary or that a targeted attack is taking place. The proposed test establishes the vulnerability or gauge dependence of the proxy metric, but does not by itself establish that the benchmark has been compromised. The answer requires additional experiments.
This leads to a question that remains unanswered theoretically and experimentally: does the non-invariance of an NTK-based proxy necessarily imply that the proxy is a poor predictor of architecture quality? Non-invariance establishes that the proxy is not an intrinsic property of the predictive function, but it does not by itself establish weak predictive performance. Answering the latter question would require additional experiments comparing raw NTK proxies and Quotient NTK directly in terms of their ability to predict the final performance of trained architectures.
Thus, while the theoretical contribution of the paper is significant, the practical significance of the proposed Quotient NTK for NAS remains insufficiently established.