acceptodds
Under review as a conference paper at ICLR 2027

TANGENT KERNELS MEASURE THE GAUGE: NON-IDENTIFIABILITY AND FULL-SPACE AUDITS OF TRAINING-FREE NAS PROXIES

Abstract

Architectures are increasingly compared without training them, by reading spectral summaries (effective rank, condition number, kernel–target alignment) off an empirical tangent kernel as architectural properties. For softmax classifiers they are not. We prove that a function- and training-preserving softmax gauge injects arbitrary PSD contamination into the trace NTK while leaving the softmax, cross-entropy loss, gradient and training trajectory untouched; its same-architecture form leaves every gradient unchanged. The class-quotient NTK, the class-projected kernel used to analyse cross-entropy dynamics, is exactly invariant under both forms; sample centering removes the injected block only when its tangent features are constant across the probe set. Auditing whole search spaces (all NAS-Bench-201 cells and the -cell TransNAS-Bench-101 micro space) shows the consequence is practical. Editing one weight matrix leaves the spoofed top-10 shortlist holding 2 of its 10 architectures, and an adversary choosing the gauge strength per candidate forces the selection of a network 34.6 accuracy points worse than the proxy's choice, while predictions, loss and gradients are preserved to in float32. On the quotient kernel the same sweep leaves every shortlist and the selected architecture fixed, with drift below . The effect persists on a -parameter vision transformer and across graph architectures, where sample centering amplifies rather than repairs drift; a pre-specified control on softmax-free heads bounds the claim. Consequently, a tangent-kernel number reported for a softmax model is a claim about a gauge choice unless it is quotiented, so proxy comparisons built on one are not falsifiable as stated. The repair is cheap (one extra evaluation of the diagnostic under a random function-preserving gauge, released as an audit) but precision-critical: default reduced-precision GPU matmuls alone make an exact invariance look violated.

Primary area:
datasets and benchmarks
Keywords:
neural tangent kernel, neural architecture search, zero-cost proxies, reparameterization invariance, benchmark integrity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VDaAOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates training-free metrics for evaluating neural network classifiers without training (training-free NAS proxies) based on spectral characteristics of the Neural Tangent Kernel (NTK). The authors show that these characteristics, in particular effective rank and condition number, are not uniquely determined properties of an architecture or of the function it performs. In particular, two functionally equivalent networks can have substantially different values of NTK-based proxy metrics.

The authors focus on the invariance of the softmax function to adding the same shift to all logits and show that such a shift maintains the network outputs, loss, gradients, and training trajectory unchanged, while potentially changing the spectral characteristics of the NTK and the resulting proxy metrics substantially. Thus, NTK-based proxy values can depend on a functionally irrelevant degree of freedom associated with the softmax (a gauge degree of freedom).

The practical consequences of this effect are demonstrated on the full TransNAS-Bench-201 architecture space. The authors show that a logit shift can substantially alter the ranking of architectures according to proxy metrics while preserving their classification accuracy. In particular, targeted manipulation of these metrics can result in the selection of an architecture with substantially worse actual accuracy. The work therefore demonstrates both the theoretical non-identifiability of NTK-based proxies and its potential impact on the NAS procedure.

As a way to eliminate this dependence, the authors consider the Quotient NTK, which removes the contribution of the considered softmax gauge freedom. The idea of class projection that Quotient NTK uses has already been considered in Plenk et al. (2026); the contribution of the given paper is the proof of the non-identifiability of trace-NTK spectral characteristics, the construction and theoretical analysis of a gradient-preserving modification of the neural network's last-layer parameters, and an audit of the full NAS architecture space on benchmark datasets. The authors also propose a simple test for detecting vulnerability of a training-free proxy metrics: if a small function-preserving transformation results in a substantial change in the proxy metric value, this indicates that the metric depends on an unidentifiable gauge degree of freedom.

The main contribution of the paper is the theoretical proof that NTK-based proxy metrics are not intrinsic properties of a neural network architecture or of the function it performs. However, the paper does not demonstrate that the proposed Quotient NTK is a stronger predictor of architectural quality. In particular, the paper does not answer the important practical question of how well the Quotient NTK predicts the actual performance or potential of architecture prior to training. The authors demonstrate that raw NTK-based proxies can produce different, and potentially misleading, rankings under function-preserving gauge transformations, but they do not show that removing the softmax gauge degree of freedom makes the Quotient NTK a more reliable criterion for architecture selection.

Critical strengths
  1. The paper challenges a fundamental assumption of training-free NAS that the value of a proxy metric is an intrinsic property of an architecture and can therefore be used to compare architectures prior to training. The authors show that this assumption is, in general, invalid for NTK-based proxies.

  2. The paper provides a strong theoretical result: it shows that a function- and training-preserving gauge transformation can introduce a PSD component into the NTK and thereby alter NTK-based proxy metrics without changing the network's predictions, loss, gradients, or training trajectory.

  3. A full audit of the TransNAS-Bench-201 search space is conducted, covering all 15,625 architectures. The authors demonstrate that gauge transformations can substantially change the values and rankings of NTK-based proxy metrics, potentially causing the NAS procedure to select architecture with substantially worse final accuracy.

  4. The paper shows that sample centering is not a universal solution to the non-identifiability problem of NTK-based proxies. In particular, the authors provide a criterion for when sample centering removes the gauge component.

  5. The softmax-free/regression-head experiment provides an important negative control. It shows that the observed effect is specifically related to the redundancy of softmax classification: the analogous modification of a regression head does not preserve the model outputs and does not produce the same proxy instability.

Critical weaknesses
  1. The novelty is partially based on a previously known idea. The projection underlying the Quotient NTK was already considered in Plenk et al. (2026). However, the authors do not claim novelty for the projection itself; rather, the main contribution is showing that this projection removes the softmax gauge degree of freedom from the considered NTK-based proxies.

  2. The theoretical results are specific to the additive gauge symmetry of softmax classifiers and to particular training dynamics for which the gauge transformation preserves the relevant gradients and parameter updates. In particular, the equivalence of training trajectories does not generally hold for optimization schemes that depend explicitly on parameter values, e.g. those involving weight decay or other parameter-dependent regularization. It is also unclear whether the claimed equivalence extends without modification to architectures involving batch normalization, layer normalization, or other components that may interact with the modified final-layer parameters.

  3. The claim that sample centering is “not a repair” seems too categorical. The authors show that sample centering does not remove the gauge component in general and provide an exact criterion for when it does. However, this does not imply that sample centering cannot serve as an effective method for eliminating gauge dependence in general.

  4. The statistical robustness of the experimental results is insufficiently investigated. In particular, there is no systematic analysis of sensitivity to random initialization or other sources of experimental variability. The experiments use a single seed, which limits the assessment of the reproducibility of the quantitative results and the robustness of the observed effects.

Questions for authors

The paper does not answer the important practical question of how well the Quotient NTK predicts the actual performance or potential of architecture prior to training. It is also unclear how the proposed gauge-sensitivity test can be used to detect that a benchmark has actually been compromised by an adversary or that a targeted attack is taking place. The proposed test establishes the vulnerability or gauge dependence of the proxy metric, but does not by itself establish that the benchmark has been compromised. The answer requires additional experiments.

This leads to a question that remains unanswered theoretically and experimentally: does the non-invariance of an NTK-based proxy necessarily imply that the proxy is a poor predictor of architecture quality? Non-invariance establishes that the proxy is not an intrinsic property of the predictive function, but it does not by itself establish weak predictive performance. Answering the latter question would require additional experiments comparing raw NTK proxies and Quotient NTK directly in terms of their ability to predict the final performance of trained architectures.

Thus, while the theoretical contribution of the paper is significant, the practical significance of the proposed Quotient NTK for NAS remains insufficiently established.

Reviewer_GfprOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that the NTK spectral summaries used as training-free architecture proxies are not intrinsic properties of softmax classifiers. Using the softmax gauge freedom, arbitrary PSD contamination can be injected into the trace NTK without changing predictions, loss, or gradients, while the class-quotient NTK remains invariant. Audits of NAS-Bench-201 and TransNAS-Bench-101 show that this substantially changes architecture rankings and selections. The failure mode is clear and practically relevant, but the remedy builds on an existing class-projection method. The ranking results lack robustness checks across seeds, probe batches, and intervention directions. The paper also does not show that the corrected diagnostics select better architectures. For some architectures, the strongest attack goes beyond the verified invariance regime, and the theory covers only additive softmax shifts under cross-entropy training.

Critical strengths
  1. The paper shows that arbitrary positive semidefinite contamination can be injected into the raw trace NTK on a finite probe set without changing softmax predictions. This exposes a fundamental limitation of such diagnostics.

  2. It constructs a same-architecture intervention that preserves both predictions and cross-entropy gradients while changing NTK-based scores. This rules out architectural differences as an explanation.

  3. It audits complete NAS search spaces and shows that the contamination substantially changes architecture rankings and shortlists, while quotient-based diagnostics remain stable.

Critical weaknesses
  1. The proposed remedy builds on an existing class-projection method. The main novelty therefore lies in identifying and auditing the failure mode, not in developing a new correction.

  2. The NAS ranking results are not validated across multiple initialization seeds, probe batches, or intervention directions, so their robustness is not yet established.

  3. The experiments establish gauge invariance, but they do not show that the corrected diagnostics predict final accuracy more reliably or lead to better architecture selection.

  4. For some architectures, the strongest selection attack uses intervention strengths beyond the numerically verified invariance regime. This weakens the claim that the attack is strictly behavior-preserving.

  5. The TransNAS audit uses reconstructed backbones and Gaussian-noise probes instead of the benchmark's original task-specific models and real inputs. This limits how well it represents practical use.

Questions for authors
  1. What is new beyond the existing class-projection method?
  2. Are the ranking results consistent across different random seeds and probe batches?
  3. Does the corrected NTK predict final accuracy more reliably?
  4. How effective is the attack within the numerically verified invariance range?
  5. Do the TransNAS findings hold with real images and the original task-specific models?
Reviewer_1ixSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the gauge non-identifiability of Neural Tangent Kernel (NTK) based training-free NAS proxies. It demonstrates that function- and gradient-preserving transformations of softmax classifiers can substantially alter raw trace-NTK spectral diagnostics without changing the underlying predictive behavior. The authors theoretically characterize this phenomenon, adopt the gauge-invariant quotient NTK, and propose a full-space audit protocol to assess its practical consequences. Experiments across NAS-Bench-201, TransNAS-Bench-101, ViT, and graph architectures demonstrate substantial changes in raw NTK diagnostics and architecture rankings, while quotient NTK diagnostics remain stable.

Critical strengths
  1. An interesting and fundamental question about NTK-based NAS proxies. Most existing studies focus on whether training-free proxies correlate with final model accuracy. This paper takes a different angle and asks whether NTK-based scores are even well-defined for functionally equivalent models. I find this perspective valuable, as it reveals a potential limitation that cannot be identified simply by evaluating prediction accuracy or ranking correlations.

  2. Clear theoretical results with a convincing construction. The authors show that raw trace-NTK diagnostics can change substantially under transformations that preserve model predictions and training gradients. Importantly, the same-architecture construction demonstrates that this issue does not require adding artificial components to the network. The analysis of quotient NTK invariance and the limitations of sample centering also helps clarify why the problem occurs and under what conditions it can be avoided.

  3. Extensive experiments that go beyond illustrative examples. The authors evaluate the phenomenon across complete NAS search spaces, a large pretrained ViT, and graph architectures. I particularly appreciate the non-softmax control experiments and the careful treatment of numerical precision, which help distinguish the claimed invariance from numerical artifacts. These experiments provide useful evidence that the theoretical issue is not restricted to a small synthetic example.

Critical weaknesses
  1. The strongest architecture-selection result needs more careful validation. The paper reports a substantial 34.6-point increase in selection regret under gauge manipulation, which is one of its most striking findings. However, this result uses gauge strengths up to γ = 100, while the stated float32 prediction-preservation bound is only verified for γ ≤ 1. As acknowledged in Appendix K and L, some architectures exceed this bound at larger strengths. I would like to see whether a comparable selection impact can still be observed within the verified range. Without this, I find the practical significance of the worst-case result somewhat less convincing.
  2. The practical impact on NAS remains unclear. The paper convincingly shows that raw NTK scores can change under function-preserving transformations, but I am less convinced that this necessarily translates into a serious problem for practical NAS. The evaluated proxies already show rather weak correlations with final accuracy, and the main ranking experiments use only one initialization and probe batch. Moreover, the targeted attack allows different gauge strengths for different architectures, which is not how standard NAS evaluations are typically conducted. It would be helpful to better understand how much these findings matter under a fixed and consistent evaluation protocol.
Questions for authors
  1. Can the authors report the worst-case selection regret when restricting the attack to γ ≤ 1? Alternatively, can they verify that the architectures responsible for the reported 34.6-point regret increase still satisfy the prediction-preservation bound? This would help clarify how much of the reported selection impact is supported by the numerical experiments.
  2. How stable are the reported ranking changes across different initialization seeds, probe batches, and gauge directions? Even a small multi-seed experiment on a subset of NAS-Bench-201 would help assess the robustness of the results.
  3. While the paper demonstrates that gauge transformations can manipulate NAS rankings, it remains unclear under what practical circumstances such attacks would occur. In particular, who would be able to modify the gauge of individual architectures, and how realistic is this threat in standard NAS workflows where initialization and evaluation protocols are typically fixed? Clarifying the practical attack scenarios would help establish the real-world significance of the findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.