acceptodds
Withdrawn from ICLR 2027

Geometry is Not the Trap: A Spectral‑Reopening Diagnostic for Protocol-Bounded Optimization Locking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While practice often equates the geometric collapse of deep representations with a loss of recoverability, and existing analyses frequently rely on proxies such as Neural Collapse, the relationship between whole-manifold rank-one geometric cliffs and persistent protocol lock under cross-entropy (CE)—the inability of CE to recover under a given protocol and budget—remains unclear. We show that what distinguishes recoverability under CE is not static forward geometry, but a dynamical object: whether the full-extractor Feature Neural Tangent Kernel (Feature-NTK) reopens. Across the convolutional networks we test, trained on standard image-classification benchmarks and covering residual networks, Visual Geometry Group (VGG) networks, and Group Normalization, standard CE recovers from the tested whole-manifold rank-one geometric cliffs. Among checkpoints with severe Feature-NTK spectral cut-offs induced by structured auxiliary regularizers, some checkpoints can still reopen the Feature-NTK and recover to approximately accuracy, so a spectral cut-off is not sufficient for persistent protocol lock. The critical bifurcation between persistent protocol lock and recoverable cut-off is whether the Feature-NTK reopens under a controlled -minibatch CE spectral-reopening probe. Through controlled parameter interventions, we further show that maintaining the lock relies on a layer-structured batch normalization (BN) scale configuration. Finally, after freezing the rule, we prospectively evaluate unseen cut-off checkpoints across architectures. On the checkpoints we test, persistent protocol lock tracks whether the Feature-NTK reopens rather than static geometry.

Primary area:
optimization
Keywords:
Training Dynamics, Neural Collapse, Neural Tangent Kernel

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eb9NOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether severe representation collapse predicts failure to recover under a given cross-entropy (CE) training setup and computational budget. The authors construct collapsed checkpoints of CIFAR classification models using auxiliary regularization and report that subsequent CE training can achieve recovery even when the feature representations are nearly rank one or the static Feature-NTK spectrum is severely compressed. Based on these observations, they propose a diagnostic rule that measures whether the Feature-NTK spectrum re-expands after 50 minibatches of CE training. This rule agrees with the recovery outcomes of eight checkpoints used to develop the criterion and five new cutoff checkpoints evaluated prospectively after the criterion was fixed. The authors also perform interventions on BN-ResNet-18 that alter the distribution of batch normalization (BN) scales while keeping their overall RMS unchanged. Preserving the layer-wise RMS differences maintains training stagnation across four checkpoints, whereas removing these differences enables recovery. These results suggest that recovery depends not only on the overall scale magnitude but also on the checkpoint-specific distribution of BN scales across layers.

Critical strengths
  1. Concrete Counterexamples: The paper distinguishes the current state of representation collapse from the ability to recover by demonstrating cases where CE training achieves recovery despite severe representation collapse or a static Feature-NTK cutoff.

  2. Controlled BN Interventions: By keeping the overall RMS of BN scale parameters γ\gamma unchanged, the authors investigate how the distribution of these scales affects recovery beyond what can be explained by their overall magnitude alone.

Critical weaknesses
  1. The substantive novelty relative to prior work remains unclear. The distinction between static feature rank and learning capacity, as well as the importance of NTK evolution during training, has already been discussed in prior research. Although this paper examines these ideas in a specific representation collapse setting, it does not sufficiently explain how its observations extend existing understanding. It remains unclear whether the differences in experimental setting and measurement procedure constitute a meaningful scientific advance.

  2. The central diagnostic rule lacks sufficient justification and validation. The 50-step probe and its thresholds constitute an empirical heuristic derived from retrospective cases, without theoretical guarantees linking them to long-term recovery. Although an empirical diagnostic does not necessarily require a theoretical guarantee, only five new cutoff checkpoints were evaluated after fixing the rule. Furthermore, its additional value over simply monitoring loss or accuracy, as well as its computational efficiency advantages, have not been sufficiently established.

  3. The BN interventions provide limited mechanistic insight. Although the matched-RMS comparison is appropriate, the main results are restricted to four locked checkpoints of BN-ResNet-18. The experiments demonstrate that changing the arrangement of BN scale parameters γ\gamma can alter recovery outcomes, but do not sufficiently explain why the original arrangement maintains locking during CE training. The fact that some checkpoints recover even under permutations that preserve layerwise RMS further indicates that the proposed profile-based explanation alone is insufficient to account for the observed outcomes.

Questions for authors
  1. What are the new insights provided by this paper relative to prior work? Given existing research on the distinction between static feature rank and learning capacity, as well as the temporal evolution of the NTK, please clarify the key advances this paper offers beyond differences in experimental settings and measurement procedures.

  2. What motivates the choice of the 50-step diagnostic rule, and what is its additional predictive value? How were the number of updates and the thresholds determined, and do the diagnostic outcomes remain consistent when the measurement images or projection directions are changed? Beyond the agreement observed on five new cutoff checkpoints, can the authors provide evidence that the proposed diagnostic offers advantages over simply monitoring loss or accuracy under the same computational budget?

  3. What specific mechanism allows the arrangement of BN scales to maintain locking? If some checkpoints recover even after permutations that preserve layerwise RMS, which properties of the layerwise or channelwise scale arrangement are actually important? Beyond demonstrating the effects of modifying γ\gamma, please explain why CE training fails to escape the original configuration but achieves recovery after the intervention.

  4. How are the execution states of the probe and long-term recovery separated? Do the two runs start from independent copies of the original checkpoint, or does long-term recovery continue from the state reached after the probe? Additionally, please clarify whether the probe restores momentum from auxiliary training and how optimizer states and BN buffers are initialized or preserved.

  5. How is independence maintained between checkpoint selection and diagnostic evaluation? Since the spectrum measured on a fixed test subset is used both for determining cutoff and for stopping auxiliary training, evaluating new checkpoints alone does not ensure independence of the measurement data. Please demonstrate whether the same diagnostic decisions hold on a separate subset that was not used for checkpoint selection.

  6. Is the reported locking behavior also confirmed on the training objective? Can the authors provide the training CE loss and training accuracy trajectories for the locked checkpoints? Please explain how they distinguish genuine optimization stagnation from evaluation failures caused by differences between BN training and evaluation modes or by running statistics, given that low test accuracy and a collapsed spectrum at evaluation time do not necessarily establish optimization failure.

Reviewer_4Y3DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when a collapsed image classifier can be retrained back to useful accuracy with plain cross-entropy (CE). The authors call a collapsed checkpoint "locked" if it stays collapsed and at chance accuracy after a fixed amount of retraining (Section 3.2). Collapsed checkpoints are built on purpose: a CE warmup, then a five-term auxiliary penalty (feature anchor, residual-map suppression, input-sensitivity penalty, L2 and L1 penalties on BN scales; Section 5, Eq. 1; Appendix B.3). On CIFAR-10 with BN-ResNet-18, ResNet-34, VGG-11-BN and ResNet-18-GN, the authors report:

  • all nine cross-architecture rank-one cliffs (1.001 ≤ rPR ≤ 1.023) leave the cliff under 20 block-reset CE rounds, reaching 0.241 to 0.453 accuracy (Section 4; Table 2);
  • among five severe Feature-NTK cut-offs with similar static ratios (3×10^-17 to 1.3×10^-16), four stay locked and one (seed 9) recovers to 0.792 (Section 5);
  • a 50-minibatch CE probe with the rule "recover if ρ0 > 10^-6 or ρ50 > 10^-3" matches all 8 retrospective outcomes (Eq. 2; Table 4) and, with thresholds frozen, all 5 prospective outcomes (Tables 5, 6);
  • at matched global BN-scale RMS, keeping each layer's RMS keeps four checkpoints locked at 0.100, while flattening RMS across layers reopens the spectrum and allows recovery to 0.782 to 0.842 (Section 6; Table 7).
Critical strengths
  • The BN experiments are well designed. Keeping the overall scale size fixed while changing only how it is spread across layers cleanly shows that the spread, not the size, keeps the network stuck (Section 6; Table 7).
  • The claims are carefully limited. The paper calls its test "an operational criterion, not a universal trainability certificate" (Section 3.2), and reports the settings where it fails to produce a stuck network (Sections 3.1, 5).
Critical weaknesses

1. The problem is artificial.

  • Every stuck network is made by the authors' own five-term penalty (Section 5, Eq. 1), which they say is not an attack (Sections 2, 5). Without it, no network gets stuck; the federated tests produce none (Appendix D.6).
  • No naturally occurring collapse is studied, such as unstable training or plasticity loss in continual learning, which the paper names as related (Section 2).
  • The motivating belief ("practice often equates the geometric collapse ... with a loss of recoverability", Abstract) is not supported by a citation or an example.
  • Recovery is not clearly worth it. Training from scratch reaches 0.807 in 100 rounds (Appendix D.1), about the same as the one recovered stuck-looking network (0.792).

2. The geometric-collapse result is expected. All nine collapsed networks keep a working kernel (ρ from 1.77×10^-3 to 0.883; Table 2), so ordinary training is expected to recover them. Section 1 itself notes that prior work does "not make every severe collapse a certificate of irrecoverability."

3. The BN finding is unexplained and may not transfer.

  • No explanation. The paper shows that how BN scales are spread across layers keeps a network stuck, but not why (Section 7). The edits follow no consistent rule: shuffling scales within a layer frees two of four stuck networks (Appendix D.4).
  • Only BN ResNets get stuck. With GroupNorm or without skip connections, no network gets stuck (Section 5; Appendix D.1), so the effect is unlikely to appear in current architectures. The paper should test at least one, for example a small ViT.

4. Presentation issues.

  • Inconsistent numbers. Seed 9's starting value is 6.24×10^-17 in Table 3 but 4.10×10^-17 in Table 4, with no explanation.
  • A missing result. Seed 15 is reported under only one of the two retraining setups; the other four new cases have both (Section 5).
  • Hard to compare across tables. The spectrum is measured on 10, 30 or 60 images depending on the experiment (Appendix C.1). On one checkpoint, two of these differ by about 2× (6.77×10^-4 vs 1.32×10^-3; Appendix C.2).
  • Heavy terminology. Terms such as "whole-manifold rank-one geometric cliff" and "persistent protocol lock" could be replaced with plain descriptions.
  • A loose definition of "recovers." Any improvement in accuracy counts (Section 3.1), so a network at 24% accuracy on CIFAR-10 is called recovered (Table 2).
Questions for authors
  1. In this paper, "collapse" is produced on purpose with a five-term penalty (Section 5, Eq. 1). No practitioner would collapse a model this way and then retrain it back. Can you define collapse as a state that occurs in real training, for example after a diverging learning rate or during continual learning, and show that your stuck checkpoints match it?
  2. If the goal is to understand collapse in modern models such as transformers or large vision-language models, why are small BN CNNs on CIFAR a valid proxy, given that the stuck state disappears with GroupNorm or without skip connections (Section 5; Appendix D.1)?
  3. When is recovering a collapsed checkpoint better than retraining from scratch, given that training from scratch reaches 0.807 accuracy (Appendix D.1)?
  4. Why does the way BN scales are spread across layers keep a network stuck? Without an explanation, what is the main takeaway beyond the checkpoints you constructed?
Reviewer_VgcDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors of this work challenge the consensus of previous works that neural collapse of neural networks can be statically connected with the ability of a network to ``escape" a bad feature geometry and achieve improved performance. Specifically, they show that networks in which feature energy is concentrated in a single dimension can exhibit different behaviors as training progresses, with some escaping the bad feature geometry. Contrary to that, they propose a dynamic test for such feature collapse and protocol lock (as they refer the inability of the network to recover), where they monitor the feature geometry over a batch of 50 iterations. The authors show that this window is enough to separate networks that are able to escape the bad feature geometry state and networks that are not. Finally, they showcase how interventions in the Batch Normalization scaling can allow even networks under protocol lock to recover.

Critical strengths
  • The proposed approach for monitoring the training state of a network is a topic that would be of significant interest to the community. Additionally, both the proposed diagnostic for protocol lock and interventions through Batch Norm rescaling are easy to implement and adopt, which increases the significance of the contribution.
  • The presentation of the idea is clear and accessible
Critical weaknesses
  • While the proposed protocol lock detection heuristic along with the proposed intervention can be interesting to the community, the authors do not provide any specific hypothesis about the mechanism underlying these phenomena. As such, the proposed approaches might seem anecdotal and only applicable to the specific set of architectures.
Questions for authors
  • What might be the cause that the BN intervention allows for reopening of the NTK dimensions while other interventions fail?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.