- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether severe representation collapse predicts failure to recover under a given cross-entropy (CE) training setup and computational budget. The authors construct collapsed checkpoints of CIFAR classification models using auxiliary regularization and report that subsequent CE training can achieve recovery even when the feature representations are nearly rank one or the static Feature-NTK spectrum is severely compressed. Based on these observations, they propose a diagnostic rule that measures whether the Feature-NTK spectrum re-expands after 50 minibatches of CE training. This rule agrees with the recovery outcomes of eight checkpoints used to develop the criterion and five new cutoff checkpoints evaluated prospectively after the criterion was fixed. The authors also perform interventions on BN-ResNet-18 that alter the distribution of batch normalization (BN) scales while keeping their overall RMS unchanged. Preserving the layer-wise RMS differences maintains training stagnation across four checkpoints, whereas removing these differences enables recovery. These results suggest that recovery depends not only on the overall scale magnitude but also on the checkpoint-specific distribution of BN scales across layers.
Critical strengths
-
Concrete Counterexamples: The paper distinguishes the current state of representation collapse from the ability to recover by demonstrating cases where CE training achieves recovery despite severe representation collapse or a static Feature-NTK cutoff.
-
Controlled BN Interventions: By keeping the overall RMS of BN scale parameters unchanged, the authors investigate how the distribution of these scales affects recovery beyond what can be explained by their overall magnitude alone.
Critical weaknesses
-
The substantive novelty relative to prior work remains unclear. The distinction between static feature rank and learning capacity, as well as the importance of NTK evolution during training, has already been discussed in prior research. Although this paper examines these ideas in a specific representation collapse setting, it does not sufficiently explain how its observations extend existing understanding. It remains unclear whether the differences in experimental setting and measurement procedure constitute a meaningful scientific advance.
-
The central diagnostic rule lacks sufficient justification and validation. The 50-step probe and its thresholds constitute an empirical heuristic derived from retrospective cases, without theoretical guarantees linking them to long-term recovery. Although an empirical diagnostic does not necessarily require a theoretical guarantee, only five new cutoff checkpoints were evaluated after fixing the rule. Furthermore, its additional value over simply monitoring loss or accuracy, as well as its computational efficiency advantages, have not been sufficiently established.
-
The BN interventions provide limited mechanistic insight. Although the matched-RMS comparison is appropriate, the main results are restricted to four locked checkpoints of BN-ResNet-18. The experiments demonstrate that changing the arrangement of BN scale parameters can alter recovery outcomes, but do not sufficiently explain why the original arrangement maintains locking during CE training. The fact that some checkpoints recover even under permutations that preserve layerwise RMS further indicates that the proposed profile-based explanation alone is insufficient to account for the observed outcomes.
Questions for authors
-
What are the new insights provided by this paper relative to prior work? Given existing research on the distinction between static feature rank and learning capacity, as well as the temporal evolution of the NTK, please clarify the key advances this paper offers beyond differences in experimental settings and measurement procedures.
-
What motivates the choice of the 50-step diagnostic rule, and what is its additional predictive value? How were the number of updates and the thresholds determined, and do the diagnostic outcomes remain consistent when the measurement images or projection directions are changed? Beyond the agreement observed on five new cutoff checkpoints, can the authors provide evidence that the proposed diagnostic offers advantages over simply monitoring loss or accuracy under the same computational budget?
-
What specific mechanism allows the arrangement of BN scales to maintain locking? If some checkpoints recover even after permutations that preserve layerwise RMS, which properties of the layerwise or channelwise scale arrangement are actually important? Beyond demonstrating the effects of modifying , please explain why CE training fails to escape the original configuration but achieves recovery after the intervention.
-
How are the execution states of the probe and long-term recovery separated? Do the two runs start from independent copies of the original checkpoint, or does long-term recovery continue from the state reached after the probe? Additionally, please clarify whether the probe restores momentum from auxiliary training and how optimizer states and BN buffers are initialized or preserved.
-
How is independence maintained between checkpoint selection and diagnostic evaluation? Since the spectrum measured on a fixed test subset is used both for determining cutoff and for stopping auxiliary training, evaluating new checkpoints alone does not ensure independence of the measurement data. Please demonstrate whether the same diagnostic decisions hold on a separate subset that was not used for checkpoint selection.
-
Is the reported locking behavior also confirmed on the training objective? Can the authors provide the training CE loss and training accuracy trajectories for the locked checkpoints? Please explain how they distinguish genuine optimization stagnation from evaluation failures caused by differences between BN training and evaluation modes or by running statistics, given that low test accuracy and a collapsed spectrum at evaluation time do not necessarily establish optimization failure.