- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether spectral-rank dynamics can guide label-free early stopping for Transformers used as frozen encoders. Building on prior findings of representation compression and expansion, the authors propose a rate-to-peak stopping rule based on the slowdown of spectral-rank growth after rebound. Experiments on O-RAN, chess, and Pythia examine the relationship between spectral dynamics and downstream linear-probe performance. The results show that probe gains often concentrate near spectral rebound but remain task-dependent. Comparisons between raw and standardized probes further reveal differences between improved readout accessibility and gains that persist after feature standardization. The proposed method reduces training costs substantially on Pythia, while showing limitations on chess tasks.
Critical strengths
-
The paper addresses a practical and meaningful problem of reducing training costs when using pretrained Transformers as frozen encoders.
-
The proposed label-free stopping criterion is simple and does not require downstream task labels or probe training to make stopping decisions.
-
The experiments cover multiple architectures across three distinct domains (O-RAN, chess, and Pythia), providing useful insights into the task-dependent relationship between spectral dynamics and downstream probe performance.
Critical weaknesses
-
The methodological novelty is somewhat limited, as the proposed stopping rule builds on existing spectral-rank measurements and previously studied compression–expansion dynamics.
-
The stopping criterion shows inconsistent reliability across domains. While it performs well on Pythia, the results on O-RAN and chess reveal limitations in its robustness and practical applicability.
-
The stopping performance is sensitive to the choice of the rate threshold. Further analysis of its robustness to spectral estimation and checkpoint sampling would strengthen the method's reliability.
Questions for authors
-
Have the authors considered alternative spectral metrics, such as effective rank or spectral entropy, for the stopping criterion? It would be helpful to clarify whether the observed benefits are specific to the proposed k95-based rate-to-peak rule or can be achieved with other spectral measures.
-
How was the threshold q = 0.1 selected? Given the variation in stopping performance across thresholds and domains, could the authors clarify whether this choice was made independently of downstream labels and how robust a single fixed threshold is across different settings?
2 comments
Thank you for the detailed rebuttal and additional experimental results. I appreciate the authors' efforts in addressing my questions, particularly the comparisons with alternative spectral metrics, the cross-domain threshold transfer experiments, and the additional robustness analyses.
These results provide useful clarifications and strengthen the empirical evaluation of the proposed stopping criterion. However, my main concerns regarding the limited methodological novelty and the consistency of the stopping criterion across different domains remain. In particular, while the additional experiments demonstrate encouraging results in certain settings, the effectiveness and computational benefits of the approach still appear to be domain-dependent. Furthermore, the clarification that the threshold was selected after examining labeled sweeps raises some remaining questions about the extent to which the overall procedure can be considered fully label-free.
Overall, while I appreciate the additional evidence provided in the rebuttal, I do not believe it substantially changes my original assessment of the paper. Therefore, I will maintain my initial rating.
Thank you for a careful review! (The ethics flag reads "Yes" with the note "No ethics review needed". We assume the flag was unintended.)
Novelty (W1): The spectral trajectory is prior work. What the paper adds is the downstream question: probe acquisition is target dependent, a fast raw-probe gain can be readout accessibility with no new information, and a label-free stopping rule is evaluated against label-using oracles with its failures reported.
Other spectral measures (Q1): The same rule on all 78 runs at q = 0.1, as median stop / layer-runs passing:
On Pythia and O-RAN the three measures give similar trade-offs, though not at a common threshold. On chess only k95 is safe and the others fire near the rebound, 12 to 18 points short on the worst task.
Choice of q and robustness (Q2, W2, W3): q is not a hyperparameter with a correct value. It picks an operating point on a trade-off between compute saved and worst-task shortfall, and each value of q is a legitimate point on that curve. We report q = 0.1, which we fixed after seeing labeled sweeps over q. The sweep used 47 of the 78 runs and the other 31 were held out. The value is also stable under held-out tuning: tuning q on two domains and testing on the third selects 0.08, 0.10 and 0.08, under our shortfall and cost model. A different shortfall and cost model would find a different but equally valid range of q.
Applied to each held-out domain, q tuned on the other two (0.08 for chess, 0.10 for O-RAN, 0.08 for Pythia) gives a worst-task shortfall of 0.2, 2.2 and 0.6 points respectively.
Baselines tuned the same way transfer worse. On held-out chess, a fixed fraction or loss threshold stops at about 38% of training, 4.5-8.6 points short. On held-out O-RAN and Pythia they train to 86–100%, saving almost nothing. The one exception is the loss threshold on Pythia, which stops earlier than the rule (5% vs 14%) but passes fewer layer-runs (77% vs 93%). On O-RAN with held-out loss (ten replays), the loss baselines stop at 95% (transferred threshold) and 58% (patience-based early stopping), against the rule's 36%, for 0.7 and 0.4 points worst-task shortfall lower than the rule's.
On O-RAN the stop is stable to the smoothing half-life (33 to 37% of training for half-lives of 2.5 to 10 updates) and to seeing only every every second or fourth spectrum. On Pythia, thinning the checkpoints moves the stop later. It is sensitive to the variance level - with k90 or k99 the chess pass rate falls from 85% to 64% or 35%.