acceptodds
Under review as a conference paper at ICLR 2027

Early-Stopping Frozen Transformer Encoders via Spectral Rank

Abstract

Selecting a transformer backbone for use as a frozen encoder requires committing to depth, head count, width, positional encoding scheme and normalization style. Each combination is conventionally evaluated by training the backbone to convergence and probing it on downstream tasks. Recent theory, validated on synthetic data, predicts a two-phase geometric trajectory for transformers when trained on standard next-token prediction. Transformer representations first compress into low-dimensional, factored subspaces. They then re-expand to capture joint correlations and reduce loss. We investigate a downstream consequence the original theory does not address: whether gains in linear probe accuracy on held-out tasks concentrate near the compression-to-expansion transition, and how these gains depend on the probed target. We test this on two structured-sequence domains: chess and network traffic trace-derived tokens from a specification-compliant Open Radio Access Network (O-RAN) testbed generating data on a variety of telecom equipment combinations, and on natural language using published Pythia training trajectories. We train a family of small next-token language models on these domains, varying architectural choices such as depth, normalization style, RoPE layout and RoPE subspace allocation. Using intra-epoch checkpoints we track both the 95%-variance spectral rank of layer-1 representations and the test accuracy of linear probes trained on held-out tasks whose labels the language model does not see. We show that across the position-aware architectural variants tested, with the no-position control behaving as an expected exception, gains for particular probes concentrate near the point of maximal spectral compression. For these probes, continued language-model training contributes marginal, and in some configurations negative, improvement.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation learning, transformers, linear probing, training dynamics, representation geometry, spectral analysis, self-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mDhPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether spectral-rank dynamics can guide label-free early stopping for Transformers used as frozen encoders. Building on prior findings of representation compression and expansion, the authors propose a rate-to-peak stopping rule based on the slowdown of spectral-rank growth after rebound. Experiments on O-RAN, chess, and Pythia examine the relationship between spectral dynamics and downstream linear-probe performance. The results show that probe gains often concentrate near spectral rebound but remain task-dependent. Comparisons between raw and standardized probes further reveal differences between improved readout accessibility and gains that persist after feature standardization. The proposed method reduces training costs substantially on Pythia, while showing limitations on chess tasks.

Critical strengths
  1. The paper addresses a practical and meaningful problem of reducing training costs when using pretrained Transformers as frozen encoders.

  2. The proposed label-free stopping criterion is simple and does not require downstream task labels or probe training to make stopping decisions.

  3. The experiments cover multiple architectures across three distinct domains (O-RAN, chess, and Pythia), providing useful insights into the task-dependent relationship between spectral dynamics and downstream probe performance.

Critical weaknesses
  1. The methodological novelty is somewhat limited, as the proposed stopping rule builds on existing spectral-rank measurements and previously studied compression–expansion dynamics.

  2. The stopping criterion shows inconsistent reliability across domains. While it performs well on Pythia, the results on O-RAN and chess reveal limitations in its robustness and practical applicability.

  3. The stopping performance is sensitive to the choice of the rate threshold. Further analysis of its robustness to spectral estimation and checkpoint sampling would strengthen the method's reliability.

Questions for authors
  1. Have the authors considered alternative spectral metrics, such as effective rank or spectral entropy, for the stopping criterion? It would be helpful to clarify whether the observed benefits are specific to the proposed k95-based rate-to-peak rule or can be achieved with other spectral measures.

  2. How was the threshold q = 0.1 selected? Given the variation in stopping performance across thresholds and domains, could the authors clarify whether this choice was made independently of downstream labels and how robust a single fixed threshold is across different settings?

2 comments

Thank you for a careful review! (The ethics flag reads "Yes" with the note "No ethics review needed". We assume the flag was unintended.)

Novelty (W1): The spectral trajectory is prior work. What the paper adds is the downstream question: probe acquisition is target dependent, a fast raw-probe gain can be readout accessibility with no new information, and a label-free stopping rule is evaluated against label-using oracles with its failures reported.

Other spectral measures (Q1): The same rule on all 78 runs at q = 0.1, as median stop / layer-runs passing:

k95 (paper)Participation ratioEntropy effective rank
Pythia14% / 92.3%4% / 76.9%14% / 89.2%
O-RAN35% / 60.5%19% / 55.0%25% / 55.5%
Chess100% / 85.3%2% / 7.4%3% / 21.1%

On Pythia and O-RAN the three measures give similar trade-offs, though not at a common threshold. On chess only k95 is safe and the others fire near the rebound, 12 to 18 points short on the worst task.

Choice of q and robustness (Q2, W2, W3): q is not a hyperparameter with a correct value. It picks an operating point on a trade-off between compute saved and worst-task shortfall, and each value of q is a legitimate point on that curve. We report q = 0.1, which we fixed after seeing labeled sweeps over q. The sweep used 47 of the 78 runs and the other 31 were held out. The value is also stable under held-out tuning: tuning q on two domains and testing on the third selects 0.08, 0.10 and 0.08, under our shortfall and cost model. A different shortfall and cost model would find a different but equally valid range of q.

Applied to each held-out domain, q tuned on the other two (0.08 for chess, 0.10 for O-RAN, 0.08 for Pythia) gives a worst-task shortfall of 0.2, 2.2 and 0.6 points respectively.

Baselines tuned the same way transfer worse. On held-out chess, a fixed fraction or loss threshold stops at about 38% of training, 4.5-8.6 points short. On held-out O-RAN and Pythia they train to 86–100%, saving almost nothing. The one exception is the loss threshold on Pythia, which stops earlier than the rule (5% vs 14%) but passes fewer layer-runs (77% vs 93%). On O-RAN with held-out loss (ten replays), the loss baselines stop at 95% (transferred threshold) and 58% (patience-based early stopping), against the rule's 36%, for 0.7 and 0.4 points worst-task shortfall lower than the rule's.

On O-RAN the stop is stable to the smoothing half-life (33 to 37% of training for half-lives of 2.5 to 10 updates) and to seeing only every every second or fourth spectrum. On Pythia, thinning the checkpoints moves the stop later. It is sensitive to the variance level - with k90 or k99 the chess pass rate falls from 85% to 64% or 35%.

Thank you for the detailed rebuttal and additional experimental results. I appreciate the authors' efforts in addressing my questions, particularly the comparisons with alternative spectral metrics, the cross-domain threshold transfer experiments, and the additional robustness analyses.

These results provide useful clarifications and strengthen the empirical evaluation of the proposed stopping criterion. However, my main concerns regarding the limited methodological novelty and the consistency of the stopping criterion across different domains remain. In particular, while the additional experiments demonstrate encouraging results in certain settings, the effectiveness and computational benefits of the approach still appear to be domain-dependent. Furthermore, the clarification that the threshold was selected after examining labeled sweeps raises some remaining questions about the extent to which the overall procedure can be considered fully label-free.

Overall, while I appreciate the additional evidence provided in the rebuttal, I do not believe it substantially changes my original assessment of the paper. Therefore, I will maintain my initial rating.

Reviewer_TrjnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the performance of a linear probe on target data during the training of a transformer model. Prior work, validated on synthetic data, has shown that transformers undergo a two-phase geometric transition when trained with next token prediction loss, where the transformer representations first compress into a low dimensional space and then expand into high dimensional space. This paper particularly studies the gains in the performance of linear probes during this transition of the transformer representation from compressed to expanded representations, and show that most of the gains in linear probe performance is obtained around this transition. The paper trains several small transformer models on two small structured datasets: Open Radio Access Network (O-RAN), and chess dataset, as well as studies trajectories for natural language tasks from the Pythia training trajectories. The results show similar patterns with most probe gains coming near the transition point from compressed to expanded representations. Based on these experiments the paper proposes a label-free stopping condition based on this geometric transition of representations instead of the classical and expensive way training linear probes at different points in the training of a transformer.

Critical strengths
  • This paper studies the relationship between the performance of a linear probe on downstream task and the geometric representation of the transformer backbone the linear probes are trained on. The paper finds that most of the gains in performance of the linear probes come around a transition period from compressed to expanded representation. This is an interesting observation.
  • The paper proposes a label-free stopping criteria for stopping the training of a transformer backbone based on the geometric representation of a transformer during its training instead of requiring to train linear probes for downstream tasks at different stages of training of the transformer backbone.
  • The paper provides experiments on 2 structured datasets: a) chess and b)Open Radio Access Network (O-RAN), and for natural language tasks the paper uses trajectories directly from the Pythia open sourced training checkpoints for the spectral analysis. Across these experiments, the paper observes that most of the gains in linear probes comes around the geometric transition of representation.
Critical weaknesses
  • The paper provides an interesting observation between correlation of the geometric transition in transformer representations and downstream accuracy of linear probe. However, the paper does not provide any theoretical/algorithmic intuition behind it. Without any intuition, it is unclear whether the results will generalize beyond the presented transformer backbones or datasets. Is it possible to provide any intuition behind the observations or under what condition it does or does not hold?
  • Minor point: in Fig. 2, the “grey strips: smoothed k_95” in the figure is not clearly visible. Please make the table easier to interpret.
Questions for authors
  • Please address the weaknesses.
1 comment

Thank you for a careful review!

Intuition (W1): The intuition is about readouts. A linear probe trained on a budget learns along high-variance directions first, and k95 counts how many directions carry the variance. When rank collapses, information can be present but out of such a probe's reach. At chess update 50, where k95 is 1.6 (68 at initialization), the paper's probe reads 12 points below a probe solved to convergence on the same features, and the gap closes as rank recovers (one run, final layer). The rule stops when the average growth of rank since the rebound has fallen to a fraction q of its peak, so it is a test for "expansion has slowed", and it is safe to the extent that probe gains end when expansion slows. What we do not have is a theory that predicts for which targets they do. On Pythia they do, at the final layer, gains finish a median of two doublings after the rebound, and at 14% of training the stopped encoder is within 0.2 points of the final one on average. On chess they do not, rank keeps expanding, probes keep improving for a median of six doublings, and the rule declines to stop 10 of 15 runs. On O-RAN the raw-probe gain is a change in feature conditioning, not new information (Sections 5.2 and 5.4). On 50 retrained runs a converged probe falls about 10 points over training on the two multi-class tasks while the fixed-budget probe rises. So we can state the conditions under which the rule was safe in three domains but we lack a predictive theory of their behavior.

Figure 2. (W2): The grey k95 strips will be redrawn larger, with their own axis and values.

Reviewer_oACaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when to stop next-token pretraining for a transformer that will later serve as a frozen encoder. It tracks the covariance spectral rank of hidden representations and relates their compression and subsequent expansion to downstream linear-probe performance. The proposed stopping rule detects when spectral growth has slowed relative to its earlier peak, without using downstream labels. Experiments on O-RAN traces, chess, and Pythia trajectories show that acquisition timing depends on the task. The paper reports substantial training reductions on O-RAN and Pythia and uses standardized probes to distinguish learned information from changes in readout accessibility.

Critical strengths

The comparison across domains reveals an interesting difference in acquisition timing. The chess results show that useful concepts can continue improving long after the initial spectral rebound, while the evaluated Pythia syntactic tasks saturate much earlier. This makes the observational study more informative than a claim that one geometric transition universally marks the end of learning. The repeated seeds and measurements across layers help support this distinction.

The standardized-probe control is valuable. On O-RAN, the standardized probe already reaches 79.5% at initialization, exceeding the trained raw probe’s 64.6%. Reporting this result substantially changes how the apparent benefit of pretraining should be understood. The distinction between acquiring information and making existing information easier for a particular readout to access is useful beyond this application.

Critical weaknesses

The reported accuracy at the stopping time can come from a different, label-selected checkpoint. Section 4.3 defines accuracy at step (t) using the probe with the highest validation accuracy among all checkpoints up to (t). Consequently, different tasks can be evaluated using different earlier backbones. The stopping decision itself uses no labels; my concern is which encoder the reported result evaluates. For the stated goal of producing a reusable frozen encoder, the paper needs to evaluate the actual stopped checkpoint, with every task using that same backbone. The current protocol establishes what can be recovered from a saved checkpoint history using downstream validation labels. It leaves the quality of the encoder returned at the stopping time unresolved.

The reliability evidence falls short of the paper’s stated success criterion. At (q=0.1), only 61% of O-RAN layer-runs meet the requirement that every task remain within two percentage points of its reference performance. One chess run stops at step 440 with a reported 22.7-point shortfall, while ten of fifteen chess runs obtain no early stopping benefit. Table 9’s non-inferiority test uses a five-point margin on shortfall averaged across tasks and layers, which does not establish the stricter per-task criterion. Small test sets could contribute to the O-RAN failures, but attributing them to probe noise requires paired uncertainty estimates. These failures matter when the intended use is to stop training before the downstream tasks are known.

Questions for authors

Checkpoint selection: What are the results when each task is evaluated using the backbone at the actual spectral stopping time, without selecting among earlier backbone checkpoints? How much do the reported savings and pass rates change?

Probe optimization: Section 5.4 reports that 88% of O-RAN raw-probe fits reach the 200-epoch cap. Do the acquisition curves and architecture rankings persist with converged, regularization-tuned linear probes? Please include the standardized, untrained encoder as a reference.

Stopping reliability: Can you report paired uncertainty for individual-task shortfalls and the frequency of premature stops under the stated two-point tolerance? In particular, what evidence supports attributing the O-RAN failures primarily to evaluation noise?

Threshold selection and baselines: How was (q=0.1) chosen, and which runs or domains were untouched during that choice? Does its advantage persist against equally tuned loss-slowdown and fixed-fraction baselines on held-out evaluation runs? For O-RAN, the loss baseline should also be checked using held-out loss, since Appendix D currently uses training loss on a repeatedly visited corpus.

1 comment

Thank you for a careful review!

Which encoder is scored (W1, Q1): Yes, tasks are credited at different saved checkpoints. We still think the protocol answers a real question. Stopping ends training, a downstream user selects among saved checkpoints by validation score, just as classic early stopping returns its best, not last, checkpoint. Training to the end is scored the same way, so the comparison isolates what further training buys.

With one shared checkpoint for all tasks, accuracy at the stop changes by +0.01 points on O-RAN and +0.03 on Pythia, chess is 1.3 lower at the stop and equally lower at the end of training. With the encoder at the actual stop and no selection, Pythia is unchanged (0.00; 96.9% of layer-runs pass). On O-RAN, where we replayed all 50 runs and probed at the stop, the stopped encoder is 1.46 points (95% interval 1.08 to 1.86) below the end of training, against 0.52 under per-task selection. Its 2-point pass rate is 32.5% against 68.0% under per-task selection (on the original runs, test noise alone fails 38% under this scoring) savings are unchanged.

Reliability (W2, Q3): We agree that a margin on the mean did not establish a per-task 2-point criterion. In the requested paired bootstrap, median O-RAN task-shortfall SE (test sampling only) is 1.96 points. Test noise alone would fail 21% of layer-runs; 39.5% fail. So about half of the O-RAN failures are noise and half are real (mean shortfall 0.77 points), and our sentence attributing them to noise "rather than the stopping point" was too strong. On chess we agree that the rule declines to stop 10 of 15 runs, stops four late and one early.

Converged probes (Q2): A converged probe measures what is linearly decodable while the paper's fixed-budget probe measured what a budgeted readout can extract, which also depends on the geometry the rule monitors.

As requested, we solved the same linear probe to its optimum with a validation-tuned L2 penalty, on identical features. On chess and Pythia (one new run each, final layer) the converged probe keeps 89% (86 to 91) and 106% (101 to 111) of the summed fixed-budget gain from initialization to the end of training. On the untrained Pythia encoder a converged standardized probe reads 57 to 79%, against 93 to 98% trained. On chess, two small-gain tasks keep half or less of their gain, and 4 of 14 tasks are read better from the untrained encoder. At chess update 50, where k95 is at its minimum (1.6, from 68 at initialization), the fixed-budget probe reads 12 points below the converged one on average and the gap closes by update 500 as rank recovers. On O-RAN we retrained all 50 runs and fitted both probes on the same layer-1 features. From initialization to the end of training the converged probe falls from 77.2% to 65.8% on the intervention task and from 75.0% to 65.2% on configured variant, in 49 and 48 of the 50 runs, where the fixed-budget probe rises by 10 and 14 points. The O-RAN gain is readout accessibility, as Section 5.2 argues.

So fixed-budget accuracy mixes information and accessibility, converged probes attribute the chess and Pythia gains to the first, and the O-RAN gain and the early chess dip to the second. On architecture rankings (Section 5.7), agreement with the final architecture order at initialization / rebound / twice the rebound is 0.03 / 0.08 / 0.22 with the converged probe on the 50 retrained runs (fixed-budget probe in the paper: 0.19 / 0.62 / 0.83). The ranking result is therefore specific to the fixed-budget probe/accessibility.

Threshold and baselines (Q4): q is not a hyperparameter with a correct value. It picks an operating point on a trade-off between compute saved and worst-task shortfall, and each value of q is a legitimate point on that curve. We report q = 0.1, which we fixed after seeing labeled sweeps over q. The sweep used 47 of the 78 runs and the other 31 were held out. The value is also stable under held-out tuning: tuning q on two domains and testing on the third selects 0.08, 0.10 and 0.08, under our shortfall and cost model. A different shortfall and cost model would find a different but equally valid range of q.

Applied to each held-out domain, q tuned on the other two (0.08 for chess, 0.10 for O-RAN, 0.08 for Pythia) gives a worst-task shortfall of 0.2, 2.2 and 0.6 points respectively.

Baselines tuned the same way transfer worse. On held-out chess, a fixed fraction or loss threshold stops at about 38% of training, 4.5-8.6 points short. On held-out O-RAN and Pythia they train to 86–100%, saving almost nothing. The one exception is the loss threshold on Pythia, which stops earlier than the rule (5% vs 14%) but passes fewer layer-runs (77% vs 93%). On O-RAN with held-out loss (ten replays), the loss baselines stop at 95% (transferred threshold) and 58% (patience), against the rule's 36%, for 0.7 and 0.4 points worst-task shortfall lower than the rule's.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.