- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes -Align, a label-free metric that measures the alignment between representation variance and prediction-head sensitivity. The authors analyze its dynamics during pre-training and report strong negative correlations with downstream performance across several LLM families. However, the practical value of the metric and the reasons behind these correlations remain insufficiently established.
Critical strengths
S1. A meaningful extension of spectrum-based representation analysis. The paper identifies a genuine limitation of spectrum-only metrics: covariance eigenvalues do not capture the orientation of representation variance relative to the prediction head. The proposed metric offers a mathematically motivated way to characterize this relationship.
S2. Evaluation across training checkpoints and multiple model families. The experiments cover pre-training dynamics and fully trained models from several model families. The reported consistency of the negative association with downstream performance provides an interesting empirical observation that warrants further investigation.
Critical weaknesses
W1: The significance of the proposed metric is unclear. (\Phi)-Align measures the alignment between representation variance and prediction-head sensitivity, but this does not necessarily indicate whether the representations contain useful information for prediction. The paper does not sufficiently demonstrate what additional insights the metric provides beyond existing metrics and pre-training loss.
W2: The empirical evidence is insufficient to support the broader claims. The three-phase training trajectory is mainly associated with SciQ and LAMBADA, while the downstream correlation analysis involves only a few models per family. It remains unclear whether these findings generalize to other capabilities and unseen models.
W3: The paper needs substantial improvement in presentation and clarity. The abstract mainly lists model families, correlation coefficients, and experimental results without clearly conveying the research question and key findings. The paper also contains recurring grammatical errors, inconsistent punctuation around equations, and unusual spacing around section transitions, while some claims about capability emergence are stronger than the correlational evidence supports.
Questions for authors
Q1: Does (\Phi)-Align provide additional predictive value beyond model scale, pre-training loss, RankMe, and (\alpha)-ReQ when evaluated on unseen models?
Q2: Do the observed three-phase dynamics hold for a broader range of downstream tasks, beyond SciQ and LAMBADA?