- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies spectral concentration in language-model output heads and analyzes its relationship with hidden dimension and vocabulary size. The paper proposes controlling spectral concentration through Spectral Skew Regularization (SSR), which applies a gated penalty. The experiments show that SSR can reduce spectral skew. However, its effectiveness depends on the specific model and dataset, and reducing spectral skew alone does not consistently improve predictive performance.
Critical strengths
-
The paper provides several controlled comparisons and attempts to distinguish reductions in spectral skew from actual improvements in predictive performance.
-
SSR is a simple regularization method with a reasonably motivated design, and it is evaluated across multiple model scales.
Critical weaknesses
-
The paper is written in an extremely obscure and difficult-to-follow manner. Even researchers working at the forefront of matrix decomposition and spectral analysis may find it difficult to understand what the authors are trying to communicate.
-
Even after reading the entire paper, I could not identify what serious problems are caused by spectral concentration or spectral skew. In the experiments, reducing spectral skew does not necessarily improve predictive performance. The authors should define the harmful consequences of this form of spectral collapse more clearly, design experiments that directly demonstrate these consequences and their mitigation, and explain the resulting observations more clearly. I believe the paper would benefit substantially from doing so.
Questions for authors
See the weakness. Overall, I do not find this to be a paper from which human readers can easily understand the main ideas or learn useful insights.