- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the geometric asymmetry between input embeddings and output unembeddings in weight-tied language models. Based on geometric analyses of public models, the authors propose Asymmetric Geometric Decoupling (AGD), which introduces lightweight input-side bias and low-rank transformations while preserving the shared output matrix. Experiments show consistent perplexity improvements with minimal parameter overhead, including on a 0.6B model pretrained on 20B tokens. The improvements also persist after SFT.
Critical strengths
- Practical and efficient. The method is simple to implement and achieves consistent improvements with negligible additional parameters.
- Solid empirical evaluation. The paper includes extensive geometric analyses, controlled ablations, and multi-seed pretraining experiments, with improvements persisting after SFT.
Critical weaknesses
- Limited novelty. The method mainly consists of simple bias and low-rank transformations, while some of the geometric observations were already discussed in prior work. The distinct contribution beyond existing approaches remains somewhat unclear.
- Insufficient mechanistic validation. Although the geometric analyses are extensive, the paper does not directly demonstrate that AGD corrects the identified asymmetry or that such correction explains the performance improvements.
Questions for authors
-
Could the authors clarify how the geometric observations motivate AGD? In particular, why should input-side bias correction address the limitations identified in the analysis?
-
What is the key insight beyond the simple architectural modification? Since Input-Bias already achieves much of the improvement, a clearer discussion of the role of additional low-rank flexibility would strengthen the contribution.
I appreciate the simplicity and consistent empirical results, and am open to raising my rating if these concerns are convincingly addressed.