acceptodds
Under review as a conference paper at ICLR 2027

Efficient Untying of Weight-Tied Language Models via Asymmetric Geometric Decoupling

Abstract

Weight tying reduces vocabulary-layer parameters, but forces input embeddings and output unembeddings to share one vocabulary geometry. Modern language models increasingly untie these matrices at larger scales, raising the question of whether parameter-constrained models can relax tying without paying the full cost of an independent vocabulary matrix. We diagnose this trade-off through training dynamics and embedding geometry across public model families. Tied shared matrices are consistently closer to untied output unembeddings than to input embeddings; output spaces also show stronger cross-model cosine alignment and stronger mean/rank-1 spectral structure. These patterns match dense softmax gradients on the output side versus sparse lookup gradients on the input side, suggesting that hard tying makes the shared matrix output-dominated and constrains independent input-side geometry. We propose Asymmetric Geometric Decoupling (AGD), which keeps a shared vocabulary matrix on the output side while adding small structured freedom to the input side through bias correction and hidden-dimensional low-rank deformation. Input-side AGD improves tied baselines in MiniMind and FineWeb-Edu pretraining, including a 0.6B configuration trained from scratch on 20B tokens, with minimal parameter overhead. Its gains persist in held-out assistant-token likelihood after full-parameter SmolTalk fine-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Weight Tying, Language Modeling, Input Embeddings, Output Unembeddings, Embedding-Head Decoupling, Parameter-Efficient Training, Asymmetric Geometric Decoupling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ppBSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the geometric asymmetry between input embeddings and output unembeddings in weight-tied language models. Based on geometric analyses of public models, the authors propose Asymmetric Geometric Decoupling (AGD), which introduces lightweight input-side bias and low-rank transformations while preserving the shared output matrix. Experiments show consistent perplexity improvements with minimal parameter overhead, including on a 0.6B model pretrained on 20B tokens. The improvements also persist after SFT.

Critical strengths
  1. Practical and efficient. The method is simple to implement and achieves consistent improvements with negligible additional parameters.
  2. Solid empirical evaluation. The paper includes extensive geometric analyses, controlled ablations, and multi-seed pretraining experiments, with improvements persisting after SFT.
Critical weaknesses
  1. Limited novelty. The method mainly consists of simple bias and low-rank transformations, while some of the geometric observations were already discussed in prior work. The distinct contribution beyond existing approaches remains somewhat unclear.
  2. Insufficient mechanistic validation. Although the geometric analyses are extensive, the paper does not directly demonstrate that AGD corrects the identified asymmetry or that such correction explains the performance improvements.
Questions for authors
  1. Could the authors clarify how the geometric observations motivate AGD? In particular, why should input-side bias correction address the limitations identified in the analysis?

  2. What is the key insight beyond the simple architectural modification? Since Input-Bias already achieves much of the improvement, a clearer discussion of the role of additional low-rank flexibility would strengthen the contribution.

I appreciate the simplicity and consistent empirical results, and am open to raising my rating if these concerns are convincingly addressed.

Reviewer_CLRvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates weight tying (WT) dynamics, showing across 82 models that tied matrices mirror output unembeddings rather than input embeddings due to dense softmax versus sparse lookup gradient dynamics. To decouple representations cheaply, the authors propose Asymmetric Geometric Decoupling (AGD), using lightweight input-side transformations (e.g., input bias or low-rank affine adapters). Experiments show modest, consistent perplexity gains over tied baselines.

Critical strengths

Extensive Empirical Diagnostic: Compelling multi-model analysis across 82 checkpoints confirming that tied matrices consistently collapse to output geometry.

Grounded Literature: Thorough contextualization within existing work on representational capacity and vocabulary anisotropy.

Practical Scope: Directly addresses parameter efficiency for small-to-medium LLMs where vocabulary parameters dominate.

Methodical Validation: Reliable multi-seed evaluation across 6B and 20B token regimes.

Critical weaknesses

Marginal Gains: Absolute perplexity reductions are small (≈0.13\approx 0.13–0.200.20 PPL), with no validation on downstream benchmarks to confirm real-world impact.

Ad-Hoc Solution: The spectral analysis shows divergence is concentrated in the mean axis / rank-1 energy, making the proposed rank-32 hidden deformation feel empirically patched rather than theoretically motivated.

Surface-Level Theory: The gradient update disparity (dense softmax vs. sparse lookup) remains an intuitive, qualitative argument lacking formal theoretical characterization.

Questions for authors

Spectral Justification: If the geometric divergence is primarily a rank-1/mean-axis phenomenon, what motivates a rank-32 deformation over pure centroid shifts or diagonal scalings?

Downstream Utility: Do the modest perplexity improvements translate into measurable gains on standard downstream tasks?

Mechanistic Depth: Can you formally characterize how output head gradients actively degrade or constrain input representations during joint training?

Reviewer_j4XvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies that in tied-embedding language models, the shared matrix becomes heavily `output-dominated', which limits the input representation freedom. However, fully untying is expensive particularly for small models in which the vocabulary matrix accounts for a significant portion of entire model size. Instead, based on the diagnosis, the work proposes to still share the matrix but allow a controlled freedom for the input side (AGD). Empirical results show AGD seems effective.

Critical strengths

S1. Paper is easy and simple to follow (simple is a plus) S2. The solution is based on practical discovery S3. The solution looks effective from the experimental study

Critical weaknesses
  1. The discovery that the shared matrix is closer to the output is not not entirely surprising, and also discussed in Lopardo, Antonio, et al. "Weight Tying Biases Token Embeddings Towards the Output Space". ACL 2026.

  2. The exact benefit of AGD can be strengthened. (a) What's the implication of adopting AGD? on training throughput, inference overhead for instances. (b) Currently the evaluation used held-out cross-entropy. It would be clearer to see the benefit of AGD if we can see its impact on some downstream tasks directly.

  3. It would be interesting to study the limitation of input side AGD variants w.r.t. how far they can approximate fully untying.

Questions for authors

The exact source of improvement of AGD w.r.t. weight tying. Would it come from different tuning, optimization due to the extra parameterization across different model setups, or due to the extra freedom of the input side?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.