acceptodds
Desk rejected at ICLR 2027

Rethinking LoRA Placement at the Vocabulary Boundary with A-LoRA

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-Rank Adaptation (LoRA) is typically applied to internal Transformer layers, while vocabulary-boundary layers such as input embeddings and output LM heads have long lacked systematic evaluation. We revisit this placement question and propose A-LoRA, a hidden-space parameterization that does not learn free residuals directly on vocabulary matrices, but instead induces vocabulary-matrix updates via a shared low-rank affine transform, so parameter count depends mainly on hidden dimension rather than vocabulary size . Analyzing 30 Base→Instruct model pairs, we find that output LM heads and tied shared matrices better match this affine structure, whereas untied input embeddings should not be treated symmetrically under the same assumption. Downstream LoRA post-training further shows that optimal vocabulary-boundary placement is conditional: A-LoRA as a standalone small-budget adapter is stronger on the input side; when combined with standard hidden LoRA, the `lm_head` side becomes a more stable low-cost supplement. Our results indicate that vocabulary-boundary layers should not be excluded from LoRA design space by default, but their topology should adapt to existing adaptation capacity rather than being added symmetrically to input and output sides unconditionally.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-Efficient Fine-Tuning, PEFT, LoRA, Vocabulary Boundary, Embedding Layer, Language Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_poWFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes A-LoRA, a parameter-efficient adaptation method for the vocabulary-boundary layers of language models. A-LoRA leverages a frozen pretrained vocabulary matrix and a low-rank hidden-space transform, making the number of trainable parameters independent of vocabulary size. For models with tied input and output weights, A-LoRA further shares the same transform between the input embedding and output head. Experimental results show that A-LoRA outperforms W-form LoRA on two structured tasks.

Critical strengths
  1. This paper investigates parameter-efficient adaptation of input embeddings and output heads, offering an alternative perspective on LoRA parameterization and placement.

  2. The method is clearly designed. A-LoRA uses a low-rank hidden-space transform to reduce the number of trainable parameters for vocabulary matrices from (r(V+d)) to (2dr). For models with tied weights, the shared transform acts on both the input and output paths while preserving weight tying after merging.

Critical weaknesses
  1. The paper uses pretrained vocabulary structure to constrain low-rank updates as an important motivation. However, the experiments lack a controlled ablation that isolates the effect of using pretrained vocabulary structure while keeping other settings unchanged. Therefore, it remains unclear whether A-LoRA's performance gains actually result from leveraging pretrained vocabulary structure.

  2. The methodological novelty is relatively limited. The core parameterization of A-LoRA is highly similar in mathematical form to the existing LoRMA+ (Bihany et al., 2025). Although this paper applies the parameterization to vocabulary-boundary layers and investigates input–output sharing, these designs mainly extend existing methods. Therefore, the novelty of the core low-rank adaptation mechanism is relatively limited.

Questions for authors

Please conduct controlled experiments to verify whether leveraging pretrained vocabulary structure contributes to A-LoRA's performance improvements.

Please clarify the key methodological differences between A-LoRA and LoRMA+ (Bihany et al., 2025), and explain the novel contributions of A-LoRA beyond applying a similar low-rank parameterization to vocabulary-boundary layers and introducing input–output sharing.

Reviewer_FkLFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces A-LoRA, which approximates the vocabulary matrix, tied or untied, into a frozen and low-rank component W = W0 (I + s PQ) where P, Q are low rank matrices and W0 is frozen but full-rank. The trainable parameters are not dependent on vocabulary dimension. The proposed method A-LoRA is compared against LoRA (W-form in the paper) in different ways. The paper trains Qwen3-0.6B on two tasks CLUENER and WikiSQL and see that A-LoRA performs better than W-form LoRA. And that the shared A-LoRA is better than the unshared one.

Critical strengths

The idea to make the vocabulary matrix low-rank factorization to be independent of vocabulary dimension is very interesting. The paper shows A-LoRA placement has a major effect on how the model adapts to a source task, shared E+U placement A-LoRA adapts ~10 times better than U placement of A-LoRA on one of the source tasks WikiSQL.

Critical weaknesses
  • One of the main concerns is with the clarity of the paper. The paper is hard to read with sections not being very clear on what the main claim is and how various metrics support it. For example, section 2, there are many notations without discussing them in the text. Section 3 introduces alignment metrics without enough background on why the specific formula is used. Experimental protocol and results mention different setup but it is hard to understand how they are connected. The paper needs adding clear descriptions on what each section is about, main takeaways and how results support the takeaways.

  • Experimental setup choices: There are design choices which need more explanation and reasoning on why they were chosen. For example a) Why A-LoRA is evaluated with rank 16 and not other rank with W-dense LoRA with rank 1? Where as near budget analysis uses rank of 19 for A-LoRA and rank of 4 for W-form. b) Why CLUENER and WikiSQL were used as tasks?

The core idea of the paper has merit as stated in the strength section, but it is hard to read with some of the important details missing.

Questions for authors

None

Reviewer_1E26OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies low-rank adaptation at input embeddings and output LM heads. A-LoRA applies a low-rank transform in hidden space, giving a trainable parameter count independent of vocabulary size. On tied models, the same transform can adapt both boundaries while preserving a mergeable shared matrix. The paper combines weight-space analysis of 30 base/instruction-tuned model pairs with a five-seed placement study on Qwen3-0.6B, supplementary experiments, and an analysis of retained generation behavior.

Critical strengths
  1. The hidden-space parameterization is simple and provides a vocabulary-independent parameter budget. Sharing one transform across tied input and output boundaries preserves a useful deployment property: the adapter can be merged into a single tied matrix.

  2. The shared-versus-input-only comparison controls parameter count, rank, and selected learning rate. The five-seed results, supplied prediction records, and analysis of generation retention provide useful evidence within the evaluated setting.

Critical weaknesses
  1. Limited distinctive novelty. Embedding adaptation is already supported by the official LoRA implementation. Parameter sharing and restrictions on low-rank factors have also been explored by Tied-LoRA, VeRA, and Uni-LoRA, although their parameterizations differ. More directly, A-LoRA's multiplicative form is closely related to LoRMA, which also examines post-multiplication. I acknowledge the discussion in Section 6 and Appendix A.13 and the paper's specific focus on vocabulary matrices and shared input–output adaptation. Nevertheless, I am not convinced that combining these ideas establishes a sufficiently distinctive methodological contribution, particularly without systematic comparisons to the closest alternatives.

  2. Insufficient experimental support. The main controlled study is limited to Qwen3-0.6B, two tasks, and 64 training steps. The appendix contains broader historical experiments, but these do not establish the shared E+U advantage under comparably controlled and tuned protocols at larger scales or longer training horizons. Systematic comparisons with compressed PEFT, multiplicative adaptation, and hidden-layer LoRA at matched parameter budgets are missing. The main A-LoRA versus vocabulary-LoRA comparisons differ in rank, parameter count, and initialization; the near-budget comparison between these parameterizations covers only one task and input-only placement. This limitation is distinct from the well-controlled shared-versus-input-only A-LoRA comparison. The observed tradeoff between source-task performance and retained generation also calls for comparisons at matched source-task quality. Overall, the current evidence does not compensate for the limited methodological novelty or establish broad effectiveness, which motivates my clear-rejection recommendation.

Questions for authors
  1. What is the precise methodological advance beyond applying multiplicative low-rank adaptation to vocabulary matrices and sharing the transform across tied input and output layers? How does it differ from post-multiplicative LoRMA+, beyond placement and parameter-budget choices?

  2. Under matched trainable-parameter budgets and comparable hyperparameter tuning, how does A-LoRA compare with representative compressed PEFT methods, LoRMA, and hidden-layer LoRA? Can such comparisons establish an advantage beyond the single near-budget input-only control?

  3. Does the shared E+U advantage persist on larger models, additional tasks, and longer training runs? In the existing results, how do retention conclusions change when methods are compared at similar source-task performance?

These questions target the distinctiveness and empirical scope of the contribution, which are the main factors in my recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.