- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes A-LoRA, a parameter-efficient adaptation method for the vocabulary-boundary layers of language models. A-LoRA leverages a frozen pretrained vocabulary matrix and a low-rank hidden-space transform, making the number of trainable parameters independent of vocabulary size. For models with tied input and output weights, A-LoRA further shares the same transform between the input embedding and output head. Experimental results show that A-LoRA outperforms W-form LoRA on two structured tasks.
Critical strengths
-
This paper investigates parameter-efficient adaptation of input embeddings and output heads, offering an alternative perspective on LoRA parameterization and placement.
-
The method is clearly designed. A-LoRA uses a low-rank hidden-space transform to reduce the number of trainable parameters for vocabulary matrices from (r(V+d)) to (2dr). For models with tied weights, the shared transform acts on both the input and output paths while preserving weight tying after merging.
Critical weaknesses
-
The paper uses pretrained vocabulary structure to constrain low-rank updates as an important motivation. However, the experiments lack a controlled ablation that isolates the effect of using pretrained vocabulary structure while keeping other settings unchanged. Therefore, it remains unclear whether A-LoRA's performance gains actually result from leveraging pretrained vocabulary structure.
-
The methodological novelty is relatively limited. The core parameterization of A-LoRA is highly similar in mathematical form to the existing LoRMA+ (Bihany et al., 2025). Although this paper applies the parameterization to vocabulary-boundary layers and investigates input–output sharing, these designs mainly extend existing methods. Therefore, the novelty of the core low-rank adaptation mechanism is relatively limited.
Questions for authors
Please conduct controlled experiments to verify whether leveraging pretrained vocabulary structure contributes to A-LoRA's performance improvements.
Please clarify the key methodological differences between A-LoRA and LoRMA+ (Bihany et al., 2025), and explain the novel contributions of A-LoRA beyond applying a similar low-rank parameterization to vocabulary-boundary layers and introducing input–output sharing.