- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LLARA, a novel parameter-efficient adaptation method that integrates low-rank techniques into attention residuals, enabling efficient cross-layer information aggregation during post-training. While the proposed approach is interesting and promising, the experimental evaluation does not sufficiently validate the effectiveness and necessity of its core design.
Critical strengths
-
The paper enables the efficient use of attention residuals during post-training without costly pretraining from scratch.
-
The paper provides substantial theoretical analysis to motivate the proposed method and support its architectural design.
Critical weaknesses
-
The experiments mainly compare LLARA with standard LoRA, making it unclear whether the performance gains stem from the additional trainable parameters or the proposed cross-layer residual aggregation mechanism. Parameter-matched baselines and targeted ablation studies are needed to establish the effectiveness of the proposed design.
-
Although the paper provides extensive theoretical analysis of depth-wise residual dilution, the experiments do not directly verify this phenomenon or demonstrate how LLARA mitigates it. This weakens the connection between the theoretical analysis and the empirical improvements.
Questions for authors
-
Can the authors provide parameter-matched comparisons or ablation studies to clarify whether the performance gains primarily arise from the additional trainable parameters or the proposed cross-layer residual aggregation mechanism?
-
Can the authors provide empirical evidence supporting the theoretical analysis of depth-wise residual dilution, particularly demonstrating whether this phenomenon affects model performance and how LLARA mitigates it?
3 comments
Thank you for your detailed responses and clarifications. I appreciate the authors' efforts in addressing my concerns. I will maintain my positive rating.
Official Comment by Authors
We are very pleased to receive your feedback and sincerely appreciate your recognition of our work! Once again, we would like to express our gratitude for the time and effort you have devoted to our manuscript.
Official Comment by Authors
Dear Reviewer wFUU,
We sincerely appreciate your time and expertise in evaluating our work. We particularly appreciate your recognition of our method as interesting and promising, and of the substantial theoretical analysis. We have carefully addressed your concerns below:
W1 & Q1: Capacity versus Cross-Layer Aggregation
A: We thank the reviewer for raising this point. To separate additional trainable parameters from the proposed cross-layer residual aggregation, we first compare LoRA baselines given slightly more trainable parameters than the corresponding LoRA+LLARA models. All comparisons use the same Qwen2.5-7B mathematical SFT protocol.
The results above show that increasing LoRA's parameter count does not improve accuracy. This finding is consistent with the experimental results in the original LoRA paper [1]. LoRA+LLARA improves over a LoRA baseline with slightly more trainable parameters, because LLARA's parameters do not directly modify the within-block transformation of each token's hidden state, as LoRA does, and instead learn token-dependent cross-layer composition of the realized residual increments through a low-rank linear attention residual mechanism.
Figure 7 in our manuscript shows that increasing LLARA's rank and parameter count also does not consistently improve accuracy.
W2 & Q2: Depth-Wise Residual Dilution
A: We thank the reviewer for raising this point. To test the predicted depth-wise dilution, we use a controlled signal-plus-noise model in which the signal lies in a single depth. We compare standard accumulation, uniform averaging, and a selective normalized readout that places mass 0.8 on the informative depth. Standard accumulation and uniform averaging have identical visibility.
The table shows the predicted contrast. From depth 16 to 1024, standard accumulation reduces directional visibility by about a factor of eight. The log-log slopes are −0.494 under independent noise and −0.507 under correlated noise, close to the predicted O(1/L) rate. The selective readout keeps mass 0.8 on the informative depth, and its visibility stays near 0.707 at every reported depth. Uniform averaging matches standard accumulation exactly, since rescaling cancels in the visibility ratio. The preserved visibility therefore comes from keeping constant mass on the informative depth, which is the condition that motivates LLARA's selective normalized correction.
We then track an early residual component in the trained Qwen2.5-7B checkpoint with LLARA rank 16 on held-out GSM8K and MATH prompts. In the unperturbed backbone, its visibility decreases from 0.9056 at block 8 to 0.0940 at block 28. Under added late-depth interference, retention compares the perturbed visibility with the clean visibility, and adaptive gain is the increase from recomputing the learned depth weights.
Under the same late-depth interference, the trained LLARA readout retains more of the probed early component than standard accumulation. This ordering holds for both GSM8K and MATH prompts. Recomputing the input-dependent weights further increases visibility over the clean-input weights, so the learned depth weights contribute to the protection when later residuals change. Together with the decay from block 8 to block 28, early residual components are diluted across depth in the trained backbone, and LLARA preserves a larger share of them under late-depth interference.
Reference
[1] Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022.
Once again, we deeply appreciate your valuable comments and thoughtful efforts. We sincerely hope our clarifications have addressed your concerns and look forward to your response.