acceptodds
Desk rejected at ICLR 2027

LLARA: Low-Rank Linear Attention Residual Adaptation for Large Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern large language models (LLMs) rely critically on residual connections with PreNorm, yet their uniform aggregation of layer outputs with fixed unit weights causes hidden states to expand uncontrollably with depth, gradually diluting the contribution of each layer. Recent research introduces Attention Residuals (AttnRes) to enable each layer to selectively aggregate earlier representations on a per-token basis through learned weights, by replacing the fixed accumulation with softmax attention over preceding layer outputs, yielding consistent empirical gains across model scales. However, pre-training AttnRes-based LLMs demands prohibitively high computational resources and expenses, preventing a wide range of open-source models from reaping the benefits of this novel architecture. To bridge this gap, we propose **L**ow-Rank **L**inear **A**ttention **R**esidual **A**daptation, or **LLARA**, which freezes the pre-trained model weights and introduces a learnable low-rank linear attention residual mechanism to fine-tune depth-wise aggregation of per-block residual contributions, allowing pre-trained LLMs to robustly benefit from token-wise attention residuals during downstream task adaptation. Jointly with LoRA for parameter-efficient fine-tuning, LLARA maintains a compact linear attention state along the decoder depth through low-rank key–value trajectories, reads it out at the final block via a learnable query, and adds the resulting residual back to the hidden representations, thereby fine-tuning the model’s residual architecture. Both theoretical analysis and empirical experiments have demonstrated the effectiveness, efficiency, and robustness of LLARA. Our code is available at https://anonymous.4open.science/r/LLARA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Model Architecture Fine-Tuning, Low-Rank Linear Attention Residuals

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wFUUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LLARA, a novel parameter-efficient adaptation method that integrates low-rank techniques into attention residuals, enabling efficient cross-layer information aggregation during post-training. While the proposed approach is interesting and promising, the experimental evaluation does not sufficiently validate the effectiveness and necessity of its core design.

Critical strengths
  1. The paper enables the efficient use of attention residuals during post-training without costly pretraining from scratch.

  2. The paper provides substantial theoretical analysis to motivate the proposed method and support its architectural design.

Critical weaknesses
  1. The experiments mainly compare LLARA with standard LoRA, making it unclear whether the performance gains stem from the additional trainable parameters or the proposed cross-layer residual aggregation mechanism. Parameter-matched baselines and targeted ablation studies are needed to establish the effectiveness of the proposed design.

  2. Although the paper provides extensive theoretical analysis of depth-wise residual dilution, the experiments do not directly verify this phenomenon or demonstrate how LLARA mitigates it. This weakens the connection between the theoretical analysis and the empirical improvements.

Questions for authors
  1. Can the authors provide parameter-matched comparisons or ablation studies to clarify whether the performance gains primarily arise from the additional trainable parameters or the proposed cross-layer residual aggregation mechanism?

  2. Can the authors provide empirical evidence supporting the theoretical analysis of depth-wise residual dilution, particularly demonstrating whether this phenomenon affects model performance and how LLARA mitigates it?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer wFUU,

We sincerely appreciate your time and expertise in evaluating our work. We particularly appreciate your recognition of our method as interesting and promising, and of the substantial theoretical analysis. We have carefully addressed your concerns below:

W1 & Q1: Capacity versus Cross-Layer Aggregation

A: We thank the reviewer for raising this point. To separate additional trainable parameters from the proposed cross-layer residual aggregation, we first compare LoRA baselines given slightly more trainable parameters than the corresponding LoRA+LLARA models. All comparisons use the same Qwen2.5-7B mathematical SFT protocol.

MethodrLoRAr_{\mathrm{LoRA}}rLLARAr_{\mathrm{LLARA}}Trainable paramsGSM8K (%)MATH (%)
LoRA11—3.47M82.347.5
LoRA + LLARA883.35M84.049.3
LoRA14—4.42M81.747.1
LoRA + LLARA8164.19M83.749.1
LoRA19—5.99M82.147.5
LoRA + LLARA8325.85M84.349.9

The results above show that increasing LoRA's parameter count does not improve accuracy. This finding is consistent with the experimental results in the original LoRA paper [1]. LoRA+LLARA improves over a LoRA baseline with slightly more trainable parameters, because LLARA's parameters do not directly modify the within-block transformation of each token's hidden state, as LoRA does, and instead learn token-dependent cross-layer composition of the realized residual increments through a low-rank linear attention residual mechanism.

Figure 7 in our manuscript shows that increasing LLARA's rank and parameter count also does not consistently improve accuracy.

W2 & Q2: Depth-Wise Residual Dilution

A: We thank the reviewer for raising this point. To test the predicted depth-wise dilution, we use a controlled signal-plus-noise model in which the signal lies in a single depth. We compare standard accumulation, uniform averaging, and a selective normalized readout that places mass 0.8 on the informative depth. Standard accumulation and uniform averaging have identical visibility.

DepthIndependent: standard/uniformIndependent: selectiveCorrelated: standard/uniformCorrelated: selective
160.24260.70620.18630.7031
640.12400.70670.09070.7061
2560.06240.70680.04500.7068
10240.03120.70690.02250.7070

The table shows the predicted contrast. From depth 16 to 1024, standard accumulation reduces directional visibility by about a factor of eight. The log-log slopes are −0.494-0.494 under independent noise and −0.507-0.507 under correlated noise, close to the predicted O(1/L)O(1/\sqrt{L}) rate. The selective readout keeps mass 0.8 on the informative depth, and its visibility stays near 0.707 at every reported depth. Uniform averaging matches standard accumulation exactly, since rescaling cancels in the visibility ratio. The preserved visibility therefore comes from keeping constant mass on the informative depth, which is the condition that motivates LLARA's selective normalized correction.

We then track an early residual component in the trained Qwen2.5-7B checkpoint with LLARA rank 16 on held-out GSM8K and MATH prompts. In the unperturbed backbone, its visibility decreases from 0.9056 at block 8 to 0.0940 at block 28. Under added late-depth interference, retention compares the perturbed visibility with the clean visibility, and adaptive gain is the increase from recomputing the learned depth weights.

Prompt sourceStandard stream retained (%)LLARA readout retained (%)Adaptive gain (%)
GSM8K68.3283.65+14.07
MATH69.2980.93+11.66

Under the same late-depth interference, the trained LLARA readout retains more of the probed early component than standard accumulation. This ordering holds for both GSM8K and MATH prompts. Recomputing the input-dependent weights further increases visibility over the clean-input weights, so the learned depth weights contribute to the protection when later residuals change. Together with the decay from block 8 to block 28, early residual components are diluted across depth in the trained backbone, and LLARA preserves a larger share of them under late-depth interference.

Reference

[1] Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022.

Once again, we deeply appreciate your valuable comments and thoughtful efforts. We sincerely hope our clarifications have addressed your concerns and look forward to your response.

Thank you for your detailed responses and clarifications. I appreciate the authors' efforts in addressing my concerns. I will maintain my positive rating.

AuthorsOpenReview ↗

Official Comment by Authors

We are very pleased to receive your feedback and sincerely appreciate your recognition of our work! Once again, we would like to express our gratitude for the time and effort you have devoted to our manuscript.

Reviewer_4EXYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LLARA, which accumulates block-level residual increments of a frozen backbone via low-rank linear attention over depth and adds a single correction to the final hidden state, trained jointly with LoRA. The authors claim effectiveness, efficiency, and robustness.

Critical strengths
  1. Easy to adopt in existing PEFT pipelines since the backbone and block interface are untouched.
  2. Small parameter and memory overhead.
  3. Consistent gains across backbones, scales, and training methods (SFT/RL).
Critical weaknesses
  1. Missing ablations: eg. a static layer-wise scalar mix. Without them, it’s unclear whether gains come from token-wise depth mixing or just extra capacity.

  2. The abstract claims robustness, but it’s unclear what this means and no experiment tests it.

  3. The adapter cannot be merged (App. F.3), so inference latency matters, but it is not reported.

  4. The in-place AttnRes failure may be due to initialization rather than gradient structure.

Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 4EXY,

Thank you very much for your careful review and helpful suggestions. We particularly appreciate your recognition of the compatibility with PEFT pipelines of our method, the parameter efficiency, and the extensive experiments. We have carefully addressed your concerns below:

W1: Layer-Wise Scalar Mix Ablation

A: We thank the reviewer for suggesting this ablation. We replace LLARA's token-dependent weights with a learned, input-independent layer-wise scalar mix over the same block residuals, Δht=∑l=0Lsoftmax(c)lvl,t\Delta h_t=\sum_{l=0}^{L}\mathrm{softmax}(c)_l v_{l,t}. The vector c∈RL+1c\in\mathbb{R}^{L+1} is optimized jointly with LoRA and shared across all tokens and inputs. This baseline retains the additive output correction and uses input-independent depth weights. All experiments use the same Qwen2.5-7B mathematical SFT protocol. We evaluate the static mix at the original LoRA rank and at a higher rank that gives it slightly more trainable parameters than LoRA + LLARA.

MethodrLoRAr_{\mathrm{LoRA}}Trainable paramsGSM8K (%)MATH (%)
LoRA82.52M82.047.4
LoRA + input-independent residual weights82.52M81.647.5
LoRA + input-independent residual weights144.42M82.247.1
LoRA + LLARA (r=16r=16)84.19M83.749.1

These results support the benefit of token-dependent weighting over learned input-independent weights, even at a comparable parameter budget. Our experiments find that the learned softmax(c)\mathrm{softmax}(c) weights remain relatively uniform across depths, whereas LLARA exhibits more pronounced preferences for particular layers.

W2: The Robustness Claim

A: We thank the reviewer for pointing out this ambiguity. To eliminate this ambiguity, we have replaced “robustness” with “stability” in the abstract to clarify the scope of the claim. Specifically, we refer to the observed stability of LLARA during adaptation of a pretrained residual backbone, as evaluated against in-place AttnRes.

This claim is supported by the training-dynamics experiments in Section 4.4 and informed by the gradient analysis in Section 3.5. The analysis shows that in-place AttnRes introduces trainable queries within the depth recurrence, so their gradients can involve products of downstream block Jacobians. LLARA preserves the pretrained recurrence and applies a single readout after the final block, avoiding this source of depth-dependent gradient amplification for its architecture parameters. In Qwen2.5-7B mathematical SFT, in-place AttnRes exhibits substantially higher training loss and gradient norms than LLARA, indicating unstable optimization under the evaluated setting. LLARA maintains a loss trajectory close to the matched LoRA baseline, and its gradient norm approaches the baseline level. These observations support the specific claim of stability.

W3: Inference Latency

A: We thank the reviewer for raising this point. We have added a description of the resulting execution path in Appendix F.3 to improve inference efficiency. Prefill keeps the existing dynamic cache, the completed key-value state is copied once into a fixed buffer, and each decoding step is replayed from a CUDA graph. Inputs without padding use Flash Attention on the valid key-value length.

We measure inference cost on GSM8K for the Qwen2.5-7B mathematical checkpoint with LLARA rank 16 and LoRA rank 8. The three models are the original backbone, the same checkpoint with LoRA merged and LLARA removed, and that merged backbone with LLARA attached. All three use the same execution configuration. Each setting generates 128 tokens per sequence with end-of-sequence stopping disabled. Latency is the median of five CUDA-synchronized runs after two warmups. It includes prefill, the key-value copy or reset, and the full decode, and it excludes model loading, tokenization, input transfer, and the one-time CUDA-graph preparation.

BatchInputModelLatency (s)Tokens/sMemory (GiB)Prep. (s)
1256Qwen2.5-7B1.85668.9814.270.683
1256LoRA merged1.86368.6914.300.710
1256LoRA merged + LLARA1.89367.6214.330.743
11024Qwen2.5-7B1.92066.6614.470.958
11024LoRA merged1.91566.8414.490.975
11024LoRA merged + LLARA1.94765.7314.531.057
8256Qwen2.5-7B2.168472.2714.771.388
8256LoRA merged2.171471.6714.791.454
8256LoRA merged + LLARA2.202465.0014.821.469

Relative to the original model, median latency is higher by at most 2.0%. Relative to the LoRA-merged, the increase is between 1.4% and 1.7%. Peak allocated memory stays within 0.06 GiB of the original model. The attached LLARA readout therefore does not add a noticeable inference latency or computational cost.

AuthorsOpenReview ↗

Official Comment by Authors

W4: Cause of the In-Place AttnRes Failure

A: We thank the reviewer for raising this question. We have added Appendix H to analyze the structural mismatch underlying the observed in-place AttnRes failure. Specifically, we clarify the initialization used in our experiments, explain why changing the query initialization alone cannot generally recover the pretrained residual recurrence, and connect this distinction to the gradient analysis and observed training dynamics.

In our in-place AttnRes comparison, every layer query is initialized to the zero vector, following the initialization requirement in Attention Residuals [1]: "Crucially, all pseudo-query vectors must be initialized to zero." This yields uniform initial attention weights across source layers, without imposing an initial preference for any particular source.

However, uniform attention weights do not recover standard residual accumulation. To make this distinction explicit, consider the two aggregation operators applied to the same collection of residual values Vl=(v0,…,vl)V_l=(v_0,\ldots,v_l). Standard residual accumulation assigns each contribution a coefficient of one, giving Sl(Vl)=∑i=0lviS_l(V_l)=\sum_{i=0}^{l}v_i. Our in-place AttnRes instead uses the normalized aggregate Al(ql,Vl)=∑i=0lαi→l(ql,Vl)viA_l(q_l,V_l)=\sum_{i=0}^{l}\alpha_{i\to l}(q_l,V_l)v_i, where αi→l=exp⁡(ql⊤k~i)/∑j=0lexp⁡(ql⊤k~j)\alpha_{i\to l}=\exp(q_l^\top\widetilde{k}_i)/\sum_{j=0}^{l}\exp(q_l^\top\widetilde{k}_j) and k~i=RMSNorm(vi)\widetilde{k}_i=\mathrm{RMSNorm}(v_i).

At zero initialization, ql=0q_l=0 yields αi→l=1/(l+1)\alpha_{i\to l}=1/(l+1) and consequently Al(0,Vl)=Sl(Vl)/(l+1)A_l(0,V_l)=S_l(V_l)/(l+1). Thus, zero initialization produces an equal-weight average, rather than the unit-weight sum used by the pretrained residual recurrence.

Changing the query initialization alone cannot generally eliminate this operator-level difference while retaining the same aggregation rule. For any query, the softmax weights sum to one, whereas the coefficients in standard residual accumulation sum to l+1l+1. For example, if all source values equal a nonzero vector uu, then, for any query and l≥1l\geq1, we obtain Al(ql,(u,…,u))=uA_l(q_l,(u,\ldots,u))=u but Sl((u,…,u))=(l+1)uS_l((u,\ldots,u))=(l+1)u. Therefore, no query initialization can make the two aggregation operators identical for arbitrary residual values. A nonzero query changes the relative weighting of the sources but does not remove the normalization constraint.

Reference

[1]Kimi Team. Attention Residuals. arXiv preprint arXiv:2603.15031, 2026.

Once again, we deeply appreciate your valuable comments and thoughtful efforts. We sincerely hope our clarifications have addressed your concerns and look forward to your response.

Thank you for the detailed response and new experiments. My main concern (W1) is addressed, so I am raising my rating to weak acceptance.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your encouraging feedback! We are pleased to hear that our rebuttal has effectively addressed your concerns. Once again, we would like to express our gratitude for the time and effort you have devoted to our manuscript.

Reviewer_E4iqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LLARA, a parameter-efficient method that learns a token-dependent aggregation of block residuals and adds the result to the final hidden state. It preserves the pretrained block recurrence and is trained jointly with LoRA. Experiments report consistent improvements over LoRA across several SFT tasks, model families, and RL settings.

Critical strengths
  1. The approach addresses a useful adaptation setting. Adding a learned readout over intermediate residuals is a sensible way to complement LoRA while preserving the pretrained block interfaces.

  2. The results are consistent across tasks and backbones. The improvements on mathematical SFT and under both PPO and GRPO are encouraging. The appendix also provides useful ablations and measured training costs.

Critical weaknesses
  1. The zero initialization raises a concern about the effective rank. Both the key projections and the query are initialized to zero. Under the stated operations, the projection rows appear to remain identical throughout training, making the readout effectively equivalent to a rank-one scalar gate, apart from the numerical stabilizer. I could not identify what breaks this symmetry. This needs clarification, particularly for interpreting the rank ablation.

  2. The baselines do not isolate the benefit of the proposed construction. The default configuration trains 4.19M parameters versus 2.52M for LoRA (Table 5). A parameter-matched LoRA baseline and a simple output-only depth-pooling baseline would help establish whether the gains require the proposed linear-attention mechanism.

  3. The application of Theorem 8 needs additional justification. Eq. (41) takes the aggregation weights outside the expectation, which is valid for fixed weights. LLARA’s weights depend on the residuals themselves and may therefore correlate with the noise. The stated assumptions do not appear sufficient to extend this result to the learned readout.

Questions for authors
  1. Does the implementation include any symmetry-breaking initialization or operation? How does (r=1) perform?
  2. How important is token-dependent weighting for the observed gains? Would a simple set of learned, input-independent weights over block residuals achieve similar improvements?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer E4iq,

We sincerely appreciate your time and expertise in evaluating our work. We particularly appreciate your recognition of the usefulness of our method, the clear presentation, and the encouraging experimental results. Your concerns have been carefully addressed below:

W1 & Q1: Clarification of Initialization and Effective Rank

A: We thank the reviewer for raising this important point. We have revised the initialization description and the corresponding theoretical analysis to align with the actual implementation used in our experiments. In the initial manuscript, the theoretical analysis assumed zero initialization to make the analysis of LLARA more tractable. In our implementation and experiments, however, the query qq is initialized to zero, whereas each projection matrix Al∈Rr×dA_l\in\mathbb{R}^{r\times d} is initialized with independent Gaussian entries, (Al)ij∼N(0,σ2/d)(A_l)_{ij}\sim\mathcal N(0,\sigma^2/d), where dd is the hidden dimension and σ=0.1\sigma=0.1 by default. This independent Gaussian initialization breaks row symmetry, so the effective rank does not collapse to one in the actual implementation.

Your comment helped us recognize the importance of measuring effective rank. Keeping all other experimental settings the same as in the main experiments, we fix the LLARA rank at r=16r=16 and evaluate two initializations over σ∈{0.03,0.1,1}\sigma\in \lbrace 0.03,0.1,1\rbrace: Gaussian initialization, nn.init.normal_(A, mean=0.0, std=σ / math.sqrt(d)), and full-rank orthogonal initialization, nn.init.orthogonal_(A, gain=σ). We report task accuracy and the effective rank of the learned LLARA projection matrices after SFT of Qwen2.5-7B on math and code tasks. For each projection matrix AlA_l, let sl,is_{l,i} denote its singular values and pl,i=sl,i∑j=1rsl,jp_{l,i}=\frac{s_{l,i}}{\sum_{j=1}^{r}s_{l,j}} their normalized values. The effective rank is computed as reff(Al)=exp⁡(−∑i=1rpl,iln⁡pl,i)r_{\mathrm{eff}}(A_l)=\exp\left(-\sum_{i=1}^{r}p_{l,i}\ln p_{l,i}\right) [1], with the convention 0ln⁡0=00\ln 0=0. For each task, we report the mean effective rank across the 29 LLARA projection matrices.

Initializationσ\sigmaGSM8K (%)MATH (%)Math eff. rankHumanEval (%)MBPP (%)Code eff. rank
Gaussian0.0383.449.02.6258.564.82.94
Gaussian0.183.749.13.9358.765.26.89
Gaussian183.648.411.7059.864.615.61
Orthogonal0.0383.349.33.6459.365.02.94
Orthogonal0.183.949.63.9558.564.96.89
Orthogonal184.049.111.4660.464.815.62

The results above show that effective rank varies substantially across initialization scales, but higher rank does not consistently improve accuracy. Gaussian and orthogonal initialization yield similar performance at matched scales. Under the same initialization method and scale, the effective rank after training differs markedly across different tasks.

We also fix the initialization to the Gaussian scheme with σ=0.1\sigma=0.1, and report GSM8K and MATH accuracy together with the effective rank for r∈{1,4,8,16,32,64}r\in \lbrace1,4,8,16,32,64 \rbrace.

rrGSM8K (%)MATH (%)Effective rank
180.946.61.00
483.248.81.74
884.049.32.57
1683.749.13.93
3284.349.96.28
6484.049.611.19

These results show that r=1r=1 performs substantially worse (even worse than LoRA baseline), while for larger rr the effective rank after training remains far below the nominal rank even though each projection matrix is full rank at initialization. We have added a theoretical analysis in Appendix G.5 to explain the degeneration of the LLARA mechanism at r=1r=1.An effective rank below the nominal rank is not unique to LLARA: prior work [2] also reports LoRA updates with effective ranks substantially below their configured ranks, depending on the training setup.

AuthorsOpenReview ↗

Official Comment by Authors

W2 & Q2: Parameter Budget and Token-Dependent Weighting

A: Thank you for these helpful suggestions. We have added experiments to answer two questions and address your concerns: (1) whether the gains come from additional parameters, and (2) whether token-dependent weighting improves over learned, input-independent weighting. All comparisons use the same Qwen2.5-7B mathematical SFT protocol.

(1) Comparable parameter budgets. We compare three budget levels by increasing rLoRAr_{\mathrm{LoRA}}, giving each LoRA baseline slightly more parameters than its LLARA counterpart.

MethodrLoRAr_{\mathrm{LoRA}}rLLARAr_{\mathrm{LLARA}}Trainable paramsGSM8K (%)MATH (%)
LoRA11—3.47M82.347.5
LoRA + LLARA883.35M84.049.3
LoRA14—4.42M81.747.1
LoRA + LLARA8164.19M83.749.1
LoRA19—5.99M82.147.5
LoRA + LLARA8325.85M84.349.9

The results above show that increasing LoRA's parameter count does not improve accuracy. This finding is consistent with the experimental results in the original LoRA paper [3]. LoRA+LLARA improves over a LoRA baseline with slightly more trainable parameters, because LLARA's parameters do not directly modify the within-block transformation of each token's hidden state, as LoRA does, and instead learn token-dependent cross-block composition of the realized residual increments through a low-rank linear attention residual mechanism.

Figure 7 in our manuscript shows that increasing LLARA's rank and parameter count also does not consistently improve accuracy.

(2) LLARA versus input-independent weighting. We replace LLARA's token-dependent weights with learned, input-independent weights over the same block residuals, Δht=∑l=0Lsoftmax(c)lvl,t\Delta h_t=\sum_{l=0}^{L}\mathrm{softmax}(c)_l v_{l,t}. The vector cc is optimized during training and shared across all tokens. We evaluate this baseline with both the original LoRA rank and a comparable total parameter budget.

MethodrLoRAr_{\mathrm{LoRA}}Trainable paramsGSM8K (%)MATH (%)
LoRA82.52M82.047.4
LoRA + input-independent residual weights82.52M81.647.5
LoRA + input-independent residual weights144.42M82.247.1
LoRA + LLARA (r=16r=16)84.19M83.749.1

These results support the benefit of token-dependent weighting over learned input-independent weights, even at a comparable parameter budget. The learned softmax(c)\mathrm{softmax}(c) weights remain relatively uniform across depths, whereas LLARA exhibits more pronounced preferences for particular layers.

W3: Applicability of Theorem 8 to Residual-Dependent Weights

A: We thank the reviewer for identifying this gap. To address this issue, we have revised Theorem 8 in Appendix C to explicitly restrict its scope to deterministic weights and added Theorem 9 for nonnegative weights that sum to at most one, including weights that depend on the residuals. Specifically, under the same signal-plus-noise assumption, Theorem 9 decomposes each weight into its expectation and a fluctuation. The mean weights yield the fixed-weight signal and noise bounds, while an additional remainder accounts for the interaction between weight fluctuations and noise. This remainder reduces the signal lower bound and increases the noise upper bound. These theoretical results aim to provide insight into attention residuals and method design under reasonable assumptions.

Reference

[1] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In Proceedings of the 15th European Signal Processing Conference (EUSIPCO), 2007.

[2] Shuttleworth et al., LoRA vs Full Fine-tuning: An Illusion of Equivalence. In NeurIPS, 2025

[3] Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022.

Once again, we deeply appreciate your valuable comments and thoughtful efforts. We sincerely hope our clarifications have addressed your concerns and look forward to your response.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer E4iq,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of our paper. With the discussion period closing in three days, we would appreciate any further feedback on whether our response addresses your concerns. We would be happy to provide further clarification.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript. We are especially grateful for the reviewers' recognition of the usefulness and PEFT compatibility of LLARA, its parameter efficiency and encouraging experimental results, and the supporting theoretical analysis. For clarity and convenience, we summarize below the key strengths highlighted by the reviewers, followed by our corresponding revisions and responses to their concerns.

*Reviewer IDs in square brackets indicate the reviewers associated with each point.

Key strengths noted by the reviewers:

S1: Interface-Preserving Adaptation [E4iq, 4EXY] A learned readout over intermediate residuals complements LoRA while the pretrained backbone and block interface remain unchanged, and the method is easy to adopt in existing PEFT pipelines.

S2: Consistent Empirical Gains [E4iq, 4EXY] The improvements are consistent across tasks, backbones, and scales, including mathematical SFT and both supervised fine-tuning and reinforcement learning with PPO and GRPO.

S3: Efficiency and Theoretical Support [4EXY, wFUU] The added parameter and memory cost is small. Attention residuals can be used during post-training without pretraining from scratch, and the theoretical analysis motivates the method and supports its architectural design.

Key responses and revisions to the comments: We have comprehensively responded to the reviewers' feedback and revised the manuscript accordingly. The key responses and revisions are summarized below:

  • [E4iq] (Revision): We corrected the projection initialization in Section 3.3 and Appendix F.1 to independent Gaussian initialization. Appendix G.5 now analyzes this initialization and explains why the rank-one setting loses query-controlled relative depth selection.

  • [E4iq] (Revision): We added Appendix K.2 and Tables 5–6 to examine initialization type, initialization scale, and learned effective rank. The results show that effective rank varies across scales and tasks, while higher effective rank does not consistently improve accuracy.

  • [E4iq, 4EXY, wFUU] (Revision): We added parameter-budget comparisons in Appendix K.3, Table 7. Across three budget levels, LoRA + LLARA outperforms LoRA-only baselines with slightly more trainable parameters, supporting gains beyond additional parameter count alone.

  • [E4iq, 4EXY] (Revision): We added a learned static layer-wise scalar mix in Appendix K.3, Table 8. LLARA exceeds the static baseline with a larger total parameter budget by 1.5 percentage points on GSM8K and 2.0 points on MATH, supporting the benefit of token-dependent depth weighting.

  • [E4iq] (Revision): We restricted Theorem 8 in Appendix C to deterministic weights and added Theorem 9 for residual-dependent weights. The new theorem explicitly includes the interaction between weight fluctuations and noise in both bounds, providing sufficient conditions for signal preservation without assuming weight–noise independence.

  • [4EXY] (Revision): We replaced “robustness” with “stability” in the abstract to clarify that the claim concerns training stability. We linked this clarification to the architectural-gradient analysis in Section 3.5 and the loss and gradient-norm comparisons in Section 4.4, Figure 3.

  • [4EXY] (Revision): We documented the optimized inference path in Appendix F.3 and added latency, throughput, memory, and preparation-time measurements in Appendix K.5.2, Table 14. Under the same execution configuration, LLARA adds 1.4–1.7% median latency over the LoRA-merged control across the tested settings, excluding one-time CUDA-graph preparation.

  • [4EXY] (Revision): We added Appendix H to analyze the structural mismatch in our in-place AttnRes comparison. We show that zero-initialized queries yield uniform averaging and that changing query initialization alone cannot generally recover standard residual accumulation. We connect this analysis to the architectural-gradient analysis in Section 3.5 and the training instability observed in Section 4.4, Figure 3.

  • [wFUU] (Revision): We added controlled signal-plus-noise experiments in Appendix K.4.1, Table 9 and residual interventions on a trained Qwen2.5-7B checkpoint in Appendix K.4.2, Table 10. Standard accumulation and uniform averaging decay as approximately O(1/L)O(1/\sqrt{L}), while a selective readout remains approximately constant. Under late-depth interference, the LLARA readout retains more early-component visibility than standard accumulation, and recomputing its input-dependent weights improves visibility further.

All the revised contents are highlighted in the updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.