acceptodds
Under review as a conference paper at ICLR 2027

Low-Rank Attention Residuals

Abstract

Attention Residuals (AttnRes) replace the fixed residual sum with depth-wise attention over previous sub-layer outputs in Large Language Models (LLMs), but use each output as both a full-dimensional key and value. This couples routing with representation and makes the cost of computing depth-routing scores scale with hidden width . We propose Low-Rank Attention Residuals (LR-AttnRes), which keep full-dimensional residual values while using -dimensional keys, with , for routing. LR-AttnRes uses the last dimensions of each value as the routing key, reducing total residual-side FLOPs while still improving performance. Comprehensive sweeps across the number of blocks () and show that depth-wise routing can be effective with far fewer dimensions than the model width. At both B and B parameters with , LR-AttnRes achieves lower final validation loss, higher average downstream accuracy, and higher measured training-step throughput than standard AttnRes. We also provide a fused kernel supporting standard and low-rank routing. We release all code, the kernel, and all trained models to facilitate future research.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention Residuals, Low-Rank Routing, Depth-Wise Attention, Residual Connections, Transformers, Large Language Models, Training Efficiency, Training Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UpP7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper modifies Attention Residuals to weight earlier layers’ outputs using fewer coordinates while keeping the full outputs, and reports incremental improvements in prediction quality and training speed on models up to 4B parameters.

Critical strengths

The structure is a simple modification of Attention Residuals, with the clear goal of reducing computation time.

Critical weaknesses
  1. The paper does not clearly explain why using fewer coordinates leads to better predictions. The theory shows when fewer coordinates can preserve the model’s outputs, but it does not explain why the results improve.

  2. The change to Attention Residuals is small, and so are the gains: about 1% faster training and 0.4–0.6 percentage points higher average accuracy on the 1B and 4B models. I am not convinced these improvements are enough to justify acceptance.

  3. The presentation makes the main idea harder to follow than it needs to be. The change from Attention Residuals could be explained more directly, and many numerical results are reported with unnecessary decimal places.

Questions for authors

Can you explain why using fewer routing coordinates improves prediction quality, and what evidence supports this explanation?

Reviewer_ybW7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LR-AttnRes to reduce the computational cost of AttnRes. Specifically, the method lowers the dimensionality of the routing keys in AttnRes while preserving the original dimensionality of the values (i.e., the sub-layer outputs). The study demonstrates that reducing the routing key dimension maintains the performance of AttnRes while improving efficiency. Extensive experiments across multiple model scales, experimental settings, and configurations strongly support both the effectiveness and efficiency of the proposed approach.

Critical strengths

This paper provides strong, reliable empirical results supported by thorough ablation studies, and transparent disclosure of the training process. For example, the authors report comprehensive FLOPs and validation loss metrics across 32 configurations varying in rank and block count, alongside training throughput across different model sizes and GPU architectures.

Critical weaknesses

The current framing and narrative flow do not yet fully establish the conceptual novelty of applying a low-rank paradigm to attention residuals. While the paper highlights potential issues arising from the "dual role" of sub-layer outputs, it does not empirically verify whether this interference actually occurs in practice, or to what extent the low-rank approach mitigates it. Providing direct empirical evidence, such as quantifying the interference caused by this dual role and measuring how much LR-AttnRes reduces it, would make this proposed motivation much more convincing. Without such verification, the motivation remains somewhat unsubstantiated, particularly because LR-AttnRes still utilizes a subset of the sub-layer output as a key, meaning the dual role persists within those dimensions. While the gradient norm analysis offers useful insights into general training dynamics, it does not directly demonstrate that LR-AttnRes outperforms standard AttnRes by resolving dual-role interference.

Additionally, it is difficult to gauge how this work is situated within the broader literature due to the relatively brief related work section. A more comprehensive discussion of prior studies, particularly those closely aligned with the proposed architectural design, would help readers better appreciate both the significance of the problem being addressed and the originality of the proposed method.

Questions for authors
  1. Could the authors provide a more detailed empirical analysis quantifying the interference caused by the dual role of the sub-layer output, and directly evaluate whether LR-AttnRes mitigates this specific effect?
  2. Since LR-AttnRes uses a slice of the sub-layer output as the key representation, it appears that the dual role still exists within that subspace. Does restricting the rank of the routing keys actually reduce the interference between the two roles, or could the observed improvements simply arise from constraining the routing dimensionality?
  3. Could the authors expand the discussion of related works to better clarify the positioning of this paper and highlight its architectural novelty relative to existing methods?
  4. While I acknowledge the validity of the empirical results, I find the technical contribution of this work rather limited without a convincing explanation of why low-rank routing keys should lead to better performance, or why the simple slicing strategy is effective beyond its computational benefits. I would appreciate further evidence or justification from the authors to clarify these points.
Reviewer_AqR5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper modifies AttnRes to use only last rr coordinates of each output as the routing key, while the values stay full width.

Critical strengths
  1. The change is simple and essentially free to implement.
  2. The sweep is systematic, and the paper includes seed replication and a full-RMS control.
  3. The paper is forward about what its analyses do and do not show.
Critical weaknesses
  1. The novelty is very small. The whole method is basically slicing. It also contradicts the paper’s own motivation: the keys are still value coordinates, so routing and representation are not decoupled. The variant that actually decouples them (Projected LR-AttnRes) does no better (2.944).
  2. The best result reflects selection. r=128 was picked from 32 single-seed runs judged on the same validation split, and only that configuration was replicated. Many sweep differences are about 0.001, which is within seed noise.
  3. The only comparisons are AttnRes and a vanilla Transformer. There are no compute-matched comparisons with DenseFormer, hyper-connections or Deep Delta Learning.
Questions for authors

See weaknesses.

Comments on the paper

1 comment

Thank you to the authors for the detailed response.

I appreciate the new interference analyses, which provide useful evidence of reduced gradient opposition and routing-induced disruption. However, these results do not fully establish whether the reduction in interference within the shared subspace is responsible for the observed performance gains, or whether the improvements primarily arise from constraining the routing dimensionality. This distinction is important, particularly given the full-width RMS control and projected key results already presented in the manuscript. While the training-free conversion is an interesting additional contribution, it does not explain why low-rank routing improves performance when trained from scratch. The expanded related work is also appreciated, although the architectural novelty remains somewhat limited, I think. Overall, while the rebuttal addresses some of my concerns, I still remain unconvinced by the proposed explanation for the performance gains, thus i will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.