acceptodds
Under review as a conference paper at ICLR 2027

ROLE-DECOUPLED ATTENTION RESIDUALS: SEPARATE QK AND V READS IN BLOCK ATTNRES

Abstract

Block Attention Residuals use one content-dependent depth mixture to construct queries, keys, and values. This couples the representations that determine attention scores to the value content aggregated under those scores. Role-Decoupled Attention Residuals (RD-AttnRes) retain the parent route and learn a second route for over the same Block sources. Tying the route queries recovers Block AttnRes; untying them adds one model-width vector per attention layer and no second token-to-token attention operation. After 2B FineWeb-Edu tokens, RD-AttnRes lowers fixed-prefix validation NLL in all five paired seeds at both 120M and 343M parameters, with mean within-seed perplexity reductions of 2.97% and 2.43%. In a single-seed 1.5B pair trained for 10B tokens, RD-AttnRes lowers endpoint perplexity by 2.22%, leads at all 20 shared quick-validation checkpoints, and reaches the same validation NLL with 8.89% fewer training tokens on average across 17 interpolation-supported targets. The four primary zero-shot scores improve by 1.77 percentage points on average. Tested controls and fixed-checkpoint interventions show that trained models learn and use the separation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention Residuals, Role-Decoupled Routing, Residual Connections, Depth Routing, Transformer Architectures, Language Model Pretraining, Training Efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oQvCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Role-Decoupled Attention Residuals (RD-AttnRes), giving Q/K and V separate depth routes within Block AttnRes while preserving its source collection and token-attention operation. The modification adds one model-width query vector per attention layer, and tying the routes recovers the parent architecture. Paired experiments at 120M and 343M parameters and a single-seed 1.5B experiment demonstrate lower validation loss. For the 1.5B pair, the authors estimate an average 8.89% reduction in training tokens needed to reach matched validation loss. Role-partition ablations and checkpoint interventions further support the usefulness of the learned separation.

Critical strengths
  1. The design is simple, well motivated, and recovers Block AttnRes when the routes are tied.
  2. Paired-seed experiments, equal-cost role partitions, and checkpoint interventions provide convincing evidence that the modification helps in the tested settings.
Critical weaknesses
  1. Motivation. As mentioned in Intro, MUDDFormer and DeepCrossAttention have already discussed role-specific cross layer reads. The authors should first clarify the challenges and differences when applying RD-AttnRes to Block AttnRes.
  2. Limited training-scale validation. The largest experiment uses only 10B training tokens and one seed. This provides insufficient evidence that the gains persist under substantially longer pretraining, such as 100B tokens.
  3. Computational overhead. Despite its minimal parameter increase, RD-AttnRes increases downstream scoring time by 43%, while improving the four primary zero-shot scores by 1.77 percentage points on average. This substantial overhead limits its practical appeal.
Questions for authors

See weakness.

Reviewer_t1i7OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Role-Decoupled Attention Residuals (RD-AttnRes), which assigns separate depth-routing queries to Q/K and V within Block AttnRes. The modification preserves the parent’s source construction and token-attention operation while adding only one model-width vector per attention layer. The authors evaluate the method through paired multi-seed training, controlled ablations, routing diagnostics, functional interventions, and transfer experiments. The results support the effectiveness of adaptive QK/V separation across the tested settings, with additional evidence from a 1.5B-parameter model trained on 10B tokens.

Critical strengths

The carefully matched multi-seed comparisons, targeted controls, alternative role partitions, and functional interventions form a strong and coherent body of evidence. Cross-corpus evaluations and the larger-scale experiment further strengthen the empirical case, making the study particularly solid.

Critical weaknesses

Training efficiency receives substantial attention, while inference evaluation is limited to downstream scoring timings. The reported 43% overhead warrants separate prefill/decode benchmarks and analysis across batch sizes and context lengths.

Questions for authors

Could the authors provide a small inference benchmark separating prefill and decoding costs? This would help contextualize the reported 43% overhead in downstream scoring and clarify the practical cost during autoregressive generation.

Reviewer_65QgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In Block AttnRes, each attention layer uses one token-shared depth query to form a softmax mixture over the embedding, the completed block outputs, and the partial sum of the current block. The same mixed state then produces Q, K, and V. The authors propose RD-AttnRes, which adds a second depth query for V over the same sources. Q/K and V can thus read from different depth distributions. When the two queries are equal, the model reduces exactly to Block AttnRes. In paired experiments with 5 seeds each at 120M and 343M and 2B FineWeb-Edu tokens, RD wins in all 10 pairs, with mean PPL reductions of 2.97% and 2.43%. Under the M-scale protocol, RD reaches a mean NLL of 3.001769, below Block-MHAR (3.027965) and Projected LR-Block (3.016320).

Critical strengths
  • The change is minimal and reduces exactly to the baseline model. RD only changes the depth read on the attention input side. Source construction, MLP routing, write-back, the final mixer, and token attention all stay the same. Tying the queries recovers the baseline model.
  • The paired design is careful, and the statistics are complete. Each pair shares the initialization of all common parameters and the data order, and both route queries start from zero. All 5 seeds at both S and M favor RD.
  • The controls and role partitions support role assignment as the key factor. The study extends to 1.5B parameters and 10B tokens and reports the L-scale results openly.
Critical weaknesses
  • The novelty is limited, and role-specific baselines are missing. Section 2 admits that Delta-V already applies QK|V routing inside Delta AttnRes. MUDDFormer and DeepCrossAttention learn separate cross-layer reads for Q, K, and V, and Value Residual Learning and Depth-Attention mix V across depth separately from Q and K. The new step here is mainly placing QK|V inside Block AttnRes.
  • The efficiency numbers are inconsistent, and the inference cost is understated. The introduction reports a 4.96% increase in time per token at M scale. Appendix G refers to a three-round M timing benchmark in Section 5.7, but that section reports only L-scale timing and gives no M number. Downstream evaluation is 43.0% slower for RD (Table 10), while L-scale training is only 1.51% slower (Table 11). The paper does not explain this gap.
  • The unusual model configuration limits how far the results extend. At all three scales the attention head dimension is 16 (36 by 16 at S, 52 by 16 at M, 111 by 16 at L). Mainstream models use 64 to 128. With such small heads the V capacity per head is limited, so a separate depth input for V may matter more than usual. Part of the gain may come from the setting itself.
Questions for authors
  • Can you add a compute-matched comparison at M scale, where Block trains on about 5% to 10% more tokens or for the same wall-clock time?
  • How does the 4.96% figure in the introduction relate to the 9% to 10% throughput drop in Appendix G? What causes the 43% slower inference in Table 10, and how much of the overhead remains with a fused kernel?
  • Does the RD gain hold with a standard head dimension of 64 or 128 (for example 13 heads of size 64 at M scale)?
  • Could you compare against Delta-V, MUDDFormer, or DeepCrossAttention under the common M-scale protocol? Does the QK|V split also improve Full AttnRes, or input-independent depth aggregation such as DenseFormer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.