- Rating
- 2
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Role-Decoupled Attention Residuals (RD-AttnRes), giving Q/K and V separate depth routes within Block AttnRes while preserving its source collection and token-attention operation. The modification adds one model-width query vector per attention layer, and tying the routes recovers the parent architecture. Paired experiments at 120M and 343M parameters and a single-seed 1.5B experiment demonstrate lower validation loss. For the 1.5B pair, the authors estimate an average 8.89% reduction in training tokens needed to reach matched validation loss. Role-partition ablations and checkpoint interventions further support the usefulness of the learned separation.
Critical strengths
- The design is simple, well motivated, and recovers Block AttnRes when the routes are tied.
- Paired-seed experiments, equal-cost role partitions, and checkpoint interventions provide convincing evidence that the modification helps in the tested settings.
Critical weaknesses
- Motivation. As mentioned in Intro, MUDDFormer and DeepCrossAttention have already discussed role-specific cross layer reads. The authors should first clarify the challenges and differences when applying RD-AttnRes to Block AttnRes.
- Limited training-scale validation. The largest experiment uses only 10B training tokens and one seed. This provides insufficient evidence that the gains persist under substantially longer pretraining, such as 100B tokens.
- Computational overhead. Despite its minimal parameter increase, RD-AttnRes increases downstream scoring time by 43%, while improving the four primary zero-shot scores by 1.77 percentage points on average. This substantial overhead limits its practical appeal.
Questions for authors
See weakness.