- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper modifies Attention Residuals to weight earlier layers’ outputs using fewer coordinates while keeping the full outputs, and reports incremental improvements in prediction quality and training speed on models up to 4B parameters.
Critical strengths
The structure is a simple modification of Attention Residuals, with the clear goal of reducing computation time.
Critical weaknesses
-
The paper does not clearly explain why using fewer coordinates leads to better predictions. The theory shows when fewer coordinates can preserve the model’s outputs, but it does not explain why the results improve.
-
The change to Attention Residuals is small, and so are the gains: about 1% faster training and 0.4–0.6 percentage points higher average accuracy on the 1B and 4B models. I am not convinced these improvements are enough to justify acceptance.
-
The presentation makes the main idea harder to follow than it needs to be. The change from Attention Residuals could be explained more directly, and many numerical results are reported with unnecessary decimal places.
Questions for authors
Can you explain why using fewer routing coordinates improves prediction quality, and what evidence supports this explanation?
Thank you to the authors for the detailed response.
I appreciate the new interference analyses, which provide useful evidence of reduced gradient opposition and routing-induced disruption. However, these results do not fully establish whether the reduction in interference within the shared subspace is responsible for the observed performance gains, or whether the improvements primarily arise from constraining the routing dimensionality. This distinction is important, particularly given the full-width RMS control and projected key results already presented in the manuscript. While the training-free conversion is an interesting additional contribution, it does not explain why low-rank routing improves performance when trained from scratch. The expanded related work is also appreciated, although the architectural novelty remains somewhat limited, I think. Overall, while the rebuttal addresses some of my concerns, I still remain unconvinced by the proposed explanation for the performance gains, thus i will maintain my original score.