1. Presentation and Novelty:
The presentation of the paper feels rushed and lacks the level of explanation and polish expected for an ICLR submission. For instance, Figure 1 is not particularly clear or informative compared with Figure 2 of the closely related MUDDFormer paper, while Figure 2 has inconsistent fonts and sizing. More importantly, several mathematical formulations and design choices are introduced without sufficient motivation or explanation. For example, the statement about Sinkhorn-Knopp iterations and doubly stochastic residual mixing around line 124 is difficult to understand without further context.
Regarding novelty, the proposed approach appears quite similar to MUDDFormer, which already explores separate aggregation for queries, keys, and values, including static and dynamic variants. Although DSR operates on parallel mHC streams rather than historical layer representations, the paper does not sufficiently explain the mathematical and architectural differences or establish the advantages of the proposed approach over related work. Given the limited conceptual novelty, I believe a more detailed comparison is necessary.
2. Experimentation and Mathematical Formulation:
The experimental evaluation is limited to only two model configurations, 1B and 3B, trained for 1.64B tokens, with most ablations conducted at 1B scale. Although some evaluation statistics are reported, there is no statistical analysis across independent training seeds to establish the robustness of the relatively small improvements.
The computational overhead is also wroth noting, with Full DSR requiring approximately 50% more training time and 59% more peak GPU memory at 1B scale. I believe a compute-matched comparison is important. For instance, using the same GPUs and training time, the mHC baseline could process approximately 2.45B tokens instead of 1.64B, based on the reported throughput. It would be interesting to see whether DSR still outperforms mHC under the same computation usage.
Mathematical formulation: I also have concerns about the mathematical motivation behind the proposed approach.
First, the gradient probe in Equation (11) uses the perturbation
xr=x+∑i=1nδi(r)(Xi−x),
which gives
=\left<\nabla_{x_r}\mathcal L,X_i-x\right>.$$
However, DSR in Equation (4) uses normalized representations, so the corresponding perturbation directions would be $\operatorname{RMSNorm}(X_i)-x_{\mathrm{base}}$, rather than $X_i-x$? These are not necessarily equivalent, and it is unclear whether the reported gradient alignment would still hold for the actual representations used by DSR.
Second, the interpretation of negative centered-gradient cosine as conflicting routing preferences is not clear. For example, consider two gradient vectors
$$g_Q=(1,2,3,4),\qquad g_V=(4,3,2,1).$$
After mean-centering, their cosine similarity is $-1$, indicating perfectly opposing centered gradients. However, all components of both original gradients are positive. Therefore, moving from the base toward any candidate using small nonnegative mixture weights would increase the loss to first order for both roles. Both would locally prefer remaining at the original base representation, despite their negative centered-gradient cosine. Thus, negative cosine alone does not establish that separate routing decisions would improve the objective. I believe this interpretation needs further justification.
Finally, looking at Table 2, the validation-loss results across the different routing variants are very similar, and the improvements over the mHC baseline are relatively small compared with the substantial increase in computational resources. For instance, Full DSR improves validation loss from 3.06608 to 3.05631 (approximately 0.32%), while requiring around 50% more training time and 59% more peak GPU memory. Even the simpler static and partial-routing variants require substantially more resources while achieving similar validation losses. This raises concerns about the practical significance and computational efficiency of the proposed approach. A comparison under the same GPU-hours and computational budget would be important to establish whether these improvements justify the additional cost.
Overall, while the results suggest that role-specific aggregation may be useful, I do not find the current experimental evidence and mathematical justification sufficiently convincing to establish the significance of the proposed contribution.