acceptodds
Under review as a conference paper at ICLR 2027

Decoupled Semantic Routing: Role-Specific Residual Stream Readout for Attention

Abstract

Queries, keys, and values serve different purposes in attention, yet multi-stream Transformers typically construct all three from the same mixture of residual streams. We study whether they benefit from separate mixtures. A gradient probe on a trained Manifold-Constrained Hyper-Connections (mHC) model reveals aligned query and key preferences but opposing value preferences in deeper layers. Motivated by this observation, we introduce Decoupled Semantic Routing (DSR), which forms a separate stream mixture for each attention role. DSR computes the mixture weights in a low-dimensional space and applies them to the original hidden representations, using only streams at the current layer. On 1B and 3B language models trained for 1.64B tokens, DSR lowers validation loss relative to mHC by 0.00977 and 0.01830. Static mixtures and partial routing also retain the improvement, with several variants outperforming full dynamic routing. The results distinguish two choices in residual readout: which roles share a mixture, and whether that mixture varies across tokens. Separate mixtures remain effective even when their weights are static.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Stream Residual Networks, Hyper-Connections, Decoupled Semantic Routing, Attention Mechanisms, Query-Key-Value Decoupling, Content-Adaptive Routing, Deep Transformers, Language Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_k4d9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper finds that models with Manifold-Constrained Hyper-Connections (mHC) exhibit stream preference alignment and misalignment: gradient probes are aligned for queries and keys but misaligned between queries/keys and values. The authors propose Decoupled Semantic Routing (DSR), which essentially splits the input into multiple streams and performs learned, context-aware routing of input signals to attention. Models with DSR achieve lower perplexity than models using mHC alone.

Critical strengths

None

Critical weaknesses
  • The method is motivated by the observed gradient misalignment, but the authors do not show whether their method actually resolves this misalignment. Moreover, the authors do not address the fundamental question of why such disagreement is harmful.

  • I found the paper difficult to read. Some parts require considerable effort to understand, even though the underlying ideas turn out to be relatively simple.

  • The experimental setup is not well suited to evaluating the proposed method. (1) The 1B and 3B models are trained on only 1.64B tokens. (2) Recent studies have shown that cross-entropy loss and perplexity are not always reliable indicators of downstream performance. However, the authors report only loss-based results, without evaluating any downstream tasks.

  • The ablation results raise questions about the effectiveness of the proposed method. QK-only routing achieves the best results, while "Static weights" (without context-aware routing) outperform Full DSR. Nevertheless, the authors use Full DSR for their main experiments. This suggests that the proposed context-aware routing mechanism may not be responsible for the observed improvements. Moreover, DSR introduces additional parameters, which could potentially explain the performance gains.

Questions for authors

NA

Reviewer_EpwpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper extends Manifold-Constrained Hyper-Connections (mHC) by proposing to compute the query, key and values with role-specific mixture weights, resulting in Decoupled Sematic Routing (DSR). The approach is to first compute the normalized aggregation across streams as usual, then concatenate it with the RMSNorm of each stream. Then this "pool" feature is then used to compute the query, key and value, and the role-specific weights are computed with softmax attention. Then agggregation is computed for each role. A small-scale fineweb experiment with Qwen2.5 3B and 1B are conducted over 1.64B tokens. Validation loss is compared with mHC and no clear gains are observed. Some ablation is done to disentangle which components are important.

Critical strengths

None

Critical weaknesses
  • The number of training tokens of 1.64B is tiny, far lower than what is usually used for pretraining on FineWeb and if you consider Chinchilla scaling law of roughly 1 parameter for 20 tokens. With this few tokens, the model may struggle to unlock its true expressiveness
  • No downstream experiments at all, only validation loss is examined. And the validaiton loss differences are very small and do not show clear winners, whilst you lose a lot on token/s and memory usage due to the additional parameters and operations.
Questions for authors
  • Is it possible to provide some downstream experiments? No post-training required if you use say commonsense reasoning suite of table 3 in the Gated Deltanet paper (Yang et al. 2025)
Reviewer_JAxzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Decoupled Semantic Routing (DSR), which builds upon Manifold-Constrained Hyper-Connections (mHC). Rather than using the same aggregated representation for the query, key, and value projections, DSR allows each attention role (i.e., Q, K,V) to construct its own mixture from the shared mHC representation and the individual residual streams. This provides each attention component with a separate, role-specific representation, motivating the name Decoupled Semantic Routing.

Critical strengths

Decoupling the attention inputs and giving the attention mechanism greater flexibility to use different mixtures of residual streams is an interesting and potentially beneficial direction. I believe exploring such approaches is worthwhile. Furthermore, I find the gradient-perturbation analysis an interesting idea that, with further development, could provide a stronger justification for the proposed approach and more convincing evidence that using different mixtures for queries, keys, and values can be beneficial.

Critical weaknesses

1. Presentation and Novelty:

The presentation of the paper feels rushed and lacks the level of explanation and polish expected for an ICLR submission. For instance, Figure 1 is not particularly clear or informative compared with Figure 2 of the closely related MUDDFormer paper, while Figure 2 has inconsistent fonts and sizing. More importantly, several mathematical formulations and design choices are introduced without sufficient motivation or explanation. For example, the statement about Sinkhorn-Knopp iterations and doubly stochastic residual mixing around line 124 is difficult to understand without further context.

Regarding novelty, the proposed approach appears quite similar to MUDDFormer, which already explores separate aggregation for queries, keys, and values, including static and dynamic variants. Although DSR operates on parallel mHC streams rather than historical layer representations, the paper does not sufficiently explain the mathematical and architectural differences or establish the advantages of the proposed approach over related work. Given the limited conceptual novelty, I believe a more detailed comparison is necessary.

2. Experimentation and Mathematical Formulation:

The experimental evaluation is limited to only two model configurations, 1B and 3B, trained for 1.64B tokens, with most ablations conducted at 1B scale. Although some evaluation statistics are reported, there is no statistical analysis across independent training seeds to establish the robustness of the relatively small improvements.

The computational overhead is also wroth noting, with Full DSR requiring approximately 50% more training time and 59% more peak GPU memory at 1B scale. I believe a compute-matched comparison is important. For instance, using the same GPUs and training time, the mHC baseline could process approximately 2.45B tokens instead of 1.64B, based on the reported throughput. It would be interesting to see whether DSR still outperforms mHC under the same computation usage.

Mathematical formulation: I also have concerns about the mathematical motivation behind the proposed approach.

First, the gradient probe in Equation (11) uses the perturbation

xr=x+∑i=1nδi(r)(Xi−x),x_r=x+\sum_{i=1}^n\delta_i^{(r)}(X_i-x),

which gives

=\left<\nabla_{x_r}\mathcal L,X_i-x\right>.$$ However, DSR in Equation (4) uses normalized representations, so the corresponding perturbation directions would be $\operatorname{RMSNorm}(X_i)-x_{\mathrm{base}}$, rather than $X_i-x$? These are not necessarily equivalent, and it is unclear whether the reported gradient alignment would still hold for the actual representations used by DSR. Second, the interpretation of negative centered-gradient cosine as conflicting routing preferences is not clear. For example, consider two gradient vectors $$g_Q=(1,2,3,4),\qquad g_V=(4,3,2,1).$$ After mean-centering, their cosine similarity is $-1$, indicating perfectly opposing centered gradients. However, all components of both original gradients are positive. Therefore, moving from the base toward any candidate using small nonnegative mixture weights would increase the loss to first order for both roles. Both would locally prefer remaining at the original base representation, despite their negative centered-gradient cosine. Thus, negative cosine alone does not establish that separate routing decisions would improve the objective. I believe this interpretation needs further justification. Finally, looking at Table 2, the validation-loss results across the different routing variants are very similar, and the improvements over the mHC baseline are relatively small compared with the substantial increase in computational resources. For instance, Full DSR improves validation loss from 3.06608 to 3.05631 (approximately 0.32%), while requiring around 50% more training time and 59% more peak GPU memory. Even the simpler static and partial-routing variants require substantially more resources while achieving similar validation losses. This raises concerns about the practical significance and computational efficiency of the proposed approach. A comparison under the same GPU-hours and computational budget would be important to establish whether these improvements justify the additional cost. Overall, while the results suggest that role-specific aggregation may be useful, I do not find the current experimental evidence and mathematical justification sufficiently convincing to establish the significance of the proposed contribution.
Questions for authors

Most of my question are basically justifying or answering the concerns above in particular to name a few:

  1. What is the main novelty of DSR compared with MUDDFormer, beyond applying role-specific aggregation to parallel residual streams? What are the main mathematical and architectural differences, and what advantages does DSR provide?

  2. Given that Full DSR achieves only a small percentage (0.32%) relative improvement in validation loss while requiring approximately 50% more GPU training time and 59% more peak GPU memory at 1B scale, how do the authors justify this trade-off? Would DSR still outperform mHC under the same GPU-hours and computational resources?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.