acceptodds
Under review as a conference paper at ICLR 2027

Dual Attention Residuals

11 views

Abstract

Recent work extends Transformer residual pathways along two complementary axes: historical retrieval provides direct access to earlier computation, whereas multi-stream methods maintain multiple evolving representations. These direc- tions have largely developed independently, leaving unexplored how depth-wise retrieval and multi-stream representation can complement each other within a uni- fied residual architecture. We propose Dual Attention Residuals (DAR), which combines these capabilities by allowing the two trajectories to participate recip- rocally in depth retrieval. For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream’s own history. The retrieved states are combined as input to a Transformer branch. To control overhead, DAR retrieves over completed block histories while maintaining two temporary stream states within each block. Under matched training-token budgets, DAR achieves lower validation loss than the eval- uated residual baselines across dense models from 0.1B to 1B parameters and a 7B sparse-MoE model. Parameter-matched ablations show that DAR benefits from both its dual-stream representation and reciprocal cross-stream retrieval. Rep- resentation and intervention analyses further show that reciprocal cross-stream selection preserves depth-wise diversity and avoids the redundancy or functional imbalance observed in alternative two-stream designs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer Architecture, Residual Connections, Multi-Stream Residuals

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eix6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Dual Attention Residuals (DAR), combining retrieval over historical residual states with two evolving residual streams. Each stream uses the opposite stream's normalized states as keys and its own states as values. Learned gates combine the retrieved states for an unchanged Transformer branch and distribute its output through a two-stream residual update. A block formulation limits history overhead. Experiments cover 0.1B–1B dense models and a 7B sparse-MoE model under matched training-token budgets, with language-modeling, downstream, efficiency, and retrieval-rule evaluations.

Critical strengths
  1. The architecture is well specified and addresses a meaningful design question: how historical retrieval and multiple residual trajectories can interact. The SelfKV control retains the two-stream machinery while changing the key source, providing a useful test beyond comparison with an ordinary Transformer. Appendix A also correctly identifies CrossV as an equivalent parameterization rather than treating it as an independent improvement.

  2. The empirical evidence is reasonably broad and consistently positive. DAR improves validation loss across the tested dense sizes and sparse-MoE setting. At 1B, Paloma token perplexity improves from 20.707 to 20.264 over AttnRes, and DAR is better on 15 of 16 reported domains. Table 4 further favors DAR over SelfKV and FixedKV. Table 5 transparently reports the cost: 94K versus 104K tokens/s and 70.5 versus 67.4 GiB compared with AttnRes. These results support a useful quality–cost tradeoff and make me lean toward acceptance.

Critical weaknesses
  1. Some experimental accounting is not yet reproducible from the description. Section 4.4 assigns an independent d-by-d value projection to each historical source, but Section 5.1 claims parameter matching by reducing the 0.5B FFN width only from 5120 to 4992. With 20 layers and d=1280, this saves only 9.83M parameters even for a three-matrix FFN, whereas twenty such projections alone add 32.77M. Table 8 does not give the VProj total. The actual sharing or projection placement may differ, but this needs clarification before interpreting VProj's worse result as evidence against value projection. The supplementary default training configuration also differs from Table 11, making the exact experimental version unclear. These issues affect specific controls and reproducibility; they do not invalidate the more direct DAR–SelfKV comparison.

  2. The mechanistic conclusions are stronger than the current diagnostics establish. Zeroing a DAR stream affects its own value path and the other stream's key path, so the rescue intervention does not separate content loss from disrupted addressing. It supports joint functional dependence, but cannot alone establish complementary content or lower effective depth in competing designs. CKA and gate analyses use only four sequences' final 128 tokens, and Appendix C states that every result is from a single training run. Thus, the 0.025 loss advantage over SelfKV and the observed stream imbalance lack evidence of stability across initializations. The consistent cross-scale improvements remain persuasive, but the causal and general architectural interpretations should be qualified.

Questions for authors
  1. Clarification: What are the exact number, dimensions, sharing pattern, and total parameter count of the VProj projections? Please reconcile these with the stated FFN-width adjustment and specify the actual Table 4 configurations.

  2. Clarification: In the rescue experiment, which current states and cached history entries are modified? Can the reported effect be distinguished from perturbing the opposite stream's addressing? Targeted validation, if available: key-only/value-only interventions and a small number of repeated 0.5B DAR/SelfKV runs would materially strengthen the mechanism claim; otherwise, please narrow it to the observed models.

  3. Reproducibility clarification: Please provide the exact commands and overrides for Tables 5 and 11. The supplied YAML defaults use 100K iterations and microbatch 4, while Table 11 specifies 60K and 2; the launcher also defaults to no LR scheduler. Additionally, static inspection of the per-branch query/gate shapes suggests about 1.084M extra DAR parameters over AttnRes at 24 layers and d=2048, rather than the 0.363M in Section 5.5. Which implementation and parameter-sharing configuration produced the reported results?

2 comments

Dual Attention Residuals | Response to Reviewer eix6

We appreciate your constructive feedback on our manuscript.

1. Parameter counts and reproducibility (Weakness 1, Q1, Q3)

Thank you for reviewing our paper so carefully. You identified errors in our manuscript by calculating VProj's cost and checking DAR's parameter increase against the supplementary code. We mistakenly reported the VProj FFN width and the parameter increase of DAR over AttnRes from earlier configurations. We apologize for this oversight. We corrected both values for the reported runs and added the VProj total to Table 8.

Weakness 1 / Q1. Table 4 configurations and VProj parameter counts

All five methods in Table 4 use the 20-layer 0.5B Full configuration with d=1280d=1280. The FFN width is 5120 for AttnRes, DAR-SelfKV, DAR-FixedKV, and DAR, and 4704 for AttnRes + VProj. Each source layer has its own value projection matrix. For candidate viv_i (i≥1i\geq 1), L(i)L(i) denotes the Transformer layer that produced it, and its projection matrix is Wv,L(i)W_{v,L(i)}. Attention and FFN branches share this matrix when reading candidates from that layer. The initial embedding is used directly. The 20 matrices of size 1280×12801280\times1280 add 32.768M parameters. Reducing the FFN width from 5120 to 4704 saves 31.9488M parameters, giving a net increase of 0.8192M over AttnRes.

The original 4992 came from an earlier configuration with four layers per block and six matrices, including one for the embedding. We clarified how the projections are shared in Section 4.4 and corrected the FFN width in Section 5.1. Table 8 now gives the VProj total of 492.444M parameters excluding vocabulary parameters and 660.216M including them.

Weakness 1 / Q3. Launch commands and overrides for Tables 5 and 11

We used the settings in Table 11, training for 60,000 steps with a microbatch size of 2 and a global batch size of 128. We warmed up the learning rate to 3×10−43\times10^{-4} over the first 2,000 steps and then decayed it to 3×10−53\times10^{-5} on a cosine schedule. We updated the defaults in the supplementary code. The README's Training commands section lists all three launch commands and their overrides.

Q3. Which implementation and sharing configuration produced Table 5?

Table 5 uses the fused implementations. In both AttnRes and DAR, the attention and FFN branches have independent queries. DAR's gates are also branch-specific. At 24 layers and d=2048d=2048, including the output queries gives 100,352 residual-module parameters for AttnRes and 1,184,128 for DAR, a difference of 1,083,776 (about 1.084M). The original 0.363M came from an earlier experiment with within-block sharing. We corrected Section 5.5 and Table 10.

2. Rescue interventions and mechanistic interpretation (Weakness 2, Q2)

Q2. Which current states and cached history entries are modified?

We intervene once on the two-stream residual state immediately after the MLP at layer nn, testing three conditions for all tokens. We retain only stream 0, retain only stream 1, or zero both. For DAR and its retrieval variants, the modified output passes to later layers and supplies the keys and values for the new history entry at layer nn. Previously cached entries, including the initial embedding, retain their original values. Subsequent layers run as usual. For mHC-2, the intervention affects only the two-stream output at that layer (Section 5.4, Figure 5).

Weakness 2 / Q2. Can content loss be separated from disruption of the other stream's addressing?

The current rescue intervention cannot separate these effects. For example, the same state supplies the value for stream 0 and the key for stream 1. Zeroing it changes both paths at once, making a key-only or value-only intervention difficult. As suggested, we narrowed the conclusion in Section 5.4 to the joint functional dependence on both streams observed in the evaluated DAR model and limited our mechanistic interpretation to the behavior observed in this comparison.

Weakness 2. Effective depth and the diagnostic samples

Both analyses use token positions from the four validation sequences as observations. For CKA, each sampled position contributes a row of hidden representations. We compare the two streams at matched positions across pairs of layers. In Figure 4, mHC-2 shows high cross-stream CKA for many pairs of late layers at different depths. Section 5.4 presents lower effective depth as one possible explanation.

For the gate analysis, we average the write gates across positions. DAR and DAR-SelfKV both use the final 128 positions of each sequence, so each per-layer, per-stream mean uses the same 512 positions. On these inputs, the gap in gate values between streams is smaller for DAR than for DAR-SelfKV, consistent with the rescue results in Figure 5(b).

Thank you for the response. The SelfKV comparison shows that swapping keys helps in the tested setting. What remains unclear to me is why keys from one stream should be better at identifying useful historical states in the other stream. The validation loss comparison shows that the design can work, but it does not directly test that explanation. I still consider this a gap in the paper’s core motivation. On balance, I’m keeping my original rating of 3.

Reviewer_mNbuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Dual Residual Attention (DAR), which combines a modified version of Attention Residuals (AttnRes) (Kimi Team et al., 2026) with Manifold-Constrained Hyper-Connections (mHC) (Xie et al., 2025). The modified AttnRes uses two streams instead of one, as in the original work, to make it compatible with mHC. The authors propose to mix the attention streams of AttnRes by swapping keys between streams: each stream uses keys from the other stream to compute attention weights, while values are kept from the current stream.

The authors train and evaluate models with AttnRes, mHC, DAR, and variants of DAR, such as DAR-selfKV, where keys are not swapped. They train 0.1B–1B-parameter transformers and a 7B-parameter MoE transformer (1.4B active parameters), showing that DAR models achieve lower validation loss. Moreover, the 1B model outperforms the other baselines on average across short LM benchmarks.

The paper demonstrates that DAR streams learn representations that are less similar in terms of Centered Kernel Alignment (CKA) compared to those learned by mHC and DAR-selfKV models. Additionally, the authors analyze the behavior of the models when one stream is suppressed at a single layer.

Critical strengths
  • The proposed method is potentially useful, as it extends both AttnRes and mHC. The original works demonstrated that both methods substantially improve model performance. Since the two methods can be combined by design, exploring their combination is a natural direction.
Critical weaknesses
  • Cross-stream interaction occurs through two mechanisms: mHC and key swapping in AttnRes. Since mHC already allows information exchange across streams, it is not clear why mHC, together with interactions across multiple layers, is insufficient. The authors do not sufficiently discuss this or explain why cross-stream retrieval is necessary.

  • The evaluation appears incomplete. (1) For the 7B models, only validation loss is reported. (2) In the LM results for the 1B models, only three baselines are included in the table, but I would expect to see at least DAR-selfKV and mHC as well. (3) It would also be useful to see a comparison between mHC-4 and mHC-2. The latter operates with two streams, similarly to DAR, but the authors report results only for mHC with four streams. (4) There are not enough results on downstream tasks to draw strong conclusions in favor of DAR. Most experiments focus on validation loss, which does not necessarily reflect downstream performance. The models could be evaluated on additional LM benchmarks and/or, for example, RULER, which is suitable for base models.

  • The interpretation of the results in Section 5.4 seems too strong. The experiment can be interpreted as measuring "how the final outputs change when one stream is removed at a single layer" rather than "how much one stream contributes to the final output." Removing a stream can disrupt subsequent operations in later layers, even when the streams contain redundant information. Therefore, the rescue scores do not necessarily demonstrate that DAR learns complementary representations.

  • The presentation of the results in Section 5.4 could be improved. (1) The authors should better explain how to interpret negative scores in Figure 5. (2) The authors write, "DAR-SelfKV’s KL rescue consistently favors stream 1." However, since DAR mixes streams, the stream indices are nominal, and their order in the plot should not matter. I suggest reporting the minimum and maximum rescue gains, or their absolute difference, at each layer.

Questions for authors

NA

2 comments

Thank you for your careful review and thoughtful suggestions.

Critical Weaknesses

1. Why exchange keys when mHC already mixes information across streams?

Thank you for pointing out that mHC mixing across multiple layers already lets the two streams exchange information. We chose to exchange keys for two reasons. Table 4 gives the corresponding comparisons under the same 0.5B Full setting.

  1. Decoupling keys and values while preserving symmetry between the streams. Both DAR-FixedKV and DAR use one stream's representations to score historical layers and read content from the other. DAR-FixedKV always takes all keys from stream 0 and all values from stream 1. In DAR, each stream supplies keys to the other and reads its own values. The two designs have validation losses of 2.690 and 2.653, respectively.

  2. Allowing the streams to interact directly when selecting historical layers. DAR-SelfKV retains mHC-style mixing at every layer, but each stream still scores historical layers with its own keys. DAR keeps the same mixing and write mechanisms and changes only the source of the keys used for scoring. Each stream can thus directly influence which layers the other reads. Validation loss falls from 2.678 (DAR-SelfKV) to 2.653 (DAR).

2. Evaluation

Thank you for suggesting additional evaluations. We agree with these suggestions and have added the results we were able to obtain.

2.1 Downstream evaluation of the 7B models

We agree that downstream results for the 7B models would strengthen the comparison. We are sorry that we cannot run this evaluation at present because we do not have access to the company server that holds the model checkpoints.

2.2 Including DAR-SelfKV and mHC in the 1B evaluation

Thank you for the suggestion. We chose AttnRes as the main residual baseline for the 1B experiments based in part on the Kimi Team's original comparison, in which AttnRes generally outperformed mHC.

We agree that adding DAR-SelfKV and mHC under the same 1B setting would make the comparison more complete. Given the compute currently available to us, we have not completed these two experiments. Our 1B conclusions are limited to the methods evaluated.

2.3 Performance of mHC-2 and mHC-4

Thank you for the suggestion. We added the mHC-2 validation losses to Table 3. At 0.1B, 0.3B, and 0.5B, they are 3.557, 2.848, and 2.779, respectively. The corresponding mHC-4 losses are 3.531, 2.844, and 2.764.

2.4 Additional downstream tasks, LM benchmarks, and RULER

Thank you for the suggestion. We added RULER results for the three 1B models using a 4K context budget, the base prompt template, and greedy decoding. The evaluation covers all 13 tasks, with 500 examples per task per model. The table gives the mean scores across the 13 tasks.

ModelMean RULER score (%)
Baseline31.37
AttnRes33.31
DAR35.12

DAR scores 3.75 percentage points higher than Baseline and 1.81 points higher than AttnRes on average, providing a comparison beyond validation loss. We added the mean scores to the main text and the full per-task results and evaluation settings to Appendix Table 13.

3. Interpretation of the results in Section 5.4

Thank you for pointing this out. We clarified in Section 5.4 that rescue measures how much of the original output is recovered when only one stream is retained at the selected layer, relative to zeroing both streams at that layer. We narrowed our conclusion about complementary representations accordingly. The observed recovery patterns suggest that DAR's predictions depend in part on the two streams working together.

4.1 Meaning of negative rescue scores in Figure 5

As you suggested, we clarified in Section 5.4 that a negative score in Figure 5(a) means that retaining only one stream at that layer makes the final output deviate farther from the original output than zeroing both streams at the same layer does.

4.2 Stream indices and the description of rescue asymmetry

Thank you for the suggestion. Figure 5(a) shows the rescue score for each retained stream. The new Figure 5(b) shows the absolute difference ∣Rn,1−Rn,0∣|R_{n,1}-R_{n,0}| between the two scores at each layer. Exchanging the stream labels leaves this difference unchanged. We replaced the statement that “DAR-SelfKV's KL rescue consistently favors stream 1” with a comparison of within-layer differences. Taking the absolute difference at each layer and then averaging across layers gives 0.60 for DAR-SelfKV and 0.29 for DAR. Under this intervention, the two streams differ more in output recovery for DAR-SelfKV.

Thank you for your response. The additional experiments address some of my concerns. However, downstream performance at 7B remains untested, and the 1B downstream comparison does not include DAR-SelfKV. As a result, it is still unclear how much cross-stream retrieval contributes to the observed gains. I will keep my score unchanged.

Reviewer_o2xEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at attention architecture. The motivation is that some improvements attends residual streams to several layers back, others use multiple residual streams at each layer, but there is no interaction between the two types of improvements. The paper proposes dual attention residuals, and backs it up with scaling experiments.

Critical strengths

The motivation of the paper is clear, as mentioned in the summary above, and the methodology makes sense to me. The authors also back up the proposed methodology with extensive evaluation and scaling experiments, which is a plus.

Critical weaknesses

Significance

  1. In the comparison between different methods, all methods use the same parameter. Do you also have comparison of their FLOP, memory, and wall-clock time at training/inference? The only answer I could find is a statement on "the dominant component of total training FLOPs remains comparable across methods", but there is no qualitative report.

Clarity

  1. The methodology part is heavy on notations, and some of the derivations focus on what the operations are without motivating why we want to do them in the first place. This is the part that I am lost. Specifically,
    1. Line 166-167: What is PbiP_b^i ? In the text, it was said that they are "temporary states within block b" -- but that does not tell how they are defined. Perhaps the more important question is, why do we need these temporary states PbiP_b^i? Why are we not happy with only using the original states HbiH_b^i ?
    2. Line 175: There is a CrC_r and a Cl\mathcal{C}_l, where the rr indexes the history (which i suppose is the index of the layer), and ll indexes branch, which is confusing. Should not both CC and C\mathcal{C} have both rr and ll as indexes?
  2. Since I was confused by the text, I moved on to the figure. However, the figure did not make it clear either. The figure seems to be a stacking of many boxes, without clearly explaining what each box does and how different boxes are related.
    1. There is a history box on the left. What is inside them? Are they different layers in a transformer? Why is one box blue and the other orange -- initially i thought they are different sub-layers within a transformer layer, but the the paper uses dual residual streams which is another possibility, so I don't know any more.
    2. In the middle, there are two giant boxes with blue/orange background. How are they related to the "history box" on the left? There are also many smaller H/v boxes, which are not connected at all.
Questions for authors

Please see the questions above.

1 comment

Thank you for reading our paper carefully and for your helpful comments. Some parts of the original manuscript were unclear, which made DAR harder to understand. In response to your comments, we have rewritten the Methods section and redrawn the schematic in Figure 1.

Significance 1. Training and inference costs

For the 7B sparse MoE models, Table 5 reports measured training throughput and peak memory per GPU for Baseline, AttnRes, and DAR.

Clarity 1.1. What is PbiP_b^i, and why is it needed?

In the original notation, PbiP_b^i is the pair of residual states accumulated through the ii-th branch of block bb. We called it a “temporary state” but did not clearly explain why later branches need it. Consider block bb in revised Figure 1(c). The first attention branch writes its output to both streams, forming Pb1P_b^1. At this point the history bank still contains only H0,…,Hb−1H_0,\ldots,H_{b-1}. The MLP reads Pb1P_b^1 to access the attention update. Its output is then used to update the pair to Pb2P_b^2. Section 4.1 now uses ss for the branch position within a block, and Section 4.2 uses ii for the position of a candidate source. Equations (13) and (5) give the update and the block-end storage rule, respectively.

Clarity 1.2. Candidate indices at line 175

You asked whether CrC_r and Cℓ\mathcal C_\ell both need two indices. We omitted the index of the reading branch from CrC_r. In the revised Section 4.2, Eq. (6) defines the entire candidate list Cℓ\mathcal C_\ell for branch ℓ\ell, so one index suffices for the list. Immediately after the equation, we denote the entry at position ii by Cℓ,iC_{\ell,i}. Its two indices identify the reading branch and the position in its list.

Clarity 2.1. The History box and the two colors

Each blue-orange pair in the original History box represents one history entry. In the notation of the original Eq. (5), Hr=[vr0,vr1]H_r=[v_r^0,v_r^1]. Blue denotes residual stream 0 (vr0v_r^0), and orange denotes residual stream 1 (vr1v_r^1). Both components of the first entry start from the same token embedding, originally written as H0=[h1,h1]H_0=[h_1,h_1]. Each subsequent pair stores the final state of one completed block of KK Transformer layers. We left these boxes unlabeled, so it was easy to read the colors as two sublayers. In the redrawn Figure 1(c), Attention and MLP are shown as separate branches. The History bank at the bottom starts with H0=[e,e]H_0=[e,e]. The Block bb output at the top, Hb=Pb4H_b=P_b^4, is added to the bank when the block ends.

Clarity 2.2. How the two middle regions relate to History

In the original figure, the blue and orange regions show the stream-0 and stream-1 components of the entries in History. The last row adds the current block's partial state. For example, H10H_1^0 and v10v_1^0 in the blue region denote the same stream-0 component of H1H_1, used directly as a value. We drew it twice without making that correspondence clear. In revised Figure 1(c), the first MLP in block bb reads the history entries H0,…,Hb−1H_0,\ldots,H_{b-1} and the current partial state Pb1P_b^1. The inset shows the cross stream attention operation αcross0\alpha_{\mathrm{cross}}^0 for stream 0. For the entry Hb−1H_{b-1}, stream 0 uses the orange component vb−11v_{b-1}^1 as its key and the blue component vb−10v_{b-1}^0 as its value. Stream 1 uses the same pair with the roles reversed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.