- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Dual Attention Residuals (DAR), combining retrieval over historical residual states with two evolving residual streams. Each stream uses the opposite stream's normalized states as keys and its own states as values. Learned gates combine the retrieved states for an unchanged Transformer branch and distribute its output through a two-stream residual update. A block formulation limits history overhead. Experiments cover 0.1B–1B dense models and a 7B sparse-MoE model under matched training-token budgets, with language-modeling, downstream, efficiency, and retrieval-rule evaluations.
Critical strengths
-
The architecture is well specified and addresses a meaningful design question: how historical retrieval and multiple residual trajectories can interact. The SelfKV control retains the two-stream machinery while changing the key source, providing a useful test beyond comparison with an ordinary Transformer. Appendix A also correctly identifies CrossV as an equivalent parameterization rather than treating it as an independent improvement.
-
The empirical evidence is reasonably broad and consistently positive. DAR improves validation loss across the tested dense sizes and sparse-MoE setting. At 1B, Paloma token perplexity improves from 20.707 to 20.264 over AttnRes, and DAR is better on 15 of 16 reported domains. Table 4 further favors DAR over SelfKV and FixedKV. Table 5 transparently reports the cost: 94K versus 104K tokens/s and 70.5 versus 67.4 GiB compared with AttnRes. These results support a useful quality–cost tradeoff and make me lean toward acceptance.
Critical weaknesses
-
Some experimental accounting is not yet reproducible from the description. Section 4.4 assigns an independent d-by-d value projection to each historical source, but Section 5.1 claims parameter matching by reducing the 0.5B FFN width only from 5120 to 4992. With 20 layers and d=1280, this saves only 9.83M parameters even for a three-matrix FFN, whereas twenty such projections alone add 32.77M. Table 8 does not give the VProj total. The actual sharing or projection placement may differ, but this needs clarification before interpreting VProj's worse result as evidence against value projection. The supplementary default training configuration also differs from Table 11, making the exact experimental version unclear. These issues affect specific controls and reproducibility; they do not invalidate the more direct DAR–SelfKV comparison.
-
The mechanistic conclusions are stronger than the current diagnostics establish. Zeroing a DAR stream affects its own value path and the other stream's key path, so the rescue intervention does not separate content loss from disrupted addressing. It supports joint functional dependence, but cannot alone establish complementary content or lower effective depth in competing designs. CKA and gate analyses use only four sequences' final 128 tokens, and Appendix C states that every result is from a single training run. Thus, the 0.025 loss advantage over SelfKV and the observed stream imbalance lack evidence of stability across initializations. The consistent cross-scale improvements remain persuasive, but the causal and general architectural interpretations should be qualified.
Questions for authors
-
Clarification: What are the exact number, dimensions, sharing pattern, and total parameter count of the VProj projections? Please reconcile these with the stated FFN-width adjustment and specify the actual Table 4 configurations.
-
Clarification: In the rescue experiment, which current states and cached history entries are modified? Can the reported effect be distinguished from perturbing the opposite stream's addressing? Targeted validation, if available: key-only/value-only interventions and a small number of repeated 0.5B DAR/SelfKV runs would materially strengthen the mechanism claim; otherwise, please narrow it to the observed models.
-
Reproducibility clarification: Please provide the exact commands and overrides for Tables 5 and 11. The supplied YAML defaults use 100K iterations and microbatch 4, while Table 11 specifies 60K and 2; the launcher also defaults to no LR scheduler. Additionally, static inspection of the per-branch query/gate shapes suggests about 1.084M extra DAR parameters over AttnRes at 24 layers and d=2048, rather than the 0.363M in Section 5.5. Which implementation and parameter-sharing configuration produced the reported results?
2 comments
Thank you for the response. The SelfKV comparison shows that swapping keys helps in the tested setting. What remains unclear to me is why keys from one stream should be better at identifying useful historical states in the other stream. The validation loss comparison shows that the design can work, but it does not directly test that explanation. I still consider this a gap in the paper’s core motivation. On balance, I’m keeping my original rating of 3.
Dual Attention Residuals | Response to Reviewer eix6
We appreciate your constructive feedback on our manuscript.
1. Parameter counts and reproducibility (Weakness 1, Q1, Q3)
Thank you for reviewing our paper so carefully. You identified errors in our manuscript by calculating VProj's cost and checking DAR's parameter increase against the supplementary code. We mistakenly reported the VProj FFN width and the parameter increase of DAR over AttnRes from earlier configurations. We apologize for this oversight. We corrected both values for the reported runs and added the VProj total to Table 8.
Weakness 1 / Q1. Table 4 configurations and VProj parameter counts
All five methods in Table 4 use the 20-layer 0.5B Full configuration with d=1280. The FFN width is 5120 for AttnRes, DAR-SelfKV, DAR-FixedKV, and DAR, and 4704 for AttnRes + VProj. Each source layer has its own value projection matrix. For candidate vi (i≥1), L(i) denotes the Transformer layer that produced it, and its projection matrix is Wv,L(i). Attention and FFN branches share this matrix when reading candidates from that layer. The initial embedding is used directly. The 20 matrices of size 1280×1280 add 32.768M parameters. Reducing the FFN width from 5120 to 4704 saves 31.9488M parameters, giving a net increase of 0.8192M over AttnRes.
The original 4992 came from an earlier configuration with four layers per block and six matrices, including one for the embedding. We clarified how the projections are shared in Section 4.4 and corrected the FFN width in Section 5.1. Table 8 now gives the VProj total of 492.444M parameters excluding vocabulary parameters and 660.216M including them.
Weakness 1 / Q3. Launch commands and overrides for Tables 5 and 11
We used the settings in Table 11, training for 60,000 steps with a microbatch size of 2 and a global batch size of 128. We warmed up the learning rate to 3×10−4 over the first 2,000 steps and then decayed it to 3×10−5 on a cosine schedule. We updated the defaults in the supplementary code. The README's
Training commandssection lists all three launch commands and their overrides.Q3. Which implementation and sharing configuration produced Table 5?
Table 5 uses the fused implementations. In both AttnRes and DAR, the attention and FFN branches have independent queries. DAR's gates are also branch-specific. At 24 layers and d=2048, including the output queries gives 100,352 residual-module parameters for AttnRes and 1,184,128 for DAR, a difference of 1,083,776 (about 1.084M). The original 0.363M came from an earlier experiment with within-block sharing. We corrected Section 5.5 and Table 10.
2. Rescue interventions and mechanistic interpretation (Weakness 2, Q2)
Q2. Which current states and cached history entries are modified?
We intervene once on the two-stream residual state immediately after the MLP at layer n, testing three conditions for all tokens. We retain only stream 0, retain only stream 1, or zero both. For DAR and its retrieval variants, the modified output passes to later layers and supplies the keys and values for the new history entry at layer n. Previously cached entries, including the initial embedding, retain their original values. Subsequent layers run as usual. For mHC-2, the intervention affects only the two-stream output at that layer (Section 5.4, Figure 5).
Weakness 2 / Q2. Can content loss be separated from disruption of the other stream's addressing?
The current rescue intervention cannot separate these effects. For example, the same state supplies the value for stream 0 and the key for stream 1. Zeroing it changes both paths at once, making a key-only or value-only intervention difficult. As suggested, we narrowed the conclusion in Section 5.4 to the joint functional dependence on both streams observed in the evaluated DAR model and limited our mechanistic interpretation to the behavior observed in this comparison.
Weakness 2. Effective depth and the diagnostic samples
Both analyses use token positions from the four validation sequences as observations. For CKA, each sampled position contributes a row of hidden representations. We compare the two streams at matched positions across pairs of layers. In Figure 4, mHC-2 shows high cross-stream CKA for many pairs of late layers at different depths. Section 5.4 presents lower effective depth as one possible explanation.
For the gate analysis, we average the write gates across positions. DAR and DAR-SelfKV both use the final 128 positions of each sequence, so each per-layer, per-stream mean uses the same 512 positions. On these inputs, the gap in gate values between streams is smaller for DAR than for DAR-SelfKV, consistent with the rescue results in Figure 5(b).