acceptodds
Under review as a conference paper at ICLR 2027

Role-Conditioned Residual Computation in Diffusion Language Models

Abstract

Text diffusion is emerging as a competitive alternative to autoregressive language modeling: it reconstructs masked tokens over successive steps and can predict many positions in parallel. Yet standard Transformer backbones carry visible evidence and evolving masked predictions through a single residual state at each position. We ask whether learned routes through depth can better support denoising. We add persistent residual streams with token-conditioned read, transport, and write maps around unchanged Transformer branches. In matched 337M-parameter pretraining on 3.5B tokens, two streams capture 86.6% of the best additional in-domain gain beyond learned single-stream control and achieve the strongest mean transfer across four external corpora. Without assigned stream roles, visible and masked positions learn different read and write routes. Causal interventions establish the importance of these routes and of token-specific write strength; a role-factorized model trained from scratch closes 50.2% of the gap from one stream to the full two-stream model. Stream-state interventions reveal a preservation–transfer–reuse mechanism in the two-stream model: one stream retains an input-dependent early state, the other carries visible-source influence to masked targets, and late layers reuse the retained state. The preservation-and-reuse signature recurs across diffusion widths and is nearly absent in autoregressive controls. We hope these findings establish multi-stream residual transport as both a practical architecture for text diffusion and a useful lens for understanding how denoising models organize computation between evidence and prediction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mhc, manifold-constrained hyperconnections, hyperconnections, diffusion language models, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MK3xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the two distinct roles in masked diffusion language models—visible positions (providing evidence) and masked positions (requiring prediction)—should share the single residual stream found in standard Transformers. The authors introduce learnable multi-stream residual transmission and discover that using two residual streams captures nearly all performance gains while yielding the best transferability to external corpora; notably, the model spontaneously allocates read/write paths based on the "masked" versus "visible" roles. Further causal intervention reveals a "preserve-transmit-reuse" mechanism: one stream preserves early-stage states, the other transmits visible evidence to masked targets, and later layers reuse the preserved states—a pattern prevalent in diffusion models but virtually absent in autoregressive models.

Critical strengths
  1. the experimental design is rigorous: all comparisons were conducted under matched parameter counts, data ordering, masking strategies, and training schedules, and were supplemented by autoregressive baselines and cross-corpus transfer evaluations, ensuring the reliability of the conclusions.
  2. the study goes beyond merely reporting performance gains; through systematic causal interventions—such as mean substitution, role swapping, intra-stream shuffling, and layer-wise patching—it traces these improvements back to specific mechanisms and validates, using models trained from scratch, that these mechanisms can indeed guide architectural design.
Critical weaknesses
  1. the architectural conclusions rest on a single model scale (337M parameters), a solitary training run, and a marginal Cross-Entropy (CE) difference of less than 0.06; the absence of variance estimation across multiple random seeds or an accounting of additional parameter, computational, and VRAM overheads leaves the claims of "two streams being sufficient" and the design being a "practical architecture" unsupported.

  2. the mechanistic analysis relies on causal interventions performed at only one checkpoint (n=2, t=0.5), while the crucial contrast—noting the phenomenon's prevalence across widths versus its near-absence in autoregressive settings—is supported merely by descriptive statistics regarding routing and similarity rather than replicated interventions across other widths or AR baselines; furthermore, the fact that the "Forward-Routing Ratio" (FRR) explains only half the performance gain, combined with the lack of generation quality metrics or downstream evaluations, undermines the persuasiveness of the study's core contribution: offering a "new perspective on understanding denoising computation."

Questions for authors

How much do parameters, FLOPs, memory usage, and wall-clock time increase for two-stream and multi-stream interfaces? Under iso-compute or iso-parameter conditions, does the two-stream model still outperform widened or deepened single-stream baselines?

The causal interventions in Sections 5–7 (role mean substitution, role swapping, intra-stream shuffling, source-to-target patching) were performed only on models with n=2 and t=0.5; could these interventions be repeated for n=3/4 and different t values ​​to support the conclusion that the mechanism is universal across widths?

FRR bridges only 50.2% of the gap between single-stream and two-stream models; where does the remaining half come from (token-dependent read/write directions? learnable stream transport?)? Have any ablations been conducted to isolate these individual factors?

1 comment

Thank you for highlighting our matched comparisons and causal interventions. We address each concern below.

[W1] Repeatability and scale. 337M runs share the FineWeb-Edu token pool, 53,400 updates, and fixed validation blocks and masks. Seeds vary initialization, block order, and training masks.

SeedStandard CELearned n=1n=1n=2n=2n=3n=3
13373.602603.577503.544603.55109
13383.594943.590623.539273.54524
13393.589283.584993.538573.54087
Mean ± sample SD3.59561 ± .006693.58437 ± .006583.54081 ± .003303.54573 ± .00513

Paired gain over standard is 0.05479 ± 0.00372 CE (5.33% lower reconstruction PPL). The original sweep favored n=3n=3 by 0.0052 CE. n=2n=2 captured 86.6% of the best gain beyond learned n=1n=1 and led mean PPL reduction across four corpora. It beats n=3n=3 on all three repeat seeds, supporting our choice of two streams.

At 1.028B. With 124,000 matched updates, n=2n=2 lowers validation CE from 3.27752 to 3.24224 (3.47% lower reconstruction PPL). Causal tests reproduce preservation, transfer, and late reuse (3q3F [W1]).

[Q1] Cost and matched compute. Standard forward matrix operations cost 0.739 GFLOPs/token at length 1,024. Router projections add 0.11%, 0.32%, and 1.13% for n=2,3,4n=2,3,4. Measurements use one H100 80 GB, BF16, and 65,536 tokens/update. Timing includes routing kernels and stream traffic. Throughput optimization is future work.

ModelParams (M)Training s/updatePeak allocated (GiB)
Standard336.10.64636.7
n=1n=1336.20.94545.2
n=2n=2336.61.68954.2
n=3n=3337.41.81263.3
n=4n=4340.42.15972.4

To test the architectural gain, we match widened and deepened learned n=1n=1 to n=2n=2 at 7.77×10187.77\times10^{18} estimated training FLOPs (3 × forward FLOPs/token × input tokens), sharing corpus, seed, and validation masks. n=2n=2 beats the better control by 0.03783 CE.

ModelParams (M)Tokens (B)Validation CE ↓Mean CE, 4 external corpora ↓
Widened, d=1152d=1152403.62.9363.612694.3969
Deepened, 20 blocks381.43.0613.582434.3870
Two streams336.63.5003.544604.3457

[W2/Q2] Causal tests across widths and mask ratios. We test n=2,3,4n=2,3,4 at t=0.1,0.5,0.9t=0.1,0.5,0.9 (128 blocks each). In all nine settings, replacing visible states with their own earlier states costs at most about 8% of shuffle damage. Clean-state rescue before late layers reverses 83–98%.

nnttEarly replacement ΔCEShuffle ΔCEDamage rescued
20.1+0.005+0.88186%
20.9+0.003+0.22283%
30.1+0.006+0.87391%
30.9+0.003+0.22988%
40.1+0.007+0.11696%
40.9+0.005+0.06298%

In all nine settings, role means beat global means and swapping roles raises CE for read and for write destination at fixed write volume. Masked late-read cuts cost 0.099–2.099 CE.

For n=3,4n=3,4 at each tt, we select 65–101 correctly predicted targets whose log-probability drops ≥0.5 nat on masking a visible source. Source patches recover more of the drop at 6.25% depth. At 75%, target patches lead by 34.5–70.1 pp. Paired 95% CIs exclude zero for both contrasts.

At matched positions, early-state replacement costs +.010/+.030/+.036 masked CE (diffusion, t=0.5t=0.5) and +.676/+.074/+.081 next-token CE (AR) for n=2/3/4n=2/3/4. The contrast is largest at n=2n=2.

[Q3] Dynamic reads close most of the FRR gap. Each variant restores one component, keeps dynamic write volume, and trains from scratch (seed 1337, 53,400 updates). Shares use this repeat's FRR-to-full n=2n=2 validation CE gap.

Component restoredFRR-to-full gap recovered
Token-dependent reads76.9%
Token-dependent write destination60.7%
Learned stream mixing48.7%

Recoveries can overlap. Roles largely organize reads and destinations (Fig. 2), and token-state refinements add gains. Mean transport can mix streams. Learning a mixer improves on FRR's identity.

[W2] Downstream and generation. We add tasks and open-ended generation beyond reconstruction CE. LAMBADA and ReCoRD use seed 1337. n=2n=2 gains over standard have paired 95% CIs (ReCoRD clustered by passage).

Benchmark337M gain, pp1B gain, pp
LAMBADA [1] (5,149 passages, target length given)+3.53 [2.48, 4.59]+6.74 [5.58, 7.89]
ReCoRD [2] (10,000 questions)+1.80 [0.99, 2.61]+0.98 [0.18, 1.78]
PIQA [3] (1,838 questions)+0.82 [0.04, 1.59]+1.96 [0.99, 2.93]

At 337M, n=2n=2 leads GenPPL in 9/9 settings and MAUVE in 7/9 (3q3F [W2]).

We will add these results and protocols to the revised paper.

[1] Paperno et al. (2016). https://arxiv.org/abs/1606.06031

[2] Zhang et al. (2018). https://arxiv.org/abs/1810.12885

[3] Bisk et al. (2020). https://arxiv.org/abs/1911.11641

Reviewer_3q3FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates role-dependent residual routing in diffusion language models using multi-stream residual connections, showing that masked and visible tokens learn different routing behaviors and that the two streams specialize in state preservation and information transfer.

Critical strengths
  1. Investigating whether different token roles require different residual routing strategies is an interesting and well-motivated idea.

  2. The mechanistic analysis is thorough, combining routing weight analysis with multiple causal interventions to understand the behavior of different residual streams.

Critical weaknesses
  1. All experiments are conducted on relatively small models (~337M parameters). I'm curious how the findings would change as model size increases. Evaluations on larger models would be valuable.

  2. Performance is mainly evaluated using masked-token CE and perplexity. The paper would benefit from evaluations on actual generation tasks, such as text generation, instruction following, or reasoning benchmarks.

  3. FRR recovers only about half of the additional improvement over the learned single-stream model, while its computational advantages remain unclear. Training and inference throughput comparisons are missing.

Questions for authors

See Weakness

3 comments

Thank you for your thoughtful feedback on our causal analysis. We address your three questions below.

[W1] Scaling to 1B. We trained 32-block, 1.028B standard-residual and two-stream models. At 124,000 matched updates, two streams lower validation CE from 3.27752 to 3.24224 (3.47% lower reconstruction perplexity, versus 5.71% for the original 337M checkpoint).

The same preservation–transfer–reuse organization emerges at both scales, with a longer state-preparation phase at 1B. Early attention writes to the stream that retains position-specific states. Evidence reaches masked targets while those states are preserved, and later layers reuse the retained visible states.

Probe337M1B
Share of first attention write gain to the preserved stream (masked positions)97.95%99.67%
Mid-depth replacement of visible states, own early state / within-role shuffle (ΔCE)+0.0059 / +0.3721+0.0076 / +1.9266
Source-patch recovery near 6% depth96.2%92.2%
Patch recovery at 75% depth, target / source99.6% / 26.3%99.8% / 31.2%
Late duplication into both streams at visible positions, preserved / other (ΔCE)+0.0104 / +0.1475+0.0225 / +0.2106

At 337M, the first block yields a reusable state. At 1B, using the later a8a_8 state instead of m1m_1 lowers replacement cost from +0.0164 to +0.0076 CE. The preserved stream is s1s_1 at 337M and s0s_0 at 1B.

Patch recovery measures how much lost target log-probability is restored after masking an influential visible source. At 1B, the target patch recovers 16.6% near 6% depth. By 75% depth, it leads the source patch by 68.6 pp (58 selected pairs, paired 95% CI [62.0, 75.4]).

[W2] Does this improve generation and downstream accuracy? At 337M, we generate 4,000 256-token continuations per condition from the same 128-token prefixes, with linear reveal, temperature 1, and top-p=0.9p=0.9. Across three corpora and 64/128/256 steps, n=2n=2 ranks first among five implementations in GPT-2 Large GenPPL in 9/9 conditions (3.4–7.2% below standard) and in MAUVE in 7/9.

Corpus, 256 stepsGenPPL ↓, standard → n=2n=2MAUVE ↑, standard → n=2n=2
FineWeb-Edu78.131 → 73.8820.751 → 0.780
C490.428 → 86.4560.666 → 0.677
OpenWebText140.312 → 130.2240.380 → 0.425

A separate generation evaluation with 256 matched prefixes per corpus and a fixed reveal order confirms lower 256-step GenPPL on C4 (92.45 → 87.45) and OpenWebText (127.59 → 122.23). At 128 steps, the C4 reduction is 7.94% (paired 95% CI [1.64, 13.70]).

On the replication checkpoints (seed 1337), we measure exact match on LAMBADA by greedy last-word decoding (5,149 passages, target length given) and on SuperGLUE ReCoRD by scoring masked candidates (10,000 validation questions). The separate PIQA evaluation uses 1,838 questions. The table gives n=2n=2 gains over standard with paired 95% CIs.

Benchmark337M gain, pp1B gain, pp
LAMBADA [1]+3.53 [2.48, 4.59]+6.74 [5.58, 7.89]
ReCoRD [2]+1.80 [0.99, 2.61]+0.98 [0.18, 1.78]
PIQA [3]+0.82 [0.04, 1.59]+1.96 [0.99, 2.93]

[W3] FRR as a mechanism ablation. FRR tests how much of the two-stream gain survives when read weights and write destinations depend only on whether a token is visible or masked. Write volume remains token-dependent, and stream mixing is fixed to identity. Trained from scratch, FRR recovers 50.2% of the full n=2n=2 CE gain over learned n=1n=1 and 71.6% over standard residual.

Training on H100 80 GB. All variants use BF16 and 65,536 input tokens per update.

VariantSeconds/update ↓Input tokens/s ↑Peak GPU memory (GiB) ↓
Standard0.646101.4k36.7
Learned n=1n=10.94569.4k45.2
Full n=2n=21.68938.8k54.2
FRR1.61240.7k48.2

FRR cuts step time by 4.5% and peak CUDA allocation by 11.1% relative to full n=2n=2.

Generation on H100 80 GB. We report medians of three compiled BF16 runs on one GPU, with 50 continuations of 256 tokens, shared C4 prefixes, and fixed reveal order.

Variants/batch, 128 steps ↓s/batch, 256 steps ↓Tokens/s, 256 steps ↑
Standard3.7297.2971754
Full n=2n=27.81715.418830
FRR5.19010.1711258
FRR with cached role weights4.5468.9151436

At 256 steps, FRR cuts batch time by 34.0% vs. full n=2n=2, or 42.2% with cached role weights. In separate eager BF16 tests, 128-step FRR takes 10.299 s vs. 11.065 s for 256-step standard. At those budgets, GenPPL falls from 127.59 to 112.88 on OpenWebText (11.5%, paired 95% CI [4.6, 18.5]) and from 92.45 to 88.12 on C4.

We will report all new results and protocols in the revised manuscript.

[1] Paperno et al. (2016). https://arxiv.org/abs/1606.06031

[2] Zhang et al. (2018). https://arxiv.org/abs/1810.12885

[3] Bisk et al. (2020). https://arxiv.org/abs/1911.11641

Thank you for the extensive experiments. I appreciate the larger-scale evaluation and the newly reported downstream results. The full two-stream model requires more than twice the training and inference time of the standard model, and I am still not sure whether the performance gains justify these additional costs. FRR reduces this overhead, but its practical advantage remains unclear. Setting aside these practical concerns, I personally like this paper from the perspective of mechanistic analysis and find its findings interesting. Given the remaining practical concerns, I tend to maintain my rating now

Thank you for recognizing the value of our mechanistic analysis, which we regard as the paper’s main contribution. To address your question about practical applicability, we measured the cost of two-stream mHC during both training and generation.

Training optimization. For mHC n=2, we combine the read, write, and mixing projections into one linear operation and express stream mixing as two weighted sums. This reduces the number of separate GPU operations while keeping all routing weights trainable. For FRR, we compute the read weights and write directions once per sublayer for each token role (masked or visible), then reuse them across positions. Write volume remains token-dependent.

VariantSeconds/update, before → after ↓Reduction in update time
Standard (reference)0.558—
mHC n=10.640 → 0.6104.7%
mHC n=21.193 → 0.69441.8%
FRR n=20.865 → 0.66623.0%

For mHC n=2, this brings training time to 24.4% above Standard, down from 113.8%. We report median time per optimizer update over ten measured updates after four warm-up updates. All variants use the same backbone, compiled BF16 training on H200, and 65,536 input tokens per update. The timings come from separate H200 allocations.

Inference optimization. At each denoising step, all variants apply the final normalization and vocabulary projection only to positions selected for reveal. mHC n=2 also uses the combined routing projection and direct stream mixing described above. For FRR, the checkpoint weights are fixed during generation, so we cache the masked and visible routing weights across denoising steps.

At budget 1/1, mHC n=2 takes 23.2–25.2% more generation time than optimized Standard, with 4.3–7.1% lower GenPPL. Each two-stream cell reports GenPPL followed by the increase in generation time over Standard at the same batch size, corpus, and budget.

BatchCorpusStandard GenPPL ↓mHC n=2 · GenPPL / extra timeFRR n=2 · GenPPL / extra time
32C488.0784.01 / +25.2%88.97 / +20.4%
32OpenWebText132.54126.88 / +25.0%129.49 / +20.4%
64C488.5283.97 / +23.5%88.84 / +18.6%
64OpenWebText136.73127.01 / +23.2%128.80 / +18.2%

At budgets 1/4 and 1/2, mHC n=2 also lowers GenPPL in all eight comparisons, with 20.7–24.7% additional generation time. We use compiled BF16 inference on H200 and score 1,024 continuations of 256 tokens per condition with GPT-2 Large. Models share the same prefixes. Budgets 1/4, 1/2, and 1/1 correspond to 64, 128, and 256 denoising steps. We calibrate median generation times using Standard reference runs on each GPU.

We will include these measurements and implementation details in the revised manuscript.

Reviewer_WRguOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces multi-stream residual computation in masked diffusion language models. The paper proposes learned residual routing and analyze how visible and masked tokens utilize different computational pathways. The results suggest that two residual streams improve modeling performance and exhibit role-dependent routing behavior.

Critical strengths
  1. The paper explores an interesting architectural direction for diffusion language models.
  2. The causal interventions provide useful insights on how residual streams organize information flow during denoising.
Critical weaknesses
  1. Unclear mathematical formulation in Section 3. The notation is insufficiently defined. In particular, nn in Equation (1) should be explicitly introduced as the number of persistent residual streams, rather than leaving readers to infer its meaning from Section 4. The motivation for Equation (2) is unclear. Although it decomposes write gains into volume and destination, the paper should explain why this decomposition is necessary and explicitly derive its connection to the mean residual update.

  2. The takeaway of Section 3 largely shows the definitions of write volume and destination, rather than presenting a finding.

  3. Suboptimal paper organization. The contributions are listed in the Related Work section, which is unconventional. They would be more appropriately placed in the Introduction to establish the paper's novelty and scope.

  4. Insufficient clarity and coherence of the methodology. Despite the interesting research direction, the overall presentation is difficult to follow. In particular, the progression from the architectural formulation to the mechanistic analysis lacks a clear logical structure. The authors should first establish the proposed architecture and its mathematical properties before introducing the corresponding analyses and interpretations.

Questions for authors

Figure 2 suggests that replacing read and write-destination controls with role-conditioned means produces relatively small performance degradation when these controls are modified individually. However, replacing all four controls jointly with role-conditioned means causes a substantially larger loss increase. Could the authors clarify how much of this degradation is attributable to removing token-dependent write volume, as opposed to interactions among the different routing components?

1 comment

Thank you for highlighting the causal interventions and for your feedback on the presentation. We clarify the paper's contribution and address each concern below.

Contribution and informativeness. Prior work left open whether learned multi-stream residual routing benefits masked diffusion language models and how denoising would organize the resulting streams. We show that two streams improve reconstruction and learn distinct routing patterns for masked and visible tokens. Causal interventions reveal a preservation–transfer–reuse mechanism: an early state is retained while visible evidence reaches masked targets, then reused in later layers. Both the performance gain and the qualitative mechanism recur at 1B (3q3F [W1]).

W1. Equation (1) follows the standard mHC convention of representing each token's residual state as a matrix xℓ∈Rn×dx_\ell\in\mathbb{R}^{n\times d}. We will define nn as the number of persistent residual streams and dd as the width of each stream immediately before Eq. (1).

Equation (2) separates the overall strength of a write from its allocation across streams so we can vary them independently in the later interventions. Lines 132–136 describe the effect on the stream mean. We will make the averaging step explicit. Let xˉℓ=1n⊤xℓ/n\bar{x}_\ell=\mathbf{1}_n^\top x_\ell/n and let Δℓ\Delta_\ell be the sublayer output for this token. The columns of the doubly stochastic transport matrix sum to one, so 1n⊤Hℓres=1n⊤\mathbf{1}_n^\top H_\ell^{\mathrm{res}}=\mathbf{1}_n^\top. Averaging Eq. (1) then gives

xˉℓ+1−xˉℓ=(1n∑igℓi)Δℓ=vℓΔℓ.\bar{x}_{\ell+1}-\bar{x}_\ell =\left(\frac{1}{n}\sum_i g_{\ell i}\right)\Delta_\ell =v_\ell\Delta_\ell.

Thus vℓv_\ell scales the sublayer output in the mean update, and the normalized write gains determine how that output is distributed across individual streams.

W2. Section 3 provides the mathematical setup. Its takeaway summarizes the distinction between write strength and destination so readers can interpret the interventions that follow. The empirical contribution comes from those interventions in Sections 5 and 6. We show that the corruption mask largely organizes read and write destination and that write strength depends on the individual token state even within a given role.

W3. The Introduction (lines 45–70) states the research question and previews our main findings. Our organization follows an established approach in which Related Work culminates in an explicit statement of contributions, allowing readers to see how our findings extend prior work on masked diffusion and learned residual routing. We appreciate the reviewer's suggestion and will make the contributions more explicit in the Introduction as well.

W4. Our paper is organized to establish the architecture before examining its learned behavior. Section 3 introduces the residual architecture and its mathematical properties, Section 4 motivates the focus on two streams through controlled width comparisons, and Sections 5–7 build on this foundation to analyze role-dependent routing, token-specific write strength, and the preservation–transfer–reuse mechanism through causal interventions. We will make this progression more explicit through clearer transitions between sections.

Q (Fig. 2). We extended the role-mean interventions in Appendix D with a 2×22\times2 ablation on the original two-stream checkpoint at t=0.5t=0.5 (1,024 evaluation sequences). One factor replaces write volume with its masked/visible role mean. The other jointly replaces read, write destination, and transport with their role means. All four conditions use the same sequences and masks, with role means estimated on disjoint calibration sequences. We use the submission's bounded destination coordinate (Appendix A). The table shows increases in masked-target CE over the intact model.

Read, write destination, and transportDynamic volumeRole-mean volume
Dynamic0.0000+0.2012
Role means+0.1197+1.6456

Averaging volume alone adds 0.2012 CE, consistent with the 0.2003 CE increase in Appendix D. Averaging the other three controls together, with volume dynamic, adds 0.1197 CE. If the effects were additive, averaging both groups would add 0.2012+0.1197=0.32090.2012+0.1197=0.3209 CE. The measured joint increase is 1.6456 CE. The excess, 1.6456−0.3209=1.32471.6456-0.3209=1.3247 CE, is the interaction (paired 95% sequence-bootstrap CI [1.3066, 1.3430]). This is 1.3247/1.6456≈80.51.3247/1.6456\approx80.5\\% of the joint increase. With the other controls set to role means, averaging volume adds 1.5259 CE.

We will include these clarifications and the factorial results in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.