Role-Conditioned Residual Computation in Diffusion Language Models

Text diffusion is emerging as a competitive alternative to autoregressive language modeling: it reconstructs masked tokens over successive steps and can predict many positions in parallel. Yet standard Transformer backbones carry visible evidence and evolving masked predictions through a single residual state at each position. We ask whether learned routes through depth can better support denoising. We add persistent residual streams with token-conditioned read, transport, and write maps around unchanged Transformer branches. In matched 337M-parameter pretraining on 3.5B tokens, two streams capture 86.6% of the best additional in-domain gain beyond learned single-stream control and achieve the strongest mean transfer across four external corpora. Without assigned stream roles, visible and masked positions learn different read and write routes. Causal interventions establish the importance of these routes and of token-specific write strength; a role-factorized model trained from scratch closes 50.2% of the gap from one stream to the full two-stream model. Stream-state interventions reveal a preservation–transfer–reuse mechanism in the two-stream model: one stream retains an input-dependent early state, the other carries visible-source influence to masked targets, and late layers reuse the retained state. The preservation-and-reuse signature recurs across diffusion widths and is nearly absent in autoregressive controls. We hope these findings establish multi-stream residual transport as both a practical architecture for text diffusion and a useful lens for understanding how denoising models organize computation between evidence and prediction.