- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LaLanD, that converts a causal RWKV-7 linear attention model into a bidirectional denoiser to perform linear-cost masked diffusion language model. To make it bidirectional, it learns from sequences aligned both forward and backward, combining them with sigmoid gate. The fusion initially gives more favor to the causal direction. During training, the model combines a cross entropy loss for clean token prediction at masked positions with a separate forward-only next token prediction loss to preserve its existing causal ability. Reverse attention and fusion gate parameters are first learned followed by upper layer parameters and the whole network. During inference, the unmasking policy used in transformer-based masked diffusion language models. Compared to original masked diffusion language models, the computation and memory complexity of LaLanD attention reduced from quadratic form to the linear form of the sequence length. Evaluation shows that LaLanD is more accurate than RWKV-7 and higher throughput compared to similar sized autoregressive models.
Critical strengths
- Provided recipe to adapt RWKV-7 linear attention into masked diffusion language models.
- Good motivation considering diffusion language models have computation complexity which becomes the largest bottleneck in longer sequences.
Critical weaknesses
- Applying linear attention to diffusion language model is not new. This paper does not explain why RWKV-7 should be adapted into diffusion language models.
- Throughput results are compared with autoregressive models, which means the improvements are not solely due to LaLanD. Accuracy and throughput results should show the comparison between DiffuMamba, Triplet block Diffusion RWKV and LaLanD.
Questions for authors
- Why is adapting RWKV-7 into diffusion language model a important problem?
- Could you provide more comparisons with DiffuMamba and Triplet block Diffusion RWKV in terms of methods and in experiments?
- Do we need both and % columns in Table 1?
- How are the tok/s throughput measured in Table 3? Is ignore_eos flag activated to make sure all the models make the same number of tokens for fair comparison? I think 192.3/15.5 should match with 33.1/3.6.
- As diffusion language models can decode all tokens in one iteration in the most extreme case, Table 3 should also compare accuracy results along with the throughput results.
5 comments
3. Long-context generalization
LaLanD performs bidirectional recurrent mixing with a fixed-size state, allowing each denoising pass to process a full canvas with linear sequence-mixing complexity.
We evaluate length extrapolation from 512 to 64K tokens using packed-text perplexity normalized to each model's 1K reference.
Table R5. Length extrapolation (Main Table 4). Relative perplexity; lower is better.
| Model | Size | 512 | 1K | 2K | 4K | 8K | 16K | 32K | 64K |
|---|---|---|---|---|---|---|---|---|---|
| LaLanD | 1.3B | 1.115 | 1.000 | 0.947 | 0.913 | 0.908 | 0.902 | 0.895 | 0.901 |
| Bidirectional Mamba-2 | 2.74B | 1.139 | 1.000 | 0.928 | 0.962 | 1.289 | 2.445 | 7.537 | 21.720 |
| B³D-RWKV | 7.2B | 1.159 | 1.000 | 0.935 | 0.897 | 0.891 | 0.862 | 0.897 | 0.922 |
Although trained on 1K-token windows, LaLanD maintains a relative perplexity of 0.901 at 64K. The result demonstrates stable masked-token modeling across substantially longer sequences and complements the QA comparisons above.
4. Decoding efficiency and generation quality
We have reorganized the efficiency analysis to distinguish the computational cost of a single recurrent denoising pass from that of complete multi-round generation.
For context length , answer length , and refinement rounds, LaLanD requires sequence-mixing operations. Its recurrent state size is independent of sequence length, while the total number of full-canvas passes determines the overall generation cost.
On one H100, LaLanD 1.3B processes a 1K-token canvas in 35.1 ms and a 64K-token canvas in 1,737.1 ms, corresponding to 29,209 and 37,728 input tokens/s, respectively.
Table R6. LaLanD 1.3B single-call processing (Appendix E.4).
| Canvas length | Time per call | Input tokens/s |
|---|---|---|
| 1K | 35.1 ms | 29,209 |
| 16K | 436.4 ms | 37,545 |
| 64K | 1,737.1 ms | 37,728 |
For the full-canvas schedule, these single-call measurements also determine the corresponding schedule-based decoding rates. We distinguish these calculated rates from independently measured end-to-end request latency. The fixed-length calculation does not involve EOS-based early stopping.
The separate 3.3B World3 adaptation experiment uses a complete-request protocol with a 512-token prompt, 512-token answer budget, and 16 denoising rounds; its results are now reported in Appendix G.1.
Generation quality is evaluated through the text-modeling, QA, and continuation experiments rather than inferred from throughput alone. At 0.24B, LaLanD improves News and Wiki MAUVE and reduces four-gram repetition across News, Story, and Wiki relative to the locally trained DiffuMamba baseline (Main Table 2).
The revised main tables focus on cross-architecture modeling and downstream capabilities, while the supplementary World3 tables preserve detailed absolute and relative performance changes.
Taken together, the expanded comparisons demonstrate LaLanD's distinctive content-adaptive bidirectional RWKV-7 architecture, strong from-scratch performance at 1.3B, competitive downstream QA, and stable long-sequence modeling with linear-cost mixing per denoising pass.
As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our response and let us know whether any further clarification is needed.
Many thanks for your time and consideration.
The Authors of LaLand
We thank the reviewer for the constructive suggestions regarding the motivation for bidirectional RWKV-7, comparisons with existing recurrent diffusion architectures, and the relationship between decoding efficiency and generation quality. We have strengthened the revised manuscript with new from-scratch experiments, controlled architectural ablations, and expanded comparisons with full-attention and recurrent diffusion models.
1. Why bidirectional RWKV-7 for masked diffusion?
LaLanD introduces a content-adaptive bidirectional recurrent denoiser that combines independently learned forward and reverse RWKV-7 Time Mix streams through token-wise, channel-wise directional gating. RWKV-7 provides input-dependent recurrent state updates and linear-cost sequence mixing, while its bidirectional construction enables masked positions to integrate preceding and succeeding contextual information within each denoising pass. Across diffusion rounds, newly revealed tokens provide additional conditioning information for subsequent predictions.
This architecture differs from DiffuMamba's bidirectional selective state-space modeling and B³D-RWKV's triplet-block bidirectional construction. LaLanD directly combines two RWKV-7 recurrent streams with learned directional fusion and noise-conditioned prediction.
We conducted a controlled ablation at the 0.24B configuration, with each variant trained on the same 2B-token budget.
Table R1. Controlled architectural ablation (Main Table 3). Four-corpus denoising objective in nats/token; lower is better.
Bidirectional recurrence reduces the mean objective from 6.699 to 5.836 nats/token, improving all four corpora. Noise-conditioned modulation further reduces it to 5.799. These controlled results identify bidirectional contextual integration as the principal source of improvement, while content-adaptive fusion provides learnable directional weighting within the recurrent denoiser.
2. Comparisons with full-attention and recurrent diffusion models
We extended LaLanD with from-scratch pretraining at 0.24B and 1.3B configuration scales using 25B and 120B DCLM tokens, respectively. The principal comparisons align training corpus, GPT-2 tokenizer, 1K context window, and training-token budget across LaLanD, full-attention diffusion, and DiffuMamba.
Table R2. From-scratch text modeling at 1.3B (Main Table 1(a)). Masked-diffusion perplexity estimates; lower is better.
LaLanD achieves the lowest perplexity estimate on five of seven corpora, including PTB, WikiText-2, LM1B, LAMBADA, and AG News. These results demonstrate the effectiveness of bidirectional RWKV-7 denoising against both dense-attention and selective state-space diffusion architectures.
We additionally evaluate zero-shot question answering under a common normalized conditional-choice scoring procedure.
Table R3. Extended normalized conditional-choice accuracy (Appendix E.3, %, higher is better).
LaLanD reaches 40.11% Mean6, compared with 39.55% for the 2.74B bidirectional Mamba-2 and 43.02% for the 7.2B B³D-RWKV. The larger-model comparison shares the normalized choice scorer; the 1.3B DCLM reference experiment is the aligned pretraining comparison. Architecturally, LaLanD uses independently learned forward/reverse RWKV-7 recurrent streams with adaptive fusion, versus Mamba's selective state-space updates and B³D-RWKV's triplet-block construction.
Table R4. Extended zero-shot QA (AppE.3).
LaLanD 1.3B achieves 40.1% mean accuracy, exceeding the 2.74B Diffmamba's 39.5% and approaching the 7.2B B³D-RWKV.