acceptodds
Under review as a conference paper at ICLR 2027

Solve One Problem with Another: When Linear Attention Meets Masked Diffusion

Abstract

Long-context language modeling needs a global view of the answer and economical token-level computation. We present LaLanD (Linear Attention Language Diffusion), which unifies discrete diffusion with linear attention: diffusion organizes global answer refinement across rounds, while a bidirectional RWKV-7 denoiser integrates the full context to supply token predictions at every unresolved position. We connect the denoising objective to linear-cost sequence mixing and derive a cost–error criterion that states when an additional denoising round improves the generated distribution under fixed-partition stochastic sampling; Lean-verified proofs establish nested refinement and a training-coverage bound, and an enumerable task confirms the predicted regimes. Following this theory, we build a 3.3B model from RWKV-7 World v3 data with 30B tokens resampled from its own pretraining corpus. It raises the 20-task LongBench mean of its initialization from 11.8 to 19.5 with gains on 18 of 20 tasks, reaches a 68.3% ten-task average, and improves RACE and 4k bAbILong QA1, while retaining a fixed-size recurrent state and mixing for context length , answer length , and rounds, and decoding all answer positions in parallel within each round.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear Attention, Diffusion Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3BdFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LaLanD, that converts a causal RWKV-7 linear attention model into a bidirectional denoiser to perform linear-cost masked diffusion language model. To make it bidirectional, it learns from sequences aligned both forward and backward, combining them with sigmoid gate. The fusion initially gives more favor to the causal direction. During training, the model combines a cross entropy loss for clean token prediction at masked positions with a separate forward-only next token prediction loss to preserve its existing causal ability. Reverse attention and fusion gate parameters are first learned followed by upper layer parameters and the whole network. During inference, the unmasking policy used in transformer-based masked diffusion language models. Compared to original masked diffusion language models, the computation and memory complexity of LaLanD attention reduced from quadratic form to the linear form of the sequence length. Evaluation shows that LaLanD is more accurate than RWKV-7 and higher throughput compared to similar sized autoregressive models.

Critical strengths
  • Provided recipe to adapt RWKV-7 linear attention into masked diffusion language models.
  • Good motivation considering diffusion language models have O(L2)O(L^2) computation complexity which becomes the largest bottleneck in longer sequences.
Critical weaknesses
  • Applying linear attention to diffusion language model is not new. This paper does not explain why RWKV-7 should be adapted into diffusion language models.
  • Throughput results are compared with autoregressive models, which means the improvements are not solely due to LaLanD. Accuracy and throughput results should show the comparison between DiffuMamba, Triplet block Diffusion RWKV and LaLanD.
Questions for authors
  • Why is adapting RWKV-7 into diffusion language model a important problem?
  • Could you provide more comparisons with DiffuMamba and Triplet block Diffusion RWKV in terms of methods and in experiments?
  • Do we need both Δ\Delta and Δ\Delta % columns in Table 1?
  • How are the tok/s throughput measured in Table 3? Is ignore_eos flag activated to make sure all the models make the same number of tokens for fair comparison? I think 192.3/15.5 should match with 33.1/3.6.
  • As diffusion language models can decode all tokens in one iteration in the most extreme case, Table 3 should also compare accuracy results along with the throughput results.
5 comments

We thank the reviewer for the constructive suggestions regarding the motivation for bidirectional RWKV-7, comparisons with existing recurrent diffusion architectures, and the relationship between decoding efficiency and generation quality. We have strengthened the revised manuscript with new from-scratch experiments, controlled architectural ablations, and expanded comparisons with full-attention and recurrent diffusion models.

1. Why bidirectional RWKV-7 for masked diffusion?

LaLanD introduces a content-adaptive bidirectional recurrent denoiser that combines independently learned forward and reverse RWKV-7 Time Mix streams through token-wise, channel-wise directional gating. RWKV-7 provides input-dependent recurrent state updates and linear-cost sequence mixing, while its bidirectional construction enables masked positions to integrate preceding and succeeding contextual information within each denoising pass. Across diffusion rounds, newly revealed tokens provide additional conditioning information for subsequent predictions.

This architecture differs from DiffuMamba's bidirectional selective state-space modeling and B³D-RWKV's triplet-block bidirectional construction. LaLanD directly combines two RWKV-7 recurrent streams with learned directional fusion and noise-conditioned prediction.

We conducted a controlled ablation at the 0.24B configuration, with each variant trained on the same 2B-token budget.

Table R1. Controlled architectural ablation (Main Table 3). Four-corpus denoising objective in nats/token; lower is better.

ConfigurationPTB ↓LAMBADA ↓AG News ↓WikiText-2 ↓Mean ↓
Forward-only7.2365.5587.1296.8726.699
Bidirectional, without FiLM6.4634.5306.2916.0615.836
Full LaLanD6.4454.5056.2446.0005.799

Bidirectional recurrence reduces the mean objective from 6.699 to 5.836 nats/token, improving all four corpora. Noise-conditioned modulation further reduces it to 5.799. These controlled results identify bidirectional contextual integration as the principal source of improvement, while content-adaptive fusion provides learnable directional weighting within the recurrent denoiser.

2. Comparisons with full-attention and recurrent diffusion models

We extended LaLanD with from-scratch pretraining at 0.24B and 1.3B configuration scales using 25B and 120B DCLM tokens, respectively. The principal comparisons align training corpus, GPT-2 tokenizer, 1K context window, and training-token budget across LaLanD, full-attention diffusion, and DiffuMamba.

Table R2. From-scratch text modeling at 1.3B (Main Table 1(a)). Masked-diffusion perplexity estimates; lower is better.

ModelPTBWikiText-2LM1BLAMBADAAG NewsPubMedarXiv
MDLM101.1436.6399.7236.7367.5323.0323.25
DiffuMamba99.0434.74102.0136.0468.7522.3522.98
LaLanD48.1729.9763.9328.6246.7024.2724.90

LaLanD achieves the lowest perplexity estimate on five of seven corpora, including PTB, WikiText-2, LM1B, LAMBADA, and AG News. These results demonstrate the effectiveness of bidirectional RWKV-7 denoising against both dense-attention and selective state-space diffusion architectures.

We additionally evaluate zero-shot question answering under a common normalized conditional-choice scoring procedure.

Table R3. Extended normalized conditional-choice accuracy (Appendix E.3, %, higher is better).

ModelSizeARC-EasyARC-ChallengeHellaSwagPIQAOBQALogiQAMean6
LaLanD1.3B48.9525.0041.8466.8132.4025.6540.11
DiffuMamba2.74B49.4025.5042.7064.5029.8025.4039.55
B³D-RWKV7.2B53.3035.1044.2063.2035.0027.3043.02

LaLanD reaches 40.11% Mean6, compared with 39.55% for the 2.74B bidirectional Mamba-2 and 43.02% for the 7.2B B³D-RWKV. The larger-model comparison shares the normalized choice scorer; the 1.3B DCLM reference experiment is the aligned pretraining comparison. Architecturally, LaLanD uses independently learned forward/reverse RWKV-7 recurrent streams with adaptive fusion, versus Mamba's selective state-space updates and B³D-RWKV's triplet-block construction.

Table R4. Extended zero-shot QA (AppE.3).

ModelSizeARC-EasyARC-ChallengeHellaSwagPIQAOBQALogiQAMean6
DiffuMamba2.74B49.4025.5042.7064.5029.8025.4039.55
B³D-RWKV7.2B53.3035.1044.2063.2035.0027.3043.02
LaLanD1.3B48.9525.0041.8466.8132.4025.6540.11

LaLanD 1.3B achieves 40.1% mean accuracy, exceeding the 2.74B Diffmamba's 39.5% and approaching the 7.2B B³D-RWKV.

3. Long-context generalization

LaLanD performs bidirectional recurrent mixing with a fixed-size state, allowing each denoising pass to process a full canvas with linear sequence-mixing complexity.

We evaluate length extrapolation from 512 to 64K tokens using packed-text perplexity normalized to each model's 1K reference.

Table R5. Length extrapolation (Main Table 4). Relative perplexity; lower is better.

ModelSize5121K2K4K8K16K32K64K
LaLanD1.3B1.1151.0000.9470.9130.9080.9020.8950.901
Bidirectional Mamba-22.74B1.1391.0000.9280.9621.2892.4457.53721.720
B³D-RWKV7.2B1.1591.0000.9350.8970.8910.8620.8970.922

Although trained on 1K-token windows, LaLanD maintains a relative perplexity of 0.901 at 64K. The result demonstrates stable masked-token modeling across substantially longer sequences and complements the QA comparisons above.

4. Decoding efficiency and generation quality

We have reorganized the efficiency analysis to distinguish the computational cost of a single recurrent denoising pass from that of complete multi-round generation.

For context length HH, answer length NN, and KK refinement rounds, LaLanD requires O(K(H+N))O(K(H+N)) sequence-mixing operations. Its recurrent state size is independent of sequence length, while the total number of full-canvas passes determines the overall generation cost.

On one H100, LaLanD 1.3B processes a 1K-token canvas in 35.1 ms and a 64K-token canvas in 1,737.1 ms, corresponding to 29,209 and 37,728 input tokens/s, respectively.

Table R6. LaLanD 1.3B single-call processing (Appendix E.4).

Canvas lengthTime per callInput tokens/s
1K35.1 ms29,209
16K436.4 ms37,545
64K1,737.1 ms37,728

For the K=S/16K=S/16 full-canvas schedule, these single-call measurements also determine the corresponding schedule-based decoding rates. We distinguish these calculated rates from independently measured end-to-end request latency. The fixed-length calculation does not involve EOS-based early stopping.

The separate 3.3B World3 adaptation experiment uses a complete-request protocol with a 512-token prompt, 512-token answer budget, and 16 denoising rounds; its results are now reported in Appendix G.1.

Generation quality is evaluated through the text-modeling, QA, and continuation experiments rather than inferred from throughput alone. At 0.24B, LaLanD improves News and Wiki MAUVE and reduces four-gram repetition across News, Story, and Wiki relative to the locally trained DiffuMamba baseline (Main Table 2).

The revised main tables focus on cross-architecture modeling and downstream capabilities, while the supplementary World3 tables preserve detailed absolute and relative performance changes.

Taken together, the expanded comparisons demonstrate LaLanD's distinctive content-adaptive bidirectional RWKV-7 architecture, strong from-scratch performance at 1.3B, competitive downstream QA, and stable long-sequence modeling with linear-cost mixing per denoising pass.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our response and let us know whether any further clarification is needed.

Many thanks for your time and consideration.

The Authors of LaLand

Reviewer_d7EWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LaLanD (Linear Attention Language Diffusion), which combines discrete diffusion with a bidirectional RWKV-7 linear-attention denoiser. The motivation is full-sequence diffusion LMs re-read the context every round at quadratic cost, while linear-attention/recurrent models are cheap but decode one token at a time and never revise. LaLanD does full-canvas masked-diffusion refinement where each round is a linear-cost bidirectional recurrent scan predicting all masked positions, committing the most confident ones, yielding O(K(H+N)) mixing for K rounds, context H, answer N, with a fixed-size recurrent state. The model is 3.3B parameters, initialized from RWKV-7 World3. Experiments on various benchmarks and tasks verify the method's performance.

Critical strengths

1.Instantiating a D3PM-style denoiser as a bidirectional linear-attention (RWKV-7) stack to get full-context refinement at O(K(H+N)) cost with a fixed-size state is a sensible and well-motivated combination for long-context generation. The zero-init gated-fusion initialization that starts from a pretrained causal model is an elegant way to bootstrap a bidirectional diffusion denoiser from an AR checkpoint.

Critical weaknesses

1.The diffusion baselines are 8–16× smaller and differently trained, there is no same-scale dense-attention diffusion comparator (the method's actual target), and the 14× speedup rests on an implausibly slow RWKV-7 baseline in a single environment.

2.What are the results of a compute-matched causal control — i.e., RWKV-7 World3 further trained on the same 30B resampled tokens with its native AR objective — on LongBench and Avg10? Without this, how do you attribute the gains to recurrent diffusion rather than to additional training?

3.The Lean-verified guarantees assume fixed-partition stochastic sampling with true conditionals, but the system uses greedy confidence selection. What, if anything, do the formal results imply for the actual decoder?

4.Can you add a dense-attention masked-diffusion denoiser at comparable scale and training budget, so the "linear-cost vs. quadratic-cost diffusion" claim is tested directly?

Questions for authors

See weakness.

4 comments

We thank the reviewer for the thoughtful suggestions on matched-scale diffusion baselines, training controls, the scope of the refinement analysis, and decoding efficiency. In response, we have reorganized LaLanD around a new from-scratch study, controlled architectural ablations, and direct evaluations against full-attention and recurrent diffusion models. The revised manuscript distinguishes the main 0.24B/1.3B pretraining study from the earlier 3.3B World3 adaptation.

1. Scale-aligned diffusion comparisons and decoding efficiency

We now train LaLanD from random initialization at the 0.24B and 1.3B configuration scales on 25B and 120B DCLM tokens, respectively. This substantially extends the previous evaluation against much smaller diffusion models. The comparisons include the full-attention MDLM-style DiffuTran reference and the bidirectional DiffuMamba model at each scale, with aligned training corpus, GPT-2 tokenizer, 1,024-token context, and token budget. Their actual parameter totals are reported separately from nominal configuration labels.

Table R1. Model configurations and training budgets (Appendix E.1).

ModelConfigurationTotal parametersTraining tokens
MDLM-style DiffuTran0.24B0.279B25B
DiffuMamba0.24B0.300B25B
LaLanD0.24B0.312B25B
MDLM-style DiffuTran1.3B1.291B120B
DiffuMamba1.3B1.681B120B
LaLanD1.3B1.579B120B

We additionally extend the efficiency comparison beyond the original World3-versus-causal-RWKV throughput ratio. Our throughput analysis normalizes full-attention MDLM to 1.0×1.0\times within each decoding regime and calculates the corresponding ratios for the recurrent architectures.

Table R2. Throughput ratios relative to MDLM (MDLM = 1.0×1.0\times, higher is better). The LaLanD entries are our schedule-based projections, not separate measured end-to-end request speedups.

ModelConfigurationFull-canvas / 64KBlock-cache / 256K
MDLM (full attention)1.3B1.0×1.0\times1.0×1.0\times
DiffuMamba1.3B8.2×8.2\times2.3×2.3\times
LaLanD (projected)1.3B7.3×7.3\times2.5×2.5\times

Our inference benchmarks demonstrate substantial throughput improvements over full-attention MDLM. At 64K full-canvas decoding, LaLanD achieves 7.3× MDLM throughput, compared with 8.2× for DiffuMamba. At 256K block-cache decoding, LaLanD reaches 2.5×, exceeding DiffuMamba's 2.3×. These results establish competitive inference efficiency across both decoding regimes, alongside LaLanD's advantages in masked-token modeling and downstream question answering.

2. Causal continued-training control and the source of improvement

The original 3.3B World3 experiment adapts RWKV-7 using 30B additional tokens. Appendix G reports the resulting LongBench increase from 11.79 to 19.52 and Avg10 increase from 52.9% to 68.3% as outcomes of that complete adaptation recipe. The requested 30B-token causal World3 continuation is not included in these measurements. We use the new random-initialization comparisons and controlled component ablations to evaluate the architecture, rather than attributing the World3 adaptation gains specifically to diffusion.

The new from-scratch experiments isolate a different, central question: whether LaLanD's denoising architecture learns strong clean-token conditionals without an inherited World3 initialization. In addition, a controlled 0.24B ablation uses the same 2B-token budget for each architecture and isolates directional access and noise modulation.

Table R3. Controlled architecture ablation (Main Table 3). Masked denoising objective (nats/token, ↓); 2B training tokens per configuration.

ConfigurationPTBLAMBADAAG NewsWikiText-2Four-corpus mean
Forward-only7.2365.5587.1296.8726.699
Bidirectional, without FiLM6.4634.5306.2916.0615.836
Full LaLanD6.4454.5056.2446.0005.799

Bidirectional recurrence lowers the mean objective by 0.863 nats/token relative to forward-only processing, improving all four corpora; noise conditioning reduces it by another 0.037. Thus the controlled evidence identifies bidirectional contextual access as the main measured contributor, with noise modulation providing an additional gain. Taken together with the independently pretrained 1.3B result, these experiments establish the value of LaLanD's architecture across two scales.

3. Refinement theory and the implemented sampling policies

We have clarified the exact sampling process analyzed by the theory and the distinction between our two experimental regimes.

The new from-scratch LaLanD sampler performs absorbing-state stochastic reverse diffusion. At step kk, a still-masked position is revealed independently of prediction confidence with probability

preveal=tk−tk+1tk,p_{\mathrm{reveal}}=\frac{t_k-t_{k+1}}{t_k},

and its token is sampled from the denoiser's categorical prediction. Revealed tokens remain visible in subsequent rounds. Since the reveal decisions do not depend on sampled token values, conditioning on a realized reveal schedule gives an exogenous partition of positions of the type considered in Section 3.2.

Our analysis expresses the forward KL under such a fixed partition as

KL(Px∥Qθ,π)=Dπ(x)+Eθ,π(x),\mathrm{KL}(P_x\|Q_{\theta,\pi})=D_\pi(x)+E_{\theta,\pi}(x),

where DπD_\pi is the conditional dependence omitted by simultaneous independent predictions and Eθ,πE_{\theta,\pi} is the learned conditional-prediction error. An additional refinement improves forward KL when its recovered dependence exceeds the increase in conditional error. The Lean verification establishes the fixed-partition decomposition and monotonicity under true conditionals, together with a training-coverage bound for learned conditionals.

The earlier World3 adaptation also used confidence-ranked greedy commitment, in which reveal order depends on model predictions. We describe this as a distinct empirical decoding procedure: the fixed-partition theorem is used for the stochastic sampler and is not extended into a monotonicity guarantee for confidence-greedy commitment. Appendix A.5 provides finite-law verification of the criterion, while Appendix A.6 provides controlled neural evidence on conditional error and long-distance evidence use.

4. Direct full-attention masked-diffusion comparison

The requested dense-attention comparator is now part of the main 1.3B from-scratch evaluation. We compare LaLanD against the MDLM-style DiffuTran full-attention denoiser and bidirectional DiffuMamba, aligned in DCLM data, tokenizer, 1K training context, and 120B-token budget.

Table R4. Seven-corpus masked-diffusion perplexity at 1.3B (Main Table 1(a)). Lower is better.

ModelPTBWikiText-2LM1BLAMBADAAG NewsPubMedarXiv
MDLM-style DiffuTran101.1436.6399.7236.7367.5323.0323.25
DiffuMamba99.0434.74102.0136.0468.7522.3522.98
LaLanD48.1729.9763.9328.6246.7024.2724.90

LaLanD achieves the lowest perplexity estimate on five of seven held-out corpora; the Mamba model leads on PubMed and arXiv. We further test whether the modeling improvements translate into zero-shot selection accuracy.

Table R5. Four-task zero-shot QA at 1.3B (Main Table 1(b)). Accuracy (%, ↑) under the common conditional-choice scoring protocol.

ModelOpenBookQAHellaSwagPIQALogiQA
MDLM-style DiffuTran26.6134.9760.6421.86
DiffuMamba32.1237.7462.5629.87
LaLanD32.4041.8466.8125.65

LaLanD exceeds the full-attention reference on all four tasks and DiffuMamba on three of four, with a particularly clear advantage on HellaSwag and PIQA. Together, the PPL and QA results demonstrate that LaLanD's content-adaptive bidirectional RWKV-7 denoiser can combine strong conditional modeling and downstream prediction with linear per-pass sequence mixing.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our response and let us know whether any further clarification is needed.

Many thanks for your time and consideration.

The Authors of LaLand

Reviewer_zVraOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LaLanD couples absorbing-mask discrete diffusion with a bidirectional RWKV-7 denoiser so each refinement round can mix a full context+answer canvas at linear cost. The paper includes a cost–error analysis (Eq. 7) with Lean-checked propositions, a small enumerable 4-bit diagnostic (Table 5), and empirical results for a ~3.3B model adapted from RWKV-7 World3 on additional pretraining tokens (LongBench / capability averages; decode throughput on H100).

Critical strengths

Connecting masked diffusion to a linear recurrent backbone is a timely direction, and the Lean-checked propositions / Table 5 finite-law regimes show some analytical ambition beyond a pure engineering swap.

Critical weaknesses
  1. Presentation and Claim Discipline: The paper lacks a clear narrative thread, often oscillating between motivational slogans ("solve one problem with another"), dense theoretical derivations, and sprawling empirical tables. Critical limitations, such as the absence of compute-matched causal baselines, are relegated to late sections of the paper. Meanwhile, the abstract and introduction present diffusion-based refinement as a definitively proven mechanism rather than a tested hypothesis.

  2. Severe Confounders in Main Results: The reported performance gains are heavily confounded by added capacity and extensive continued training. LaLanD introduces approximately 0.4B extra parameters by duplicating attention weights into bidirectional pathways (App. C.1) and undergoes additional pretraining on 30B tokens. However, the core evaluations on LongBench and general capabilities in Tables 1 and 4 compare directly against the original, unadapted RWKV-7 World3 checkpoint. Without a compute- and parameter-matched causal baseline or a bidirectional autoregressive counterpart trained on the same 30B tokens, the attribution of improvements specifically to masked diffusion refinement remains unverified.

Questions for authors
  1. Can you report a compute-matched continued-training baseline that keeps causal RWKV-7 (same 30B tokens, same stages / LR) and, separately, a bidirectional RWKV without diffusion? How much of Table 1’s LongBench lift remains?

  2. For the deployed confidence-greedy schedule, do you have any diagnostic analogous to Table 5 showing that Eq. 7’s “helpful” regime holds under greedy commits?

  3. Please tighten the writing so theory assumptions (fixed partitions) and the deployed decoder (greedy commits) are stated up front as a limitation rather than elided.

  4. How does LaLanD compare to DiffuMamba / Triplet-Block Diffusion RWKV under matched data and decode budget?

4 comments

We thank the reviewer for the thoughtful feedback on experimental attribution, the relationship between refinement theory and sampling, and comparisons with recurrent diffusion models. The feedback has helped us reorganize the revised manuscript around three contributions: content-adaptive bidirectional RWKV-7 denoising, a new from-scratch evaluation across model scales, and a conditional-dependence/error analysis of iterative refinement. The revised paper places the random-initialization study at the center of its empirical evaluation and presents the earlier World3 adaptation as a separate supplementary experiment.

1. Training controls

We introduce from-scratch LaLanD pretraining at 0.24B and 1.3B configuration scales, comprising 0.312B and 1.579B total parameters and trained on 25B and 120B DCLM tokens, respectively. The corresponding full-attention diffusion and DiffuMamba references use matched training corpora, GPT-2 tokenization, 1,024-token windows, and scale-aligned token budgets. This evaluates the bidirectional recurrent denoiser from random initialization, separately from any World3 pretrained weights.

Table R1. Model scales and training budgets (Appendix E.1). The full-attention baseline is denoted MDLM in the revised manuscript; the corresponding reference configuration is called DiffuTran in Singh et al. (2026).

ModelConfigurationTotal parametersDataTraining tokens
MDLM (full-attention reference)0.24B0.279BDCLM25B
DiffuMamba0.24B0.300BDCLM25B
LaLanD0.24B0.312BDCLM25B
MDLM (full-attention reference)1.3B1.291BDCLM120B
DiffuMamba1.3B1.681BDCLM120B
LaLanD1.3B1.579BDCLM120B

We additionally isolate directional information flow and noise conditioning through an equal-budget 0.24B ablation, with 2B training tokens for each variant.

Table R2. Directionality and noise-conditioning ablation (Main Table 3). Four-corpus masked-token objective (nats/token, lower is better).

ConfigurationPTBLAMBADAAG NewsWikiText-2Mean
Forward-only7.2365.5587.1296.8726.699
Bidirectional, without FiLM6.4634.5306.2916.0615.836
Full LaLanD6.4454.5056.2446.0005.799

Bidirectional recurrence reduces the mean objective from 6.699 to 5.836, with improvements on all four corpora. Noise-conditioned modulation yields a further reduction to 5.799. These controls identify access to both contextual directions as the principal measured source of the gain.

The 3.3B World3 results are now confined to Appendix G as an adaptation study (LongBench mean: 11.79 to 19.52; Avg10: 52.9% to 68.3%). These gains describe the full 30B-token adaptation recipe, not a diffusion-only causal effect. The requested 30B-token matched causal and non-diffusion World3 controls are not part of this study; our architectural evidence instead comes from the from-scratch comparisons and equal-budget component ablations above.

2. Sampling procedure and the refinement mechanism

The revised main experiments use absorbing-state stochastic reverse sampling, as implemented for the from-scratch LaLanD models. At each denoising step, still-masked positions are selected for revelation according to the reverse diffusion schedule, independently of model confidence. Their token values are sampled from the learned categorical distributions, and revealed positions remain visible as conditioning for subsequent steps.

For successive reverse times tk−1>tkt_{k-1}>t_k, the reveal probability for an unresolved position is

rk=tk−1−tktk−1.r_k=\frac{t_{k-1}-t_k}{t_{k-1}}.

The reveal decisions depend only on the schedule and which positions remain masked, not on sampled token identities. Conditioning on the realized sequence of reveal groups therefore produces the fixed-partition stochastic sampling structure used in the analysis of Eq. (7). This directly aligns the primary LaLanD sampling implementation with the theoretical refinement framework.

The earlier World3 adaptation also explored confidence-ranked commitment. Its generated outputs are presented as supplementary results in Appendix G, separately from the stochastic sampling procedure used by our principal from-scratch experiments. The fixed-partition theorem applies to the stochastic sampler; it is not stated as a guarantee for confidence-ranked selection.

3. Alignment between theoretical assumptions and the implemented sampling procedure

We have revised Sections 3.2 and 3.5 to establish a direct connection between the theoretical assumptions and our primary LaLanD generation procedure.

Section 3.5 now specifies absorbing-state stochastic reverse sampling, where unresolved positions are revealed according to the diffusion time schedule, independently of generated token values. Conditioning on the realized reveal groups yields the fixed ordered partition assumed in Section 3.2. Consequently, our primary from-scratch generation procedure follows the stochastic factorization analyzed in Eq. (7).

For a fixed reveal partition π\pi, the forward KL decomposes as

=D_\pi(\mathbf{x})+E_{\theta,\pi}(\mathbf{x}),$$ where $D_\pi$ measures conditional dependence discarded through simultaneous prediction, and $E_{\theta,\pi}$ measures learned conditional prediction error. Additional refinement improves forward KL precisely when $$E_{\theta,\mathrm{multi}}-E_{\theta,\mathrm{one}} < D_{\mathrm{one}}-D_{\mathrm{multi}}.$$ **Appendix A.5 directly validates the criterion through exhaustive finite-law enumeration.** In the four-bit diagnostic, the one-call sampler has forward KL of 0.9893 nats. An informative two-call reveal reduces this to 0 under accurate conditionals, whereas an uninformative reveal retains 0.9893 and inaccurate conditional predictions can increase it to 2.6832. These results demonstrate the beneficial, neutral, and harmful refinement regimes characterized by the theoretical criterion. The Lean development further verifies the finite-law decomposition, the nested-refinement property under true conditionals, and the training-coverage bound for learned conditional prediction error. Together, these results connect the mathematical criterion, the implemented stochastic reverse sampler, and the conditions under which iterative refinement improves the generated distribution.

4. Comparison with DiffuMamba and B³D-RWKV

We extend the evaluation across full-attention diffusion, bidirectional state-space diffusion, and RWKV-based diffusion architectures. The from-scratch 1.3B comparison matches the DCLM corpus, GPT-2 tokenizer, 1K training window, and 120B-token pretraining budget; downstream comparisons with larger models use their reported model sizes and a common task-scoring protocol.

Table R4. Extended six-task normalized conditional-choice accuracy (Appendix E.3, Table 12). Accuracy (%; higher is better).

ModelSizeARC-EasyARC-ChallengeHellaSwagPIQAOBQALogiQAMean6
LaLanD1.3B48.9525.0041.8466.8132.4025.6540.11
DiffuMamba2.74B49.4025.5042.7064.5029.8025.4039.55
B³D-RWKV7.2B53.3035.1044.2063.2035.0027.3043.02

The LaLanD 1.3B mean is 40.11%, versus 39.55% for the 2.74B bidirectional Mamba-2 and 43.02% for B³D-RWKV at 7.2B. The bidirectional Mamba-2 model is an additional model-scale comparison, distinct from the DCLM-matched 1.3B DiffuMamba experiment.

Table R5. Extended six-task normalized conditional-choice accuracy (Appendix E.3, Table 12). Accuracy (%; higher is better).

ModelSizeARC-EasyARC-ChallengeHellaSwagPIQAOBQALogiQAMean6
LaLanD1.3B48.9525.0041.8466.8132.4025.6540.11
DiffuMamba2.74B49.4025.5042.7064.5029.8025.4039.55
B³D-RWKV7.2B53.3035.1044.2063.2035.0027.3043.02

The LaLanD 1.3B mean is 40.11%, versus 39.55% for the 2.74B bidirectional Mamba-2 and 43.02% for B³D-RWKV at 7.2B. The bidirectional Mamba-2 model is an additional model-scale comparison, distinct from the DCLM-matched 1.3B DiffuMamba experiment.

Table R6. Packed-text length extrapolation (Main Table 4(a)). Perplexity ratio relative to 1K (lower is better).

ModelSize1K4K8K16K32K64K
LaLanD1.3B1.0000.9130.9080.9020.8950.901
DiffuMamba2.74B1.0000.9621.2892.4457.53721.720
B³D-RWKV7.2B1.0000.8970.8910.8620.8970.922

LaLanD maintains a 64K-to-1K relative perplexity ratio of 0.901 after pretraining on 1K windows. This measures long-sequence masked-token modeling stability and complements the direct QA comparison.

Table R7. Measured inference throughput relative to full-attention MDLM (Main Table 4(b)). MDLM = 1.0×1.0\times in each measured regime; higher is better.

ModelSizeFull-canvas, 64KBlock-cache, 256K
MDLM1.3B1.0×1.0×
DiffuMamba1.3B8.2×2.3×
LaLanD1.3B7.3×2.5×

Our measurements show that LaLanD provides 7.3× MDLM throughput in the 64K full-canvas regime and 2.5× at 256K with block caching, compared with 8.2× and 2.3× for DiffuMamba, respectively. These two decoding regimes are reported separately. Together with the modeling results, they demonstrate LaLanD's accuracy–efficiency characteristics relative to dense-attention and recurrent diffusion baselines.

The architectural distinction is equally important: DiffuMamba employs bidirectional selective state-space updates, B³D-RWKV uses a triplet-block construction, and LaLanD combines independently learned forward and reverse RWKV-7 Time Mix streams through content-adaptive token-wise and channel-wise fusion. The controlled ablation in Table R2 identifies bidirectional contextual integration as the main measured source of improvement, while the new 1.3B results establish its effectiveness at larger scale.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our response and let us know whether any further clarification is needed.

Many thanks for your time and consideration.

The Authors of LaLand

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Responses

We thank all three reviewers for their thoughtful and constructive feedback. Their suggestions have helped us sharpen the central contributions of LaLanD, improve the clarity and organization of the manuscript, and strengthen our comparisons with leading diffusion language modeling approaches, including MDLM, DiffuMamba, and B³D-RWKV.

Building on these suggestions, we have substantially revised the manuscript and introduced a new from-scratch pretraining study at 0.24B and 1.3B model scales, using 25B and 120B DCLM tokens, respectively. Both models are trained from random initialization and compared with full-attention MDLM and bidirectional DiffuMamba under matched training corpora, tokenization, context lengths, and token budgets. These new experiments, together with expanded architectural ablations, zero-shot QA, document understanding, and long-context evaluations, provide a comprehensive assessment of LaLanD's content-adaptive bidirectional recurrent denoising architecture and its scaling behavior.


Shared experimental tables

Training configurations (revised manuscript Appendix E.1, Table 10, PDF p. 24)

ModelConfiguration (B)Total parameters (B)CorpusPretraining tokensTokenizerEvidence source
LaLanD0.240.312DCLM25BGPT-2This work
MDLM0.240.279DCLM25BGPT-2Published reference
DiffuMamba0.240.300DCLM25BGPT-2Published reference
LaLanD1.31.579DCLM120BGPT-2This work
MDLM1.31.291DCLM120BGPT-2Published reference
DiffuMamba1.31.681DCLM120BGPT-2Published reference

Table 1. Text modeling and zero-shot QA (revised manuscript PDF p. 8)

(a) Seven-corpus masked-diffusion perplexity estimate (lower is better)

ModelSize (B)TokensPTBWikiText-2LM1BLAMBADAAG NewsPubMedarXiv
MDLM1.3120B101.1436.6399.7236.7367.5323.0323.25
DiffuMamba1.3120B99.0434.74102.0136.0468.7522.3522.98
LaLanD1.3120B48.1729.9763.9328.6246.7024.2724.90

(b) Common-protocol zero-shot normalized conditional-choice accuracy (%, higher is better)

ModelSize (B)TokensOpenBookQAHellaSwagPIQALogiQA
BD3-LM0.17524B28.2032.5059.3027.80
LaLanD0.2425B27.4028.6459.0321.20
MDLM1.3120B26.6134.9760.6421.86
DiffuMamba1.3120B32.1237.7462.56**29.87 **
LaLanD1.3120B32.4041.8466.8125.65

At 1.3B, LaLanD leads the masked-diffusion perplexity estimates on PTB, WikiText-2, LM1B, LAMBADA, and AG News. The normalized choice results in (b).

Table 2. Native denoising and continuation (revised manuscript PDF p. 9)

(a) Native masked-token text objective (nats/token, lower is better)

ModelSize (B)PTBWikiText-2LAMBADAAG News
MDLM-OWT0.174.433.873.894.38
BD3-LM0.174.483.903.804.24
LaLanD0.245.034.483.854.98
Bidirectional Mamba-22.744.303.924.014.55
B3D-RWKV7.23.703.364.084.23
LaLanD1.33.873.403.353.84

(b) 0.24B corpus-conditioned continuations (16-token prefix, 64-token continuation)

ModelSize (B)News MAUVE ↑News Rep-4 ↓Story MAUVE ↑Story Rep-4 ↓Wiki MAUVE ↑Wiki Rep-4 ↓
DiffuMamba (local)0.240.0420.4840.3650.4230.1560.424
LaLanD0.240.0510.4690.3550.3760.1840.423

News and Story use 5,000 examples each; Wiki uses 1,792. LaLanD achieves higher MAUVE on News and Wiki, with lower four-gram repetition on all three corpora. Table 2(a) reports a native denoising objective in nats/token, distinct from the perplexity estimates in Table 1(a).

Main Table 3 — Controlled component ablation

0.24B configurations, 2B training tokens per variant; four-corpus masked-token objective (nats/token, ↓).

ConfigurationPTBLAMBADAAG NewsWikiText-2Mean
Full LaLanD6.4454.5056.2446.0005.799
Forward-only7.2365.5587.1296.8726.699
Without noise modulation (FiLM)6.4634.5306.2916.0615.836

Bidirectionality reduces the mean objective from 6.699 to 5.836, and noise-conditioned modulation further reduces it to 5.799. The controlled comparison identifies bidirectional contextual access as the largest measured contribution.

Main Table 4 — Length extrapolation and inference efficiency

(a) Packed-text perplexity ratio relative to 1K (↓)

ModelSize5121K2K4K8K16K32K64K
LaLanD1.3B1.1151.0000.9470.9130.9080.9020.8950.901
Bidirectional Mamba-22.74B1.1391.0000.9280.9621.2892.4457.53721.720
B³D-RWKV7.2B1.1591.0000.9350.8970.8910.8620.8970.922

(b) Measured inference throughput relative to MDLM (↑; MDLM = 1.0×1.0\times)

ModelConfigurationFull-canvas 64KBlock-cache 256K
MDLM (full attention)1.3B1.0×1.0\times1.0×1.0\times
DiffuMamba1.3B8.2×8.2\times2.3×2.3\times
LaLanD1.3B7.3×7.3\times2.5×2.5\times

LaLanD maintains a 64K-to-1K packed-text perplexity ratio of 0.901, demonstrating stable long-sequence masked-token modeling. Our measured inference benchmarks use MDLM as the 1.0×1.0\times reference in each setting. At 64K full-canvas decoding, LaLanD achieves 7.3×7.3\times MDLM throughput, compared with 8.2×8.2\times for DiffuMamba; at 256K block-cache decoding, LaLanD reaches 2.5×2.5\times, compared with 2.3×2.3\times for DiffuMamba. These results show competitive throughput across full-canvas and cached decoding, complementing LaLanD's masked-token modeling and QA performance.

AuthorsOpenReview ↗

Clarifying the Motivation and Contributions of LaLanD

We thank all reviewers for their thoughtful feedback. We would like to further clarify the research motivation and architectural contributions of LaLanD, particularly how our approach differs from existing recurrent diffusion language models.

Our research starts from the question of how recurrent memory can efficiently support long-context masked diffusion. We focus on two complementary challenges: how to preserve and update contextual information across distant token positions within each denoising pass, and how to integrate information from both directions as the answer canvas is progressively revealed. These challenges motivate our design of a bidirectional recurrent denoiser with expressive memory updates, content-adaptive directional fusion, and diffusion noise conditioning.

At the recurrent-state level, LaLanD builds on RWKV-7's generalized delta-rule mechanism. Each directional stream maintains a matrix-valued state updated through

Si=Si−1[diag⁡(wi)+aibi⊤]+viki⊤.S_i=S_{i-1}\left[\operatorname{diag}(w_i)+a_i b_i^\top\right]+v_i k_i^\top.

This transition combines channel-dependent decay with an input-dependent rank-one correction, allowing the model to selectively transform previously stored information along different feature directions.

This differs from the Mamba-2 SSD recurrence employed by DiffuMamba, which takes the form

Si=αiSi−1+Bixi⊤,S_i=\alpha_i S_{i-1}+B_i x_i^\top,

where αi\alpha_i is an input-dependent scalar state-retention coefficient within each head. Consequently, the existing memory is propagated through cumulative scalar transitions. Both architectures provide linear-cost sequence mixing, but they differ in how recurrent memory is retained, transformed, and updated as new contextual information arrives.

Our second architectural contribution concerns bidirectional information integration. DiffuMamba combines its directional representations through additive fusion,

hi=hi→+hi←,h_i=h_i^\rightarrow+h_i^\leftarrow,

whereas LaLanD introduces independently learned forward and backward RWKV-7 Time Mix streams with content-adaptive directional gating:

g_i &= \operatorname{sigmoid}(W_g z_i+b_g),\\ h_i &= g_i\odot h_i^\rightarrow +(1-g_i)\odot h_i^\leftarrow. \end{aligned}$$ This formulation allows each token and feature channel to adjust the relative contributions of preceding and succeeding contextual information. Noise-conditioned modulation further adapts these representations to the current diffusion step. Together, these components define LaLanD's bidirectional recurrent denoising architecture. We evaluate this design through new from-scratch pretraining experiments at 0.24B and 1.3B, using aligned DCLM corpora, tokenization, context lengths, and training-token budgets against full-attention MDLM and DiffuMamba. At the 1.3B configuration scale, LaLanD achieves the lowest masked-diffusion perplexity estimates on five of seven corpora, including 48.17 versus 99.04 on PTB compared with DiffuMamba. Our separate long-context evaluation further shows that LaLanD maintains a 64K-to-1K relative perplexity ratio of 0.901, compared with 21.720 for the evaluated 2.74B bidirectional Mamba-2 checkpoint. These results demonstrate distinct length-generalization behavior and support our investigation of richer recurrent memory updates for diffusion. The controlled 0.24B experiments further identify the contribution of bidirectional contextual integration. Under an identical 2B-token training budget, the four-corpus denoising objective decreases from 6.699 with forward-only processing to 5.836 with bidirectional recurrence, and further to 5.799 with noise-conditioned modulation. Beyond the architectural design, LaLanD provides a conditional-dependence/error analysis of iterative refinement. Our analysis characterizes when the information recovered by additional reveal rounds outweighs the error of learned conditional predictions. The resulting criterion is supported by Lean-verified propositions and finite-law validation, with the stochastic absorbing-state sampling procedure specified in the revised Method. The revised manuscript therefore brings together three contributions: a content-adaptive bidirectional RWKV-7 denoising architecture, controlled from-scratch evaluation across model scales, and a mathematical analysis of iterative refinement. We believe these contributions establish a distinct approach to combining recurrent memory dynamics and masked diffusion for efficient long-context language modeling. We appreciate the reviewers' feedback and consideration of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.