acceptodds
Under review as a conference paper at ICLR 2027

Causal Inductive Biases for In-Context Learning in Diffusion Language Models

Abstract

Why does next-token pretraining produce in-context learners? Autoregressive (AR) training directly supervises prediction from the observed prefix, whereas masked and diffusion objectives distribute supervision across many observation patterns. We connect AR pretraining to Bayes-optimal episodic prediction and quantify the uncertainty reduction obtained by observing the future. These results motivate a supervision-allocation hypothesis: training more often on earlier context and later targets should promote sequential contextual adaptation. We test this hypothesis with matched AR, MLM and diffusion transformers, then introduce a minimal position- dependent corruption process. This intervention improves few-shot adaptation across diverse algorithmic tasks, survives native iterative generation under multiple diffusion objectives, and transfers to LLaDA-8B through continued pretraining. Together, these results identify training-objective factorisation as a central inductive bias for the emergence of in-context learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
In-Context-Learning, Discrete Diffusion LLMs, Meta-Learning, Inductive Bias, role of objective functions

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aaipOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether the distribution of conditional prediction problems presented during pretraining influences in-context learning. It connects population autoregressive log loss to Bayes-optimal episodic prediction and characterizes the entropy difference between prefix-only and future-conditioned prediction.

Motivated by this perspective, it introduces position-dependent corruption that makes earlier tokens relatively cleaner and later tokens more frequently masked. Experiments compare training objectives in matched small models, evaluate synthetic contextual tasks and constrained iterative decoding, and apply the intervention through 500M-token continued pretraining of LLaDA-8B.

The intervention improves several contextual evaluations. At 8B scale, its relative effect changes with query position in a manner consistent with the imposed corruption profile. A positional loss-weighting control does not reproduce this pattern. These results support a useful supervision-allocation hypothesis, although the theoretical interpretation and the isolation of conditioning structure from other training changes require qualification.

Critical strengths
  1. The training intervention provides a useful controlled test of supervision allocation. The positional corruption profile changes the training observation patterns without introducing a new architecture or changing the decoder within the principal comparisons. In LLaDA-8B, FORWARD and SYM receive 500M tokens of continued pretraining under matched data and schedules. At K=16, retrieval accuracy is 0.458 versus 0.422, while the paired margin effect changes from negative to positive as the query moves through the imposed positional profile. This signed positional pattern is stronger evidence of an intervention-specific effect than aggregate improvement alone.

  2. The study evaluates several distinct manifestations of contextual behavior and reports important failures. The synthetic battery includes retrieval, permutation inference, conflicting bindings, and finite-state prediction. At the largest reported context sizes, the alpha=1 model improves Key–Value accuracy from 0.234 to 0.354 and Cipher accuracy from 0.140 to 0.222. Under the specified 64-step contextual decoding protocol, MDLM OverrideKV lift improves from 0.332 to 0.510 at K=32. The appendices also report near-chance tasks, failures of the stronger tilt, disagreements between scoring and decoding, and the restricted GIDD replication. This makes the empirical contribution more credible than a presentation based only on favorable evaluations.

Critical weaknesses
  1. The theoretical interpretation is stronger than the formal result, and one strictness claim needs correction. Theorem 2.1 correctly identifies a conditional mutual information as an entropy gap. However, higher uncertainty when the future is unobserved is not itself evidence that a diffusion model cannot directly learn the correct prefix conditional, nor a bound on sample complexity or inference cost. Appendix O acknowledges this limitation, but the "barrier" terminology and parts of the main discussion still suggest a stronger explanation of the empirical AR–diffusion difference.

    In addition, Proposition C.2 does not follow from the assumptions in C.1 as written. A nondegenerate posterior over latent hypotheses and globally non-identical component distributions are insufficient. For example, let p_1=p_2 differ from p_3=p_4, and let the prefix reveal which pair contains the latent hypothesis. The posterior over hypotheses remains nondegenerate, but all hypotheses remaining possible after a given prefix induce the same token distribution. The target and future are then conditionally independent, so the claimed strict entropy gap is zero. Strictness requires residual uncertainty over distinct predictive distributions.

  2. The key control does not yet fully isolate conditioning structure as the explanation. Appendix G reports that mean-normalized MDLM corruption produces a realized masking rate of approximately 0.452 rather than 0.5, while GIDD uses a separately reduced corruption budget. Less corruption changes task difficulty as well as context availability; its direction is not automatically conservative for every learning comparison. The 8B loss-weighted control is well motivated, but matches expected target mass only away from clipping and is reported as unpaired condition means without uncertainty estimates. Its failure to reproduce FORWARD supports the proposed explanation, but does not justify the categorical statement that target allocation is not a cause and context allocation is the cause.

Questions for authors
  1. Theoretical scope and strictness: Would you clarify that Theorem 2.1 is an information identity motivating the supervision-allocation hypothesis, rather than a computational or statistical barrier for diffusion learning? Please also strengthen the assumptions of Proposition C.2 to require residual posterior variation over distinct predictive distributions; the assumptions in C.1 permit the zero-gap counterexample described above.

  2. Mechanism-discriminating control: Can the loss-weighted arm be evaluated on the same episodes as SYM and FORWARD, with paired uncertainty estimates? Please distinguish what is established by the current interior-matched control from what remains unresolved because of clipping and changed realized corruption. If an exactly clipping-aware target-matched or realized-corruption-matched control is unavailable, please narrow the causal attribution accordingly.

  3. Interpretation of iterative contextual performance: Please report acc(real) and acc(control) separately for the principal OverrideKV results, rather than lift alone. A larger difference can arise through changes in either component. Please also specify whether candidate restriction is applied during denoising or only at final selection, and label the principal result explicitly as performance under the stated constrained decoding protocol.

3 comments

We thank the reviewer for an exceptionally careful reading of the paper, including the appendices. The counterexample to Proposition C.2 made the result tighter, and the two requests on OverrideKV reporting and on isolating conditioning structure led to the most significant changes in the revision: a rewritten Section 6.1 with a new figure reporting accuracy, and the new direction controls in Section 7.


W1 / Q1a: Scope of Theorem 2.1

We agree. Section 2.2 now states directly after the theorem that "barrier" denotes only the additional uncertainty when the future is unobserved, that the identity establishes no computational or statistical separation between AR and diffusion learning, and that it motivates the supervision-allocation hypothesis; the AR–diffusion difference is attributed only through the causal experiments. The Discussion is revised accordingly.


W1 / Q1b: Proposition C.2

The counterexample is correct. The construction in Appendix C.1 now assumes that, with positive probability over prefixes, the posterior places mass on two hypotheses with distinct predictive distributions ph≠ph′p_h \neq p_{h'}. The proof now uses this condition directly: given hh and the prefix, the next and first future tokens are independent draws from php_h, so their covariance at any token aa equals the posterior variance of ph(a)p_h(a), which is positive under the condition; hence I(Y;F1∣C)>0I(Y; F_1 \mid C) > 0. The condition excludes the reviewer's construction, where all hypotheses compatible with a prefix share one predictive distribution.


W2 / Q2: Isolating conditioning structure

We agree that the earlier controls did not fully isolate conditioning structure, and we have added two new LLaDA-8B controls. SHUFFLED applies the α=1\alpha=1 rates to a fresh random permutation of positions for every sequence; conditioned on the noise level it has exactly the same multiset of rates, mask-count distribution, realised budget and clipping as α=1\alpha=1, but no direction. REVERSE keeps the rates, budget and clipping and flips the orientation. The loss-weighted model is now evaluated on the same 500 examples per KK as all other models, with paired 5000-resample bootstrap intervals (Appendix G.1).

What the controls establish (Figure 5, Appendix G.1): at q=2048q=2048, only α=1\alpha=1 gains as competing bindings increase (ΔR=+0.334\Delta R = +0.334 and +0.289+0.289 at K=8, 16K = 8,\, 16). SHUFFLED is worse than α=0\alpha=0 from K=4K=4 on, so neither the reduced realised budget nor the heterogeneity of rates produces the gain. REVERSE is worse under interference. Loss weighting on symmetric inputs is slightly negative throughout. We have narrowed the attribution accordingly: the forward direction of the corruption profile is necessary for the interference gain; matched budget, heterogeneity or target weighting is not sufficient.

What remains open: the loss-weighted match is approximate where corruption probabilities are clipped, although SHUFFLED controls clipping exactly; intervals are over examples, not training runs; and the realised-budget difference the reviewer notes for the 170M MDLM and GIDD models is controlled at 8B but not at small scale, where we describe the models as nominally rather than exactly budget-matched (Appendix G).


Q3: OverrideKV under native generation

The concern was well founded, since a difference in lift can move through either component, so we tested it directly. Section 6.1 and Appendix L.1 have been rewritten. Figure 4(a, b) now plots exact-match accuracy on the real prompts, and Table 8 reports acc(real), acc(control) and lift separately for the principal variants. Control accuracy never exceeds 0.010 for any model or setting, so lift is within a percentage point of acc(real) and the gains come from the real prompts.

No candidate restriction is applied at any point in generation: intermediate proposals are drawn over the full vocabulary, and the final step is an unrestricted greedy argmax. Candidate labels are used only in the one-step scoring of Section 5. The principal result is labelled throughout as accuracy under clamped-prefix, remasked-query, 64-step greedy decoding over the unrestricted vocabulary; Appendix L.1 reports the full decoding grid.


We are grateful for a review that substantially improved the paper, and we would be glad to clarify anything further.

Reviewer_aaipOpenReview ↗

Response to Authors

Dear authors,

Thank you for your reply and effort. After reading your responses and clarifications, I think the scope of THM 2.1 and the concern about OverrideKV under native generation are clear to me. After considering the responses carefully, I decide to maintain my score 3.

AuthorsOpenReview ↗

Thank you, and summary of new results strengthening the contribution

Dear Reviewer aaip,

Thank you for your valuable feedback and the constructive discussion. We are glad that our responses clarified the issues raised. We would like to briefly report new results that provide additional evidence for the supervision-allocation hypothesis and strengthen the paper's contribution.

Supervision allocation within block diffusion. Following Reviewer aSz9's request, we continued pretraining LLaDA-8B under block diffusion with symmetric (α=0\alpha=0) and causal-bias (α=1\alpha=1) within-block corruption. Causal bias transfers to block diffusion, raising the retrieval margin under interference by +0.48 and +0.52 (K=8,16K=8, 16), with accuracy gains of 9.8 and 5.6 points. The result is consistent with the scope the revision gives Theorem 2.1: the identity motivates a supervision-allocation hypothesis, and at the block-final position, where the within-block information term is zero, causal bias still improves retrieval, so supervision allocation alone produces the gain.

A practical application. Block diffusion is more expensive to train, limits revision and parallel decoding, and at the same budget remains below the starting checkpoint on LAMBADA and five-shot MMLU. In contrast, from the same checkpoint and data, our full-sequence intervention improves key–value retrieval and both benchmarks, while preserving all properties of diffusion. Causal bias is therefore an efficient post-training recipe for improving ICL in existing diffusion language models.

For more details, please see our general comment and the discussion with Reviewer aSz9. Thank you again for a review that substantially improved the paper.

Reviewer_aSz9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies how masked prediction training objectives affect ICL. Using a Bayesian view of sequential prediction, the work argues that predicting later tokens from earlier context may help models learn to use examples. The authors test this idea by assigning higher masking rates to later positions during training. The experiments cover small models and continued pretraining of LLaDA-8B. The results show gains on several ICL tasks and suggest that the training objective can affect how models use context.

Critical strengths
  1. The research question is interesting. It may help us understand how different language modeling methods affect ICL. The findings may guide the design of better training methods and model architectures.

  2. The experiments include both small-scale models and continued pretraining of LLaDA-8B. This helps test whether the findings hold across different model scales.

Critical weaknesses
  1. The idea that masking later tokens in a sequence may improve performance on masked prediction objectives appears to have been introduced in prior work [1, 2, 3]. The authors should discuss the relationship between their work and these studies.

  2. Regarding the contribution, as the authors acknowledge, the Bayesian prediction result builds on prior work [4], and the main theorem follows from a standard entropy identity. The theory does not explain why finite-capacity masked prediction models should learn ICL less effectively than AR models when given the same context.

  3. The experiments do not fully isolate the effect of masking direction, as both the noise level and prediction difficulty change. Shuffling positions while keeping the noise budget fixed would strengthen the claim.

[1] AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation. Wu et al. arXiv: 2305.09515.

[2] Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed. Fu et al. arXiv: 2512.14067.

[3] On the Trainability of Masked Diffusion Language Models via Blockwise Locality. Wang et al. arXiv: 2604.24832.

[4] Meta-learning of Sequential Strategies. Ortega et al. arXiv: 1905.03030.

Questions for authors
  1. This work shows that masked prediction appears to be less effective than AR for ICL. The paper mentions the flexibility of masked prediction, but it is unclear what benefits this flexibility brings. Why should we retain masked prediction instead of using AR? Can the authors show where it offers a clear benefit, or would AR alone be sufficient for these tasks?

  2. Block diffusion uses earlier blocks as clean context when predicting the next block. Does the same ICL gap appear in block diffusion models? Would the proposed masking schedule help?

7 comments

We thank the reviewer for the constructive review, and especially for suggesting the shuffled control: together with the reverse control, it now completely isolates the direction of the corruption profile (Section 7).


W1: Relation to prior position-dependent masking [1–3]

We now discuss all three in Section 9. Each adds directional structure to diffusion, with a different motivation and by changing more than the corruption.

  • AR-Diffusion [1] ties each token's diffusion time to its position in continuous embedding diffusion, at training and inference, for generation quality and speed.
  • Efficient-DLM [2] converts AR models into block diffusion and masks later positions within each block more often, to match the left-to-right order of confidence-based decoding; the goal is throughput.
  • Blockwise locality [3] enforces autoregressive order within blocks, which repairs the trainability of random-masking models on in-context regression.

Each also changes the attention or sampler, and none asks why direction matters for ICL or separates it from budget and loss weighting. Our intervention changes only the corruption distribution of full-sequence masked diffusion, derives its direction from a theory of ICL, and isolates that direction with matched controls. A directional bias in diffusion is not new in itself; to our knowledge, ours is the first to introduce it purely through the corruption distribution and test it this way. [2] and [3] are independent evidence for our mechanism.


W2: Contribution of the theory

We agree that Proposition B.1 builds on Ortega et al. [4] and that Theorem 2.1 is an entropy identity; the revision says so explicitly (Section 2.2). The contribution is its use: Proposition B.1 makes episodic ICL risk a conditional subproblem of AR log loss; Propositions B.3–B.4 show that symmetric masking makes prefix-clean supervision combinatorially rare and that any monotone tilt shifts mass towards it; together they give a falsifiable prediction we test causally.

We also agree that the theory does not explain finite-capacity learning; Section 2.2 now states that the identity establishes no computational or statistical separation and only motivates the hypothesis we test. For finite models, the new mechanistic analysis gives an empirical account at 8B: the intervention amplifies the value path of an existing retrieval circuit (Section 7).


W3: Isolating the masking direction

Following the reviewer's suggestion, we trained two new LLaDA-8B controls. SHUFFLED applies the α=1\alpha=1 rates to a fresh random permutation of positions per sequence, so it has exactly the same mask-count distribution, realised budget and clipping as α=1\alpha=1, but no direction. REVERSE flips the profile.

On the same 500 examples per KK with paired bootstrap intervals, SHUFFLED is worse than α=0\alpha=0 from K=4K=4 on, REVERSE is worse under interference, and only α=1\alpha=1 improves as competing bindings increase (Figure 5, Appendix G.1). Neither the noise budget nor the heterogeneity of rates produces the gain; removing the direction removes it.


Q1: Why retain masked prediction?

Masked diffusion offers parallel decoding, infilling and revision of earlier predictions, with gains in throughput [2] and on planning tasks such as path-finding [3]. Its bidirectional factorisation is also better matched to modalities without a privileged order, such as protein and genomic sequences, where function depends on interactions in both directions. Language, by contrast, has a strong sequential structure, and our results suggest this is where the AR factorisation confers its advantage for ICL. Our aim is therefore not to argue that AR is sufficient, but to identify which ingredient of the AR objective drives ICL, so that it can be added to diffusion without giving up these properties. Our intervention keeps the architecture and sampler unchanged and does not reduce generation quality (Section 6.2) or benchmark accuracy (Section 7).


Q2: Block diffusion

Block diffusion conditions on clean earlier blocks, so the only future tokens that can be visible during training are those later in the active block. The corresponding information gap can be positive within a block and is zero at its final token. Under symmetric absorbing corruption, Appendix B.5 gives the exact loss mass on the within-block prefix pattern as 1/[B(B−1r−1)]1/[B\binom{B-1}{r-1}] at offset rr in a block of size BB. Proposition B.6 shows that an endpoint-complete monotone schedule with schedule-matched weights cannot decrease this coverage. For our clipped schedule, coverage can increase or decrease depending on block size and offset. The α=1\alpha=1 schedule increases exact-prefix supervision at interior block positions. Whether the schedule improves ICL in block diffusion under matched compute remains an empirical question beyond the scope of this paper.

I thank the authors for discussing the related work and adding the SHUFFLED and REVERSE controls to rule out noise-budget confounds. However, given the limited theoretical contribution and the lack of empirical validation on block diffusion, I will maintain my score.

We thank the reviewer for confirming that the related-work discussion and the SHUFFLED and REVERSE controls resolve W1 and W3. These controls rule out more than the noise budget: at every noise level, SHUFFLED matches α=1\alpha=1 in mask-count distribution, budget, clipping and the multiset of rates, and differs only in their placement. We address the two remaining concerns, the theoretical contribution and block diffusion, below.

Aim and theoretical contribution

The paper asks why ordinary language-model pretraining produces an in-context learner, and answers at the level of the training objective's factorisation. To our knowledge, it is the first work to study ICL through this lens, and the first to unify three views of ICL usually treated separately: few-shot task adaptation [1], the fall in predictive loss as context grows [2, 3], and Bayesian sequential prediction under a latent task distribution [4].

  • Formalisation: Bayes-optimal episodic ICL is a conditional subproblem of population AR log loss (Proposition B.1). AR pretraining therefore directly supervises the map ICL requires.
  • Consequence for diffusion: a non-causal objective must recover that conditional by marginalising unknown futures, and the cost is exactly a conditional mutual information (Theorem 2.1).
  • Synthetic instantiation: Appendix C makes the gap concrete and shows that iterative denoising does not remove it and can increase prediction loss.
  • Link to training: symmetric masking makes prefix-clean supervision combinatorially rare, and a monotone tilt favours it over its reversed counterpart (Propositions B.3–B.4).

Together, these turn an informal intuition into a testable theory: it identifies the supervision ICL requires, explains why non-causal objectives under-supply it, and specifies the minimal intervention that should restore it and the matched alternatives that should not. The reviewer's own SHUFFLED control could have refuted that prediction; it did not.

Empirical validation

The intervention changes only the corruption distribution, and it improves ICL across objectives, tasks, inference procedures and scale:

  • four controlled ICL tasks in two diffusion objectives (MDLM, GIDD), toy models and natural-text context-use benchmarks;
  • native iterative generation (Section 6.1) and natural-text continuation (Section 6.2);
  • at 8B, key–value retrieval under interference and both LAMBADA and five-shot MMLU (Section 7);
  • attention-routing and value-path analyses across model scales that locate the mechanism (Section 7, Appendix N);
  • ablations over iterative unmasking, noise level, appended masks and query position, and the direction controls above.

On block diffusion

Block diffusion is a different model family, not a variant of the model we study. It changes the factorisation and the attention mask, generates blocks serially, confines parallel denoising to the active block, cannot revise committed blocks, and adds block size as a hyperparameter.

Appendix B.5 already answers the central theoretical question. Block causality removes the part of the gap carried by later blocks, but the within-block gap persists at interior positions and vanishes only at the block-final token. Yet even there, symmetric masking gives exact-prefix supervision a weight of only 1/B1/B, and Proposition B.6 shows that an endpoint-complete monotone tilt cannot decrease it. Clean earlier blocks therefore address part of the information gap but leave the supervision-allocation problem inside each block, which is exactly the problem our intervention targets.

No claim in the paper depends on block diffusion, so no outcome of a block-diffusion study would change a claim the paper makes. At close to 50 pages, the manuscript already carries substantial theoretical and experimental appendices. A block-diffusion study would add a separate axis, varying the factorisation, attention and decoding that our experiments hold fixed. Including it would make the central result harder to interpret: changing only the placement of masking rates improves ICL in full-sequence models. We see two promising follow-ups: whether the tilt transfers to block diffusion, and how much of the gap inference-time procedures can close in a symmetrically trained model, which Appendix C.3 suggests is limited. Both warrant controlled studies of their own. If the reviewer has a specific block-diffusion prediction in mind, we would gladly discuss it here.

We therefore ask that block diffusion be weighed as a follow-up direction, not as a gap in the support for the paper's claims.

References

[1] Brown et al. Language Models are Few-Shot Learners. NeurIPS 2020.

[2] Kaplan et al. Scaling Laws for Neural Language Models. arXiv 2020.

[3] Olsson et al. In-context Learning and Induction Heads. 2022.

[4] Ortega et al. Meta-learning of Sequential Strategies. arXiv 2019.

Reviewer_Qd77OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper argues that the emergence of in-context learning (ICL) is driven by how the training objective factorises the sequence distribution, not by which joint distribution a model can represent. The authors show that Bayes-optimal episodic ICL is a conditional subproblem of the population autoregressive (AR) objective. Non-causal objectives such as masked language modelling and masked diffusion, by contrast, must recover the prefix-conditioned predictor by marginalising over the unobserved future, which incurs an information-theoretic gap. Motivated by this analysis, the authors propose a position-dependent masking schedule, controlled by a single hyperparameter α, that corrupts later positions more frequently while leaving the architecture, loss weighting and decoding unchanged.

Critical strengths
  1. The theoretical analysis is mathematically sound.
Critical weaknesses
  1. The practical significance of this work is unclear. The paper is largely conceptual: it reaches a conclusion the authors find convincing but does not turn it into a practical improvement. A position-dependent masking schedule is proposed, but the authors explicitly frame it as a diagnostic probe rather than a method.
  2. Controlling the masking distribution through a power-law hyperparameter α\alpha appears ad hoc. Block diffusion (Arriola et al., 2025) is a well-known alternative that enforces autoregressive structure across blocks while retaining diffusion within each block, and thus provides clean-prefix conditioning by construction.
  3. The central result is the continued pretraining of LLaDA-8B, yet critical details are missing: (i) The 500M-token training corpus is not specified; (ii) Adam is named as the optimizer, but β1\beta_1, β2\beta_2, the learning rate, batch size and warmup schedule are not reported; (iii) LLaDA was pretrained on 2.3T tokens, so it is surprising that only 500M tokens of continued pretraining substantially change its key-value retrieval behavior under distractors.
Questions for authors
  1. Beyond serving as a diagnostic probe, can the proposed schedule improve LLaDA-8B on standard downstream benchmarks, compared with existing diffusion LM training methods?
  2. How does the α-based schedule compare with block diffusion under the same compute budget, and why is a power-law profile preferable?
  3. What corpus and hyperparameters (β1\beta_1, β2\beta_2, learning rate, batch size) were used for the 500M-token continued pretraining? Given that LLaDA was pretrained on 2.3T tokens, how do the authors explain the substantial change in key-value retrieval?
8 comments

We thank the reviewer for the careful assessment and for questions that led directly to three additions to the revision: downstream benchmarks, full continued-pretraining details, and a mechanistic account of how 500M tokens change an 8B model. Section 7 has been rewritten around these results.


W1 / Q1: Practical significance and standard benchmarks

The paper's primary aim is a theory of ICL: which property of the training objective drives it. We designed the intervention as a probe of that account, not as an optimised method. The empirical results nonetheless show a clear practical gain: the intervention improves ICL across models, scales and tasks, as well as on standard benchmarks and native generation, while preserving all properties of full-sequence diffusion.

Following the reviewer's question, Section 7 and Figure 6 now report LAMBADA and five-shot MMLU. At 500M tokens, α=1\alpha=1 reaches 67.0% on LAMBADA (α=0\alpha=0: 66.3%; base: 64.9%) and 66.6% on MMLU (α=0\alpha=0: 66.1%; base: 65.9%); paired gains over α=0\alpha=0 are 0.8 and 0.45 points. The α=0\alpha=0 model is LLaDA's own training objective continued under identical data, schedule and random draws, so this is a like-for-like comparison against the standard diffusion training method.


W2 / Q2: Is the power law ad hoc? Comparison with block diffusion

The theory does not depend on the power law. Proposition B.4 holds for any monotone non-decreasing profile; the power law is simply the smallest one-parameter family that contains the symmetric baseline (α=0\alpha=0). The new controls in Section 7 also show that what matters is the direction of the profile, not its exact form or budget.

Block diffusion is a different way of introducing directional structure, with its own design choices. It changes the factorisation: a block-causal attention mask and matching sampler, parallel decoding and revision confined to the active block, a training objective that is a sum of per-block bounds, and a block size that must be tuned. Our intervention keeps all of these fixed. All experiments are therefore deliberately restricted to full-sequence masked diffusion.

The new Appendix B.5 analyses block diffusion theoretically: block structure removes the part of the future-marginalisation gap carried by later blocks, but not the within-block part; supervision on the exact within-block prefix decays combinatorially with block size and a monotone tilt can only increase it (Proposition B.6). The two are therefore complementary rather than competing.


W3 / Q3: Continued-pretraining details

We thank the reviewer for pointing this out. Appendix F now gives the full setup: the data mixture and held-out split, the optimiser and all its hyperparameters (β\beta, ϵ\epsilon, weight decay, learning rate, warmup and schedule), the batch size, and the shared data order and random draws across models.


Q3: How can 500M tokens change an 8B model?

Part of the change is adaptation to new data. LLaDA's pretraining corpus is undisclosed, so both models are continued on a different mixture, which moves them away from the base checkpoint, and our key–value probe seems to be particularly sensitive to that adaptation. We also report gains on two other benchmarks. Prior work, including post-training on LLaDA itself, shows that a similar amount of supervision suffices to alter a model's behaviour [1, 2]. Continued pretraining with LLaDA's own objective (α=0\alpha=0) already raises K=16K=16 accuracy from 35.4% to 42.2%. The comparison of interest is therefore between α=0\alpha=0 and α=1\alpha=1, which share this adaptation and differ only in the corruption profile.

That difference is modest and targeted. Held-out loss under a common symmetric corruption is nearly identical (2.726 for α=0\alpha=0, 2.733 for α=1\alpha=1). The new mechanistic analysis (Section 7, Figure 7) locates the effect: α=1\alpha=1 moves attention at the answer position away from the masked suffix towards the clean prefix (7% less suffix attention at K=16K=16), and strengthens the value-path contribution of the queried binding by about 5%. Attention to that binding itself changes by less than 3%. Continued pretraining therefore amplifies an existing retrieval circuit rather than building a new one, consistent with mechanistic studies of fine-tuning [3]. The shuffled and reverse controls weaken these value contributions, and the same value-path signature appears in the small models (Appendix N).


We hope these additions address the reviewer's concerns, and we would be glad to clarify any remaining point.

References

[1] Fu et al. Data Engineering for Scaling Language Models to 128K Context. ICML 2024.

[2] He et al. UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models. ICLR 2026.

[3] Prakash et al. Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking. ICLR 2024.

We would like to point the reviewer to new results that additionally address two of their main concerns, beyond our earlier response.

Practical significance (W1/Q1). The new experiment identifies a clear practical application: causal bias is an efficient post-training recipe for improving ICL in existing diffusion language models. From the same LLaDA-8B checkpoint and data, it improves key–value retrieval and both LAMBADA and five-shot MMLU, while preserving the architecture, attention, sampler, parallel decoding and revision.

Comparison with block diffusion at matched budget (W2/Q2). We now ran this comparison. Causal bias also transfers within block diffusion, raising the retrieval margin by +0.48 and +0.52 (K=8,16K=8, 16) with the offset structure predicted in Appendix B.5. As a post-training method, however, block diffusion at the same training data budget remains below the starting checkpoint on both benchmarks, while our full-sequence intervention improves on it.

For more details, please see our general comment and the discussion with Reviewer aSz9. We hope these results resolve the reviewer's remaining concerns.

I appreciate the authors' effort during the rebuttal, which resolved some of my questions and concerns. I will maintain my score for now. My remaining questions are:

  1. I feel the results on LAMBADA and MMLU are insufficient to demonstrate the impact of CT w/ causal bias on LLaDA performance, more comprehensive downstream evaluation is needed.
  2. The authors point out in the rebuttal that CT w/ block diffusion causes LLaDA to drop points, which is completely incomparable to causal bias. This confuses me, because LLaDA uses block diffusion sampling by default. Why would a more consistent training and inference method lead to a drop in points? Or did the authors use other sampling methods? I think a more comprehensive experimental setting is needed.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: Summary of new results addressing raised concerns and complete revisions made in the manuscript

We thank all reviewers for their constructive feedback. We recognise that the manuscript is dense: it combines theory, a controlled intervention, and evaluations across objectives, inference procedures and scales. Several clarifications needed to read the main results were placed in the appendix, where they required close familiarity with both the ICL and the diffusion literature. To enhance clarity, our revision moves these points forward and substantially rewrites Sections 6 and 7.

We are especially grateful to aaip for a thorough reading including the appendices, and to aSz9 and aaip for proposing targeted controls. Following their suggestions and Qd77's questions on scale and benchmarks, we made the following changes.

Direction controls at 8B (aSz9, aaip). To test whether the gain comes from the direction of the profile rather than its budget or weighting, we trained two new LLaDA-8B controls for 500M tokens under the identical protocol. SHUFFLED exactly matches the α=1\alpha=1 mask-count distribution at every noise level, including clipping, but randomly permutes the masking rates for each sequence. REVERSE flips the positional profile. All models, including the loss-weighted control, are evaluated on the same 500 examples per KK with paired bootstrap intervals. None of the three controls reproduces the α=1\alpha=1 improvement under interference, isolating forward direction as the mechanism behind gain. (Section 7, Figure 5, Appendix G.1).

Downstream benchmarks (Qd77). Following the reviewer's suggestion, we further evaluated the 8B checkpoints on LAMBADA and five-shot MMLU (Figure 6). The models show the same signature as on retrieval: α=1\alpha=1 again outperforms the symmetric α=0\alpha=0 variant, reaching 67.0% on LAMBADA (α=0\alpha=0: 66.3%; base: 64.9%) and 66.6% on MMLU (α=0\alpha=0: 66.1%; base: 65.9%).

Mechanistic analysis (Qd77). To address how 500M tokens can change an 8B model's behaviour, we decompose attention at the answer position into routing and value paths (Figure 7). α=1\alpha=1 shifts attention away from the masked suffix towards the clean prefix and strengthens the value-path contribution of the queried binding by about 5%, while attention to that binding changes by less than 3%. Continued pretraining therefore amplifies an existing retrieval circuit rather than building a new one. Neither SHUFFLED nor REVERSE shows this signature, and the small models show the same value-path signature (Appendix N).

Continued-pretraining details (Qd77). Appendix F now gives the full setup: data mixture, optimiser and all hyperparameters, batch size, and the shared data order and random draws, so the 8B experiments are reproducible.

OverrideKV under native generation (aaip). The concern was well founded: a change in lift could come from either the real or the control condition, so we tested it directly. Section 6.1 and Appendix L.1 now report acc(real) and acc(control) separately, and Figure 4 plots accuracy. Control accuracy never exceeds 0.010 in any configuration we evaluate, so the gains come from the real prompts. No candidate restriction is applied at any step; the protocol is stated explicitly, with full tables in the appendix.

Theory (aaip, aSz9). Theorem 2.1 is now stated explicitly as an information identity that motivates the supervision-allocation hypothesis; it does not establish a computational or statistical separation between AR and diffusion learning. The assumptions of Proposition C.2 now require the posterior to place mass on two hypotheses with distinct predictive distributions, which excludes the reviewer's counterexample; the proof is updated.

Related work and block diffusion (aSz9, Qd77). We now discuss AR-Diffusion, Efficient-DLM and blockwise locality. Each changes the attention or sampler along with the corruption. Appendix B.5 now analyses block diffusion formally. Block structure removes the part of the gap carried by later blocks but not the part within a block, where prefix supervision decays combinatorially with block size and a monotone directional tilt can only increase it (Proposition B.6). The two approaches are therefore complementary rather than competing.

Aim of the work. The paper makes three connected contributions.

  • (i) A theory: the population AR objective contains Bayes-optimal episodic ICL as a conditional subproblem, while non-causal objectives must recover the prefix predictor by marginalising unknown futures.
  • (ii) A minimal intervention testing this account: it changes only the corruption distribution of masked diffusion, leaving the architecture, loss weight and sampler unchanged.
  • (iii) Controlled evidence across tasks, objectives, inference procedures and scale.
AuthorsOpenReview ↗

Empirical test on block diffusion and the practical application it revealed

We would like to sincerely thank Reviewers aSz9 and Qd77 for insisting on an empirical test of block diffusion, which led to an unexpected finding that significantly strengthens the paper's contribution.

We stand by our original scope. The paper studies how ICL depends on the factorisation of the training objective, comparing AR, MLM and full-sequence masked diffusion under matched conditions, with a minimal intervention that changes only the corruption distribution. Block diffusion is a different factorisation: it imposes block-causal attention, generates blocks sequentially, confines parallel decoding and revision to the active block, never revises committed blocks, adds block size as a hyperparameter, and is more expensive to train. Including it would have confounded the paper's central comparison.

What the experiment shows. We continued pretraining LLaDA-8B under block diffusion with the same 500M-token protocol as Section 7:

  • Causal bias transfers to block diffusion. Within blocks, it raises the retrieval margin under interference by +0.48 and +0.52 (K=8,16K=8, 16), with accuracy gains of 9.8 and 5.6 points, following the offset structure predicted in Appendix B.5.
  • Block diffusion is not an efficient post-training method for ICL adaptation. Besides being more expensive to train and limiting revision and parallel decoding, it remains below the starting checkpoint on all six downstream settings at the same budget: its block-causal mask restricts the attention of a model pretrained bidirectionally, removing context the model was trained to use, in line with the theoretical prediction (see our response to Reviewer Qd77).
  • In contrast, our intervention leaves attention untouched. From the same checkpoint and data, causal bias in full-sequence diffusion improves key–value retrieval, LAMBADA, five-shot MMLU and one-shot AG News over both the starting checkpoint and the symmetric continuation, and 16-shot SST-2 over the symmetric continuation, while preserving the architecture, sampler, bidirectional context, parallel decoding and revision.
  • The gain is largest where the task can only be solved in context. Following [1], we replace the AG News label names with semantically unrelated ones, so a model can only beat chance (25%) by learning the input–label mapping from the demonstrations. The causal-bias continuation reaches 41.0% at k=8k=8 and 58.4% at k=16k=16 demonstrations, above both the starting checkpoint (38.8%, 49.8%) and the symmetric continuation (37.4%, 53.8%).

Our causal bias intervention is therefore a clear-cut post-training recipe for improving ICL capability in existing diffusion language models. The full protocol, offset-resolved results and benchmark comparison are given in our response to Reviewer aSz9 and in our final response to Reviewer Qd77. We will include these results, together with the revised contributions, in the camera-ready version.

References

[1] Wei et al. Larger Language Models Do In-Context Learning Differently. arXiv:2303.03846, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.