- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether the distribution of conditional prediction problems presented during pretraining influences in-context learning. It connects population autoregressive log loss to Bayes-optimal episodic prediction and characterizes the entropy difference between prefix-only and future-conditioned prediction.
Motivated by this perspective, it introduces position-dependent corruption that makes earlier tokens relatively cleaner and later tokens more frequently masked. Experiments compare training objectives in matched small models, evaluate synthetic contextual tasks and constrained iterative decoding, and apply the intervention through 500M-token continued pretraining of LLaDA-8B.
The intervention improves several contextual evaluations. At 8B scale, its relative effect changes with query position in a manner consistent with the imposed corruption profile. A positional loss-weighting control does not reproduce this pattern. These results support a useful supervision-allocation hypothesis, although the theoretical interpretation and the isolation of conditioning structure from other training changes require qualification.
Critical strengths
-
The training intervention provides a useful controlled test of supervision allocation. The positional corruption profile changes the training observation patterns without introducing a new architecture or changing the decoder within the principal comparisons. In LLaDA-8B, FORWARD and SYM receive 500M tokens of continued pretraining under matched data and schedules. At K=16, retrieval accuracy is 0.458 versus 0.422, while the paired margin effect changes from negative to positive as the query moves through the imposed positional profile. This signed positional pattern is stronger evidence of an intervention-specific effect than aggregate improvement alone.
-
The study evaluates several distinct manifestations of contextual behavior and reports important failures. The synthetic battery includes retrieval, permutation inference, conflicting bindings, and finite-state prediction. At the largest reported context sizes, the alpha=1 model improves Key–Value accuracy from 0.234 to 0.354 and Cipher accuracy from 0.140 to 0.222. Under the specified 64-step contextual decoding protocol, MDLM OverrideKV lift improves from 0.332 to 0.510 at K=32. The appendices also report near-chance tasks, failures of the stronger tilt, disagreements between scoring and decoding, and the restricted GIDD replication. This makes the empirical contribution more credible than a presentation based only on favorable evaluations.
Critical weaknesses
-
The theoretical interpretation is stronger than the formal result, and one strictness claim needs correction. Theorem 2.1 correctly identifies a conditional mutual information as an entropy gap. However, higher uncertainty when the future is unobserved is not itself evidence that a diffusion model cannot directly learn the correct prefix conditional, nor a bound on sample complexity or inference cost. Appendix O acknowledges this limitation, but the "barrier" terminology and parts of the main discussion still suggest a stronger explanation of the empirical AR–diffusion difference.
In addition, Proposition C.2 does not follow from the assumptions in C.1 as written. A nondegenerate posterior over latent hypotheses and globally non-identical component distributions are insufficient. For example, let p_1=p_2 differ from p_3=p_4, and let the prefix reveal which pair contains the latent hypothesis. The posterior over hypotheses remains nondegenerate, but all hypotheses remaining possible after a given prefix induce the same token distribution. The target and future are then conditionally independent, so the claimed strict entropy gap is zero. Strictness requires residual uncertainty over distinct predictive distributions.
-
The key control does not yet fully isolate conditioning structure as the explanation. Appendix G reports that mean-normalized MDLM corruption produces a realized masking rate of approximately 0.452 rather than 0.5, while GIDD uses a separately reduced corruption budget. Less corruption changes task difficulty as well as context availability; its direction is not automatically conservative for every learning comparison. The 8B loss-weighted control is well motivated, but matches expected target mass only away from clipping and is reported as unpaired condition means without uncertainty estimates. Its failure to reproduce FORWARD supports the proposed explanation, but does not justify the categorical statement that target allocation is not a cause and context allocation is the cause.
Questions for authors
-
Theoretical scope and strictness: Would you clarify that Theorem 2.1 is an information identity motivating the supervision-allocation hypothesis, rather than a computational or statistical barrier for diffusion learning? Please also strengthen the assumptions of Proposition C.2 to require residual posterior variation over distinct predictive distributions; the assumptions in C.1 permit the zero-gap counterexample described above.
-
Mechanism-discriminating control: Can the loss-weighted arm be evaluated on the same episodes as SYM and FORWARD, with paired uncertainty estimates? Please distinguish what is established by the current interior-matched control from what remains unresolved because of clipping and changed realized corruption. If an exactly clipping-aware target-matched or realized-corruption-matched control is unavailable, please narrow the causal attribution accordingly.
-
Interpretation of iterative contextual performance: Please report acc(real) and acc(control) separately for the principal OverrideKV results, rather than lift alone. A larger difference can arise through changes in either component. Please also specify whether candidate restriction is applied during denoising or only at final selection, and label the principal result explicitly as performance under the stated constrained decoding protocol.
3 comments
Response to Authors
Dear authors,
Thank you for your reply and effort. After reading your responses and clarifications, I think the scope of THM 2.1 and the concern about OverrideKV under native generation are clear to me. After considering the responses carefully, I decide to maintain my score 3.
Thank you, and summary of new results strengthening the contribution
Dear Reviewer aaip,
Thank you for your valuable feedback and the constructive discussion. We are glad that our responses clarified the issues raised. We would like to briefly report new results that provide additional evidence for the supervision-allocation hypothesis and strengthen the paper's contribution.
Supervision allocation within block diffusion. Following Reviewer aSz9's request, we continued pretraining LLaDA-8B under block diffusion with symmetric () and causal-bias () within-block corruption. Causal bias transfers to block diffusion, raising the retrieval margin under interference by +0.48 and +0.52 (), with accuracy gains of 9.8 and 5.6 points. The result is consistent with the scope the revision gives Theorem 2.1: the identity motivates a supervision-allocation hypothesis, and at the block-final position, where the within-block information term is zero, causal bias still improves retrieval, so supervision allocation alone produces the gain.
A practical application. Block diffusion is more expensive to train, limits revision and parallel decoding, and at the same budget remains below the starting checkpoint on LAMBADA and five-shot MMLU. In contrast, from the same checkpoint and data, our full-sequence intervention improves key–value retrieval and both benchmarks, while preserving all properties of diffusion. Causal bias is therefore an efficient post-training recipe for improving ICL in existing diffusion language models.
For more details, please see our general comment and the discussion with Reviewer aSz9. Thank you again for a review that substantially improved the paper.
We thank the reviewer for an exceptionally careful reading of the paper, including the appendices. The counterexample to Proposition C.2 made the result tighter, and the two requests on OverrideKV reporting and on isolating conditioning structure led to the most significant changes in the revision: a rewritten Section 6.1 with a new figure reporting accuracy, and the new direction controls in Section 7.
W1 / Q1a: Scope of Theorem 2.1
We agree. Section 2.2 now states directly after the theorem that "barrier" denotes only the additional uncertainty when the future is unobserved, that the identity establishes no computational or statistical separation between AR and diffusion learning, and that it motivates the supervision-allocation hypothesis; the AR–diffusion difference is attributed only through the causal experiments. The Discussion is revised accordingly.
W1 / Q1b: Proposition C.2
The counterexample is correct. The construction in Appendix C.1 now assumes that, with positive probability over prefixes, the posterior places mass on two hypotheses with distinct predictive distributions ph=ph′. The proof now uses this condition directly: given h and the prefix, the next and first future tokens are independent draws from ph, so their covariance at any token a equals the posterior variance of ph(a), which is positive under the condition; hence I(Y;F1∣C)>0. The condition excludes the reviewer's construction, where all hypotheses compatible with a prefix share one predictive distribution.
W2 / Q2: Isolating conditioning structure
We agree that the earlier controls did not fully isolate conditioning structure, and we have added two new LLaDA-8B controls. SHUFFLED applies the α=1 rates to a fresh random permutation of positions for every sequence; conditioned on the noise level it has exactly the same multiset of rates, mask-count distribution, realised budget and clipping as α=1, but no direction. REVERSE keeps the rates, budget and clipping and flips the orientation. The loss-weighted model is now evaluated on the same 500 examples per K as all other models, with paired 5000-resample bootstrap intervals (Appendix G.1).
What the controls establish (Figure 5, Appendix G.1): at q=2048, only α=1 gains as competing bindings increase (ΔR=+0.334 and +0.289 at K=8,16). SHUFFLED is worse than α=0 from K=4 on, so neither the reduced realised budget nor the heterogeneity of rates produces the gain. REVERSE is worse under interference. Loss weighting on symmetric inputs is slightly negative throughout. We have narrowed the attribution accordingly: the forward direction of the corruption profile is necessary for the interference gain; matched budget, heterogeneity or target weighting is not sufficient.
What remains open: the loss-weighted match is approximate where corruption probabilities are clipped, although SHUFFLED controls clipping exactly; intervals are over examples, not training runs; and the realised-budget difference the reviewer notes for the 170M MDLM and GIDD models is controlled at 8B but not at small scale, where we describe the models as nominally rather than exactly budget-matched (Appendix G).
Q3: OverrideKV under native generation
The concern was well founded, since a difference in lift can move through either component, so we tested it directly. Section 6.1 and Appendix L.1 have been rewritten. Figure 4(a, b) now plots exact-match accuracy on the real prompts, and Table 8 reports acc(real), acc(control) and lift separately for the principal variants. Control accuracy never exceeds 0.010 for any model or setting, so lift is within a percentage point of acc(real) and the gains come from the real prompts.
No candidate restriction is applied at any point in generation: intermediate proposals are drawn over the full vocabulary, and the final step is an unrestricted greedy argmax. Candidate labels are used only in the one-step scoring of Section 5. The principal result is labelled throughout as accuracy under clamped-prefix, remasked-query, 64-step greedy decoding over the unrestricted vocabulary; Appendix L.1 reports the full decoding grid.
We are grateful for a review that substantially improved the paper, and we would be glad to clarify anything further.