acceptodds
Under review as a conference paper at ICLR 2027

Latent-Plan Block Diffusion: Sampling a Block-Level Intent for Few-Step Parallel Decoding

12 views

Abstract

Block diffusion language models such as BD3-LM use a masked-diffusion denoiser to decode a block of tokens in parallel while keeping the key–value (KV) cache of autoregressive models, but every denoising step samples the block from a product of per-token marginals. We show that the least error of drawing a block in one such step is its conditional total correlation given the context, a cost that unmasking schedules can only trade for more steps. *Latent-Plan Block Diffusion* (LP-BD) supplies the missing shared variable: before a block is decoded, a small conditional flow samples a low-dimensional *plan* given the generated prefix, and the denoiser decodes the block conditioned on it. The plan is learned from the denoiser's own loss, without supervision, and LP-BD equals BD3-LM at initialization, so it can be added to a pretrained model. Trained from scratch on TinyStories with 16-token blocks, LP-BD lowers the negative log-likelihood of one-step samples under an autoregressive judge by 31% and reaches 16-step BD3-LM quality with 8 steps. The gains are similar at 28M and 85M parameters. Added to the released 170M BD3-LM checkpoints on OpenWebText, LP-BD lowers generative perplexity at matched entropy by 14–55% at every step budget of a fixed-step sampler, and by 38–66% with guidance. The plan acts as a sampled *intent* shared by the block rather than a prediction from the context, and its gains require steps that decode several tokens jointly.

Primary area:
generative models
Keywords:
Block Diffusion, Diffusion Language Models, Parallel Decoding, Latent Variable Models, Flow Matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uADeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Latent-Plan Block Diffusion (LP-BD), a framework designed to improve few-step parallel decoding in block diffusion language models by capturing dependencies among simultaneously generated tokens. The authors show that the minimum KL divergence of one-step factorized block sampling equals the conditional total correlation among block tokens, motivating the introduction of a shared block-level latent plan. LP-BD learns this plan without explicit supervision by extracting representations from clean blocks and training a conditional rectified flow to match an EMA target encoder. During inference, the flow samples a plan conditioned on the generated prefix, which remains fixed throughout block decoding. The method can be integrated into pretrained BD3-LM models without changing the underlying architecture. Experiments on TinyStories, text8, and OpenWebText demonstrate improvements in generation quality across fixed-step decoding budgets, often achieving comparable or better quality with fewer denoising steps. Ablation studies further highlight the importance of stochastic plan sampling, position-aware conditioning, and jointly decoding multiple tokens.

Critical strengths
  1. Clear motivation and well-organized presentation. The paper is clearly written and well structured, with a logical progression from identifying the limitations of independent token sampling to proposing a latent-variable-based solution. The overall methodology is intuitive and easy to follow.

  2. Strong theoretical motivation. The paper provides a principled theoretical analysis by characterizing the minimum single-step decoding error in terms of conditional total correlation. The subsequent propositions offer useful insights into how latent variables can address dependencies among simultaneously generated tokens and under what conditions they may fail to help.

  3. Simple and practical methodology. LP-BD introduces a relatively lightweight latent planning mechanism that can be integrated into pretrained BD3-LM models. The use of zero-initialized conditioning layers and a conditional flow predictor makes the approach conceptually simple and practically appealing.

  4. Systematic experiments and convincing empirical results. The paper evaluates the proposed method across multiple datasets, model sizes, and denoising step budgets, demonstrating consistent improvements in few-step generation quality. The extensive ablation studies, null-vector controls, and matched-entropy evaluation provide additional evidence supporting the effectiveness of the proposed approach.

Critical weaknesses

Major Weaknesses

  1. Lack of direct empirical validation of conditional total correlation reduction.

    The theoretical motivation centers on reducing conditional total correlation (TC), but the experiments do not directly measure this quantity. Although Figure 4 provides indirect evidence through plan utilization and residual entropy, these results do not necessarily demonstrate that the learned latent plans reduce TC. An empirical analysis, potentially using controlled synthetic settings or approximate TC measurements, would help strengthen the connection between the theoretical motivation and the proposed method.

  2. Unclear necessity of conditional flow modeling.

    The experiments in Section 5.4 suggest that the sampled plans depend only weakly on the prefix. Shuffling the prefix increases the flow-matching loss by only 4–7%, and using plans generated from different prefixes preserves most of the performance gains. These findings raise the question of whether conditioning the flow on the prefix is necessary. Although the paper compares against uniform sampling on the sphere, a comparison with a learned unconditional prior would help clarify the contribution of conditional flow modeling.

Minor Weakness

  1. Dependence on the decoding schedule.

    The effectiveness of LP-BD depends on decoding multiple tokens jointly. As discussed in Section 5.4, the method performs worse than BD3-LM without guidance when using the original one-token-per-step sampler. Although this observation is consistent with the theoretical motivation, it suggests that the benefits of LP-BD depend on the choice of decoding schedule. Further experiments with alternative unmasking strategies would help clarify the conditions under which the proposed method is most effective.

  2. Limited evaluation of scalability and generalization.

    The experiments are conducted on relatively small models, with the largest backbone containing 170M parameters. Furthermore, the evaluation focuses exclusively on unconditional text generation. It remains unclear whether the proposed approach would provide similar benefits for larger language models and conditional generation tasks, such as instruction following, reasoning, or code generation. Additional experiments in these settings would strengthen the practical relevance and generalizability of the proposed method.

Questions for authors
  1. Interpretability of latent plans. What kinds of information do the learned latent plans actually encode? For example, do they capture semantic content, syntactic structure, or specific lexical choices? It would be interesting to see whether different sampled plans correspond to interpretable differences in generated blocks.

  2. Planning beyond individual blocks. LP-BD samples a separate latent plan for each block. Could the authors discuss how this design affects coherence across multiple blocks? Would sharing information between consecutive plans potentially improve longer-range consistency?

2 comments

We thank the reviewer for the positive assessment and constructive suggestions. The revision adds the requested analyses (in blue).

W1. Direct validation of TC reduction. Exact, synthetic (Appendix E): where TC is computed by enumeration, the one-step error of the product sampler equals TC (1.8954 vs. 1.895 nats), and sampling with the plan reduces it to 0.12 nats (−94%). Real data (new Table R1): by the identity of Appendix A, TC−EzTC(x∣z,c)=∑iI(xi;z∣c)−I(x;z∣c)\mathrm{TC}-\mathbb E_z\mathrm{TC}(x\mid z,c)=\sum_i I(x_i;z\mid c)-I(x;z\mid c). The first term is the plan utilization of Figure 4b, the second the drop in NELBO when the denoiser receives the plan, so trained models estimate TC with and without the plan (nats per token):

settingTC: BD3-LMnull vectorplanone-step gain (w=1w{=}1)
TinyStories B=16B{=}16 (3 seeds)2.782.620.40 (−85%)−31%
TinyStories B=16B{=}16, 85M2.792.640.33 (−87%)−29%
TinyStories B=64B{=}64, sub-block3.713.481.33 (−62%)−11%
text8 B=16B{=}161.241.180.09 (−92%)−14%
text8 B=16B{=}16, adaLN only–1.120.37 (−67%)−7%
text8 B=16B{=}16, d=8d{=}8–1.180.56 (−53%)−5%
text8 B=64B{=}64, sub-block1.451.471.00 (−32%)−1%
text8 B=64B{=}64, broadcast1.451.461.56 (+7%)−1%
OpenWebText B=8B{=}82.312.300.69 (−70%)−44%
OpenWebText B=16B{=}162.852.831.89 (−33%)−14%

The null vector stays close to BD3-LM's own TC, and within each dataset the one-step gain tracks the reduction. At B=64B{=}64, only plans broadcast to all 64 positions fail to lower TC (Remark 1) and bring no gain. Sub-block plans, our default there, lower it.

W2. Necessity of the conditional flow. We trained a learned unconditional prior, the same flow without the prefix input, next to a conditional flow with the same recipe (Table R2). On TinyStories it keeps 84% of the one-step gain but only 51% at 16 steps (text8: 84% and 62%), matching the conditional flow sampled with another sequence's prefix (Table 9). The plan thus acts mainly as a sampled intent at one step, and conditioning on the prefix contributes half of the gain at 16 steps, at no extra cost, as hbh_b comes from the KV-cache pass.

Decoding schedule. A step pays the TC of the tokens it decodes jointly (Proposition 1), so the plan helps whenever a step decodes several tokens. At one token per step only the plan model's own error remains, and better flows shrink this gap: from +51% to +27% at B=8B{=}8 and to +1% at B=16B{=}16 (Table 12). New (Table R5): with confidence-based unmasking, LP-BD has higher MAUVE than BD3-LM at every step budget (0.82 vs. 0.16 at 4 steps) and lowers the judge NLL by 39% and 23% at 2 and 4 steps. At 16 steps this schedule makes BD3-LM's text stereotyped (judge NLL 0.86, below held-out text's 1.39; distinct-3 0.38 vs. 0.65), while LP-BD keeps the diversity of held-out text (0.58).

Scale and conditional generation. The relative gains hold from 28M to 85M (Table 14) and grow at matched entropy (16 steps: −1% at 28M, −6% at 85M, Appendix D.11), and the plan helps when added to the largest released BD3-LM (170M). LP-BD is conditional by construction, as plans are sampled from the prefix state. In prompted continuation, the gain matches that of unconditional generation, for held-out stories (judge NLL −33% at one step and −15% at sixteen, MAUVE 0.94 vs. 0.69 at sixteen, Table R4) and for 256-token OpenWebText prompts (Gen PPL at matched entropy −4% to −34%, Table R9). Since LP-BD equals the base model at initialization, larger models only need fine-tuning, which we leave to future work.

Q1. What do plans encode? At B=16B{=}16 the plan fixes most of the block: given the plan, the NELBO drops from 1.43 to 0.31 nats/char (text8) and from 1.80 to 0.49 nats/token (TinyStories). Decoding a TinyStories prefix twice with the same plan and different sampling noise gives 3× the token agreement of two different plans (33% vs. 11%, null vector 9%), so different plans give different continuations (Table R7). Linear probes on 22.5k blocks show that the plan carries block-level choices the prefix leaves open: adding it to the prefix state raises the AUC for every probed property (Table R6), e.g., whether a sentence ends in the block (0.86 → 0.98), contains dialogue (0.92 → 0.98) or mentions a frequent object noun (0.70 → 0.80).

Q2. Planning beyond blocks. Each plan is sampled given hbh_b, which summarizes all previous blocks, so plans interact through the generated text, and the autoregressive factorization models cross-block structure exactly. New (Table R2): a flow that also receives the previous block's plan has the same validation loss as the flow given hbh_b alone (1.414 vs. 1.414, three seeds) and the same judge NLL (within 0.5%), as that plan is encoded from a block hbh_b already summarizes. A sequence-level plan conditioning the block plans is a natural extension (Section 7).

Reviewer_uADeOpenReview ↗

Response to Author Rebuttal

Thank you for the detailed rebuttal and extensive additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the direct analysis of total correlation reduction and the comparison with a learned unconditional prior.

The additional experiments substantially strengthen the connection between the theoretical motivation and the proposed method. In particular, the synthetic validation and estimated total correlation reductions on real datasets provide convincing evidence that latent plans help mitigate dependencies among jointly decoded tokens. The unconditional prior experiments also clarify the contribution of prefix conditioning, especially at larger decoding step budgets.

I also appreciate the additional evaluations with confidence-based unmasking, prompted generation, and the analysis of what latent plans encode. These results provide useful insights into the method's behavior and address several of my remaining questions.

Some limitations remain, particularly regarding generalization to substantially larger language models and more complex downstream tasks such as reasoning and instruction following. However, I consider these reasonable directions for future work rather than fundamental weaknesses of the current contribution.

Overall, I believe the rebuttal has substantially addressed my major concerns and further strengthened the theoretical and empirical contributions of the paper. I maintain my positive assessment and will keep my current rating.

Thank you again for the thoughtful and comprehensive response.

Reviewer_Lc9jOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper identifies a precise information-theoretic limit of block diffusion LMs (BD3-LM) and proposes a targeted fix. The key observation (Proposition 1): because a masked-diffusion step samples the masked positions of a block independently from a product of marginals, the least possible single-step KL error of decoding a block is exactly its conditional total correlation TC(x_b | c) — the information the block's tokens share given the context. Unmasking schedules can only trade this cost for more steps; it cannot be removed by a better schedule. Specifically, the authors propose Latent-Plan Block Diffusion (LP-BD): before decoding a block, a small conditional flow samples a low-dimensional plan z_b from the prefix hidden state. The denoiser then decodes the block conditioned on z_b, so the tokens become near-conditionally-independent given the plan and can be drawn well in few steps. The method is supported by the proposed theoretical analysis. LP-BD is validated by training from scratch on TinyStories/text8 (28M/85M/12M backbones) and by adding it—via zero-initialized injection—to the public 170M BD3-LM OpenWebText checkpoint and fine-tuning, demonstrating that sampling a single latent plan per block lets the model match or exceed BD3-LM's generation quality with fewer denoising steps under a fixed-step sampler. It also has an extensive set of mechanism ablations (null-vector control, deterministic/per-step-redrawn plans, cross-prefix plans, sub-block injection) establishing that the gains come from the plan rather than from added capacity.

Critical strengths

1.Theoretically motivated method design. Proving that single-step block-parallel error equals the conditional total correlation turns a folk intuition into an actionable principle, and the paper then derives the method (a shared latent reducing within-block dependence) directly from it, including the precise conditions under which it helps (large within-block TC, steps that decode several tokens jointly) and when it does not (one token per step, large residual entropy).

2.Method is evaluated with various metrics. An extensive set of mechanism ablations are conducted to analyze the gains of the method.

Critical weaknesses

1.Judge NLL, Gen-PPL, and MAUVE are all proxies, and the judge shares the same architecture and training data as the evaluated model, raising a self-preference risk.
2.No downstream or conditional-generation metrics. All results are on unconditional generation, with nothing beyond perplexity-style measures (no QA, continuation accuracy, or instruction following), so the claim of "better quality" holds only in the sense of the generative distribution.
3.Throughput (tokens/s) is reported, but the quality advantage at "fewer steps" partly relies on guidance (which doubles the denoiser forward passes per step), and the wall-clock benefit is further eroded by the flow's sequential steps; the metrics are reported honestly, but the "half the steps at equal quality" framing should be discounted once the guidance overhead is taken into account.

Questions for authors

See weakness.

1 comment

We thank the reviewer for a summary that captures our contributions precisely, and for recognizing the theory and the mechanism ablations. We address the three concerns below and believe they resolve the soundness question.

W1. Proxy metrics and self-preference.

  • The judge is a separately trained AR model. It shares no weights with any evaluated model, and it shares its architecture and training data with both BD3-LM and LP-BD, which use the same backbone, so it has no reason to prefer either, and a 31% gap between them cannot come from self-preference. We also track repetition, which can fool likelihood judges, for every cell (Appendix C).
  • On OpenWebText the judge is GPT-2 Large, an external model with a different architecture and training data, under the BD3-LM protocol (our pipeline reproduces the published BD3-LM numbers within 2%). We compare at matched entropy, so that lower diversity cannot pass for quality.
  • MAUVE against held-out text, a distributional metric that also penalizes lost diversity, ranks the systems the same way, also at matched entropy: LP-BD scores higher at every step budget (0.41 vs. 0.10 at 4 steps, new Table R8), and 8 guided steps above 16 BD3-LM steps at equal passes (0.80 vs. 0.66).
  • New, external judges (Table R3). GPT-2 Large and Llama-3.1-8B share neither architecture nor training data with our models. Under both, LP-BD has lower Gen PPL than BD3-LM at every step budget without guidance (GPT-2 Large: 345 vs. 2180 at one step, 31.7 vs. 47.6 at sixteen, real text 9.2; Llama-3.1-8B: 273 vs. 1423 and 29.4 vs. 43.9, real text 5.9), and the 28M LP-BD beats the 85M BD3-LM, also at matched entropy. LP-BD's token entropy is also closer to that of real text than BD3-LM's.
  • New, blind pairwise preference. Shown each pair in both orders, Qwen2.5-14B-Instruct prefers LP-BD over BD3-LM in 64% of judgments at 4 and at 16 steps, and Mistral-Small-24B-Instruct, a judge from another model family, in 79% and 63% (77% at one step). Both prefer 8-step LP-BD over 16-step BD3-LM (56% and 58%), and all 95% bootstrap intervals exclude 50%.
  • The results are statistically reliable: three training seeds on TinyStories, with standard deviations of at most 0.18 nats against gaps of up to 3 nats (Table 1), and on OpenWebText 95% bootstrap intervals that exclude zero for every cell (Table 10) and a second training seed that reproduces the gains (new Table R10).
  • The gain is not added capacity: the same checkpoints sampled with the null vector match BD3-LM within 2% (Figure 1), and an 85M BD3-LM with 3× the parameters is worse than the 28M LP-BD at every step budget (Table 14), in Gen PPL and MAUVE also at matched entropy.

W2. Downstream and conditional generation. Unconditional generation is the standard protocol for diffusion LMs at this scale (MDLM, SEDD, BD3-LM), and our question, the error of a parallel decoding step, is a property of the generative distribution that these metrics measure. LP-BD is also conditional by construction: every block is generated given its prefix, and the plan is sampled from the prefix state, so a prompt is simply a prefix. New (Table R4): given the first 64 tokens of 256 held-out TinyStories sequences as prompts, LP-BD's continuations have 33% lower judge NLL than BD3-LM's at one step and 15% at sixteen (three seeds, w=1w{=}1), and LP-BD also has lower GPT-2 Large Gen PPL and higher MAUVE against the true continuations at every step budget. With 256-token OpenWebText prompts and the 170M model, the continuation's Gen PPL at matched entropy is 4–34% lower without guidance and 21–52% lower with it, and its MAUVE is higher at every step budget (Table R9). Instruction following and reasoning require billion-parameter instruction-tuned models. Since LP-BD equals the base model at initialization, it can be added to such models by fine-tuning, which we leave to future work.

W3. Guidance and "half the steps". We agree that compute should be counted in denoiser passes. At equal passes LP-BD is better, also at matched entropy:

  • Without guidance, at equal steps: on TinyStories (B=16B{=}16), against BD3-LM's nucleus frontier at the same entropy, the judge NLL is 27% lower at one step and the gain persists at every budget (1% at 16 steps, new Table R8). On OpenWebText, Gen PPL is 14–55% lower at every budget (Table 2).
  • With guidance, half the steps costs the same denoiser passes. At matched entropy, 8 guided steps beat 16 BD3-LM steps on TinyStories (judge NLL −4%, GPT-2 Large Gen PPL −12%, Appendix D.11), and on OpenWebText half the steps beat BD3-LM at every budget and both block sizes (by 8–43%, Table 2).
  • Wall-clock times are measured end to end, flow included: on OpenWebText, guided LP-BD with half the steps is also faster than BD3-LM in 6 of 7 settings (Table 11).

We hope these results resolve the three concerns, and we are glad to discuss any remaining questions.

Reviewer_v753OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a limitation of block diffusion: tokens decoded in parallel are sampled independently, making it difficult to preserve dependencies within a block. The authors theoretically characterize the minimum one-step approximation error and propose LP-BD, which introduces a shared stochastic latent plan to coordinate token generation. The method can be added to pretrained block diffusion models without changing their initial behavior. Experiments on several corpus show improved generation quality with fewer denoising steps. Further analysis suggests that the plan primarily serves as a shared stochastic intent rather than a prediction of the next block, and that its benefits depend on decoding tokens jointly.

Critical strengths

1. Theoretical characterization and a concrete solution of parallel decoding

The paper identifies independent token sampling as a key limitation of parallel generation and characterizes the minimum KL div. of one-step generation in terms of conditional total correlation. While the formulation of shared latents to shape token dependencies has been explored in prior work, yet this paper adapts the idea to block diffusion and presents a concrete solution. It also discusses cases where introducing a latent variable does not necessarily reduce token dependencies.

2. Ease of integration and careful experimental evaluation

Proposed method, LP-BD, can be integrated into existing BD3-LM models with relatively minor modifications, and its zero-initialized connections preserve the pretrained model's behavior at initialization. Notably, LP-BD outperforms the baseline with half the denoising steps on tinygsm, and the improvements was also shown when added to a pretrained 170M model. These results provide concrete evidence of the mehthods' effectiveness.

Critical weaknesses

1. technical scope and novelty related to existing work

The difficulty of capturing token dependencies is not specific to block diffusion, but is a general challenge in parallel generation. Peng et al. [1], for example, have already explored one-step generation with a shared latent variable and few-step refinement with a fixed latent, which overlaps with the central idea of this work. Guo et al. [2] also explore latent-based text generation, using VAE and a block-causal diffusion prior to model continuous representations before decoding text. Although these differ in their implementations, It remains unclear what is gained by restricting generation to the block level, or how the proposed method differs technically from prior approaches. A more careful comparison with related methods would help clarify the motivation, novelty, and contributions for the block-level design.

  • [1] "Coupling Models for One-Step Discrete Generation", Peng et al., 2026.
  • [2] "Continuous Latent Diffusion Language Model" Guo et al., 2026.

2. Necessity of the planning and justification

Sec.5.4 shows that shuffling the context causes only a small increase in flow-matching loss, and that plans sampled using a different context retain most of the improvement in generation quality. These results raise questions about how necessary it is to learn a context-conditioned flow, given the additional complexity of formulation and training costs it introduces. The comparison with uniform sampling on the sphere does not rule out the possibility that a learned unconditional prior could achieve similar results. Moreover, when only one token is unmasked per step, adding the plan makes generation quality worse rather than leaving it unchanged, and the reason for this degradation remains unclear.

Questions for authors

Parameter and compute efficiency.

LP-BD introduces additional trainable parameters and training costs. It is releted to Weaknesses, could you provide a parameter- or compute-matched comparison with BD3-LM to clarify whether the improvements justify these additional resources?

3 comments

We thank the reviewer for recognizing the theoretical characterization, the concrete solution, and the careful evaluation. The revision adds the comparisons below (in blue).

W1. Relation to Coupling Models (Peng et al.) and Cola DLM (Guo et al.). We agree that dependence among parallel tokens is general: Propositions 1–3 hold for any parallel decoding step. Coordinating parallel predictions through a latent goes back to latent-variable non-autoregressive models (Kaiser et al., 2018; Ma et al., 2019), and we do not claim that idea. We contribute (i) the exact cost of a parallel step and when a latent reduces it, (ii) a design that makes a latent work inside block diffusion LMs, and (iii) evidence for when and why it helps. Coupling Models samples a sequence-level latent and decodes the whole sequence in one step (or a few masked steps with the latent fixed), a pattern LP-BD shares at the block level. Cola DLM generates per-token VAE latents with a block-causal DiT and decodes them into text, so its latent replaces the tokens. LP-BD keeps the discrete denoiser and adds a small plan per block:

Coupling ModelsCola DLMLP-BD
Latentsequence of 256-d codes from a frozen Qwen2.5-0.5Bthe text itself (128-d per token)32-d plan per 16-token block
Generatorone-step decoder, distilled from a 0.5B LM~2B latent DiT + decoderthe masked-diffusion denoiser, any steps
Trainingtwo stagesVAE, then DiTend to end, denoiser's own loss
Prefix / KV cachenoyesyes, plan from the cached prefix state
Added to a pretrained diffusion LMnonoyes, identical at initialization
Analysisfactorized family not universalMarkov-path viewmin one-step KL = TC, and when a latent raises it

What is gained at the block level. Across blocks the autoregressive factorization is exact, so the plan only carries the dependence within a block, which is what one parallel step cannot represent (Proposition 1). This keeps the latent tiny (32 numbers per 16 tokens, a 0.7M flow), lets it be sampled from the prefix state the KV cache already computes, and keeps variable-length generation. A sequence-level latent must carry all dependence and a per-token latent all content, hence the large codes and priors of both. Section 6 now includes this discussion.

W2. Is the context-conditioned flow necessary? We trained the suggested learned unconditional prior: the same flow with its prefix input removed, fit to the frozen model next to a conditional flow with the same recipe (TinyStories, three seeds, w=1w{=}1, judge NLL, Table R2):

plan priorT=1T{=}124816
none (BD3-LM)9.687.345.474.243.56
unconditional flow7.065.244.153.593.29
conditional flow6.564.803.793.293.04

The unconditional prior keeps 84% of the one-step gain but only 51% at 16 steps (text8: 84% and 62%), as does the conditional flow sampled with another sequence's prefix (Table 9). A sampled intent thus carries most of the one-step gain, and conditioning on the prefix half of the 16-step gain. Conditioning adds no cost: hbh_b comes from the KV-cache pass, and the flow has the same size. The 1.4–1.7× training cost comes from the clean encoder pass, which any learned plan prior needs.

W2. Why one token per step is worse. With one token per step there is no TC to remove (Proposition 1), so the plan can only add the error of the plan model. Proposition 3 holds only approximately, as the flow fits EMA-target plans while the denoiser sees noised online plans. The data follow this: improving only the flow (refit, FM loss 1.485 → 1.434) shrinks the gap at B=8B{=}8 from +51% to +30% and turns the guided B=16B{=}16 result from −6% into −15%. A refit trained 10× longer (FM loss 1.410) brings the unguided gap to +27% at B=8B{=}8 and +1% at B=16B{=}16, while the null vector stays within 11% of BD3-LM (Table 12). This regime costs BB passes per block, whereas LP-BD targets the parallel one.

Q. Parameter- and compute-matched comparison. LP-BD adds 1.0M parameters (+3.6%, Table 4). The 85M BD3-LM, with 3× the parameters and 1.3× the training GPU time (12.3 vs. 9.6 H100 hours), is worse than the 28M LP-BD without guidance at every step budget (Table 14, three seeds: 9.67 vs. 6.71 at one step, 3.46 vs. 3.08 at sixteen). At matched entropy it is still worse at every budget in GPT-2 Large Gen PPL and MAUVE, and up to 8 steps in judge NLL (equal at 16, Appendix D.11). Scaling BD3-LM 3× improves its one-step judge NLL by only 0.1%, as Proposition 1 predicts for a sampling error. At inference, w=1w{=}1 uses the same denoiser passes (the flow adds <1% FLOPs), and at equal passes and matched entropy guided LP-BD is still 3–14% better on TinyStories (Appendix D.11) and 8–43% on OpenWebText.

We hope this addresses the concerns, and we are glad to discuss any remaining questions.

Reviewer_v753OpenReview ↗

Response to Rebuttal

Thank you for the detailed response.

The added comparisons with Coupling Models and COLA make the technical differences clearer, and the new unconditional-prior and compute-matched experiments address my experimental questions. I also find the explanation for the one-token-per-step degradation reasonable.

My remaining concern is still novelty. It offers a practical way to introduce shared latents into block diffusion, but the underlying idea of coordinating parallel predictions through a shared latent is not new in my understanding. In addition, unconditional prior also retains most of the gain at low step counts, so I am not fully convinced that context conditioning is essential in the few-step regime. The evaluation remains limited to relatively small sized models, without downstream results.

I appreciate the provided additional evidence, yet these concerns still affect my assessment.

Thank you for the follow-up, and for confirming that the comparisons, the unconditional prior, the compute-matched comparison and the one-token-per-step analysis address your questions. On the remaining points:

Novelty. We agree that coordinating parallel predictions through a shared latent is an established idea, and we cite it as such. What the paper contributes is which latent removes the cost of a parallel step, and the answer is not "any shared latent". Proposition 1 gives that cost exactly, the conditional total correlation, and Proposition 2 shows that a latent lowers it by ∑iI(xi;z)−I(x;z)\sum_i I(x_i;z)-I(x;z), which can be negative (Remark 1). The experiments follow this: a plan broadcast to all 64 positions of a text8 block raises TC and brings no gain (Table R1), a plan redrawn at every denoising step, as the latent of VADD is, falls behind BD3-LM from four steps on (Table 9), a deterministic plan collapses into repetition (Figure 5), and without noise on the training plan quality stops improving with more steps (Table 3). The design that avoids these failures is what makes the latent a component that can be added to a pretrained block diffusion LM, which it equals at initialization, and that beats BD3-LM at matched entropy with equal or fewer denoiser passes (Section 5.2, Appendix D.11).

Context conditioning. We agree, and the paper reaches the same conclusion: the plan acts as a sampled intent rather than a prediction from the context (abstract and Section 5.4). None of our claims rests on conditioning. With the unconditional prior, LP-BD still lowers the one-step judge NLL from 9.68 to 7.06 (Table R2). We keep the conditional flow as the default because it costs nothing and contributes up to half of the gain as the number of steps grows (51% of the gain without it at 16 steps).

Scale and downstream tasks. We agree that this is the main limitation of the study (Section 7). Within our budget, the relative gains hold from 28M to 85M (three seeds each) and grow at matched entropy, and the plan helps when added to the largest released BD3-LM (170M), including for prompted continuation.

We appreciate the careful reading and the constructive discussion.

Comments on the paper

2 comments

We thank all reviewers for their thoughtful feedback. We are encouraged that they found the theory principled and actionable ("turns a folk intuition into an actionable principle", Lc9j; "a principled theoretical analysis", uADe; "a concrete solution", v753), the method simple to add to pretrained BD3-LMs (v753, uADe), and the evaluation careful and systematic, with "an extensive set of mechanism ablations establishing that the gains come from the plan rather than from added capacity" (Lc9j; also v753, uADe).

We uploaded a revision (changes in blue). The main additions, each detailed in the individual replies:

1. The plan directly reduces total correlation (uADe). By the identity TC−EzTC=∑iI(xi;z)−I(x;z)\mathrm{TC}-\mathbb E_z\mathrm{TC}=\sum_i I(x_i;z)-I(x;z) (Appendix A), plan utilization (Figure 4b) and the NELBO with and without the plan give a direct estimate of TC. At B=16B{=}16 the plan removes 85% (TinyStories) and 92% (text8) of it, and within each dataset the one-step gain follows the reduction. The only plans that do not lower TC, one plan broadcast to all 64 positions of a text8 block (Remark 1), bring no gain, while sub-block plans, our default at B=64B{=}64, lower it (Table R1).

2. A learned unconditional prior (v753, uADe). A separately trained unconditional flow keeps 84% of the one-step gain but only 51% at 16 steps (TinyStories; text8: 84% and 62%, Table R2), as does the conditional flow sampled with another sequence's prefix (Table 9). The plan acts as a sampled intent, and conditioning on the prefix contributes half of the gain at 16 steps, at no extra cost.

3. Parameter- and compute-matched comparison (v753, Lc9j). The 28M LP-BD (+1.0M parameters) outperforms an 85M BD3-LM, which has 3× the parameters and 1.3× the training GPU time, at every step budget and without guidance (Table 14), also at matched entropy except for an equal judge NLL at 16 steps (Appendix D.11). Scaling BD3-LM 3× improves its one-step quality by only 0.1% (three seeds each), as Proposition 1 predicts for a sampling error that capacity cannot remove.

4. Guidance, compute and matched entropy (Lc9j). At equal denoiser passes, LP-BD is better at matched entropy: without guidance at every step budget (TinyStories at B=16B{=}16, new Table R8; OpenWebText, Table 2), and with guidance at half the steps, which costs the same passes, on both datasets. On OpenWebText, guided half-step sampling is also faster in measured wall-clock time in 6 of 7 settings (Table 11).

5. One token per step (v753, uADe). This regime has no TC to remove (Proposition 1), so only the plan model's own error remains (Proposition 3). Improving the flow alone shrinks the gap at B=8B{=}8 from +51% to +30%, and a flow trained 10× longer to +27% at B=8B{=}8 and +1% at B=16B{=}16 (Table 12).

6. External judges and blind preference (Lc9j). Scored by GPT-2 Large and Llama-3.1-8B, which share neither architecture nor training data with our models, unguided LP-BD lowers the Gen PPL of TinyStories samples by 81–84% at one step and 33% at sixteen, and the 28M LP-BD beats the 85M BD3-LM (Table R3). In blind pairwise comparisons, Qwen2.5-14B-Instruct and Mistral-Small-24B-Instruct prefer LP-BD in 63–79% of judgments at 4 and 16 steps (Mistral: 77% even at one step).

7. Prompted generation, other schedules, and what plans encode (Lc9j, uADe). Given held-out story openings, LP-BD's continuations gain as much as unconditional samples (MAUVE against the true continuations 0.94 vs. 0.69 at 16 steps, Table R4), and with 256-token OpenWebText prompts its Gen PPL at matched entropy is 4–34% lower (21–52% with guidance, Table R9). With confidence-based unmasking, LP-BD has higher MAUVE at every step budget and keeps the diversity of held-out text where BD3-LM's samples become stereotyped (Table R5). Different plans give different continuations of a prefix, and linear probes show that plans carry block-level choices, such as whether the block ends a sentence, that the prefix leaves open.

8. Related work (v753). We added a comparison with Coupling Models (Peng et al.) and Cola DLM (Guo et al.). LP-BD learns a 32-d block-level plan end to end from the denoiser's own loss, samples it from the KV-cache prefix state at no extra pass, and can be added to a pretrained block diffusion LM, which it equals at initialization.

9. Presentation (all). A notation table (Appendix B), a description of both samplers and their costs in Section 2, and a pipeline overview in Section 3.

We are happy to answer further questions during the discussion.

Dear reviewers,

The author-reviewer discussion period is ending soon. If you have not done so yet, please read the authors’ rebuttal and reply to it. Tell the authors whether it addresses your concerns, and update your score if your assessment has changed.

The ICLR 2027 Reviewer Guidelines list « discussing with the authors to clarify and improve the paper » as one of the core duties you accepted as a reviewer. With the high number of submissions this year, your input matters more than ever. Recall that reviewers that do not fulfill their duties may also face penalties on their own submissions.

Thank you for your time and help.

Best regards,


Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.