- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Latent-Plan Block Diffusion (LP-BD), a framework designed to improve few-step parallel decoding in block diffusion language models by capturing dependencies among simultaneously generated tokens. The authors show that the minimum KL divergence of one-step factorized block sampling equals the conditional total correlation among block tokens, motivating the introduction of a shared block-level latent plan. LP-BD learns this plan without explicit supervision by extracting representations from clean blocks and training a conditional rectified flow to match an EMA target encoder. During inference, the flow samples a plan conditioned on the generated prefix, which remains fixed throughout block decoding. The method can be integrated into pretrained BD3-LM models without changing the underlying architecture. Experiments on TinyStories, text8, and OpenWebText demonstrate improvements in generation quality across fixed-step decoding budgets, often achieving comparable or better quality with fewer denoising steps. Ablation studies further highlight the importance of stochastic plan sampling, position-aware conditioning, and jointly decoding multiple tokens.
Critical strengths
-
Clear motivation and well-organized presentation. The paper is clearly written and well structured, with a logical progression from identifying the limitations of independent token sampling to proposing a latent-variable-based solution. The overall methodology is intuitive and easy to follow.
-
Strong theoretical motivation. The paper provides a principled theoretical analysis by characterizing the minimum single-step decoding error in terms of conditional total correlation. The subsequent propositions offer useful insights into how latent variables can address dependencies among simultaneously generated tokens and under what conditions they may fail to help.
-
Simple and practical methodology. LP-BD introduces a relatively lightweight latent planning mechanism that can be integrated into pretrained BD3-LM models. The use of zero-initialized conditioning layers and a conditional flow predictor makes the approach conceptually simple and practically appealing.
-
Systematic experiments and convincing empirical results. The paper evaluates the proposed method across multiple datasets, model sizes, and denoising step budgets, demonstrating consistent improvements in few-step generation quality. The extensive ablation studies, null-vector controls, and matched-entropy evaluation provide additional evidence supporting the effectiveness of the proposed approach.
Critical weaknesses
Major Weaknesses
-
Lack of direct empirical validation of conditional total correlation reduction.
The theoretical motivation centers on reducing conditional total correlation (TC), but the experiments do not directly measure this quantity. Although Figure 4 provides indirect evidence through plan utilization and residual entropy, these results do not necessarily demonstrate that the learned latent plans reduce TC. An empirical analysis, potentially using controlled synthetic settings or approximate TC measurements, would help strengthen the connection between the theoretical motivation and the proposed method.
-
Unclear necessity of conditional flow modeling.
The experiments in Section 5.4 suggest that the sampled plans depend only weakly on the prefix. Shuffling the prefix increases the flow-matching loss by only 4–7%, and using plans generated from different prefixes preserves most of the performance gains. These findings raise the question of whether conditioning the flow on the prefix is necessary. Although the paper compares against uniform sampling on the sphere, a comparison with a learned unconditional prior would help clarify the contribution of conditional flow modeling.
Minor Weakness
-
Dependence on the decoding schedule.
The effectiveness of LP-BD depends on decoding multiple tokens jointly. As discussed in Section 5.4, the method performs worse than BD3-LM without guidance when using the original one-token-per-step sampler. Although this observation is consistent with the theoretical motivation, it suggests that the benefits of LP-BD depend on the choice of decoding schedule. Further experiments with alternative unmasking strategies would help clarify the conditions under which the proposed method is most effective.
-
Limited evaluation of scalability and generalization.
The experiments are conducted on relatively small models, with the largest backbone containing 170M parameters. Furthermore, the evaluation focuses exclusively on unconditional text generation. It remains unclear whether the proposed approach would provide similar benefits for larger language models and conditional generation tasks, such as instruction following, reasoning, or code generation. Additional experiments in these settings would strengthen the practical relevance and generalizability of the proposed method.
Questions for authors
-
Interpretability of latent plans. What kinds of information do the learned latent plans actually encode? For example, do they capture semantic content, syntactic structure, or specific lexical choices? It would be interesting to see whether different sampled plans correspond to interpretable differences in generated blocks.
-
Planning beyond individual blocks. LP-BD samples a separate latent plan for each block. Could the authors discuss how this design affects coherence across multiple blocks? Would sharing information between consecutive plans potentially improve longer-range consistency?
2 comments
Response to Author Rebuttal
Thank you for the detailed rebuttal and extensive additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the direct analysis of total correlation reduction and the comparison with a learned unconditional prior.
The additional experiments substantially strengthen the connection between the theoretical motivation and the proposed method. In particular, the synthetic validation and estimated total correlation reductions on real datasets provide convincing evidence that latent plans help mitigate dependencies among jointly decoded tokens. The unconditional prior experiments also clarify the contribution of prefix conditioning, especially at larger decoding step budgets.
I also appreciate the additional evaluations with confidence-based unmasking, prompted generation, and the analysis of what latent plans encode. These results provide useful insights into the method's behavior and address several of my remaining questions.
Some limitations remain, particularly regarding generalization to substantially larger language models and more complex downstream tasks such as reasoning and instruction following. However, I consider these reasonable directions for future work rather than fundamental weaknesses of the current contribution.
Overall, I believe the rebuttal has substantially addressed my major concerns and further strengthened the theoretical and empirical contributions of the paper. I maintain my positive assessment and will keep my current rating.
Thank you again for the thoughtful and comprehensive response.
We thank the reviewer for the positive assessment and constructive suggestions. The revision adds the requested analyses (in blue).
W1. Direct validation of TC reduction. Exact, synthetic (Appendix E): where TC is computed by enumeration, the one-step error of the product sampler equals TC (1.8954 vs. 1.895 nats), and sampling with the plan reduces it to 0.12 nats (−94%). Real data (new Table R1): by the identity of Appendix A, TC−EzTC(x∣z,c)=∑iI(xi;z∣c)−I(x;z∣c). The first term is the plan utilization of Figure 4b, the second the drop in NELBO when the denoiser receives the plan, so trained models estimate TC with and without the plan (nats per token):
The null vector stays close to BD3-LM's own TC, and within each dataset the one-step gain tracks the reduction. At B=64, only plans broadcast to all 64 positions fail to lower TC (Remark 1) and bring no gain. Sub-block plans, our default there, lower it.
W2. Necessity of the conditional flow. We trained a learned unconditional prior, the same flow without the prefix input, next to a conditional flow with the same recipe (Table R2). On TinyStories it keeps 84% of the one-step gain but only 51% at 16 steps (text8: 84% and 62%), matching the conditional flow sampled with another sequence's prefix (Table 9). The plan thus acts mainly as a sampled intent at one step, and conditioning on the prefix contributes half of the gain at 16 steps, at no extra cost, as hb comes from the KV-cache pass.
Decoding schedule. A step pays the TC of the tokens it decodes jointly (Proposition 1), so the plan helps whenever a step decodes several tokens. At one token per step only the plan model's own error remains, and better flows shrink this gap: from +51% to +27% at B=8 and to +1% at B=16 (Table 12). New (Table R5): with confidence-based unmasking, LP-BD has higher MAUVE than BD3-LM at every step budget (0.82 vs. 0.16 at 4 steps) and lowers the judge NLL by 39% and 23% at 2 and 4 steps. At 16 steps this schedule makes BD3-LM's text stereotyped (judge NLL 0.86, below held-out text's 1.39; distinct-3 0.38 vs. 0.65), while LP-BD keeps the diversity of held-out text (0.58).
Scale and conditional generation. The relative gains hold from 28M to 85M (Table 14) and grow at matched entropy (16 steps: −1% at 28M, −6% at 85M, Appendix D.11), and the plan helps when added to the largest released BD3-LM (170M). LP-BD is conditional by construction, as plans are sampled from the prefix state. In prompted continuation, the gain matches that of unconditional generation, for held-out stories (judge NLL −33% at one step and −15% at sixteen, MAUVE 0.94 vs. 0.69 at sixteen, Table R4) and for 256-token OpenWebText prompts (Gen PPL at matched entropy −4% to −34%, Table R9). Since LP-BD equals the base model at initialization, larger models only need fine-tuning, which we leave to future work.
Q1. What do plans encode? At B=16 the plan fixes most of the block: given the plan, the NELBO drops from 1.43 to 0.31 nats/char (text8) and from 1.80 to 0.49 nats/token (TinyStories). Decoding a TinyStories prefix twice with the same plan and different sampling noise gives 3× the token agreement of two different plans (33% vs. 11%, null vector 9%), so different plans give different continuations (Table R7). Linear probes on 22.5k blocks show that the plan carries block-level choices the prefix leaves open: adding it to the prefix state raises the AUC for every probed property (Table R6), e.g., whether a sentence ends in the block (0.86 → 0.98), contains dialogue (0.92 → 0.98) or mentions a frequent object noun (0.70 → 0.80).
Q2. Planning beyond blocks. Each plan is sampled given hb, which summarizes all previous blocks, so plans interact through the generated text, and the autoregressive factorization models cross-block structure exactly. New (Table R2): a flow that also receives the previous block's plan has the same validation loss as the flow given hb alone (1.414 vs. 1.414, three seeds) and the same judge NLL (within 0.5%), as that plan is encoded from a block hb already summarizes. A sequence-level plan conditioning the block plans is a natural extension (Section 7).