acceptodds
Under review as a conference paper at ICLR 2027

Learning to Couple: Context-Adaptive Joint Denoising for Diffusion Language Models

Abstract

Diffusion language models accelerate generation by predicting multiple tokens in parallel, but factorized denoisers treat simultaneous predictions as conditionally independent. Decoding heuristics reduce the resulting errors by controlling which tokens are committed, but leave the underlying distribution factorized. Joint modeling addresses the source of the problem, yet existing methods attach separately learned layers to frozen denoisers, leaving the backbone unadapted to the joint objective and the coupling parameters fixed across contexts. We propose Codpula, a context-parameterized denoiser combining a Transformer with a tractable probabilistic inference layer. Given a partially masked input, the Transformer produces token potentials that shape per-position predictions and compact gates that modulate the layer's coupling parameters, yielding a context-dependent joint distribution while retaining exact conditional inference. We jointly optimize both components under a conditional denoising objective using a dual-flow multiplicative update that preserves valid probabilistic parameters. At inference, Codpula supports standard heuristics and uses conditional queries to adapt commitments to dependencies among proposed tokens without extra Transformer evaluations within a step. Codpula sets new state-of-the-art results across a broad suite of reasoning benchmarks while advancing the accuracy–latency Pareto frontier in mathematical reasoning and code generation.

Primary area:
generative models
Keywords:
Diffusion Language Models, Copula, Probabilistic Circuits

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DSPnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Codpula, which combines a probabilistic inference layer with diffusion large language models. Codpula includes context-dependent token potentials and coupling gates, and uses a dual-flow update for training. Token dependencies are also used for guiding the decoding. Experiments show that Codpula achieves better efficiency-quality trade-offs.

Critical strengths
  1. The proposed method targets a critical problem for diffusion large language models. The authors also provide a theoretical analysis to show that the proposed update preserves valid probability parameters. In some settings, the proposed method achieves good performance and better accuracy-latency trade-offs.
  2. The proposed method addresses two limitations of CoDD. CoDD uses a frozen backbone and fixed coupling parameters. Instead, Codpula introduces backbone adaptation and context-dependent coupling gates.
Critical weaknesses
  1. The experiments are not comprehensive. The benefits of context-dependent coupling are unclear. The experiments on LLaDA only use Input Modulation without Sum Modulation. In Table 2, adding Sum Modulation sometimes makes performance worse. The experiments are only on math and coding tasks. There are no evaluations on general tasks like QA. And CoDD is absent in Figure 3 and Figure 4.
  2. The proposed method is limited. The authors contend that they jointly optimize both components, but for LLaDA 1.0, they train the PC and adapter in two stages. And Codpula uses separate models for math and code tasks. So the generalizability of Codpula may be limited.
Questions for authors
  1. Please see the weaknesses.
  2. Is the proposed Codpula compatible with KV cache?
  3. How are the checkpoints in Table 5 selected? Are they selected based on the test sets?
5 comments

We thank the reviewer for recognizing the significance of the problem, our theoretical analysis of the proposed learning algorithm, the improved performance and accuracy-latency trade-offs, and Codpula's contributions in addressing limitations other work did not. We next address the reviewer's concerns through additional training ablations, evaluations on general QA tasks, and direct comparisons of inference overhead.

(W1, Q1). Broader evaluation and clearer evidence for context-dependent coupling.

We provide additional LLaDA-1.0 results comparing Input-only and Input + Sum modulation under both staged and joint training, along with evaluations on the general QA benchmarks MMLU and GPQA. We add CoDD to Figures 3-4 and show that Codpula's implementation incurs roughly half CoDD's per-step overhead (43-61% lower at batch size 64) across all four backbones.

Sum modulation improves LLaDA-1.0 in 11 of 12 settings under joint training. We now evaluate both modulation variants under both staged and joint training. Compared with jointly trained Input-only Codpula, Input + Sum improves all four benchmarks at 4 and 8 tokens per step, including 13.4% →\to 22.4% on MATH500 at 8 tokens/step and 32.3% →\to 41.6% on MBPP at 4 tokens/step. The jointly trained Input + Sum model also matches or exceeds the paper's staged Input-only model in 9 of 12 settings. These results demonstrate the benefit of context-dependent coupling, particularly at higher parallelism, while the mixed staged-training results show that the training schedule matters.

Table 1. LLaDA-1.0 accuracy (%). Column numbers indicate tokens per step. CE-SFT results are from Table 3 of the paper. Bold indicates the best result in each column.

MethodGSM8K 248MATH500 248MBPP 248HumanEval 248
Base69.165.437.841.030.210.041.325.710.133.525.011.6
CE-SFT66.256.025.637.827.610.241.628.012.834.222.011.6
Codpula (Input), staged71.769.052.441.635.220.848.331.915.637.229.316.5
Codpula (Input + Sum), staged71.668.051.341.034.219.246.735.017.536.626.217.7
Codpula (Input), joint71.365.545.437.432.813.447.932.317.138.430.513.4
Codpula (Input + Sum), joint72.567.651.338.635.622.449.841.622.637.231.116.5

The paper's checkpoints improve QA accuracy without additional training. We provide additional QA results on MMLU and GPQA by reusing the checkpoints from the paper, without additional training. On GPQA, Codpula improves LLaDA-2.0 from 18.3% to 31.5% at 4 tokens/step and SDAR from 29.7% to 34.2% at 2 tokens/step over their respective base models.

Table 2. GPQA accuracy (%). Column numbers indicate tokens per step. Bold indicates the best result in each column.

MethodLLaDA-2.0 24816SDAR 1248
Base22.818.39.44.733.029.720.14.0
CE-SFT30.818.86.22.533.531.517.94.9
CoDD25.421.211.65.8--------
Codpula (Input)32.831.518.114.5--------
Codpula (Input + Sum)--------36.234.221.25.8

On MMLU, we reuse the Dream and LLaDA-2.0 checkpoints from the paper without additional training. Codpula achieves the highest accuracy in all seven settings. With entropy-based decoding at 8 tokens/step, Input + Sum modulation improves Dream's accuracy from 18.2% to 45.9% over the base model and exceeds CE-SFT by 5.9 percentage points.

Table 3. MMLU accuracy (%). Column numbers indicate tokens per step. Dream uses entropy-based decoding. Bold indicates the best result in each column.

MethodDream 248LLaDA-2.0 24816
Base52.437.618.272.360.631.39.2
CE-SFT58.948.140.072.559.321.07.9
CoDD63.249.928.072.666.036.715.5
Codpula (Input)60.957.045.173.071.054.733.9
Codpula (Input + Sum)64.359.345.9--------

(Continued W1, Q1) Codpula's has lower per-step overhead than the CoDD across all four backbones. We compare both methods at batch sizes 1 and 64 and include CoDD in revised Figures 3-4. At batch size 1, Codpula (Input) adds 3.4-15.4% overhead, compared with 62.8-287.8% for CoDD. On SDAR, Codpula (Input + Sum) adds 14.8% versus CoDD's 187.9% at batch size 1 and 22.1% versus 53.3% at batch size 64. Adding Sum modulation increases overhead by at most 1.9 percentage points across the measured configurations.

Table 4. Per-step overhead over the base model (%); B denotes batch size.

BackboneTaskCoDD, B=1Codpula Input, B=1Codpula Input + Sum, B=1CoDD, B=64Codpula Input, B=64Codpula Input + Sum, B=64
LLaDA-1.0GSM8K266.310.412.05.02.52.6
LLaDA-1.0MATH500184.67.28.32.81.41.4
LLaDA-1.0HumanEval184.87.18.22.41.21.2
LLaDA-1.0MBPP185.07.18.22.51.21.3
DreamGSM8K287.815.417.36.13.33.4
DreamMATH500212.011.312.83.51.91.9
DreamHumanEval184.79.911.13.11.61.7
DreamMBPP209.311.212.63.21.71.8
LLaDA-2.0GSM8K65.23.5—10.86.1—
LLaDA-2.0MATH50062.83.4—7.04.0—
SDARCode187.913.414.853.321.022.1

Overhead is per decoding step (one NFE: one backbone call and one circuit call). The circuit costs about the same on every backbone (2.6-3.8 ms at B=1, 14-35 ms at B=64); the backbone step does not. LLaDA-1.0 and Dream re-encode the full sequence at every step, so their step cost grows with sequence length (GSM8K generates 256 tokens, the other tasks 512, hence the higher GSM8K overhead) and with batch size (24-38 ms at B=1, 1.0-2.3 s at B=64). SDAR decodes one 16-token block against a KV cache, so its step is short (19.5 ms at B=1, 66 ms at B=64) and the same for every task; the same circuit cost is therefore a larger fraction, most of all at B=64. The LLaDA-2.0 forward grows only 5-8$\times$ from B=1 to B=64, so its overhead also rises with batch size.

(W2, Q1). Joint optimization is not a requirement for Codpula, and generalization is supported by more evidence.

Joint optimization. Our new LLaDA-1.0 experiments jointly optimize the backbone adapter, PC, and router, matching or exceeding the paper's staged Input-only model in 9 of 12 settings. Thus, two-stage training is not a requirement of Codpula. The original schedule was a backbone-specific optimization choice: Appendix A.1 discusses inconsistent or negative effects of standard SFT on LLaDA reported by d1 [1], DSFT [2], and Blockwise SFT [3], alongside related fine-tuning sensitivity reported by CDLM [4].

Generalization. Our protocol follows established practice in diffusion LM post-training: d1 (Section 4.1) [1] trains separate models per task, while JustGRPO (Section 5) [5] uses task-specific mathematical training splits and a coding dataset. We also directly evaluate generalization beyond the training domains: we reuse the paper's checkpoints on general QA benchmarks (MMLU and GPQA) without additional training, obtaining improvements across three backbones. For example, Dream improves from 18.2% to 45.9% on MMLU at 8 tokens/step, and LLaDA-2.0 improves from 18.3% to 31.5% on GPQA at 4 tokens/step.

Q2. KV caching is compatible with Codpula and orthogonal to our contribution. Circuit refinement operates on the backbone's outputs without changing its attention structure. Existing PC algorithms [6] also provide a basis for caching circuit computations. Fully exploiting these caching opportunities requires additional engineering, which we leave to future work.

Q3. We select a single checkpoint for each backbone, method variant, and task family, rather than independently selecting checkpoints for each evaluation benchmark or decoding budget. Specifically, we use MATH500 for math and MBPP for code, screening intermediate checkpoints on subsets before evaluating the candidates on the full selection benchmark. The selected checkpoints are then reused across the remaining benchmarks in each task family. We also evaluate these existing checkpoints on MMLU and GPQA without additional training or checkpoint selection.

We apply the same accuracy-based checkpoint-selection protocol to the independently tuned CE-SFT and CoDD baselines. This is consistent with best-checkpoint reporting in prior diffusion LM post-training work, including d1 [1], wd1 [7], and SPG [8]. We acknowledge that MATH500 and MBPP serve as model-selection benchmarks rather than strictly held-out evaluations. Importantly, improvements on the other benchmarks provide independent evidence beyond checkpoint selection.

We hope these clarifications and additional results address your concerns. Please let us know if any points would benefit from further clarification. We would appreciate it if you could consider raising your score.

[1] d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning, Zhao et al.

[2] DSFT: Inspiring Diffusion Large Language Models to Comprehend Mathematical and Logical Patterns, Chen et al.

[3] Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding, Sun et al.

[4] CDLM: Consistency Diffusion Language Models for Faster Sampling, Kim et al.

[5] The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models, Ni et al.

[6] Lossless Compression with Probabilistic Circuits, Liu et al.

[7] wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models, Tang et al.

[8] SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models, Wang et al.

Reviewer_4DnCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Codpula combines a Transformer with a tractable probabilistic circuit. Neural outputs modulate token potentials and optionally coupling weights. A dual-flow update preserves valid circuit parameters during conditional training. Joint conditional queries guide adaptive commitments. Evaluation covers four backbones, mathematical reasoning, and code generation.

Critical strengths
  1. The method trains token predictions through a joint conditional model. Exact circuit queries remain available. The dual-flow update handles signed derivatives while preserving positivity. This advances the existing CoDD formulation.

  2. The evaluation is broad. It includes CE-SFT, CoDD, decoding baselines, measured latency, and component ablations. Appendix A also accounts for generation-length effects. These checks strengthen the empirical case.

Critical weaknesses
  1. The benefit of context-dependent coupling weights is uneven. Tables 2 and 3 contain gains and regressions relative to input-only modulation. This variant is not evaluated on LLaDA. The broad improvement mainly establishes the complete trained system’s value. It does not yet establish that dynamic coupling is a generally useful component.

  2. Selection uncertainty is insufficiently documented. Appendix A.1 selects checkpoints using downstream generation accuracy. Appendix B.5 selects evidence temperature separately by decoding budget. The development/test separation is unclear. AIME uses only 30 problems per year. Large speedup ratios there are tied to low accuracy anchors and shorter generations.

Questions for authors
  1. Which datasets select checkpoints, temperatures, and deferral thresholds? Please report final results on untouched evaluation data with paired uncertainty.

  2. What is the gain from coupling modulation alone under matched training and decoding settings? Does it persist across seeds and a LLaDA backbone?

  3. The discussion should distinguish learned joint prediction from inference-time coordination and correction more clearly. Relevant approaches [1-4] provide training-free revision and continuous refinement, respectively. Which benefits specifically require a learned joint distribution? CoDD remains the primary matched modeling comparator. Additional comparisons should account for different training costs.

[1] Breaking the Factorization Barrier in Diffusion Language Models

[2] Dependency-Guided Parallel Decoding in Discrete Diffusion Language Models

[3] Locally Coherent Parallel Decoding in Diffusion Language Models

[4] Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs

5 comments

We thank the reviewer for recognizing our methodological advance over existing work and the breadth of our evaluation. Below, we provide additional ablations of coupling modulation, clarify the selection protocol and report paired uncertainty, and distinguish learned joint prediction from inference-time coordination while comparing their training costs.

(W1, Q2). Coupling modulation improves 11 of 12 LLaDA-1.0 settings under joint training.

Adding Sum modulation improves accuracy by approximately 3.5 percentage points on average, reaching +9.0 points on MATH500 at eight tokens per step and +9.3 points on MBPP at four tokens per step. The jointly trained Input + Sum model achieves the highest accuracy in seven of twelve settings.

We compare both modulation variants under staged and joint training (Table 1 below; Table 16 in the revised paper). The joint variants share the hyperparameters in Appendix A.1, Table 7, apart from router training. Staged Input + Sum adds a router-only stage with the circuit and backbone adapter frozen.

Table 1. LLaDA-1.0 accuracy (%). Column numbers indicate tokens per step. CE-SFT and Codpula (Input), staged results are from Table 3 of the paper. Bold indicates the best result in each column.

MethodGSM8K 248MATH500 248MBPP 248HumanEval 248
Base69.165.437.841.030.210.041.325.710.133.525.011.6
CE-SFT66.256.025.637.827.610.241.628.012.834.222.011.6
Codpula (Input), staged71.769.052.441.635.220.848.331.915.637.229.316.5
Codpula (Input + Sum), staged71.668.051.341.034.219.246.735.017.536.626.217.7
Codpula (Input), joint71.365.545.437.432.813.447.932.317.138.430.513.4
Codpula (Input + Sum), joint72.567.651.338.635.622.449.841.622.637.231.116.5

The benefit extends to general QA without additional training. We reuse the paper's checkpoints on MMLU's 1,531 validation questions. On Dream, Input + Sum improves over Input-only at all three decoding budgets, adding 3.4, 2.3, and 0.8 points, respectively.

Table 2. MMLU accuracy (%). Column numbers indicate tokens per step. Dream uses entropy-based decoding. Bold indicates the best result in each column.

MethodDream 248LLaDA-2.0 24816
Base52.437.618.272.360.631.39.2
CE-SFT58.948.140.072.559.321.07.9
CoDD63.249.928.072.666.036.715.5
Codpula (Input)60.957.045.173.071.054.733.9
Codpula (Input + Sum)64.359.345.9--------

These gains require little additional computation. Across the measured Dream configurations, Sum modulation adds 1.2-1.9 percentage points of per-step overhead at batch size 1, and at most 0.1 point at batch size 64.

Table 3. Per-step overhead over the base model (%); B denotes batch size.

BackboneTaskCoDD, B=1Codpula Input, B=1Codpula Input + Sum, B=1CoDD, B=64Codpula Input, B=64Codpula Input + Sum, B=64
DreamGSM8K287.815.417.36.13.33.4
DreamMATH500212.011.312.83.51.91.9
DreamHumanEval184.79.911.13.11.61.7
DreamMBPP209.311.212.63.21.71.8

Paired comparisons support coupling gains on both MBPP and MMLU. Relative to jointly trained Input-only Codpula, Sum modulation adds 9.3 ± 2.9 points on LLaDA-1.0 MBPP at four tokens per step. On Dream MMLU, it adds 3.4 ± 1.3 points at two tokens per step. Both gains exceed two paired standard errors.

Table 4. Paired accuracy differences on LLaDA-1.0 MBPP. Entries are percentage points, A - B, ± one standard error, pairing predictions on the same problems. Column numbers indicate tokens per step. Bold indicates a gain exceeding two standard errors.

A - B248
CE-SFT - Base+0.4 ± 2.4+2.3 ± 2.9+2.7 ± 2.0
CoDD - Base+1.9 ± 2.6+3.5 ± 3.0−0.4 ± 2.1
Codpula (Input), staged - Base+7.0 ± 2.6+6.2 ± 3.3+5.4 ± 2.4
Codpula (Input), joint - Base+6.6 ± 2.7+6.6 ± 3.0+7.0 ± 2.5
Codpula (Input + Sum), joint - Base+8.6 ± 2.8+16.0 ± 3.0+12.5 ± 2.8
Codpula (Input + Sum), joint - Codpula (Input), staged+1.6 ± 2.7+9.7 ± 2.9+7.0 ± 2.9
Codpula (Input + Sum), joint - Codpula (Input), joint+1.9 ± 2.6+9.3 ± 2.9+5.4 ± 2.8
Codpula (Input), joint - Codpula (Input), staged−0.4 ± 2.6+0.4 ± 3.2+1.6 ± 2.9

(W2, Q1). Selection protocol, generalization, and paired uncertainty.

Checkpoint selection. Appendix A.1 now specifies one checkpoint per backbone and task family, selected using a subset of MATH500 for math and a subset of MBPP for code, except SDAR, which was selected on HumanEval+. GSM8K, HumanEval, AIME, MMLU, and GPQA are not used for checkpoint selection. CE-SFT and CoDD are independently tuned using the same accuracy-based selection principle. This is consistent with best-checkpoint reporting in d1 [1], wd1 [2], and SPG [3]. Evidence temperatures are selected separately for each benchmark and decoding budget; adaptive operating points come from sweeps on that benchmark.

The selected checkpoints transfer to general QA without additional training. Task-specific post-training also appears in d1 (Section 4.1) [1] and JustGRPO (Section 5) [4]. We directly evaluate transfer by reusing the math checkpoints for Dream and LLaDA-2.0 and the code checkpoint for SDAR on MMLU and GPQA. No checkpoint selection uses these benchmarks. Improvements extend across all three backbones: for example, Dream improves from 18.2% to 45.9% on MMLU at eight tokens per step, and LLaDA-2.0 from 18.3% to 31.5% on GPQA at four tokens per step.

Paired comparisons support gains beyond CE-SFT. On Dream MMLU, Codpula (Input + Sum) improves over CE-SFT by 5.4-11.2 percentage points, with each gain exceeding three paired standard errors. We compare predictions on the same 1,531 validation questions and will add paired uncertainty to the revised result tables.

Table 5. Paired accuracy differences relative to CE-SFT on Dream MMLU, in percentage points ± one standard error. Column numbers indicate tokens per step. Bold marks positive gains exceeding two standard errors.

Method - CE-SFT248
Base-6.5 ± 1.5-10.5 ± 1.5-21.8 ± 1.4
CoDD+4.2 ± 1.4+1.8 ± 1.5-12.0 ± 1.5
Codpula (Input)+2.0 ± 1.4+8.9 ± 1.5+5.1 ± 1.6
Codpula (Input + Sum)+5.4 ± 1.4+11.2 ± 1.5+5.9 ± 1.5

The QA results therefore establish checkpoint transfer.

Coding and math gains persist when temperature is selected on another benchmark. We retain the existing checkpoints and select evidence temperature per method and decoding budget on a source benchmark, then apply it to the target benchmark without further temperature tuning. On Dream GSM8K, Input + Sum improves over Base by 5.1-13.1 points and over CE-SFT by 9.0-17.7 points. On SDAR MBPP+, its mean gain over CE-SFT is 8.73 points across four budgets (95% paired CI: 6.02-11.51; Holm-adjusted p<0.0001).

Table 6. Temperature-transfer accuracy (%). Dream transfers temperatures from MATH500 to GSM8K and MMLU using entropy decoding; SDAR transfers from HumanEval+ to MBPP+. Column numbers indicate tokens per step. Bold marks the highest accuracy in each column.

MethodDream GSM8K 248Dream MMLU 248SDAR MBPP+ 1248
Base72.649.316.752.437.618.262.241.524.68.5
CE-SFT69.144.79.658.948.140.061.942.623.06.6
CoDD75.462.927.163.249.928.065.142.924.98.5
Codpula (Input)75.561.715.160.957.044.766.952.933.117.5
Codpula (Input + Sum)78.262.421.864.359.343.867.552.131.717.7

For Dream, the MATH500-selected temperatures at 2/4/8 tokens per step are 0.1/0.1/0.3 for both Codpula variants and 0.1 throughout for CoDD. These are temperature-transfer checks on previously examined benchmarks; SDAR checkpoint selection also involved the code benchmarks.

Our efficiency evidence extends beyond the small AIME benchmarks and shorter outputs. We agree that AIME ratios need qualification: each year contains 30 problems, one answer changes accuracy by 3.3 points, and the AIME25 speedup is anchored at 10% accuracy. We will present these ratios as illustrative. Appendix A.3.2 distinguishes their generation-length contribution: Codpula produces 15-48% fewer tokens at matched tokens per step (Table 13).

Independently, fixed-length evaluations demonstrate parallel-decoding gains. On GSM8K, LLaDA-2.0 achieves 75.3% versus 70.3% accuracy with half as many NFEs, while both methods generate 512 tokens. Dream's 4.02$\times$ MBPP speedup likewise uses a fixed length of 512 at matched accuracy. Our additional fixed-length SDAR evaluation improves HumanEval+ and MBPP+ accuracy by 10.3 and 9.8 points, respectively, at the same 163 NFEs, further showing that the accuracy gains persist without shorter generations.

Q3. Codpula learns dependencies within each proposal, complementing inference-time coordination and revision. Inference-time methods improve how predictions are committed or refined. Codpula learns a joint conditional distribution before commitment, providing three capabilities:

  • Dependent multi-token proposals from one backbone evaluation. The circuit couples token predictions within the same step. This is particularly useful at high parallelism: on LLaDA-2.0 GSM8K at 16 tokens per step, Codpula achieves 12.7%, compared with 2.4% for ReMDM and 2.8% for Tolerator (Table 1).
  • Exact conditional queries for scheduling. The circuit computes the dependence scores in Eq. 12 from the existing backbone outputs, requiring additional circuit queries but no additional backbone evaluation.
  • Backbone adaptation through a joint likelihood. With the circuit held fixed, adding the trained adapter improves LLaDA-1.0 MBPP by 4.7, 4.7, and 2.3 points at 2, 4, and 8 tokens per step (Table 20). These capabilities complement decoding strategies: Codpula supports confidence, margin, and entropy rules, and can be combined with revision. Training-free methods remain competitive in some regimes; for example, DUS exceeds Codpula on LLaDA-2.0 GSM8K at 8 tokens per step (60.3% vs. 57.9%). We will clarify this distinction and discuss reviewer's [1–4] accordingly.

CoDD provides the direct modeling comparison. On Dream MMLU, Codpula (Input + Sum) improves over CoDD by 1.1, 9.4, and 17.9 points at 2, 4, and 8 tokens per step. On LLaDA-1.0 and Dream at batch size 1, Codpula's uniform-decoding overhead is 7.1-17.3%, versus 184.6-287.8% for CoDD. Codpula's well-optimized kernels contribute to its lower per-step overhead.

Table 7. Codpula's training cost is comparable to the trained baselines. We report Dream math training costs using the same GPU model. CoDD's totals include both precomputation and circuit training; checkpoint-evaluation costs are excluded for all methods.

MethodGPUsGPU-hours to evaluated checkpointGPU-hours for full run
CE-SFT69.518.3
CoDD619.321.9
Codpula (Input)29.123.9
Codpula (Input + Sum)613.128.0

Full-run costs are 1.3-1.5$\times$ CE-SFT's and 1.1-1.3$\times$ CoDD's; both evaluated Codpula checkpoints require fewer GPU-hours than CoDD's. CE-SFT was also trained beyond its selected checkpoint, which occurs at 9.5 of 18.3 GPU-hours. Training-free methods incur no training cost; Codpula incurs this cost once per backbone and task family. Figures 3, 4, and 9 include its inference overhead in the accuracy–latency comparisons.

We hope our response has helped clarify these points and we would appreciate it if you could consider raising your score.

[1] d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning, Zhao et al.

[2] wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models, Tang et al.

[3] SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models, Wang et al.

[4] The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models, Ni et al.

Reviewer_TSo1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Codpula, a context-adaptive joint denoising framework for diffusion language models (DLMs). The method combines a Transformer with a probabilistic circuit to model dependencies among simultaneously predicted tokens. The authors introduce a dual-flow multiplicative update for training and a dependency-aware decoding strategy to guide token commitment. Experiments across four DLM backbones on mathematical reasoning and code generation benchmarks demonstrate improvements in prediction accuracy and accuracy–latency trade-offs.

Critical strengths
  1. The paper addresses the factorization barrier in DLMs by explicitly modeling dependencies among simultaneously predicted tokens. This is a well-motivated problem, particularly for improving the reliability of parallel decoding.

  2. The paper evaluates the proposed method across four DLM backbones and multiple mathematical reasoning and code generation benchmarks. The experiments include comparisons with strong baselines, different decoding configurations, and ablation studies, demonstrating competitive performance and improved accuracy–latency trade-offs.

Critical weaknesses
  1. The proposed decoding strategy defers tokens with high conditional dependence scores. However, strong dependence does not necessarily indicate unreliable predictions, as highly dependent tokens may also form correct and consistent joint predictions. The paper could better justify why the magnitude of conditional dependence is an appropriate criterion for deferring token commitments and provide more direct empirical evidence supporting this design choice.

  2. The reported speedups appear to be partly driven by shorter generated sequences rather than more efficient parallel decoding. For example, Codpula generates substantially fewer tokens than the baseline on AIME24. It would be valuable to further disentangle the contributions of reduced generation length and improved decoding efficiency to the reported speedups.

Questions for authors

See Weaknesses.

3 comments

We thank the reviewer for recognizing the motivation, the breadth of our evaluation, and the significance of our performance in accuracy-latency trade-offs. Below, we clarify why our adaptive decoder postpones tokens with high dependence scores, and show that the efficiency gains extend beyond shorter outputs.

W1: Dependence guides contextual refinement; our additional experiments demonstrate its practical benefit.

The magnitude of conditional dependence is a good deferral criterion because it separates tokens that are safe to commit in parallel and requires only low-cost circuit queries and no additional backbone evaluation. Updated Figures 3-4 show that adaptive commitment improves Codpula’s accuracy-latency Pareto frontier, achieving higher accuracy at matched runtime. We provide additional experiments below to further demonstrate the practical value.

We agree that highly dependent tokens can also form correct joint predictions. Representing them is the purpose of our joint denoiser: if the valid alternatives are (A, A) and (B, B), a factorized sampler must either risk the inconsistent combinations (A, B) and (B, A) or collapse onto one alternative, sacrificing diversity. Codpula can represent and sample both alternatives consistently. Our dependence-adaptive decoding complements this capability. When the dependence is large, the choice between alternatives rests on the learned coupling, so we commit the less dependent tokens first and let the backbone re-predict the rest with that context visible.

Specifically, δi\delta_i in Eq. 12 is dependence conditional on the visible context. If the context already determines a group of tokens, each proposal has probability near one with or without the other proposals, so δi≈0\delta_i \approx 0 and the group is committed together, however tightly its tokens are linked. Conversely, because probabilities are at most one, Eq. 12 implies min⁡{p(x^i∣xt), p(x^i∣x^−i,xt)}≤e−δi\min\{p(\hat{x}^i \mid x_t),\ p(\hat{x}^i \mid \hat{x}^{-i}, x_t)\} \le e^{-\delta_i}. A large score therefore means one of two things: the context alone leaves the token undetermined (positive dependence), or the proposal is unlikely given the other proposals (negative dependence). For example, δi≈0.74\delta_i \approx 0.74 in Fig. 12 bounds one of these probabilities by 0.480.48. Fig. 13 shows both regimes for the same token: ' string' scores 0.00000.0000 in prose and 1.131.13 inside the identifier.

Joint sampling captures dependencies under the current distribution. Deferral lets the backbone re-predict the tokens whose value is not yet settled, after the accepted tokens become visible and the neural potentials are updated. In Fig. 12, the deferred tokens' scores decrease by $20-1000 \times $ after one step, and they are then committed. Dependence thus serves as a scheduling heuristic.

Our results support dependence as a useful scheduling signal: adaptive Codpula improves over uniform decoding and outperforms existing adaptive methods at several evaluated time budgets. We compare these Codpula variants with the base model, Dynamic Confidence, and EB Sampler on LLaDA-2.0 and SDAR. At batch size 1, each method reports its highest observed accuracy within the same wall-clock budget.

Table 1. Accuracy (%); time budgets are seconds per question. Codpula uses Input modulation for LLaDA-2.0 and SDAR/HumanEval+, and Input + Sum modulation for SDAR/MBPP+, consistently between uniform and adaptive decoding.

Backbone / benchmarkTime budget (s/q)BaseDynamic ConfidenceEB SamplerCodpula uniformCodpula adaptive
LLaDA-2.0 / GSM8K7.536.764.3—57.972.3
LLaDA-2.0 / GSM8K10.036.770.3—57.974.1
LLaDA-2.0 / MATH50020.028.254.2—41.062.8
SDAR / MBPP+0.5016.415.324.334.439.7
SDAR / MBPP+1.0041.556.148.452.960.1
SDAR / HumanEval+0.7028.137.827.436.040.9
SDAR / HumanEval+1.2551.268.373.259.863.4

Adaptive commitment improves both accuracy and runtime at selected operating points. On GSM8K, it increases accuracy from 57.9% to 72.3% while reducing time from 7.16 to 6.18 seconds. On MATH500, it achieves the same 62.8% accuracy in 17.96 versus 28.80 seconds, a 37.6% time reduction. The comparisons with Dynamic Confidence and EB Sampler further demonstrate competitive accuracy-latency trade-offs. Revised Figures 3-4 provide the full curves, including CoDD and non-adaptive Codpula.

W2: Fixed-length evaluations demonstrate parallel-decoding gains independently of shorter outputs. Following the reviewer's suggestion, we distinguish these gains from the generation-length reductions on AIME in Appendix A.3.2.

Fixed-length comparisons. In Fig. 4, LLaDA-2.0 improves the accuracy-latency frontier on GSM8K and MATH500 with fixed generation lengths of 512 and 1,024 tokens, without early stopping. On GSM8K, Codpula achieves 75.3% accuracy at four tokens per step, exceeding the base model's 70.3% at two tokens per step while using half as many NFEs (Table 1 above). Dream's $4.02 \times $ speedup on MBPP likewise compares both methods at matched accuracy with the same fixed generation length of 512. These gains cannot arise from shorter outputs.

Table 2. SDAR with fixed-length generation. Every question decodes all 512 tokens without EOS-based early stopping. NFE is identical across methods at each tokens-per-step setting. Each cell reports accuracy (%) / seconds per question.

BenchmarkTokens/stepNFEBaseCodpula (Input)Codpula (Input + Sum)
HumanEval+155268.9 / 10.972.6 / 12.773.8 / 12.7
HumanEval+229251.2 / 5.859.8 / 6.659.8 / 6.7
HumanEval+416328.1 / 3.336.0 / 3.738.4 / 3.7
HumanEval+89810.4 / 2.017.1 / 2.217.7 / 2.2
MBPP+155262.2 / 11.867.7 / 12.769.3 / 12.7
MBPP+229241.5 / 5.955.6 / 7.152.9 / 6.6
MBPP+416324.6 / 3.234.4 / 3.934.4 / 3.6
MBPP+8988.5 / 2.018.5 / 2.318.3 / 2.1

The gains also persist when early stopping is disabled on SDAR. We additionally decode all 512 tokens for every question, using identical NFE budgets across methods. At four tokens per step (163 NFEs), Codpula (Both) improves HumanEval+ accuracy from 28.1% to 38.4% and MBPP+ from 24.6% to 34.4%. Runtime increases modestly from 3.3 to 3.7 seconds and 3.2 to 3.6 seconds, respectively. These accuracy gains therefore persist without any reduction in generation length or backbone evaluations.

Shorter outputs additionally contribute on AIME. Codpula generates 15-48% fewer tokens across the four settings in paper's Table 13. Both Codpula and CE-SFT produce fewer self-correction phrases, including after normalization by output length, suggesting that fine-tuning changes generation behavior. On AIME24 at two tokens per step, Codpula achieves 53.3% versus 40.0% accuracy with 27.8% fewer tokens. We now distinguish these end-to-end gains, which include reduced generation length, from the fixed-length parallel-decoding gains above.

Per-step costs are reported separately. Updated Table 15 measures static and adaptive decoding overhead across all four backbones at batch sizes 1 and 64. The summary below separates per-step computation from the number of steps and generated tokens.

Table 3. Per-NFE overhead over the base model (%). B denotes batch size; adaptive ranges cover the query widths evaluated in Table 15.

BackboneCoDD, B=1Codpula static, B=1Codpula adaptive, B=1CoDD, B=64Codpula static, B=64Codpula adaptive, B=64
LLaDA-1.0184.67.210.8-13.42.81.45.6-8.5
Dream212.012.818.2-19.13.51.98.3-9.4
LLaDA-2.062.83.44.7-6.37.04.014.9-27.7
SDAR187.914.829.4-36.053.322.175.0-155.6

LLaDA models use Input modulation; Dream and SDAR use Input + Sum. Adaptive costs assume a full-width dependence query at every step, giving the upper-bound estimates described in Table 15. Figures 3-4 include circuit and adaptive-query costs in the overall decoding time, demonstrating Codpula's Pareto dominance over CoDD and improved accuracy-latency trade-offs against the other baselines.

We hope these clarifications and additional results address your concerns. Please let us know if any points would benefit from further clarification. We would appreciate it if you could consider raising your score.

We thank the reviewer again for the thoughtful questions and hope our rebuttal and additional results have addressed the concerns. As the discussion period is close to end, we would be grateful if the reviewer could revisit their review considering our revisions. We are happy to answer any follow-up questions or clarify any remaining points.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Responses to Reviewers and Revisions on Submission (1/2)

We thank all reviewers for their constructive feedback. TSo1 and DSPn recognize the importance of addressing the factorization barrier and the improved accuracy-latency trade-offs. 4DnC and DSPn recognize the methodological advance over existing work and the soundness of the proposed learning algorithm. TSo1 and 4DnC highlight the breadth of evaluation, including strong baselines and component ablations.

The concerns center on component effectiveness, selection and generalization, and the interpretation of efficiency gains. We have strengthened the evidence as follows:

1. Sum modulation adds accuracy gains beyond Input-only Codpula (4DnC W1/Q2, DSPn W1)

Extending the paper's Dream and SDAR evaluation, we added LLaDA-1.0 comparisons under both staged and joint training. Under joint training, Sum modulation improves 11 of 12 settings, by 3.5 points on average (4DnC and DSPn responses, Table 1; revised PDF, Table 16). Paired comparisons support these gains: +9.3 ± 2.9 points on MBPP at four tokens per step and +3.4 ± 1.3 on Dream MMLU at two tokens per step, with ± denoting one standard error (4DnC response, Tables 2 and 4 and accompanying paired analysis). Sum modulation adds at most 1.9 percentage points of per-step overhead across the measured configurations (DSPn response, Table 4; revised PDF, Table 15).

2. Evaluation breadth and generality (DSPn W1/W2)

  • Existing checkpoints improve general QA beyond CE-SFT, without additional training. We reuse the paper's math and code checkpoints on MMLU and GPQA. At eight tokens per step on MMLU, Codpula achieves 45.9% versus CE-SFT's 40.0% on Dream, and 54.7% versus 21.0% on LLaDA-2.0. On GPQA at four tokens per step, LLaDA-2.0 achieves 31.5% versus CE-SFT's 18.8% (DSPn response, Tables 2-3; revised PDF, Table 4).
  • LLaDA-1.0 supports joint optimization; staged training is optional. Jointly trained Input + Sum matches or exceeds the paper's staged Input-only model in 9 of 12 settings (DSPn response, Table 1; revised PDF, Table 16).
  • Direct CoDD comparisons demonstrate better accuracy-latency trade-offs. Revised Figures 3-4 now include CoDD, and we report overhead across all four backbones. On LLaDA-1.0 and Dream at batch size 1, uniform-decoding overhead is 7.1-17.3% for optimized Codpula versus 184.6-287.8% for CoDD's original implementation (DSPn response, Table 4; revised PDF, Table 15).
  • Codpula is compatible with backbone KV caching. Circuit refinement preserves the backbone's attention structure; further caching optimizations remain for future work (DSPn response, Q2).

3. Gains persist when temperatures are selected on another benchmark. (4DnC W2/Q1, DSPn Q3)

Following the best-checkpoint reporting practice in prior diffusion-LM post-training work (d1, wd1, and SPG), we select one checkpoint per backbone and task family using downstream generation accuracy. CE-SFT and CoDD are independently selected using the same accuracy-based principle. Math selection uses a subset of MATH500; code selection uses a subset of MBPP, except SDAR, which was selected on HumanEval+. Evidence temperatures and adaptive operating points use benchmark results, so we additionally report paired uncertainty and retrospective temperature-transfer checks. On Dream MMLU, Codpula (Input + Sum) exceeds CE-SFT by 5.4-11.2 points, each exceeding three paired standard errors (4DnC response, Table 5). When temperatures are instead selected on MATH500, gains persist: 5.1-13.1 points over Base on GSM8K and 3.8-11.2 points over CE-SFT on MMLU (4DnC response, Table 6, temperature transfer). These checks strengthen the evidence while leaving the benchmark-based selection explicit.

4. Fixed-length controls demonstrate gains independently of shorter outputs. (TSo1 W2, 4DnC W2)

With fixed generation length and no early stopping, LLaDA-2.0 achieves 75.3% versus 70.3% on GSM8K with half as many backbone evaluations (revised PDF, Table 1 and Figure 4). New SDAR evaluations likewise hold generation length and NFE fixed, improving HumanEval+ and MBPP+ by 10.3 and 9.8 points, respectively, at 163 NFEs (TSo1 response, Table 2). For AIME, we explicitly acknowledge that shorter generations contribute 15-48% fewer tokens (revised PDF, Appendix A.3.2 and Table 13).

5. Dependence guides scheduling, with empirical gains over uniform decoding. (TSo1 W1)

We clarify that the score measures a proposal's sensitivity to other provisional tokens, conditional on the visible context; it provides a scheduling signal without guaranteeing correctness. At a selected LLaDA-2.0 GSM8K operating point, adaptive Codpula improves accuracy from 57.9% to 72.3% while reducing runtime from 7.16 to 6.18 seconds. We also compare against existing adaptive methods under shared time budgets (TSo1 response, Table 1 and accompanying discussion; full curves in revised Figures 3-4).

AuthorsOpenReview ↗

Summary of Responses to Reviewers and Revisions on Submission (2/2)

6. Learned joint prediction complements inference-time coordination, with training costs reported explicitly. (4DnC Q3)

Codpula learns dependencies within multi-token proposals, supports exact circuit queries without additional backbone evaluations, and adapts the backbone through a joint likelihood. These capabilities complement inference-time commitment and revision (4DnC response, Q3; backbone-adaptation ablation in revised Table 20). On Dream math, evaluated Codpula checkpoints require 9.1-13.1 GPU-hours, versus 19.3 for CoDD. Full runs require 1.1-1.3$\times$ CoDD's and 1.3-1.5$\times$ CE-SFT's GPU-hours, with checkpoint-evaluation costs excluded consistently (4DnC response, Q3 training-cost table).

Revisions. For ease of review, we have marked all changes in blue in the revised PDF. The revised paper includes the new LLaDA-1.0 ablation, QA results, fixed-length SDAR results, CoDD in Figures 3-4, the overhead and training-cost tables, the explicit selection protocol, and discussion on AIME benchmarks.

We thank all reviewers and the Area Chair again for their time and constructive feedback, which have helped us strengthen the paper. We welcome any further questions or requests for clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.