acceptodds
Under review as a conference paper at ICLR 2027

Inter-Trajectory Dependence Limits Answer Coverage in Local dLLM Repair

10 views

Abstract

Test-time compute scaling improves large language model performance by allocating more computation during inference. For diffusion large language models (dLLMs), sequential repair offers a natural instantiation: it iteratively remasks low-confidence tokens and regenerates them while retaining the surrounding completion. However, additional repair rounds often yield limited performance gains. While recent work has studied inference dynamics within individual trajectories, our focus is inter-trajectory dependence: whether successive repair candidates explore distinct answer states. We introduce a diagnostic framework that combines answer-level metrics, hyperparameter sweeps, and remasking interventions to characterize this dependence. We assess pairwise answer agreement, unique-answer counts, oracle pass@, and correction/regression rates across four dLLMs and three verifiable tasks. We find that sequential local repair produces highly correlated candidates with substantially lower answer coverage than independent sampling. The evaluated local-remasking variants do not reliably close this gap. These findings suggest prioritizing trajectory-level diversification when allocating test-time compute for answer coverage, with local repair used for refinement. Our study adds an inter-trajectory perspective to the analysis of dLLM inference dynamics and provides practical guidance for allocating inference compute.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Models, Test-Time Compute Scaling, Local Repair

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_LP6FOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares repeated local remasking of a completed diffusion-language-model response with independent sampling. Across four models and three main reasoning/code benchmarks, repair produces fewer distinct answers and substantially lower oracle pass@kk, despite sometimes improving accuracy. The contribution is an empirical diagnosis using answer agreement, coverage, correction or regression rates, remasking interventions, and budget comparisons. The results favor spending inference compute on independent trajectories before extending a single repair chain.

Critical strengths
  1. The empirical gap is substantial and practically relevant. At similar initial accuracy, LLaDA-8B reaches 82.8% oracle pass@16 with independent sampling versus 29.0% with repair on Countdown (Table 2). Approximately compute-matched comparisons also favor independent sampling, even when repair can select from more candidate states. This provides useful evidence for allocating inference compute.

  2. The paper checks several plausible alternative explanations. Chain-level selection, semantic canonicalization, parameter sweeps, and guided-remasking interventions strengthen the observation beyond a simple accuracy comparison. Reporting exceptions, including Dream's Countdown gains and favorable creative-writing results, also helps establish the finding's scope.

Critical weaknesses
  1. The central dependence explanation remains unresolved. Partial repair uses joint regeneration, whereas independent sampling uses 32-token blockwise decoding. In Table 18, changing the full-remask regeneration schedule alone raises Countdown pass@16 from 46.8% to 80.0%, close to the 83.0% independent baseline. This is a large effect without retained-state dependence. Matching the first candidate's accuracy does not match later candidates' distributions, and ρ^\hat{\rho} also reflects marginal answer concentration. The authors acknowledge these limitations, but they leave unclear how much dependence contributes to the measured gap.

  2. The contribution beyond closely related work needs stronger support. Frković et al. [1] already examine limited remasking benefits, decoding-setting sensitivity, and diversity loss. This work is absent from the discussion. The present focus on successive states within one repair chain is distinguishable, but the experiments use a generic post-hoc repair operator without directly evaluating an established decoder such as WINO [2] or CORE [3]. The resulting advance over existing remasking analyses is not yet clear.

References

[1] Frkovic, Stipe, et al. "Re-evaluating Confidence Remasking in Masked Diffusion Language Models." arXiv preprint arXiv:2606.12232 (2026).

[2] Hong, Feng, et al. "Wide-in, narrow-out: Revokable decoding for efficient and effective dllms." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Zhai, Kevin, et al. "Core: Context-robust remasking for diffusion language models." arXiv preprint arXiv:2602.04096 (2026).

Questions for authors
  1. Evidence needed for the central explanation: Does the partial-remasking coverage gap persist when repair uses the same blockwise regeneration kernel and comparable denoising allocation? A comparison between within-chain candidate sets and candidates drawn across independently initialized repair chains at matching round indices would help separate serial dependence from marginal concentration.

  2. Contribution and scope clarification: Which findings go beyond Frković et al. [1], and do those findings hold for an established remasking method, such as WINO [2] or CORE [3], beyond the post-hoc operator evaluated here?

  3. Implementation clarification: Does Dream's confidence recomputation apply the one-position logit shift used in its official sampler? Table 9 reports answer confidences of only 0.008–0.014. Please explain these values and confirm the alignment in both the repair operator and the confidence audit. These values warrant clarification but do not, by themselves, establish an implementation error.

References

[1] Frkovic, Stipe, et al. "Re-evaluating Confidence Remasking in Masked Diffusion Language Models." arXiv preprint arXiv:2606.12232 (2026).

[2] Hong, Feng, et al. "Wide-in, narrow-out: Revokable decoding for efficient and effective dllms." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Zhai, Kevin, et al. "Core: Context-robust remasking for diffusion language models." arXiv preprint arXiv:2602.04096 (2026).

4 comments
AuthorsOpenReview ↗

Response to Reviewer LP6F (1/2)

We thank the reviewer for the careful review. The shared evidence is in the General Response, GR-1 to GR-4.


Comment/Question:

  1. The central dependence explanation remains unresolved. Partial repair uses joint regeneration, whereas independent sampling uses 32-token blockwise decoding. ...

Answer: We ran both controls the reviewer proposes, and they separate the explanations.

Kernel control (GR-1). With the independent sampler's blockwise kernel and a matched denoising budget, partial repair at α=0.2 still falls 17–54 pp below independent sampling on all three tasks, with every interval excluding zero; Dream-7B is similar. The schedule effect the reviewer cites from Table 18, now Table 19, is real at α=1, but with 80% retained the same swap moves coverage by only 0.5 pp.

Serial dependence versus marginal concentration (GR-2). Pooling the final states of eight independently initialized chains gives coverage with no detectable difference from independent sampling on all three tasks, and nearly the same agreement. Pooling the states within one chain loses 18–50 pp. The marginal distribution of repaired states is therefore not concentrated; the gap comes from dependence between successive states of the same chain. This is the comparison the reviewer suggested, and it is in Appendix G.2.

The revised Section 5.2 states the dependence claim at this level, together with the limitation that retained context, bidirectional attention and schedule are not fully decomposed.


Comment/Question:

  1. The contribution beyond closely related work needs stronger support. Frković et al. already examine limited remasking benefits, decoding-setting sensitivity, and diversity loss. ...

Answer: Thank you for the reference; the revision now cites and discusses Frković et al. They compare decoders that remask during generation, namely WINO, Saber and COVER, against confidence-based unmasking alone. They find that remasking matches the baseline at short block lengths and helps only at longer blocks, that WINO loses diversity under stochastic decoding, and that the dLLM rather than the remasking rule is the bottleneck.

Our question is different: whether the successive states of a post-completion repair chain form a useful candidate pool for test-time scaling. Three findings go beyond their work.

  1. We measure the pool directly, with oracle pass@k, agreement and unique answers on the ordered chain. This separates whether a correct answer is present from how it would be selected.
  2. We show that the coverage loss is a chain-level effect rather than a decoding-setting effect. It persists under a matched blockwise kernel and budget (GR-1), and it is not detected when states are pooled across chains (GR-2).
  3. We evaluate established methods in both roles (GR-3). The released CORE, WINO, Saber, RCR and COVER decoders produce independent trajectories and keep their pools close to independent sampling, whereas chains that use CORE- or WINO-inspired selectors concentrate as much as our operator.

The third point places the effect in sequentially re-editing one retained chain, not in remasking as such. This agrees with Frković et al.'s observation that remasking is not the bottleneck, and it answers a question their decoder-level comparison does not ask.


AuthorsOpenReview ↗

Response to Reviewer LP6F (2/2)

Comment/Question:

  1. Evidence needed for the central explanation: Does the partial-remasking coverage gap persist when repair uses the same blockwise regeneration kernel ...

Answer: Yes to both parts; see our answer to Weakness 1 above, with the tables in GR-1 and GR-2. Under the same kernel and denoising allocation the gap persists, and pools compared at matching round index show that across-chain pools behave like independent sampling while within-chain pools do not.


Comment/Question:

  1. Contribution and scope clarification: Which findings go beyond Frković et al. [1], and do those findings hold for an established remasking method ...

Answer: See our answer to Weakness 2 above and GR-3. In short, the chain-level separation through the matched kernel and the within- versus across-chain pools goes beyond Frković et al. Among established methods, the released decoders that produce independent trajectories do not concentrate, while repair chains using CORE- or WINO-inspired selectors do.


Comment/Question:

  1. Implementation clarification: Does Dream's confidence recomputation apply the one-position logit shift used in its official sampler? ...

Answer: The repair operator applies the shift; the separate audit did not. Dream predicts token i from position i−1, and our repair operator shifts the logits accordingly before the softmax, so all Dream repair results are unaffected. The audit's omission explains the anomalous confidence values. With the shift, answer confidence is about 0.95 on both tasks, and the exposure–agreement correlation is unchanged. GR-4 gives the corrected values, and Tables 10–11 and the text are corrected. Thank you for raising this.

Thank you for providing additional evidence and addressing my concerns about the paper’s novelty. I have no further questions. I will maintain my rating and increase my confidence score from 3 to 4.

Dear Reviewer LP6F,

Thank you for your follow-up and for engaging so carefully with our response. Your suggestions, in particular the within-chain versus across-chain comparison, the blockwise kernel control and the Dream alignment check, made the paper clearer and more rigorous, and the revision now reflects them.

We are grateful for your support and for the increased confidence.

Best regards,

The Authors

Reviewer_ALT2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies sequential local repair for diffusion large language models, a test‑time scaling method that iteratively remasks low‑confidence tokens while preserving most of the existing generation. The core finding is inter‑trajectory dependence: repeated local repair produces heavily correlated candidate answers, leading to answer‑coverage collapse even with substantial additional inference compute.

The authors develop a set of answer‑level diagnostics including pairwise answer agreement, unique‑answer count, oracle pass@k, and correction/regression transition metrics. Across four dLLMs and multiple benchmarks, they compare sequential repair against independent best‑of‑N sampling, ODD, and various remasking variants. Their ablation shows standard local remasking strategies cannot close the coverage gap versus independent sampling. Only full‑remask restarts using block‑wise generation recover comparable diversity. The paper advocates a compute‑allocation principle: prioritize independent trajectories for exploration, and apply local repair for subsequent refinement.

Critical strengths
  1. This work brings a valuable inter‑trajectory perspective, complementing existing literature that mainly analyzes intra‑trajectory dynamics in dLLMs. It highlights a real practical pitfall: simply increasing repair rounds does not guarantee broader answer exploration.
  2. The evaluation carefully disentangles candidate‑pool coverage from selector quality via oracle pass@k. Semantic canonicalization is used to rule out superficial text variation as a confounder.
  3. Comprehensive ablations cover remasking heuristics, remask ratios and temperature. Controlled interventions including attention‑blocking and full‑remask restarts reveal the importance of generation schedule alongside retained context conditioning. Budget accounting uses measured forward passes rather than nominal step counts.
  4. Empirical findings are supported across multiple model families and tasks. The proposed resampling‑then‑repair hybrid provides a concrete, actionable baseline for practitioners.
Critical weaknesses
  1. The relative contribution of retained context, bidirectional attention and generation schedule under partial remasking (α<1\alpha<1) cannot be quantitatively decomposed. This limitation is largely confined to appendices and deserves more prominent discussion in the main text. The observed correlation between answer‑token exposure and answer agreement is observational and should not be interpreted as causal.
  2. Strong quantitative evidence is limited to verifiable tasks with extractable ground‑truth answers. Open‑ended generation results are only descriptive. The manuscript should better articulate the scenarios where local repair remains useful beyond post‑sampling refinement. Format corruption under repair on multiple‑choice benchmarks also warrants further practical discussion.
  3. Existing dLLM search algorithms (Prism, S³, RFG) that combine local edits and trajectory‑level exploration are only briefly mentioned in discussion. Direct empirical comparison with these methods is absent, which weakens the broader applicability of the conclusions.
  4. Dream‑7B exhibits task‑dependent repair performance: large gains on Countdown but degradation on other tasks. The model‑task interaction driving this anomaly is not sufficiently analyzed in the main body.
Questions for authors
  1. For partial‑remask repair, attention‑blocking yields only modest improvements in answer diversity. Aside from conditioning on retained context, what are the remaining drivers of high inter‑candidate answer correlation?
  2. Why does joint‑update full‑remask restart still produce low answer diversity even with no prior generation retained, while block‑wise restart nearly matches independent sampling performance?
  3. Dream‑7B shows high answer‑token exposure but only sees meaningful repair gains on Countdown. What model‑task interaction explains this discrepancy?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ALT2 (1/2)

We thank the reviewer for the careful review. The shared evidence is in the General Response, GR-1 to GR-4.


Comment/Question:

  1. The relative contribution of retained context, bidirectional attention and generation schedule under partial remasking (α<1) cannot be quantitatively decomposed. ...

Answer: We agree that a full decomposition is not possible, and the revised Section 5.2 and Discussion now say so directly. The new controls do bound the role of the schedule. Without retained context the schedule matters a great deal: a blockwise instead of a joint restart raises Countdown pass@16 from 46.8% to 80.0% (Appendix C.6). With 80% retained, the same swap changes coverage by 0.5 pp, and blockwise partial repair remains far below independent sampling (GR-1). The schedule thus matters a lot without retention and little with it. We do not separate retained context from bidirectional attention, because the model attends bidirectionally in every condition.

On exposure, we agree that its relationship with agreement is observational. The main text now presents it as a descriptive association, with the Dream values corrected as described in GR-4, and makes no causal claim.


Comment/Question:

  1. Strong quantitative evidence is limited to verifiable tasks with extractable ground-truth answers. Open-ended generation results are only descriptive. ...

Answer: Coverage requires an extractable answer, so our quantitative claims are limited to verifiable tasks, and our open-ended results are descriptive. The revised Discussion and Limitations now state this scope explicitly.

Where repair remains useful. We see three cases. The first is refining a single answer: the states of a repair chain are on average more accurate than independent samples on all three tasks, even though the chain covers fewer answers (GR-3a). The second is refining a diverse pool, where resample-then-repair slightly improves Countdown coverage (Table 3). The third is a favorable model–task pair such as Dream on Countdown, which gains 14 pp over greedy decoding after three repair rounds (Table 8). In practice this means exploring with independent candidates and using repair to refine a selected one. The revised Discussion now describes these uses.

Multiple-choice format. On ARC-Challenge, OpenBookQA and CommonsenseQA, repair roughly triples the rate of unparseable option labels, while accuracy among parseable answers barely changes (Table 28). The loss there is mainly format corruption, so practical use requires protecting the answer-label template from remasking; we recommend this but have not evaluated it. Format corruption does not explain the coverage gap on the verifiable tasks, since two thirds or more of the gap remains after semantic canonicalization (Table 20).


Comment/Question:

  1. Existing dLLM search algorithms (Prism, S³, RFG) that combine local edits and trajectory-level exploration are only briefly mentioned in discussion. ...

Answer: We added a direct comparison with Prism, using the official code without modification in our matched setting (Appendix G.5). S³ and RFG have no released code, so we could not evaluate them. Prism keeps four of its sixteen candidates through self-verification, so we compare its final pool with four independent samples drawn from the same sixteen.

TaskMethodSelected accuracy, %Pool coverage, %ρ̂
CountdownPrism16.555.00.04
CountdownIndependent@422.856.00.07
MATH500Prism34.053.00.34
MATH500Independent@433.550.30.36
HumanEvalPrism14.626.80.09
HumanEvalIndependent@428.941.40.29

Prism's pool is as diverse as independent sampling, unlike repair chains. Its coverage shows no detectable difference from independent sampling on Countdown and MATH500 and is lower on HumanEval. Under Prism's own sampling configuration, its coverage is lower than independent sampling on both Countdown and HumanEval. This supports the scope of our conclusion: combining local edits with independent exploration avoids the concentration we report, although search does not automatically beat plain independent sampling at equal pool size here.


AuthorsOpenReview ↗

Response to Reviewer ALT2 (2/2)

Comment/Question:

  1. Dream-7B exhibits task-dependent repair performance: large gains on Countdown but degradation on other tasks. ...

Answer: The original Dream table mixed two comparisons, which we now separate. Table 8 compares repair with greedy decoding, and the matched-kernel control in Table 9 compares sampled pools.

TaskW→CC→WNet change vs greedyRepair − independent pass@16Per-state accuracy, repair vs independent
Countdown21.1%7.0%+14.1 pp−49.7 pp26.7 vs 26.1%
MATH5003.8%3.6%+0.2 pp−29.3 pp33.3 vs 33.1%
HumanEval6.7%19.5%−12.8 pp−14.2 pp45.2 vs 35.9%

Task dependence appears only in the gain over greedy decoding. Against sampled pools, repair concentrates the candidates and loses coverage on all three tasks. The HumanEval "degradation" is specific to the greedy comparison: with sampled initial answers, Dream's repair states on HumanEval are more accurate than independent samples. The revised Section 4 now says this explicitly. Why the balance after a greedy start differs by task is not isolated, and we report it as an open question.


Comment/Question:

  1. For partial-remask repair, attention-blocking yields only modest improvements in answer diversity. ...

Answer: Our controls rule out several candidate drivers and point to the retained reasoning beyond the answer span as the main remaining one.

  • Not the model's marginal answer distribution. States drawn from different chains are about as diverse as independent samples, while states within one chain are not (GR-2).
  • Not mask selection. Four different selectors differ by at most 5 pp and all stay highly concentrated (GR-3a).
  • Not the regeneration schedule. Blockwise and joint repair differ by 0.5 pp at α=0.2 (GR-1).
  • Not the remask ratio or temperature alone. Across the sweeps in Tables 16–18, coverage stays far below independent sampling until α=1.

What remains is the retained context itself, and the attention-blocking result suggests that it acts in a distributed way: blocking an equal number of random retained positions lowers agreement more than blocking the answer span (Table 21). Our interpretation is that the answer is encoded redundantly in the retained reasoning, through intermediate results, the chosen approach and the format, so the model can re-derive the same answer even without access to the answer tokens. A controlled perturbation of the reasoning content would test this directly, and we leave it to future work.


Comment/Question:

  1. Why does joint-update full-remask restart still produce low answer diversity even with no prior generation retained, ...

Answer: With nothing retained, the two restarts differ only in the order in which positions are committed, and that alone separates them by 10–33 pp in pass@16 (Appendix C.6). The blockwise restart shows no detectable difference from independent sampling (GR-1).

We attribute the difference to when each schedule fixes the answer. Joint decoding commits, at each step, the most confident positions anywhere in the response. For LLaDA-Instruct, Nie et al. report that such confidence-ordered decoding over the full response tends to commit high-confidence end-of-sequence padding early, which is why they recommend semi-autoregressive decoding. Once the length, format and answer slot are fixed this early, the remaining steps fill in content consistent with them, leaving sampling noise little room to produce a different answer. Blockwise decoding commits from left to right, so the reasoning is generated before the answer and early sampling differences can lead to different answers. A per-step trace of when answer positions are committed would confirm this directly; Appendix C.6 of the revision discusses this explanation.


Comment/Question:

  1. Dream-7B shows high answer-token exposure but only sees meaningful repair gains on Countdown. ...

Answer: After the alignment correction in GR-4, Dream's exposure is still high on both Countdown and MATH500, so exposure cannot explain why only Countdown gains over greedy decoding.

Exposure determines whether an answer can change; the gain depends on whether the changes are corrections. On Countdown, wrong-to-correct transitions outnumber correct-to-wrong ones three to one, while on MATH500 the two balance (Table 8). The gain over greedy decoding therefore reflects a task-dependent balance of corrections and regressions, not the level of exposure. We have not isolated what drives this balance, and the revision reports it as an open question.


Dear Reviewer ALT2,

Thank you again for your thoughtful review. We have completed our rebuttal and updated the manuscript, adding the Prism comparison, a clearer account of the Dream model–task interaction and a more explicit statement of scope (General Response and Appendix G).

We would be grateful if you could look at our responses and the revised paper when convenient, and let us know whether anything remains unclear. We are happy to answer further questions before the discussion period ends.

Best,

The Authors of Submission 842

Reviewer_6CCcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors examine test-time compute strategies such as BoN with independent resampling and repairs, which is a popular approach. They attempt to better understand the source of the performance gains. The authors show that independent resampling with a block kernel preserves answer diversity, while the popular confidence-based repair approach with joint resampling largely re-rationalizes through reasoning and adds little answer diversity. They apply several repair interventions (like remasking the answer token specifically) to close the gap, but show that answer diversity still suffers, and even the best possible selector (oracle at pass@16) lacks diverse candidates to choose from.

Critical strengths

Strengths:

  1. Shows that confidence-based repair with joint unmasking suffers from answer diversity, which heavily correlates with performance. They demonstrate this approach and it is largely replicable across several diffusion models.
  2. They have best-practice recommendations for future test-time scaling approaches that modify selectors and approaches that modify/repair strategies. This includes metrics like answer coverage and answer agreement.
Critical weaknesses

Weaknesses

  1. They don’t provide any strong new methodology or levers that steer test-time compute budget to meaningful improvement.
  2. Nie et al. recommend semi-AR (blockwise decoding) specifically for LLaDA-Instruct. We also know from Kim et al. (https://icml.cc/virtual/2025/oral/47251 ICML Oral) that the decoding order is paramount. The current paper states that diversity-preserving independent sampling outperforms local repair strategies. However, independent sampling always uses a blockwise kernel, while local repair is not done blockwise. In Appendix C, they evaluate blockwise at \alpha = 1, but there it’s an independent restart, with a blockwise kernel variant outperforming a restart with a joint update. There is no blockwise partial repair for alpha lower than 1. The statement would be stronger if they also observed the effect on blockwise local repairs: blockwise remasking based on confidence and repair.
Questions for authors

n/a

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6CCc

We thank the reviewer for the careful review. The shared evidence is in the General Response, GR-1 to GR-4.


Comment/Question:

They don't provide any strong new methodology or levers that steer test-time compute budget to meaningful improvement.

Answer: The paper is diagnostic, and the lever it identifies is how a fixed candidate budget is allocated. Spending candidates on successive states of one retained chain sharply limits coverage, even with a matched kernel and budget (GR-1), and even against independent sampling with a smaller budget, as shown in our FLOPs reply to mSra. The same eight chains give 75% pass@8 on Countdown when their final states are pooled across chains, but 25% when the states of one chain are pooled (GR-2). The allocation alone moves coverage by about 50 pp.

The constructive recommendation is resample-then-repair: draw candidates independently, then refine them. On Countdown this keeps the diversity of independent sampling and adds a small gain, from 72.3% to 75.0% pass@8 after three repair rounds (Table 3). We present it as a safe default rather than a new method. The paper also contributes diagnostics that any new repair method should report, namely coverage, agreement, unique answers and correctness transitions, and GR-3 shows that they separate methods that preserve diversity from methods that do not.


Comment/Question:

Nie et al. recommend semi-AR (blockwise decoding) specifically for LLaDA-Instruct, and Kim et al. show that decoding order is paramount. ...

Answer: We agree that decoding order matters. Without any retained context, a blockwise restart reaches 80.0% pass@16 on Countdown against 46.8% for a joint restart (Appendix C.6). This is exactly why the comparison needed the control the reviewer asks for.

We ran it in GR-1. Positions are selected by confidence and regenerated with the same semi-autoregressive kernel and denoising steps as independent sampling. At α=0.2 and 0.5, blockwise repair stays far below independent sampling on all three tasks, with every interval excluding zero, and Dream-7B behaves the same way; at α=1 the gap disappears. Once most of the response is retained, the decoding order of the regenerated part has little effect, and the deficit comes from repairing a retained chain.


Dear Reviewer 6CCc,

Thank you again for your review. We have completed our rebuttal and updated the manuscript. In particular, we ran the blockwise partial-repair control you asked for (General Response GR-1).

We would be grateful if you could look at our response and the revised paper when convenient, and let us know whether it resolves your concern. We are happy to answer further questions before the discussion period ends.

Best,

The Authors of Submission 842

AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer 6CCc,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a very busy period for you. We wanted to briefly let you know that we have posted a point-by-point response to your concerns and uploaded a revised manuscript with the changes highlighted. In particular, we ran the blockwise partial-repair control you asked for (General Response, GR-1): positions are selected by confidence and regenerated with the same semi-autoregressive kernel and denoising steps as independent sampling, at α<1.

Whenever you have a chance, we would be grateful for your feedback on whether the response addresses your concerns. We would be happy to clarify anything that remains open before the discussion period ends.

Thank you!

Best,

The Authors of Submission 842

AuthorsOpenReview ↗

Follow-Up to Your Review (Second Reminder)

Dear Reviewer 6CCc,

Thank you again for the time and care you put into reviewing our work!

We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.

If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.

Thanks again for your support!

Sincerely yours,

The Authors of Submission 842

Reviewer_mSraOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of sequential local repair in dLLMs compared with independent sampling when allocating additional test-time compute. The authors use several metrics to analyze answer-level dependence and find that although repair and independent sampling start from candidates of similar quality, successive repair rounds largely revisit the same answers, resulting in much lower answer coverage. The findings are validated across four dLLM checkpoints and three main tasks. The paper also examines different remasking strategies, attention-blocking interventions, and full-remask controls to better understand the factors contributing to limited answer coverage. Based on these results, the paper recommends spending compute on independent candidates first and using local repair mainly for refinement.

Critical strengths
  1. The paper provides a useful perspective on understanding whether repair and resampling actually explores new answer hypotheses.

  2. The paper uses multiple metrics to analyze answer quality, diversity, and coverage. The findings are further supported by extensive experiments and controlled comparisons across different settings.

Critical weaknesses
  1. Insufficient control of the generation procedure. The main finding that independent sampling achieve better performance needs to be further validated.
  • Independent sampling uses blockwise decoding, while sequential repair uses joint-update regeneration. Appendix C.6 shows that the generation strategy alone can significantly affect answer coverage. Joint updates are far more concentrated than blockwise even both with no retained state. However, the paper does not compare these two strategies under partial remasking (e.g., α\alpha=0.2). It's unclear how much of the coverage gap is due to retained context rather than the generation strategy.
  • The paper uses forward-pass counts to compare inference budgets. Intuitively, repair may have fewer FLOPs considering that only 20% tokens would be regenerated but more latency due to serial generation. Reporting approximate FLOPs or wall-clock latency would provide a more practical efficiency comparison, especially given the different decoding schedules or possibly used KV caching or approximate caching.
  1. The paper draws broad conclusions about local dLLM repair, but most experiments rely on the same sequential repair framework with different token-selection heuristics. Existing remasking-based refinement methods cited in the paper are not evaluated using the proposed diagnostics. Moreover, Table 11 shows relatively small differences among remasking strategies (37.0–42.0% pass@16) only on MATH500 with LLaDA-8B, without confidence intervals to establish their significance. The paper also lacks an oracle masking strategy experiment to assess whether better confidence estimation and accurate identification of incorrect reasoning steps could improve repair.

  2. The figure 2 largely repeats the results in Table 2. A more focused visualization could improve clarity. Some curves (CD, repair) in figure 3 (left) are difficult to distinguish due to overlap. Using different markers or line styles would improve readability. The results in figure 3 is based only on LLaDA-8B and one seed. Averaging the curves across multiple seeds and showing the curves of different models would also make the results more robust.

Questions for authors
  1. Have the authors considered partial repair (e.g., α\alpha=0.2) using blockwise regeneration instead of joint updates, while keeping the masking positions and denoising budget comparable?

  2. Do blockwise decoding and joint-update regeneration process the same sequence length per forward pass? Is KV caching or any other caching mechanism used?

  3. How exactly is the token confidence (ci=pθ(x^i∣x^)c_i = p_\theta(\hat{x}_i \mid \hat{x})) computed? Is it evaluated directly on the complete generated sequence, or is the target position masked before computing its probability? Since masked diffusion models are trained to predict masked tokens, how reliable is this confidence for selecting repair positions? Table 9 reports answer-span confidences of 0.014 and 0.008 for Dream, versus 0.599–0.986 for the LLaDA models. Could the authors verify that Dream's shifted-logit alignment is correctly handled when computing confidence? If not, this could affect Dream's repair results and the cross-model exposure analysis in Tables 9–10.

  4. Could the authors evaluate repair with oracle or ground-truth error localization on tasks where intermediate reasoning steps can be verified? This would help determine whether the limited improvement comes from inaccurate token selection or from the regeneration process itself.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mSra (1/2)

We thank the reviewer for the careful review. The shared evidence is in the General Response, GR-1 to GR-4.


Comment/Question:

Insufficient control of the generation procedure. Independent sampling uses blockwise decoding, while sequential repair uses joint-update regeneration. ...

Answer: We ran this control in GR-1. With repair using the independent sampler's blockwise kernel and a matched denoising budget, partial repair at α=0.2 stays 17–54 pp below independent sampling on all three tasks, with every interval excluding zero, and Dream-7B behaves the same way. The gap disappears only at α=1, when nothing is retained. Holding the regeneration fixed and switching between joint and blockwise decoding changes Countdown coverage by 0.5 pp. The generation schedule therefore matters a great deal for a full restart, but very little once most of the response is retained.


Comment/Question:

The paper uses forward-pass counts to compare inference budgets. Intuitively, repair may have fewer FLOPs ...

Answer: In our implementation, remasking 20% of positions does not reduce FLOPs. Every forward pass in both methods processes the full prompt and response, masked positions included, and neither method uses caching. Forward counts are therefore proportional to FLOPs, which we confirmed with a profiler. These results are now in the main text and in Appendix G.3.

FLOPs. Table 1 is approximately FLOPs-matched: repair at r=38 costs 16.4 times greedy decoding, against 16 times for BoN-16. In Table 2, repair actually uses less than half the FLOPs of independent sampling. To make sure this smaller budget does not explain the gap, we compared repair with seven independent samples, which cost slightly less than repair. Independent sampling still has higher coverage in all nine task–seed conditions, and every interval excludes zero.

Latency. The reviewer's intuition holds at matched FLOPs. Median single-request latency in seconds on one H200:

TaskIndependent-16, batchedRepair, Table 2 settingFLOPs-matched repair
Countdown10.207.4917.02
MATH50027.7516.5337.12
HumanEval27.8216.5037.16

At the budget of Table 2 repair is faster. At matched FLOPs it is 1.3–1.7 times slower, because independent samples can be batched while repair rounds run one after another. Throughput with several concurrent requests shows the same ordering.


Comment/Question:

The paper draws broad conclusions about local dLLM repair, but most experiments rely on the same sequential repair framework ...

Answer: We address the three parts in turn.

Other methods. GR-3 applies our diagnostics to other position selectors on the same chain, including CORE- and WINO-inspired ones, and to five released remasking decoders. Every selector chain concentrates, while the released decoders, which produce independent trajectories, stay close to independent sampling. We agree that our original wording was too broad. The conclusion concerns post-completion repair chains used as candidate pools, not remasking during decoding, and the revised abstract, introduction and conclusion now say so.

Table 11 intervals. The original table mixed a three-seed mean with single-seed values. In the revision, where it is now Table 12, every entry is a three-seed mean with a t-interval:

Heuristicpass@16, % [95% CI]Δ vs low-confidence, pp [95% CI]
Low-confidence37.00 [28.04, 45.96]—
Random36.33 [30.08, 42.58]−0.67 [−3.54, +2.20]
Answer-span38.17 [29.71, 46.62]+1.17 [+0.45, +1.88]
PRM first-bad-step36.67 [27.85, 45.48]−0.33 [−1.05, +0.38]
PRM + answer-span38.17 [29.90, 46.44]+1.17 [−0.73, +3.06]

The heuristics differ from one another by at most 1.8 pp. Only answer-span's small gain excludes zero, and that heuristic also regenerates more tokens. All five remain about 26–28 pp below independent sampling, with every interval excluding zero.

Oracle masking. We ran a verified error-localization experiment, described in our answer to Question 4.


Comment/Question:

The figure 2 largely repeats the results in Table 2. A more focused visualization could improve clarity. ...

Answer: Thank you. Figure 3 has been redrawn so that the curves no longer overlap. Independent-sampling curves are distinguished by color and marker, repair is drawn as one band, and all curves are averaged over decoding seeds; the other three models appear in appendix Figures 8 and 9. Figure 2, which repeated Table 2, has been replaced by a figure on pool construction that shows the GR-2 result: within-chain pools lose coverage on all three tasks, while across-chain pools show no detectable difference from independent sampling.

AuthorsOpenReview ↗

Response to Reviewer mSra (2/2)

Comment/Question:

  1. Have the authors considered partial repair (e.g., α=0.2) using blockwise regeneration instead of joint updates, ...

Answer: Yes; see GR-1. With the same B32 kernel and per-round denoising steps, α=0.2 repair still has far lower pass@16 than independent sampling on all three tasks, for example 29% against 83% on Countdown. A second control also matches the masked positions and the regeneration forwards; there, switching from joint to blockwise regeneration changes Countdown coverage by 0.5 pp.


Comment/Question:

  1. Do blockwise decoding and joint-update regeneration process the same sequence length per forward pass? ...

Answer: Yes. Both process the full prompt and response at every forward. Blockwise decoding only restricts which positions may be committed at each step; it does not shorten the input. Neither uses KV caching or any approximate cache.


Comment/Question:

  1. How exactly is the token confidence computed? Is it evaluated directly on the complete generated sequence, or is the target position masked before computing its probability? ...

Answer: Computation. We run one forward over the complete sequence with position i visible and take c_i = pθ(x̂_i | x̂). We then remask the lowest-scoring response positions. The score is not a probability computed after masking position i.

Reliability. The reviewer's concern is well founded. Masked diffusion models are trained to predict masked positions, so the probability of a visible token is not a calibrated estimate of whether it is wrong. The evidence suggests, however, that score quality is not what limits coverage. Random selection shows no detectable coverage difference from low-confidence selection, and CORE-inspired selection, which scores positions by masking perturbations, is no better (GR-3a). Even a verified error location helps only modestly, as our answer to Question 4 shows. A better confidence estimate could change which errors get fixed, but by itself it does not make successive states of a chain diverse.

Dream alignment. The repair operator applied Dream's logit shift, but the separate audit did not. GR-4 gives the corrected values. Tables 10–11 are corrected, and the Dream repair results are unchanged. Thank you for catching this.


Comment/Question:

  1. Could the authors evaluate repair with oracle or ground-truth error localization on tasks where intermediate reasoning steps can be verified? ...

Answer: Yes; see Appendix G.6. On Countdown, 42 of 200 initial responses are wrong and contain an arithmetic step that our verifier proves false. We remask everything from that step onward and regenerate it without supplying a correction. This is a verified error location, though not necessarily the earliest error. We compare it with confidence repair on the same responses, averaging over three repair seeds, and report the fraction of problems solved by at least one state:

RegenerationConfidence repairVerified-step repairΔ, pp [95% CI]
Joint2.4%12.7%+10.3 [0.0, +21.4]
Blockwise2.4%7.9%+5.6 [−0.8, +12.7]

Accurate localization helps, but most of these problems, 87–92%, remain unsolved after 15 rounds. The limit therefore lies largely in what regeneration produces from the retained context, not only in where it is applied.


Dear Reviewer mSra,

Thank you again for your detailed review. We have completed our rebuttal and updated the manuscript, adding the blockwise partial-repair control, FLOPs and latency comparisons, the oracle error-localization experiment and the corrected Dream audit.

We would be grateful if you could look at our responses and the revised paper when convenient, and let us know whether any concerns remain. We are happy to answer further questions before the discussion period ends.

Best regards,

The Authors

Thanks for your detailed responses. Most of my concerns have been addressed and I increased the score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers. Several questions concern the same four pieces of new evidence, which we report once here and refer to as GR-1 to GR-4 in the individual replies. All of it is in the revised paper, mainly in the new Appendix G. Throughout, α denotes the remasked fraction, so 1−α of the response is retained.

GR-1. Blockwise partial repair with a matched kernel and budget. Addresses mSra W1/Q1, 6CCc W2 and LP6F W1/Q1. Details are in Appendix G.1.

Repair now regenerates with the independent sampler's own 32-token blockwise kernel and runs the same number of denoising steps per round as one independent generation. Positions are still chosen by token confidence, and retained tokens stay fixed. The only extra cost is one confidence-scoring forward per round, so the two arms differ by less than 1.5% in forward passes. We use LLaDA-8B-Instruct with N=16, temperature 0.4 and three decoding seeds. The table gives repair minus independent pass@16 with 95% paired problem-bootstrap intervals.

TaskIndependent pass@16Δ at α=0.2 [95% CI]Δ at α=0.5 [95% CI]Δ at α=1.0 [95% CI]
Countdown82.67%−53.50 [−58.00, −48.83]−51.67 [−56.17, −47.17]−0.33 [−2.83, +2.17]
MATH50064.00%−26.00 [−30.67, −21.33]−25.67 [−30.50, −21.00]+0.17 [−2.67, +3.00]
HumanEval53.05%−17.28 [−22.56, −12.20]−14.43 [−19.72, −9.15]−0.61 [−3.05, +1.83]

At α=0.2 and 0.5, partial repair still falls far below independent sampling and every interval excludes zero; at α=1 the gap disappears. The kernel mismatch in the original experiments therefore does not explain the partial-repair deficit. Dream-7B shows the same pattern under this control (Table 9).

A second Countdown control holds the regeneration itself fixed. From identical initial answers, joint and blockwise repair remask 25 positions per round and use the same number of forwards. Their pass@16 differs by only 0.5 pp. Once most of the response is retained, the regeneration schedule hardly matters.

GR-2. Within-chain versus across-chain pools. Addresses LP6F Q1 and ALT2 Q1. Details are in Appendix G.2.

For every problem we ran eight independently initialized repair chains of eight states each. A within-chain pool is the eight states of one chain; an across-chain pool takes the final state of each of the eight chains. Every pool has eight candidates.

TaskPoolPass@8Unique/8ρ̂
Countdownwithin-chain25.44%1.310.891
Countdownacross-chain75.00%6.510.072
Countdownindependent71.50%6.530.079
MATH500within-chain35.38%1.250.933
MATH500across-chain60.00%4.090.378
MATH500independent60.50%4.100.373
HumanEvalwithin-chain33.38%2.080.735
HumanEvalacross-chain51.83%4.870.328
HumanEvalindependent48.78%5.100.295

Across-chain pools show no detectable coverage difference from independent sampling on any task, whereas within-chain pools lose 18–50 pp of pass@8, and all paired bootstrap intervals for that loss exclude zero. Repaired states are therefore not concentrated as a population; the concentration lies between successive states of the same chain. The comparison matches candidate count rather than compute, but the within-chain deficit remains when one chain is compared with independent samples of the same generation cost.

AuthorsOpenReview ↗

General Response

GR-3. Other selectors and released remasking decoders. Addresses mSra W3, LP6F W2/Q2 and ALT2 Q1. Details are in Appendix G.4.

Both evaluations use LLaDA-8B-Instruct in the matched setting with a single seed. Each cell gives pass@16 and pairwise answer agreement ρ̂.

(a) Other position selectors on the same repair chain. Starting from the same initial answers, we replaced low-confidence selection with random, CORE-inspired and WINO-inspired selection. The latter two adapt the selection ideas to our operator and are not full reproductions of those methods.

PoolCountdownMATH500HumanEval
Independent reference81.5 / 0.0763.5 / 0.3651.8 / 0.29
Low-confidence chain29.5 / 0.9041.0 / 0.9434.2 / 0.80
Random chain28.0 / 0.8438.5 / 0.9434.2 / 0.66
CORE-inspired chain27.5 / 0.9340.0 / 0.9634.2 / 0.79
WINO-inspired chain32.0 / 0.8143.5 / 0.9137.2 / 0.72

Every selector chain stays concentrated, yet each has higher per-state accuracy than independent sampling, so its lower coverage comes from concentration rather than weaker candidates. The best selector, WINO-inspired, improves on low-confidence by only 2.5–3 pp and still trails independent sampling by 15–50 pp.

(b) Released remasking decoders. We ran CORE, WINO, Saber, RCR and COVER from their released code, each with the author control that disables remasking where one exists. These decoders remask inside each generation but still produce independent trajectories.

MethodCountdownMATH500HumanEval
Independent reference81.5 / 0.0763.5 / 0.3651.8 / 0.29
CORE81.5 / 0.0768.5 / 0.3639.0 / 0.29
CORE, unmask control82.5 / 0.0764.0 / 0.3753.7 / 0.29
WINO80.0 / 0.1160.5 / 0.4545.1 / 0.48
WINO, default control84.5 / 0.0767.5 / 0.3953.1 / 0.30
Saber81.0 / 0.1062.0 / 0.4453.1 / 0.38
Saber, default control83.5 / 0.0764.5 / 0.3753.1 / 0.29
RCR85.5 / 0.0862.0 / 0.4556.7 / 0.30
RCR, unmask control84.5 / 0.0962.0 / 0.4050.0 / 0.38
COVER78.5 / 0.0865.0 / 0.3754.3 / 0.29

All of these pools stay close to independent sampling in agreement. Their costs differ by method, so this is a descriptive single-seed comparison.

Together, (a) and (b) show where the loss arises. Remasking within an independent trajectory does not concentrate the pool, whereas pooling successive states of one retained chain does, whichever selector is used. Our claim concerns the latter.

GR-4. Dream logit alignment. Addresses mSra Q3, LP6F Q3 and ALT2 Q3. Details are in Tables 10–11.

Dream predicts token i from position i−1. Our repair operator applied this shift, so all Dream repair results are unaffected. The separate confidence and exposure audit did not, and the reviewers were right that its confidence values were anomalous. We reran the audit on the same generated sequences. Without the shift we reproduce the original values exactly, which confirms the cause; with the shift we obtain:

TaskAnswer confidence, old → correctedExposure, old → corrected
Countdown0.014 → 0.9504.25 → 2.65
MATH5000.008 → 0.9573.99 → 1.31

The exposure–agreement correlation across the eight model–task cells stays at −0.881 after the correction, and we report it as a descriptive association. Tables 10–11 and the related text are corrected in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.