- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper distinguishes between decision tokens and rationales in diffusion language models, it finds that if previously generated rationales are retained, the model reconstructs the prior decision, whereas otherwise, control shifts to the current evidence. The authors validated this by comparing three types of interventions (no reset, answer reset, and path reset) across three dLLMs (Dream, LLaDA, and iLLaDA).
Critical strengths
- The paper systematically distinguishes, for the first time, between "open-ended answers" and "revisable trajectories."
- Replication across three different dLLM checkpoints enhances generalizability.
Critical weaknesses
- The primary experiments rely on Deep Arithmetic Chains and HotpotQA multi-hop question answering, that are tasks characterized by short answers, multiple-choice formats, or single-token decisions. What about open-ended generation tasks (e.g., long-form reasoning, code generation, multi-turn dialogue)?
- The paper repeatedly emphasizes that "path reset" represents a causal endpoint rather than a deployment strategy. However, the best partial reset during testing achieved only 70%, far below the 99% seen with "path reset", indicating a significant discrepancy.
- The three models (Dream, LLaDA, iLLaDA) all belong to the class of masked diffusion language models, what about other types of diffusion language models?
- While the "top-50" and "mass-80" direct attribution methods outperformed the random control, the "top-25" and "decision-mask" methods did not. The rank correlation between the two attribution scores is merely 0.115, suggesting they select different tokens—a point that lacks comprehensive analysis or explanation.
Questions for authors
see weakness
3 comments
Thank you for the detailed response. The new free-form evaluation partly addresses my W1, and I agree that the attribution–repair gap is better framed as a negative result. However, after reading the full discussion, including Reviewer UzAb’s review and your reply to it, I have come to share the concern about what is specific to diffusion. Now you say that AR models may show the same anchoring, and that path reset is operationally close to regenerating on the same canvas. Without a matched AR comparison or a restart-with-evidence baseline, it is unclear how much of the finding is specific to diffusion models rather than a general property of retained rationale. My own concerns are also only partly resolved. The free-form gain depends strongly on the checkpoint (Dream: +1 [−3, 5]) and is still extractive short-answer HotpotQA (W1), and there is no evidence beyond masked dLLMs (W3).
Taken together, I am lowering my score from 3 to 2. I mention them as what would most change my assessment, not as a condition for this round.
Prompt-restart results and relation to AR research
We have completed the prompt-restart comparison you highlighted.
1. Why study dLLMs when AR research reports related phenomena?
AR models generate tokens sequentially; masked dLLMs predict multiple positions on a shared canvas and allow generated positions to be remasked. Whether that editing operation enables revision with new evidence requires direct testing.
Related AR studies examine CoT dependence [1] and self-correction without external feedback [2]. Our answer/path interventions, rationale transplantation and selective repair test which retained state obstructs revision, how its content affects decisions, and whether influential tokens suffice for repair. Related AR findings do not replace these tests. This is the contribution of our work.
2. Prompt-restart comparison
On the same 100 questions and three checkpoints, we placed evidence in the user prompt and generated from scratch, retaining the 64/16-token rationale/answer fields, 32-step greedy decoding and the same parser. All 900 outputs across correct, corrupt and absent evidence are included; every cell has 100% first-turn format pass.
Correct-evidence answer EM:
| Model | Answer reset | Path reset | Prompt restart |
|---|---|---|---|
| Dream | 65% | 66% | 64% |
| LLaDA | 8% | 54% | 57% |
| iLLaDA | 4% | 67% | 70% |
Restart-minus-path differences are −2 [−8, 3], +3 [−3, 10] and +3 [−3, 10] percentage points (paired 95% intervals).
3. What does the comparison show?
LLaDA and iLLaDA reach 57% and 70% with prompt restart, versus 8% and 4% with answer reset. Reopening the rationale restores 54% and 67%, close to the restart estimates. Together with the matched answer/path intervention, this shows that reopening the answer can leave the decision tied to earlier rationale, even when the model can answer correctly from the current evidence. The contribution is identifying this revision barrier and testing its transfer and repair properties.
4. Checkpoint and task coverage
Dream shows little separation in this interface. The barrier therefore occurs in some tested checkpoints, rather than universally. The experiments cover extractive short-answer QA and three masked dLLM checkpoints. Long-form generation and other diffusion families remain untested; these boundaries limit generalization while leaving the observed revision barrier in LLaDA and iLLaDA intact.
With this additional comparison and clarification of the contribution, we hope you will reconsider the decision to lower your rating.
[1] Lanham et al. (2023), Measuring Faithfulness in Chain-of-Thought Reasoning. https://arxiv.org/abs/2307.13702
[2] Huang et al. (ICLR 2024), Large Language Models Cannot Self-Correct Reasoning Yet. https://arxiv.org/abs/2310.01798
Response to concerns on scope and attribution
Thank you for the concrete questions. We respond to each weakness below.
W1: The evaluation uses short or one-token decisions; what about open-ended generation?
We froze a new 100-question source-disjoint HotpotQA evaluation before inference. It provides no answer choices or proposed answer; gold and controlled false targets contain 2–8 tokens. All 3,600 expected outputs are retained, and the minimum per-cell format-pass rate is 98%.
This extends the result to reliably parsed free-form short answers and reveals checkpoint dependence: the additional path-reset effect is large in LLaDA/iLLaDA and negligible in Dream. It does not establish long-form reasoning, code generation, or multi-turn dialogue, and we revised the scope accordingly.
W2: The best partial reset reaches 70%, far below the 99% path-reset result.
We agree that the gap is large; it is the mechanism result we call the attribution–repair gap. Direct top-50 and mass-80 exceed matched random selection by +16.8 [9.0, 25.2] and +18.2 [11.0, 26.0] points, so the score improves token prioritization. Yet the best partial reset reaches 70% versus 99% after path reset. We now present this as a negative result on compact repair sufficiency, not as a near-deployable method.
W3: What about other diffusion families?
The evidence covers Dream, LLaDA, and iLLaDA, all fully bidirectional masked dLLM checkpoints. The 64-step and reset-time robustness studies vary decoding within this family; they do not replace cross-family validation. We removed broader architectural implications and do not claim coverage of continuous diffusion or all diffusion language models.
W4: Why do top-50/mass-80 help while top-25/decision-masking do not, and why is rank correlation only 0.115?
The scores measure opposite directions. Decision masking asks how the answer supports an observed rationale token; direct leave-one-out asks how removing a rationale token changes the answer distribution. Their mean Spearman correlation is therefore low (0.115; top-quarter overlap 30%). Direct top-50 and mass-80 establish a matched-random advantage, whereas top-25 and the decision-masking ranking do not. Figure 4 now presents the repair frontier and matched-random effects together; Appendix D.6 retains per-model results and all policies. Redundancy and nonlinear interactions remain possible explanations for the gap, not established conclusions.
The free-form protocol and full cells are in Section 3.5, Table 3, Appendix C.5, and Appendix D.4; attribution revisions are in Section 3.7, Figure 4, and Appendix D.6.