- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work argues for the importance of distinguishing the convergence of a discrete diffusion model and its correctness. They coin several concepts such as "stable but wrong" (SBW), and "lock-ins" (to quantify stability, even when the produced answer is wrong). They then propose to evaluate the model's hidden representations with an additional classifier to intervene with repairs, and which measurably increases the performance of the base models at a high computational cost.
Critical strengths
- This paper presents a relatively simple algorithm/method to measurably improve performance of baseline models.
- The authors identify an issue that seems important to the diffusion literature (here called “lock-in”).
Critical weaknesses
Weaknesses:
- There is a lack of clarity in the paper about the nature of the diffusion language model. I originally thought the model was a continuous diffusion language model, but I guess the citations and the models evaluated in Table 9 made it clearer that it was discrete diffusion. I would make this more explicit.
- The specification of the extraction function, , is not clear where it is introduced: “a deterministic extraction function [yielding] the current normalised answer”. In particular, what is “normalised”, precisely? (I can imagine a few things, but this should be exact immediately.)
- What is a “Surface-18” vector? I assume that the authors wanted to introduce the nomenclature, but, in which case, it should be written the other way around, unless I am unaware of this specific terminology.
- The terminology of 3.3 is very hard to read. A lot of vocabulary is implicitly introduced (i.e., without giving the actual meaning of words; e.g., “caliper”, “Surface-18”, “Matching”). Worse, the experimental benchmark should appear directly in 3.3, or else it is not understandable when first reading the paper. Most surprisingly, an explanation is finally provided at the last sentence of the subsection (“This tests whether hidden trajectories…”). I have to admit that I find this section to be a very hard read.
- What is “(selective) reasoning repair”? Used multiple times, never introduced. What is “prefix hidden trajectories”? In general, “The observed prefix hidden trajectory forms the planning state: a risk model either retains the baseline path or triggers targeted remasking to expand a local counterfactual repair space”, and other sentences are very confusing reads.
- “We define , where selects an intermediate baseline state and is the nominal remasking budget applied to an answer-related region before denoising resumes”: what is ? a scalar? a function? Likewise: what is ?
- Similarly, and there are more examples, but I will end with this last one, is used before it is introduced. I really do not think that giving an overview, that tries to be formal but isn’t, is such a good idea. I think a sort of “linear” build-up/narrative would have made things much clearer.
- An algorithm would have made the repair procedure much clearer.
- Overall, while the performance improvements in the experiments are real, I struggle to see the connection to the original motivation of the wrong lock-ins. If I understand the algorithm correctly, ranking/measuring the potentially wrong tokens with an additional head, and branching out into 8 tracks is evidently helpful; the model benefits from some additional parameters certainly too, and from some smarter sampling scheme, which relies on the rich hidden states.
- The computational cost incurred by the method is large.
Questions for authors
- Could you produce an algorithmic box for your final method to alleviate any ambiguity about what your method is doing exactly?
- How does your original understanding of the distinction between convergence and correctness relate to your proposed algorithm? I guess it is obvious in some sense, but I don't see the insight, and find the two parts quite disconnected. Adding a new model which evaluates different branches at inference sounds like a natural way to improve performance, but, again, the connection with the motivating bit is too brittle in my understanding.
I would be happy to revise my score slightly after the clarification of many of the given concepts, a large re-write (which might be out of scope for a rebuttal, but maybe feasible with some edits only), and especially the better establishment of a connection between the motivation and proposed solution -- which I find currently too disconnected.
Summary of Revisions
Dear Reviewers and Area Chair,
Thank you for your detailed feedback and the discussion. We have uploaded a revised manuscript incorporating the clarifications, additional experiments, and evaluation corrections described in our responses. For ease of reference, we summarize the main updates below.
1. Definitions and method protocol (Sections 3–4; Appendices A–D). We clarify answer extraction, normalization, correctness, and Surface-18 matching without hidden-state information. Algorithm 1 explicitly describes the complete-Base protocol, including midpoint-prefix risk estimation, eight fixed repair actions, candidate verification and selection, and acceptance or fallback to completed Base. The revised text also distinguishes the detector, verifier, and selector.
2. Final-output evaluation (Sections 5.4–5.6; Appendix F). We distinguish the matched benchmark’s checkpoint labels from completed-Base accuracy and re-evaluate all policies on their actual returned answers across 38,416 questions. We add seven-method comparisons on five new natural-distribution holdouts, each with 10,000 test questions and a separate 2,000-question validation set. The full results include ineligible questions, Rescue/Damage counts, routing rates, computational costs, and paired confidence intervals for the natural-distribution comparisons.
3. Computation and timing (Appendices A.4, D, and G.3). We explain the historical analytic 5× convention and distinguish accumulated forward-token work, profile-weighted estimates, and elapsed time. The new accounting includes individual repair actions, hidden-state extraction, required replay, candidate generation, auxiliary scoring, and routing. We also separate the nominal action bank from configuration-specific profiling and clarify that savings over always-repair do not imply a speedup over Base.
4. Prompt controls, Dream, and frozen transfer (Sections 5.7–5.9; Appendices I–K). We add answer-first/reasoning-first comparisons, late answer regeneration, and RF-LOCKR-pos, together with answer-visibility and eligibility statistics. Frozen GSM8K evaluation covers all 1,319 test questions for DG, LLaDA, and Dream without target training or calibration. Dream uses its own source-trained components, and its RF Base check is explicitly distinguished from a full RF planner evaluation.
5. Component and supervision controls (Sections 5.10–5.11; Appendices G.1 and L). We add independent trigger comparisons, clean selection comparisons with fixed repair IDs and candidates, matched-capacity and training controls, retrained shuffled-order controls, and error-enrichment diagnostics. Complete-pipeline supervision curves cover four training fractions and three seeds, with explicit supervision counts and preparation and retraining cost scopes.
6. Conclusions and limitations. We update the abstract, introduction, conclusion, limitations, captions, and cross-references to reflect the revised evidence. We retain uncertain and unfavorable comparisons and qualify claims concerning prompt dependence, shared data lineage, candidate coverage, temporal order, supervision, and computational overhead.
We appreciate the reviewers’ and area chair’s time and consideration.