- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper extends CoRe, to Local-Context-Dropout which is LCD. LDE has additional masking pass for candidate’s r=1 neighborhood. Authors experiment LCD if it helps and where it helps, but it has a minimal effect on the final performance gain.
Critical strengths
- Authors are clear and honest on the result, that there is not much of the gain in the methodology.
- A lot of ablation studies are done in the setting.
Critical weaknesses
- Figure 1 has unnecessarily many texts in the figure and some words going outside the box, which makes it harder to parse.
- LCD vs CoRe Humaneval experiment has 4 scores changed, which leads to 2.44pp gain. Authors explicitly say that gain is indistinguishable.
- The fact that is revealed, such as ‘relatively close neighborhood is important’ is trivial in language models.
- The preliminary result does not connect to the main claim of the paper.
- There is only CoRe and base experiment.
- Numbers (accuracy) of HE, MBPP, GSM8K is way behind the LLaDA paper, about 20pp.
- Table 24 has “pending: instrumented runs in progress”. The paper seems to be not finished, and not ready for the publishing. At least the paper should be containing all the content.
- The title “Testing Attribution Claims in Diffusion-LM Remasking” is too generalized compared to the experiments in this paper.
Questions for authors
- Is there any reason that we should use LCD? Most questions are already addressed in the weaknesses section.
4 comments
Response to Reviewer UFA9: part 2
(Part 2/2)
Accuracy on HumanEval, MBPP and GSM8K is about 20 pp below the LLaDA paper. We attribute most of this to CoRe's compute-limited setting rather than a pipeline fault. It decodes 512 tokens in 128 steps, four tokens per step, with greedy decoding. CoRe's own paper reports the same range (12.2% HumanEval, 15.6% MBPP and 51.4% GSM8K for vanilla decoding in its Table 1), and our vanilla row is close to it (12.2%, 16.4%, 51.8%). To check both the gap and our conclusion, we reran HumanEval at 512 steps, one token per step (new App. R, Table 24). Vanilla decoding rises to 0.3354 pass@1, close to the 0.354 that the LLaDA paper reports for this model. So on HumanEval the gap comes from the decoding budget, not the pipeline. We ran this check on HumanEval only. For MBPP and GSM8K the budget is the likely cause, not a tested one, and for MBPP the LLaDA paper also uses 4-shot prompts where CoRe's harness uses 3-shot. At 512 steps, LCD and CoRe reach the same outcome on every one of the 164 problems: LCD still acts and changes the text of 19 completions, but not one pass or fail. (CoRe itself does not beat vanilla at this budget: 1 problem fixed, 4 broken.) The setup paragraph and App. R now state this scope.
Table 24 says "pending". It (now Table 27) is filled with counts and denominators: CoRe's intended remasks displaced by LCD are 55 of 791 on HumanEval (6.95%), 222 of 2,829 on MBPP (7.85%) and 678 of 9,330 on GSM8K (7.27%). These come from runs that reproduce the reported scores exactly.
The title is too general. Changed to "Masking a Token's Neighbors in Diffusion-LM Remasking: Local Dependence Did Not Identify Useful Remasks". It now names the intervention studied and states the finding as an empirical result.
Q. Is there any reason to use LCD? Not as a decoder: it does not beat CoRe at equal compute and does not carry to held-out problems. Its value is as an instrument. With the release test, it lets one check whether a perturbation-based remasking rule is acting on errors or only on context sensitivity. We release it for that purpose.
Thank you for your rebuttal.
- Redrawn Figure 1 still has unnecessary comments, which summarizes figure but not straightforward to understand. "Median necessity share." "Changed, but rarely repaired."
- Still, Tables 2, 3, and 4 do not have an intuitive interpretation after changing. Even though Table 27 has a result, it also does not help in understanding the main claim. Knowing success/failure numbers does not help building intuition.
- In your rebuttal, "We no longer present LCD as an improvement" and "Its value is as an instrument." part -if I am understanding correctly, contribution seems to be low. In the release test, remasking an influential neighbor changed that token more often than remasking a matched distant token, but you found no reliable advantage in correcting errors. If the observation itself is now the main contribution, I am not convinced that it substantially changes our understanding of how to select tokens for remasking.
I will maintain my original score.
2nd response to reviewer UFA9
Thank you for the follow-up. We agree with all three points and revised the paper again. Changes from this round are in purple in the revised PDF.
1. Figure 1. We removed both notes. Panel (a) is now one real verify step, from MBPP/467 ("remove lowercase letters"). It shows what each method masks around the candidate upper in if char.isupper():, and the probability the model gives to lower: 0.24 under CoRe's mask, 0.63 under LCD's, and 0.24 again when only .is is put back. LCD therefore remasks upper. The decoder writes upper again, keeps .is, and the program passes. Panels (b) to (d) now carry axis titles in place of notes: pass@1 change against CoRe, share of the effect removed, and programs repaired.
2. Tables 2 to 4. Each is rebuilt around one plain question, and the text says how to read it.
- Table 2 counts solved problems. CoRe solves 23 of 164 HumanEval problems. With LCD's extra compute spent on more verify calls it solves 25, and LCD solves 27.
- Table 3 asks how much of LCD's effect disappears when one masked token is put back. For the adjacent neighbor the median is 0.45 to 0.79 across three tasks and two models. For a distant token of the same confidence it is 0.00 to 0.07. The test statistics moved to the appendix.
- Table 4 places the release test next to a new positive control (point 3a).
We agree that the slot-displacement counts (old Table 27) do not bear on the main claim. The main text no longer cites them; they stay in the appendix as bookkeeping.
3. Contribution. You are right that our reply leaned on "instrument", and that an observed dependence which does not repair programs is a weak result on its own. We added two analyses and one experiment. Together they turn the observation into a claim about how to choose tokens for remasking.
(a) A positive control for the release test (Sec. 4.3). On passing programs we replace the neighbor LCD relied on with the model's runner-up token, and keep it out of the verify candidates. In 35 events this breaks the program. Freeing the planted token then repairs 80% of them, against 11% for freeing the matched distant token. So the test does catch a wrong neighbor when one is there. The real neighbors LCD relied on are repaired in 2 of 621 failing cases, against 1 for the control. This bounds the share of those neighbors that are fixable errors at about 1.5%.
(b) Dependence against correctness. The neighbor dependence is as strong on problems that pass as on problems that fail. On LLaDA the AUROC is 0.46 to 0.60, and every 95% interval reaches 0.5. A signal that looks the same on correct and wrong programs cannot pick out the wrong ones.
(c) Which remasks help (new RQ4, Sec. 4.4). Take every problem whose outcome a rule changes. CoRe's own remasks, chosen because the re-prediction disagrees with the token in its visible context, are repairs in 68% of cases (376 against 179; 91% on held-out MBPP). The remasks LCD adds, chosen because hiding neighbors moved the prediction, are repairs in 51% (84 against 80; 30% held out), a coin flip. CoRe acts first and may take the easier errors, and the paper says so.
What this changes. CoRe frames its perturbation test around tokens "tied to hard consistency constraints", and LCD extends the same premise: if hiding context changes a prediction, the token deserves revision. Our results split that premise in two. Dependence means the token is bound to its context, as upper is bound to .is. Error means the token is wrong. In our experiments these came apart: the dependence is strong, local and equally present in correct code, while the useful remasks came from disagreement in full context. Sec. 4.4 turns this into three design rules. Score a candidate by evidence against the token in its full context. Accept a proposal made under a wider mask only if the full-context prediction supports it. Test any new selector by acting on its picks against matched controls, with a planted error as a positive control. These apply to anyone designing remasking rules, not only to LCD.
We hope the new figure, the rebuilt tables and the added evidence address your concerns. We would be grateful if you would reconsider your score in light of them.
Response to Reviewer UFA9: part 1
(Part 1/2)
Response to Reviewer UFA9
Thank you for the review. We agree with most of the weaknesses you list, and we changed the paper to address each one. The revised PDF marks new and changed text in blue. We also rewrote the main text in shorter, plainer sentences.
Figure 1 has too much text, and some overflows its boxes. Redrawn. The new Figure 1 has two schematic panels (what CoRe masks, what LCD masks) and three small result panels, one per research question. It has about a third of the previous text, larger type, and nothing crosses a frame.
The HumanEval gain is four problems and is indistinguishable at equal compute. We agree, and this is now the paper's first result rather than a caveat (Sec. 4.1). At equal compute CoRe gains 1.22 pp and LCD 2.44 pp; head to head they differ by four problems to two (p = 0.688), so LCD's lead is not statistically reliable. (We now say "not significant" rather than "indistinguishable", since a null test does not show equivalence.) The frozen method loses 0.86 pp on 464 held-out MBPP problems. We no longer present LCD as an improvement, and the conclusion says we do not recommend it as a decoder.
"A close neighborhood is important" is trivial in language models. That nearby context matters is indeed expected; Piskorz et al. (ICML 2026) document locality bias in diffusion LMs, and we cite them. Our findings go beyond it in three ways:
The third point is not trivial, because it cuts against the premise behind neighbor masking: that the neighbor a prediction depends on is a source of error worth removing. We state it as a result for this protocol, which frees one neighbor after LCD has revised the candidate, not as a general law.
The preliminary result does not connect to the main claim. We restructured the paper around one line of argument, shown in Figure 1 (c) to (e). RQ1: neighbor masking does not help once compute is matched. RQ2: what the extra pass measures is a joint, local dependence. RQ3: acting on that dependence repaired no more programs than acting on a matched control, which is consistent with neighbor masking not helping. Each section now ends by answering its question and leading to the next.
There is only CoRe and the base experiment. Table 1 compares seven methods on five benchmarks: vanilla decoding, CoRe, four other training-free overlays that we reimplemented on the same model (ReMDM, T2M, STaRR, Lookahead Unmasking), and LCD. The paper also runs a second backbone (Dream-7B), three equal-compute variants, a held-out set, count-matched masking controls, and, new in this revision, a second decoding budget. We build LCD on CoRe because CoRe is the only overlay with a verify pass whose mask can be widened. We have made the comparison set more visible in the setup paragraph.