acceptodds
Under review as a conference paper at ICLR 2027

Testing Attribution Claims in Diffusion-LM Remasking: A Localized Effect Without a Demonstrated Benefit

Abstract

Masked diffusion language models decode in parallel, and training-free overlays improve them by re-decoding committed tokens. The strongest, CoRe, jointly masks a pool of low-margin candidates and re-predicts them, so a token that confidently holds a neighbor's error in place is rarely perturbed. We ask whether widening the perturbation to each candidate's neighbors helps, and test the resulting method, Local-Context-Dropout (LCD), against the controls a headline comparison omits. It fails them: its +2.44 pp HumanEval gain over CoRe is indistinguishable from spending the same extra compute on more CoRe verify calls, no gate configuration dominates, and the frozen method loses 0.86 pp on held-out MBPP problems. What survives is an attribution result. Removing one masked neighbor recovers a median 0.08 of the full-mask effect while restoring it inside the mask removes 0.52, so neighbors act in combination, and adjacent positions carry more of the effect than non-adjacent ones from the same mask, with commit confidence balanced at mask time, across two backbones and three tasks. The association is positional, not proof that proximity is the operative variable, and has no detectable signature in program correctness. Code: https://anonymous.4open.science/r/LCD-iclr27-1EE0/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
masked diffusion language models, inference-time remasking, perturbation-based attribution, controlled ablations, negative results

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UFA9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper extends CoRe, to Local-Context-Dropout which is LCD. LDE has additional masking pass for candidate’s r=1 neighborhood. Authors experiment LCD if it helps and where it helps, but it has a minimal effect on the final performance gain.

Critical strengths
  1. Authors are clear and honest on the result, that there is not much of the gain in the methodology.
  2. A lot of ablation studies are done in the setting.
Critical weaknesses
  1. Figure 1 has unnecessarily many texts in the figure and some words going outside the box, which makes it harder to parse.
  2. LCD vs CoRe Humaneval experiment has 4 scores changed, which leads to 2.44pp gain. Authors explicitly say that gain is indistinguishable.
  3. The fact that is revealed, such as ‘relatively close neighborhood is important’ is trivial in language models.
  4. The preliminary result does not connect to the main claim of the paper.
  5. There is only CoRe and base experiment.
  6. Numbers (accuracy) of HE, MBPP, GSM8K is way behind the LLaDA paper, about 20pp.
  7. Table 24 has “pending: instrumented runs in progress”. The paper seems to be not finished, and not ready for the publishing. At least the paper should be containing all the content.
  8. The title “Testing Attribution Claims in Diffusion-LM Remasking” is too generalized compared to the experiments in this paper.
Questions for authors
  1. Is there any reason that we should use LCD? Most questions are already addressed in the weaknesses section.
4 comments
AuthorsOpenReview ↗

Response to Reviewer UFA9: part 1

(Part 1/2)

Response to Reviewer UFA9

Thank you for the review. We agree with most of the weaknesses you list, and we changed the paper to address each one. The revised PDF marks new and changed text in blue. We also rewrote the main text in shorter, plainer sentences.

Figure 1 has too much text, and some overflows its boxes. Redrawn. The new Figure 1 has two schematic panels (what CoRe masks, what LCD masks) and three small result panels, one per research question. It has about a third of the previous text, larger type, and nothing crosses a frame.

The HumanEval gain is four problems and is indistinguishable at equal compute. We agree, and this is now the paper's first result rather than a caveat (Sec. 4.1). At equal compute CoRe gains 1.22 pp and LCD 2.44 pp; head to head they differ by four problems to two (p = 0.688), so LCD's lead is not statistically reliable. (We now say "not significant" rather than "indistinguishable", since a null test does not show equivalence.) The frozen method loses 0.86 pp on 464 held-out MBPP problems. We no longer present LCD as an improvement, and the conclusion says we do not recommend it as a decoder.

"A close neighborhood is important" is trivial in language models. That nearby context matters is indeed expected; Piskorz et al. (ICML 2026) document locality bias in diffusion LMs, and we cite them. Our findings go beyond it in three ways:

  1. The dependence is joint. Restoring one neighbor removes a median 0.52 of the effect, while removing it alone recovers only 0.08. A locality curve over single positions cannot show this.
  2. It holds within one mask, against positions matched on commit confidence and, on the two largest cells, on lexical role. Distant copies of the candidate's own token carry little effect: median necessity 0.00 on every LLaDA cell (new App. O).
  3. Most importantly, in our experiments acting on it did not repair programs. We added a release test (Sec. 4.3, Table 4): we free the neighbor the probes implicate, let the decoder re-decode it, and compare with freeing a matched distant position. Over 1,097 interventions, freeing the neighbor changes it in 23% of events against 3% for the control, yet repairs 2 of 621 failing problem-events against 1 (difference +0.2 per 100, 95% CI [-0.6, +1.0]). On problems that pass, 88% of implicated tokens are unchanged in the final output.

The third point is not trivial, because it cuts against the premise behind neighbor masking: that the neighbor a prediction depends on is a source of error worth removing. We state it as a result for this protocol, which frees one neighbor after LCD has revised the candidate, not as a general law.

The preliminary result does not connect to the main claim. We restructured the paper around one line of argument, shown in Figure 1 (c) to (e). RQ1: neighbor masking does not help once compute is matched. RQ2: what the extra pass measures is a joint, local dependence. RQ3: acting on that dependence repaired no more programs than acting on a matched control, which is consistent with neighbor masking not helping. Each section now ends by answering its question and leading to the next.

There is only CoRe and the base experiment. Table 1 compares seven methods on five benchmarks: vanilla decoding, CoRe, four other training-free overlays that we reimplemented on the same model (ReMDM, T2M, STaRR, Lookahead Unmasking), and LCD. The paper also runs a second backbone (Dream-7B), three equal-compute variants, a held-out set, count-matched masking controls, and, new in this revision, a second decoding budget. We build LCD on CoRe because CoRe is the only overlay with a verify pass whose mask can be widened. We have made the comparison set more visible in the setup paragraph.

AuthorsOpenReview ↗

Response to Reviewer UFA9: part 2

(Part 2/2)

Accuracy on HumanEval, MBPP and GSM8K is about 20 pp below the LLaDA paper. We attribute most of this to CoRe's compute-limited setting rather than a pipeline fault. It decodes 512 tokens in 128 steps, four tokens per step, with greedy decoding. CoRe's own paper reports the same range (12.2% HumanEval, 15.6% MBPP and 51.4% GSM8K for vanilla decoding in its Table 1), and our vanilla row is close to it (12.2%, 16.4%, 51.8%). To check both the gap and our conclusion, we reran HumanEval at 512 steps, one token per step (new App. R, Table 24). Vanilla decoding rises to 0.3354 pass@1, close to the 0.354 that the LLaDA paper reports for this model. So on HumanEval the gap comes from the decoding budget, not the pipeline. We ran this check on HumanEval only. For MBPP and GSM8K the budget is the likely cause, not a tested one, and for MBPP the LLaDA paper also uses 4-shot prompts where CoRe's harness uses 3-shot. At 512 steps, LCD and CoRe reach the same outcome on every one of the 164 problems: LCD still acts and changes the text of 19 completions, but not one pass or fail. (CoRe itself does not beat vanilla at this budget: 1 problem fixed, 4 broken.) The setup paragraph and App. R now state this scope.

Table 24 says "pending". It (now Table 27) is filled with counts and denominators: CoRe's intended remasks displaced by LCD are 55 of 791 on HumanEval (6.95%), 222 of 2,829 on MBPP (7.85%) and 678 of 9,330 on GSM8K (7.27%). These come from runs that reproduce the reported scores exactly.

The title is too general. Changed to "Masking a Token's Neighbors in Diffusion-LM Remasking: Local Dependence Did Not Identify Useful Remasks". It now names the intervention studied and states the finding as an empirical result.

Q. Is there any reason to use LCD? Not as a decoder: it does not beat CoRe at equal compute and does not carry to held-out problems. Its value is as an instrument. With the release test, it lets one check whether a perturbation-based remasking rule is acting on errors or only on context sensitivity. We release it for that purpose.

Thank you for your rebuttal.

  1. Redrawn Figure 1 still has unnecessary comments, which summarizes figure but not straightforward to understand. "Median necessity share." "Changed, but rarely repaired."
  2. Still, Tables 2, 3, and 4 do not have an intuitive interpretation after changing. Even though Table 27 has a result, it also does not help in understanding the main claim. Knowing success/failure numbers does not help building intuition.
  3. In your rebuttal, "We no longer present LCD as an improvement" and "Its value is as an instrument." part -if I am understanding correctly, contribution seems to be low. In the release test, remasking an influential neighbor changed that token more often than remasking a matched distant token, but you found no reliable advantage in correcting errors. If the observation itself is now the main contribution, I am not convinced that it substantially changes our understanding of how to select tokens for remasking.

I will maintain my original score.

AuthorsOpenReview ↗

2nd response to reviewer UFA9

Thank you for the follow-up. We agree with all three points and revised the paper again. Changes from this round are in purple in the revised PDF.

1. Figure 1. We removed both notes. Panel (a) is now one real verify step, from MBPP/467 ("remove lowercase letters"). It shows what each method masks around the candidate upper in if char.isupper():, and the probability the model gives to lower: 0.24 under CoRe's mask, 0.63 under LCD's, and 0.24 again when only .is is put back. LCD therefore remasks upper. The decoder writes upper again, keeps .is, and the program passes. Panels (b) to (d) now carry axis titles in place of notes: pass@1 change against CoRe, share of the effect removed, and programs repaired.

2. Tables 2 to 4. Each is rebuilt around one plain question, and the text says how to read it.

  • Table 2 counts solved problems. CoRe solves 23 of 164 HumanEval problems. With LCD's extra compute spent on more verify calls it solves 25, and LCD solves 27.
  • Table 3 asks how much of LCD's effect disappears when one masked token is put back. For the adjacent neighbor the median is 0.45 to 0.79 across three tasks and two models. For a distant token of the same confidence it is 0.00 to 0.07. The test statistics moved to the appendix.
  • Table 4 places the release test next to a new positive control (point 3a).

We agree that the slot-displacement counts (old Table 27) do not bear on the main claim. The main text no longer cites them; they stay in the appendix as bookkeeping.

3. Contribution. You are right that our reply leaned on "instrument", and that an observed dependence which does not repair programs is a weak result on its own. We added two analyses and one experiment. Together they turn the observation into a claim about how to choose tokens for remasking.

(a) A positive control for the release test (Sec. 4.3). On passing programs we replace the neighbor LCD relied on with the model's runner-up token, and keep it out of the verify candidates. In 35 events this breaks the program. Freeing the planted token then repairs 80% of them, against 11% for freeing the matched distant token. So the test does catch a wrong neighbor when one is there. The real neighbors LCD relied on are repaired in 2 of 621 failing cases, against 1 for the control. This bounds the share of those neighbors that are fixable errors at about 1.5%.

(b) Dependence against correctness. The neighbor dependence is as strong on problems that pass as on problems that fail. On LLaDA the AUROC is 0.46 to 0.60, and every 95% interval reaches 0.5. A signal that looks the same on correct and wrong programs cannot pick out the wrong ones.

(c) Which remasks help (new RQ4, Sec. 4.4). Take every problem whose outcome a rule changes. CoRe's own remasks, chosen because the re-prediction disagrees with the token in its visible context, are repairs in 68% of cases (376 against 179; 91% on held-out MBPP). The remasks LCD adds, chosen because hiding neighbors moved the prediction, are repairs in 51% (84 against 80; 30% held out), a coin flip. CoRe acts first and may take the easier errors, and the paper says so.

What this changes. CoRe frames its perturbation test around tokens "tied to hard consistency constraints", and LCD extends the same premise: if hiding context changes a prediction, the token deserves revision. Our results split that premise in two. Dependence means the token is bound to its context, as upper is bound to .is. Error means the token is wrong. In our experiments these came apart: the dependence is strong, local and equally present in correct code, while the useful remasks came from disagreement in full context. Sec. 4.4 turns this into three design rules. Score a candidate by evidence against the token in its full context. Accept a proposal made under a wider mask only if the full-context prediction supports it. Test any new selector by acting on its picks against matched controls, with a planted error as a positive control. These apply to anyone designing remasking rules, not only to LCD.

We hope the new figure, the rebuilt tables and the added evidence address your concerns. We would be grateful if you would reconsider your score in light of them.

Reviewer_mpM1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper adds neighborhood masking to CoRe through Local-Context-Dropout (LCD). It compares coding accuracy under equal-compute and frozen held-out evaluation, then uses perturbation experiments to examine local influence, necessity, sufficiency, and interactions. The initial accuracy gain disappears under the stronger controls. The remaining result is a conditional local effect on token probabilities, without a demonstrated accuracy benefit or beneficial mechanism.

Critical strengths

The equal-compute and frozen held-out evaluations directly test the initial accuracy claim. Confidence-balanced within-mask controls, problem-level resampling, and additional-backbone checks strengthen the analysis of local effects. The paper also distinguishes token-probability effects from end-task accuracy.

Critical weaknesses
  1. The contribution is limited, and the statistical exclusion claim is unsupported.

LCD extends an existing remasking procedure to neighboring tokens, and its accuracy gain disappears under stronger controls. The remaining findings concern local sensitivity and interactions within selected remasking events. They do not establish that these effects produce beneficial corrections. Section 1 and Appendix M acknowledge that perturbation-based attribution and the distinction between necessity and sufficiency are established ideas. The paper needs to explain which finding changes the current understanding of remasking.

Section 4.3 and Appendix P (p. 34) use nonsignificant associations and a power simulation to argue that a strong association can be excluded, although the analyses contain only four and six outcome-changing problems. Simulated power against a specified alternative does not provide an observed-data confidence bound on the association. Report an effect-size interval or an appropriate exclusion test, or withdraw the exclusion claim.

  1. The manuscript does not present a consistent, completed set of results.

Table 24 (p. 36) contains "pending: instrumented runs in progress" where a directly counted displacement rate should appear. The surrounding prose presents this measurement as a reported result. Appendix Q provides a budget-preserving variant, but the promised direct measurement remains unreported. Provide the counts and denominators or remove the claim that this measurement has been completed.

Internal revision commentary remains throughout the manuscript. Page 8 says an "earlier version drew the opposite conclusion" and "we withdraw it". Page 10 announces "Three additions in this revision", and pp. 28 and 32 state that "a reviewer caught it". Appendix U (p. 36) says it "has been rebuilt twice, and the second rebuild is a retraction" and that earlier instrumentation "did not measure what we said it did". Page 25 describes a second occurrence within the same revision of one trace set silently replacing another's numbers. The manuscript needs one account of the final protocol, measurements, and conclusions, with any scientifically relevant correction history consolidated and superseded editorial commentary removed.

The interpretations also remain inconsistent. Appendix S (pp. 35-36) invokes a "better-chosen token", "doing no harm on reasoning", and discoveries driving the HumanEval gain, without reconciling these claims with the equal-compute and held-out results. Figure 4 (p. 19) labels a comparison "compute-matched" while reporting budgets of 1.06x and 1.13x on the same page. The equal-compute experiment elsewhere does not justify that label.

Figure 1 (p. 2) has multiple visible text/frame collisions and overflow. The middle-panel heading crosses into the right-hand panel, the middle-panel legend extends beyond its frame, and dense, very small text impairs readability. These defects make the central overview figure difficult to read.

I recommend clear rejection because the contribution remains limited and the statistical and reporting problems affect the interpretation of the results.

Questions for authors
  1. What specific conclusion about diffusion-LM remasking is established beyond the known limitations of perturbation-based attribution and this unsuccessful neighborhood extension? Please identify the supporting result and explain which existing understanding it changes.

  2. What observed-data confidence interval or exclusion test supports the statement in Section 4.3 / Appendix P that a strong association is excluded? Please distinguish that evidence from simulated power under a selected alternative.

  3. Please provide the actual Table 24 displacement result, including counts and denominators, or confirm that it was not completed. Explain its relationship to the budget-preserving experiment in Appendix Q, and identify which positive mechanism and utility claims in Appendix S remain supported after the equal-compute and held-out analyses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer mpM1: part 1

(Part 1/2)

Response to Reviewer mpM1

Thank you for a detailed and exact review. Every concern you raised was valid, and each one led to a change in the paper. The revised PDF marks new and changed text in blue. We also rewrote the main text in shorter, plainer sentences.

W1 / Q1. Which finding changes the current understanding of remasking? We agree that the submission did not answer this, because it stopped at token probabilities. We added an experiment that does: the release test (Sec. 4.3, Table 4, App. Q). For each applied intervention we free the neighbor the probes implicate most, let the decoder re-decode it, and compare the final program with one in which a confidence-matched non-adjacent position was freed instead. Over 1,097 interventions on HumanEval, MBPP and GSM8K:

  • freeing the implicated neighbor changes it in 23% of events, against 3% for the matched control, so the dependence is real;
  • but it repairs 2 of 621 failing problem-events, against 1 for the control. Repairs are rare, so we give a paired score interval (Newcombe), which stays informative when both arms have zero repairs: the difference is +0.2 per 100 failing events, 95% CI [-0.6, +1.0]. Counting each of the 435 failing problems once gives [-0.9, +1.4] per 100 problems;
  • on problems whose reference output passes, 88% of implicated tokens are unchanged in the final output (89% of those inside the scored text). This shows most are compatible with a passing output, not that each is correct;
  • restricting to the 717 events whose candidate lies inside the scored output, repairs are 1 against 1.

So the finding is a dissociation. A verify prediction depends on its committed neighbors, jointly and more than on equally confident distant positions (Sec. 4.2). But in our experiments acting on that dependence repaired no more programs than acting on a matched control. The existing understanding, in CoRe's framing and in the motivation of neighbor masking, treats context sensitivity under a masking perturbation as evidence of an error to revise. In our data the two came apart, so a perturbation-based remasking score needs its own error criterion. We state the scope plainly: the test frees one neighbor after LCD has revised the candidate and adds one remask, so it does not show that local dependence can never identify errors (App. Q). The supporting results are Table 4 and Sec. 4.2; the conclusion now states the claim in these terms. We do not recommend LCD as a decoder, and the paper now says so.

W2 / Q2. The exclusion claim. You are right that simulated power against a chosen alternative is not an observed-data bound. We removed the power argument. Table 25 now reports, for each task and statistic:

  • the probability of superiority A (chance that a changed problem has the larger statistic), with a 95% bootstrap interval that resamples problems within each group;
  • the Hodges-Lehmann shift with its exact distribution-free 95% interval from inverting the Mann-Whitney test.

Observed A is 0.34 and 0.55 on HumanEval and 0.53 and 0.30 on MBPP. The upper ends of the four intervals are 0.62, 0.81, 0.76 and 0.50. A strong positive association (A >= 0.9) lies outside every interval; a moderate one (A = 0.7) lies inside two of the four, and the exact shift intervals all include zero. The text now says exactly this, notes that the bootstrap is approximate with four and six changed problems, and points to the release test as the direct test on far more events.

W3 / Q3. Table 24 (now Table 27). The measurement was complete in our logs but not in the table, which was our error. The table is now filled with counts and denominators, read from the decoder's counters in the probe runs. Those runs reproduce the reported scores exactly, so their decoding paths are those of Table 1.

TaskCoRe's intended remasksdisplaced by LCDrateapplied LCD interventions
HumanEval791556.95%91
MBPP2,8292227.85%429
GSM8K9,3306787.27%939

(The GSM8K counts include one padded duplicate generation from distributed evaluation.) Relation to the budget-preserving variant: on HumanEval, 55 of the 91 applied interventions took a slot CoRe would have used, and 36 used a slot CoRe left empty. The budget-preserving variant forbids displacement. It loses problems 60 and 82, the two repairs that needed a displaced slot, and keeps 66 and 85, so it gains +1.22 pp instead of +2.44.

AuthorsOpenReview ↗

Response to Reviewer mpM1: part 2

(Part 2/2)

Which Appendix S claims remain (now App. V)? We rewrote the three you flagged. "Better-chosen token" is now "a different token from the one CoRe would have chosen, using CoRe's budget; whether it is better is what the equal-compute, held-out and release tests decide, and they find no evidence that it is." "Doing no harm on reasoning" and "discoveries that drive the HumanEval gain" are removed. The only positive statements left are descriptive: the gates cut LCD's action rate about six-fold without switching it off, and LCD does not increase the remask count.

W4. Internal revision commentary. Removed throughout. The paper now gives one account of the final protocol and results. Specifically:

  • p. 8, "earlier version drew the opposite conclusion ... we withdraw it": removed; the two-probe result is stated directly.
  • p. 10, "Three additions in this revision": now "Three components are worth checking first."
  • pp. 28 and 32, "a reviewer caught it": removed; the bootstrap and radius-2 passages now state the method and the result.
  • App. U (now App. X), "rebuilt twice ... retraction" and "did not measure what we said it did": rewritten as a design rationale ("why a full-mask comparison is not enough") followed by the measurement.
  • p. 25, the trace-overwrite story: removed; the checks appendix now states the provenance rule without the history.
  • All "the submitted version ..." passages in the appendices (baselines, prompt matching, gates, robustness, priority, Dream): rewritten as direct statements.

Where a correction history is scientifically relevant, we kept the fact and dropped the narrative. For example, the end-dump pitfall for fixed-step reimplementations is now a short methods note.

W5. Inconsistent interpretations. Appendix S is fixed as above. Figure 4 no longer says "compute-matched": the shaded band is labeled "near 1x (not equal)", and the appendix text says that 1.06x and 1.13x are different budgets, with the equal-compute comparison in Sec. 4.1. While checking this figure we found that it still plotted the external methods' pre-pinning GSM8K means (Lookahead 0.3360 instead of Table 1's 0.3321). It now matches Table 1. Two rounded values in Figures 5 and 8 (-0.16, -0.01) now match the tables (-0.15, -0.02). One GSM8K attribution event was counted twice, because distributed evaluation pads the last batch with a duplicate problem; removing it changes Table 3's GSM8K cell from 692 to 691 events and moves no conclusion. We also found that Table 1's ReMDM row mixed two confidence variants across tasks; its GSM8K cell came from a run with stored confidence while the other cells read confidence at the current step. We reran GSM8K with the current-step variant and pinned prompts (0.4882), so the row now uses one variant. We also found that the decoder's seed argument never reached the random draws of ReMDM or Lookahead, because lm-evaluation-harness reseeds PyTorch at start-up, so the earlier seed runs differed only through the GPU count, not the seed (Lookahead gave 0.1585 at all three). The seed table now varies the harness seed: on HumanEval, ReMDM-conf spans 2.44 pp and Lookahead 1.83 pp across three seeds, as much as LCD's whole HumanEval gain. App. B reads each ReMDM cell as one draw with that spread. Finally, we aligned every summary sentence with Table 2: at equal compute CoRe gains 1.22 pp and LCD 2.44 pp, a difference that is not significant (p = 0.688). The text now says exactly that, not that the two are equal or indistinguishable.

W6. Figure 1. Redrawn (Figure 1). It now has two schematic panels and three small result panels, one per research question, with about a third of the previous text, larger type, and no text crossing a frame.

We hope the revision addresses your concerns about soundness and reporting. The release test is, in our view, the result that the contribution now rests on, and it is reported with its validation and caveats in App. Q.

Reviewer_V4J3OpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study whether masking neighbouring tokens can improve remasking decisions in masked diffusion language models (MDLMs). They introduced Local-Context-Dropout (LCD), extension of CoRe. They introduced an additional verification pass to perturb the local context around candidate tokens. LCD improves HumanEval performance by 2.44 pp, but the gain is not able to generalize to held-out MBPP problems. The authors further analyze contextual dependencies through counterfactual masking experiments, finding that adjacent tokens have a stronger influence on verification probabilities than non-adjacent tokens across multiple tasks and model backbones. However, these attribution effects are not clearly associated with downstream correctness.

Critical strengths

They include equal-compute comparisons, held-out evaluations, gate ablations, and random masking controls. These experiments help identify which improvements are actually due to the proposed LCD method.

The counterfactual experiments provide observations about how neighbouring tokens influence verification predictions. The difference between removing a token individually and restoring it within a larger mask suggests that these dependencies are not entirely independent. Authors observe similar patterns across different benchmarks and two backbones.

Critical weaknesses
  1. LCD is a relatively straightforward extension of CoRe, and its methodological novelty appears limited. Similarly, the attribution findings largely align with existing observations about local contextual sensitivity in MDLMs. More importantly, the proposed method does not demonstrate consistent improvements across datasets, with the HumanEval gain not translating to MBPP or the reasoning benchmarks. The negative results on held-out MBPP further raise concerns about generalization. Given the limited novelty and lack of consistent downstream improvements, the overall significance of the contribution remains unclear.

  2. The paper motivates LCD by suggesting that incorrect neighbouring tokens reinforce errors and prevent CoRe from correcting them. However, the experiments mainly demonstrate changes in prediction probabilities when neighbouring tokens are masked, without directly showing whether these neighbours are incorrect or responsible for the errors. While the attribution analysis is interesting, there is limited direct evaluation of how neighbouring tokens contribute to incorrect predictions or successful corrections.

Questions for authors
  1. Piskorz et al. (ICML 2026) have already demonstrated locality bias in MDLMs, while CoRe investigates contextual sensitivity through masking-based perturbations. Can the authors please clarify what new insight the proposed attribution analysis provides beyond these existing findings, particularly regarding the role of neighbouring tokens in error correction?

  2. The paper suggests that incorrect neighbouring tokens can cause errors and prevent CoRe from correcting them. However, it is unclear whether the influential neighbouring tokens are actually incorrect. Given the inconsistent improvements across benchmarks and poor generalization on held-out MBPP, could the authors provide more evidence for this hypothesis and explain why LCD does not consistently improve performance?

  3. Could the observed differences between adjacent and non-adjacent tokens be explained by their syntactic or semantic relationships rather than positional proximity? For example, would the conclusions remain similar when comparing tokens with similar syntactic roles?

  4. Given the lack of a clear relationship between the attribution findings and downstream correctness, could the authors clarify how these observations contribute to improving remasking decisions? Is there evidence that the observed attribution effects help identify or correct errors, rather than simply reflecting the model's sensitivity to changes in context?

3 comments
AuthorsOpenReview ↗

Response to Reviewer V4J3: part 1

(Part 1/2)

Response to Reviewer V4J3

Thank you for a careful review. Your second question pointed at the gap that mattered most: we had shown what the verify prediction depends on, but not whether the neighbor it depends on is wrong. We ran a new experiment to answer it, and the answer reshapes the paper. The revised PDF marks new and changed text in blue. We also rewrote the main text in shorter, plainer sentences.

Summary of the main change. We added a release test (Sec. 3.3, Sec. 4.3, Table 4, App. Q). For each applied LCD intervention, we take the neighbor the probes implicate most (largest necessity share), replay the decoding from the start, and at the event's verify step return that neighbor to the mask pool so the decoder re-decodes it. As a control, we instead free the non-adjacent position that the balanced design matched to it on commit confidence. Both arms add one extra remask at the same step. Over 1,097 interventions on HumanEval, MBPP and GSM8K (LLaDA-8B-Base):

free the implicated neighborfree a matched non-adjacent position
re-decoded to a different token23%3%
repairs (fail to pass), of 621 failing problem-events21
repair difference per 100 failing events, 95% score CI+0.2 [-0.6, +1.0]
breaks (pass to fail)53
unchanged in final output, passing reference problems88%99%

Replays without an intervention reproduce the reference output byte for byte (767 of 770 problems), and we keep only events whose two arms reach the event step on the reference path. The token-change gap also holds when each problem counts once (213 problems against 22). Counting each of the 435 failing problems once, the repair difference is +0.2 per 100 problems, 95% CI [-0.9, +1.4].

W1 / Q1. What is new beyond Piskorz et al. (ICML 2026) and CoRe? Piskorz et al. show that MDMs favor local over distant prompt context, and that appended masks distract the model. CoRe shows that masking low-margin candidates exposes context-sensitive tokens. Neither tests two things a remasking rule depends on:

  1. Does a verify prediction on a committed output token depend on its neighbors more than on other masked positions, once commit confidence and lexical role are held fixed? Our answer is yes for confidence on every cell, and for lexical role on the two cells with enough events (Q3 below). The dependence is also joint: restoring one neighbor removes a median 0.52 of the effect, removing it alone recovers only 0.08 (Sec. 4.2).
  2. Does acting on that dependence repair programs? In our release test it did not do so more often than a matched control. Freeing the implicated neighbor repairs 2 of 621 failing cases, against 1 for the control (difference +0.2 per 100, 95% CI [-0.6, +1.0]).

The second point is the one we would ask you to weigh. Locality bias explains why the dependence is local. It does not by itself make local dependence a useful error signal, and in our experiments it was not one. A perturbation-based remasking score measures what a prediction rests on; whether that marks an error needs its own test. We state this as an empirical result for this protocol, not a general law (see the limits under W2 below). We added a paragraph on this to the introduction and a full comparison with Piskorz et al. to App. M.

We agree that LCD itself is a direct extension of CoRe. The revised paper treats it as a test of the neighbor idea, not as a method we recommend, and the conclusion says so.

W2 / Q2. Are the influential neighbors actually incorrect? Why does LCD not improve consistently? The release test answers the second part directly and the first part only in part. Freeing the implicated neighbor changes it far more often than freeing the control (23% of events against 3%), because it is coupled to the candidate that LCD has just revised. But on failing problems those changes repair 2 of 621 problem-events, against 1 for the matched control. On problems whose reference output passes, 88% of implicated tokens are unchanged in the final output (476 events; 89% of the 298 that lie inside the scored text). That shows most are compatible with a passing output, not that every implicated token is correct, and it says nothing about failing problems, where no token-level labels exist. Only 65% of applied interventions have their candidate inside the scored output; within that subset, repairs are 1 against 1. This fits LCD's uneven gains: its extra pass changes the context the candidate is re-predicted from, and in our data acting on the dependence repaired no more often than perturbing a matched position.

AuthorsOpenReview ↗

Response to Reviewer V4J3: part 2

(Part 2/2)

The test has limits, which App. Q states. It frees the neighbor after LCD has revised the candidate, it adds one extra remask, and it uses one backbone. So it measures the benefit of one extra remask under this protocol. We do not claim that local dependence can never identify errors.

Q3. Syntactic or semantic relationship instead of positional proximity? We added a fine-grained analysis (App. O, Table 22). Each probe's token gets one of ten lexical roles (keyword, identifier, number, operator, opening and closing bracket, separator, string, space, newline). Within each event we compare an adjacent and a non-adjacent probe of the same role, choosing the closest commit confidence.

  • The adjacent advantage survives on MBPP (72 pairs against 34, p = 0.0003) and GSM8K (198 against 114, p < 10^-5), with problem-level bootstrap intervals above zero.
  • On HumanEval, only 19 to 24 events have a same-role pair, and the difference is not detectable. On Dream it survives role matching alone but not role and confidence matched together.
  • Distant copies of the candidate's own token, such as the same identifier used elsewhere, have median necessity 0.00 in every LLaDA cell and 0.00 to 0.04 on Dream (5 to 21 probes per cell). So this kind of semantic tie at a distance carries little of the effect.
  • Distant positions on the candidate's own line, a coarse syntactic unit, carry almost none either: median 0.00 to 0.03 on MBPP and 0.00 on GSM8K, against 0.21 and 0.13 for adjacent positions. Within events that have both, the adjacent probe is larger on MBPP (54 events against 13, and 48 against 19) and on GSM8K (147 against 61), each at p <= 0.001. HumanEval has too few same-line positions to test.

So on the two largest cells position matters beyond lexical role, and we do not claim it on HumanEval or Dream under the confidence-balanced design (with role matched and far positions drawn uniformly, Dream is significant too). These checks do not rule out finer relations. Lexical role and line membership are proxies, and we did not parse dependency or data-flow relations, so such a link could still carry part of the effect. The abstract, Sec. 4.2 and App. O now state this scope.

Q4. How do these observations help improve remasking decisions? The direct answer is that they show what not to rely on without a further check. A perturbation-sensitivity score finds tokens whose prediction depends on their context, and in our release test acting on that dependence did not find tokens worth revising. Three practical points follow, now in the conclusion and App. K:

  1. A remasking rule built on context sensitivity needs a separate error criterion.
  2. Single-position removal underestimates dependence, because neighbors act jointly (0.08 against 0.52). Scores built on leave-one-out masking will miss it.
  3. A claim that a perturbation helps "by removing a misleading token" needs a release test and an equal-compute control, not only a probe. Here the equal-compute control alone leaves no significant gain (4 problems to 2, p = 0.688).

Thank you again. The release test turned a descriptive finding into a tested one, and we think the paper is clearer for it.

Reviewer_V4J3OpenReview ↗

Comments after rebuttal

Thank you for the detailed response and additional experiments. I appreciate the authors’ efforts to address the concerns and clarify the scope of the contribution. However, after considering the rebuttal and the other reviewers’ assessments, I share their concerns regarding the limited novelty and significance of the findings.

The new release test addresses an important gap, but it does not demonstrate that the attribution signal identifies useful corrections. Releasing the implicated neighbor repairs only 2 of 621 failing problem-events, compared with 1 for the matched control. Thus, the higher token-change rate establishes sensitivity to the intervention, but does not establish a reliable benefit in error correction.

The revised framing is more appropriately cautious. Nevertheless, given the lack of a reliable equal-compute advantage, the negative held-out result, and the limited actionable insight for selecting tokens to remask, I remain unconvinced that the contribution is sufficiently substantial. I am therefore lowering my original rating from 3 to 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.