acceptodds
Under review as a conference paper at ICLR 2027

When Anchors Commit, Coherence Breaks

Abstract

Diffusion multimodal large language models (dMLLMs) generate responses through iterative denoising, with predictive confidence determining which masked positions are committed first. Under full-sequence confidence-based decoding, we identify a three-phase commitment pattern in the evaluated trajectories: opening sentences first, tail anchors next, and middle content last. This timing imbalance leaves extended masked spans in the middle of a response whose surrounding content is already fixed, and is associated with semantic discontinuities. We investigate local attention concentration as an intervention target: sequence-wide detection can prioritize dominant peaks at already committed positions while overlooking concentrations in unresolved regions. We propose GADS (Gap-Aware Dynamic Suppression), a training-free intervention that detects concentrated attention columns within a local window and applies a reference-preserving logit bias, while retaining full-sequence token selection. GADS shifts commitment toward previously delayed middle content and reduces the tail-before-middle gap from 31.6 to 6.7 global denoising steps on LLaVA-Bench. Across LLaDA-V, MMaDA, and LaViDa, it improves LLaVA-Bench scores by 12.0, 21.3, and 2.1 points, respectively, while reducing object hallucination rates on all three backbones. These results support local attention intervention as an effective approach to improving commitment trajectories and long-form multimodal generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Semantic coherence, Long-Context dMLLM, decode order

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5DpDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a three-phase commitment pathology of dMLLMs. Structural anchors are committed prematurely, fragmenting mid-response semantics. The authors attribute this to attention sinks on structural tokens and propose GADS, a training-free local attention intervention.

Critical strengths
  1. The paper discovers the three-phase commitment pattern in dMLLMs, offering a new perspective on the internal decoding dynamics of dMLLMs.
  2. GADS reduces to logit biases on attention, is training-free, and adds <5% overhead at L=512.
Critical weaknesses
  1. The paper claims the cause of the three-phase pattern is attention sinks on structural tokens (e.g., <eos>, periods), yet GADS explicitly "does not identify semantic versus non-semantic columns". This means GADS cannot target the very cause the paper proposes. It is a blind suppressor that would equally suppress structural/non-structural tokens. This disconnect breaks the paper's overall logical coherence.
  2. If structural tokens caused the three-phase pattern, mid-sequence periods and line breaks should also commit early. Instead, the model jumps to the sequence tail, suggesting a U-shaped positional bias rather than a position-independent structural-token problem. The authors need to disentangle these two hypotheses.
  3. The paper fails to establish the causal link between attention sinks and the three-phase pattern. It provides no correlational evidence to show that stronger sinks actually lead to earlier commitment (e.g., a Pearson correlation between the strength of the attention sink on a token and how early that token is committed). Without this, its central finding and proposed GADS are disconnected.
  4. There is a mismatch between the attention intervention and the confidence-based sampling that drives the pathology. The three-phase pathology is driven by predictive confidence, whereas GADS intervenes on attention weights. Although the authors explicitly acknowledge that "incoming attention to a key position is not the same as the output confidence at that position," the paper offers no analysis to bridge the two. Why should reducing a token's attention-as-Key necessarily lower its output-layer confidence? Attention can be smooth while confidence remains high. This missing mechanistic account severely weakens the explanation of why the method works.
  5. The "long-form" claim is unsupported by the setup. The maximum tested length is L=512,** with no evaluation at L=2048**, and the paper concedes that contexts exceeding 10K tokens would require hierarchical windowing.
  6. The Figure 9 setup is ambiguous and the baselines are weak: the "Non-struct" baseline lacks a definition or construction details, and "Logit pen." appears to penalize only <eos>, whereas a fair baseline should penalize all defined structural tokens at the output layer.
Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response (Part1)

Focus: token identity versus position · attention-to-confidence pathway · causal scope

We thank you for your thoughtful review and for recognizing the three-phase diagnostic and the low-overhead intervention. Your questions highlight a distinction that our presentation should make consistently: the observed token categories, the attention statistic used for intervention, and the causal interpretation are different claims.

1. A token-agnostic intervention can address the observed concentration, but is not a structural-token detector.

GADS selects columns by their local attention-energy distribution, not by whether their predicted tokens are punctuation. Structural tokens motivate the investigation because they appear in the inspected concentration patterns; the intervention acts on the concentration itself. A structural column can therefore be selected when it meets the energy criterion, while a structural token without that pattern need not be penalized.

This makes the method compatible with the observation, but it does not establish structural specificity. You are correct that a semantic column can also be suppressed, and neither reference-logit preservation nor the algebraic propositions guarantee semantic preservation. Our intended claim is consequently an empirical one about reweighting locally concentrated attention, not that GADS recognizes and removes only harmful structural anchors. The manuscript's stronger structural-specific causal language should be replaced throughout, including the conclusion and figure annotations.

2. Token identity and position are competing or interacting explanations, not alternatives resolved by our plots.

We agree that a position-independent “punctuation commits early” account does not explain why tail punctuation would precede middle punctuation. Our observation concerns a joint configuration: tail anchors are committed while intervening content remains unresolved. The submitted trajectories do not distinguish a positional preference, structural-token predictability, or an interaction between them.

The appropriate analysis compares structural and non-structural predictions within position bins, and compares the same structural category across middle and tail positions, controlling for denoising step and unresolved context. These categories must be defined from predictions available at the measured step, rather than using future token identities to select intervention targets. Until such an analysis is available, we do not rule out a U-shaped positional bias or claim that structural identity alone causes the three-phase pattern.

3. Attention-as-key does not imply confidence at the same position.

We agree with this central point. For one attention row, write its output as oq=∑jaqjvjo_q=\sum_j a_{qj}v_j. A key-column bias changes the attending query's output according to

∂oq∂bk=aqk(vk−oq).\frac{\partial o_q}{\partial b_k}=a_{qk}(v_k-o_q).

Thus, the direct effect is on query representations. After subsequent layers and the output projection, the resulting confidence changes can have either sign; there is no general guarantee that suppressing incoming attention to key position kk lowers the output confidence at position kk. The plausible route is attention reweighting → changed query representations → changed predictions/confidence rankings → potentially changed commitments. This is a possible pathway, not a demonstrated monotonic mechanism.

The submission reports that timing and continuity change jointly under GADS: the LLaDA-V/LLaVA-Bench gap decreases from 31.6 to 6.7 steps, alongside a topic-switch reduction from 31.7% to 17.9%. These measurements do not establish the missing attention-to-confidence link. A correlation would be useful descriptive evidence, but would also need to control for position, step, token category, and shared prompt context. A stronger diagnostic is a paired forward pass from the identical partially masked state, measuring how the intervention changes confidence and rank at unresolved query positions before allowing the trajectories to diverge.

Accordingly, we do not claim that commitment reordering is the only channel of benefit. Changes to content representations can contribute as well.

Reviewer_bMQQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies full-sequence confidence-based decoding in diffusion MLLMs, where any remaining masked position can be committed at each global step (L35-38). Instrumenting LLaDA-V on 60 LLaVA-Bench In-the-Wild prompts, it reports a three-phase order (opening sentences, then tail anchors such as <eos>, then middle content) and a per-prompt tail-before-middle gap (Sec 2.1; Eq 1). Training-free GADS computes column energies in a local window of one head's pre-softmax logits (Eq 2), picks a reference column at the largest ratio gap of the sorted energies (Eqs 3-4), checks a mass condition (Eq 5), and adds non-positive log-biases to columns ranked above it (Eqs 6-7; Alg 1). On LLaDA-V, MMaDA and LaViDa it reports LLaVA-Bench gains of 12.0, 21.3 and 2.1 points and lower CHAIR (Table 1).

Critical strengths
  1. Under the paper's own decoding setup, GADS is the only method in Table 1 that lowers object hallucination without lowering LLaVA-Bench "all" on any backbone. "all" rises by 12.0, 21.3 and 2.1 points on LLaDA-V, MMaDA and LaViDa, whereas AdaBlock, Wavefront and CoTA each lower it on at least one backbone, and CHAIRs and CHAIRi fall against vanilla on each (LLaDA-V: 6.4 to 5.4 and 4.8 to 4.0). On LLaDA-V the mean output length barely moves (1.000 to 1.012, L177-178), so longer answers do not explain the gain. Lowering hallucination without losing long-form coverage is the hard direction.
  2. The commit-order diagnostic is measured, not anecdotal, and it shows where the middle of a response falls behind. Global commit steps are recorded per position and summarized by a per-prompt tail-before-middle gap (Eq 1), which GADS shrinks from 31.6 to 6.7 steps, with a paired 95% bootstrap interval of [-29.7, -20.4] for the change over the 60 prompts (Fig 8, L879-880). Early <eos> termination was already reported for LLaDA (Weakness 1), but I credit the opening, tail, middle ordering and its per-prompt measurement as the paper's most original piece, and other dMLLM decoders can be scored on the same statistic.
  3. GADS is cheap and confined, and it leaves the full-sequence sampler unchanged. It needs no training, changes only logits whose query and key both lie in the local window (Prop 1), leaves the reference column unpenalized (Prop 2), and adds under 5% latency at L=512 (24.94 s to 26.15 s, Fig 10a) while being triggered on 14.2-27.4% of steps (Fig 10b). For a decoding-time fix this cost profile matters, provided the gains hold against the baseline in Weakness 1.
Critical weaknesses
  1. LLaDA's authors published a fix for premature <eos> under full-sequence decoding, and Table 1 never runs it. They trace "early termination in low-confidence remasking" to <eos> padding in SFT data and set the <eos> confidence to zero (Nie et al., 2025, Sec 3.2, App B.4); MMaDA's authors decode in blocks, noting that non-autoregressive fixed-length generation "often produces very short responses" (Yang et al., 2025b). LLaDA-V and MMaDA build on LLaDA. Zeroing <eos> confidence would defer <eos> commitment, as GADS does (Fig 8a). Fig 9's terminator controls recover 50-76% of GADS's change, without intervals. Fix: add the zero-<eos>-confidence sampler to Table 1 on all three backbones, with intervals.
  2. Fig 9, the only test that the sink itself drives the commit order, repeats end-task numbers under commit-order labels. Its Vanilla and GADS bars equal Table 1's LLaDA-V LLaVA-Bench "all" (57.8, 69.8) and CHAIRs (6.4, 5.4) and Fig 6b's topic-switch rate (34%, 14%); its "<eos> commit step" falls from 6.4 to 5.4, where Fig 8a has 4.2 to 25.6. The paper itself says the attention-to-confidence link "requires measurements at matched intermediate states" (L202-203), yet the caption calls the sink "the causal driver" (L900-901) and the Conclusion says sinks "artificially inflate structural-token confidence" (L476-478). Fix: rerun the four controls on Fig 8's metrics.
  3. The gains do not follow the length dependence the paper states, and the largest short-form gain reaches the backbone's published score. LLaDA-V's MME cognition, a yes/no task that Sec 4.2 treats as short-form (L457), rises from 343.9 to 508.2 (Table 1), close to the 491 LLaDA-V's authors report (You et al., 2025), while MathVista, in the long-form and reasoning group (L412-414), moves at most 1.2. This contradicts "gains on very short answers ... are marginal" (L483-484) and the scope analysis invoked under Fig 10c (L934-935). The MME gain looks like recovery from a weak vanilla setup. Fix: state generation length, steps and block setting per benchmark, and report output lengths.
Questions for authors
  1. On MMaDA, what are the mean output length and the Fig 8 gap with and without GADS, and how much do 23.4 and 44.7 move over seeds and judge runs? Bootstrap intervals for the Table 1 LLaVA-Bench cells would help.
  2. Fig 9: do panels 1-2 plot LLaVA-Bench "all" and CHAIRs rather than commit confidence and <eos> commit step? Can you report the four controls on Fig 8's gap and structural-token confidence?
  3. Which generation length, step count, window and block length did each Table 1 benchmark use? LLaDA-V's authors report MME 491/1507 and CoTA reaches 490.5/1492.7 here; why is your vanilla row 343.9/1411.9?
  4. Which LaViDa checkpoint did you use, LLaDA-8B or Dream-7B based? LLaDA-V and MMaDA build on LLaDA; does the three-phase order hold on the public Dream-based (AR-initialized) LaViDa?
  5. Your CACD rows equal CACD's own Table 1, whose LLaDA-V vanilla ScienceQA is 79.23, so L461-462 mixes setups; your LaViDa AdaBlock and MMaDA Wavefront CHAIR cells round its values. Which rows did you run (L511-513)?
Reviewer_n5KNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that confidence-driven decoding in diffusion multimodal LLMs commits opening sentences first, structural tokens second, and middle content last, leaving masked gaps that correlate with topic drift. The authors proposed GADS, a training-free intervention that detects a locally over-attended column inside a window and biases its logits down toward a reference column.

Critical strengths
  • The observation is interesting and the attention axis is an under-explored angle relative to the scheduler-axis crowd.
  • The method is generally sound.
  • The method makes minor extra overhead.
Critical weaknesses
  • The construction of the intervention window W is not given. Section 3.1 states only that W is an input to the operator, and neither Section 3.2 nor Algorithm 1 defines how W is derived at each step. The constants alpha and tau, which appear only in the sweep of Fig. 11, are likewise undefined.

  • There exist some failure modes. For example, the method degenerates to the identity map in several regimes when the reference coincides with the highest-energy column.

  • The local-versus-global comparison that motivates the design varies selection rules, reference levels, and bias strengths simultaneously. The authors concede that it evaluates the complete interventions and does not by itself isolate locality.

  • It remains undetermined through which channel GADS improves generation: by reordering token commitments, by altering the residual representations of unresolved positions, or both.

  • The three-phase commitment pathology is only quantified on 60 LLaVA-Bench prompts and 500 CHAIR images, each on a single backbone.

  • On CHAIR, GADS attains the best CHAIRs/CHAIRi but its recall is well below CoTA (63.1). This pattern is consistent with more conservative generation rather than strictly more faithful outputs.

  • The stratified analysis shows gains concentrated on instances with three or more blocks and slightly negative effects for two or fewer.

  • Anonymous source code is offered to the Area Chair only, with no public anonymous link. Moreover, the ethics statement references a "small preference study", yet no protocol, participant count, or results appear anywhere in the paper or appendix.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.