- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies a three-phase commitment pathology of dMLLMs. Structural anchors are committed prematurely, fragmenting mid-response semantics. The authors attribute this to attention sinks on structural tokens and propose GADS, a training-free local attention intervention.
Critical strengths
- The paper discovers the three-phase commitment pattern in dMLLMs, offering a new perspective on the internal decoding dynamics of dMLLMs.
- GADS reduces to logit biases on attention, is training-free, and adds <5% overhead at L=512.
Critical weaknesses
- The paper claims the cause of the three-phase pattern is attention sinks on structural tokens (e.g.,
<eos>, periods), yet GADS explicitly "does not identify semantic versus non-semantic columns". This means GADS cannot target the very cause the paper proposes. It is a blind suppressor that would equally suppress structural/non-structural tokens. This disconnect breaks the paper's overall logical coherence. - If structural tokens caused the three-phase pattern, mid-sequence periods and line breaks should also commit early. Instead, the model jumps to the sequence tail, suggesting a U-shaped positional bias rather than a position-independent structural-token problem. The authors need to disentangle these two hypotheses.
- The paper fails to establish the causal link between attention sinks and the three-phase pattern. It provides no correlational evidence to show that stronger sinks actually lead to earlier commitment (e.g., a Pearson correlation between the strength of the attention sink on a token and how early that token is committed). Without this, its central finding and proposed GADS are disconnected.
- There is a mismatch between the attention intervention and the confidence-based sampling that drives the pathology. The three-phase pathology is driven by predictive confidence, whereas GADS intervenes on attention weights. Although the authors explicitly acknowledge that "incoming attention to a key position is not the same as the output confidence at that position," the paper offers no analysis to bridge the two. Why should reducing a token's attention-as-Key necessarily lower its output-layer confidence? Attention can be smooth while confidence remains high. This missing mechanistic account severely weakens the explanation of why the method works.
- The "long-form" claim is unsupported by the setup. The maximum tested length is L=512,** with no evaluation at L=2048**, and the paper concedes that contexts exceeding 10K tokens would require hierarchical windowing.
- The Figure 9 setup is ambiguous and the baselines are weak: the "Non-struct" baseline lacks a definition or construction details, and "Logit pen." appears to penalize only
<eos>, whereas a fair baseline should penalize all defined structural tokens at the output layer.
Questions for authors
None
Response (Part1)
We thank you for your thoughtful review and for recognizing the three-phase diagnostic and the low-overhead intervention. Your questions highlight a distinction that our presentation should make consistently: the observed token categories, the attention statistic used for intervention, and the causal interpretation are different claims.
1. A token-agnostic intervention can address the observed concentration, but is not a structural-token detector.
GADS selects columns by their local attention-energy distribution, not by whether their predicted tokens are punctuation. Structural tokens motivate the investigation because they appear in the inspected concentration patterns; the intervention acts on the concentration itself. A structural column can therefore be selected when it meets the energy criterion, while a structural token without that pattern need not be penalized.
This makes the method compatible with the observation, but it does not establish structural specificity. You are correct that a semantic column can also be suppressed, and neither reference-logit preservation nor the algebraic propositions guarantee semantic preservation. Our intended claim is consequently an empirical one about reweighting locally concentrated attention, not that GADS recognizes and removes only harmful structural anchors. The manuscript's stronger structural-specific causal language should be replaced throughout, including the conclusion and figure annotations.
2. Token identity and position are competing or interacting explanations, not alternatives resolved by our plots.
We agree that a position-independent “punctuation commits early” account does not explain why tail punctuation would precede middle punctuation. Our observation concerns a joint configuration: tail anchors are committed while intervening content remains unresolved. The submitted trajectories do not distinguish a positional preference, structural-token predictability, or an interaction between them.
The appropriate analysis compares structural and non-structural predictions within position bins, and compares the same structural category across middle and tail positions, controlling for denoising step and unresolved context. These categories must be defined from predictions available at the measured step, rather than using future token identities to select intervention targets. Until such an analysis is available, we do not rule out a U-shaped positional bias or claim that structural identity alone causes the three-phase pattern.
3. Attention-as-key does not imply confidence at the same position.
We agree with this central point. For one attention row, write its output as oq=∑jaqjvj. A key-column bias changes the attending query's output according to
∂bk∂oq=aqk(vk−oq).Thus, the direct effect is on query representations. After subsequent layers and the output projection, the resulting confidence changes can have either sign; there is no general guarantee that suppressing incoming attention to key position k lowers the output confidence at position k. The plausible route is attention reweighting → changed query representations → changed predictions/confidence rankings → potentially changed commitments. This is a possible pathway, not a demonstrated monotonic mechanism.
The submission reports that timing and continuity change jointly under GADS: the LLaDA-V/LLaVA-Bench gap decreases from 31.6 to 6.7 steps, alongside a topic-switch reduction from 31.7% to 17.9%. These measurements do not establish the missing attention-to-confidence link. A correlation would be useful descriptive evidence, but would also need to control for position, step, token category, and shared prompt context. A stronger diagnostic is a paired forward pass from the identical partially masked state, measuring how the intervention changes confidence and rank at unresolved query positions before allowing the trajectories to diverge.
Accordingly, we do not claim that commitment reordering is the only channel of benefit. Changes to content representations can contribute as well.