- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies "repetition collapse" in diffusion language models applied to digit-wise time-series forecasting, evaluating LLaDA-8B and Dream-7B on the Monash benchmark. The authors identify a recurring attention geometry that combines shared anchor stripes and shifted-predecessor feedback. They derive a reduced replicator dynamics for anchor mass whose finite-depth concentration is governed by cumulative gain , and they obtain a cross-position homogenization corollary. They validate the mechanism with two training-free interventions, quarter-rank SVD truncation on attention and history-sparsified classifier-free guidance (HSCFG), both of which reduce measured gain by about 3% and substantially cut repetition while keeping NMAE comparable to vanilla. A Dream replication in the appendix reproduces the main qualitative findings.
Critical strengths
- The work provides a clean, memorable empirical phenomenon. Look-ahead repetition climbs from to while draft NMAE flatlines near , and the slotwise cascade is quantified crisply.
- The attention decomposition is a non-obvious abstraction that productively ties the observed geometry to classical rank-collapse and concentration results.
- The spectral-plateau argument genuinely justifies why low-rank SVD truncation targets the predecessor channel rather than the stripe, moving the intervention from a generic regularizer to a mechanism-targeted probe.
- The two-pronged intervention design, with SVD acting within a forward pass and HSCFG acting between diffusion steps, is methodologically careful and the matched-pair protocol guards against obvious confounders.
Critical weaknesses
- The quantitative link from theory to intervention is thin. The gain reductions are only about 3% with no reported variance, while HSCFG cuts value-level repetition from to , so the gain pathway may not be doing most of the work.
- The anchor-detection pipeline has several hyperparameters with no sensitivity analysis, so it is unclear how robust the measured gain curves and correlations are to those choices.
- Causality of the slotwise cascade is not directly tested, is correctly described as not establishing causation, but no ablation with a randomized or round-robin scheduler is run to separate scheduler effects from the proposed feedback loop.
- The forecasting-accuracy claim is operationally modest. NMAE sits near across all conditions on both models, which is roughly persistence-baseline performance, and SVD/HSCFG slightly degrade NMAE on Dream.
- Related DLM-repetition accounts (Zhao et al. 2026 on context tokens as anchors, Rulli et al. 2026 on attention sinks in DLMs) are cited but not quantitatively compared, which weakens the novelty claim given overlapping terminology.
Questions for authors
- What are the standard errors or paired per-rollout difference distributions on the and cumulative-gain reductions and on the repeating-vs-non-repeating gap?
- How sensitive are the gain measurements and gain-to-anchor correlations to the anchor-detection hyperparameters?
- If you swap the scheduler for a randomized or round-robin policy and hold everything else fixed, does cumulative gain still rise and does repetition still emerge?
- Does applying SVD only to shallow layers (where the stripe-plus-predecessor fit is cleanest) have a different effect than applying it uniformly or only to deep layers?
- Are your "anchor" positions the same objects as the attention sinks and is your account complementary to or competing with Zhao et al. 2026?
3 comments
Q2: Sensitivity to anchor-detection hyperparameters
We further assessed the robustness of these measurements by varying the anchor-detection threshold, increasing the required excess above background from the default 20% to 40%, 70%, and 100%. The results are shown in Table 2.
Table 2. Anchor-detection threshold sensitivity.
| Threshold above background | Repeating gain | Non-repeating gain | Vanilla gain | SVD gain | HSCFG gain | Pearson | Spearman |
|---|---|---|---|---|---|---|---|
| 40% | 51.014 | 49.486 | 50.815 | 49.506 | 49.900 | 0.8343 | 0.8787 |
| 70% | 57.907 | 55.720 | 57.837 | 56.093 | 56.449 | 0.819 | 0.824 |
| 100% | 67.215 | 63.185 | 67.113 | 65.492 | 66.001 | 0.789 | 0.837 |
The threshold affects the absolute scale of measured gain, but all tested settings preserve the comparisons central to our analysis: repeating rollouts have higher gain than non-repeating rollouts, and both SVD and HSCFG reduce gain relative to their corresponding vanilla baseline. Gain-to-anchor correlations also remain strongly positive, with Pearson ranging from 0.789 to 0.8343 and Spearman from 0.824 to 0.8787.
Thus, the relative gain comparisons and gain–anchor association supporting our mechanism remain robust across the tested thresholds, including substantially stricter anchor criteria. We will include these sensitivity results in the revision.
Q3: Randomized and round-robin schedulers
To examine the contribution of scheduling to the slotwise cascade, we evaluated both round-robin and randomized schedulers while holding the remaining settings fixed. Table 3 reports the results.
Table 3. Scheduler ablations with the remaining settings fixed.
| Scheduler | Cumulative gain | NMAE | ||||
|---|---|---|---|---|---|---|
| Round-robin | 40.5833 | 0.4276 | 0.1228 | 0.1050 | 0.2180 | 1.372 |
| Randomized | 40.389 | 0.41867 | 0.11643 | 0.1038 | 0.2199 | 1.440 |
Repetition persists under both schedulers, but is substantially reduced. Cumulative gain also decreases from the vanilla value of 42.298 to 40.5833 and 40.389, respectively. Thus, disrupting the original reveal order weakens both cumulative reinforcement and repetition, consistent with our model's prediction.
These results support the reviewer's observation that scheduling contributes to the slotwise cascade. In particular, the original preference for revealing hundreds digits first facilitates early shared predictions that can reinforce subsequent repetition. The scheduler therefore affects the strength of the proposed feedback process, and the feedback–repetition relationship remains evident under both alternative policies.
However, using these schedulers to suppress repetition comes with a substantial forecasting cost: NMAE reaches 1.372 for round-robin and 1.440 for randomized scheduling, both exceeding the persistence baseline of 1. These scheduler changes therefore do not provide the same balance between repetition suppression and forecasting accuracy as our interventions. We will include these ablations in the revision.
Q4: SVD placement across layers
We also evaluated deep-only and uniform SVD placement against our original shallow-layer setting. For the uniform setting, we intervened on one head per layer to keep the total number of modified head–layer pairs approximately matched to the original intervention budget. Table 4 summarizes the comparison.
Table 4. Effect of SVD layer placement.
| SVD placement | Cumulative gain | Value-level repetition |
|---|---|---|
| Shallow layers (original) | 40.936 | 0.701 |
| Deep layers only | 41.273 | 0.837 |
| Uniform, one head per layer | 42.306 | 0.992 |
Deep-only SVD provides a weaker reduction in repetition, with , , and . Uniform placement produces little change in repetition, and its cumulative gain remains close to vanilla (42.306 versus 42.298).
These results support our choice to apply SVD in the shallow layers, where the stripe-plus-predecessor model provides the best fit and the intervention is most effective.
Q5: Anchors, attention sinks, and Zhao et al. (2026)
Finally, regarding attention sinks, our anchors differ from the conventional picture of a few dominant sink tokens. In our forecasting rollouts, these stripes appear on meaningful content tokens, rather than being tied to BOS or the first sequence position, and are observed across all layers. Moreover, attention is distributed across multiple positions: in many attention matrices, roughly one sixth of the key positions form elevated columns.
Zhao et al. (2026) provide related empirical background. Their setting concerns cache-induced repetition in diffusion multimodal models, which differs substantially from our uncached forecasting setting. Their information-flow account is primarily an empirical interpretation based on attention patterns and entropy trajectories. Their anchors describe nearby context tokens that aggregate information, without a formal characterization of their role in repetition dynamics or a theoretical derivation of how that role produces repetition. Their observations therefore provide contextual background instead of a competing explanation of the mechanism studied here.
We thank the reviewer for the careful reading of our paper and the thoughtful, constructive feedback.
Weakness 1: Quantitative link between gain and HSCFG
We agree that the original comparison combines two effects of HSCFG: directly changing the selected output tokens, and allowing these changes to influence subsequent forward passes through the updated canvas. The direct effect can reduce repetition without changing the cumulative gain measured in the current forward pass.
To distinguish these effects, we conducted a new ablation. At each step, we record the HSCFG-selected tokens for output evaluation, but commit the unguided selections to the actual canvas used by subsequent forward passes. When guidance changes the selected position, we likewise record the guided position and token while retaining the unguided canvas update. Only after generation is complete do we replace the tokens at the recorded positions with their recorded HSCFG selections for output evaluation. This preserves the direct output effect while preventing the guided changes from influencing subsequent model inputs.
Table 1 reports the direct-output-only ablation. Its cumulative gain remains close to vanilla (42.298), while value-level repetition decreases from vanilla's 1.000 to 0.401. Full HSCFG achieves a cumulative gain of 41.016 and further reduces repetition to 0.132 when the guided changes also enter subsequent forward passes.
Table 1. HSCFG ablation separating direct output changes from their subsequent feedback.
Your observation is therefore correct: the reduction to 0.132 reflects the combined effects of direct token changes and their feedback into subsequent forward passes.
Q1: Uncertainty in cumulative-gain differences
The standard errors of the paired per-rollout gain differences are 0.305 for quarter-rank SVD, 0.297 for HSCFG, and 0.338 for the repeating-versus-non-repeating comparison, all reported in the original cumulative-gain units. We calculated approximate 95% confidence intervals and expressed them as percentages of the corresponding reference means. The resulting intervals are [1.81%, 4.63%] for the SVD gain reduction, [1.65%, 4.41%] for the HSCFG gain reduction, and [2.93%, 6.21%] for the higher gain in repeating versus non-repeating rollouts. All three intervals lie entirely above zero, supporting both the reduction in cumulative gain under the interventions and the higher gain associated with repeating rollouts.