- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies repetition in unconditional generation of self-conditioned continuous diffusion language models such as ELF. the paper argues that self-conditioning feedback forms an attractor associated with repetition, and proposes Attractor Contrast Escape (ACE). ACE collects self-conditioning feedback from generated samples, averages it separately for high- and low-repetition groups, and takes the difference as a repetition direction. During generation, this direction is subtracted from the feedback at each denoising step. ACE reduces repetition while maintaining text quality.
Critical strengths
-
The paper quantitatively shows that ELF generations are highly repetitive and links this behavior to self-conditioning.
-
Also, the paper shows that repetition can be reduced by guiding the self-conditioning feedback along a single direction.
-
The method is training-free and lightweight. The direction is estimated once and transfers across model sizes, inference configurations, and other self-conditioned models (LangFlow, Plaid).
Critical weaknesses
-
Unjustified link between fixed points and repetition. The local analysis only predicts convergence to a fixed point, but the claim that this attractor corresponds to repetition is supported only by empirical correlations observed in a limited set of models. In recursive reasoning models (e.g., TRM, HRM, GRAM, EqR), convergence to a fixed point can instead correspond to reaching the correct solution or better answers. Without a principled explanation for why the attractor should specifically represent repetition, the conclusion risks overgeneralization, and the proposed method may not apply in settings where convergence to a fixed point is desirable.
-
Repetition reduction alone is insufficient evidence. The proposed ACE direction is constructed using REP-4 labeling (high REP-4 samples - Low REP-4 samples) and is explicitly designed to suppress repetition, so an improvement in REP-4 is largely expected by construction. Moreover, the metrics that improve, such as Rep. share, dist-1 and dist-3, are themselves closely tied to repetition, whereas repetition-independent metrics show no improvement (Grammar, Fluency) or even degrade (MAUVE, self-BLEU). Evaluation on downstream tasks would therefore be important to demonstrate that reducing repetition leads to a genuine improvement in model quality rather than merely optimizing repetition-related metrics.
-
Insufficient positioning against prior work. The abstract presents the bias of Gen-PPL toward repetitive text as a finding (“We find...”), but this is already known [1]. Similarly, self-conditioning as fixed-point iteration was studied in [2]. Although [2] is cited in the introduction, section 4.1 introduces the attracting fixed-point model (line 211) without clarifying what is new relative to [2].
[1] Zheng, Kaiwen, et al. "Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Yoo, Jaehoon, et al. "Self-conditioned Flow Map Language Models via Fixed-point Flows." arXiv preprint arXiv:2607.00714 (2026).
Questions for authors
-
Beyond the empirical correlation, is there a theoretical or empirical reason why the self-conditioning attractor should correspond to repetition? Would ACE still be applicable to models in which convergence to a fixed point is desirable?
-
Does reducing repetition with ACE lead to improvements on downstream tasks or conditional generation, rather than only on repetition-related metrics?
-
ACE consistently lowers MAUVE across settings. Could the authors explain why steering moves the generated distribution further from the reference distribution? Could this be because subtracting repetition direction also removes useful information that guides generation toward the data distribution?
-
In Figure 2, the authors scale the self-conditioning input by to show that self-conditioning causes repetition. However, a scaled prediction (e.g., α = 0.5) is likely an out-of-distribution input that the model never saw during training. How can the authors distinguish whether the reduction in repetition is caused by the weaker self-conditioning feedback, or simply by feeding out-of-distribution predictions that disrupt generation?
4 comments
Response (Part 2 of 2)
Weakness2 + Question2 + Question3
Our paper investigates repetition in unconditional generation by continuous diffusion language models, analyzes its mechanism, and develops ACE as a targeted intervention. REP-4 provides the calibration signal, while effectiveness is evaluated on independently generated held-out samples with the direction fixed beforehand. This tests whether the representation contrast translates into repetition control beyond the calibration data.
Regarding generation quality and downstream evaluation, our objective is to reduce repetition while retaining other aspects of generation quality. Grammar and Fluency assess whether repetition reduction accompanies linguistic degradation, such as that caused by indiscriminate deletion or token disruption. Distinct-1/3 help detect whether outputs degenerate into trivial text with limited vocabulary or phrase variety, while self-BLEU and MAUVE assess cross-sample diversity and distributional similarity, helping detect collapse toward a small set of similar or fixed sentences. These metrics evaluate ACE in the unconditional-generation setting addressed by the paper. We have not evaluated conditional generation or downstream tasks; investigating SC and ACE in those settings remains future work.
Regarding the MAUVE trade-off, Table 6 shows that its behavior depends on steering strength. At , REP-4 decreases from 7.22% to 5.64%, while MAUVE increases from 22.21 to 23.39 and self-BLEU remains essentially unchanged (29.24 versus 29.20). Stronger steering further reduces repetition but lowers MAUVE and increases self-BLEU. These results show that lower-strength steering can reduce repetition while maintaining MAUVE and self-BLEU. ACE is a simple, training-free linear steering method that applies a fixed directional subtraction. We hypothesize that, at excessive strengths, this uniform intervention imposes a common bias across samples, contributing to the observed distribution shift. This suggests a limitation of the simple linear intervention. However, our mechanistic analysis can motivate alternatives to fixed inference-time steering. As an initial extension, continued diffusion training with a penalty on the identified repetition-related direction reduces REP-4 to 5.42%, compared with 6.74% in the matched continued-training control, while MAUVE remains close (23.15 versus 23.65; Table 5). This illustrates how the identified feedback direction can inform further approaches to improving the repetition–quality trade-off.
Overall, ACE addresses a concrete quality defect in unconditional generation, with its effectiveness evaluated on held-out samples alongside linguistic and distributional quality. The strength sweep shows that repetition can be reduced while maintaining MAUVE and self-BLEU at a suitable strength, and characterizes the trade-off under stronger intervention.
Weakness3
We appreciate the reviewer's constructive suggestion. In the abstract, “We find” was intended to introduce our empirical observation that unconditional generations from continuous diffusion language models such as ELF are more repetitive than human text. The following Gen-PPL statement provides the evaluation context: this repetition can be overlooked because of an established limitation of the metric. We will revise the wording to separate these points and make clear that our contribution concerns identifying self-conditioning-related repetition in continuous diffusion language models, analyzing its mechanism, and developing a targeted intervention.
Yoo et al.'s Fixed-point Flows uses self-conditioning's fixed-point iteration to refine denoising predictions and develop few-step flow-map models. Our work examines a complementary aspect: the repetition-related side effects of this recurrent feedback during unconditional generation. Specifically, we characterize a local repetition-associated fixed point and its dominant feedback mode, examine how this mode tracks REP, and use this analysis to motivate ACE's intervention on recurrent self-conditioning feedback (Figs. 4 and 11; Table 4). Our contribution connects the fixed-point dynamics to a specific generation failure mode and its mitigation. We will cite Fixed-point Flows directly when introducing the analysis in Section 4.1 and clarify this distinction there and in Related Work.
Thank you again for your careful review and constructive feedback. We have revised the manuscript and incorporated additional experiments discussed in our response, with changes highlighted in blue. We summarize the updates relevant to your concerns below; section and page references refer to the revised manuscript.
W1 + Q1: Connecting the attractor to repetition. Appendix E.6 (p. 25) now reports perturbation–recovery experiments showing that the original feedback updates restore both the feedback representation and its repetitive output after displacement. Appendix E.3 (p. 22) adds interventions along the independently estimated Jacobian mode: subtracting it reduces final REP-4, whereas adding it increases REP-4. These experiments strengthen the connection between the identified feedback dynamics and repetition beyond observational correlations. Our interpretation concerns a repetition-associated regime in unconditional generation and is compatible with beneficial fixed points in other settings.
W2 + Q2–Q3: Generation quality and the MAUVE trade-off. Section 5.1 (p. 7) now presents the evaluation protocol, parameter selection, and results separately, clarifying how repetition reduction is assessed alongside linguistic quality, diversity, and distributional similarity. The strength sweep in Table 7 (p. 19) shows that weaker steering reduces repetition without lowering MAUVE, while stronger steering introduces a repetition–quality trade-off. Our evaluation remains focused on unconditional generation; conditional generation and downstream-task evaluation remain future work.
W3: Positioning relative to prior work. We revised the abstract to distinguish our observation of repetition in continuous diffusion language models from the known limitations of Gen-PPL. Section 4.1 and Related Work (Appendix H) now explicitly distinguish our contribution from Fixed-point Flows: that work exploits self-conditioning’s fixed-point iteration for few-step modeling, whereas we investigate repetition-related effects of recurrent feedback and use this analysis to motivate targeted intervention.
We have taken your concerns seriously and would be grateful if you could revisit these revisions. We welcome further discussion of any points that remain unclear or insufficiently addressed. If you find that the revisions address your concerns, we would sincerely appreciate your reconsidering your assessment and score. Thank you for the additional time involved in reviewing these changes and for helping us improve the paper.
Thank you again for your careful review and constructive suggestions. Your questions have helped us strengthen the experimental evidence and clarify the scope and interpretation of our findings. We sincerely appreciate the additional time involved in examining these results. Following our previous update, we would like to explain how the evidence addresses your central concerns and invite your feedback on any remaining questions.
Connecting the identified feedback dynamics to repetition. Your concern that fixed-point convergence alone does not explain repetitive content motivated additional controlled experiments. Across 128 paired trajectories per seed and five seeds, subtracting the Jacobian-estimated mode reduces final REP-4 from 7.52% to 6.34%, whereas adding it increases REP-4 to 9.63%, under matched sampling conditions and intervention strength (Appendix E.3, p. 22). Both effects hold across all five seeds. Complementary perturbation–recovery experiments show that unchanged feedback updates restore both the displaced representation and its repetitive output (Appendix E.6, p. 25). Together, these results provide controlled evidence connecting the identified dynamics to repetition beyond the original observational correlations. Our interpretation concerns a repetition-associated regime in unconditional generation and is compatible with beneficial fixed points in other settings.
Assessing repetition reduction alongside generation quality. We understand your concern that reducing REP should represent a meaningful improvement rather than simply optimize a repetition-related metric. ACE’s direction is fixed before evaluation on independently generated held-out samples. We assess repetition together with Grammar and Fluency to detect linguistic degradation, distinct-1/3 to detect trivial outputs with limited vocabulary or phrase variety, and self-BLEU and MAUVE to assess cross-sample diversity and distributional similarity, helping detect collapse toward similar or fixed sentences.
The strength sweep provides a concrete comparison: lower-strength steering reduces REP-4 from 7.22% to 5.64%, while MAUVE increases from 22.21 to 23.39 and self-BLEU remains essentially unchanged (29.24 versus 29.20; Table 7, p. 19). These results demonstrate repetition reduction alongside preserved measured distributional quality and diversity. Stronger steering introduces a trade-off. ACE is a simple, training-free linear intervention, and we hypothesize that excessive directional subtraction imposes a common bias across samples, contributing to the distribution shift.
Regarding conditional generation and downstream evaluation, we appreciate the importance of your suggestion: it would establish whether repetition control also improves task-specific usefulness. Our present contribution is identifying a repetitive failure mode in unconditional continuous diffusion language generation, connecting it to SC-feedback dynamics through controlled experiments, and translating this understanding into targeted mitigation. Understanding why repetition arises and demonstrating how to reduce it while preserving measured aspects of generation quality provide both scientific insight and practical value within this setting. Our conclusions concern these experimentally evaluated findings and benefits.
Conditional generation and downstream evaluation would extend this contribution by testing whether the identified dynamics persist under conditioning and whether repetition reduction improves task performance while preserving prompt relevance and accuracy. We regard this as an important next step. The current study provides a mechanistic and empirical foundation for that investigation, together with demonstrated repetition-control benefits in unconditional generation.
Clarifying the contribution relative to prior work. The revised abstract distinguishes our observation of repetition in continuous diffusion language models from the known limitations of Gen-PPL. Section 4.1 and Appendix H clarify that Fixed-point Flows exploits self-conditioning’s fixed-point iteration for improved modeling, whereas we investigate repetition-related effects of recurrent feedback and use this analysis to motivate targeted intervention.
We would be particularly grateful to know whether these results and clarifications address your concerns, and which specific issues remain unresolved in your assessment. We would welcome the opportunity to clarify any remaining gaps in the evidence or its interpretation. If these additions address your concerns, we would sincerely appreciate your reconsideration of your assessment and score.
Thank you again for your time and for the thoughtful questions that have helped us strengthen the paper.
Response (Part 1 of 2)
Weakness 1 + Question1
Our analysis concerns a repetition-associated self-conditioning attractor in unconditional generation by continuous diffusion language models. The local theory characterizes feedback convergence and modal dominance; the connection to repetitive content is examined empirically. Existing evidence links feedback convergence (Fig. 11), increasing dominance of one feedback mode (Table 4), and a leading-mode projection that tracks REP-4 (Fig. 4a). We strengthened this connection through recovery experiments and direct interventions on the Jacobian-estimated mode.
Regarding the link to repetition, we evaluated perturbation recovery on 128 states per seed across five seeds. We fixed the noisy latent and time and iterated the feedback map analyzed in Fig. 11a. Of the 640 states, 599 reached the relative fixed-point residual tolerance of 10−4. We applied one-time Gaussian perturbations to these converged references with relative L2 magnitudes of 0.1, 0.5, and 1.0, then resumed the original feedback updates for 40 iterations. 1,789/1,797 perturbation–state pairs (99.55%) jointly returned to within 1% relative distance of the reference representation and within 1 percentage point of its REP-4. For the strongest perturbation, decoded REP-4 changed from 6.45 ± 0.50% at the reference to 5.74 ± 0.51% after the first resumed update, then recovered to 6.43 ± 0.48% after 40 updates. Thus, the unchanged feedback dynamics restored both the representation and its repetitive output after displacement.
We additionally tested whether the Jacobian-estimated mode influences final repetition. Using 128 paired trajectories per seed across five seeds, with matched initial noise, sampling randomness, and intervention strength, subtracting this mode reduced final REP-4 from 7.52 ± 0.96% to 6.34 ± 0.85%, whereas adding it increased REP-4 to 9.63 ± 1.17%. Joint state–repetition recovery links repetitive outputs to the attracting state, while these opposite intervention effects establish that the identified dynamical mode contributes to repetition beyond a static embedding–output correlation.
Regarding beneficial fixed points, stability and output quality are distinct properties: convergence can sustain either a desirable solution or an undesirable repetitive output. These possibilities are compatible with our interpretation. ACE targets an empirically identified repetition-related direction while retaining the SC feedback channel. Extending ACE to recursive reasoning models would require identifying a corresponding repetitive failure mode and evaluating whether intervention preserves reasoning accuracy. Our current experiments establish its effectiveness in unconditional generation; applicability to those reasoning settings remains future work.
Question4
ELF is trained with a self-conditioning (SC) prediction on 50% of denoising examples and a zero SC input on the remaining 50%. Thus, both SC-on and SC-off input configurations are represented during training. In Figure 2, SC-off produces almost no REP, whereas SC-on produces substantial REP. This contrast links the emergence of REP to introducing SC in a comparison between training-supported feedback configurations, rather than merely to switching to an unseen SC input configuration. Intermediate SC strengths are not explicitly sampled during training, but REP changes monotonically with SC strength. Although distribution shift may contribute at these intermediate strengths, this systematic dose–response, together with the SC-on/off comparison, provides additional evidence that stronger SC feedback promotes REP.