acceptodds
Under review as a conference paper at ICLR 2027

A Dominant Self-Conditioning Direction Drives Repetition in Unconditional Continuous Diffusion Language Models

Abstract

Continuous diffusion language models offer an alternative to autoregressive generation, but their generations may suffer from repetition. We find that unconditional generations from ELF, a recent family of continuous diffusion language models, are more repetitive than human text, while Gen-PPL, a common likelihood-based metric, gives lower perplexity to repetitive generations and can conceal this problem while biasing quality evaluation. Our analysis links this behavior to a self-conditioning feedback loop in which clean-embedding predictions are repeatedly carried into subsequent denoising steps, driving representations toward an effectively one-dimensional contractive attractor associated with repetition. Based on this mechanism, we introduce Attractor-Contrast-Escape (ACE), a training-free inference-time intervention that estimates a repetition direction by contrasting denoising paths trapped in repetition with paths relatively free of repetition and subtracts it from the self-conditioning feedback during denoising. Using a direction estimated only once on ELF-B, ACE reduces mean -gram self-repetition rate from to , while retaining competitive results on several text-quality metrics beyond Gen-PPL. The direction remains effective across ELF sizes and inference configurations, and ACE also generalizes to other unconditional self-conditioned continuous diffusion language models. These results identify self-conditioning feedback as a source of repetition in continuous diffusion language models and show that ACE can directly mitigate this repetition during inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
continuous diffusion language models, self-conditioning, repetition in text generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KeMkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies repetition in unconditional generation of self-conditioned continuous diffusion language models such as ELF. the paper argues that self-conditioning feedback forms an attractor associated with repetition, and proposes Attractor Contrast Escape (ACE). ACE collects self-conditioning feedback from generated samples, averages it separately for high- and low-repetition groups, and takes the difference as a repetition direction. During generation, this direction is subtracted from the feedback at each denoising step. ACE reduces repetition while maintaining text quality.

Critical strengths
  • The paper quantitatively shows that ELF generations are highly repetitive and links this behavior to self-conditioning.

  • Also, the paper shows that repetition can be reduced by guiding the self-conditioning feedback along a single direction.

  • The method is training-free and lightweight. The direction is estimated once and transfers across model sizes, inference configurations, and other self-conditioned models (LangFlow, Plaid).

Critical weaknesses
  • Unjustified link between fixed points and repetition. The local analysis only predicts convergence to a fixed point, but the claim that this attractor corresponds to repetition is supported only by empirical correlations observed in a limited set of models. In recursive reasoning models (e.g., TRM, HRM, GRAM, EqR), convergence to a fixed point can instead correspond to reaching the correct solution or better answers. Without a principled explanation for why the attractor should specifically represent repetition, the conclusion risks overgeneralization, and the proposed method may not apply in settings where convergence to a fixed point is desirable.

  • Repetition reduction alone is insufficient evidence. The proposed ACE direction is constructed using REP-4 labeling (high REP-4 samples - Low REP-4 samples) and is explicitly designed to suppress repetition, so an improvement in REP-4 is largely expected by construction. Moreover, the metrics that improve, such as Rep. share, dist-1 and dist-3, are themselves closely tied to repetition, whereas repetition-independent metrics show no improvement (Grammar, Fluency) or even degrade (MAUVE, self-BLEU). Evaluation on downstream tasks would therefore be important to demonstrate that reducing repetition leads to a genuine improvement in model quality rather than merely optimizing repetition-related metrics.

  • Insufficient positioning against prior work. The abstract presents the bias of Gen-PPL toward repetitive text as a finding (“We find...”), but this is already known [1]. Similarly, self-conditioning as fixed-point iteration was studied in [2]. Although [2] is cited in the introduction, section 4.1 introduces the attracting fixed-point model (line 211) without clarifying what is new relative to [2].

[1] Zheng, Kaiwen, et al. "Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Yoo, Jaehoon, et al. "Self-conditioned Flow Map Language Models via Fixed-point Flows." arXiv preprint arXiv:2607.00714 (2026).

Questions for authors
  • Beyond the empirical correlation, is there a theoretical or empirical reason why the self-conditioning attractor should correspond to repetition? Would ACE still be applicable to models in which convergence to a fixed point is desirable?

  • Does reducing repetition with ACE lead to improvements on downstream tasks or conditional generation, rather than only on repetition-related metrics?

  • ACE consistently lowers MAUVE across settings. Could the authors explain why steering moves the generated distribution further from the reference distribution? Could this be because subtracting repetition direction also removes useful information that guides generation toward the data distribution?

  • In Figure 2, the authors scale the self-conditioning input by α\alpha to show that self-conditioning causes repetition. However, a scaled prediction (e.g., α = 0.5) is likely an out-of-distribution input that the model never saw during training. How can the authors distinguish whether the reduction in repetition is caused by the weaker self-conditioning feedback, or simply by feeding out-of-distribution predictions that disrupt generation?

4 comments
AuthorsOpenReview ↗

Response (Part 1 of 2)

Weakness 1 + Question1

Our analysis concerns a repetition-associated self-conditioning attractor in unconditional generation by continuous diffusion language models. The local theory characterizes feedback convergence and modal dominance; the connection to repetitive content is examined empirically. Existing evidence links feedback convergence (Fig. 11), increasing dominance of one feedback mode (Table 4), and a leading-mode projection that tracks REP-4 (Fig. 4a). We strengthened this connection through recovery experiments and direct interventions on the Jacobian-estimated mode.

Regarding the link to repetition, we evaluated perturbation recovery on 128 states per seed across five seeds. We fixed the noisy latent and time and iterated the feedback map analyzed in Fig. 11a. Of the 640 states, 599 reached the relative fixed-point residual tolerance of 10−410^{-4}. We applied one-time Gaussian perturbations to these converged references with relative L2L_2 magnitudes of 0.1, 0.5, and 1.0, then resumed the original feedback updates for 40 iterations. 1,789/1,797 perturbation–state pairs (99.55%) jointly returned to within 1% relative distance of the reference representation and within 1 percentage point of its REP-4. For the strongest perturbation, decoded REP-4 changed from 6.45 ± 0.50% at the reference to 5.74 ± 0.51% after the first resumed update, then recovered to 6.43 ± 0.48% after 40 updates. Thus, the unchanged feedback dynamics restored both the representation and its repetitive output after displacement.

We additionally tested whether the Jacobian-estimated mode influences final repetition. Using 128 paired trajectories per seed across five seeds, with matched initial noise, sampling randomness, and intervention strength, subtracting this mode reduced final REP-4 from 7.52 ± 0.96% to 6.34 ± 0.85%, whereas adding it increased REP-4 to 9.63 ± 1.17%. Joint state–repetition recovery links repetitive outputs to the attracting state, while these opposite intervention effects establish that the identified dynamical mode contributes to repetition beyond a static embedding–output correlation.

Regarding beneficial fixed points, stability and output quality are distinct properties: convergence can sustain either a desirable solution or an undesirable repetitive output. These possibilities are compatible with our interpretation. ACE targets an empirically identified repetition-related direction while retaining the SC feedback channel. Extending ACE to recursive reasoning models would require identifying a corresponding repetitive failure mode and evaluating whether intervention preserves reasoning accuracy. Our current experiments establish its effectiveness in unconditional generation; applicability to those reasoning settings remains future work.



Question4

ELF is trained with a self-conditioning (SC) prediction on 50% of denoising examples and a zero SC input on the remaining 50%. Thus, both SC-on and SC-off input configurations are represented during training. In Figure 2, SC-off produces almost no REP, whereas SC-on produces substantial REP. This contrast links the emergence of REP to introducing SC in a comparison between training-supported feedback configurations, rather than merely to switching to an unseen SC input configuration. Intermediate SC strengths are not explicitly sampled during training, but REP changes monotonically with SC strength. Although distribution shift may contribute at these intermediate strengths, this systematic dose–response, together with the SC-on/off comparison, provides additional evidence that stronger SC feedback promotes REP.

AuthorsOpenReview ↗

Response (Part 2 of 2)

Weakness2 + Question2 + Question3

Our paper investigates repetition in unconditional generation by continuous diffusion language models, analyzes its mechanism, and develops ACE as a targeted intervention. REP-4 provides the calibration signal, while effectiveness is evaluated on independently generated held-out samples with the direction fixed beforehand. This tests whether the representation contrast translates into repetition control beyond the calibration data.

Regarding generation quality and downstream evaluation, our objective is to reduce repetition while retaining other aspects of generation quality. Grammar and Fluency assess whether repetition reduction accompanies linguistic degradation, such as that caused by indiscriminate deletion or token disruption. Distinct-1/3 help detect whether outputs degenerate into trivial text with limited vocabulary or phrase variety, while self-BLEU and MAUVE assess cross-sample diversity and distributional similarity, helping detect collapse toward a small set of similar or fixed sentences. These metrics evaluate ACE in the unconditional-generation setting addressed by the paper. We have not evaluated conditional generation or downstream tasks; investigating SC and ACE in those settings remains future work.

Regarding the MAUVE trade-off, Table 6 shows that its behavior depends on steering strength. At λ=0.25\lambda=0.25, REP-4 decreases from 7.22% to 5.64%, while MAUVE increases from 22.21 to 23.39 and self-BLEU remains essentially unchanged (29.24 versus 29.20). Stronger steering further reduces repetition but lowers MAUVE and increases self-BLEU. These results show that lower-strength steering can reduce repetition while maintaining MAUVE and self-BLEU. ACE is a simple, training-free linear steering method that applies a fixed directional subtraction. We hypothesize that, at excessive strengths, this uniform intervention imposes a common bias across samples, contributing to the observed distribution shift. This suggests a limitation of the simple linear intervention. However, our mechanistic analysis can motivate alternatives to fixed inference-time steering. As an initial extension, continued diffusion training with a penalty on the identified repetition-related direction reduces REP-4 to 5.42%, compared with 6.74% in the matched continued-training control, while MAUVE remains close (23.15 versus 23.65; Table 5). This illustrates how the identified feedback direction can inform further approaches to improving the repetition–quality trade-off.

Overall, ACE addresses a concrete quality defect in unconditional generation, with its effectiveness evaluated on held-out samples alongside linguistic and distributional quality. The strength sweep shows that repetition can be reduced while maintaining MAUVE and self-BLEU at a suitable strength, and characterizes the trade-off under stronger intervention.



Weakness3

We appreciate the reviewer's constructive suggestion. In the abstract, “We find” was intended to introduce our empirical observation that unconditional generations from continuous diffusion language models such as ELF are more repetitive than human text. The following Gen-PPL statement provides the evaluation context: this repetition can be overlooked because of an established limitation of the metric. We will revise the wording to separate these points and make clear that our contribution concerns identifying self-conditioning-related repetition in continuous diffusion language models, analyzing its mechanism, and developing a targeted intervention.

Yoo et al.'s Fixed-point Flows uses self-conditioning's fixed-point iteration to refine denoising predictions and develop few-step flow-map models. Our work examines a complementary aspect: the repetition-related side effects of this recurrent feedback during unconditional generation. Specifically, we characterize a local repetition-associated fixed point and its dominant feedback mode, examine how this mode tracks REP, and use this analysis to motivate ACE's intervention on recurrent self-conditioning feedback (Figs. 4 and 11; Table 4). Our contribution connects the fixed-point dynamics to a specific generation failure mode and its mitigation. We will cite Fixed-point Flows directly when introducing the analysis in Section 4.1 and clarify this distinction there and in Related Work.


Thank you again for your careful review and constructive feedback. We have revised the manuscript and incorporated additional experiments discussed in our response, with changes highlighted in blue. We summarize the updates relevant to your concerns below; section and page references refer to the revised manuscript.

W1 + Q1: Connecting the attractor to repetition. Appendix E.6 (p. 25) now reports perturbation–recovery experiments showing that the original feedback updates restore both the feedback representation and its repetitive output after displacement. Appendix E.3 (p. 22) adds interventions along the independently estimated Jacobian mode: subtracting it reduces final REP-4, whereas adding it increases REP-4. These experiments strengthen the connection between the identified feedback dynamics and repetition beyond observational correlations. Our interpretation concerns a repetition-associated regime in unconditional generation and is compatible with beneficial fixed points in other settings.

W2 + Q2–Q3: Generation quality and the MAUVE trade-off. Section 5.1 (p. 7) now presents the evaluation protocol, parameter selection, and results separately, clarifying how repetition reduction is assessed alongside linguistic quality, diversity, and distributional similarity. The strength sweep in Table 7 (p. 19) shows that weaker steering reduces repetition without lowering MAUVE, while stronger steering introduces a repetition–quality trade-off. Our evaluation remains focused on unconditional generation; conditional generation and downstream-task evaluation remain future work.

W3: Positioning relative to prior work. We revised the abstract to distinguish our observation of repetition in continuous diffusion language models from the known limitations of Gen-PPL. Section 4.1 and Related Work (Appendix H) now explicitly distinguish our contribution from Fixed-point Flows: that work exploits self-conditioning’s fixed-point iteration for few-step modeling, whereas we investigate repetition-related effects of recurrent feedback and use this analysis to motivate targeted intervention.

We have taken your concerns seriously and would be grateful if you could revisit these revisions. We welcome further discussion of any points that remain unclear or insufficiently addressed. If you find that the revisions address your concerns, we would sincerely appreciate your reconsidering your assessment and score. Thank you for the additional time involved in reviewing these changes and for helping us improve the paper.

Thank you again for your careful review and constructive suggestions. Your questions have helped us strengthen the experimental evidence and clarify the scope and interpretation of our findings. We sincerely appreciate the additional time involved in examining these results. Following our previous update, we would like to explain how the evidence addresses your central concerns and invite your feedback on any remaining questions.

Connecting the identified feedback dynamics to repetition. Your concern that fixed-point convergence alone does not explain repetitive content motivated additional controlled experiments. Across 128 paired trajectories per seed and five seeds, subtracting the Jacobian-estimated mode reduces final REP-4 from 7.52% to 6.34%, whereas adding it increases REP-4 to 9.63%, under matched sampling conditions and intervention strength (Appendix E.3, p. 22). Both effects hold across all five seeds. Complementary perturbation–recovery experiments show that unchanged feedback updates restore both the displaced representation and its repetitive output (Appendix E.6, p. 25). Together, these results provide controlled evidence connecting the identified dynamics to repetition beyond the original observational correlations. Our interpretation concerns a repetition-associated regime in unconditional generation and is compatible with beneficial fixed points in other settings.

Assessing repetition reduction alongside generation quality. We understand your concern that reducing REP should represent a meaningful improvement rather than simply optimize a repetition-related metric. ACE’s direction is fixed before evaluation on independently generated held-out samples. We assess repetition together with Grammar and Fluency to detect linguistic degradation, distinct-1/3 to detect trivial outputs with limited vocabulary or phrase variety, and self-BLEU and MAUVE to assess cross-sample diversity and distributional similarity, helping detect collapse toward similar or fixed sentences.

The strength sweep provides a concrete comparison: lower-strength steering reduces REP-4 from 7.22% to 5.64%, while MAUVE increases from 22.21 to 23.39 and self-BLEU remains essentially unchanged (29.24 versus 29.20; Table 7, p. 19). These results demonstrate repetition reduction alongside preserved measured distributional quality and diversity. Stronger steering introduces a trade-off. ACE is a simple, training-free linear intervention, and we hypothesize that excessive directional subtraction imposes a common bias across samples, contributing to the distribution shift.

Regarding conditional generation and downstream evaluation, we appreciate the importance of your suggestion: it would establish whether repetition control also improves task-specific usefulness. Our present contribution is identifying a repetitive failure mode in unconditional continuous diffusion language generation, connecting it to SC-feedback dynamics through controlled experiments, and translating this understanding into targeted mitigation. Understanding why repetition arises and demonstrating how to reduce it while preserving measured aspects of generation quality provide both scientific insight and practical value within this setting. Our conclusions concern these experimentally evaluated findings and benefits.

Conditional generation and downstream evaluation would extend this contribution by testing whether the identified dynamics persist under conditioning and whether repetition reduction improves task performance while preserving prompt relevance and accuracy. We regard this as an important next step. The current study provides a mechanistic and empirical foundation for that investigation, together with demonstrated repetition-control benefits in unconditional generation.

Clarifying the contribution relative to prior work. The revised abstract distinguishes our observation of repetition in continuous diffusion language models from the known limitations of Gen-PPL. Section 4.1 and Appendix H clarify that Fixed-point Flows exploits self-conditioning’s fixed-point iteration for improved modeling, whereas we investigate repetition-related effects of recurrent feedback and use this analysis to motivate targeted intervention.

We would be particularly grateful to know whether these results and clarifications address your concerns, and which specific issues remain unresolved in your assessment. We would welcome the opportunity to clarify any remaining gaps in the evidence or its interpretation. If these additions address your concerns, we would sincerely appreciate your reconsideration of your assessment and score.

Thank you again for your time and for the thoughtful questions that have helped us strengthen the paper.

Reviewer_U25kOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates repetition in unconditional continuous diffusion language models, identifying self-conditioning feedback as a key contributor to repetitive generation. The authors propose a theoretical explanation based on a repetition attractor with a dominant slow-contracting direction and introduce Attractor-Contrast-Escape (ACE), a training-free method that estimates a repetition-related direction from contrasting high- and low-repetition trajectories and subtracts it from self-conditioning feedback during inference. Experiments across multiple models and sampling configurations demonstrate that ACE substantially reduces repetition while largely preserving generation quality, although some quality–diversity trade-offs remain.

Critical strengths

1, This paper proposes an empirical method that achieves significant performance in repetition containing of continuous diffusion language models. It extends the activation steering-like methods to self-conditioning, which provides an interesting view.

2, The core assumption can be partially verified by the correlation between the leading mode projection and the repetition rate.

Critical weaknesses

1, The paper is badly written and hard to follow. The ideas, settings, and results are organized and laid out in a weird way, making the key point of paragraphs hard to understand. Also, some results are lack of enough explanation. For example, the first paragraph in Section 3.1 does not explain Gen PPL enough on why it could be misled by the repetition. Another example is Section 5.1, where experimental setups and results are completely mixed in one long paragraph. Although the author claims the use of AI in writing, I believe the writing quality of the paper has not been well taken care by human.

2, The theoretical explanation are mostly unnecessary, lack of validation, and somehow misleading. The method seems to be a variant of steering, which is widely used in NLP. However, the author depends on a theory with tons of strong assumption to support the method. While many of the assumptions are too strong to me (e.g. the existence of a dominant eigenvector at each denoising step does not necessarily imply the existence of a shared dominant direction across the entire multi-step denoising process), the key proposition comes without direct validation (see my question 1). Hence, I think the theory part is misleading and should be at least partially removed.

Questions for authors

1, Proposition 1 claims that dd and v1v_1 are parallel. This should be a conclusion straightforward to validate. Could you report the cosine similarity between these two vectors for validation?

2, On the theoretical end, I wonder if v1v_1 only contains the repetition semantics. It seems that turning off SC will not only affect the repetition but also degrade the text quality. If v1v_1 also contains other semantics, then dd may not be parallel with v1v_1, which questions the whole theoretical assumption.

3, For the metric, I notice that many repetitions caused by SC take place in the form like repeated words, for example, a a a a a. Hence, I wonder if it is optimal to measure repetition by REP-4, or are there any better metrics?

I am willing to raise my score if my concerns are well addressed.

5 comments

Weakness1

We appreciate this constructive feedback. AI tools assisted with drafting, but we carefully reviewed and revised every sentence, making every effort to ensure clear organization, logical transitions, and coherence across sections. We present the empirical observation and its interpretation separately: Section 3.1 reports that Gen-PPL decreases as repetition increases (p. 3, lines 160–161), while Appendix Table 3 explains that lower Gen-PPL corresponds to greater corpus predictability (p. 13, lines 654–657). We will bring this explanation into the main text of Section 3.1 so readers can more easily follow why repetitive continuations, which are easy for a next-token scorer to predict, can receive lower Gen-PPL without indicating better generation quality.

Section 5.1 currently combines the setup, parameter selection, results, and interpretation in one paragraph, obscuring their sequence. We will separate these into clearly ordered paragraphs, retain the key settings and findings in the main text, and move detailed parameter-selection analysis to the appendix.



Weakness2 + Question1 + Question2

Our theoretical analysis connects local SC feedback dynamics, direction estimation, and intervention. The idealizing assumptions make these dynamics analytically tractable: the symmetric-Jacobian assumption enables an explicit modal decomposition, while frozen drive allows us to characterize convergence toward a fixed endpoint. This analysis motivates measurable dynamical predictions: feedback convergence, increasing modal dominance, and a relationship between the dominant mode and repetition. These are examined through the fixed-point residual in Fig. 11, the spectral gap in Table 4, and the mode–REP relationship in Fig. 4. The theory therefore provides a framework for investigating the mechanism and designing targeted interventions.

Regarding direction recovery, Table 4 reports an absolute cosine similarity of 0.45–0.56 between the ACE direction dd and the estimated local mode v1v_1. Proposition 1 establishes exact parallelism under idealized separability, which makes the groups’ orthogonal mean difference vanish. Practical ACE aggregates contrasts across samples and denoising stages, whereas v1v_1 characterizes a local feedback response. Partial alignment is compatible with this conditional recovery result and empirically connects ACE to the identified dynamical mode. The mode’s relevance is further supported by controlled interventions: using 128 paired trajectories per seed across five seeds, subtracting the estimated mode reduced final REP-4 from 7.52% to 6.34%, whereas adding it increased REP-4 to 9.63%, under matched sampling conditions and intervention strength.

Regarding sharing across denoising stages, we compared modes estimated from the feedback Jacobian averaged over 128 matched trajectories per seed across five seeds. Mean absolute cosine similarities were 0.992, 0.995, and 0.989 for progress 0.6→0.7, 0.7→0.8, and 0.8→0.9, respectively, and 0.957 between 0.6 and 0.9. These measurements directly support an approximately shared mode in the late-denoising regime.

Regarding other semantics, dd can remain parallel to v1v_1 even if v1v_1 carries information beyond repetition. Under Proposition 1’s separability condition, feedback components orthogonal to v1v_1 have equal means in the high- and low-REP groups and therefore cancel in the mean difference, leaving a contrast along v1v_1. Disabling SC removes all feedback components together, so the resulting quality degradation does not establish that this group-mean condition fails.



Question3

We thank the reviewer for raising this question.

To determine whether repetition in our outputs predominantly appears as repeated phrases or stuttering-like word runs, we compared REP-4 with Stutter-4 on the same 1,024 ELF-B Full-SC generations. REP-4 measures phrase-level repetition through recurring 4-grams, while Stutter-4 counts 4-gram windows consisting of four consecutive occurrences of the same word. Both scores use the total number of 4-gram windows as the denominator. Severity is the mean per-sample score; prevalence is the proportion of samples exceeding the corresponding human-reference 95th-percentile threshold, calibrated against the same human reference set.

MetricSeverity (%)Prevalence (%)
REP-47.2288.67
Stutter-40.00070.20

Phrase-level repetition is severe and widespread in these outputs, whereas stuttering-like repetition is rare. This comparison supports using REP-4 as the primary repetition metric for the phenomenon studied here.

Thank you for your response. However, I think my two concerns still hold:

  1. I have noticed the positive alignment between d and v, but a positive alignment of 0.6 is far away from being parallel, unless you analyze the source of systematic errors in measuring. If the theory can be directly refused by the observation, I believe the author should reconsider whether to include it.

  2. I don't see the change of the manuscript. The mentioned parts are still hard to follow.

For the above reasons, I keep my score.

Thank you for your follow-up and for reminding us to update the manuscript. As a first-time ICLR submitter, I was initially unaware that we could upload manuscript revisions during the discussion period. Thank you for pointing this out. We have now uploaded a revised version, with changes highlighted in blue.

Regarding the theoretical claim, we have reconsidered its formulation and replaced Proposition 1 and its exact-recovery proof with Remark 1 on direction alignment under imperfect separability (Appendix B.2, p. 16). The revised statement explicitly allows nonzero orthogonal group-mean differences, with exact parallelism arising as a special case when these differences cancel. Section 4.2 (p. 6) now describes ACE as an empirical trajectory-level contrast and reports its relationship to the local mode as partial alignment. We also added comparisons of feedback-direction estimates across denoising stages within individual trajectories (Appendix E.5), addressing your concern about whether a direction is approximately shared over the denoising process.

Regarding readability, Section 3.1 (p. 3) now directly explains why repetitive continuations can receive lower Gen-PPL: their predictability makes subsequent tokens easier for the scorer to predict. We reorganized Section 5.1 (p. 7) into Evaluation protocol and Results and interpretation, separating settings, parameter selection, findings, and interpretation into shorter paragraphs, with detailed parameter-selection analyses in the appendix.

Regarding your question about REP-4, Appendix A.1 (p. 13) now compares phrase-level repetition with Stutter-4 using the same generated samples, matched scoring denominators, and human-reference calibration. Phrase-level repetition is severe and prevalent, whereas stuttering-like repetition is rare, supporting REP-4 as the primary metric for the repetition observed in our outputs.

Thank you again for helping us improve both the formulation and presentation. We would sincerely appreciate your reviewing these revised passages and reconsidering your assessment and score if you find that the changes address your concerns.

Reviewer_ARv8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies repetitive generation in unconditional continuous diffusion language models and proposes a training-free inference-time intervention named Attractor-Contrast-Escape (ACE). The key idea is that self-conditioning feedback may drive denoising trajectories toward a repetition-related attractor with a dominant direction in the embedding space. ACE estimates this direction by contrasting the mean self-conditioning feedback of high-repetition and low-repetition trajectories, and subtracts the estimated direction from the feedback during subsequent denoising steps.

The paper develops a local fixed-point analysis to motivate the proposed mechanism and evaluates ACE on ELF models of different sizes, as well as LangFlow and Plaid. The experiments show that ACE reduces the mean 4-gram repetition rate (REP-4) on ELF-B from 7.28% to 4.48%, while maintaining similar grammar scores. Additional experiments examine alternative self-conditioning interventions, direction estimators, steering strengths, and inference configurations.

The paper aims to explain the relationship between self-conditioning and repetitive generation while providing a practical method to mitigate repetition without retraining the model.

Critical strengths
  1. The paper addresses a concrete generation-quality problem in continuous diffusion language models and provides empirical evidence that self-conditioning affects repetition.

  2. ACE is conceptually simple and can be applied at inference time without retraining the model.

  3. The authors conduct extensive ablations across alternative self-conditioning interventions, direction estimators, model sizes, and inference configurations.

  4. The paper attempts to connect empirical observations with a mathematical explanation of iterative self-conditioning dynamics, which could provide a useful direction for future research.

Critical weaknesses
  1. The underlying mechanism of self-conditioning remains insufficiently understood. The paper demonstrates that modifying self-conditioning feedback changes repetition, but does not fully explain why the feedback produces this behavior. Disabling or weakening self-conditioning is an informative intervention, but it does not by itself establish the proposed attractor mechanism.

  2. The repetition-related dominant direction is not sufficiently characterized. The estimated ACE direction only partially aligns with the leading local Jacobian mode. The current evidence therefore does not establish that a one-dimensional attractor is the actual mechanism responsible for repetitive generation, rather than a direction statistically associated with repetitive outputs.

  3. The theoretical analysis relies on strong assumptions. The symmetric-Jacobian and frozen-drive assumptions simplify the actual denoising dynamics. In addition, the difference-of-means estimator relies on a separability assumption that is not independently verified.

  4. The algorithmic novelty is limited. Contrastive direction estimation and inference-time directional intervention have precedents in activation steering. Applying this general idea to self-conditioning feedback in continuous diffusion language models is a useful adaptation, but the paper should clarify the substantive methodological contribution beyond this application.

  5. The quality trade-off deserves further examination. On ELF-B, MAUVE decreases from 23.63 to 19.74, while the repetition share remains 74.02%. This indicates that ACE reduces repetition but does not eliminate it, and that the improvement in repetition metrics does not translate uniformly into better generation quality.

Questions for authors
  1. The paper argues that repetitive generation is driven by a dominant self-conditioning attractor. Given the reported absolute cosine similarity of approximately 0.45–0.56 between the ACE direction and the leading local Jacobian mode, what evidence establishes that this direction represents a genuine dominant component of the denoising dynamics rather than a statistical direction correlated with repetitive outputs?

  2. Self-conditioning is widely used in diffusion models, but its underlying mechanism remains incompletely understood. What evidence distinguishes the proposed attractor mechanism from alternative explanations, such as feedback amplification, representation drift, or correlations between self-conditioning embeddings and repetitive outputs?

  3. The difference-of-means estimator relies on the assumption that components orthogonal to the target direction have the same conditional mean in high-repetition and low-repetition groups. Can the authors empirically validate this assumption and quantify how violations affect the estimated direction and ACE's effectiveness?

  4. Could the authors provide controlled perturbation experiments along the estimated direction, its opposite, and orthogonal directions under matched initial noise and sampling conditions? Such experiments would help establish whether the proposed direction specifically controls repetition and whether the observed effect is consistent with the claimed dynamical mechanism.

  5. Given existing work on contrastive activation steering, what is the main methodological novelty of ACE beyond applying directional intervention to self-conditioning feedback? Is the primary contribution intended to be the intervention itself, the discovery of the repetition-related direction, or the proposed dynamical explanation?

5 comments
AuthorsOpenReview ↗

Response (Part 1 of 2)

Weakness1 + Question2

Our attractor interpretation is supported by dynamical evidence beyond SC ablations. Under frozen denoising conditions, the relative fixed-point residual decreases from 0.116 to approximately 9×10−59\times10^{-5} (Fig. 11a), showing that feedback approaches a state that changes negligibly under further updates. The leading local gain remains below one (Fig. 11b), consistent with local stability. Meanwhile, the spectral gap increases from 1.00 to 3.04 (Table 4), indicating increasing dominance of one feedback mode, whose projection tracks REP-4 (Fig. 4a). Together, these observations link local feedback convergence to a repetition-related dominant mode.

To test attraction directly, we evaluated 128 states per seed across five seeds (640 total). We fixed the noisy latent and time and iterated the feedback map in Fig. 11a for up to 200 updates. Of these states, 599 met the relative residual tolerance of 10−410^{-4}, identifying approximate fixed points. We applied one-time Gaussian perturbations with relative L2L_2 magnitudes of 0.1, 0.5, and 1.0 to these references, then resumed the original feedback updates for 40 iterations without further intervention. 1,789/1,797 perturbation–state pairs jointly returned to within 1% relative distance of the reference representation and within 1 percentage point of its REP-4, yielding 99.55 ± 0.32% recovery, including all failed recoveries among these converged references. This return after displacement directly supports local attraction.

Repetition recovered together with the feedback state. Across all 599 converged references, the strongest perturbation changed REP-4 from 6.45 ± 0.50% → 5.74 ± 0.51% → 6.43 ± 0.48% across the reference, first post-perturbation update, and 40th update, with predictions decoded at the fixed denoising stage. Displacement reduced repetition on average, while the original feedback dynamics restored both the representation and its repetitive output.

These convergence and recovery results provide a more specific dynamical account compatible with feedback amplification and representation drift: they identify where feedback stabilizes and how it returns after displacement. Amplification and drift can participate in this mechanism. Deliberately disturbing feedback reduced REP on average, and resuming the original SC updates restored both feedback states and REP toward their reference values, providing intervention evidence beyond a static embedding–output correlation.



Weakness2 + Question1 + Question4

Our dominant-mode interpretation combines dynamical characterization with controlled directional interventions. The one-dimensional behavior in our analysis refers to the increasing dominance of one mode during local SC-feedback convergence. The estimated spectral gap increases from 1.00 to 3.04 as repetition develops (Table 4), while the leading-mode projection tracks REP-4 (Fig. 4a). Mode dominance is therefore measured from the feedback Jacobian, independently of ACE’s construction. Directly subtracting this estimated mode reduces final REP-4, whereas adding it increases REP-4 (see Weakness3 + Question3), connecting the dynamical mode to repetition through intervention. Proposition 1 gives a sufficient condition for the contrastive estimator to recover this mode exactly under separability. The measured absolute cosine similarity of 0.45–0.56 shows partial alignment between ACE’s dd and this independently identified mode v1v_1.

To test direction-specific control, we extended the Table 1 evaluation with matched positive-direction and orthogonal controls, using 1,024 trajectories per seed across five seeds. We compared no intervention, +d+d, −d-d, and an orthogonal direction, matching the ELF-B checkpoint, SDE sampler, initial noise, sampling schedule, and subsequent sampling randomness, with equal steering strength across interventions.

InterventionFinal REP-4 (%)
No intervention7.28 ± 0.35
+d+d13.85 ± 0.45
−d-d (ACE)4.48 ± 0.19
Orthogonal direction7.35 ± 0.78

Reversing the intervention reverses its effect on repetition: +d+d increases REP-4 and −d-d decreases it in every seed, while the orthogonal control remains close to baseline on average. Both signed interventions differ from the orthogonal control, with paired seed-level 95% confidence intervals excluding zero. These results establish direction- and sign-dependent causal control by dd beyond statistical association. Together, Jacobian measurements identify a repetition-related dominant mode, alignment connects ACE to it, and the predicted bidirectional intervention effects support the dominant-mode attractor interpretation.

AuthorsOpenReview ↗

Response (Part 2 of 2)

Weakness3 + Question3

We use these idealizing assumptions to make local SC feedback analytically tractable and derive empirically testable predictions. The symmetric-Jacobian assumption enables explicit modal decomposition; frozen drive permits analysis of a fixed endpoint and modes' relative decay. Proposition 1 gives a sufficient condition for exact dominant-direction recovery by the difference-of-means estimator. We examine these connections through feedback convergence, dominant-mode structure, and repetition (Fig. 11, Table 4, and Fig. 4).

Regarding empirical validation of separability, we analyzed 128 held-out trajectories per seed across five seeds (progress 0.6–0.9). We split the high–low REP mean difference into components along and orthogonal to independently estimated local modes. Across seeds, the orthogonal component averaged 1.5–1.8 times the parallel component’s magnitude. Stage-wise contrasts had mean absolute cosines of 0.49–0.56 with these modes. Thus, empirical contrasts retain partial alignment; Proposition 1 establishes exact parallelism under idealized separability.

Regarding ACE’s effectiveness, we examined the practical consequences of partial alignment by comparing interventions along ACE’s empirical direction dd and the independently estimated local mode v1v_1. We evaluated 128 paired trajectories per seed across five seeds, matching initial noise, sampling randomness, and steering strength.

Feedback interventionREP-4 (%)
Baseline7.52 ± 0.96
−d-d4.44 ± 0.57
−v1- v_16.34 ± 0.85
+v1+ v_19.63 ± 1.17

The partially aligned ACE direction produces stronger REP reduction than the estimated local mode. This is consistent with the distinction between the idealized local analysis and practical direction estimation: v1v_1 describes local feedback convergence, whereas dd captures high–low REP differences across samples and denoising stages. The dominant local convergence mode need not be optimal for reducing final repetition. The local mode nevertheless contributes to REP: subtracting/adding it consistently lowers/raises REP-4 across all five seeds (mean changes: −1.18/+2.10 pp; paired tt 95% CIs: [−1.44, −0.93]/[1.43, 2.78]). These bidirectional effects support the repetition-related dynamical mode, while ACE’s empirical contrast provides stronger repetition control.



Weakness4 + Question5

Our primary contribution is mechanism-motivated repetition control: identifying a repetition-related SC feedback mode, characterizing its local dynamics, and translating this diagnosis into ACE's targeted intervention.

Regarding methodological novelty, representative activation-steering methods modify residual-stream activations; ACE modifies the clean-embedding prediction reused as self-conditioning in subsequent denoising steps, targeting the recurrent feedback channel implicated in repetition. Calibration uses the model's own unconditional trajectories: REP-4 automatically identifies high- and low-repetition groups, requiring no manual labels or curated contrastive prompt pairs. The same framework supports alternative direction estimators, including LDA and logistic regression (Fig. 10). Effectiveness varies with estimator and steering strength; ACE provides a simple, effective implementation.

This combines mechanistic discovery, automatic trajectory-based calibration, and targeted feedback control: theory motivates where to intervene, and empirical comparisons evaluate how to intervene effectively. We will clarify this contribution structure in the Introduction and Related Work.



Weakness5

ACE is a training-free linear intervention with controllable strength; all tested strengths reduce REP (Table 6). At λ=0.25\lambda=0.25, mean REP-4 decreases from 7.22% to 5.64%, while MAUVE increases from 22.21 to 23.39 and Grammar remains comparable. Stronger steering reduces REP further but lowers MAUVE, revealing a strength-dependent repetition–quality trade-off (Fig. 6; Table 6).

For Table 1, we selected λ=0.5\lambda=0.5 for stronger repetition reduction with comparable Grammar, reducing mean REP-4 by 38.5% and repetition share from 90.00% to 74.02%. This improves severity and prevalence, though repetition remains widespread. Lower strength offers an alternative when distributional quality is the priority.

We hypothesize that excessive steering imposes a common bias across samples, shifting other aspects of the output distribution. This potential limitation of fixed linear steering motivates mechanistically informed alternatives. As an initial extension, continued diffusion training with a penalty on the identified repetition-related direction reduces REP-4 to 5.42%, compared with 6.74% in the matched continued-training control, while MAUVE remains close (23.15 versus 23.65; Table 5). The identified direction thus informs mitigation beyond inference-time steering.


Thank you again for your careful review and constructive feedback. We have revised the manuscript and incorporated additional experiments discussed in our response, with changes highlighted in blue. We summarize the updates addressing your concerns below; section and page references refer to the revised manuscript.

W1 + Q2: Evidence for the attractor mechanism. Appendix E.6 (p. 25) now reports perturbation–recovery experiments under fixed denoising conditions. After one-time perturbations, the original feedback updates restore both the feedback representation and its repetitive output. This directly tests local attraction and its connection to repetition, extending the evidence beyond SC ablations and static correlations.

W2 + Q1 + Q4: Characterization and controlled intervention. Appendix E.3 (p. 22) reports signed interventions along the independently estimated Jacobian mode: subtracting it reduces final REP-4, whereas adding it increases REP-4. Appendix E.7 (p. 26) compares positive and negative ACE interventions with an equal-norm orthogonal control under matched initial noise, sampling randomness, and steering strength. These additions test the dynamical mode’s influence on repetition and the direction-specific effects requested in your review.

W3 + Q3: Assumptions and direction recovery. We revised Section 4.2 (p. 6) and Appendix B.2 (p. 16), replacing the exact-recovery proposition with Remark 1 on alignment under imperfect separability. The revised formulation explicitly relates alignment to the groups’ along-mode and orthogonal mean differences, distinguishing idealized exact recovery from the partial alignment measured empirically.

W4 + Q5: Methodological contribution. The Introduction (p. 2) and Related Work (Appendix H) now clarify our contribution as mechanism-motivated repetition control targeting recurrent SC feedback. We distinguish this intervention from residual-stream steering and explain its calibration from unconditional trajectories without manual labels or curated contrastive prompt pairs.

W5: Generation-quality trade-off. Section 5.1 (p. 7) now separates evaluation, parameter selection, and interpretation more clearly. The strength sweep in Table 7 (p. 19) provides the relevant comparison: weaker steering reduces repetition while maintaining measured distributional quality, whereas stronger steering offers greater repetition reduction with a quality trade-off.

We have taken your concerns seriously and would be grateful if you could revisit these revisions. We welcome further discussion of any points that remain unclear or insufficiently addressed. If you find that the revisions address your concerns, we would sincerely appreciate your reconsidering your assessment and score. Thank you for the additional time involved in reviewing these changes and for helping us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.