acceptodds
Under review as a conference paper at ICLR 2027

Drifting and Merging: The Dynamics of LLM Degeneration

Abstract

An LLM can continue generating long after its output has stopped making meaningful progress. What determines whether generation continues to develop the text or becomes generic, incoherent, or repetitive? Since Holtzman et al. (2020), much research has focused on decoding and fine-tuning methods to reduce these problems, but how they develop during generation remains poorly understood. We organize these forms of text degeneration through two autoregressive effects: *drifting* moves the distribution of generated text away from the natural-language distribution, while *merging* makes predictions from different histories increasingly similar. To test this framework quantitatively, we focus on persistent repetition, whose onset is relatively easy to identify. We call this onset time the *collapse time*. We define and measure drifting and merging times, and , from prompts alone and use them to predict collapse time. Across 16 models, five corpora, and five attention temperatures, a regression using these two timescales explains 93.8% of the variance in mean log collapse time on the fitting data. Guided by this analysis, we propose a simple fine-tuning objective that increases . This increases Llama-3.1-8B's geometric-mean collapse time to its original value with zero additional data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, neural text degeneration, mode collapse, long-horizon generation, autoregressive dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Jj5HOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the fundamental dynamics behind neural text degeneration, specifically why LLMs degrade into generic, incoherent, or repetitive loops during long-form generation. The authors formalize this breakdown through two complementary autoregressive processes: drifting, where the generated distribution moves away from the natural-language manifold, and merging, where predictions from distinct histories converge and contract history-dependent constraints. To quantify these dynamics without long rollouts, they define prompt-based timescales. Modeling repetitive collapse as a first-passage event, their joint predictor explains 93.8% of the variance in mean log collapse time across 400 experimental conditions. Furthermore, they introduce a fine-tuning objective that directly enhances information persistence, delaying collapse time in Llama-3.1-8B by up to 5.52× without relying on extra training data.

Critical strengths
  1. The degradation of long text generation is systematically deconstructed into two orthogonal dimensions: Drifting deviating from the natural language manifold/distribution, and Merging where different histories converge/lose discrimination in the prediction state space.
  2. The proposed timescales: ‭τdrift\tau_{\text{drift}}‬ (derived from excess prompt cross-entropy loss) and ‭τmerge\tau_{\text{merge}}‬ (derived from the integrated decay of normalized mutual information/generalized JSD under token perturbation), can be estimated purely from short prompts without requiring expensive thousands-of-tokens rollouts.
  3. Explains why models post-trained for long-form generation (e.g., LongWriter) exhibit significantly delayed collapse under raw continuation compared to base models, despite exhibiting higher prompt loss, which is driven by their substantially longer merging timescales.
  4. Moving beyond passive observation, the authors translate their dynamical insights into a fine-tuning objective that directly maximizes local information persistence. Without any additional training data, this intervention extends Llama-3.1-8B's geometric-mean collapse time by up to 5.52×, outlasting an unmodified 300B model under greedy continuation.
Critical weaknesses
  1. The empirical evidence is considerably narrower than the conceptual framing. While the proposed drifting–merging framework is presented as a general account of LLM degeneration, the quantitative validation focuses almost exclusively on REP-4-based repetitive collapse. The claimed connections to blandness and incoherence are primarily supported through analogy with the toy dynamical system rather than direct empirical measurements.
  2. The experiments largely focus on base models under uninterrupted greedy continuation, leaving open whether the findings transfer to instruction-tuned/reasoning models, sampling-based generation, or realistic long-horizon agent settings.
  3. The results of Fine-tuning prove "delayed repetition", but do not prove "improved generation quality". Broader quality metrics and out-of-domain evaluations of the fine-tuning intervention would therefore substantially strengthen the claims.
Questions for authors
  1. Is τmerge\tau_{\text{merge}} genuinely measuring long-range information persistence, or is it merely a local sensitivity statistic that happens to correlate with another underlying property of the model?
  2. Given the stark contrast between the paper's opening motivation around modern reasoning and agent workflows and its actual reliance on uninterrupted greedy continuation with base models, could the authors bridge this gap by evaluating whether the proposed framework and persistence fine-tuning generalize to instruction-tuned or reasoning models on realistic long-form tasks (e.g., extended multi-step reasoning or constrained generation) under their native prompting and decoding settings rather than testing additional base models?
4 comments

Thank you for your careful review and constructive suggestions. We have submitted a revised manuscript with the changes marked in blue. We respond point by point below. Line numbers refer to the revised manuscript.


1. Empirical support for blandness and incoherence

The empirical evidence is considerably narrower than the conceptual framing. While the proposed drifting–merging framework is presented as a general account of LLM degeneration, the quantitative validation focuses almost exclusively on REP-4-based repetitive collapse. The claimed connections to blandness and incoherence are primarily supported through analogy with the toy dynamical system rather than direct empirical measurements.

We have added two LLM-as-judge metrics, coherence and content progression, to evaluate these forms of degeneration directly. Fine-tuning improves both scores in most evaluated conditions, including all three new domains for Qwen. These results provide empirical evidence of gains beyond repetition: more coherent text and more substantive development, addressing incoherence and blandness, respectively (see Table 3 in lines 466--480 or in below).

Unlike persistent repetition, however, blandness and incoherence do not have equally clear onset criteria. Their assessment depends on the model being evaluated and the scoring rubric. We therefore use these scores as empirical measures of the broader behaviors, rather than treating them as sharply defined collapse events. The quantitative results are summarized below in response to your third concern.


2. Instruction-tuned and reasoning models under native settings

The experiments largely focus on base models under uninterrupted greedy continuation, leaving open whether the findings transfer to instruction-tuned/reasoning models, sampling-based generation, or realistic long-horizon agent settings.

Given the stark contrast between the paper's opening motivation around modern reasoning and agent workflows and its actual reliance on uninterrupted greedy continuation with base models, could the authors bridge this gap by evaluating whether the proposed framework and persistence fine-tuning generalize to instruction-tuned or reasoning models on realistic long-form tasks (e.g., extended multi-step reasoning or constrained generation) under their native prompting and decoding settings rather than testing additional base models?

Thank you for suggesting this direction. We have also evaluated persistence optimization on mathematical reasoning tasks using MATH and DAPO. We observe more diverse reasoning trajectories and higher best-of-nn answer accuracy, suggesting that the benefits extend beyond delaying repetition to improving reasoning performance. We will include these additional results and the full evaluation details in the camera-ready version.


3. Generation quality and new-domain evaluation

The results of Fine-tuning prove "delayed repetition", but do not prove "improved generation quality". Broader quality metrics and out-of-domain evaluations of the fine-tuning intervention would therefore substantially strengthen the claims.

We agree that improved quality needs evidence beyond delayed repetition. We have added three metrics: coherence and content progression, evaluated by an LLM-as-judge, and correlation dimension (CD), a quantitative measure of predictive-state complexity that provides a judge-independent assessment. The results are reproduced below (new results: lines 466–480).

The evaluation covers 150 prompts per model from the five original corpora, plus 60 documents from each of three new domains: CNN/DailyMail, PubMed, and QMSum. Quality improves in most settings. In-domain, coherence and progression scores increase by 13.8% and 19.9% for Qwen, and 9.5% and 16.4% for Llama; CD increases by 50.6% and 32.3%, respectively. All six paired 95% intervals exclude zero. On the new domains, Qwen improves both judge scores, and mean CD increases for both models in every domain by 18.1%–65.3%.

Together, these evaluations provide quantitative evidence that the gains extend beyond avoiding repeated wording to improved coherence, content development, and predictive-state complexity.


4. Does the persistence measurement reflect more than local sensitivity?

Is τmerge\tau_{\mathrm{merge}} genuinely measuring long-range information persistence, or is it merely a local sensitivity statistic that happens to correlate with another underlying property of the model?

Thank you for this important question. We address the longer-range effect and the theoretical basis separately.

First, training Qwen on only the first eight lags also increases persistence at later lags: the response integral rises by 34.2% over lags 8–16 and 21.8% over lags 32–64 (95% intervals: 29.6%–39.5% and 15.6%–29.7%). The 8–16 gain persists after accounting for initial response strength, showing an effect beyond increased local sensitivity.

Second, we have added Proposition 1 (lines 424–435; proof in Appendix K). Under its stated conditions, which allow general response-decay forms, we prove τmerge≤E[T]\tau_{\mathrm{merge}}\leq\mathbb E[T], where TT is collapse time. In other words, τmerge\tau_{\mathrm{merge}} is a lower bound on expected collapse time. Maximizing this lower bound provides a principled training objective, beyond relying on empirical correlation alone.


For convenience, we reproduce Table 3 below. REP-4 reports collapse-time ratios; the other columns report percentage changes in means. Parentheses give 95% paired-bootstrap intervals. For all metrics, larger values indicate better quality of generated texts.

ModelEvaluation domainREP-4 onset improvementΔCDΔCoherenceΔProgression
Qwen3-4BIn-domain5.0× (4.0, 6.0)+50.6% (39.9, 61.8)+13.8% (8.0, 20.0)+19.9% (12.0, 28.8)
Llama-3.1-8BIn-domain5.5× (4.5, 6.7)+32.3% (22.2, 43.1)+9.5% (3.8, 15.7)+16.4% (8.3, 25.4)
Qwen3-4BCNN/DailyMail6.4× (4.6, 8.9)+49.9% (33.0, 69.2)+14.9% (4.6, 26.1)+40.6% (21.0, 64.0)
Qwen3-4BPubMed7.1× (5.2, 9.6)+47.0% (30.4, 66.2)+23.0% (12.3, 35.2)+33.3% (17.3, 53.1)
Qwen3-4BQMSum2.6× (1.8, 3.7)+18.1% (−3.0, 43.5)+29.1% (14.4, 46.1)+47.8% (29.4, 68.3)
Llama-3.1-8BCNN/DailyMail7.2× (5.1, 10.0)+65.3% (43.4, 88.9)+2.4% (−8.4, 13.9)+9.2% (−5.0, 25.5)
Llama-3.1-8BPubMed6.9× (4.7, 10.2)+57.2% (39.7, 77.0)+0.8% (−6.1, 8.3)+6.2% (−2.7, 16.5)
Llama-3.1-8BQMSum3.9× (2.6, 5.7)+31.0% (9.2, 55.4)−4.7% (−15.7, 6.8)+5.7% (−6.5, 19.1)

Thank you again for helping us improve the explanation and evaluation.

Thank you for the thorough rebuttal. The new out-of-domain quality evaluations (Coherence, Progression, CD) and Proposition 1 (τmerge≤E[T]\tau_{\text{merge}} \le \mathbb{E}[T]) convincingly address my concerns. While the MATH/DAPO results remain pending for the camera-ready version, the revisions clearly strengthen the paper. I will maintain my Weak Accept score.

Thank you for the follow-up. We’re glad the additional evaluations and Proposition 1 addressed your concerns.

Reviewer_GnXFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper splits LLM degeneration into drifting and merging. It estimates drifting time from prompt loss and merging time from how long the effect of a single-token replacement lasts. Both quantities depend only on the prompt, and a regression on them tracks repetition collapse time across multiple models, corpora and attention temperatures. The authors then fine-tune two models with an objective that rewards longer merging time, and under greedy raw continuation collapse comes several times later.

Critical strengths

(1) Both quantities can be estimated from prompts without generating long continuations, and together they help predict repetition onset. The two dissociation cases are fairly convincing: moderately lower attention temperature and the displayed LongWriter comparisons both show higher prompt loss with delayed collapse, so merging time does not look like loss in disguise.

(2) The intervention provides useful evidence beyond the predictive fits. Training with the persistence measure delays collapse while prompt loss barely moves, and the LM-only and LM+KL controls improve much less.

Critical weaknesses

(1) The fine-tuning results are not evaluated on a fully independent test set: Appendix G.4 says some evaluation prompts helped select training settings and LoRA strengths. The main quantitative measure of generation degeneration is REP-4, while Appendix A shows repetition and incoherence that it does not capture. Later REP-4 collapse is encouraging, but broader improvements in generation quality remain uncertain.

(2) The main fit in Section 5.1 is in-sample, with jointly fitted corpus offsets. Table 1 provides useful held-out tests and baseline comparisons, though performance varies across temperatures. Appendix G.3 also finds comparable fits with alternative feature transformations and acknowledges that the functional form is not uniquely identified. I find the predictive evidence convincing, but it leaves room for alternative explanations of the relationship.

(3) Section 4.3 measures local persistence over eight token lags, whereas the target outcome concerns much longer continuations. Section 7 acknowledges that the link between these local changes and delayed collapse remains unclear. Uncertainty estimates for the changes in Table 2 would help assess their reliability, and measurements during free generation could strengthen the proposed interpretation.

Questions for authors

See Weaknesses.

3 comments

Thank you for your careful review and constructive suggestions. We have submitted a revised manuscript with the changes marked in blue. We respond point by point below. All line numbers refer to the revised manuscript.


1. Independent evaluation and generation quality

(1) The fine-tuning results are not evaluated on a fully independent test set: Appendix G.4 says some evaluation prompts helped select training settings and LoRA strengths. The main quantitative measure of generation degeneration is REP-4, while Appendix A shows repetition and incoherence that it does not capture. Later REP-4 collapse is encouraging, but broader improvements in generation quality remain uncertain.

We appreciate this distinction. We have extended evaluation to three new domains—CNN/DailyMail, PubMed, and QMSum—with 60 documents each, none used to train the adapters. We reuse the existing trained adapters and apply the same adapter strength for each model across all three new domains, without domain-specific setting selection (lines 1523–1542).

To assess gains beyond REP-4, we have added three metrics: coherence and content progression, evaluated by an LLM-as-judge, and correlation dimension (CD), a quantitative measure of predictive-state complexity. CD provides a judge-independent check beyond repeated wording. The results are reproduced at the end of this response.

The added measurements show quality improvements in most settings. Across the five original domains, coherence and progression scores increase by 13.8% and 19.9% for Qwen, and 9.5% and 16.4% for Llama; CD increases by 50.6% and 32.3%, respectively. All six paired 95% intervals exclude zero (lines 472-480). On the three new domains, Qwen improves both judge scores, while mean CD increases for both models in every domain by 18.1%–65.3%. These results provide evidence of improved generation quality beyond simply delaying a REP-4 collapse.


2. What does the predictive relation establish?

(2) The main fit in Section 5.1 is in-sample, with jointly fitted corpus offsets. Table 1 provides useful held-out tests and baseline comparisons, though performance varies across temperatures. Appendix G.3 also finds comparable fits with alternative feature transformations and acknowledges that the functional form is not uniquely identified. I find the predictive evidence convincing, but it leaves room for alternative explanations of the relationship.

We agree that the predictive evidence does not uniquely identify a dynamical law. We now make this distinction explicit: the contribution is the complementary predictive value of drifting and merging, rather than proof that only one functional form can describe their relationship.

We have also expanded the intuition behind the chosen relation. Drifting controls opportunities to lose language structure, while merging summarizes the accumulated influence of historical constraints. These play different roles in our model, leading to a logarithmic drifting term and a linear merging term. The explanation is in lines 276–305, with assumptions and derivation in lines 1341–1383.

We regard the proposed formula as the most principled of the forms examined: it follows from explicit hypotheses about language-structure loss and the persistence of historical constraints, rather than feature transformations chosen solely for fit. It nevertheless involves simplifying assumptions, and the current data do not clearly distinguish it from the alternatives (lines 1457–1471). Our central contribution is to establish the complementary roles of drifting and merging through predictive tests and targeted fine-tuning, rather than to identify a unique functional form.

3. Local persistence, long-term behavior, and uncertainty

(3) Section 4.3 measures local persistence over eight token lags, whereas the target outcome concerns much longer continuations. Section 7 acknowledges that the link between these local changes and delayed collapse remains unclear. Uncertainty estimates for the changes in Table 2 would help assess their reliability, and measurements during free generation could strengthen the proposed interpretation.

Thank you for these suggestions. We have added three checks linking local persistence to longer-range responses, reliable collapse-time gains, and measurements during generation.

  • Changes extend beyond the training window. Training Qwen only through lag eight increases the response integral by 34.2% over lags 8–16 (95% interval: 29.6%–39.5%) and 21.8% over lags 32–64 (15.6%–29.7%). The 8–16 increase also persists after controlling initial response amplitude (lines 1713–1717 and 1759–1761).

  • Collapse-time gains are supported by bootstrap intervals. The paired 95% intervals are 2.89× (2.44, 3.41) and 4.96× (4.03, 6.03) for Qwen at strengths 0.40 and 0.55, and 5.52× (4.53, 6.74) for Llama at 0.40. All remain well above 1 under prompt resampling (lines 1597–1604).

  • The measurements can also be updated during generation. In an exploratory six-prompt test, updating merging after 32 sampled tokens while keeping the formula fixed gives an observed log-time MSE of 0.592, versus 1.260 for the stated remaining-time baseline. The complete results, uncertainty estimates, and baseline-sensitivity comparison are reported in lines 1606–1668.

Together, these additions show that the intervention affects persistence beyond the directly optimized lags and provide a concrete test of updating the measurements as generation proceeds.


Thank you again for helping us strengthen the evaluation and clarify the scope of the findings.


For convenience, we reproduce Table 3 below. REP-4 reports collapse-time ratios; the other columns report percentage changes in means. Parentheses give 95% paired-bootstrap intervals. For all metrics, larger values indicate better quality of generated texts.

ModelEvaluation domainREP-4 onset improvementΔCDΔCoherenceΔProgression
Qwen3-4BIn-domain5.0× (4.0, 6.0)+50.6% (39.9, 61.8)+13.8% (8.0, 20.0)+19.9% (12.0, 28.8)
Llama-3.1-8BIn-domain5.5× (4.5, 6.7)+32.3% (22.2, 43.1)+9.5% (3.8, 15.7)+16.4% (8.3, 25.4)
Qwen3-4BCNN/DailyMail6.4× (4.6, 8.9)+49.9% (33.0, 69.2)+14.9% (4.6, 26.1)+40.6% (21.0, 64.0)
Qwen3-4BPubMed7.1× (5.2, 9.6)+47.0% (30.4, 66.2)+23.0% (12.3, 35.2)+33.3% (17.3, 53.1)
Qwen3-4BQMSum2.6× (1.8, 3.7)+18.1% (−3.0, 43.5)+29.1% (14.4, 46.1)+47.8% (29.4, 68.3)
Llama-3.1-8BCNN/DailyMail7.2× (5.1, 10.0)+65.3% (43.4, 88.9)+2.4% (−8.4, 13.9)+9.2% (−5.0, 25.5)
Llama-3.1-8BPubMed6.9× (4.7, 10.2)+57.2% (39.7, 77.0)+0.8% (−6.1, 8.3)+6.2% (−2.7, 16.5)
Llama-3.1-8BQMSum3.9× (2.6, 5.7)+31.0% (9.2, 55.4)−4.7% (−15.7, 6.8)+5.7% (−6.5, 19.1)

Thank you for the detailed response. The new evaluation on three held-out domains goes beyond REP-4, which I appreciate. The longer-lag results and bootstrap intervals also answer my questions. I will keep my score.

Reviewer_C4wPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates whether it is possible to identify the start token position of repetition, i.e., collapse time, from the signals coming from the prompt. The proposed signal comprises two sources, drifting time and merging time. The drifting time is derived from the prediction loss, i.e., log-probability, offset by a corpus specific learned constant, indicating a drift from the natural language. The merging time measures how the different histories lead to non-distinguishable state by replacing a token with a model-predicted token and by comparing the prediction. Experiments show that the proposed signal correlated well with the repetition collapse time. The fine-tuning experiment by including the merging time as an additional loss shows better collapse time, showing less repetition.

Critical strengths
  • This work investigate an important topic of hallucination by language models, focusing on repetition. The proposed approach demonstrates that it is possible to identify when repetition will happen by analyzing the prompt part. Fine-tuning experiment also shows that it is possible to discourage repetition by adding the proposed signal as a loss term.
  • The two signals, drifting time and merging time, are well motivated and designed from the information theoretic view point. The drifting time comes from the log-probability loss with learned corpus offset. The merging time comes from the mutual information of the replaced token position and the distribution of the later position.
  • Experiments demonstrate that the proposed signal can fit well to predict the collapse time, and reduce the risk of repetition.
Critical weaknesses
  • Non intuitive combination of two signals: Each signal is well motivated, but it is not clear why the combination yield Equation 4 and 5. In particular, it is not clear why we need to multiply the drifting time with the exp of merging time. At least, an intuitive explanation is necessary.
  • No end-to-end performance is measured: This work focuses only on the collapse time to avoid repetition, and no experimental results are presented whether it could lead to the comparable performance, at least, in the fine-tuning experiment. Only coding experiments are presented in Section 6 indicating some success in avoiding repetition leading to better end-task performance, if my understanding is correct.
  • Differentiating natural repetition: This work tries to avoid repetition by prohibiting repetition at earlier position. However, it cannot differentiate natural repetition which could happen in the real-world text, such as parallel structures.
Questions for authors
  • I'd like to see the more intuitive explanation for Equation 4.
  • It is better to measure the relation with the end-to-end performance in relation to the proposed metric and collapse time. I suspect avoiding repetition may cause unnatural generation, and it is better to evaluate the experimental results quantitatively and qualitatively.
  • At least, I'd expect a discussion regarding the natural repetition.
2 comments

Thank you for your careful review and constructive suggestions. We have submitted a revised manuscript with the changes marked in blue. We respond to your concerns point by point below.


1. Why do drifting and merging enter the model differently?

Non intuitive combination of two signals: Each signal is well motivated, but it is not clear why the combination yield Equation 4 and 5. In particular, it is not clear why we need to multiply the drifting time with the exp of merging time. At least, an intuitive explanation is necessary.

I'd like to see the more intuitive explanation for Equation 4.

We agree that the original manuscript did not explain this combination clearly enough. We have rewritten Section 5 (lines 276–305) to make the intuition explicit.

The two terms describe different aspects of entering repetition: drifting creates opportunities for repetition, while merging determines whether historical constraints still guide generation away from it. Importantly, merging time summarizes the accumulated influence of past constraints. In our model, this gives an exponential dependence on merging time, whereas drifting contributes a power-law dependence. This is why mean log collapse time contains a logarithmic drifting term but a linear merging term, as now explained in lines 302–305.

The detailed derivation and assumptions are in Appendix F (lines 1341–1383). We present this as a motivated model, not a uniquely established law; Appendix G.3 (lines 1457–1471) compares alternative forms.


2. Does delayed repetition improve generation quality?

No end-to-end performance is measured: This work focuses only on the collapse time to avoid repetition, and no experimental results are presented whether it could lead to the comparable performance, at least, in the fine-tuning experiment. Only coding experiments are presented in Section 6 indicating some success in avoiding repetition leading to better end-task performance, if my understanding is correct.

It is better to measure the relation with the end-to-end performance in relation to the proposed metric and collapse time. I suspect avoiding repetition may cause unnatural generation, and it is better to evaluate the experimental results quantitatively and qualitatively.

We agree that avoiding repetition alone does not establish better generation. We have added three metrics: coherence and content progression, evaluated by an LLM-as-judge, and correlation dimension (CD), a quantitative measure of predictive-state complexity. The first two assess whether text remains coherent and develops its content; CD provides a complementary, judge-independent assessment beyond surface repetition. The results are reproduced below (revised manuscript, lines 466–481).

The evaluation covers 150 in-domain prompts per model from the five corpora already included in the original submission, plus 60 documents from each of three new domains: CNN/DailyMail, PubMed, and QMSum (Appendix H, lines 1522–1536). Quality improves in most settings. In-domain, coherence and progression scores rise by 13.8% and 19.9% for Qwen, and 9.5% and 16.4% for Llama; CD rises by 50.6% and 32.3%, respectively. All six paired 95% intervals exclude zero (lines 474–480). On the new domains, Qwen improves both judge scores, and mean CD increases for both models in every domain by 18.1%–65.3% (lines 473–476).

Together, the LLM-as-judge scores and CD measurements provide quantitative evidence of improved generation quality beyond delayed repetition.


3. How do we account for natural repetition?

Differentiating natural repetition: This work tries to avoid repetition by prohibiting repetition at earlier position. However, it cannot differentiate natural repetition which could happen in the real-world text, such as parallel structures.

At least, I'd expect a discussion regarding the natural repetition.

We agree that useful repetition, including parallel structures, should not be treated as degeneration. Our training objective does not penalize repeated tokens or REP-4; it encourages information persistence to help sustain context-dependent generation.

The added correlation-dimension evaluation addresses your concern about relying solely on surface repetition: it measures predictive-state complexity rather than counting repeated phrases. Mean CD increases in every evaluated model–domain setting, while coherence and progression improve in most settings under a judge rubric that explicitly allows useful repetition (lines 470–480). These results support improved semantic richness and content development beyond simply reducing repeated wording.


Thank you again for helping us improve the explanation and evaluation.

For convenience, we reproduce Table 3 below. REP-4 reports collapse-time ratios; the other columns report percentage changes in means. Parentheses give 95% paired-bootstrap intervals. For all metrics, larger values indicate better quality of generated texts.

ModelEvaluation domainREP-4 onset improvementΔCDΔCoherenceΔProgression
Qwen3-4BIn-domain5.0× (4.0, 6.0)+50.6% (39.9, 61.8)+13.8% (8.0, 20.0)+19.9% (12.0, 28.8)
Llama-3.1-8BIn-domain5.5× (4.5, 6.7)+32.3% (22.2, 43.1)+9.5% (3.8, 15.7)+16.4% (8.3, 25.4)
Qwen3-4BCNN/DailyMail6.4× (4.6, 8.9)+49.9% (33.0, 69.2)+14.9% (4.6, 26.1)+40.6% (21.0, 64.0)
Qwen3-4BPubMed7.1× (5.2, 9.6)+47.0% (30.4, 66.2)+23.0% (12.3, 35.2)+33.3% (17.3, 53.1)
Qwen3-4BQMSum2.6× (1.8, 3.7)+18.1% (−3.0, 43.5)+29.1% (14.4, 46.1)+47.8% (29.4, 68.3)
Llama-3.1-8BCNN/DailyMail7.2× (5.1, 10.0)+65.3% (43.4, 88.9)+2.4% (−8.4, 13.9)+9.2% (−5.0, 25.5)
Llama-3.1-8BPubMed6.9× (4.7, 10.2)+57.2% (39.7, 77.0)+0.8% (−6.1, 8.3)+6.2% (−2.7, 16.5)
Llama-3.1-8BQMSum3.9× (2.6, 5.7)+31.0% (9.2, 55.4)−4.7% (−15.7, 6.8)+5.7% (−6.5, 19.1)

Comments on the paper

1 comment

We thank all reviewers and the Area Chair for their precious time, careful reading, and constructive feedback. We have updated the manuscript, with the changes marked in blue. The main additions are:

  • Three metrics beyond REP-4: coherence and content progression, evaluated by an LLM-as-judge, and correlation dimension (CD), a judge-independent measure of predictive-state complexity. These provide evidence of quality improvements beyond delayed repetition.
  • Evaluation on three new domains: CNN/DailyMail, PubMed, and QMSum. The added evaluations show quality gains in most settings and later REP-4 onset across all tested model–domain pairs.
  • A new theoretical result: under the stated conditions, the full persistence measure τmerge\tau_{\mathrm{merge}} is a lower bound on expected collapse time. Maximizing this bound provides a principled motivation for persistence-targeted training beyond empirical correlation.
  • Response measurements beyond the training window: although Qwen is trained only over the first eight lags, its response also increases at later lags, with measurements extending to lag 64. The increase over lags 8–16 remains after controlling for initial response strength.
  • Bootstrap uncertainty estimates: paired prompt-bootstrap intervals quantify the collapse-time gains and longer-lag response changes.
  • A clearer explanation of the prediction formula: we explain why drifting and merging enter differently, producing a logarithmic drifting term and a linear merging term, and expand the derivation in the appendix.

We sincerely apologize for submitting our responses and revised manuscript so late in the discussion period. We understand that this leaves you limited time to consider the changes. If your schedules permit, we would be deeply grateful if you could review our responses and revisions. Thank you again for your time and thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.