- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the fundamental dynamics behind neural text degeneration, specifically why LLMs degrade into generic, incoherent, or repetitive loops during long-form generation. The authors formalize this breakdown through two complementary autoregressive processes: drifting, where the generated distribution moves away from the natural-language manifold, and merging, where predictions from distinct histories converge and contract history-dependent constraints. To quantify these dynamics without long rollouts, they define prompt-based timescales. Modeling repetitive collapse as a first-passage event, their joint predictor explains 93.8% of the variance in mean log collapse time across 400 experimental conditions. Furthermore, they introduce a fine-tuning objective that directly enhances information persistence, delaying collapse time in Llama-3.1-8B by up to 5.52× without relying on extra training data.
Critical strengths
- The degradation of long text generation is systematically deconstructed into two orthogonal dimensions: Drifting deviating from the natural language manifold/distribution, and Merging where different histories converge/lose discrimination in the prediction state space.
- The proposed timescales: (derived from excess prompt cross-entropy loss) and (derived from the integrated decay of normalized mutual information/generalized JSD under token perturbation), can be estimated purely from short prompts without requiring expensive thousands-of-tokens rollouts.
- Explains why models post-trained for long-form generation (e.g., LongWriter) exhibit significantly delayed collapse under raw continuation compared to base models, despite exhibiting higher prompt loss, which is driven by their substantially longer merging timescales.
- Moving beyond passive observation, the authors translate their dynamical insights into a fine-tuning objective that directly maximizes local information persistence. Without any additional training data, this intervention extends Llama-3.1-8B's geometric-mean collapse time by up to 5.52×, outlasting an unmodified 300B model under greedy continuation.
Critical weaknesses
- The empirical evidence is considerably narrower than the conceptual framing. While the proposed drifting–merging framework is presented as a general account of LLM degeneration, the quantitative validation focuses almost exclusively on REP-4-based repetitive collapse. The claimed connections to blandness and incoherence are primarily supported through analogy with the toy dynamical system rather than direct empirical measurements.
- The experiments largely focus on base models under uninterrupted greedy continuation, leaving open whether the findings transfer to instruction-tuned/reasoning models, sampling-based generation, or realistic long-horizon agent settings.
- The results of Fine-tuning prove "delayed repetition", but do not prove "improved generation quality". Broader quality metrics and out-of-domain evaluations of the fine-tuning intervention would therefore substantially strengthen the claims.
Questions for authors
- Is genuinely measuring long-range information persistence, or is it merely a local sensitivity statistic that happens to correlate with another underlying property of the model?
- Given the stark contrast between the paper's opening motivation around modern reasoning and agent workflows and its actual reliance on uninterrupted greedy continuation with base models, could the authors bridge this gap by evaluating whether the proposed framework and persistence fine-tuning generalize to instruction-tuned or reasoning models on realistic long-form tasks (e.g., extended multi-step reasoning or constrained generation) under their native prompting and decoding settings rather than testing additional base models?
4 comments
4. Does the persistence measurement reflect more than local sensitivity?
Is genuinely measuring long-range information persistence, or is it merely a local sensitivity statistic that happens to correlate with another underlying property of the model?
Thank you for this important question. We address the longer-range effect and the theoretical basis separately.
First, training Qwen on only the first eight lags also increases persistence at later lags: the response integral rises by 34.2% over lags 8–16 and 21.8% over lags 32–64 (95% intervals: 29.6%–39.5% and 15.6%–29.7%). The 8–16 gain persists after accounting for initial response strength, showing an effect beyond increased local sensitivity.
Second, we have added Proposition 1 (lines 424–435; proof in Appendix K). Under its stated conditions, which allow general response-decay forms, we prove , where is collapse time. In other words, is a lower bound on expected collapse time. Maximizing this lower bound provides a principled training objective, beyond relying on empirical correlation alone.
For convenience, we reproduce Table 3 below. REP-4 reports collapse-time ratios; the other columns report percentage changes in means. Parentheses give 95% paired-bootstrap intervals. For all metrics, larger values indicate better quality of generated texts.
| Model | Evaluation domain | REP-4 onset improvement | ΔCD | ΔCoherence | ΔProgression |
|---|---|---|---|---|---|
| Qwen3-4B | In-domain | 5.0× (4.0, 6.0) | +50.6% (39.9, 61.8) | +13.8% (8.0, 20.0) | +19.9% (12.0, 28.8) |
| Llama-3.1-8B | In-domain | 5.5× (4.5, 6.7) | +32.3% (22.2, 43.1) | +9.5% (3.8, 15.7) | +16.4% (8.3, 25.4) |
| Qwen3-4B | CNN/DailyMail | 6.4× (4.6, 8.9) | +49.9% (33.0, 69.2) | +14.9% (4.6, 26.1) | +40.6% (21.0, 64.0) |
| Qwen3-4B | PubMed | 7.1× (5.2, 9.6) | +47.0% (30.4, 66.2) | +23.0% (12.3, 35.2) | +33.3% (17.3, 53.1) |
| Qwen3-4B | QMSum | 2.6× (1.8, 3.7) | +18.1% (−3.0, 43.5) | +29.1% (14.4, 46.1) | +47.8% (29.4, 68.3) |
| Llama-3.1-8B | CNN/DailyMail | 7.2× (5.1, 10.0) | +65.3% (43.4, 88.9) | +2.4% (−8.4, 13.9) | +9.2% (−5.0, 25.5) |
| Llama-3.1-8B | PubMed | 6.9× (4.7, 10.2) | +57.2% (39.7, 77.0) | +0.8% (−6.1, 8.3) | +6.2% (−2.7, 16.5) |
| Llama-3.1-8B | QMSum | 3.9× (2.6, 5.7) | +31.0% (9.2, 55.4) | −4.7% (−15.7, 6.8) | +5.7% (−6.5, 19.1) |
Thank you again for helping us improve the explanation and evaluation.
Thank you for the thorough rebuttal. The new out-of-domain quality evaluations (Coherence, Progression, CD) and Proposition 1 () convincingly address my concerns. While the MATH/DAPO results remain pending for the camera-ready version, the revisions clearly strengthen the paper. I will maintain my Weak Accept score.
Thank you for the follow-up. We’re glad the additional evaluations and Proposition 1 addressed your concerns.
Thank you for your careful review and constructive suggestions. We have submitted a revised manuscript with the changes marked in blue. We respond point by point below. Line numbers refer to the revised manuscript.
1. Empirical support for blandness and incoherence
We have added two LLM-as-judge metrics, coherence and content progression, to evaluate these forms of degeneration directly. Fine-tuning improves both scores in most evaluated conditions, including all three new domains for Qwen. These results provide empirical evidence of gains beyond repetition: more coherent text and more substantive development, addressing incoherence and blandness, respectively (see Table 3 in lines 466--480 or in below).
Unlike persistent repetition, however, blandness and incoherence do not have equally clear onset criteria. Their assessment depends on the model being evaluated and the scoring rubric. We therefore use these scores as empirical measures of the broader behaviors, rather than treating them as sharply defined collapse events. The quantitative results are summarized below in response to your third concern.
2. Instruction-tuned and reasoning models under native settings
Thank you for suggesting this direction. We have also evaluated persistence optimization on mathematical reasoning tasks using MATH and DAPO. We observe more diverse reasoning trajectories and higher best-of-n answer accuracy, suggesting that the benefits extend beyond delaying repetition to improving reasoning performance. We will include these additional results and the full evaluation details in the camera-ready version.
3. Generation quality and new-domain evaluation
We agree that improved quality needs evidence beyond delayed repetition. We have added three metrics: coherence and content progression, evaluated by an LLM-as-judge, and correlation dimension (CD), a quantitative measure of predictive-state complexity that provides a judge-independent assessment. The results are reproduced below (new results: lines 466–480).
The evaluation covers 150 prompts per model from the five original corpora, plus 60 documents from each of three new domains: CNN/DailyMail, PubMed, and QMSum. Quality improves in most settings. In-domain, coherence and progression scores increase by 13.8% and 19.9% for Qwen, and 9.5% and 16.4% for Llama; CD increases by 50.6% and 32.3%, respectively. All six paired 95% intervals exclude zero. On the new domains, Qwen improves both judge scores, and mean CD increases for both models in every domain by 18.1%–65.3%.
Together, these evaluations provide quantitative evidence that the gains extend beyond avoiding repeated wording to improved coherence, content development, and predictive-state complexity.