acceptodds
Under review as a conference paper at ICLR 2027

Perplexity Can Reverse the Verdict on Model Collapse: A Frequency-Band Decomposition

Abstract

Recursive training on model-generated data is reported to erode the rare tail of a model's distribution, yet it is almost always evaluated with held-out perplexity. Perplexity weights each token by how often it occurs, so rare tokens barely count: on WikiText-2 the rarest band of training tokens carries 1.3% of the validation mass and the commonest 23.7%. We decompose held-out loss by each token's training frequency, an exact identity that needs no additional evaluation data. In five of ten recursive-training experiments, the aggregate and the rare bands give opposite verdicts on the same intervention, raising the sampling temperature. A conclusion drawn from perplexity can therefore be reversed by the numbers it already contains. A pre-registered forced choice that tests rare-token knowledge directly sides with the rare bands wherever the generator is not already well fit, and separates the arms by 1.95 accuracy points on checkpoints where three standard benchmarks find nothing. The mismatch cannot be outgrown. Under a Zipf corpus we prove that the validation mass on the tokens recursive training endangers decays as , the same exponent that governs how fast the tail is lost, so a larger corpus degrades the measurement as fast as it improves the process. The disagreement survives the band definition, a measured seed floor, a four-times-larger corpus, five further interventions and two controls for alternative explanations. We also prove that error stays bounded under unbiased accumulate-subsample training, resolving a case left open by Dey & Donoho (2024). We recommend reporting the band decomposition alongside perplexity. Code and results: https://anonymous.4open.science/r/band-resolved-collapse-1319.

Primary area:
learning theory
Keywords:
model collapse, recursive training, synthetic data, model-generated data, held-out perplexity, Zipf distribution, tail degradation, frequency-band decomposition, convergence bounds, language modeling, WikiText-103, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_bsb1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper works on one limitation of perplexity that underscore rare tokens. This will introduce judge damage to the rare tail texts. In a Zipf memorization model, the authors prove that error stays bounded under unbiased subsample training, and the validation weight on the rare items recursive training endangers shrinks as the corpus grows. The authors proposes splitting held-out loss by each token’s training frequency and shows sampling at τ = 1.15 always helps rare tokens in the experiment.

Critical strengths

The paper is clean and correct in its theory and rich in its experiments. The authors several controls about how temperature tau can influence loss only the token at different band to rule out alternative explanations.

Critical weaknesses

This paper's central claim is about perplexity can underweight rare tokens. A decomposition of loss by token rarity can expose it and authors demonstrate using sampling temperature (τ = 1.15 vs. 1.0) as an instrument. However, the whole section 3 containing the theoretical proofs didn't contribute too much to this main theme. The section 3 is working under one simplified memorization model (different from the LLM in experiment). Lemma 1, 2 and the Theorem 1 just says missing mass in the memorization model stays bounded under unbiased accumulate-subsample. This result is unrelated to perplexity and is neither used nor tested by the experiments. Propositions 1 and 2 are only comparatively elementary relation to perplexity. They describe how held-out loss weights rare tokens but assume the change of the rare token loss (as the most interesting and important part) was given. More broadly, the paper attempts to cover too many loosely related results or aspects within the page limit. Alongside its central measurement argument, it includes Theorem 1, a mean-field analysis, a relation between generator bias and collapse rate, a causality test, decoder–workflow interactions, and several mitigation arms. As a result, the main text is extremely compact, and I have to read 26 appendix to evaluate the main claims. This dilutes the paper’s focus and makes the core argument hard to follow from the main text alone.

Questions for authors
  1. Lots of presentations errors appeared throughout the paper, together with the dense context of section 3 and 4, making this manuscript hard to follow. Here are some of the examples:

"|a| ≤ 0.006" in line 224 is not defined in main text.

The collapse exponent for Table 1’s “aggregate” column is never defined until in the appendix.

"Bands" used in introduction and section 3, far away earlier than they’re defined in section 4.

The most important Abstract and Introduction sections are full of undefined parameter (T, beta, p_i and c) and concept that was used in section 3.

The very important verdict column is not defined in table 1 and table 2.

There are many remining incorrect pointing or symbol mismatch. I suggest the authors to read careful the whole manuscript and fix the presentation errors.

  1. Using the definition in Table 2 (final perplexity, with a ‘silent’ category), would the five disagreements in Table 1 can be classified as silent rather than opposite? Can the author provide an explanation why silent is possible at Table 2 but not enabled in Table 1 by definition.

  2. Table 1 is ordered by the overfit ratio. It is defined as generation-0 perplexity over “the best achievable for that model and corpus.” How was the best achievable perplexity determined?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer bsb1

Reply to Reviewer bsb1

Thank you for the careful reading and the specific list of problems. Every item is fixed in the revision, and your Q2 led to the largest change in the paper.

W1, Section 3 and the main theme. We agree. Theorem 1 answers a separate question (bounded error under unbiased accumulate-subsample training, left open by Dey and Donoho, 2024). The measurement does not use it, so it is now in Appendix A.1 with its lemmas. Propositions 1 and 2 are statements about held-out loss, and both are measured:

  • Proposition 1 (the rate): measured over a 42x range of WikiText-103 budgets.
  • Proposition 2 (the noise cost): measured from 11 seed-replicate pairs.

You are right that they take the change in rare-token loss as given. Section 3.2 now states how the pieces connect. Proposition 1 says the weight on at-risk tokens falls as corpora grow. Proposition 2(c) says a head cost of wR/wHw_R/w_H times the tail gain flips the aggregate's sign, a bias that more data or more seeds do not remove. Section 5.1 measures how often that happens. We did not have time to add a derivation of the band profile itself, and we do not claim one.

W2, too many results. We agree. The main text now covers one argument: the decomposition (Section 3), the setup (Section 4), the measurement under one verdict rule (Section 5.1), five further interventions (5.2), and which reading tracks knowledge (5.3). The convergence theorem, mean-field analysis, bias predictor, causality test and decoder-workflow interaction appear only in the appendices. The main text no longer relies on any of them. It is 8 pages.

Q1, presentation errors.

  • ∣a∣≤0.006|a|\le 0.006 moved with Theorem 1 to Appendix A.1, where aa is defined as the exponent of a power law fitted to excess error.
  • The collapse exponent is defined in Section 3 ("Definitions"), before Table 1.
  • Bands are introduced in plain words in the introduction and defined formally at the start of Section 3, before any use.
  • The abstract contains no symbols. TT, β\beta, pip_i and cc appear only after their definitions in Section 3.1.
  • The verdict rule is defined once in Section 3, and both table captions point to it (see Q2).
  • Three symbol clashes removed: the tilt strength is now λ\lambda (it was kk, the synthetic influx rate); the pool count is κn\kappa_n (it was cnc_n, beside the exponent cc); the region mass in Proposition 2 is wRw_R. We also checked every cross-reference against its target; the build reports none undefined.

Q2, silent versus opposite. Yes. Under Table 2's rule, four of the five would be silent, and thank you for catching the inconsistency. Table 1 scored the aggregate by the collapse exponent; Table 2 used terminal perplexity with a "silent" category. The revision applies one rule to every table: an aggregate statistic agrees with the rare bands, opposes them, or is silent, judged against the seed floor of the same model family. We ran second seeds for the families that had none (Table 10, Appendix P.3). We report three aggregate statistics:

aggregate statistic (five disagreeing conditions)oppositesilentagree
collapse exponent500
terminal perplexity140
mass-weighted loss ∑bwbA(b)\sum_b w_b A(b)410

The five conditions at overfit ratio 1.22 to 1.84 agree under all three. The rare-band advantage clears its floor in all ten. We treat silence and reversal as the same failure, since in both cases perplexity does not register a rare-band change that the decomposition shows. Table 2's verification filter is an example of the silent case, here on a mitigation. The abstract now describes the five as conditions where perplexity "gets worse or stays within run-to-run noise".

Q3, "best achievable" perplexity. The wording overstated it. The ratio is generation-0 validation perplexity divided by the lowest generation-0 validation perplexity among all recipes we ran for that model and corpus, from the per-model sweeps of Appendix C. Ratio 1.00 therefore marks the best recipe we found. For OPT-125m the reference is the two-epoch recipe on 36,000 blocks (35.64), which uses 2.4x more of WikiText-2 than the 15,000-block recipes. Section 4 and the Table 1 caption now state this. The ratios are computed from the result files rather than typed, and they reproduce the printed values.

Reviewer_LUKQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper observes that held-out perplexity weights each token by its frequency, so the rare tail that model-collapse studies care about carries little of the loss, and proposes reporting held-out NLL decomposed by training-frequency band. In recursive-training experiments on OPT-125m, Pythia and SmolLM3-3B fine-tuned on WikiText-2, raising the sampling temperature to 1.15 improves rare-band NLL in 10/10 conditions while the aggregate disagrees in sign in 5/10; a count-matched forced-choice probe, a Zipf-rate proposition and a martingale bound on missing mass under unbiased accumulate-subsample training are offered in support.

Critical strengths

Critical Strengths The count-matched forced-choice probe is a sensible calibration-immune complement to likelihood, and the null result on three standard benchmarks is reported rather than hidden.

Critical weaknesses

Critical Weaknesses

  1. Frequency-stratified loss is already used by Magnusson et al. (2024), Pinto et al. (2024), and Meister & Cotterell (2021), while tail loss under synthetic training is studied by Dohmatob et al. (2024b). Equation (3) is an identity. The remaining claimed novelty is a short Zipf integration in Proposition 1 and the observation that its exponent matches Theorem 1, although both results arise from the same sum.

  2. All five disagreeing conditions have overfit ratios between 1.00 and 1.06. In this regime, an evaluation-time unigram tilt reproduces 83--93% of the rare-band advantage (Table 4), and the paper acknowledges that calibration cannot be ruled out. The calibration-immune probe disagrees with the aggregate only in two OPT-125m conditions, by +0.37+0.37 and +0.59+0.59 accuracy points. In the best-fit 3B condition, it instead agrees with the aggregate (−0.34-0.34 points). The title therefore overstates evidence that is marginal and limited to the 125M scale.

Questions for authors
  1. For each of the five disagreeing conditions, what are AresidA_{\mathrm{resid}} and the forced-choice gain after removing the best unigram tilt? Does any reversal remain?

  2. How is the title claim reconciled with SmolLM3-3B at one epoch, where the calibration-immune probe favors τ=1.0\tau=1.0?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer LUKQ

Reply to Reviewer LUKQ

Thank you for a precise review. Your Q1 is the right test, and the revision answers it with new runs.

W1, novelty. We agree that frequency-stratified loss is not new. The submission cited the same three papers and called Eq. 3 an identity. Our claim is narrower and different. Prior work shows the aggregate is insensitive to rare tokens on a fixed model or during pretraining. We show that when comparing interventions it can give the opposite sign to the rare bands. This is not a generic property of a weighted average: on Pythia-410m's public pretraining checkpoints the two readings agree in all 4 comparisons (Appendix P.2). We then test which reading tracks rare-token knowledge (Section 5.3). Dohmatob et al. (2024b) model tail loss as a change of scaling law; they do not study how the evaluation weights the tail. On Proposition 1 we accept the point: it is an elementary Zipf calculation, and the revision presents it as a rate, measured over a 42x range of corpus sizes (Section 3.1), not as a result that coincides with Theorem 1. The theorem itself is now in Appendix A.1.

W2 and Q1, the calibration residual and the probe after the tilt. The new Table 3 covers every condition where an aggregate statistic disagrees with the rare bands. The two conditions that lacked a tilt sweep or probe were re-run with both. Each condition was re-run as a pair on one software stack, with second seeds where a family had no seed floor. The runs and six predictions were registered before launch. We report two tilt fits:

  • Rare-matched: your criterion, the tilt that best reproduces ArareA_{\text{rare}}.
  • Whole-profile: a stricter one, the tilt that best reproduces A(b)A(b) over bands 1 to 8, since a calibration account must also explain the common bands.

The probe is immune to the tilt in practice: across the sweep, it moves the control's accuracy by at most 0.006 points. So the forced-choice gain after removing the best tilt equals the gain shown. Rare-band seed floor for OPT-125m: 0.022 nats.

conditionArareA_{\text{rare}}rare-matched sharewhole-profile residualprobe, pp [95% CI]
SmolLM3-3B, 1 ep+0.11899%+0.001−0.34 [−0.49, −0.20]
same, second seed+0.151110%−0.014−0.26 [−0.42, −0.11]
Pythia-1b, 1 ep+0.20090%−0.027−0.12 [−0.28, +0.03]
same, second seed+0.20190%−0.028−0.01 [−0.16, +0.14]
OPT, 2 ep, 36k blocks+0.43092%+0.074+0.35 [+0.14, +0.57]
same, second seed+0.43691%+0.080+0.24 [+0.01, +0.47]
OPT, 1 ep+0.301100%+0.057+0.53 [+0.33, +0.74]
OPT, 2 ep+0.42579%+0.098+0.66 [+0.43, +0.88]
OPT, WikiText-103, 1 ep+0.65793%+0.171+1.27 [+0.98, +1.56]

As you suspected, the rare-matched tilt reproduces most of the rare-band advantage everywhere (79% to 110%), so that criterion cannot settle the question. The whole-profile fit does, and one rule, registered before the runs, predicts the probe in all nine rows:

  • OPT-125m: no single tilt reproduces the band profile (residual +0.057 to +0.171 against a floor of 0.022), and the probe sides with the rare bands.
  • Pythia-1b and SmolLM3-3B at one epoch: a tilt reproduces the profile within the floor, and the probe does not side with the rare bands.

So a reversal remains, in your sense, exactly where recalibration cannot explain the rare-band gain, and not elsewhere. You were right that the knowledge evidence does not extend beyond 125M. The revision states this scope in the abstract, introduction and limitations.

Q2, the title and SmolLM3-3B at one epoch. The title claims that perplexity and its own frequency decomposition can give opposite verdicts on the same intervention. It does not claim the rare bands are always right. SmolLM3-3B at one epoch is where they are not: the two readings disagree (the exponent and the mass-weighted loss oppose the rare bands beyond the 3B seed floor), while the probe sides with the aggregate on both seeds. Pythia-1b behaves the same way. The revised abstract says this in words: the test sides with the rare-token reading in three of the five disagreeing settings, with perplexity in one, and is inconclusive in one. The decomposition identifies when perplexity's verdict on the tail is in question; a calibration-immune test settles which reading tracks knowledge. If the AC prefers, we will adopt a narrower title for the camera-ready.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision (to the AC and all reviewers)

Summary of the revision (to the AC and all reviewers)

We thank the AC and both reviewers. The three reviews raise the same core issues, and we accept them: the paper was hard to read, its take-home message was not stated, the verdict rule differed between Tables 1 and 2, and the calibration evidence was incomplete in exactly the conditions that matter. The revised PDF addresses each point. The main changes are:

  1. Rewritten for readability. A new abstract (no symbols) and introduction define every term in plain words and state one take-home message: held-out perplexity can miss or reverse an effect on rare tokens that its own frequency decomposition shows, so the per-band losses should be reported next to it. Definitions (bands, advantage, collapse exponent, seed floors, verdict rule) now come before any result (Section 3).

  2. One verdict rule for every table. Each table now scores the aggregate as agree, opposite or silent against the seed floor of the same model family, under three aggregate statistics: the collapse exponent, terminal perplexity, and the mass-weighted loss ∑bwbA(b)\sum_b w_b A(b). We ran second seeds for the families that had none. In the five conditions at overfit ratio 1.06 or below, no aggregate statistic agrees with the rare bands (exponent opposite in 5; terminal perplexity opposite in 1 and silent in 4; mass-weighted loss opposite in 4 and silent in 1). In the five at 1.22 to 1.84, all three agree (Table 1).

  3. The calibration control and the probe in every disagreeing condition (new Table 3). We ran 14 new runs, registered before launch, as pairs on one software stack. The calibration-immune forced-choice test sides with the rare bands in all four OPT-125m disagreeing settings (+0.24 to +1.27 accuracy points, every CI above zero). It does not side with them at Pythia-1b (two seeds, both null) or SmolLM3-3B (two seeds, −0.34-0.34 and −0.26-0.26). A rule registered in advance predicts this: the test sides with the rare bands exactly where no unigram recalibration can reproduce the band profile. It held in all nine rows. One registered prediction failed, and we report it (Appendix H). The knowledge claim is now stated in that scope.

  4. Focus. The convergence theorem, which the measurement does not use, is now in Appendix A.1. The main text covers the decomposition, the measurement, the controls and the test. Section 3.2 states how the theory connects to the experiments, and what the decomposition costs in practice.

  5. Corrections we found ourselves. The teaser previously showed a condition whose aggregate lay inside its seed floor; it now shows one that clears it. Appendix C understated the drift between software stacks (0.01 nats); the new runs measure up to 0.26 nats in the rarest bands, which is why no comparison mixes stacks. Three symbol clashes are removed.

The main text is 8 pages. Every number is generated from the result files. The code release (anonymous link in the abstract, and the supplementary zip) now includes the new runs and the script that scores them. Point-by-point replies are in each reviewer's thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.