- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper works on one limitation of perplexity that underscore rare tokens. This will introduce judge damage to the rare tail texts. In a Zipf memorization model, the authors prove that error stays bounded under unbiased subsample training, and the validation weight on the rare items recursive training endangers shrinks as the corpus grows. The authors proposes splitting held-out loss by each token’s training frequency and shows sampling at τ = 1.15 always helps rare tokens in the experiment.
Critical strengths
The paper is clean and correct in its theory and rich in its experiments. The authors several controls about how temperature tau can influence loss only the token at different band to rule out alternative explanations.
Critical weaknesses
This paper's central claim is about perplexity can underweight rare tokens. A decomposition of loss by token rarity can expose it and authors demonstrate using sampling temperature (τ = 1.15 vs. 1.0) as an instrument. However, the whole section 3 containing the theoretical proofs didn't contribute too much to this main theme. The section 3 is working under one simplified memorization model (different from the LLM in experiment). Lemma 1, 2 and the Theorem 1 just says missing mass in the memorization model stays bounded under unbiased accumulate-subsample. This result is unrelated to perplexity and is neither used nor tested by the experiments. Propositions 1 and 2 are only comparatively elementary relation to perplexity. They describe how held-out loss weights rare tokens but assume the change of the rare token loss (as the most interesting and important part) was given. More broadly, the paper attempts to cover too many loosely related results or aspects within the page limit. Alongside its central measurement argument, it includes Theorem 1, a mean-field analysis, a relation between generator bias and collapse rate, a causality test, decoder–workflow interactions, and several mitigation arms. As a result, the main text is extremely compact, and I have to read 26 appendix to evaluate the main claims. This dilutes the paper’s focus and makes the core argument hard to follow from the main text alone.
Questions for authors
- Lots of presentations errors appeared throughout the paper, together with the dense context of section 3 and 4, making this manuscript hard to follow. Here are some of the examples:
"|a| ≤ 0.006" in line 224 is not defined in main text.
The collapse exponent for Table 1’s “aggregate” column is never defined until in the appendix.
"Bands" used in introduction and section 3, far away earlier than they’re defined in section 4.
The most important Abstract and Introduction sections are full of undefined parameter (T, beta, p_i and c) and concept that was used in section 3.
The very important verdict column is not defined in table 1 and table 2.
There are many remining incorrect pointing or symbol mismatch. I suggest the authors to read careful the whole manuscript and fix the presentation errors.
-
Using the definition in Table 2 (final perplexity, with a ‘silent’ category), would the five disagreements in Table 1 can be classified as silent rather than opposite? Can the author provide an explanation why silent is possible at Table 2 but not enabled in Table 1 by definition.
-
Table 1 is ordered by the overfit ratio. It is defined as generation-0 perplexity over “the best achievable for that model and corpus.” How was the best achievable perplexity determined?
Reply to Reviewer bsb1
Reply to Reviewer bsb1
Thank you for the careful reading and the specific list of problems. Every item is fixed in the revision, and your Q2 led to the largest change in the paper.
W1, Section 3 and the main theme. We agree. Theorem 1 answers a separate question (bounded error under unbiased accumulate-subsample training, left open by Dey and Donoho, 2024). The measurement does not use it, so it is now in Appendix A.1 with its lemmas. Propositions 1 and 2 are statements about held-out loss, and both are measured:
You are right that they take the change in rare-token loss as given. Section 3.2 now states how the pieces connect. Proposition 1 says the weight on at-risk tokens falls as corpora grow. Proposition 2(c) says a head cost of wR/wH times the tail gain flips the aggregate's sign, a bias that more data or more seeds do not remove. Section 5.1 measures how often that happens. We did not have time to add a derivation of the band profile itself, and we do not claim one.
W2, too many results. We agree. The main text now covers one argument: the decomposition (Section 3), the setup (Section 4), the measurement under one verdict rule (Section 5.1), five further interventions (5.2), and which reading tracks knowledge (5.3). The convergence theorem, mean-field analysis, bias predictor, causality test and decoder-workflow interaction appear only in the appendices. The main text no longer relies on any of them. It is 8 pages.
Q1, presentation errors.
Q2, silent versus opposite. Yes. Under Table 2's rule, four of the five would be silent, and thank you for catching the inconsistency. Table 1 scored the aggregate by the collapse exponent; Table 2 used terminal perplexity with a "silent" category. The revision applies one rule to every table: an aggregate statistic agrees with the rare bands, opposes them, or is silent, judged against the seed floor of the same model family. We ran second seeds for the families that had none (Table 10, Appendix P.3). We report three aggregate statistics:
The five conditions at overfit ratio 1.22 to 1.84 agree under all three. The rare-band advantage clears its floor in all ten. We treat silence and reversal as the same failure, since in both cases perplexity does not register a rare-band change that the decomposition shows. Table 2's verification filter is an example of the silent case, here on a mitigation. The abstract now describes the five as conditions where perplexity "gets worse or stays within run-to-run noise".
Q3, "best achievable" perplexity. The wording overstated it. The ratio is generation-0 validation perplexity divided by the lowest generation-0 validation perplexity among all recipes we ran for that model and corpus, from the per-model sweeps of Appendix C. Ratio 1.00 therefore marks the best recipe we found. For OPT-125m the reference is the two-epoch recipe on 36,000 blocks (35.64), which uses 2.4x more of WikiText-2 than the 15,000-block recipes. Section 4 and the Table 1 caption now state this. The ratios are computed from the result files rather than typed, and they reproduce the printed values.