acceptodds
Under review as a conference paper at ICLR 2027

What Does a Training Token Buy? Contextual Information Gain Beyond Perplexity Across Open Model Suites

Abstract

A lower perplexity on held-out text can come from two sources. The model may use the context better, or it may make a better guess about the text before any context is given; perplexity adds the two into one number, so a falling perplexity cannot say which improved. We separate them. For every held-out token we score the model twice, with its context and under its own unconditional next-token distribution, obtained by averaging its predictions over reference contexts. The difference, contextual information gain (CIG), is the pointwise mutual information between context and token under the model, so a higher value means the context did more of the work; the first term is the usual held-out cross-entropy, and the second tracks how the model’s unconditional expectations move during training. We measure both on 790 public checkpoints from seven model families (14M to 12B parameters, up to 1071× the Chinchilla-optimal data budget) on 206 held-out texts from ten domains, in bits per byte, and judge every trend against the model’s own checkpoint-to-checkpoint noise. In the three training runs whose learning rate and training-data composition stay constant, held-out cross-entropy keeps falling to the end of the window, at up to 173× the optimal budget, while the model’s prior moves away from held-out text by 18 to 29% of the loss improvement without becoming more peaked. The additive scaling law predicts that after the same number of training tokens every model size improves at the same rate per factor of two in tokens; instead, larger models improve faster, at all 58 matched comparisons on four model ladders across three corpora. Only the 70M to 160M Pythia models get worse late in training, whereas models trained far beyond their optimal budget keep improving. In warmup-stable-decay schedules the decay phase lowers held-out loss 17× faster per factor of two than the stable phase in three runs and 36× faster in a fourth; every documented decay phase coincides with a change of training data, but within the one decay phase with 13 checkpoints, at unchanged data, the rate rises 3.5× as the learning rate falls to zero. The measurement thus turns the model’s own noise into a stopping diagnostic, and tells, from checkpoints and held-out text alone, whether the next factor of two in training tokens still improves the use of context or only shifts the model’s expectations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
scaling laws, pretraining dynamics, language model evaluation, perplexity, contextual information gain, pointwise mutual information, overtraining, data repetition, learning rate schedules, intermediate checkpoints

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x4C6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces contextual information gain (CIG), computed by comparing token predictions under their actual contexts and a reference distribution averaged over other contexts. Using 790 checkpoints from seven open model families, it studies how held-out prediction changes with training duration, model size, data repetition, learning-rate schedules, and domains. It reports persistent loss improvements beyond Chinchilla-style data budgets, reference-prior drift, scale-dependent gains, localized late degradation, and faster gains during decay phases, then proposes a noise-based stopping diagnostic.

Critical strengths
  • A useful question with observable evidence. Instead of focusing only on final checkpoints, the paper asks what additional training actually achieves along a training trajectory.
  • Broad checkpoint coverage with careful regime separation. The study covers 790 checkpoints and distinguishes stable windows from schedule or data-mixture transitions.
Critical weaknesses
  • My main concerns lie in the gap between the CIG motivation and the evidence used for the paper's main claims. CIG is presented as a way to distinguish context use from shifts in unconditional expectations, but most headline findings are based on held-out cross-entropy slopes and would remain unchanged without the proposed decomposition. Moreover, CIG can increase simply because the reference distribution assigns lower probability to the evaluated tokens. The paper therefore does not establish that its observed training gains represent better context utilization rather than changes in predictive loss and the chosen reference.
  • The reference-prior analysis is sensitive to a narrow sampling choice. The main prior is estimated from only 12 documents, and changing the reference changes the estimated drift magnitude. The claim that prior drift reflects a model-wide effect rather than adaptation to reference contexts remains insufficiently established.
  • The scaling-law conclusion is broader than the test supports. The additive Chinchilla form was fitted for final-loss scaling, whereas the paper tests local derivatives along intermediate trajectories. The 58 comparisons are correlated, and a single matched-learning-rate size pair cannot fully control optimization differences.
Questions for authors
  • What actually moves in the prior? And which token groups gain or lose probability mass as its entropy stays nearly constant?
  • Why does the size effect reverse over training?
4 comments
AuthorsOpenReview ↗

Weakness 1&2 addressed

W1, first part: the headline findings rest on held-out cross-entropy and would not change without the decomposition Your point is correct, and by design: the duration, scaling, corpus and schedule results are statements about held-out loss and should not depend on a reference. The submission did not say clearly what the decomposition adds beyond them. The revision says it in one paragraph of §1.

  • After warm-up, against a reference that covers the evaluated vocabulary, the unconditional term is flat, so the loss improvements are not shifts in unconditional expectations.
  • In the first thousand steps it carries 40–54 % of the loss fall (twelve Pythia runs).
  • The seen-text advantage sits in the unconditional term (our reply to Reviewer NNXC gives the numbers). The revision no longer claims a motion of the model's prior after warm-up.

W1, second part: CIG can rise simply because the reference assigns lower probability to the evaluated tokens. Agreed, and in the submitted numbers this is what happened. Against the 12-document reference, CIG rose 17–26 % faster than the loss fell, because the reference term rose; and the reference term rose because that reference lacks 27–29 % of the evaluated tokens. Against a reference of 800 documents, CIG's slope equals the loss improvement to within 0.6 %, 2.8 % and 2.2 % in the three windows. So the gains we report are decreases in held-out loss, and the revision attributes them to context only in this measured sense. The abstract's closing claim about "the use of context" is removed. §4 will also say what the term does not show: it does not distinguish short from long context. In an exploratory analysis for a new appendix, the late improvement is complete within 256 tokens of context in all five runs we tested.

W2: the 12-document reference. Agreed; the paper's two references (3 and 12 documents) could not settle it. We ran the test: nested references of 12, 50, 200 and 800 held-out documents, on the three windows of the submission and on twelve Pythia runs (fifteen runs, four model families). The apparent drift is proportional to the share of evaluated tokens the reference lacks, and against 800 documents it is under 3 % of the loss improvement in thirteen of the fifteen. It was adaptation to the reference sample, not a model-wide effect, and we withdraw the claim. The two exceptions are the 70M Pythia runs (8.5 % and 6.3 %). There the remainder sits in tokens the reference does cover, in both training runs. We report it and do not explain it.

AuthorsOpenReview ↗

Weakness 3 addressed

W3: the scaling claim. We take the three points in turn.

Scope. Agreed. The additive form was fitted to final losses of runs whose learning-rate schedule was matched to their token budget; we look at slopes inside single runs. What we test is the form's implication when it is read along a training trajectory: equal gain per doubling of tokens at matched tokens, whatever the size. We do not refit a compute-optimal frontier. The revision says so in the first paragraph of §5 and retitles the finding.

Correlated comparisons. Agreed, and we counted. Each comparison uses a local window of at least four checkpoints, so neighbours share checkpoints. The 58 comparisons rest on 42 distinct windows. At most 8 of these share no checkpoint with one another: 4 on the standard Pythia ladder, 2 on the deduplicated one, 1 each for SmolLM2 and BLOOM. All 8 are positive and resolved.

We also computed one size effect per ladder over its whole evaluated range, with an interval from a bootstrap that resamples probes and checkpoints (the same checkpoints for every size where the sizes share a checkpoint grid). It is positive and excludes zero on all four ladders, in bits per byte per doubling of tokens per doubling of parameters:

  • Pythia deduplicated: +0.0061 [0.0055, 0.0072]
  • Pythia standard: +0.0020 [0.0018, 0.0022]
  • SmolLM2: +0.0022 [0.0016, 0.0027]
  • BLOOM: +0.0069 [0.0052, 0.0094] The revision reports these in place of the count of 58, with two caveats that we found on the way.
  • On three ladders the gain rises at every step in size. On the standard Pythia ladder it does not: over the range, the 70M and 160M models gain less than the 14M and 31M models. This is the 70M–160M band of our third result. For that ladder the number is a trend across six sizes, not an ordering.
  • BLOOM has only three or four checkpoints per model in the range we evaluate (52–210 B tokens), and seven of its eight local windows begin before our warm-up guard. Its whole-range value above is about a fifth of the median we had reported. BLOOM supports the direction, as the submission said, and nothing more. One matched pair. We added a second. Pythia 6.9B and 12B share a peak learning rate in the Pythia paper and in the released configuration files. Our 12B rows were bf16, which we do not pool, so we re-scored the 12B checkpoints in fp32. The larger model improves faster at 13 of 13 matched steps past the guard (9 distinct windows). The submitted statement that this pair was negative and not interpreted described the bf16 rows; it is replaced. We also found that for the first pair (410M / 1B) the two sources disagree: the Pythia paper's table gives both models the same peak learning rate, and the released configuration gives 1B a value 1.2 times smaller. The appendix documents this. Past 100 B tokens that pair is positive at 13 of 13 steps on both corpora.
AuthorsOpenReview ↗

Questions answered

Q1. What moves in the prior? The estimate, not the model's unconditional expectation. Tokens that never occur in the 12 reference documents (27–29 % of the evaluated tokens) lose mass: their averaged probability is only what the model leaks to them in contexts where they do not occur, and the leak shrinks as predictions sharpen. They carry 106–108 % of the drift in the three windows. Tokens the reference contains gain slightly. The entropy stays nearly constant because it is weighted by the averaged distribution itself, hence by frequent tokens, and those are calibrated: for tokens that an 800-document reference contains more than 300 times, the averaged probability matches their frequency to within 0.014 bits at every checkpoint. A new appendix gives the split by token frequency.

Q2. Why does the size effect reverse over training? We observe it and cannot explain it, and the revision says so in §5 instead of an appendix.

  • The observation. In the 410M / 1B pair, just after our warm-up guard, the larger model gains less per doubling. This is resolved at 16.8–29.4 B training tokens on the deduplicated corpus and at 16.8–25.2 B on the standard one. From 73 B (deduplicated) and 31 B (standard) on, every matched step is resolved in the other direction.
  • What the standard forms say. The additive form predicts no size effect at any budget. The coupled forms we know of (a single size–data interaction term, or a power of the sum of a size term and a data term) give the size effect one sign at every budget.
  • A hypothesis, with no fit. A reversal needs a data exponent that depends on size: the smaller model gains faster early and saturates sooner.
  • What we cannot exclude. The learning rate, for this early window. The 6.9B / 12B pair has a coarser checkpoint grid there (its first local window spans 23–61 B tokens) and is positive in it.
AuthorsOpenReview ↗

To Reviewer x4C6. Where your points stand in the final revision

Dear Reviewer x4C6, The final revision is uploaded. Each weakness in your review named something that could be tested, and we ran every test. We would be grateful for your assessment of the results before the discussion closes on November 18 (AoE). Here is where each point now stands.

  1. W1 and W2, the reference. Section 4, Figure 1 (left) and Appendix I. Against nested references of 12 to 800 documents the apparent drift is proportional to the share of evaluated tokens that the reference lacks. We withdrew the claim about the prior.
  2. Q1, what moves in the prior. Table 2. The tokens that the reference lacks rise in all three windows, and no group of tokens that occurs more than three times has a resolved slope.
  3. W3, the scaling claim. Section 5. The first paragraph states the scope. Table 3 gives one interval per ladder, which excludes zero on all four ladders. The 58 comparisons rest on 42 distinct windows. At most 8 of these share no checkpoint with one another, and all 8 are positive and resolved. Table 4 gives the second pair with an equal peak learning rate, 6.9B and 12B, with the 12B checkpoints scored again in fp32.
  4. Q2, the early reversal. The last paragraph of Section 5. We report it, name the scaling forms that do not produce it, and say that we cannot yet explain it.

Two direct questions. Do the reference-size test and the second pair answer W2 and W3 for you? Is there a remaining concern that keeps the paper below acceptance for you?

If the revision resolves your concerns, we ask you to update your review and your scores. If something still stands in the way, please name it, and we will answer within the remaining time.

Thank you for your time.

Reviewer_4UFvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors aim to disentangle the factors affecting the perplexity of a given token. They argue that the perplexity of a token is the combined effect of two factors : the model's ability to use the context and model's ability to correctly predict without the context. Therefore it is difficult to understand which contributes more to the perplexity. To distinguish between these two factors, the authors evaluate the models twice - first, using the context and second, by averaging over reference contexts, drawn from held out dataset.

I am not sure how these two sentences in the abstract are connected -- "We measure both on 790 public checkpoints from seven model families (14M to 12B parameters, up to 1071× the Chinchilla-optimal data budget) on 206 held-out texts from ten domains, in bits per byte, and judge every trend against the model’s own checkpoint-to-checkpoint noise.

In the three training runs whose learning rate and training-data composition stay constant, held-out cross-entropy keeps falling to the end of the window, at up to 173× the optimal budget, while the model’s prior moves away from held-out text by 18 to 29% of the loss improvement without becoming more peaked."

Overall the abstract is too long and difficult to understand.

Critical strengths

Some of the takeaways like "LARGER MODELS IMPROVE FASTER AT EQUAL DATA" are interesting though the extensive use of LLMs in the text and the use of unusual terminologies is hindering the ability of the reader to understand and focus on the main takeaways.

Critical weaknesses

As the authors also have mentioned in the AI use statement the extensive use of generative tools in writing -- it is to the extent where it can hinder a reader's ability to understand the main content and takeaways. for example: In second paragraph of the introduction the sentence -- "Part of the difficulty is the instrument." , "Held-out loss keeps falling" , "Late decline is a size band, not an overtraining threshold", "Provenance is inherited, not controlled", "plus 6 legacy probes", "Seen text: the advantage shrinks with scale", "12 register-balanced".

Next the paper cites a paper (L143-L147) which is not published and further also mentions that -- "The present paper takes the definition as given, reuses none of that paper’s results as a contribution, and asks what the instrument measures across training: duration, scale, corpus, contamination, schedule and domain, including what the reference term’s motion over training does and does not mean".

Questions for authors
  1. I could not find the paper -- "Anonymous. The baseline is the question: Degenerate empty contexts in token-level information measures for language models, 2026. Under review." cited in the submission in the sentence -- "A separate methods paper, under review (Anonymous, 2026), quantifies that failure and the stability of the marginal-by-averaging reference across architectures, corpora and tokenizers."-- where is it published?

  2. The abstract is too long and difficult to understand.

2 comments

Thank you for pointing to the exact sentences and phrases that made the paper hard to read. Reviewer NNXC gave the same presentation score, so we treat this as a defect of the paper.

1. The two abstract sentences, and the abstract's length (Summary; Question 2). You are right that the two sentences you quote are not connected. The first describes everything we measured: 790 checkpoints. The second jumps to a question that only three training runs can answer, and does not say why. The reason is this. In most public training runs the learning rate decays, or the data mixture changes, while training proceeds, so a change in held-out loss cannot be attributed to the amount of training alone. Only three runs have a documented stretch in which both stay constant, and our statement about how long the loss keeps falling is made for those stretches. The revised abstract says so, and is cut from 391 to roughly 250 words:

Here is the draft we are publishing here for you, before the revision is updated:

Held-out loss can fall because a model uses its context better, or because its expectations before seeing any context improve. Loss combines the two, so a falling loss cannot say which. We separate them: each held-out token is scored with its context and under the model's own unconditional next-token distribution, its predictions averaged over reference contexts. The difference, contextual information gain (CIG), is the pointwise mutual information between context and token under the model. We measure both terms on 790 public checkpoints from seven model families (14M–12B parameters) and 206 held-out texts, against each model's checkpoint-to-checkpoint noise. Three results concern training. Where learning rate and data are documented to be constant, loss keeps falling to 173× the Chinchilla-optimal budget. At matched training tokens, larger models improve faster per doubling in all four ladders of model sizes; an additive scaling form predicts no dependence on size. Only the 70M–160M Pythia models degrade late in training. Three concern the decomposition. The unconditional term is learned early: in the first thousand steps of twelve Pythia runs it supplies 40–54% of the loss fall; afterwards, against a reference that covers the evaluated vocabulary, under 3% in thirteen of fifteen runs. Against a small reference it appears to drift, an artifact proportional to the share of evaluated tokens the reference lacks. And the per-token advantage of articles that predate the training corpus over later ones lies in the unconditional term, appears within a thousand steps at every model size, and is reproduced by token counts with no model.

2. Undefined terms and compressed phrases (Weakness 1). Each phrase you list is rewritten.

  • "plus 6 legacy probes" was an undefined term. The six are Wikipedia texts that serve as a fixed anchor: one checkpoint is re-scored on them in every session, to confirm that the measurement has not moved. §3 now says so.
  • "12 register-balanced" was undefined too. The reference is 12 held-out documents from five kinds of text: two biomedical articles, two Python source files, two literary texts, four encyclopedia articles and two regulatory notices. The revision also evaluates references of up to 800 documents (see our comment to all reviewers).
  • "Part of the difficulty is the instrument" becomes a plain statement of the two ways a loss can fall.
  • The headings become sentences, for example "Held-out loss keeps falling at constant learning rate and data", "Only the 70M–160M models get worse late in training" and "We observe public training runs; we did not control them". On AI assistance: we disclosed it as the ICLR 2027 policy asks. The design, the measurements, the claims and the final wording are the authors' responsibility. We take your comment as a request for plainer writing, which the revision tries to meet throughout.

3. The paper cited as "Anonymous, 2026, under review" (Weakness 2; Question 1). It is under review at another venue and has no public version, which is why you could not find it. It is cited without author names so that double-blind review is preserved. Nothing in this submission depends on it: Equation 1 and §3 define the measurement completely, every number in the paper is computed here, and no result of that paper is used. The phrase you quote, "takes the definition as given", wrongly suggested that something is defined elsewhere. It is removed, and §2 now states that the paper is self-contained.

4. Soundness. The review gives soundness 1 and names no concern about the method or the evidence; the concerns it states are the writing and the citation, both addressed above. If there is a methodological concern that we have missed, we would be glad to address it during the discussion.

AuthorsOpenReview ↗

To Reviewer 4UFv. Where your points stand in the final revision

Dear Reviewer 4UFv, The final revision is uploaded. We ask you to look at it once more before the discussion closes on November 18 (AoE), because every point of your review is now addressed in the text.

  1. Readability. The whole paper is rewritten in plain sentences. Each phrase that you listed is gone, and the terms that you found undefined are explained in Section 3. The introduction now follows four plain questions.
  2. The two unconnected sentences of the abstract, and its length. The abstract is new, rewritten and shorter. The introduction explains why only three of the runs can answer the question of duration.
  3. The paper cited as under review. Section 2 states that no result of that paper is used here. Section 3 defines the measurement completely. Your review gives soundness 1 and a clear rejection, and it names no concern about the method or the evidence. If you have such a concern, please state it, and we will answer it within the remaining time. If you have none, we ask you to reconsider the soundness score and the rating in the light of the revised text.

A short reply either way would help the area chair to weigh the reviews.

Thank you for your time.

Reviewer_NNXCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper splits held-out cross entropy into a context term p and a reference term q. The latter is the model's marginal next-token distribution, computed by averaging its predictions over 21600 contexts from 12 reference documents. the difference (CIG) is the pointwise mutual info between context and token under the model. The reference term decomposes as H(P) + KL(P||q), so it makes the motion over training is KL drift.

From 790 public checkpoints, the authors find many conclusions, including the fact that at matched tokens, larger models gain more per doubling, contradicting the additive Chinchilla form.

Critical strengths
  • Good measurement care. For example, fp32 throughout with an anchor re-scored every session, noise floor per model, and pre-registration ledger that reports refuted predictions.
  • Good audit on the Pythia models
  • Careful claims with with the duration and windows where LR and data are documented to be constant.
Critical weaknesses
  • It seems that the new instrument does very little work. As section 1 says, the duration, scaling, corpus and schedule results use only the context term, which is simply just the held-out cross-entropy. I am still not sure how to read the drift result in the decomposition.
  • I am not sure how CIG goes beyond perplexity. I would be interested in a result where CIG changes a conclusion that held-out loss would give.
Questions for authors
  • Is there a case where CIG and the context term disagree and change the conclusion, like when to stop or which domain improves
  • How does drift change with reference size? If it shrinks as the reference grows, it might be mostly a property of the sample,
  • I feel that 6 findings, each with many nuances, make the paper a little bit hard to follow. Is it possible to focus on key few ones? Mostly about the writing.
5 comments
AuthorsOpenReview ↗

Questions 2&3; Weakness 1

Thank you for a review that named the right test. We answer your second question first, because the rest depends on it.

Q2. How does the drift change with reference size? It shrinks, and your suspicion was right: it is the sample. We re-measured the three constant-learning-rate windows of the submission (SmolLM2-1.7B, SmolLM3-3B, Apertus-8B; every checkpoint) against nested references of 12, 50, 200 and 800 held-out documents.

  • The apparent drift falls in proportion to the share of evaluated tokens that the reference does not contain (r² = 0.989–0.998; at zero uncovered share the fitted line is within ±0.4 % of the loss slope).
  • It goes from 16–32 % of the loss improvement at 12 documents to 0.6 %, 2.8 % and 2.2 % at 800.
  • The same proportionality holds in twelve Pythia runs.
  • In the reference we submitted, the tokens it lacks are 27–29 % of the evaluated tokens and carry 106–108 % of the drift. We withdraw the drift as a statement about the models. What remains is a rule for the instrument, which the revision states: report the uncovered share with every reference-term number, and use a reference that covers at least 98 % of the evaluated tokens.

W1. "The new instrument does very little work"; how to read the drift. You are right on both counts. Four of the six findings are statements about held-out loss, on purpose: they should not depend on any reference. And the drift could not be read because it was not a property of the models. What the decomposition does establish is now stated in one paragraph of §1 and in §4.

  • After warm-up, against a reference that covers the evaluated vocabulary, the unconditional term is flat: its slope is under 3 % of the loss slope in 13 of 15 runs. So the loss improvements we report are not shifts in the models' unconditional token expectations.
  • During warm-up it is not flat: in the first thousand steps of twelve Pythia runs it carries 40–54 % of the loss fall.

Q3. Six findings with many nuances. Agreed. The revision leads with three results about training: how long held-out loss keeps falling at constant learning rate and data; larger models improving faster at matched tokens; and the 70M–160M models that degrade late. One paragraph then says what the decomposition adds. The decay phase, the domain structure and the stopping diagnostic become one short section of further observations, with their detail in appendices. The abstract goes from 391 to about 250 words.

AuthorsOpenReview ↗

W2 and Q1. A case where CIG and the loss disagree and the conclusion changes

For the two examples you name, when to stop and which domain improves: no, and we can now say why. Scored against a covering reference, CIG's slope equals the loss improvement to within 0.6–2.8 % in the three windows. A stopping rule built on CIG's slope therefore gives the same answer there as one built on the loss, and we expect the same of a domain ranking (we have not re-measured it domain by domain against the larger reference). Where our submitted numbers differed, the difference came from the reference.

The decomposition changes the reading in two other places.

Early training. In the first thousand steps, 40–54 % of the loss fall is the model learning which tokens are frequent, not using context. That order is known (Chang and Bergen, 2022; Belrose et al., 2024); the decomposition gives its size.

Seen against unseen text, at the final checkpoint. We compare ten biomedical articles that predate the training corpus with ten published after training.

  • In the 14M–70M Pythia models the loss is lower on the older articles by 0.39–0.48 bits per token, which looks like memorisation. In the 1.4B–6.9B models the gap is no longer resolved (0.03–0.09), which looks as if the two sets were alike.
  • Neither reading is complete. The unconditional term favours the older articles by 0.56–0.61 bits per token at every model size, from step 1000 on. Raw token counts of held-out documents, with no model at all, give the same gap: 0.66 [0.38, 0.94]. The older articles use vocabulary that is more frequent in this kind of text.
  • That gap stays, and the loss gap closes with model size (the decline is resolved, per token and per byte). What the models gain from context (CIG) must be larger on the newer articles Two limits, which the revision states: this is ten articles against ten, and per byte the gap itself is not resolved at any single size.
AuthorsOpenReview ↗

To Reviewer NNXC. Where your points stand in the final revision

Dear Reviewer NNXC, The final revision is uploaded. Your second question named the test that changed this paper, so we would be grateful for your assessment of the result before the discussion closes on November 18 (AoE). Here is where each of your points now stands.

  1. Q2, reference size. Section 4, Figure 1 (left), Table 2 and Appendix I. The drift was the reference, as you suspected. Against 800 documents it is 0.6%, 2.8% and 2.2% of the loss slope in the three windows. We withdrew the finding.
  2. W1, what the instrument does. Question (4) of the introduction and Section 4. After warm-up the unconditional term is flat, with a slope under 3% of the loss slope in 13 of 15 runs. In the first thousand steps it supplies 40 to 54% of the fall in loss.
  3. W2 and Q1, a case where the decomposition changes the conclusion. Section 6 and Table 5, older against newer articles.
  4. Q3, too many findings. Three results about training and one short section of further observations.
  5. Presentation. The whole text is rewritten in plain sentences. The abstract has 262 words instead of 391. Two direct questions. Does the reference-size test answer Q2 and W1 for you? Does the rewritten text change your score for presentation?

If the revision resolves your concerns, we ask you to update your review and your scores. If something still stands in the way, please name it, and we will answer within the remaining time.

Thank you for your time.

Comments on the paper

3 comments
AuthorsOpenReview ↗

**To all reviewers: what we did, what we withdraw, what we correct**

We thank the three reviewers. Two of the reviews named tests we had not run. We ran them, and one overturned a finding of the submission. This comment lists what changes; the replies under each review give the detail. A revised PDF follows within the discussion period.

New measurements, each asked for in a review.

  1. Reference size (NNXC Q2, x4C6 W2). The unconditional term re-measured against nested references of 12, 50, 200 and 800 held-out documents, in fifteen training runs from four model families.
  2. A second pair of model sizes with equal peak learning rate (x4C6 W3): Pythia 6.9B and 12B, with the 12B checkpoints re-scored in fp32.
  3. What the decomposition shows across training (NNXC W1–W2, x4C6 W1): the unconditional term's share of the loss change during and after warm-up in twelve Pythia runs, and the seen-text contrast split by term.

What we withdraw. The submission's second finding, that the model's prior drifts away from held-out text by 18–29 % of the loss improvement, was an artifact of our 12-document reference. The apparent drift is proportional to the share of evaluated tokens that the reference does not contain. Against 800 documents it is under 3 % of the loss improvement in thirteen of the fifteen runs. NNXC's second question anticipated this.

What we correct. While preparing the revision we found an error in our unit. Bits per byte were divided by the bytes of each whole stored text, although only its first 1,800 tokens are scored. All values are recomputed. Levels rise by a factor of 1.56–1.81, depending on the tokenizer; source-code texts, which are short and scored whole, do not change. No count of signs in the paper changes (all 58 comparisons behind "larger models improve faster" stay positive and resolved), and the decay-phase factors stay at 17× and 36×. Comparisons between code and the other nine domains do change. Two sentences that singled out code in the domain section are removed. "The repetition penalty is largest in code" holds at 160M and 410M and no longer at 70M. One registered test, for domain structure in that penalty at 70M, is no longer significant. The revision lists every changed number.

What stays. Held-out loss keeps falling where learning rate and data are constant. Larger models improve faster at matched training tokens. Only the 70M–160M models degrade late in training.

The revision has an abstract of about 250 words instead of 391, three results about training up front, one paragraph on what the decomposition adds, and new appendices for the reference-size test and for the two terms across training.

AuthorsOpenReview ↗

Summary of changes, to all reviewers

The revised paper is uploaded. A summary of the changes

We have uploaded the revised paper and its supplementary material. This comment says what changed and where to find it.

Why the text differs so much while the study is the same. Two reviewers scored the presentation 1, and NNXC asked us to concentrate on fewer findings. So we rewrote the prose of every section in plainer language and reduced six findings to three results about training and one section of further observations. The correction of the unit changed every number in bits per byte. The pool (790 checkpoints, 27 series), the 206 probes, the two defining equations and the three results about training are those of the submission.

The text. The abstract has 262 words instead of 391. The introduction states three results about training and then says what the decomposition adds. The whole text is rewritten in plainer language, with one name for each quantity. The main text ends on page 10, within the limit of the discussion phase. Three tables that answer points of the reviews now stand in it (Table 2 for x4C6 Q1, Table 4 for x4C6 W3 and Table 5 for NNXC Q1). Appendices A to H are rewritten and Appendices I to L are new.

What is withdrawn. The second finding of the submission, a drift of the model's prior, is withdrawn (Section 4, Appendix I). Against references of 12 to 800 held-out documents the apparent drift is proportional to the share of evaluated tokens that the reference lacks. Against 800 documents the unconditional term moves by less than 3% of the loss slope in 13 of 15 runs. The analyses that read this drift by entropy and by domain are withdrawn with it (Appendix C).

What is corrected. The unit (Appendix B). Bits per byte are now divided by the bytes of the scored text. Levels and slopes of the prose probes rise by a factor of 1.56 to 1.81, and the source-code probes do not change. Beyond the scale the following changed. Comparisons between code and the other domains changed. One registered test at 70M lost its significance. Three early comparisons of the 410M and 1B pair became resolved, two of them negative. Three more intervals changed their status at the margin. No resolved slope changed its sign. The supplementary material holds the full list and gives every numerical output of the analysis in both units.

New analyses, by the review that asked for them.

  1. Reference size (NNXC Q2, x4C6 W2). Section 4, Figure 1, Table 2 and Appendix I. Fifteen runs from four model families, references of 12, 50, 200 and 800 documents, and a split by how often a token occurs in the reference.
  2. The two terms across training (NNXC W1 and W2, x4C6 W1). Sections 4 and 6, Table 5 and Appendix J. In the first thousand steps of twelve Pythia runs the unconditional term supplies 40 to 54% of the fall in loss. The advantage of older over newer articles sits in the unconditional term, and token counts with no model reproduce it.
  3. A second pair of sizes with equal peak learning rate (x4C6 W3). Section 5, Table 4 and Appendix D. We scored Pythia-12B again in fp32. The larger model gains more at 13 of 13 matched token counts from 23 B tokens on. Every table and figure now shows the fp32 scores of that model.
  4. The independence of the 58 comparisons (x4C6). Section 5 and Appendix K. They rest on 42 distinct windows, and 8 windows share no checkpoint. All 8 are positive and resolved. Table 3 now gives one interval per ladder over the whole evaluated range, and it excludes zero on all four ladders.
  5. The two Pythia corpora on the four sizes that they share (Section 5, Appendix K). Their size effects differ by a factor of 1.10 [1.06, 1.14]. The larger factor of the submission came from ladders of different sizes.
  6. Context range (Appendix L). This part is exploratory and no claim rests on it. The record of predictions. The ledger of the 29 predictions of the submission is now in the supplementary material, with corrected numbers. No verdict changed. During the revision we registered 101 further predictions before the analyses that test them. 57 held, 43 failed and one could not be tested. The supplementary material lists each one with its date, its statement and its verdict.

The supplementary material adds the tables that the paper refers to, the notebooks and scripts of every new analysis with their tests on synthetic data, and the scripts that derive every number, table and figure of the paper from the result files.

What the revision does not do. Six predictions were registered for analyses that we did not run. The paper does not explain the early reversal of the size effect in the 410M and 1B pair, nor the remainder in the two 70M runs.

We thank the reviewers for the tests they asked for. One of them overturned a finding of the submission.

AuthorsOpenReview ↗

Final Revision Nov 17th. What changed since November 13

**Final revision uploaded. **

We have uploaded the final revision. It differs from the revision of November 13 in presentation only. No number, table, figure or appendix changed.

  1. The introduction is now built around four questions. Each names the runs that serve as its control and gives its answer in one sentence. Three questions concern training. The fourth says where the decomposition adds information to the loss and where it does not.
  2. The definition of the measure (Eq. 1) moved from the introduction to Section 3. The introduction states it in words.
  3. In Section 5 the paragraph on the early reversal of the size effect has a new heading and the same content. It still says that we cannot yet explain the reversal.
  4. The limitations (Section 8) now name the runs that each limit concerns.
  5. A few statements that stood twice were removed. The main text ends on page 10. Our summary of changes of November 13 describes everything else and still holds. Under each review we have posted a short note that lists where its points are answered in the final PDF. We would be grateful for a response from each reviewer before the discussion closes on November 18 (AoE).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.