- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces contextual information gain (CIG), computed by comparing token predictions under their actual contexts and a reference distribution averaged over other contexts. Using 790 checkpoints from seven open model families, it studies how held-out prediction changes with training duration, model size, data repetition, learning-rate schedules, and domains. It reports persistent loss improvements beyond Chinchilla-style data budgets, reference-prior drift, scale-dependent gains, localized late degradation, and faster gains during decay phases, then proposes a noise-based stopping diagnostic.
Critical strengths
- A useful question with observable evidence. Instead of focusing only on final checkpoints, the paper asks what additional training actually achieves along a training trajectory.
- Broad checkpoint coverage with careful regime separation. The study covers 790 checkpoints and distinguishes stable windows from schedule or data-mixture transitions.
Critical weaknesses
- My main concerns lie in the gap between the CIG motivation and the evidence used for the paper's main claims. CIG is presented as a way to distinguish context use from shifts in unconditional expectations, but most headline findings are based on held-out cross-entropy slopes and would remain unchanged without the proposed decomposition. Moreover, CIG can increase simply because the reference distribution assigns lower probability to the evaluated tokens. The paper therefore does not establish that its observed training gains represent better context utilization rather than changes in predictive loss and the chosen reference.
- The reference-prior analysis is sensitive to a narrow sampling choice. The main prior is estimated from only 12 documents, and changing the reference changes the estimated drift magnitude. The claim that prior drift reflects a model-wide effect rather than adaptation to reference contexts remains insufficiently established.
- The scaling-law conclusion is broader than the test supports. The additive Chinchilla form was fitted for final-loss scaling, whereas the paper tests local derivatives along intermediate trajectories. The 58 comparisons are correlated, and a single matched-learning-rate size pair cannot fully control optimization differences.
Questions for authors
- What actually moves in the prior? And which token groups gain or lose probability mass as its entropy stays nearly constant?
- Why does the size effect reverse over training?
4 comments
Weakness 3 addressed
W3: the scaling claim. We take the three points in turn.
Scope. Agreed. The additive form was fitted to final losses of runs whose learning-rate schedule was matched to their token budget; we look at slopes inside single runs. What we test is the form's implication when it is read along a training trajectory: equal gain per doubling of tokens at matched tokens, whatever the size. We do not refit a compute-optimal frontier. The revision says so in the first paragraph of §5 and retitles the finding.
Correlated comparisons. Agreed, and we counted. Each comparison uses a local window of at least four checkpoints, so neighbours share checkpoints. The 58 comparisons rest on 42 distinct windows. At most 8 of these share no checkpoint with one another: 4 on the standard Pythia ladder, 2 on the deduplicated one, 1 each for SmolLM2 and BLOOM. All 8 are positive and resolved.
We also computed one size effect per ladder over its whole evaluated range, with an interval from a bootstrap that resamples probes and checkpoints (the same checkpoints for every size where the sizes share a checkpoint grid). It is positive and excludes zero on all four ladders, in bits per byte per doubling of tokens per doubling of parameters:
- Pythia deduplicated: +0.0061 [0.0055, 0.0072]
- Pythia standard: +0.0020 [0.0018, 0.0022]
- SmolLM2: +0.0022 [0.0016, 0.0027]
- BLOOM: +0.0069 [0.0052, 0.0094] The revision reports these in place of the count of 58, with two caveats that we found on the way.
- On three ladders the gain rises at every step in size. On the standard Pythia ladder it does not: over the range, the 70M and 160M models gain less than the 14M and 31M models. This is the 70M–160M band of our third result. For that ladder the number is a trend across six sizes, not an ordering.
- BLOOM has only three or four checkpoints per model in the range we evaluate (52–210 B tokens), and seven of its eight local windows begin before our warm-up guard. Its whole-range value above is about a fifth of the median we had reported. BLOOM supports the direction, as the submission said, and nothing more. One matched pair. We added a second. Pythia 6.9B and 12B share a peak learning rate in the Pythia paper and in the released configuration files. Our 12B rows were bf16, which we do not pool, so we re-scored the 12B checkpoints in fp32. The larger model improves faster at 13 of 13 matched steps past the guard (9 distinct windows). The submitted statement that this pair was negative and not interpreted described the bf16 rows; it is replaced. We also found that for the first pair (410M / 1B) the two sources disagree: the Pythia paper's table gives both models the same peak learning rate, and the released configuration gives 1B a value 1.2 times smaller. The appendix documents this. Past 100 B tokens that pair is positive at 13 of 13 steps on both corpora.
Questions answered
Q1. What moves in the prior? The estimate, not the model's unconditional expectation. Tokens that never occur in the 12 reference documents (27–29 % of the evaluated tokens) lose mass: their averaged probability is only what the model leaks to them in contexts where they do not occur, and the leak shrinks as predictions sharpen. They carry 106–108 % of the drift in the three windows. Tokens the reference contains gain slightly. The entropy stays nearly constant because it is weighted by the averaged distribution itself, hence by frequent tokens, and those are calibrated: for tokens that an 800-document reference contains more than 300 times, the averaged probability matches their frequency to within 0.014 bits at every checkpoint. A new appendix gives the split by token frequency.
Q2. Why does the size effect reverse over training? We observe it and cannot explain it, and the revision says so in §5 instead of an appendix.
- The observation. In the 410M / 1B pair, just after our warm-up guard, the larger model gains less per doubling. This is resolved at 16.8–29.4 B training tokens on the deduplicated corpus and at 16.8–25.2 B on the standard one. From 73 B (deduplicated) and 31 B (standard) on, every matched step is resolved in the other direction.
- What the standard forms say. The additive form predicts no size effect at any budget. The coupled forms we know of (a single size–data interaction term, or a power of the sum of a size term and a data term) give the size effect one sign at every budget.
- A hypothesis, with no fit. A reversal needs a data exponent that depends on size: the smaller model gains faster early and saturates sooner.
- What we cannot exclude. The learning rate, for this early window. The 6.9B / 12B pair has a coarser checkpoint grid there (its first local window spans 23–61 B tokens) and is positive in it.
To Reviewer x4C6. Where your points stand in the final revision
Dear Reviewer x4C6, The final revision is uploaded. Each weakness in your review named something that could be tested, and we ran every test. We would be grateful for your assessment of the results before the discussion closes on November 18 (AoE). Here is where each point now stands.
- W1 and W2, the reference. Section 4, Figure 1 (left) and Appendix I. Against nested references of 12 to 800 documents the apparent drift is proportional to the share of evaluated tokens that the reference lacks. We withdrew the claim about the prior.
- Q1, what moves in the prior. Table 2. The tokens that the reference lacks rise in all three windows, and no group of tokens that occurs more than three times has a resolved slope.
- W3, the scaling claim. Section 5. The first paragraph states the scope. Table 3 gives one interval per ladder, which excludes zero on all four ladders. The 58 comparisons rest on 42 distinct windows. At most 8 of these share no checkpoint with one another, and all 8 are positive and resolved. Table 4 gives the second pair with an equal peak learning rate, 6.9B and 12B, with the 12B checkpoints scored again in fp32.
- Q2, the early reversal. The last paragraph of Section 5. We report it, name the scaling forms that do not produce it, and say that we cannot yet explain it.
Two direct questions. Do the reference-size test and the second pair answer W2 and W3 for you? Is there a remaining concern that keeps the paper below acceptance for you?
If the revision resolves your concerns, we ask you to update your review and your scores. If something still stands in the way, please name it, and we will answer within the remaining time.
Thank you for your time.
Weakness 1&2 addressed
W1, first part: the headline findings rest on held-out cross-entropy and would not change without the decomposition Your point is correct, and by design: the duration, scaling, corpus and schedule results are statements about held-out loss and should not depend on a reference. The submission did not say clearly what the decomposition adds beyond them. The revision says it in one paragraph of §1.
W1, second part: CIG can rise simply because the reference assigns lower probability to the evaluated tokens. Agreed, and in the submitted numbers this is what happened. Against the 12-document reference, CIG rose 17–26 % faster than the loss fell, because the reference term rose; and the reference term rose because that reference lacks 27–29 % of the evaluated tokens. Against a reference of 800 documents, CIG's slope equals the loss improvement to within 0.6 %, 2.8 % and 2.2 % in the three windows. So the gains we report are decreases in held-out loss, and the revision attributes them to context only in this measured sense. The abstract's closing claim about "the use of context" is removed. §4 will also say what the term does not show: it does not distinguish short from long context. In an exploratory analysis for a new appendix, the late improvement is complete within 256 tokens of context in all five runs we tested.
W2: the 12-document reference. Agreed; the paper's two references (3 and 12 documents) could not settle it. We ran the test: nested references of 12, 50, 200 and 800 held-out documents, on the three windows of the submission and on twelve Pythia runs (fifteen runs, four model families). The apparent drift is proportional to the share of evaluated tokens the reference lacks, and against 800 documents it is under 3 % of the loss improvement in thirteen of the fifteen. It was adaptation to the reference sample, not a model-wide effect, and we withdraw the claim. The two exceptions are the 70M Pythia runs (8.5 % and 6.3 %). There the remainder sits in tokens the reference does cover, in both training runs. We report it and do not explain it.