acceptodds
Under review as a conference paper at ICLR 2027

Parameter Conventions Confound Compute-Optimal Scaling Exponents of Dense Language Models

Abstract

Compute-optimal scaling exponents are routinely compared across labs as if they measured the same quantity. We audit every public language-model scaling sweep we could find whose metadata determine, run by run, exactly how many parameters sit in vocabulary-sized matrices: 18 sweeps (1,673 runs) from six independent research groups. Changing only whether those matrices are counted raises the fitted data-model exponent, r, in every sweep and every group, by 0.54 on average across groups in held-out fits, comparable to the entire Kaplan-Chinchilla gap, under both separable and non-separable loss forms. Public data do not consistently settle which count is right: small-to-large extrapolation leans toward transformer-only counting in most groups but not significantly, holding out the largest models splits the groups three to three, and only one lab's frontier holdout favors one count clearly. The convention changes each exponent, but we find no evidence that it explains why labs disagree: harmonizing every sweep to one convention does not detectably reduce the cross-lab spread relative to the mean exponent. The two conventions nonetheless prescribe different frontier allocations, a median 2.3 times difference in tokens per parameter at 100 times the fitted compute. We release exact reconstructions, a convention registry, and a reporting standard.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling laws, compute-optimal training, large language models, parameter counting, reproducibility, meta-analysis, Chinchilla, held-out evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tXeiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the choice of parameter count affects fitted compute-optimal scaling laws. It reconstructs parameter counts for 18 public sweeps, covering 1,673 runs from six research groups, and compares fits using total parameters and transformer-only parameters.

Including vocabulary-related parameters increases the fitted data–model exponent in every primary sweep. Across the six groups, the mean increase from fits to the lower-compute subsets is about 0.54. However, held-out prediction does not consistently favour one counting convention across groups.

The paper also examines differences between labs and shows that the fitted conventions can recommend quite different model and data allocations at larger compute budgets. It proposes clearer reporting of parameter counts, compute definitions and recommended allocations.

Critical strengths
  1. The main strength is the systematic comparison across several sources. The effect of parameter counting was already known, but this paper checks it across a reasonably broad collection of public results. The source registry and reconstruction of parameter counts are useful contributions. The paper also keeps sources that do not meet its primary inclusion rules separate from the main analysis.

  2. The paired comparisons are carefully designed. The authors use the same runs, fitting procedure and loss form when comparing the two counting conventions. The exponent increases under both loss forms, which gives confidence that the main observation is not specific to one form.

  3. Grouping related sweeps by research group and codebase is sensible. It avoids treating all sweeps as fully independent evidence. I also appreciate that the paper distinguishes disagreement between predicted allocations from compute savings demonstrated through actual training.

Critical weaknesses
  1. The connection between the reported exponent and the compute constraint needs more clarity.

    Section 3 defines the data–model exponent as r=(α+θ)/βr=(\alpha+\theta)/\beta, or α/β\alpha/\beta for the separable form. This expression is valid under C∝NDC\propto ND, using the same NN as in the fitted loss.

    Section 4.4 instead optimises both fitted surfaces under the shared constraint C=6NtotalDC=6N_{\mathrm{total}}D. For a fit expressed in core parameters, the mapping to total parameters is nonlinear. The slope of the resulting optimal allocation curve therefore need not equal the reported exponent ratio. Both quantities can be useful, but their difference needs more clear explanation. At present, it is difficult to tell which conclusions concern a fitted exponent and which concern allocation under a common compute budget.

    There is a related issue with calling this compute axis “physical”. Section 3 describes core plus output-head parameters as the FLOP-faithful count, whereas Table 2 uses total parameters. For untied embeddings, counting the input lookup table in 6ND6ND assigns it the same cost per parameter as dense matrix operations. Porian et al. also discuss the output head and attention costs separately. A shared compute approximation is useful for comparison, but the paper should explain how closely it represents the actual training cost.

    Finally, an exact change of parameter coordinates, with the same underlying loss and compute functions, would preserve the physical optimum. Here, the loss forms are fitted again after changing the parameter count. The different recommendations can therefore reflect changes in the fit and its extrapolation also. This distinction is important for interpreting the practical claims.

  2. The allocation and between-lab conclusions need stronger support than the main exponent-shift result.

    The reported 2.3× difference in tokens per parameter and approximately 7–11% excess-compute estimates are important practical claims. However, Table 2 does not clearly identify which loss form gives these numbers or show whether they remain similar under the other form. This needs clarification, particularly because Section 6 states that none of the conclusions depend on the non-separable form.

    Appendix C reports that the non-separable form loses to the separable form in all 11 evaluable joint small-to-large extrapolation tests. This is a separate diagnostic panel, including non-primary sources, so it does not establish that the main allocation estimates are wrong. Still, it raises a reasonable concern about extrapolation sensitivity, especially when Table 2 goes much further beyond the observed budgets. Please specify the loss form used and show how sensitive the allocation conclusions are to this choice, the compute calculation and the vocabulary-count mapping.

    The conclusion about differences between labs also needs some care. Section 4.3 reports a standard-deviation ratio interval of [0.52, 3.98]. This interval allows a reduction of about 48% in standard deviation, or an increase to nearly four times the original value. If the disagreement is measured by variance, the lower endpoint allows a reduction of about 73%. So the statement that the convention does not account for “most” of the spread needs to specify the measure and reflect this uncertainty.

    The two loss forms also use different starting points for this comparison. The non-separable analysis moves from each source’s released convention to core counting, whereas the separable analysis moves from total to core counting. In the latter case, the absolute spread halves, while the spread relative to the mean remains broadly similar. These results need a matched comparison before drawing a general conclusion about disagreement between labs.

Presentation

Some terms need more precise wording. “Which count is right” is too broad for what the held-out tests establish. These tests compare counting conventions within particular fitted loss forms; they do not identify a parameter definition that is best for every purpose.

“Held-out Δr\Delta r” is also slightly confusing. The exponent is fitted on the lower-compute training subset, while loss prediction is evaluated on held-out runs. “Δr\Delta r from lower-compute fits” would make this clearer.

Please use different symbols for vocabulary size and the number of parameters in vocabulary matrices. Both are represented by VV in different parts of the paper. A short explanation connecting the fitted exponent, the compute constraint and the allocation curve would also help readers who are less familiar with scaling-law fitting.

Questions for authors
  1. Which compute constraint defines the reported r=(α+θ)/βr=(\alpha+\theta)/\beta? Please distinguish this quantity from the slope obtained when both fitted surfaces are optimised under the same compute constraint. For a direct comparison, can you report the slope dlog⁡D∗dlog⁡Ntotal∗\frac{d\log D^{\ast}}{d\log N_{\mathrm{total}}^{\ast}} for both fits at common compute budgets? This would help separate the effect of changing the parameter coordinate from differences between the fitted allocation curves.

  2. Which loss form produces the estimates in Table 2, and do the conclusions remain similar under the other form? Where the source information permits, can you also check a compute calculation that accounts separately for the output head, attention and input embedding lookup? Please show how sensitive the allocation differences and excess-compute estimates are to the vocabulary-count mapping beyond the observed model sizes.

  3. Can the between-lab comparison use the same starting convention for both loss forms? Please report uncertainty for both absolute spread and spread relative to the mean. This would make it easier to judge how much disagreement is reduced by using a common parameter convention.

  4. Can you clarify the vocabulary-matrix reconstruction for Farseer and Kempner? Farseer reports a vocabulary of 65,536, and its parameter-count difference is consistent with two matrices of this size. The manuscript describes one 131,072-row matrix. The total subtraction may be correct, but the distinction matters when separating input and output costs. The public Kempner base configuration specifies a vocabulary of 32,000 with weight tying disabled. If these settings apply to the analysed runs, why is the input/output split treated as unidentifiable?

  5. Farseer’s Appendix G already compares fits with and without embeddings at a 25.1B validation point, using its own scaling-law formulation. It reports more than four times smaller prediction error for the embedding-excluding fit. How does this relate to the discussion of Delphi as the exceptional frontier evidence? This is not a matched reanalysis, but it seems relevant to the discussion. If the necessary data are not available for a matched comparison, please explain this also.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tXei (1/2)

(1/2)

Thank you for a careful and constructive review. Two of your points were factual corrections that we confirmed against the sources' own code; several others led to new analyses.

How to read this reply: we number your five questions Q1–Q5 in the order they appear. Part 1 answers Q4 (the reconstructions) first, then Q1; part 2 answers Q2, Q3 and Q5, then Presentation. Your weaknesses map onto these: exponent vs. constraint and the coordinate change (Q1), the compute axis and the allocation claims (Q2), and the between-lab comparison (Q3).

Q4: Farseer and Kempner reconstruction (corrections).

  • Farseer: you are correct. Farseer's counting code uses V = 65,536 with untied embedding and head, so N_add_emb − exact_N = 2 × 65,536·h. Our subtraction and all reported numbers are unchanged; we have corrected the description. Because the head is identifiable, Farseer now enters the core+head analysis.
  • Kempner: also correct. The sweep script itself, not only the base config, sets vocab_size 32,000 and weight_tying False, so 64,000·d is a 32,000·d embedding plus a 32,000·d head. No number changes. We removed "not identifiable," and core+head now covers all six groups and all 18 sweeps. With full coverage, core+head is most accurate in 1 of 17 slices (Farseer, marginally); we have corrected the earlier "never" accordingly.
  • Re-auditing every reconstruction after your comment also led us to correct the Gadre counts (general comment, point 4).

On "the effect of parameter counting was already known." Its existence was known from one historical comparison (Kaplan vs. Chinchilla). What was not known is that the two conventions are mutually exclusive power-law assumptions that are not reliably distinguished by in-sample fit but prescribe different allocations for larger models. That difference is detectable on observed optima after only a 2.2× extrapolation, and which convention extrapolates better depends on the type of test (general comment, point 3).

Q1: exponent vs. compute constraint. The revision now separates these explicitly. r̂ is the slope under each convention's own C = 6·N·D, while Table 2 optimises both fitted surfaces under one shared constraint. The new decomposition separates the two exactly: axis relabelling, change of constraint, and difference between fitted surfaces on matched axes (table in the general comment). On a common N_total axis and a common constraint (C = 6·N_total·D), the slope d log D*/d log N_total averaged over groups is 1.38 for the transformer-only fit and 1.23 for the total fit within the fitted range, and 0.77 vs 1.23 at 10–1000× the largest fitted budget (non-separable form). Put simply: the total fit's slope is 1.23 both within the fitted range and at the frontier, constant because it is a power law in N_total. The transformer-only fit, re-expressed on the same axis, bends from 1.38 to 0.77 as the vocabulary share falls. Your point that an exact change of coordinates preserves the physical optimum is the right lens. Within the fitted range the two fits nearly coincide on matched axes (surface term −0.16, CI includes 0). The practical disagreement comes from the fits extrapolating different power laws (surface term +0.45 at 10–1000×).

AuthorsOpenReview ↗

Response to Reviewer tXei (2/2)

(2/2)

Q2: which loss form, compute calculation and sensitivity. The submitted Table 2 used the non-separable form; we now report both. We replaced 6·N_total·D with a FLOP count that charges the head and attention and treats the input embedding as a lookup. It matches Marin's measured FLOPs to within 0.2% and Kempner's released FLOP column exactly, so we now call it a "FLOP-faithful" axis rather than "physical." The intervals refit the vocabulary, head and attention mappings beyond the observed sizes.

  • At 100×: 2.39× (non-separable; BCa [2.04, 3.32]; 14/17 same direction) and 2.64× (separable; BCa [1.84, 6.25]; 12/17).
  • Excess compute at 100× (non-separable): 8.2% and 14.4% for the two reference conventions (submitted: 7.1% and 11.4%; the change comes mostly from the Gadre correction, see the bridge table in the general comment).
  • The full table at 10×, 100× and 1000× under both forms, with BCa intervals, is in our reply to fN6b (W2).

On Appendix C: the allocation gap is of similar size under the separable form, which extrapolates no worse, so it is not an artifact of the relaxation.

Q3: matched dispersion comparison. With the same starting convention (total → core) for both forms:

  • Non-separable — SD ratio: 1.10 [0.69, 1.45]; Variance ratio: 1.21 [0.47, 2.11]
  • Separable — SD ratio: 0.51 [0.42, 0.97]; Variance ratio: 0.26 [0.18, 0.95]

Under the separable form, the absolute spread does fall significantly. The spread relative to the mean rises under the non-separable form (coefficient-of-variation ratio, core ÷ total, 2.03 [1.49, 2.71]), because the exponents fall while the absolute spread is unchanged, and does not change detectably under the separable form (1.11 [0.92, 1.96]). §4.4 of the revision (§4.3 as submitted) now names the measure, and the word "most" is replaced by these intervals.

Q5: Farseer Appendix G. Thank you for pointing this out. Farseer's own validation favours the embedding-excluding fit at 25.1B. Because the seven validation runs (2.3B–25.1B) are released on the same metric as the fitting set, we ran the matched reanalysis you suggest. Transformer-only counting predicts the held-out large models best under both forms: mean relative error 0.0082 vs 0.0105 (non-separable) and 0.0324 vs 0.0346 (Chinchilla). That is the same direction as Farseer's appendix but a smaller typical gain (about 1.3×). At the single 25.1B point the gap is much larger (0.0001 vs 0.0128), but one point is noisy, so we base the comparison on all seven runs. Delphi is therefore no longer an isolated exception: with the Tao et al. intervention (response to fN6b, Q6), the frontier and controlled evidence favours transformer-only counting in Marin, Farseer and Tao. Kempner's released 3.3B frontier runs (about 20× beyond the sweep) also favour transformer-only counting, while Kempner's near-range holdouts and our new allocation backtest favour total counting (general comment, point 3). The revision therefore says that frontier-scale and controlled tests mostly favour transformer-only counting, that near-range tests are mixed, and that no test establishes either convention universally.

Presentation. We adopted all of your suggestions:

  • "Which count is right" → "which convention extrapolates better," and the §4.3 heading now reads "Public tests do not establish either convention universally."
  • "Held-out Δr" → "Δr from lower-80% fits."
  • Separate symbols for vocabulary size (V) and vocabulary-matrix parameters (N_vocab).
  • A short paragraph connecting the fitted exponent, the compute constraint and the allocation curve.

We believe these address the two concerns behind your rating: the distinction between fitted exponents and common-budget allocations, and the support for the allocation and between-lab conclusions. We would be glad to answer any remaining questions.

Thank you for the detailed response and the substantial revision. The revised manuscript addresses most of my original concerns, particularly the distinction between fitted exponents and common-budget allocations, and the support for the practical allocation claims.

The clarification in Section 3 and the decomposition in Section 4.2 are helpful. The manuscript now distinguishes the exponent under each convention’s own compute constraint from the allocation curve obtained under a shared constraint. Separating axis relabelling, changes in the constraint, and differences between fitted surfaces also makes the interpretation clearer. In particular, the practical disagreement is now appropriately connected to extrapolating different fitted power laws.

Table 2 now reports allocation differences under both loss forms, with uncertainty intervals. At 100× the fitted budget, the median tokens-per-parameter ratios are 2.39 and 2.64 under the non-separable and separable forms, respectively. This substantially addresses my concern that the allocation result might depend on the non-separable relaxation. The revised FLOP accounting and the explicit description of excess compute as a counterfactual estimate also improve the presentation. The new allocation backtest provides useful additional evidence that convention choice can affect decisions over relatively short extrapolations.

The corrected Farseer and Kempner descriptions, the disclosure of the additional Gadre corrections, and the inclusion of Farseer’s released validation runs address my reconstruction and frontier-evidence questions.

Two qualifications remain. First, the heading and opening of Section 4.4 should consistently distinguish absolute from relative dispersion. Under the separable form, the reported standard-deviation ratio of 0.51 [0.42, 0.97] supports a reduction in absolute between-group spread. Therefore, the broad statement that there is no evidence of reduced disagreement should be restricted to spread relative to the mean exponent.

Second, the distant allocation and excess-compute estimates remain conditional on the fitted loss forms and extrapolated architectural mappings. Refitting these mappings in the bootstrap is useful, but does not establish their validity at 100–1000× the observed budgets. Similarly, the clearest allocation-backtest result comes from Kempner; it supports decision sensitivity without establishing a generally preferable convention or validating the distant compute estimates.

Overall, the revision strengthens the paper sufficiently to change my assessment. The contribution is a useful systematic empirical audit, supported by clearer interpretation and additional validation. I am therefore increasing my overall rating from 2 (Weak rejection) to 3 (Weak acceptance).

AuthorsOpenReview ↗

Response to Reviewer tXei

Thank you for the careful re-reading and for raising your score. We agree with both qualifications and have revised the paper accordingly.

  1. Absolute vs. relative dispersion. The §4.4 heading now reads "Harmonizing conventions does not reduce spread relative to the exponent", and its opening and closing restrict the claim to spread relative to the size of the exponents. The section states that the absolute between-group spread does fall under the separable form (SD ratio 0.51 [0.42, 0.97]), while the spread relative to the mean exponent does not detectably fall under either form. We made the same change to finding (iii) in the introduction and to the conclusion.

  2. Conditionality of the distant estimates. We agree that the allocation and excess-compute estimates at 100–1000× are conditional on the fitted loss forms and on the vocabulary, head and attention mappings extrapolated beyond the observed sizes. The bootstrap refits these mappings but cannot validate them that far out. We also agree that the clearest backtest result comes from one group (Kempner): it shows that the convention changes decisions over short extrapolations, not which convention is preferable in general, and it does not validate the distant estimates. We now say this in Section 4.5 and in the Limitations.

We appreciate the reviewer's engagement throughout the discussion, which substantially improved the paper.

Reviewer_bY2qOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Question: Conventional approaches to scaling laws (e.g, Kaplan et al. discount vocab params). Now the question here is: Does counting vocab matrices (embed and unembed) in N change the compute-optimal results?
  • They take a bunch of public data for scaling parameters and redo the scaling analysis.
  • Finding 1: Counting vocabulary matrices raises r̂ across the board.
  • Finding 2: Held-out tests don’t consistently favor either convention (w/ or w/o vocab counting).
Critical strengths
  • A useful reflection on the conventional scaling analysis.
  • Very good use of existing datasets/models. Very nice!
  • I also appreciated the honest mixed findings.
Critical weaknesses
  • Some of the findings are expected. The fact that alpha is going to increase as you fold in more items in N is expected from math. The magnitude is the question, but we also know that is architecture/scale dependent. So not sure why we even highlight finding 1 as a "finding".

  • Why does this matter? Most people no longer use these analysis since we entered the inference-time regime and we like to over-train smaller models. So why bother with revisiting Kaplan?

    • Another reason to be skeptical is that, at larger scale embed/unembed are just a small fraction of model total params.
Questions for authors

Previous box.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bY2q

Thank you for the positive assessment, and for the three questions, which made us sharpen the paper.

In short: the sign of the shift is expected, and we now say so. What matters is that the two counting conventions are not reliably distinguished by in-sample fit but give different answers when a scaling law is used to plan larger models, and observed data can already tell them apart after a 2.2× extrapolation.

"The increase in α is expected from the math." You are right that the sign is expected, and the revision now says so directly: our new decomposition shows that within the fitted range, Δr is almost entirely axis relabelling and the change of compute constraint (general comment, point 1). What is not expected is the consequence. Because the vocabulary block grows more slowly than the transformer body, the loss cannot be a power law in both counts at once: if L = A·N_core^(−α), then L = A·(N_total − N_vocab)^(−α), which is not a power law in N_total. The two conventions are therefore mutually exclusive modelling assumptions, not relabellings, and scaling-law papers make this choice without stating it. In-sample fit does not reliably tell them apart (in Kempner, total counting fits the sweep better yet predicts the 3.3B frontier runs worse), and they extrapolate to larger models differently. Averaged over 10–1000× the fitted compute, the non-bookkeeping part of Δr is +0.45 [+0.27, +0.64] (6/6 groups), and on a FLOP-faithful axis the two fits prescribe a median 2.4× difference in tokens per parameter (BCa [2.0, 3.3]).

"Why does this matter in the over-training era?" First, we should have made clearer that the paper is not a revisit of Kaplan: every sweep in the panel is a modern public ladder released in 2024–2026 (Marin, the OLMo ladder, Farseer, Kempner, OpenLM, (Mis)Fitting). Second, we tested your scenario directly (new analysis). For each sweep's smallest model (median vocabulary share 84%) trained to 20–2,000 tokens per parameter, the two fits' predicted losses differ by less than their own fitting error (0.4–0.85×). For an inference-aware choice of model size (Sardana et al., 2024), with inference demand 10× or 100× the training tokens, the two conventions choose sizes within about 2% of each other, at a median 0.3% cost difference. So for over-training decisions within the range of model sizes a ladder covers, the convention makes little difference, and we now say so in the paper. It matters when the fitted law is used to plan models larger than the ladder, which is the main use of a scaling law. Llama 3, for example, fitted the optimal number of training tokens against compute on runs between 6×10¹⁸ and 10²² FLOPs and extrapolated about 3,800× to 3.8×10²⁵ FLOPs to size its flagship. There the convention enters through how training compute is counted, which, as far as we can find, the paper does not specify. The effect is also checkable on observed data, and over short ranges: in a new backtest, a 2.2× extrapolation in Kempner's sweeps already moves the predicted compute-optimal model size by about 1.4×, and the observed optimum distinguishes the two conventions in 6/6 corpora.

"At larger scale, embeddings are a small fraction of parameters." Agreed, and this turns out to be the mechanism by which the convention matters. That the magnitude depends on architecture and scale was known qualitatively, but it had not been measured across modern public ladders, and its effect on decisions had not been tested; the paper now does both. The vocabulary share falls from 52–97% at the smallest model of each ladder to 6–19% at the largest. Scaling laws are fitted mostly on the small end, where the two counts differ most, so the convention decides which power law the fit commits to. That commitment persists when the law is extrapolated to sizes where the share itself is negligible.

We also added tests of which convention extrapolates better. The frontier-scale tests mostly favour transformer-only counting, but near-range tests are mixed and Kempner's allocation backtest favours total counting, so no single convention can be assumed, one more reason it must be reported (general comment, point 3).

Thank you again; we are happy to clarify anything further.

AuthorsOpenReview ↗

Follow-up comment

Thank you again for the review. Before the discussion period ends, we wanted to point briefly to the two additions that respond most directly to your main concerns: the argument that the two conventions are mutually exclusive power-law assumptions rather than relabellings, so that in-sample fit does not reliably tell them apart but they extrapolate differently (Sections 4.1–4.2), and the new over-training analysis, which shows that the convention barely affects decisions within a ladder’s model sizes and matters when the law is used to plan larger models (Section 4.5). We have also uploaded a further revision in response to Reviewer tXei’s follow-up comments. We hope these address your concerns, and we would be glad to clarify anything that remains unclear.

I appreciate the authors for the response. I like your new addition that takes inference-time cost into account. That's where the world is and the draft should be centered around that scenario.

I will maintain my positive-ish score.

We thank the reviewer for the follow up and are glad about the inference-aware analysis was useful. We also agree this is practically more relevant setting, and in the final version we will put more emphasis on it per your suggestions.

Reviewer_fN6bOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper refits 18 public dense language-model scaling sweeps (1,673 runs from six research groups) whose metadata determine the vocabulary-matrix parameter count exactly, once counting and once excluding those matrices, and reports that counting them raises the fitted compute-optimal data-model exponent r = d log D*/d log N* in every sweep and group (mean +0.54 in lower-80%-compute fits of the non-separable form, 95% CI [+0.34, +0.73] over six groups; the separable form is positive in 18/18 sweeps, range +0.12 to +1.41), that public held-out tests do not consistently identify which count is right, that harmonising conventions does not detectably reduce the between-lab spread in r, and that the two conventions imply a median 2.3x difference in tokens per parameter at 100x the fitted compute, together with released reconstructions, a source registry and a reporting standard (Table 3).

Critical strengths

A careful, well-scoped measurement. There is a fixed inclusion rule and a registry of every source examined and why it was excluded (Sec 3, App. A). Exact parameter decompositions are asserted in released code, and a single command regenerates every number from pinned public data. The convention comparison is paired within each sweep (same runs, same fit, same number of free parameters, only N redefined), so cross-lab differences in data, recipe and design do not enter delta-r. The mechanism is checked directly: alpha-hat rises in all 18 Chinchilla-form fits while beta-hat moves by a median of 1.2%.

Negative and ambiguous results are reported plainly. The largest-model holdout splits the groups (Sec 4.2). Sec 4.3 finds no evidence that the convention explains cross-lab disagreement. App. C concedes the non-separable form does not extrapolate better. The Delphi example (r-hat = 1.57 under total vs 0.81 under transformer-only) shows the practical stakes. The recommendation to report allocations in physical units and to state the N convention (Table 3) is concrete and something I would use.

Critical weaknesses
  1. The sign of the headline shift is largely expected by construction, and delta-r mixes several effects. Sec 4.1 derives alpha_total = alpha/(1 - (1 - gamma)s), which exceeds alpha whenever gamma < 1, and verifies 0 <= gamma <= 0.53 in all 18 sweeps. Applied to the optimal path, the same chain rule gives r_total = r_core/(1 - (1 - gamma)s): the same allocation path, re-expressed against a different x-axis. In addition, the two r-hat values come from two different fitted surfaces, and each is the optimum under C = 6ND with that convention's own N, so they are optima under different compute constraints. "All 18 sweeps / all six groups positive" therefore mainly confirms the paper's own prediction. The informative results are the magnitude of the shift in real ladders, which the paper usefully measures, and whether the fitted surfaces prescribe different allocations on a common compute axis (Table 2). Please decompose delta-r into the pure x-axis part (computable from gamma and s along each fitted path) and the remainder, and centre the headline on the remainder and on allocations, as Sec 5 itself recommends.

  2. The common compute axis in Table 2 is only approximately FLOP-faithful, and the allocation claim depends on it. C = 6N_totalD omits attention FLOPs. For untied ladders (Marin, OLMo, (Mis)Fitting) it also charges matmul FLOPs for the input embedding, an overcount that is largest at the small-model end where the fits are anchored. Sec 3 calls core+head "the FLOP-faithful count", and Table 3 asks authors to state head and attention FLOPs, yet Table 2 uses neither; core+head appears only as an App. B diagnostic. The allocation gaps (median TPP ratio 1.39x/2.31x/4.71x at 10x/100x/1000x; excess compute 7.1%/11.4% at 100x) should be recomputed on a core+head+attention FLOP axis where the head is identifiable, with a stated range for Kempner. They should also carry bootstrap intervals that include the uncertainty of the extrapolated N_vocab = a*N_total^b mapping.

  3. The inference rests on very few effective units. In the largest-model holdout, OLMo (five sizes) and (Mis)Fitting (one slice) contribute noisy ratios (0.37, 1.58) that decide the three/three split. Delphi's frontier holdout has seven rows at three budgets, four of them seed replicates. The dispersion test's CI on the SD ratio is [0.52, 3.98]. Sec 4.3 also compares "released" against core for the non-separable form (0.25 vs 0.27), but total against core for the separable form (ratio 0.50). The comparisons should be like-for-like, ideally with a direct variance decomposition, which would show that a near-uniform shift cannot by itself reduce spread.

  4. The abstract says the +0.54 shift holds "under both separable and non-separable loss forms". The group mean and CI are given only for the non-separable relaxation. For the separable form only "18 of 18, range +0.12 to +1.41" is given, and App. C evaluates the relaxation on a different 15-sweep panel, where it does not extrapolate better than Chinchilla. Please report the separable-form group mean and CI alongside the non-separable ones.

Notes on the Presentation, Informativeness and Soundness scores

Presentation: The paper is well organised and candid about its limits, and the negative and ambiguous results are reported in the main text rather than in an appendix, which I appreciate.

It is dense: Sec 4.1 and 4.2 pack many statistics into single sentences, and the units of analysis (group, slice, codebase) are easy to lose track of. A compact table listing every test with its unit, n, statistic, sidedness and p-value would help considerably. The denominators also vary (18 sweeps in the text and Table 1, 17 primary slices in Figure 1 and Table 2, 17/17 in the held-out column); the Figure 1 caption explains the 18th slice (Delphi, which adds +0.76 and is fitted on its own split), but stating this once, early, would avoid confusion. The Table 1 caption states that the held-out column uses lower-80%-compute fits, but the column label "Held-out delta-r" on its own reads as a held-out measurement, and because the r-hat columns are all-data fits, r-hat total minus r-hat core does not equal that column (e.g. (Mis)Fitting 0.63 vs 0.28); a clearer column label would fix this.

Informativeness: Measuring the size of the convention effect across many modern public ladders is useful, and the registry of examined sources and the reporting checklist are practical contributions that I would use.

The informativeness is limited by two things. First, the direction of the shift follows from the chain-rule argument the authors give themselves in Sec 4.1, so the universality of its sign adds less than the headline suggests; the magnitude in real ladders and its decision consequences are the informative parts. Second, the question practitioners most need answered, which count actually governs scaling, is left open, and the released vocabulary-intervention runs that could address it (Tao et al., 2024) are excluded from the panel under criterion (b). Relative to Pearce and Song (2024) and Porian et al. (2024), the contribution is breadth of measurement and a reporting standard rather than a new mechanism.

Soundness: The internal arithmetic is consistent wherever it can be checked from the reported tables: the six group values in Table 1 give a mean of 0.537 with a t-interval of about [0.34, 0.73]; the 17 slice values in Figure 1 average +0.59; the four-codebase mean of +0.60 is reproduced with slice weighting within codebases (about +0.57 without it); and the between-group standard deviation of r-hat for core (about 0.27) matches Sec 4.3; the paper's 0.25 is for the conventions each source released, not for total.

My concerns are about what is being estimated rather than about the computations. (1) r is defined relative to each convention's own N, so the sign of its shift is largely predicted by construction (Sec 4.1); moreover delta-r mixes an x-axis relabelling with two different fitted surfaces and two different compute constraints, which are not separated. (2) The common compute axis used for the allocation results, C = 6 N_total D, is only approximately FLOP-faithful: it omits attention FLOPs and, for the untied ladders (Marin, OLMo, (Mis)Fitting), charges input-embedding lookups as matrix-multiply compute. (3) Several held-out comparisons rest on very few model sizes or budgets (OLMo has five sizes, (Mis)Fitting one slice, and the Delphi frontier holdout seven rows at three budgets, four of them seed replicates), and no per-group uncertainty is given for the RMSE ratios. (4) The Sec 4.3 dispersion test has little power with six groups (ratio CI [0.52, 3.98]). These seem fixable, hence a score of 2 rather than 1.

Questions for authors
  1. In Sec 4.3, which convention does "the conventions each source released" mean for each of the six groups? Why does the non-separable comparison use released vs core while the separable one uses total vs core?

  2. For the four-codebase mean (+0.60), are slices weighted within a codebase? Averaging the Table 1 held-out group values by codebase gives about +0.57; slice weighting gives about +0.60.

  3. For OLMo (25 runs, five sizes) and (Mis)Fitting (81 runs), how many distinct sizes are held out and how many remain for fitting the six-parameter surface?

  4. What fraction of fits end at the bounds of the NLS search? Please report per-sweep bootstrap intervals for r-hat and delta-r, to support "the same fits and folds estimate delta-r precisely" (Sec 4.2).

  5. Can you redo Table 2 and the dispersion analysis on a core+head+attention FLOP axis, and report the between-lab spread of allocations (e.g., tokens per parameter at fixed FLOPs) under each convention?

  6. App. A excludes Tao et al. (2024) because "the released runs fix non-vocabulary parameters and vary vocabulary size". That is the controlled intervention that Sec 4.2 and Sec 6 say is needed. Can those released runs be used to test which count better predicts held-out loss? If not, why not?

  7. How much of each delta-r is the pure x-axis term r_core*[1/(1 - (1 - gamma)s) - 1] evaluated along the fitted path, and how much comes from the different fitted surfaces and compute constraints?

4 comments
AuthorsOpenReview ↗

Response to Reviewer fN6b (1/3)

(1/3)

Thank you for an exceptionally careful review, including re-deriving our group statistics from the tables. Your suggestions changed the paper's framing, and we believe for the better.

How to read this reply: we label your four weaknesses W1–W4 and your seven questions Q1–Q7 in the order they appear in your review. Parts 1–2 cover W1–W4 (with Q5 and Q7, which overlap them) and Presentation; part 3 covers Q1–Q4, Q6 and Informativeness. Your Soundness points (1)–(4) are W1, W2, W3 and the dispersion paragraph under W3.

W1 / Q7: decompose Δr. Done; see the table in the general comment. Within the fitted range, Δr is mostly relabelling (+0.40) and the change of compute constraint (+0.31), while the fitted surfaces barely differ (−0.16, CI includes 0). Averaged over 10–1000× the fitted budget those terms fall to +0.04 and +0.06 (combined, +0.19 at 10× and +0.04 at 1000×), and the surface term is +0.45 [+0.27, +0.64] (17/17 slices, 6/6 groups). We now lead with this remainder and with allocations, alongside the measured size of the shift, as you suggested, and we state explicitly that the surface term's dominance at small vocabulary share is expected by construction. Under the separable form, the frontier surface term is +0.51 [+0.01, +1.01] (5/6 groups, one-sided p = 0.047).

W2 / Q5: FLOP-faithful compute. We recomputed Table 2 with C = 6·D·(N_core + N_head) + 12·D·n_layers·seq_len·d_model. This count matches Marin's measured FLOPs to within 0.2% on all 258 IsoFLOP runs and Kempner's released FLOP column exactly; attention is up to 35% of compute for the smallest models. The intervals are 95% shape-bootstrap intervals that refit the vocabulary, head and attention mappings. Because the bootstrap is biased upward (only 28–41% of resamples fall below the point estimate), we report BCa intervals. Kempner's head is now identifiable (see our response to tXei), so no separate range is needed for it.

(TPP ratio, total ÷ core [BCa 95%]; excess compute, core ref. / total ref.; slices in the same direction)

  • Non-separable, 10×: 1.40 [1.21, 1.66]; 2.0% [0.7, 7.0] / 2.6% [1.3, 13.1]; 14/17
  • Non-separable, 100×: 2.39 [2.04, 3.32]; 8.2% [6.2, 10.7] / 14.4% [9.1, 60.0]; 14/17
  • Non-separable, 1000×: 4.89 [3.57, 9.83]; 26.9% [17.8, 50.2] / 59.0% [33.9, 216]; 17/17
  • Separable, 10×: 1.46 [1.20, 1.71]; 9.4% [6.4, 13.0] / 9.7% [4.2, 13.5]; 11/17
  • Separable, 100×: 2.64 [1.84, 6.25]; 11.4% [10.3, 13.2] / 15.6% [11.4, 20.0]; 12/17
  • Separable, 1000×: 4.64 [2.85, 7.25]; 25.5% [16.5, 39.1] / 37.3% [17.8, 77.9]; 12/17

No optimum lies on the search boundary. The bridge from the submitted Table 2 (2.31× at 100×) is in the general comment, point 2: most of the change comes from the Gadre correction, and on a fixed panel the change of axis moves the ratio only from 2.46× to 2.39×. Under the separable form, slices disagree in both directions, so the direction-free gap is larger (5.4× at 100×) while the sign count is weaker.

Between-lab spread of allocations (Q5). At 10²¹ FLOPs and without Kempner, the spread of log tokens-per-parameter across groups is about the same under both conventions (SD ratio, total ÷ core, 1.15 [0.92, 1.51]). We exclude Kempner here because its transformer-only path leaves the physical range: it prescribes 0.2 tokens per parameter at 10²³. Including it makes total counting look less dispersed (total ÷ core 0.65 [0.28, 1.24]), and we do not rely on that result. Allocations, like exponents, do not converge under a common convention.

AuthorsOpenReview ↗

Response to Reviewer fN6b (2/3)

(2/3)

W3: few effective units. The revision states each unit explicitly in a new table listing every test with its unit, n, statistic, sidedness and p-value. In the largest-model holdout, (Mis)Fitting fits 8 of 11 sizes (3 held out). OLMo fits only 4 sizes (20 runs) with 1 held out, which is thin for a six-parameter surface; we now say so and give its 0.37 ratio little weight. We now give per-group bootstrap intervals for the largest-model RMSE ratios (total ÷ core; resampling training model sizes and refitting): Marin 1.93 [1.57, 2.55]; (Mis)Fitting 1.58 [0.65, 2.38]; Kempner 0.34 [0.32, 0.44]; OLMo 0.37 [0.18, 8.5]; OpenLM 1.50 [0.76, 1.90]; Farseer 0.98 [0.77, 1.69]. Two of the six exclude 1, in opposite directions: Marin favours transformer-only counting and Kempner favours total counting; the other four are inconclusive, OLMo especially so. The Delphi frontier holdout has 7 rows at 3 budgets, 4 of them seed replicates, as the paper states. We no longer present it as an isolated exception: the Farseer reanalysis, the Tao intervention and Kempner's 3.3B frontier runs point the same way, while the new allocation backtest favours total counting in Kempner's sweeps (general comment, point 3).

Dispersion, like for like. The dispersion section (§4.4 of the revision, §4.3 as submitted) now uses one starting convention for both forms. With total → core for both forms: under the non-separable form the SD ratio is 1.10 [0.69, 1.45]; under the separable form it is 0.51 [0.42, 0.97], a variance ratio of 0.26 [0.18, 0.95]. Your point that a near-uniform shift cannot by itself reduce spread is exactly what the variance decomposition shows: Var(r̂_core) = Var(r̂_total) + Var(Δr) − 2·Cov(r̂_total, Δr). Non-separable: 0.074 = 0.061 + 0.013 − 2·0.000. The shift is uncorrelated with the exponent, so harmonizing can only add variance. Separable: 0.156 = 0.593 + 0.169 − 2·0.303. Groups with larger exponents shift more, which removes absolute spread. Relative to the mean exponent, the spread rises under the non-separable form and does not change detectably under the separable form (details in our response to tXei, Q3). §4.4 now names the measure.

W4: separable-form group mean. +0.72 [+0.19, +1.24] on lower-80% fits (+0.70 [+0.27, +1.14] on all data); 6/6 groups, one-sided p = 0.016. The shift is now reported with a group-level interval under each form.

Presentation. We now state the 17-vs-18 slice convention once in §3 (Delphi is fitted on its own split). We relabel the column "Δr from lower-80% fits," and add the tests table.

AuthorsOpenReview ↗

Response to Reviewer fN6b (3/3)

(3/3)

Q1: released conventions. Marin: total (parameter_count includes both untied matrices). Kempner: total (params). OLMo: core + head (released MODEL_PARAMS exclude only the input embedding). Farseer: transformer-only (exact_N). OpenLM: Gadre et al. total (params, the count their fitting code uses); Porian et al. core + head (their standard count includes the output head only). (Mis)Fitting: total (N). The OpenLM group therefore mixes two released conventions. Using released conventions for one form and total for the other was an inconsistency on our part; the like-for-like comparison above replaces both.

Q2: codebase mean. You are correct: the codebase mean is slice-weighted within codebases (+0.60 as submitted; +0.61 after the Gadre correction described in the general comment). Equal weighting of groups within each codebase gives +0.575, consistent with your figure; we now report both.

Q3: held-out sizes. See W3.

Q4: fit diagnostics. All grid starts converge to the same optimum in every non-separable fit (r̂ varies by ≤ 0.0005 among near-optimal starts). E sits at its lower bound in 11 of 34 fits (the OpenLM-codebase sweeps); freeing it changes r̂ by ≤ 0.002. No other parameter ends at a bound, under either loss form. Per-sweep bootstrap intervals for Δr exclude zero in 17/17 slices; median 95% widths are in Appendix D (0.56 for r̂ under total counting, 0.22 under transformer-only, 0.36 for Δr; non-separable form), and per-sweep intervals are in the supplementary package (results/fit_diagnostics.json). The Appendix C correction (11/14 → 10/14) is described in the general comment.

Q6: Tao et al. Thank you for this suggestion. First, a correction: our registry gave the wrong reason for excluding these runs. They vary six transformer sizes, so criterion (b) is met. The correct reason is criterion (c): they use a constant learning rate, with no checkpoint annealed to its own budget. The registry is corrected. These runs are a controlled vocabulary intervention: 60 runs, 6 transformer sizes (33M–1.13B) × 10 vocabularies (4,096–96,256), untied. They stay outside the primary panel for that reason. Your question can still be answered with them. Fitting on 9 vocabularies and predicting the 10th, with characters as the data axis and Tao et al.'s normalised loss:

  • Transformer-only counting beats total counting in 9 of 10 held-out vocabularies (RMSE 0.113 vs 0.123; two-sided Wilcoxon p = 0.004, descriptive because folds share training data).
  • Held-out error is higher than at λ = 0 for every positive vocabulary weight we tried (up to 2), where N_eff = N_core + λ·N_vocab. The best-predicting weight is slightly negative (λ ≈ −0.25, the value that minimises held-out error; it is the minimiser in every vocabulary resample on a 0.05 grid, and N_eff stays positive down to −0.34, so it is an interior minimum), consistent with the U-shaped dependence on vocabulary size that Tao et al. report. The one fold core loses is the smallest vocabulary.
  • So vocabulary effects in these runs are not captured by counting vocabulary parameters. The result is unchanged across the constant used to make the normalised loss positive (+10, +20, +40). With final checkpoints only (60 rows), the comparison is inconclusive. The separable form fits these data poorly (parameters at their bounds in most fits, 449 of 470), so we report the non-separable result.

On informativeness. Pearce & Song (2024) and Porian et al. (2024) explain one historical discrepancy. The revised paper establishes three things neither does: (i) the two conventions are mutually exclusive power-law assumptions, and the decomposition shows exactly where the choice is bookkeeping (within the fitted range) and where it changes decisions (in extrapolation to larger models); (ii) on observed data, a 2.2× extrapolation already moves the predicted compute-optimal size by about 1.4×, and the observed optimum distinguishes the conventions (Kempner, 6/6 corpora); (iii) which convention extrapolates better depends on the type of test: frontier-scale and controlled tests mostly favour transformer-only counting (Delphi, Farseer, Tao, Kempner's 3.3B runs), while near-range tests are mixed and Kempner's allocation backtest favours total, so no universal convention can be adopted, and it must be reported. The breadth of measurement and the reporting standard support these claims rather than being the whole contribution.

AuthorsOpenReview ↗

Follow-up comment

Thank you again for the detailed review. Before the discussion period ends, we wanted to point briefly to the two analyses that respond most directly to your main questions: the decomposition of the shift into axis relabelling, change of compute constraint and differences between the fitted surfaces (W1/Q7; Section 4.2), and the allocation comparison on a FLOP-faithful compute axis under both loss forms, with BCa intervals (W2/Q5; Table 2). Appendix D collects the supporting analyses, including the backtest on observed optima and the frontier holdouts. We have also uploaded a further revision in response to Reviewer tXei’s follow-up comments. We hope these address your concerns, and we would be glad to clarify anything that remains unclear.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers (1/3)

(1/3)

Summary of changes and new analyses

We thank all three reviewers for careful, specific reviews. Several of their suggestions led to new analyses that sharpen the paper's claims, and two led to factual corrections.

What the revised paper claims.

  1. Transformer-only and total counting are mutually exclusive power-law assumptions. In-sample fit does not reliably indicate which one extrapolates better, and within the fitted range the exponent shift is bookkeeping.
  2. The two conventions prescribe different allocations for larger models, and observed data can detect this: holding out the top IsoFLOP budget in Kempner's sweeps, a 2.2× extrapolation moves the predicted compute-optimal size by about 1.4×, and the observed optimum separates the conventions in 6/6 corpora.
  3. No test establishes either convention universally: frontier-scale and controlled tests mostly favour transformer-only counting and near-range tests are mixed, so the convention must be reported rather than assumed.

The revised abstract now reads: "Within the fitted range this shift is accounted for by a change of coordinates and of the compute constraint. The two conventions are nevertheless mutually exclusive power-law assumptions, and in-sample fit does not reliably indicate which extrapolates better. Extrapolated, they prescribe different allocations: a median 2.4× difference in tokens per parameter at 100× the fitted compute on a FLOP-faithful axis, and in one group's IsoFLOP sweeps a held-out backtest detects the difference after only a 2.2× extrapolation. Public data do not settle which convention extrapolates better: the available frontier-scale and controlled tests mostly favor transformer-only counting, while near-range tests are mixed.". Details follow; reviewer-specific responses are posted separately.

1. What Δr measures (fN6b, bY2q, tXei). All three reviewers note that the sign of Δr follows from the chain-rule argument in §4.1. It does, and the question is where the shift reflects more than bookkeeping. Following fN6b, we decompose each Δr into (a) pure x-axis relabelling, (b) the change of compute constraint, and (c) the difference between the two fitted surfaces on matched axes:

(Within the fitted range | averaged over 10–1000× the largest fitted budget)

  • Relabelling: +0.40 [+0.18, +0.61] | +0.04 [+0.01, +0.06]
  • Compute constraint: +0.31 [+0.18, +0.44] | +0.06 [+0.02, +0.10]
  • Fitted surfaces: −0.16 [−0.50, +0.19] | +0.45 [+0.27, +0.64], 17/17 slices, 6/6 groups

By budget, the bookkeeping terms (relabelling + constraint) fall from +0.19 at 10× to +0.09 at 100× and +0.04 at 1000×, while the surface term rises from +0.35 to +0.46 and +0.51.

The decomposition is itself a finding: within the fitted range, the change of coordinates and compute constraint accounts for the whole shift. So exponents published under different conventions differ by about 0.55 for reasons unrelated to scaling behaviour, and cannot be compared directly. We no longer present the universality of the sign as a finding, but the measured size of the shift (+0.55, comparable to the Kaplan–Chinchilla gap) is unchanged and remains a primary result. The two fits nevertheless commit to different power laws. Where the vocabulary share has fallen (from 12–99% along the fitted paths to at most 43% at 10× and 13% at 1000×; the maximum is Porian-RW on the transformer-only path, whose compute-optimal models stay small; others are lower), the bookkeeping terms vanish and the surface term remains. That this term tends to Δr as the vocabulary share goes to zero is expected by construction; the empirical content is how fast the bookkeeping terms vanish (above) and that fits which public held-out tests do not consistently separate (§4.3 of the revision) prescribe different frontier allocations.

2. Allocations on a FLOP-faithful axis (fN6b, tXei). We recount training compute as transformer and LM-head matmuls plus attention, with the input-embedding lookup free. This count reproduces Marin's measured FLOPs on all 258 IsoFLOP runs to within 0.2% and matches Kempner's released FLOP column exactly. At 100× the fitted compute, the two conventions' fits prescribe a median 2.4× difference in tokens per parameter (BCa 95% [2.0, 3.3]), in the same direction in 14/17 slices. The separable form gives 2.6× (12/17). From the submitted Table 2 to the new numbers, at 100×:

(TPP ratio; excess compute, core / total reference)

  • Submitted (6·N_total·D, original panel): 2.31×; 7.1% / 11.4%
  • Corrected panel, same 6·N_total·D axis: 2.46×; 8.3% / 14.1%
  • Corrected panel, FLOP-faithful axis: 2.39×; 8.2% / 14.4%

Most of the change comes from the Gadre correction (point 4), not the change of axis; the embedding overcharge and the omitted attention term roughly cancel.

AuthorsOpenReview ↗

General response to all reviewers (2/3)

(2/3)

3. Does the convention change decisions that can be checked, and which convention extrapolates better? (all reviewers) We added a direct backtest of the allocation itself. In IsoFLOP sweeps we fit each convention on the lower budgets, predict the compute-optimal model size N* at the held-out top budget, and compare the prediction with the observed optimum.

  • Kempner (six corpora), after only a 2.2× extrapolation: the two conventions' predictions of N* differ by about 1.4×, and the observed optimum separates them in 6/6 corpora. Total counting is closer (median error 0.021 dex vs 0.17 dex for transformer-only).
  • Marin: the three conventions' predictions are too close relative to the noise in the observed optimum to separate them.

The result is the same when the top two budgets are held out and under the separable form. So the convention changes the allocation by an amount that observed optima can detect, even over short extrapolations.

On which convention extrapolates better, the evidence depends on the type of test:

  • Toward transformer-only counting:
    • Marin, in both Delphi's frontier holdout (RMSE 0.026 vs 0.091) and the largest-model holdout (ratio 1.93 [1.57, 2.55]).
    • Farseer, in a matched reanalysis of its released 2.3B–25.1B validation runs (modest margin).
    • The one public controlled vocabulary intervention (Tao et al., 2024), where adding vocabulary parameters to N at every positive exchange rate we tried (up to 2) worsens held-out-vocabulary predictions (9/10 folds). These runs use constant-learning-rate intermediate checkpoints, which is why they are outside the primary panel, and with final checkpoints only the comparison is inconclusive.
  • Kempner is mixed. Its near-range tests favour total counting: the allocation backtest, the largest-model holdout (ratio 0.34 [0.32, 0.44]) and the top-20% compute holdout. Its joint small-to-large split favours transformer-only counting (6 of 6 corpora). Its release also contains one 3.3B run per corpus at 10²¹ FLOPs, about 20× the largest sweep budget and trained with the same configuration, which our submitted panel did not use. Fitted on the lower 80% of compute (the same protocol as the headline; make kempner-frontier, entries lower80), transformer-only counting predicts these runs better in 5 of 6 corpora (mean error 0.072 vs 0.112; the exception is SlimPajama), and in 6 of 6 under the separable form (0.063 vs 0.131), although total counting fits the sweep better in-sample. The 2.2× backtest stays within the observed range (11 of 12 predicted optima lie within the observed model sizes); Kempner's transformer-only path becomes unphysical only far beyond the data (0.2 tokens per parameter at 10²³ FLOPs), which is why we exclude it from the 10²¹ spread comparison (our reply to fN6b, W2).

The paper now says that no test establishes either convention universally. The frontier-scale and controlled tests mostly favour transformer-only counting (Delphi; Farseer, by a modest margin; Tao, outside the primary panel; Kempner's 3.3B runs, 5 of 6 corpora), while the near-range tests are mixed and Kempner's allocation backtest favours total counting. In-sample fit does not reliably identify the better extrapolator. We leave explaining this pattern to future work. To be explicit: "no universal convention" does not mean "the convention does not matter." The two conventions prescribe similar allocations within the range a ladder covers, but they prescribe different compute-optimal models as soon as the law is extrapolated. Observed optima detect the difference after a 2.2× extrapolation (Kempner), and frontier models are planned by extrapolating far further (e.g., about 3,800× for Llama 3). Because no single convention can be assumed, the convention used must be reported.

AuthorsOpenReview ↗

General response to all reviewers (3/3)

(3/3)

4. Corrections we found ourselves. While re-auditing every vocabulary reconstruction in response to tXei, we found that the released Gadre et al. models are untied and that our vendored table used a count that included the head but not the input embedding. The paper's statement that Gadre's released N contains one 50,432·d matrix described our vendored table, not the release: the released params is the total count. We also restored nine 1B/7B Gadre runs, which our loader had silently dropped because it read model width from the run name and those runs' names do not encode it; their counts are now taken from the released per-model records. The panel is now 1,682 runs. No conclusion changes: the group-mean Δr is +0.55 [+0.35, +0.75], 6/6 groups, one-sided p = 0.016.

Separately, and unrelated to the Gadre fix: in Appendix C's interpolating compute-stratified folds, the non-separable form now improves held-out RMSE over Chinchilla in 10 of 14 sweeps (median +7.4%, CI [−2.6%, +13.5%]), not 11 of 14. We traced the change to library versions after an environment rebuild; 300 extra random starts per fit reproduce the new optima exactly, and versions are now pinned. This is a different test from the joint small-to-large split tXei cites, which is unchanged (the non-separable form loses on all 11 evaluable sweeps). Farseer's and Kempner's descriptions are corrected (see the response to tXei); no previously reported number changes.

The revised PDF adds a new §4.2 (what the shift measures), so later subsections move by one (held-out tests §4.3, dispersion §4.4, allocations §4.5), and a new Appendix D with every new table; all affected numbers are updated. Changed and new text in the revised PDF is marked in blue. The revised supplementary zip replaces the submitted one: CHANGES.md lists every correction and addition, and CLAIMS.md maps each number in the paper and in this response to the file that stores it and the command that regenerates it (pip install -r requirements.txt, then e.g. make decomposition or make kempner-frontier; make all runs everything).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.