- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how the choice of parameter count affects fitted compute-optimal scaling laws. It reconstructs parameter counts for 18 public sweeps, covering 1,673 runs from six research groups, and compares fits using total parameters and transformer-only parameters.
Including vocabulary-related parameters increases the fitted data–model exponent in every primary sweep. Across the six groups, the mean increase from fits to the lower-compute subsets is about 0.54. However, held-out prediction does not consistently favour one counting convention across groups.
The paper also examines differences between labs and shows that the fitted conventions can recommend quite different model and data allocations at larger compute budgets. It proposes clearer reporting of parameter counts, compute definitions and recommended allocations.
Critical strengths
-
The main strength is the systematic comparison across several sources. The effect of parameter counting was already known, but this paper checks it across a reasonably broad collection of public results. The source registry and reconstruction of parameter counts are useful contributions. The paper also keeps sources that do not meet its primary inclusion rules separate from the main analysis.
-
The paired comparisons are carefully designed. The authors use the same runs, fitting procedure and loss form when comparing the two counting conventions. The exponent increases under both loss forms, which gives confidence that the main observation is not specific to one form.
-
Grouping related sweeps by research group and codebase is sensible. It avoids treating all sweeps as fully independent evidence. I also appreciate that the paper distinguishes disagreement between predicted allocations from compute savings demonstrated through actual training.
Critical weaknesses
-
The connection between the reported exponent and the compute constraint needs more clarity.
Section 3 defines the data–model exponent as , or for the separable form. This expression is valid under , using the same as in the fitted loss.
Section 4.4 instead optimises both fitted surfaces under the shared constraint . For a fit expressed in core parameters, the mapping to total parameters is nonlinear. The slope of the resulting optimal allocation curve therefore need not equal the reported exponent ratio. Both quantities can be useful, but their difference needs more clear explanation. At present, it is difficult to tell which conclusions concern a fitted exponent and which concern allocation under a common compute budget.
There is a related issue with calling this compute axis “physical”. Section 3 describes core plus output-head parameters as the FLOP-faithful count, whereas Table 2 uses total parameters. For untied embeddings, counting the input lookup table in assigns it the same cost per parameter as dense matrix operations. Porian et al. also discuss the output head and attention costs separately. A shared compute approximation is useful for comparison, but the paper should explain how closely it represents the actual training cost.
Finally, an exact change of parameter coordinates, with the same underlying loss and compute functions, would preserve the physical optimum. Here, the loss forms are fitted again after changing the parameter count. The different recommendations can therefore reflect changes in the fit and its extrapolation also. This distinction is important for interpreting the practical claims.
-
The allocation and between-lab conclusions need stronger support than the main exponent-shift result.
The reported 2.3× difference in tokens per parameter and approximately 7–11% excess-compute estimates are important practical claims. However, Table 2 does not clearly identify which loss form gives these numbers or show whether they remain similar under the other form. This needs clarification, particularly because Section 6 states that none of the conclusions depend on the non-separable form.
Appendix C reports that the non-separable form loses to the separable form in all 11 evaluable joint small-to-large extrapolation tests. This is a separate diagnostic panel, including non-primary sources, so it does not establish that the main allocation estimates are wrong. Still, it raises a reasonable concern about extrapolation sensitivity, especially when Table 2 goes much further beyond the observed budgets. Please specify the loss form used and show how sensitive the allocation conclusions are to this choice, the compute calculation and the vocabulary-count mapping.
The conclusion about differences between labs also needs some care. Section 4.3 reports a standard-deviation ratio interval of [0.52, 3.98]. This interval allows a reduction of about 48% in standard deviation, or an increase to nearly four times the original value. If the disagreement is measured by variance, the lower endpoint allows a reduction of about 73%. So the statement that the convention does not account for “most” of the spread needs to specify the measure and reflect this uncertainty.
The two loss forms also use different starting points for this comparison. The non-separable analysis moves from each source’s released convention to core counting, whereas the separable analysis moves from total to core counting. In the latter case, the absolute spread halves, while the spread relative to the mean remains broadly similar. These results need a matched comparison before drawing a general conclusion about disagreement between labs.
Presentation
Some terms need more precise wording. “Which count is right” is too broad for what the held-out tests establish. These tests compare counting conventions within particular fitted loss forms; they do not identify a parameter definition that is best for every purpose.
“Held-out ” is also slightly confusing. The exponent is fitted on the lower-compute training subset, while loss prediction is evaluated on held-out runs. “ from lower-compute fits” would make this clearer.
Please use different symbols for vocabulary size and the number of parameters in vocabulary matrices. Both are represented by in different parts of the paper. A short explanation connecting the fitted exponent, the compute constraint and the allocation curve would also help readers who are less familiar with scaling-law fitting.
Questions for authors
-
Which compute constraint defines the reported ? Please distinguish this quantity from the slope obtained when both fitted surfaces are optimised under the same compute constraint. For a direct comparison, can you report the slope for both fits at common compute budgets? This would help separate the effect of changing the parameter coordinate from differences between the fitted allocation curves.
-
Which loss form produces the estimates in Table 2, and do the conclusions remain similar under the other form? Where the source information permits, can you also check a compute calculation that accounts separately for the output head, attention and input embedding lookup? Please show how sensitive the allocation differences and excess-compute estimates are to the vocabulary-count mapping beyond the observed model sizes.
-
Can the between-lab comparison use the same starting convention for both loss forms? Please report uncertainty for both absolute spread and spread relative to the mean. This would make it easier to judge how much disagreement is reduced by using a common parameter convention.
-
Can you clarify the vocabulary-matrix reconstruction for Farseer and Kempner? Farseer reports a vocabulary of 65,536, and its parameter-count difference is consistent with two matrices of this size. The manuscript describes one 131,072-row matrix. The total subtraction may be correct, but the distinction matters when separating input and output costs. The public Kempner base configuration specifies a vocabulary of 32,000 with weight tying disabled. If these settings apply to the analysed runs, why is the input/output split treated as unidentifiable?
-
Farseer’s Appendix G already compares fits with and without embeddings at a 25.1B validation point, using its own scaling-law formulation. It reports more than four times smaller prediction error for the embedding-excluding fit. How does this relate to the discussion of Delphi as the exceptional frontier evidence? This is not a matched reanalysis, but it seems relevant to the discussion. If the necessary data are not available for a matched comparison, please explain this also.
4 comments
Response to Reviewer tXei (2/2)
(2/2)
Q2: which loss form, compute calculation and sensitivity. The submitted Table 2 used the non-separable form; we now report both. We replaced 6·N_total·D with a FLOP count that charges the head and attention and treats the input embedding as a lookup. It matches Marin's measured FLOPs to within 0.2% and Kempner's released FLOP column exactly, so we now call it a "FLOP-faithful" axis rather than "physical." The intervals refit the vocabulary, head and attention mappings beyond the observed sizes.
- At 100×: 2.39× (non-separable; BCa [2.04, 3.32]; 14/17 same direction) and 2.64× (separable; BCa [1.84, 6.25]; 12/17).
- Excess compute at 100× (non-separable): 8.2% and 14.4% for the two reference conventions (submitted: 7.1% and 11.4%; the change comes mostly from the Gadre correction, see the bridge table in the general comment).
- The full table at 10×, 100× and 1000× under both forms, with BCa intervals, is in our reply to fN6b (W2).
On Appendix C: the allocation gap is of similar size under the separable form, which extrapolates no worse, so it is not an artifact of the relaxation.
Q3: matched dispersion comparison. With the same starting convention (total → core) for both forms:
- Non-separable — SD ratio: 1.10 [0.69, 1.45]; Variance ratio: 1.21 [0.47, 2.11]
- Separable — SD ratio: 0.51 [0.42, 0.97]; Variance ratio: 0.26 [0.18, 0.95]
Under the separable form, the absolute spread does fall significantly. The spread relative to the mean rises under the non-separable form (coefficient-of-variation ratio, core ÷ total, 2.03 [1.49, 2.71]), because the exponents fall while the absolute spread is unchanged, and does not change detectably under the separable form (1.11 [0.92, 1.96]). §4.4 of the revision (§4.3 as submitted) now names the measure, and the word "most" is replaced by these intervals.
Q5: Farseer Appendix G. Thank you for pointing this out. Farseer's own validation favours the embedding-excluding fit at 25.1B. Because the seven validation runs (2.3B–25.1B) are released on the same metric as the fitting set, we ran the matched reanalysis you suggest. Transformer-only counting predicts the held-out large models best under both forms: mean relative error 0.0082 vs 0.0105 (non-separable) and 0.0324 vs 0.0346 (Chinchilla). That is the same direction as Farseer's appendix but a smaller typical gain (about 1.3×). At the single 25.1B point the gap is much larger (0.0001 vs 0.0128), but one point is noisy, so we base the comparison on all seven runs. Delphi is therefore no longer an isolated exception: with the Tao et al. intervention (response to fN6b, Q6), the frontier and controlled evidence favours transformer-only counting in Marin, Farseer and Tao. Kempner's released 3.3B frontier runs (about 20× beyond the sweep) also favour transformer-only counting, while Kempner's near-range holdouts and our new allocation backtest favour total counting (general comment, point 3). The revision therefore says that frontier-scale and controlled tests mostly favour transformer-only counting, that near-range tests are mixed, and that no test establishes either convention universally.
Presentation. We adopted all of your suggestions:
- "Which count is right" → "which convention extrapolates better," and the §4.3 heading now reads "Public tests do not establish either convention universally."
- "Held-out Δr" → "Δr from lower-80% fits."
- Separate symbols for vocabulary size (V) and vocabulary-matrix parameters (N_vocab).
- A short paragraph connecting the fitted exponent, the compute constraint and the allocation curve.
We believe these address the two concerns behind your rating: the distinction between fitted exponents and common-budget allocations, and the support for the allocation and between-lab conclusions. We would be glad to answer any remaining questions.
Thank you for the detailed response and the substantial revision. The revised manuscript addresses most of my original concerns, particularly the distinction between fitted exponents and common-budget allocations, and the support for the practical allocation claims.
The clarification in Section 3 and the decomposition in Section 4.2 are helpful. The manuscript now distinguishes the exponent under each convention’s own compute constraint from the allocation curve obtained under a shared constraint. Separating axis relabelling, changes in the constraint, and differences between fitted surfaces also makes the interpretation clearer. In particular, the practical disagreement is now appropriately connected to extrapolating different fitted power laws.
Table 2 now reports allocation differences under both loss forms, with uncertainty intervals. At 100× the fitted budget, the median tokens-per-parameter ratios are 2.39 and 2.64 under the non-separable and separable forms, respectively. This substantially addresses my concern that the allocation result might depend on the non-separable relaxation. The revised FLOP accounting and the explicit description of excess compute as a counterfactual estimate also improve the presentation. The new allocation backtest provides useful additional evidence that convention choice can affect decisions over relatively short extrapolations.
The corrected Farseer and Kempner descriptions, the disclosure of the additional Gadre corrections, and the inclusion of Farseer’s released validation runs address my reconstruction and frontier-evidence questions.
Two qualifications remain. First, the heading and opening of Section 4.4 should consistently distinguish absolute from relative dispersion. Under the separable form, the reported standard-deviation ratio of 0.51 [0.42, 0.97] supports a reduction in absolute between-group spread. Therefore, the broad statement that there is no evidence of reduced disagreement should be restricted to spread relative to the mean exponent.
Second, the distant allocation and excess-compute estimates remain conditional on the fitted loss forms and extrapolated architectural mappings. Refitting these mappings in the bootstrap is useful, but does not establish their validity at 100–1000× the observed budgets. Similarly, the clearest allocation-backtest result comes from Kempner; it supports decision sensitivity without establishing a generally preferable convention or validating the distant compute estimates.
Overall, the revision strengthens the paper sufficiently to change my assessment. The contribution is a useful systematic empirical audit, supported by clearer interpretation and additional validation. I am therefore increasing my overall rating from 2 (Weak rejection) to 3 (Weak acceptance).
Response to Reviewer tXei
Thank you for the careful re-reading and for raising your score. We agree with both qualifications and have revised the paper accordingly.
-
Absolute vs. relative dispersion. The §4.4 heading now reads "Harmonizing conventions does not reduce spread relative to the exponent", and its opening and closing restrict the claim to spread relative to the size of the exponents. The section states that the absolute between-group spread does fall under the separable form (SD ratio 0.51 [0.42, 0.97]), while the spread relative to the mean exponent does not detectably fall under either form. We made the same change to finding (iii) in the introduction and to the conclusion.
-
Conditionality of the distant estimates. We agree that the allocation and excess-compute estimates at 100–1000× are conditional on the fitted loss forms and on the vocabulary, head and attention mappings extrapolated beyond the observed sizes. The bootstrap refits these mappings but cannot validate them that far out. We also agree that the clearest backtest result comes from one group (Kempner): it shows that the convention changes decisions over short extrapolations, not which convention is preferable in general, and it does not validate the distant estimates. We now say this in Section 4.5 and in the Limitations.
We appreciate the reviewer's engagement throughout the discussion, which substantially improved the paper.
Response to Reviewer tXei (1/2)
(1/2)
Thank you for a careful and constructive review. Two of your points were factual corrections that we confirmed against the sources' own code; several others led to new analyses.
How to read this reply: we number your five questions Q1–Q5 in the order they appear. Part 1 answers Q4 (the reconstructions) first, then Q1; part 2 answers Q2, Q3 and Q5, then Presentation. Your weaknesses map onto these: exponent vs. constraint and the coordinate change (Q1), the compute axis and the allocation claims (Q2), and the between-lab comparison (Q3).
Q4: Farseer and Kempner reconstruction (corrections).
On "the effect of parameter counting was already known." Its existence was known from one historical comparison (Kaplan vs. Chinchilla). What was not known is that the two conventions are mutually exclusive power-law assumptions that are not reliably distinguished by in-sample fit but prescribe different allocations for larger models. That difference is detectable on observed optima after only a 2.2× extrapolation, and which convention extrapolates better depends on the type of test (general comment, point 3).
Q1: exponent vs. compute constraint. The revision now separates these explicitly. r̂ is the slope under each convention's own C = 6·N·D, while Table 2 optimises both fitted surfaces under one shared constraint. The new decomposition separates the two exactly: axis relabelling, change of constraint, and difference between fitted surfaces on matched axes (table in the general comment). On a common N_total axis and a common constraint (C = 6·N_total·D), the slope d log D*/d log N_total averaged over groups is 1.38 for the transformer-only fit and 1.23 for the total fit within the fitted range, and 0.77 vs 1.23 at 10–1000× the largest fitted budget (non-separable form). Put simply: the total fit's slope is 1.23 both within the fitted range and at the frontier, constant because it is a power law in N_total. The transformer-only fit, re-expressed on the same axis, bends from 1.38 to 0.77 as the vocabulary share falls. Your point that an exact change of coordinates preserves the physical optimum is the right lens. Within the fitted range the two fits nearly coincide on matched axes (surface term −0.16, CI includes 0). The practical disagreement comes from the fits extrapolating different power laws (surface term +0.45 at 10–1000×).