- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In one sentence: the paper studies how a model's test loss falls with more training data and more storage bits, and argues that both are set by one list: the energy of each fact the model must learn, roughly how much extra test loss the model suffers if it never learns that fact.
Problem:
Scaling laws usually ask how test loss falls with more training data or a bigger model. This paper replaces "bigger model" with "more bits": the total storage for what the model learned, roughly parameters × bits per parameter (the paper calls this "memory"). Its key idea is that some facts matter more than others. It measures this with each fact's energy (the paper's "predictive energy"): how often the fact is asked × how far its answer can be from a coin flip, squared. A fact's energy is about the extra loss you suffer if the model never learns it: guessing 50/50 on a near-certain "yes" costs a lot, while guessing on a near coin flip costs little. The list of all fact energies is the energy list (the paper's "energy spectrum"). With limited data, the model can only learn the high-energy facts; with limited bits, it should store the high-energy facts first.
Goal:
The paper wants a formula for the lowest excess test loss (loss above the best possible) that any model can guarantee with n training examples and B bits, computed from the energy list, whatever its shape (not only the usual power law). From it, the paper wants to explain how fast test loss falls as data or bits grow (the scaling exponents), and to show that a simple attention layer trained by SGD reaches this lowest loss.
Method:
Theory. Test loss has two parts. The data part: with n training examples, a fact can be learned only if its energy is above about 1/n, so lower-energy facts are never learned and cost their full energy, while each learned fact still costs about 1/n from noisy estimates. The bits part: each learned fact must still be stored, so the best plan gives bits to the highest-energy facts first, and each extra bit stores that fact more precisely. The paper proves that the lowest excess test loss is roughly the data part plus the bits part (up to a constant factor), both computed from the same energy list.
Experiments. First, on synthetic tasks generated from the paper's own model, it checks three things: whether test loss falls with the predicted scaling exponents; whether its energy formula predicts test loss at larger, unseen budgets (amounts of data and bits) better than a simple power-law fit; and whether it correctly says which helps more, 4× more data or 4× more bits. Second, on six Pythia language models, it adds data by training longer and changes bits by quantizing the weights to 2–8 bits, then fits simple power laws of test loss against tokens and bits.
Results:
What they found. On synthetic tasks, test loss falls with about the predicted scaling exponents, and the energy formula chooses between 4× more data and 4× more bits better than always choosing one; on Pythia, power laws in both data and bits mostly fit test loss better than power laws in data alone.
Does it match the theory? Partly, and only on synthetic tasks built from the theory's own model. There, the data exponents of two test tasks are 0.509 and 0.339 against predicted 1/2 and 1/3, but the energy formula predicts test loss at unseen budgets worse than a simple power-law fit unless data and bits grow together. On Pythia, the paper never estimates an energy list, so the theory is never checked on a real model.
Difference between this paper and the closest baseline:
Closest work: Zhu & Lafferty, "Quantized Nonparametric Estimation over Sobolev Ellipsoids" (arXiv:1503.07368). It shows that when an estimate must be stored in limited bits, the lowest error is an estimation part plus a rounding part, with bits given to the most important parts first. This paper applies the same idea to yes/no facts, allows any energy list (not only power laws), and adds a result showing that how energies are grouped can change the scaling exponents, plus an attention-layer learner.
Critical strengths
S1: The main theorem is clean and appears to be correct. It gives the lowest possible excess test loss for any energy list, not only power laws, and shows that the data part and the bits part are both read from the same energy list.
Why it matters: A simple statement of how data and bits trade off is a useful reference point for work on scaling laws.
Critical weaknesses
W1: The main theorem mostly reuses known results. Zhu & Lafferty (arXiv:1503.07368), which the paper cites, already show that the lowest error with limited bits is an estimation part plus a rounding part, with more bits given to the parts with larger signal. This paper combines standard tools (an Assouad lower bound for the data part and reverse water-filling for the bits part) for yes/no facts instead of Gaussian estimation. The compute–precision laws in Section 6 follow directly from scaling laws the paper assumes ("Assume model error Θ(M^−χ) … Under these two-sided laws").
Why it matters: The paper's main contribution is a new link between data and bits, but the core argument is largely an existing one in a new setting.
W2: The attention learner is too small and simplified to say anything about real models. It is a single hand-built attention head that is given most of the answer (fixed keys, the correct candidate revealed during training, and known fact frequencies). To support its claims, the paper should instead pre-train a standard Transformer on next-token prediction and check whether it reaches the predicted loss.
Why it matters: The claims about autoregressive and Transformer learning do not follow from such a simplified learner.
W3: Even on its own synthetic tasks, the energy formula usually predicts loss worse than a simple curve fit. When only data or only bits grows, a plain power-law fit predicts the new loss more accurately on every one of the six tasks (Appendix, "Response functions and held-out comparisons"). The energy formula is better only at the single test point per task where data and bits both grow. For choosing between 4× more data and 4× more bits, it is compared only with "always add data" and "always add bits", not with the choice the plain power-law fit would make.
Why it matters: The paper presents loss prediction and the data-versus-bits choice as practical uses of its theory, but a simple curve fit does better on most predictions and is never tested on the choice.
W4: The Pythia experiments do not test the theory. The paper never estimates an energy list for Pythia; it only fits generic power laws of loss against tokens and bits. Quantizing a trained network's weights also does not give each fact its own bits, which is what the theory assumes. Even these generic fits give mixed results: at the two smallest models (70M and 160M), adding bits to the fit makes 6-bit predictions worse, and at 70M a simpler fixed-penalty baseline predicts better. The closest real-model work, Kumar et al., "Scaling Laws for Precision" (arXiv:2411.04330), is not cited or compared. The baseline losses at 70M and 160M (6.20 and 4.39 nats on WikiText-103, against 2.97 at 410M) also look unusually high.
Why it matters: Nothing in the paper shows that the energy formula predicts or explains how real language models behave.
Questions for authors
See weaknesses.
4 comments
Response to Xcqj — Part 2/3: extrapolation and resource decisions
W3: Beyond a single joint-budget corner. We repeated the held-out analysis at four fixed calibration boundaries: the first 3 or 4 data budgets crossed with the first 4 or 5 storage budgets. Models are fitted to validation losses and evaluated on test losses at larger held-out budgets. We average boundary-specific errors within each seed before paired bootstrap, preserving the dependence among overlapping configurations. Thus, four boundaries contribute within each seed, while the seed remains the independent statistical unit. Please refer to Revision 5 in Line 3305-3326.
For weighted-average SGD, the spectrum response has lower joint-extrapolation normalized RMSE than the free joint power law on all six sources, with all six pointwise paired 95% BCa intervals below zero. On broken-power, the difference (spectrum minus free power) is -0.05048 [-0.05763, -0.04154]. Normalization divides RMSE by that source's zero-predictor excess CE. The comparison uses 16 independent paired seeds and 5,000 resamples. The advantage therefore persists across the expanded joint-extrapolation regions, rather than depending on the original single corner. Please refer to Revision 5 in Line 3329-3338.
The two holdout tasks show different behavior: the spectrum response has lower joint-extrapolation error across all six sources, while free power has lower error for each single-axis holdout. The common-spectrum theorem characterizes minimax risk up to constants; these measurements compare finite-budget response predictions for weighted-average SGD.
Fitted free-power resource choice. We evaluated the free-power response using the same validation fits, starting configurations, test outcomes and three actions: quadruple data, quadruple storage, or keep both unchanged. Mean regret is lower for free power than the unfitted spectral rule on all six weighted-SGD sources; all six paired intervals exclude zero. Against the calibrated spectral policy, free power also has lower mean regret on all six sources, with four intervals including zero. Please refer to Revision 5 in Line 3348-3357.
For example, on broken-power, mean regrets in nats are 1.1927 for unfitted spectrum, 0.3280 for calibrated spectrum and 0.3018 for free power. The unfitted rule uses source information without response calibration; calibrated spectrum and free power use the same validation resource configurations.
Where does the decision difference occur? We now split decisions by all candidate endpoints: nine origins have every action within the calibration rectangle; seven include an extrapolated endpoint. All origins themselves are inside. Averaging six fixed sources within each of 16 seeds, spectrum-minus-free-power regret is 0.001592 [0.001326, 0.001865] nats in range and 0.000310 [0.000232, 0.000417] beyond it for weighted SGD. Action agreement is 65.05% and 75.89%. Thus the free-power advantage is present in both strata.
The direct estimator shows a different pattern: spectrum-minus-free-power regret is 0.000345 [0.000197, 0.000564] in range and -0.000066 [-0.000109, -0.000023] for extrapolated candidates, with agreement 86.92% and 90.48%. Calibrated spectrum and free power are also reported. Gains use the same three actions, including staying; their paired contrast is exactly the negative regret contrast. This separates learner dependence, calibration support and fitting information instead of attributing the gap to extrapolation alone. Please refer to Revision 5 in Line 3834-3867.
Response to Xcqj — Part 3/3: numerical reference and precision responses
W4: Numerical baseline and the six-scale comparison. We evaluated the numerical baseline directly and reran the six-scale continuation study in FP32. On the same step143000 checkpoints and 96 WikiText test windows, FP32 evaluation gives 3.821960/3.288221 nats for 70M/160M, compared with 6.200417/4.390311 under BF16 eager evaluation. For 70M, BF16-rounded parameter storage with FP32 computation gives 3.827063, separating the large arithmetic-path effect from storage rounding alone. Please refer to Revision 6 in Line 3343-3347.
A separate same-parameter intervention protects attention modules and the output projection in FP32 under outer BF16 autocast. After choosing it on eight exploratory windows, confirmation on the other 88 windows across all six scales gives mean CE differences from FP32 between 0.000056 and 0.000195 nats. These paired-window comparisons localize the numerical sensitivity in the tested environment. Please refer to Revision 6 in Line 3360-3366.
All 48 trajectories use FP32 training/evaluation: six scales, eight seeds, four continuation budgets and six formats. Per-seed fits use validation losses at the first three budgets and 4/8-bit/BF16. Table entries are test RMSE (nats): extrapolation at the largest budget for 4/6/8-bit/BF16; interpolation at 6-bit for the first three budgets.
| Model | Data-only: extrap. | Additive: extrap. | Fixed penalty: extrap. | PTQ response: extrap. | Additive: 6-bit | PTQ response: 6-bit |
|---|---|---|---|---|---|---|
| 70M | 1.99641 | 1.15471 | 1.10936 | 0.22802 | 2.02965 | 0.03262 |
| 160M | 0.77399 | 0.36539 | 0.47597 | 0.16723 | 0.66090 | 0.01897 |
| 410M | 0.08527 | 0.06334 | 0.11069 | 0.08156 | 0.09346 | 0.02206 |
| 1B | 0.05984 | 0.03865 | 0.04846 | 0.03395 | 0.04705 | 0.02311 |
| 1.4B | 0.05768 | 0.03406 | 0.04716 | 0.03198 | 0.04227 | 0.01918 |
| 2.8B | 0.05435 | 0.03247 | 0.04515 | 0.03189 | 0.03906 | 0.02110 |
Additive improves data-only extrapolation at all six scales, with all paired 95% BCa intervals below zero. Fixed penalty remains better than additive at 70M. For 6-bit interpolation, data-only remains better than additive at 70M/160M; the direction reverses at 410M–2.8B. Including an interaction improves data extrapolation over additive at 70M, 160M and 410M; the paired intervals include zero at 1B–2.8B.
Comparison with the precision-response law. Kumar et al., Scaling Laws for Precision, is directly relevant to this comparison. Their data axis is total pretraining tokens; ours is additional WikiText tokens after a fixed checkpoint. We therefore fit an explicit continuation analogue, with the same validation/held-out split, instead of inserting additional tokens into their published law. Its penalty is relative to a fitted BF16 response, with tokens, , , , and bit depth (16 for BF16).
PTQ response improves 6-bit interpolation over additive at all scales. For data extrapolation it improves at 70M, 160M, 1B and 1.4B, is worse at 410M, and has a paired interval including zero at 2.8B. Intervals use eight paired seeds and 5,000 BCa resamples. All 48 fitted PTQ penalty exponents reach zero; fixing changes no seed-level RMSE by more than . The zero-exponent control identifies a bit-depth fit without a positive continuation exponent. Please refer to Revision 7 in Line 3371-3408.
The 70M/160M 2-bit losses exceed the primary response family's specified 0–20-nat prediction range, and that calibration produced no accepted fit. The full scan includes 2/3-bit measurements; the table above uses the predefined 4/6/8-bit/BF16 comparison. Please refer to Revision 7 in Line 3402-3408 and 3430-3433.
Measured penalty versus training. We now plot using the same seed, checkpoint and windows. BF16 denotes stored weights; arithmetic is FP32. From to continuation tokens, 4/6-bit penalties decrease at all six scales, with every pointwise endpoint-change interval below zero. The 70M 2/3-bit penalties increase; 8-bit changes are small and mixed. A constrained fit can therefore reach zero while measured penalties vary. This differs from testing a total-pretraining-token exponent. Please refer to Revision 7 in Line 3868-3914.
Connection to the theory. Controlled sources directly test the spectral functional and pairing effect. Pythia measures empirical data/format responses and held-out predictability across six scales; its predictive-energy spectrum and theoretical crossover are not estimated. The minimax converse permits arbitrary joint encodings; individual coordinate bits supply its achievability construction.
Follow-up on our response
Dear Reviewer Xcqj, thank you again for your detailed review; we would appreciate your thoughts on our response when you have a chance. The thread above covers the comparison with Zhu–Lafferty, joint attention learning, the free-power resource-choice baseline, and the FP32 Pythia baseline and precision-response comparisons. Please let us know which points, if any, remain unresolved so that we can focus our clarification on them while the discussion is open.
Response to Xcqj — Part 1/3: theory and joint attention learning
Thank you for the detailed questions. This is Part 1/3: the theoretical contribution and joint attention learning. Part 2 covers held-out prediction and resource decisions; Part 3 covers the numerical baseline and the six-scale Pythia comparison.
W1: What is new relative to Zhu–Lafferty? Their work already treats finite codebooks and a Gaussian hypercube, and uses posterior decomposition and water filling. We now give an explicit comparison rather than relying on a difference in application terminology. For noise variance e, define ai=qie/(qi+e), ri=qi2/(qi+e), S=∑imin(qi,e), and Du(B)=inf∑bi≤B∑iui2−2bi with bi≥0. Then
21(S+Dq)≤∑iai+Dr≤S+Dq.
The appendix proves this coordinatewise for every summable spectrum, including countably many coordinates and zero budget. Thus the posterior-information objective already connects to the additive spectrum functional. Our result establishes the operational minimax characterization under random-coordinate Bernoulli observations and bounded-logit parameters, with constants uniform in both resources. Its proof handles binomial visit counts, a box-supported product prior for the joint-codebook converse, and integer-bit encoding. Theorem 3.3 adds the explicit same-marginal construction: changing energy–dimension pairing changes both resource exponents while preserving both complete block marginals. These are the characterization and source-construction contributions. Please refer to Revision 2 in Line 3510-3552.
With ε2=1/n and χ=2m, the power-law rates coincide. The parameter classes differ: qj≍j−(1+2m) has divergent Sobolev-weighted sum ∑jj2mqj, so the correspondence is not a literal substitution of our box into their ellipsoid theorem.
For the conditional compute law, we give a concrete bounded-logit population encoded by a uniform scalar format. Rescaling by κ(M) gives grid spacing h=2κ(M)2−b. When h≤a/2, integration over full quantizer cells and Bernoulli KL curvature give two-sided expected excess CE of order M−χ+κ(M)22−2b. Matching-precision choices enter this range as M grows. This supplies an explicit representation response; the arithmetic cost and admissible response class remain the conditions of Theorem 6.1. Please refer to Revision 3 in Line 3554-3595.
W2: Learning attention jointly from prediction labels. Fixed keys, public addresses and routing supervision belong to the proven construction. To test whether the pairing effect survives beyond it, we trained a two-layer causal Transformer, width 128 with four heads, with all Q/K/V matrices learned from binary next-output labels and no route labels. The study uses fixed finite easy/hard sources, five sample budgets, each trained from a reset initialization, and eight paired seeds.
At 65,536 training examples, hard-minus-easy excess CE is 0.006868 [0.006374, 0.007292] for χ=1 and 0.001871 [0.001352, 0.002386] for χ=2. These are paired 95% BCa intervals with 5,000 resamples. Each seed additionally uses 65,536 independent validation contexts for selecting among 20 epochs and another 65,536 test contexts. The horizontal budget counts training examples, not validation labels. Please refer to Revision 4 in Line 3276-3300.
The pairing effect therefore persists under joint attention learning without routing supervision on these finite sources. Theorem 4.3 establishes the rate for the specified learner; the new experiment tests whether the effect survives learning all attention parameters from prediction labels. The Pythia continuation study addresses the separate empirical question of data/storage responses in pretrained language models.
The complete five-budget analysis gives positive FP32 hard-minus-easy means in all ten profile–budget cells, with nine pointwise BCa intervals excluding zero (the χ=2,n=256 interval includes zero). The exact finite-source S(n) is larger for hard in every cell. BF16 and 4/6/8-bit storage retain nine positive gap intervals, 3-bit eight and 2-bit five. The full curves test finite-budget pairing under learned attention, rather than fitting an infinite-tail rate. Please refer to Revision 4 in Line 3597-3614.