acceptodds
Under review as a conference paper at ICLR 2027

One Spectrum, Two Resources: Data-Memory Scaling in Autoregressive Prediction

Abstract

How much learned memory is needed to benefit from more data? We show that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source. Each coordinate contributes its query probability times the squared radius of its unknown logit. Writing \(\mu\) for the resulting energy spectrum, we prove the minimax law for \(n\) prediction blocks and a learned state with at most \(2^B\) values. Data set the resolution \(1/n\); memory sets the level \(\tau_B\) reached by optimal bit allocation. The complete curve also recovers the positive spectrum. Energy-dimension pairing is essential: two causal sources with identical block-energy and block-dimension marginals have different data and memory exponents. A masked query-key attention head learns the route and values, realizing the law with explicit routing, format, and arithmetic errors. Further results give exponent-adaptive allocation, finite-precision realization, and compute-precision laws under two-sided arithmetic assumptions. Experiments recover the data-memory collapse and coupling exponents, explain the routing and allocation mechanisms, and examine weight-only quantization across six pretrained-model scales.

Primary area:
learning theory
Keywords:
Scaling Laws, Memory-Constrained Learning, Autoregressive Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XcqjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In one sentence: the paper studies how a model's test loss falls with more training data and more storage bits, and argues that both are set by one list: the energy of each fact the model must learn, roughly how much extra test loss the model suffers if it never learns that fact.

Problem:

Scaling laws usually ask how test loss falls with more training data or a bigger model. This paper replaces "bigger model" with "more bits": the total storage for what the model learned, roughly parameters × bits per parameter (the paper calls this "memory"). Its key idea is that some facts matter more than others. It measures this with each fact's energy (the paper's "predictive energy"): how often the fact is asked × how far its answer can be from a coin flip, squared. A fact's energy is about the extra loss you suffer if the model never learns it: guessing 50/50 on a near-certain "yes" costs a lot, while guessing on a near coin flip costs little. The list of all fact energies is the energy list (the paper's "energy spectrum"). With limited data, the model can only learn the high-energy facts; with limited bits, it should store the high-energy facts first.

Goal:

The paper wants a formula for the lowest excess test loss (loss above the best possible) that any model can guarantee with n training examples and B bits, computed from the energy list, whatever its shape (not only the usual power law). From it, the paper wants to explain how fast test loss falls as data or bits grow (the scaling exponents), and to show that a simple attention layer trained by SGD reaches this lowest loss.

Method:

Theory. Test loss has two parts. The data part: with n training examples, a fact can be learned only if its energy is above about 1/n, so lower-energy facts are never learned and cost their full energy, while each learned fact still costs about 1/n from noisy estimates. The bits part: each learned fact must still be stored, so the best plan gives bits to the highest-energy facts first, and each extra bit stores that fact more precisely. The paper proves that the lowest excess test loss is roughly the data part plus the bits part (up to a constant factor), both computed from the same energy list.

Experiments. First, on synthetic tasks generated from the paper's own model, it checks three things: whether test loss falls with the predicted scaling exponents; whether its energy formula predicts test loss at larger, unseen budgets (amounts of data and bits) better than a simple power-law fit; and whether it correctly says which helps more, 4× more data or 4× more bits. Second, on six Pythia language models, it adds data by training longer and changes bits by quantizing the weights to 2–8 bits, then fits simple power laws of test loss against tokens and bits.

Results:

What they found. On synthetic tasks, test loss falls with about the predicted scaling exponents, and the energy formula chooses between 4× more data and 4× more bits better than always choosing one; on Pythia, power laws in both data and bits mostly fit test loss better than power laws in data alone.

Does it match the theory? Partly, and only on synthetic tasks built from the theory's own model. There, the data exponents of two test tasks are 0.509 and 0.339 against predicted 1/2 and 1/3, but the energy formula predicts test loss at unseen budgets worse than a simple power-law fit unless data and bits grow together. On Pythia, the paper never estimates an energy list, so the theory is never checked on a real model.

Difference between this paper and the closest baseline:

Closest work: Zhu & Lafferty, "Quantized Nonparametric Estimation over Sobolev Ellipsoids" (arXiv:1503.07368). It shows that when an estimate must be stored in limited bits, the lowest error is an estimation part plus a rounding part, with bits given to the most important parts first. This paper applies the same idea to yes/no facts, allows any energy list (not only power laws), and adds a result showing that how energies are grouped can change the scaling exponents, plus an attention-layer learner.

Critical strengths

S1: The main theorem is clean and appears to be correct. It gives the lowest possible excess test loss for any energy list, not only power laws, and shows that the data part and the bits part are both read from the same energy list.

Why it matters: A simple statement of how data and bits trade off is a useful reference point for work on scaling laws.

Critical weaknesses

W1: The main theorem mostly reuses known results. Zhu & Lafferty (arXiv:1503.07368), which the paper cites, already show that the lowest error with limited bits is an estimation part plus a rounding part, with more bits given to the parts with larger signal. This paper combines standard tools (an Assouad lower bound for the data part and reverse water-filling for the bits part) for yes/no facts instead of Gaussian estimation. The compute–precision laws in Section 6 follow directly from scaling laws the paper assumes ("Assume model error Θ(M^−χ) … Under these two-sided laws").

Why it matters: The paper's main contribution is a new link between data and bits, but the core argument is largely an existing one in a new setting.

W2: The attention learner is too small and simplified to say anything about real models. It is a single hand-built attention head that is given most of the answer (fixed keys, the correct candidate revealed during training, and known fact frequencies). To support its claims, the paper should instead pre-train a standard Transformer on next-token prediction and check whether it reaches the predicted loss.

Why it matters: The claims about autoregressive and Transformer learning do not follow from such a simplified learner.

W3: Even on its own synthetic tasks, the energy formula usually predicts loss worse than a simple curve fit. When only data or only bits grows, a plain power-law fit predicts the new loss more accurately on every one of the six tasks (Appendix, "Response functions and held-out comparisons"). The energy formula is better only at the single test point per task where data and bits both grow. For choosing between 4× more data and 4× more bits, it is compared only with "always add data" and "always add bits", not with the choice the plain power-law fit would make.

Why it matters: The paper presents loss prediction and the data-versus-bits choice as practical uses of its theory, but a simple curve fit does better on most predictions and is never tested on the choice.

W4: The Pythia experiments do not test the theory. The paper never estimates an energy list for Pythia; it only fits generic power laws of loss against tokens and bits. Quantizing a trained network's weights also does not give each fact its own bits, which is what the theory assumes. Even these generic fits give mixed results: at the two smallest models (70M and 160M), adding bits to the fit makes 6-bit predictions worse, and at 70M a simpler fixed-penalty baseline predicts better. The closest real-model work, Kumar et al., "Scaling Laws for Precision" (arXiv:2411.04330), is not cited or compared. The baseline losses at 70M and 160M (6.20 and 4.39 nats on WikiText-103, against 2.97 at 410M) also look unusually high.

Why it matters: Nothing in the paper shows that the energy formula predicts or explains how real language models behave.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Xcqj — Part 1/3: theory and joint attention learning

Thank you for the detailed questions. This is Part 1/3: the theoretical contribution and joint attention learning. Part 2 covers held-out prediction and resource decisions; Part 3 covers the numerical baseline and the six-scale Pythia comparison.

W1: What is new relative to Zhu–Lafferty? Their work already treats finite codebooks and a Gaussian hypercube, and uses posterior decomposition and water filling. We now give an explicit comparison rather than relying on a difference in application terminology. For noise variance ee, define ai=qie/(qi+e)a_i=q_ie/(q_i+e), ri=qi2/(qi+e)r_i=q_i^2/(q_i+e), S=∑imin⁡(qi,e)S=\sum_i\min(q_i,e), and Du(B)=inf⁡∑bi≤B∑iui2−2biD_u(B)=\inf_{\sum b_i\le B}\sum_i u_i2^{-2b_i} with bi≥0b_i\ge0. Then

12(S+Dq)≤∑iai+Dr≤S+Dq.\tfrac12(S+D_q)\le\sum_i a_i+D_r\le S+D_q.

The appendix proves this coordinatewise for every summable spectrum, including countably many coordinates and zero budget. Thus the posterior-information objective already connects to the additive spectrum functional. Our result establishes the operational minimax characterization under random-coordinate Bernoulli observations and bounded-logit parameters, with constants uniform in both resources. Its proof handles binomial visit counts, a box-supported product prior for the joint-codebook converse, and integer-bit encoding. Theorem 3.3 adds the explicit same-marginal construction: changing energy–dimension pairing changes both resource exponents while preserving both complete block marginals. These are the characterization and source-construction contributions. Please refer to Revision 2 in Line 3510-3552.

With ε2=1/n\varepsilon^2=1/n and χ=2m\chi=2m, the power-law rates coincide. The parameter classes differ: qj≍j−(1+2m)q_j\asymp j^{-(1+2m)} has divergent Sobolev-weighted sum ∑jj2mqj\sum_j j^{2m}q_j, so the correspondence is not a literal substitution of our box into their ellipsoid theorem.

For the conditional compute law, we give a concrete bounded-logit population encoded by a uniform scalar format. Rescaling by κ(M)\kappa(M) gives grid spacing h=2κ(M)2−bh=2\kappa(M)2^{-b}. When h≤a/2h\le a/2, integration over full quantizer cells and Bernoulli KL curvature give two-sided expected excess CE of order M−χ+κ(M)22−2bM^{-\chi}+\kappa(M)^22^{-2b}. Matching-precision choices enter this range as MM grows. This supplies an explicit representation response; the arithmetic cost and admissible response class remain the conditions of Theorem 6.1. Please refer to Revision 3 in Line 3554-3595.

W2: Learning attention jointly from prediction labels. Fixed keys, public addresses and routing supervision belong to the proven construction. To test whether the pairing effect survives beyond it, we trained a two-layer causal Transformer, width 128 with four heads, with all Q/K/V matrices learned from binary next-output labels and no route labels. The study uses fixed finite easy/hard sources, five sample budgets, each trained from a reset initialization, and eight paired seeds.

At 65,536 training examples, hard-minus-easy excess CE is 0.006868 [0.006374, 0.007292] for χ=1\chi=1 and 0.001871 [0.001352, 0.002386] for χ=2\chi=2. These are paired 95% BCa intervals with 5,000 resamples. Each seed additionally uses 65,536 independent validation contexts for selecting among 20 epochs and another 65,536 test contexts. The horizontal budget counts training examples, not validation labels. Please refer to Revision 4 in Line 3276-3300.

The pairing effect therefore persists under joint attention learning without routing supervision on these finite sources. Theorem 4.3 establishes the rate for the specified learner; the new experiment tests whether the effect survives learning all attention parameters from prediction labels. The Pythia continuation study addresses the separate empirical question of data/storage responses in pretrained language models.

The complete five-budget analysis gives positive FP32 hard-minus-easy means in all ten profile–budget cells, with nine pointwise BCa intervals excluding zero (the χ=2,n=256\chi=2,n=256 interval includes zero). The exact finite-source S(n)S(n) is larger for hard in every cell. BF16 and 4/6/8-bit storage retain nine positive gap intervals, 3-bit eight and 2-bit five. The full curves test finite-budget pairing under learned attention, rather than fitting an infinite-tail rate. Please refer to Revision 4 in Line 3597-3614.

AuthorsOpenReview ↗

Response to Xcqj — Part 2/3: extrapolation and resource decisions

W3: Beyond a single joint-budget corner. We repeated the held-out analysis at four fixed calibration boundaries: the first 3 or 4 data budgets crossed with the first 4 or 5 storage budgets. Models are fitted to validation losses and evaluated on test losses at larger held-out budgets. We average boundary-specific errors within each seed before paired bootstrap, preserving the dependence among overlapping configurations. Thus, four boundaries contribute within each seed, while the seed remains the independent statistical unit. Please refer to Revision 5 in Line 3305-3326.

For weighted-average SGD, the spectrum response has lower joint-extrapolation normalized RMSE than the free joint power law on all six sources, with all six pointwise paired 95% BCa intervals below zero. On broken-power, the difference (spectrum minus free power) is -0.05048 [-0.05763, -0.04154]. Normalization divides RMSE by that source's zero-predictor excess CE. The comparison uses 16 independent paired seeds and 5,000 resamples. The advantage therefore persists across the expanded joint-extrapolation regions, rather than depending on the original single corner. Please refer to Revision 5 in Line 3329-3338.

The two holdout tasks show different behavior: the spectrum response has lower joint-extrapolation error across all six sources, while free power has lower error for each single-axis holdout. The common-spectrum theorem characterizes minimax risk up to constants; these measurements compare finite-budget response predictions for weighted-average SGD.

Fitted free-power resource choice. We evaluated the free-power response using the same validation fits, starting configurations, test outcomes and three actions: quadruple data, quadruple storage, or keep both unchanged. Mean regret is lower for free power than the unfitted spectral rule on all six weighted-SGD sources; all six paired intervals exclude zero. Against the calibrated spectral policy, free power also has lower mean regret on all six sources, with four intervals including zero. Please refer to Revision 5 in Line 3348-3357.

For example, on broken-power, mean regrets in 10−310^{-3} nats are 1.1927 for unfitted spectrum, 0.3280 for calibrated spectrum and 0.3018 for free power. The unfitted rule uses source information without response calibration; calibrated spectrum and free power use the same validation resource configurations.

Where does the decision difference occur? We now split decisions by all candidate endpoints: nine origins have every action within the calibration rectangle; seven include an extrapolated endpoint. All origins themselves are inside. Averaging six fixed sources within each of 16 seeds, spectrum-minus-free-power regret is 0.001592 [0.001326, 0.001865] nats in range and 0.000310 [0.000232, 0.000417] beyond it for weighted SGD. Action agreement is 65.05% and 75.89%. Thus the free-power advantage is present in both strata.

The direct estimator shows a different pattern: spectrum-minus-free-power regret is 0.000345 [0.000197, 0.000564] in range and -0.000066 [-0.000109, -0.000023] for extrapolated candidates, with agreement 86.92% and 90.48%. Calibrated spectrum and free power are also reported. Gains use the same three actions, including staying; their paired contrast is exactly the negative regret contrast. This separates learner dependence, calibration support and fitting information instead of attributing the gap to extrapolation alone. Please refer to Revision 5 in Line 3834-3867.

AuthorsOpenReview ↗

Response to Xcqj — Part 3/3: numerical reference and precision responses

W4: Numerical baseline and the six-scale comparison. We evaluated the numerical baseline directly and reran the six-scale continuation study in FP32. On the same step143000 checkpoints and 96 WikiText test windows, FP32 evaluation gives 3.821960/3.288221 nats for 70M/160M, compared with 6.200417/4.390311 under BF16 eager evaluation. For 70M, BF16-rounded parameter storage with FP32 computation gives 3.827063, separating the large arithmetic-path effect from storage rounding alone. Please refer to Revision 6 in Line 3343-3347.

A separate same-parameter intervention protects attention modules and the output projection in FP32 under outer BF16 autocast. After choosing it on eight exploratory windows, confirmation on the other 88 windows across all six scales gives mean CE differences from FP32 between 0.000056 and 0.000195 nats. These paired-window comparisons localize the numerical sensitivity in the tested environment. Please refer to Revision 6 in Line 3360-3366.

All 48 trajectories use FP32 training/evaluation: six scales, eight seeds, four continuation budgets and six formats. Per-seed fits use validation losses at the first three budgets and 4/8-bit/BF16. Table entries are test RMSE (nats): extrapolation at the largest budget for 4/6/8-bit/BF16; interpolation at 6-bit for the first three budgets.

ModelData-only: extrap.Additive: extrap.Fixed penalty: extrap.PTQ response: extrap.Additive: 6-bitPTQ response: 6-bit
70M1.996411.154711.109360.228022.029650.03262
160M0.773990.365390.475970.167230.660900.01897
410M0.085270.063340.110690.081560.093460.02206
1B0.059840.038650.048460.033950.047050.02311
1.4B0.057680.034060.047160.031980.042270.01918
2.8B0.054350.032470.045150.031890.039060.02110

Additive improves data-only extrapolation at all six scales, with all paired 95% BCa intervals below zero. Fixed penalty remains better than additive at 70M. For 6-bit interpolation, data-only remains better than additive at 70M/160M; the direction reverses at 410M–2.8B. Including an interaction improves data extrapolation over additive at 70M, 160M and 410M; the paired intervals include zero at 1B–2.8B.

Comparison with the precision-response law. Kumar et al., Scaling Laws for Precision, is directly relevant to this comparison. Their data axis is total pretraining tokens; ours is additional WikiText tokens after a fixed checkpoint. We therefore fit an explicit continuation analogue, with the same validation/held-out split, instead of inserting additional tokens into their published law. Its penalty is C(T/T0)γ[exp⁡(−P/λ)−exp⁡(−16/λ)]C(T/T_0)^\gamma[\exp(-P/\lambda)-\exp(-16/\lambda)] relative to a fitted BF16 response, with T0=220T_0=2^{20} tokens, C≥0C\ge0, γ∈[0,4]\gamma\in[0,4], λ∈[0.1,16]\lambda\in[0.1,16], and bit depth PP (16 for BF16).

PTQ response improves 6-bit interpolation over additive at all scales. For data extrapolation it improves at 70M, 160M, 1B and 1.4B, is worse at 410M, and has a paired interval including zero at 2.8B. Intervals use eight paired seeds and 5,000 BCa resamples. All 48 fitted PTQ penalty exponents γ\gamma reach zero; fixing γ=0\gamma=0 changes no seed-level RMSE by more than 8×10−98\times10^{-9}. The zero-exponent control identifies a bit-depth fit without a positive continuation exponent. Please refer to Revision 7 in Line 3371-3408.

The 70M/160M 2-bit losses exceed the primary response family's specified 0–20-nat prediction range, and that calibration produced no accepted fit. The full scan includes 2/3-bit measurements; the table above uses the predefined 4/6/8-bit/BF16 comparison. Please refer to Revision 7 in Line 3402-3408 and 3430-3433.

Measured penalty versus training. We now plot Lf(T)−LBF16(T)L_f(T)-L_{\mathrm{BF16}}(T) using the same seed, checkpoint and windows. BF16 denotes stored weights; arithmetic is FP32. From 2202^{20} to 2262^{26} continuation tokens, 4/6-bit penalties decrease at all six scales, with every pointwise endpoint-change interval below zero. The 70M 2/3-bit penalties increase; 8-bit changes are small and mixed. A constrained γ≥0\gamma\ge0 fit can therefore reach zero while measured penalties vary. This differs from testing a total-pretraining-token exponent. Please refer to Revision 7 in Line 3868-3914.

Connection to the theory. Controlled sources directly test the spectral functional and pairing effect. Pythia measures empirical data/format responses and held-out predictability across six scales; its predictive-energy spectrum and theoretical crossover are not estimated. The minimax converse permits arbitrary joint encodings; individual coordinate bits supply its achievability construction.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer Xcqj, thank you again for your detailed review; we would appreciate your thoughts on our response when you have a chance. The thread above covers the comparison with Zhu–Lafferty, joint attention learning, the free-power resource-choice baseline, and the FP32 Pythia baseline and precision-response comparisons. Please let us know which points, if any, remain unresolved so that we can focus our clarification on them while the discussion is open.

Reviewer_K7xsOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates how the amount of data and the number of bits retained after learning jointly determine the autoregressive prediction error. In a retrieval generative model with independent and finite logit parameters, the authors define the predictive energy and prove that the same energy spectrum can control both statistical error and representation error simultaneously.

Critical strengths
  1. The presentation and notation are clear.

  2. The design of the synthetic experiments is explicit and careful.

Critical weaknesses
  1. The contribution is not quite clear. I am not familiar with this theoretical framework. Is this framework developed by the authors themselves, or does it follow previous work? Some existing work has shown that the data budget and coding budget can jointly limit estimation accuracy. What new conclusions do the spectrum control, attention construction and minimax rate in this paper provide compared with previous work [1]?

  2. Section 4 uses orthogonal addresses and a fixed-key matrix. Is this assumption reasonable? Do your experiments also use fixed keys?

  3. The results mainly prove that the spectral mechanism can be implemented through a special attention model, but there is still a lack of conclusions regarding joint training under next-token supervision.

[1]: Zhu, Yuancheng, and John Lafferty. "Quantized minimax estimation over Sobolev ellipsoids."

Questions for authors

Please see the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to K7xs: closest theory and joint attention learning

The central additions are the operational minimax characterization for random-coordinate bounded-logit prediction and the pairing separation. We explain their relationship to Zhu–Lafferty, then report the new joint-attention experiment.

W1. Relationship to Zhu–Lafferty. The shared foundation is finite-codebook estimation, posterior decomposition and water filling. Their work also contains a Gaussian hypercube example; the new appendix explicitly relates its posterior-information objective to the spectrum expression. With Gaussian observation variance ee, define ai=qie/(qi+e)a_i=q_ie/(q_i+e), ri=qi2/(qi+e)r_i=q_i^2/(q_i+e), S=∑imin⁡(qi,e)S=\sum_i\min(q_i,e) and optimally allocated continuous-bit distortion Du(B)D_u(B), we prove

12(S+Dq)≤∑iai+Dr≤S+Dq.\tfrac12(S+D_q)\le\sum_i a_i+D_r\le S+D_q.

Our theorem establishes the operational minimax characterization in random-coordinate bounded-logit prediction for every summable energy sequence. Its random coordinate counts, supported box prior for the arbitrary-codebook converse and integer-bit construction provide the observation-model-specific steps. Theorem 3.3 then holds both complete group marginals fixed while changing their pairing, and obtains different data and memory exponents. The contribution is this joint characterization and explicit descriptor separation. Please refer to Revision 2 in Line 3510-3552.

Under ε2=1/n,χ=2m\varepsilon^2=1/n,\chi=2m, the power-law exponents match the Sobolev rates. This correspondence does not equate parameter classes: a fixed spectrum qj≍j−(1+2m)q_j\asymp j^{-(1+2m)} has divergent Sobolev-weighted sum. Section 4 connects the characterized source risk to a specified attention learner with routing and numerical costs.

W2. Why fixed keys, and which experiments use them? Orthogonal addresses encode public candidate positions, while the routing map and prediction logits remain unknown. Fixing the keys makes routing dynamics explicit and allows the cost of learning the route to be analyzed. The original synthetic construction and theorem-matched SGD experiments use this parameterization, including separate routing supervision. The joint-Q/K/V experiment below trains keys as well. Please refer to Revision 3 in Line 194-198.

W3. New joint-Q/K/V experiment. We tested the pairing effect with a two-layer causal Transformer (width 128, four heads), training all Q/K/V matrices from binary next-output labels, with no route labels. There are five training budgets and eight paired seeds per easy/hard source pair. Training samples are nested across budgets, and each budget resets to the same seed initialization. Epoch selection among 20 epochs uses 65,536 independent validation contexts, and evaluation uses a separate 65,536 contexts; these validation examples are additional to the reported training budget. Please refer to Revision 4 in Line 3276-3293.

At 65,536 training examples, the FP32 hard-minus-easy excess CE is:

Source profileDifference, natsPaired 95% BCa interval
χ=1\chi=10.006868[0.006374, 0.007292]
χ=2\chi=20.001871[0.001352, 0.002386]

Intervals use eight paired seeds and 5,000 resamples. The effect therefore persists when attention parameters are learned jointly without routing labels on these finite sources. Theorem 4.3 establishes the rate for its specified learner; this new experiment establishes the finite-source pairing effect under joint Q/K/V learning without route labels. Please refer to Revision 4 in Line 3294-3300.

Across all five budgets, FP32 hard-minus-easy means are positive in all ten profile–budget cells; nine pointwise BCa intervals exclude zero. The exact finite-source functional has the same direction in every cell. BF16 and 4/6/8-bit storage retain nine positive gap intervals, 3-bit eight and 2-bit five. The full curves report every budget and format. Please refer to Revision 4 in Line 3597-3614.

We thank the reviewers' for their response which fully address my concerns.

AuthorsOpenReview ↗

Thank you for your feedback

Dear Reviewer K7xs, thank you for reading our response and for confirming that it fully addresses your concerns. We appreciate your time and feedback.

Reviewer_NDkGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the minimax excess risk of any learner using n prediction blocks and a learned state of ≤2B\leq 2^B values. The claim is that data resolve the spectrum at resolution 1/n; memory at level τ_B.

Critical strengths

This is a dense theory paper whose mathematical proofs involve techniques such as water-filling KKT, entropy lower bound, Assouad step, SGD bound structure, adaptive schedule, floating-point error constants.

Critical weaknesses

The paper is very difficult to follow. For example, it is unclear to me what the sentence in the conclusion "The complete curve identifies positive energies beyond a fitted exponent; energy–dimension pairing gives identical marginals different resource laws" means.

The headline framing ("one spectrum, two resources") is established only for known spectra, fixed K, and synthetic sources, while the sole real-model evidence is a quantization probe whose connection to the theory's crossover law is qualitative.

Questions for authors

What are the assumptions in the adaptive result (Thm 5.1/I.1) and the learner of Thm 4.3/D.5?

Could you justify the assumptions in Theorem 6.1/N.1?

Could you elaborate on the sentence "The complete curve identifies positive energies beyond a fitted exponent; energy–dimension pairing gives identical marginals different resource laws"?

4 comments
AuthorsOpenReview ↗

Response to NDkG: interpretation, assumptions and empirical connections

Observations determine what can be learned; bits determine stored accuracy.

W1. What the conclusion means. A coordinate has predictive energy qi=piRi2q_i=p_iR_i^2, where pip_i is its query probability and RiR_i bounds its unknown logit. The curve Φ(t)=∑imin⁡qi,t\Phi(t)=\sum_i\min\\{q_i,t\\} retains the entire energy distribution: its right derivative counts coordinates with energy strictly greater than tt. Thus the exact curve determines positive energies and their multiplicities, up to relabeling; a tail exponent retains much less information.

Q3. Pairing concerns a different loss of information. Consider group energies (0.8,0.2)(0.8,0.2) and dimensions (1,4)(1,4). One pairing gives coordinate energies (0.8,0.05,0.05,0.05,0.05)(0.8,0.05,0.05,0.05,0.05); the other gives five coordinates of energy 0.20.2. Both group marginals are unchanged, but the coordinate spectra differ. Theorem 3.3 extends this pairing mechanism to an infinite construction with different data and memory exponents.

For the core result, “minimax” means the smallest worst-case expected excess loss achievable by any allowed learner over the stated parameter class. Excess loss subtracts the oracle loss obtained when the true parameters are known; this oracle loss is positive because the targets are stochastic. The two-term formula characterizes that optimum up to constants, rather than assigning an exact loss to every trained model. Please refer to Revision 1 in Line 108-117 and 156-161.

Q1. Known structure and adaptation (Theorems 5.1/I.1). Frequencies and radii specify the parameter class; the logits are unknown and must be learned. Adaptation assumes known canonical ordering, common power-law comparison constants, and an unknown exponent in a fixed positive compact interval. One clipped likelihood estimator and one integer bit schedule, both independent of that exponent, attain the stated rate uniformly. Please refer to Revision 3 in Line 254-257.

Q1. The learner (Theorems 4.3/D.5). Public orthogonal addresses and fixed keys encode candidate positions. Query parameters learn the unknown route from independent route-labelled calibration blocks; after the route is frozen, output logits are learned from independent prediction blocks using frequency-preconditioned projected SGD, the stated step sizes and weighted iterate average. Scalar formats encode the output. The bound includes calibration and deployment errors. The fixed-KK power-law total-resource corollary counts calibration samples and route-state bits in addition to prediction samples and value bits. Please refer to Revision 3 in Line 194-198.

Q2. Compute assumptions (Theorems 6.1/N.1). Theorem N.1 assumes E(M,b)≍M−χ+κ(M)22−2bE(M,b)\asymp M^{-\chi}+\kappa(M)^2 2^{-2b} and cost(M,b)≍Mf+1+χbν\mathrm{cost}(M,b)\asymp M^{f+1+\chi}b^\nu. Here MM is model size and bb is arithmetic precision; sample requirements and operation counts enter the cost. The converse assumes a numerical lower response matching the forward-error order; Theorem 5.2 supplies the upper bound. Matching numerical and model errors gives a feasible precision, and the converse establishes its cost order for systems satisfying this two-sided response. We add a concrete scalar-format instance: logits uniform on [−a,a][-a,a], retained on MM power-law-frequency coordinates, are rounded on a decoded grid of spacing h=2κ(M)2−bh=2\kappa(M)2^{-b}. For h≤a/2h\le a/2, full-cell integration and KL curvature prove expected excess CE of order M−χ+κ(M)22−2bM^{-\chi}+\kappa(M)^22^{-2b}. Matching precision eventually enters this range. This illustrates representation error along matching precision. The stipulated per-sample work times sample count gives the cost law. Please refer to Revision 3 in Line 3554-3595.

W2. Learning and real-model evidence. We additionally trained all Q/K/V matrices in a two-layer causal Transformer on binary next-output labels, without routing labels. At 65,536 training examples, hard-minus-easy excess CE is 0.006868 [0.006374, 0.007292] and 0.001871 [0.001352, 0.002386] for the two source profiles (eight paired seeds; 95% BCa). Each seed has separate 65,536-context validation and test sets; the training budget excludes validation. This supports the pairing effect under joint attention learning on fixed finite sources. Please refer to Revision 4 in Line 3294-3300.

The completed six-scale Pythia continuation study examines empirical data/storage responses. With FP32 training/evaluation, additive data/storage fits improve data-extrapolation RMSE over data-only at all six scales, with paired intervals excluding zero. Controlled sources test the spectral functional; Pythia quantifies data/format responses, without estimating its spectrum or crossover. Please refer to Revision 7 in Line 3371-3408 and 3916-3931.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer NDkG, thank you again for your thoughtful review; we would appreciate your feedback on whether our response clarifies the spectral interpretation and the assumptions connecting the results. For ease of reference, the response above walks through a concrete pairing example, the learning and compute assumptions, and the additional joint-attention evidence. Please let us know if any point would benefit from further clarification while the discussion remains open.

Thank you for your response and the revisions. However, I don't think my questions have been directly addressed. Therefore, I maintain my original score.

AuthorsOpenReview ↗

Clarification on your remaining questions

Dear Reviewer NDkG, thank you for reading the revision and for letting us know that our response did not get to the heart of your questions. Could you point us to the specific question or part of a question you would most like us to address more directly? That would help us give you a focused answer and guide our efforts to strengthen the paper’s contribution.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global response — Part 1/2: Revision and shared theoretical questions

The blue revision preserves ten main-text pages and all original theorem statements, assumptions, proofs and numbering. Seven Revision groups locate changes; Part 2 summarizes the evidence.

Revision:

  1. Line 035-039, 091-093, 115-123, 156-161, 504-509: We modify the Introduction, setup, theorem explanation and Conclusion to define the objects and explain the two resource bottlenecks.
  2. Line 071-079, 3252-3264, 3510-3552: We modify Related Work and add the detailed Zhu–Lafferty comparison, including an explicit posterior-water-filling comparison with our operational risk characterization and pairing result.
  3. Line 194-198, 254-257, 311-314, 3266-3271, 3554-3595: We modify the explanations of inputs, supervision and assumptions of the learning, adaptation, realization and compute results, and derive a fixed-format numerical response.
  4. Line 427-431, 3276-3300, 3597-3614: We add joint-Q/K/V learning from prediction labels, its full methods, five-budget/format curves and paired results.
  5. Line 374-377, 390-394, 417-420, 2852-2856, 3305-3338, 3348-3357, 3834-3867: We add four-boundary extrapolation and fitted free-power resource decisions; update Figure 2 and report the comparisons in Tables 16–17, with decision-support stratification.
  6. Line 396-400, 2395-2399, 3343-3347, 3360-3366: We add numerical interventions separating parameter rounding from arithmetic effects, and identify the original BF16 measurements in the captions.
  7. Line 475-479, 492-500, 3371-3408, 3430-3444, 3456-3487, 3868-3931: We add the six-scale FP32 reference, PTQ response and zero-exponent control, with Figures 4/19 and Tables 18–21/23 reporting comparisons; quantify penalties against simultaneous BF16-stored weights.

Shared questions

1. What does the common spectrum explain? Samples determine which parameters can be estimated; bits determine their stored resolution. In our source model, coordinate ii specifies a binary target probability through an unknown logit in [−Ri,Ri][-R_i,R_i] and is queried with probability pip_i. Its energy qi=piRi2q_i=p_iR_i^2 measures potential prediction difficulty over this parameter class. The same curve, Φ(t)=∑imin⁡(qi,t)\Phi(t)=\sum_i\min(q_i,t), determines both terms of the joint minimax risk: Φ(1/n)+Φ(τB)\Phi(1/n)+\Phi(\tau_B), up to constants depending on the logit bound. Here nn is the sample count and τB\tau_B is the resolution afforded by BB retained bits. The minimax risk permits arbitrary joint codebooks; coordinate allocation gives an achievable construction.

Keeping both group-energy and group-dimension marginals fixed, changing their pairing can change both resource exponents. The marginals do not determine the joint law; derivatives of the exact curve recover coordinate energies and multiplicities. Please refer to Revision 1 in Line 156-161 and Revision 2 in Line 3259-3264.

2. What is new relative to Zhu–Lafferty? Their work already includes finite codebooks, a Gaussian hypercube example and posterior water filling. We make this connection explicit: with Gaussian noise variance ee, posterior residual variances ai=qie/(qi+e)a_i=q_ie/(q_i+e) and posterior-mean variances ri=qi2/(qi+e)r_i=q_i^2/(q_i+e), the classical information-relaxation objective obeys

12S+Dq(B)≤∑iai+Dr(B)≤S+Dq(B),\tfrac12\\{S+D_q(B)\\}\le\sum_i a_i+D_r(B)\le S+D_q(B),

where S=∑imin⁡(qi,e)S=\sum_i\min(q_i,e) and DuD_u is the continuous allocation distortion for spectrum uu. Our contribution gives this functional an operational, nonasymptotic minimax characterization for random-coordinate bounded-logit prediction, then constructs identical block marginals with different resource exponents. The prediction proof addresses random visit counts, a prior supported in the logit box and integer-bit achievability. Under ε2=1/n,χ=2m\varepsilon^2=1/n,\chi=2m, the power-law exponents match; this is an exponent correspondence, not an identification of the Sobolev and box classes. Please refer to Revision 2 in Line 3510-3552.

3. How do the learning and compute results connect? Section 4 learns retrieval using public orthogonal keys and route-labelled calibration, then learns values from independent prediction examples using frequency-preconditioned weighted-average SGD. Its fixed-KK total-resource result counts calibration samples and route-state bits as well as prediction samples and value bits. Theorem 5.1 adapts to an unknown decay exponent using known coordinate order and common comparison constants over a fixed exponent interval. Theorem 5.2 evaluates supplied logits with finite storage and arithmetic. Theorem 6.1 optimizes within its stated two-sided error–cost response, including the numerical lower response; the forward-error theorem supplies an upper bound. A new fixed-format construction derives the two-sided representation response in its resolved range; operation cost remains separately specified. Please refer to Revision 3 in Line 3554-3595.

AuthorsOpenReview ↗

Global response — Part 2/2: supplementary evidence

Global response, Part 2: supplementary evidence. Revision numbers refer to the list in Global response Part 1.

1. Does the pairing effect survive joint attention learning? A two-layer causal Transformer (width 128, four heads) learns all Q/K/V matrices from binary next-output labels, without route labels. At 65,536 training examples, hard-minus-easy excess CE is 0.006868 [0.006374, 0.007292] and 0.001871 [0.001352, 0.002386] nats for χ=1,2\chi=1,2. These paired 95% BCa intervals use eight seeds. Each budget trains from initialization, with an additional independent validation set of 65,536 contexts and a separate test set of the same size. This supports the pairing effect under joint attention learning on fixed finite sources. Across all five budgets, FP32 hard-minus-easy means are positive in all ten profile–budget cells; nine pointwise intervals exclude zero. The finite-source functional has the same direction at every budget. BF16 and 4/6/8-bit storage retain nine positive intervals, versus eight for 3-bit and five for 2-bit. Please refer to Revision 4 in Line 3597-3614.

2. Does the resource evidence extend beyond one held-out corner, and how does free power choose resources? We compare four validation-calibration rectangles, averaging their test errors within each of 16 seeds before paired bootstrap. For weighted-average SGD, spectrum has lower joint-extrapolation NRMSE than free power on all six sources, with each paired interval excluding zero; free power has lower data-only and memory-only holdout errors. We also add the requested fitted free-power policy for staying, quadrupling data or quadrupling memory. It has lower mean regret than both spectral policies across the six sources; four contrasts with the calibrated spectral policy have intervals containing zero. Stratifying candidate endpoints shows that free power retains lower SGD regret both inside and beyond calibration support. Direct clipped estimation reverses the comparison in the extrapolated stratum. Prediction and resource choice are separate tasks. Please refer to Revision 5 in Line 3834-3867. Please refer to Revision 5 in Line 3305-3338. Please refer to Revision 5 in Line 3348-3357.

3. What explains the high small-model reference losses? On the same checkpoints and 96 test windows, FP32 evaluation gives 3.821960/3.288221 nats for 70M/160M, versus 6.200417/4.390311 under the tested BF16 eager path. BF16-rounded 70M parameters computed in FP32 give 3.827063. A same-parameter intervention protecting attention and output projection in FP32 brings mean CE within 0.000056–0.000195 nats of FP32 on 88 held-out windows across six scales. These comparisons separate parameter rounding from arithmetic-path sensitivity. Please refer to Revision 6 in Line 3343-3347. Please refer to Revision 6 in Line 3360-3366.

4. What does the completed Pythia comparison establish? We add 48 new FP32 training/evaluation trajectories: six scales, eight seeds, four additional-token budgets and six deployment formats. Responses fit each seed's validation losses at the first three budgets and 4/8-bit/BF16; the largest budget and 6-bit format are held out. Additive improves data-only extrapolation at every scale. The PTQ response, adapted from Kumar et al. (2025) to continuation data, improves 6-bit interpolation over additive at all six scales. Its extrapolation improves at 70M, 160M, 1B and 1.4B, is worse at 410M, and has a paired interval containing zero at 2.8B. All 48 PTQ penalty exponents γ\gamma reach zero; a zero-exponent control reproduces the errors. Fixed-penalty and interaction comparisons, small-model interpolation reversals, and all 2/3-bit measurements are reported alongside these results.

Measured 4/6-bit penalties relative to simultaneous BF16-stored weights decrease between the smallest and largest continuation budgets at all six scales. The 70M 2/3-bit penalties increase, and 8-bit changes are mixed. This helps interpret the constrained zero-exponent fit without treating it as constant measured quantization error. Please refer to Revision 7 in Line 3868-3914.

The controlled-source experiments directly evaluate the spectral functional. Pythia tests empirical responses to added training and deployment format; it does not estimate the pretrained models' predictive-energy spectra or theoretical crossover. Please refer to Revision 7 in Line 3371-3408. Please refer to Revision 7 in Line 3402-3408 and 3430-3433. The PTQ model is named descriptively in figures and tables, with its source cited in the caption and methods.

Every addition addresses a question raised in the reviews. The reviewer-specific responses provide the theoretical comparison, experimental definitions and per-scale numbers; the numbered Revision list locates the corresponding manuscript changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.