acceptodds
Under review as a conference paper at ICLR 2027

Scaling Laws for Quantized Neural Computation: Geometry, Synthesis, and Execution

Abstract

Classical empirical neural scaling laws commonly treat scaling exponents as fitted constants, limiting mechanistic prediction before costly resource sweeps are executed. We develop a first-principles theory of quantized sequential neural computation that derives joint scaling surfaces directly from independently measurable structural profiles. First, we prove that marginal depth and library-size scaling rates cannot identify their joint law because depth dynamically reuses a finite operation library as a decoder. We resolve this bottleneck by introducing the task-quotiented shared-library reachable width. For a canonical spectral residual class, we solve the finite-resource minimax problem exactly, obtaining , where is depth, is library cardinality, and is the task-visible operation spectrum. This formula shows that depth resolves a clipped Cesàro profile rather than a primitive power law. Monotone regular variation yields a sharp summability transition in which nonsummable spectra retain spectrum-dependent rates, summable spectra enter a universal synthesis regime, and critical spectra are governed by an integrated slowly varying profile. Routed nonlinear residual adapters realize these phases, while task norm and loss geometry transform the observed exponent. Furthermore, execution semantics supplies a second structural law for arithmetic transport. Under fixed-horizon residual refinement, exact Green-operator identities show that full-state write-back has arithmetic amplification via backward-propagator mass, whereas first-order error feedback remains via adjoint semivariation. On a frozen pretrained ResNet-18 representation, a task-visible spectrum estimated from 5,000 calibration examples predicts all 49 registered depth–library errors on untouched test data with 1.02% median and 2.49% worst relative error without fitting a held-out exponent. Holding the ideal route fixed while altering only arithmetic semantics shifts the measured depth slope from 0.90 under write-back to 0.00 under error feedback, isolating a bit-width law of . Consequently, neural scaling exponents emerge not as primitive constants, but as identifiable, predictable summaries of operation geometry, finite synthesis, task geometry, and execution-error transport.

Primary area:
learning theory
Keywords:
scaling laws, quantization, neural computation, residual networks, approximation theory, neural scaling, error feedback, representation learning, deep learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VFXTOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors theoretically investigate scaling laws for quantized neural computations. They especially focus on the relationship between the depth D and the cardinality M of an operation library. They show the joint law of D and M is not described by their marginal laws, and for a weighted operation simplex, M=2D − 1 is the smallest universal saturation library. They also show practical executions change the scaling law.

Critical strengths

Theoretical investigation of the scaling law of neural computations is an important topic. The manuscript is mathematically well-written. Each statement is clear, and provides some meaningful observations, such as the joint law is not described by marginal laws, there is a saturation boundary M = 2D−1, and the scaling law is different if the execution is different.

Critical weaknesses

The presentation should be improved for a wide range of readers. For example, the description of each notion defined in Section 2 is too brief. Providing some examples of Y and Z would help readers to understand the setting. Also, the connection between each section should be explained clearly. For example, in my understanding, ℓ1\ell_1 in Section 3 is an example of Z, and how it is impremented by a practical network is described in Section 4. The authors should clearly explain what is discussed in each section.

Questions for authors
  • Theorem 2 shows that the interaction between depth and library size matters. However, in the paper’s model a single library is reused at every layer, which differs from standard architectures where each layer has its own weights. What do D and M correspond to in practical architectures?

  • In Theorem 4, is the choice of ℓ2\ell_2 mainly for technical tractability, or is there a task-level motivation for it? For ℓq\ell_q (including ℓ2), Theorem 6 provides only two-sided bounds up to constants. Does an exact characterization, in particular the saturation threshold M=2D−1M = 2D − 1, hold in ℓq\ell_q?

1 comment
AuthorsOpenReview ↗

Depth and library size in practice, task norms, and section links

Thank you for the positive assessment and for the questions on how the quantities map to real networks.

Q1. What are DD and MM in practical architectures? DD is the number of sequential residual updates (route decisions), and MM the number of distinct operations available to them (Section 1, p. 2; Appendix A.3, p. 14):

  • calling one tied block DD times gives depth DD with M=1M=1;
  • routed adapters and unrolled solvers choose among a small reusable set;
  • a standard network whose layers all have their own weights is the case M=DM=D, with each operation used once.

The shared-library effects in our results therefore concern architectures that reuse operations. Appendix M.3 also tests routed libraries built from 22 and 35 pretrained ResNet blocks.

Q2. Why ℓ1\ell_1, and does M=2D−1M=2D-1 hold in ℓq\ell_q? In ℓ1\ell_1 the per-mode errors add. This matches tasks that sum independent per-coordinate errors, and it makes the minimax problem exactly solvable (Theorem 4). The threshold M=2D−1M=2D-1 does not transfer to other norms. Proposition 21 (Appendix D.5, pp. 23–24) solves the two-mode Euclidean case at D=1D=1: one mode suffices if and only if a2≤a1/3a_2\le a_1/\sqrt3. For (a1,a2)=(1,0.9)(a_1,a_2)=(1,0.9), the Euclidean radius falls from 0.9 with one mode to about 0.673 with two, while the ℓ1\ell_1 radius is (1+0.9)/2=0.95(1+0.9)/2=0.95 either way, by Theorem 4. Theorems 6 and 7 give two-sided rates in ℓq\ell_q; an exact frontier for general norms remains open.

W1. Examples of YY and ZZ. A behavior in YY can be the hidden trajectory (h0,…,hD)(h_0,\dots,h_D). Its observation in ZZ can be the final state hDh_D or the logits WhDWh_D (Section 2, p. 3). The task distance compares observations only; this is what "task-quotiented" means.

W2. How the sections connect. Your reading is right: ℓ1\ell_1 in Section 3 is the observation space ZZ of Section 2, and Section 4 realizes it with a network. Section 1 now ends with a reading map (Sections 2–7 follow Figure 1). The end of Section 2 states that Section 3 solves the problem with Z=ℓ1Z=\ell_1 and that Section 4 implements it in a network. Table 2 (p. 16) records which assumptions support each result.

Reviewer_8ETROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows how joint scaling changes with depth and it studies operation-library size and arithmetic precision. It gives an exact mini max approximation formula for a weighted spectral class and identifies different spectrum-dependent regime. It analyzes how writeback and error feedback execution transports arithmetic errors.

Critical strengths
  1. The given exact finite resource formulation is a good theoretical contribution and gives more explanations than an asymptotic exponent
  2. The paper is able to distinguish geometry of approximation, arethmetic assumptions and geometry of task.
  3. The papers lists the limitations clearly
Critical weaknesses

One of my main concerns is that paper is not well presented and is hard to follow. The presentation is dense and it is difficult to track which assumptions apply to each result and how the each experiment is supporting broader claims

  1. The main emperical prediction provides only limited validation beyond spectrum stability
  2. the predictive claim is lacking simple baselines, as there is no comparision against additive or max type resource fits or against a predictor just using a fitted spectral power law.
  3. Novelty needs clear positioning against spectrum based learning curve thoeries (See [1,2,3]) the paper should clearly show what its exact spectral formula and the Green identities add beyond the results in above.
  4. The connection to standard quantized networks is not established sufficiently, since the theory and frozen SVD experiment retain coordinate-separable or orthogonal modes, than general interacting operations.
  5. The comparison does not hold total precision resources fixed, since error feedback uses a carry and high-precision accumulation. Equal quantizer bit widths alone do not establish an equal-resource advantage. [3]

[1] Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks - Blake Bordelon, 2020. [2] Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks - Abdulkadir Canatar, 2021. [3] The Accuracy of Floating Point Summation - Nicholas J. Higham, 1993.

Questions for authors

I have questions primarily around the experiments - please see weaknesses for some questions.

  1. Can the authors explain how the tie breaking rule in the 49 point grid affect which teacher is registered (app I.2.2)? Also, it is not clear to me what this experiment is establishing beyond stability. Can the authors confirm that?
  2. what is the precision used for the carry and the accumulator?
  3. which of the predictions are applying to one fixed learned target rather than the resource-dependent minimax witnesses?
  4. can the author explain why there are no simple baselines?
5 comments
AuthorsOpenReview ↗

Answers to Q1–Q4: registered teachers, carry precision, fixed targets, baselines

Thank you for the precise questions. We answer them in order; page numbers refer to the revised PDF.

Q1. How does the tie rule affect which teacher is registered, and what does the grid establish? At each of the 49 cells we compute two candidate witness teachers from calibration data only: an omitted-mode teacher and a half-grid teacher. We register the one with the larger calibration error, and an exact tie would go to the omitted-mode teacher. No tie occurred. The smallest gap between the two calibration-error scores was 0.0067, and 39 omitted-mode and 10 half-grid teachers were registered.

Both registered routes are idle, meaning they apply no operation. For the half-grid teacher, each weight 1/(2D)1/(2D) is equally far from 00 and 1/D1/D, and the lexicographic tie rule rounds it to 00. The omitted-mode teacher uses a mode outside the library. The grid therefore tests how well the measured spectrum transfers from calibration to test data, not routed execution. Section 7 (p. 8) and Appendix I.2.2 (p. 37) now say so.

Q2. What precision do the carry and accumulator use? In the submitted intervention both are FP32. That comparison shares the write grid with write-back but stores more state. Two new controls equalize storage (Proposition 12, p. 7; Section 7, p. 9; Appendix L.2, p. 44), and write-back wins all of them (see our W5 reply):

  • a native control: one signed 16-bit write-back lane against two signed 8-bit feedback lanes (accumulator and carry);
  • a bit-packed control that charges both feedback buffers at their actual widths.

Q3. Which predictions use one fixed target? The 49 registered teachers change with (D,M)(D,M). The new experiment holds 32 target coefficient vectors fixed across all 49 cells. It scores predictions on held-out cells and disjoint test images (Section 7, p. 9; Appendix L.1 and Table 3, pp. 43–44).

Q4. Why were there no simple baselines? The submitted grid targeted spectral transfer, so we had not fitted resource laws. The revision adds a fitted power-law spectrum and additive, maximum and product resource laws, on two splits:

  • the modular split fits 17 cells and scores the other 32;
  • the joint corner fits the other 45 cells, which include both corner depths, and scores the 4 cells with D,M≥128D,M\ge128.

With 64 modes, both corner library sizes equal the full library, so each target has only two distinct corner responses. Appendix L.1 states this limitation.

AuthorsOpenReview ↗

W1–W3: predictive evidence, baselines and relation to prior work

Presentation. We agree the submitted version was dense. Section 2 (p. 3) defines each object in words and follows Definition 1 with a two-step example. Section 5 (p. 6) defines write-back, feedback, the carry and the transport before Theorem 8. Table 2 (p. 16) maps each claim to its assumptions and evidence.

W1. Evidence beyond spectrum stability. We agree that a high calibration–test correlation of the spectrum would, alone, be limited evidence. In the new fixed-target experiment, 3,000 images estimate the operation profile, 2,000 disjoint images supply the errors used to fit the resource laws, and 10,000 test images evaluate the predictions. Each of 32 targets keeps the same coefficients at every budget (Section 7, p. 9; Appendix L.1, pp. 43–44).

W2. Simple baselines. On the modular holdout, pooled median absolute relative error is 0.295% for the measured profile and 2.008% for a fitted power-law spectrum. The additive, maximum and product resource fits reach 21.131%, 18.697% and 33.442%. Both spectral predictors receive the same target and route; the resource fits receive observed errors at training cells. Not every comparison favors us: the fitted spectrum wins on 2 of 32 within-target medians. Table 3 (p. 44) also gives 90th-percentile and worst errors.

W3. Relation to Bordelon et al. (2020), Canatar et al. (2021) and Higham (1993). The first two works relate kernel spectra and target alignment to generalization as the number of training samples grows. Here DD counts sequential route decisions and MM bounds a reusable library chosen before the target; no sample size appears. For separated writes and additive task error, Theorem 4 gives the exact worst-target error at every finite (D,M)(D,M), Δa(D,M)=12D∑i=12Dmax⁡{ai,aM+1}\Delta_a(D,M)=\frac{1}{2D}\sum_{i=1}^{2D}\max\lbrace a_i,a_{M+1}\rbrace. This includes the smallest universal saturation library, M=2D−1M=2D-1.

Higham analyzes floating-point summation, including compensated summation, which carries a correction forward. Unlike Higham, Theorem 8 does not model rounding of the carry; Proposition 12 charges it separately. Theorem 8 instead treats fixed, possibly non-commuting transport and a fixed readout, and gives the exact worst-case readout gain:

  • write-back adds the magnitudes of disturbances pulled back to the readout;
  • feedback adds their successive differences and a terminal term.

Proposition 12 then charges the stored carry. Appendix K.6 (pp. 42–43) gives the details.

AuthorsOpenReview ↗

W4–W5: interacting operations and equal storage

W4. Interacting operations. We now treat two mechanisms separately.

  • Operations that read the same input. Their cross-terms enter through a Gram matrix. For 16 learned, strongly overlapping operations (maximum output cosine 0.934), the full Gram matrix lowers median fixed-route prediction error from 54.01% (the diagonal, i.e. the separated spectrum) to 0.340% (Appendix L.1, p. 43). On the pretrained ResNet blocks below, which also interact through the state, the input-state Gram and diagonal medians are 11.8% and 17.2% (Appendix M.3).
  • Operations that read the updated state. Here order matters. Assume each operation has size at most aia_i and Lipschitz constant at most γai\gamma a_i on the reachable region, and let L=γa1L=\gamma a_1. Theorem 32 (Appendix M.1, pp. 45–46) shows that a prefix-balanced route then stays within c(L)Δa(D,M)c(L)\Delta_a(D,M), with c(L)=(3+3L)eL−2c(L)=(3+3L)e^L-2. Internal quantization adds an explicit term. Appendix M.2 (p. 46) shows that a field with a hard threshold can leave an error floor under every order, so the Lipschitz assumption cannot simply be dropped.

We tested this empirically, without certifying the Lipschitz assumption, on libraries of 22 ResNet-101 and 35 ResNet-152 pretrained bottleneck blocks, used as scaled residual updates in FP32 and W8A8. At depths 64 and 128 with the full library, balanced order beat blocked order in all 32 fixed-target comparisons on each network. One ResNet-101 target favored blocked order at depth 32. For ResNet-152, six criteria were set from ResNet-101 and written into the notebook before the run (no independent timestamp); five held. In the failed one, all 576 fixed-target rows lie below the calibration frontier, but 8 of 708 rows overall exceed it, by up to 7.3%. All eight come from one omitted-block witness whose calibration amplitude underestimates its flow (Table 5, p. 48). These are measured results for routed circuits of pretrained blocks; we do not claim a frontier for arbitrary trained networks.

W5. Equal storage. We agree. The submitted feedback advantage uses an FP32 carry and accumulator, so it is an advantage at a common write grid, not at equal resources. Proposition 12 (p. 7) charges the carry to storage. The stored carry is itself rounded, so write-back with the same total bits has strictly smaller worst-case error whenever its grid is finer than the carry grid. This holds under fixed transport, a nonzero readout, independent disturbance balls and non-overloaded grids. The experiments agree (Appendix L.2, p. 44):

  • Bit-packed storage. At equal persistent storage, write-back won all 680 pairs on trained refiner blocks. The pairs reuse write-back runs, so they are not independent. The median feedback/write-back ratio was 4.644 for RMS and 4.687 for mean per-image error.
  • Native storage. In the native equal-byte control, write-back won all 64 active pairs, and the 16 idle pairs tied. Every active native feedback arm clipped or saturated, so these pairs lie outside the proposition's assumptions.

Equal stored bytes do not imply equal compute or workspace.

Reviewer_YaZwOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a scaling law for quantization that is derived analytically rather than fitted to data. It assumes a linear correction map decomposed into rank-one operations and derives a closed-form expression for the approximation error as a function of depth and library size (which I did not verify). Two cases are considered: write-back, where quantization is applied to the residual stream after every step, and error feedback, where only the increment added at each step is quantized and the rounding residual is carried unquantized into the next step. As depth increases, the write-back error grows linearly while the error-feedback error stays bounded. Similarly, as the correction map is applied in more steps (larger D), the write-back feature error eventually diverges, whereas the error-feedback error keeps decreasing. The paper also presents experiments on ResNet-18 features: after collecting features from clean and disrupted data, the authors fit a linear map A on the disrupted features and decompose it into one-dimensional correction steps. These experiments show the same trend and additionally that, under error feedback, the mean feature error scales exponentially with the number of bits.

Critical strengths

The paper seems to validate the theory that they present with carefully designed experiments.

Critical weaknesses

The paper is very hard to follow, they use many terms before they are introduced. They do not give a clear example so that readers can understand the paper more easily.

Questions for authors
  1. In figure 4c the legend does not distinguish the difference between write-back and error-feedback. Both the shape of the marker (circle vs square) and the color specify the number of bits (10 vs 12). The legend should specify that the dashed line is the error-feedback and solid line is the write back.
  2. The experiments quantize the residual stream or the correction step, not a network. Can this study be demonstrated on a quantized model (layers from ResNet). Does the method work when we have non-linearities?
1 comment
AuthorsOpenReview ↗

Readability, Figure 4(c) and a quantized nonlinear block

Thank you for the clear suggestions on readability and on testing a quantized network.

Readability and an example. Terms are now defined in words where they first appear. Section 2 (p. 3) works through a two-step, two-operation example right after the definition of reachability. Section 5 (p. 6) defines write-back, error feedback and the carry before using them. Table 1 (p. 15) collects the notation, and Table 2 (p. 16) the assumptions behind each claim.

Q1. Figure 4(c). Solid lines are write-back and dashed lines are error feedback; circles mark the 10-bit grid and squares the 12-bit grid. The revised caption states both encodings, and Appendix I.2.3 (p. 38) adds a key. The comparison fixes the write grid and gives feedback an FP32 carry.

Q2. A quantized nonlinear block. We quantized both convolution weight tensors and the first post-ReLU activation of the last pretrained ResNet-18 BasicBlock, kept the outer ReLU, and repeated the block up to depth 32. Each execution is compared with a reference that uses the same internal quantizer and depth (Appendix L.3 and Table 4, pp. 44–45).

Feedback tracks that reference more closely. At depth 32 and 8 bits, its feature RMSE is 0.935 against 2.549 for write-back, and its logit RMSE 0.130 against 0.333. It lowers both errors in all 12 repeated-depth comparisons, although at depth 1 it increases both at all three widths.

Feedback does not improve true-label accuracy, however. Accuracy is lower in 11 of the 12 comparisons and tied in one; at 8 bits and depth 32 it is 23.44% against 35.16%. Feedback also stores twice the persistent state. The quantized reference itself falls from 85.16% accuracy at depth 1 to 18.75% at depth 32.

We also routed libraries of 22 ResNet-101 and 35 ResNet-152 pretrained bottleneck blocks, which are nonlinear, in FP32 and W8A8. Every fixed target stayed below the frontier computed from calibration amplitudes (Appendix M.3, pp. 47–48).

Hard changes of activation bin also break the local linearization behind Theorem 8, which is why Theorem 32 assumes Lipschitz operations. We therefore present this as a scoped test of the theory on a real nonlinear block, not as a general guarantee.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision and its scope

We thank the Area Chair and the reviewers for their careful reading and concrete questions. The revised PDF keeps the submitted theory, experiments and figures, and adds what the reviews asked for. Page numbers refer to the revised PDF.

Readability. Section 2 (p. 3) now gives concrete examples of behaviors and observations and works through a two-step example. Appendix A (pp. 14–16) collects the resource units, a glossary (Table 1), and the assumptions and evidence behind each claim (Table 2).

Prediction beyond spectrum transfer. We agree that the submitted 49-cell grid, whose worst-case witness targets change with depth and library size, mainly tests whether the measured spectrum transfers to held-out data. We therefore added a test in which 32 targets stay fixed across all budgets, with disjoint images for profiling, fitting and testing. On held-out cells the measured profile has 0.295% pooled median error. A fitted power-law spectrum has 2.008%, and additive, maximum and product resource laws have at least 18.697% (Section 7, p. 9; Table 3, p. 44).

Interacting operations. Theorem 32 (Appendix M.1, p. 45) extends the frontier to overlapping, state-dependent operations with Lipschitz bounds, executed in a balanced order (each operation's running count stays within one step of its proportional share). It is an upper bound up to an explicit constant. Appendix M.2 shows that a hard threshold can defeat every order. On libraries of 22 ResNet-101 and 35 ResNet-152 pretrained blocks, in FP32 and W8A8, balanced order beats blocked order for every fixed target at depths 64 and 128 with the full library. Of six ResNet-152 criteria written into the notebook before the run (no independent timestamp), five held. In the failed one, a single witness target exceeds the frontier Δa\Delta_a computed from calibration amplitudes by up to 7.3%, while all 576 fixed-target rows remain below it (Table 5, p. 48).

Equal resources. We agree that the submitted feedback advantage holds at a common write grid with an FP32 carry, not at equal storage. Proposition 12 (p. 7) charges the carry to storage. At equal persistent storage, write-back has the smaller error in all 680 bit-packed and all 64 native comparisons; the native feedback arms clip and so fall outside the proposition (Appendix L.2, p. 44).

Quantized nonlinear block. In a repeated, internally quantized ResNet-18 block, feedback lowers feature and logit error in all 12 repeated-depth comparisons. It does not improve true-label accuracy, and the quantized reference itself degrades with depth (Appendix L.3 and Table 4, pp. 44–45). We report these results as measured.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.