- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors theoretically investigate scaling laws for quantized neural computations. They especially focus on the relationship between the depth D and the cardinality M of an operation library. They show the joint law of D and M is not described by their marginal laws, and for a weighted operation simplex, M=2D − 1 is the smallest universal saturation library. They also show practical executions change the scaling law.
Critical strengths
Theoretical investigation of the scaling law of neural computations is an important topic. The manuscript is mathematically well-written. Each statement is clear, and provides some meaningful observations, such as the joint law is not described by marginal laws, there is a saturation boundary M = 2D−1, and the scaling law is different if the execution is different.
Critical weaknesses
The presentation should be improved for a wide range of readers. For example, the description of each notion defined in Section 2 is too brief. Providing some examples of Y and Z would help readers to understand the setting. Also, the connection between each section should be explained clearly. For example, in my understanding, in Section 3 is an example of Z, and how it is impremented by a practical network is described in Section 4. The authors should clearly explain what is discussed in each section.
Questions for authors
-
Theorem 2 shows that the interaction between depth and library size matters. However, in the paper’s model a single library is reused at every layer, which differs from standard architectures where each layer has its own weights. What do D and M correspond to in practical architectures?
-
In Theorem 4, is the choice of mainly for technical tractability, or is there a task-level motivation for it? For (including ℓ2), Theorem 6 provides only two-sided bounds up to constants. Does an exact characterization, in particular the saturation threshold , hold in ?
Depth and library size in practice, task norms, and section links
Thank you for the positive assessment and for the questions on how the quantities map to real networks.
Q1. What are D and M in practical architectures? D is the number of sequential residual updates (route decisions), and M the number of distinct operations available to them (Section 1, p. 2; Appendix A.3, p. 14):
The shared-library effects in our results therefore concern architectures that reuse operations. Appendix M.3 also tests routed libraries built from 22 and 35 pretrained ResNet blocks.
Q2. Why ℓ1, and does M=2D−1 hold in ℓq? In ℓ1 the per-mode errors add. This matches tasks that sum independent per-coordinate errors, and it makes the minimax problem exactly solvable (Theorem 4). The threshold M=2D−1 does not transfer to other norms. Proposition 21 (Appendix D.5, pp. 23–24) solves the two-mode Euclidean case at D=1: one mode suffices if and only if a2≤a1/3. For (a1,a2)=(1,0.9), the Euclidean radius falls from 0.9 with one mode to about 0.673 with two, while the ℓ1 radius is (1+0.9)/2=0.95 either way, by Theorem 4. Theorems 6 and 7 give two-sided rates in ℓq; an exact frontier for general norms remains open.
W1. Examples of Y and Z. A behavior in Y can be the hidden trajectory (h0,…,hD). Its observation in Z can be the final state hD or the logits WhD (Section 2, p. 3). The task distance compares observations only; this is what "task-quotiented" means.
W2. How the sections connect. Your reading is right: ℓ1 in Section 3 is the observation space Z of Section 2, and Section 4 realizes it with a network. Section 1 now ends with a reading map (Sections 2–7 follow Figure 1). The end of Section 2 states that Section 3 solves the problem with Z=ℓ1 and that Section 4 implements it in a network. Table 2 (p. 16) records which assumptions support each result.