- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors provide a differentiable representation of how many ordered components a neural network should use. This differentiable representation allows to learn the component count together with the weights via a regularized loss minimization and therefore is usable in cases when both the weights and component count is unknown. Authors provide many controlled studies across different domains and ablations of hyper-parameters. They also show that a small adjustments to the method lead also to a method to learn a number of layers of a recursive network.
Critical strengths
The motivation is clear and method is well described. It is easy to implement and hyper-parameters of the method have a clear meaning and intuition. The method is also applicable in a wide range of domains.
Critical weaknesses
The method has a serious limitations
- (see question 3). If the authors do not clarify this question, we may conclude that the method is just a replacement of one unknown quantity (the components count) with another hyper-parameter ()
- (see question 6). Performance of the method seem to depend on the parameter which is not a priori known.
Some parts of the method are not properly evaluated (see questions 1, 4, 5).
The paper contains vague sentences ("The controlled study isolates the behavior of one boundary within a common function bank.", another example is in question 2 below).
Last but not least, the parameter (first mentioned in Section 4.3) seems not to be defined, this is a limitation to presentation.
Questions for authors
- what is the advantage of forcing to be a near integer in (5) when the formulas (3) and (4) seem to work with general ? Table 1 indeed shows that dropping the sinus term is worse (MSE of ) compared to keeping it (MSE of ), but when we compare the numbers to Figure 2, it seems that the case is still better then wrong choice of (that is influenced by the choice of ). Can you provide more empirical evidence that would show the benefit of including the sinus term and/or some explanation?
- what do you mean by “ being concentrated at the boundary” and how does it imply that “the task gradient is dominated by the current frontier”? The whole text around these formulations seems to be vague and also not supported by evidence
- Figure 2 shows that the learned depends substantially on ( has transition points and even though small changes to typically lead to no changes in , we do not know a priori at which scale the should be varied). Can you explain what is the benefit of the your method beyond moving the decision from “what is the optimal model size” to “what is the optimal ”? This limitation is somehow covered in the list of limitations, however, I consider this as a serious one and more attention should be put on it.
- as expected, Figure 3 indicates that the model performs worse with given when the was learned compared to when it was fixed manually. Do you have an empirical study that would show how far is the grown NGN model from the fixed-size baseline i.e. by looking at how many additional fine-tuning epochs do we need to get to the fixed-size baseline performance when we fix the learned (after it has been learned)? This study would help us to understand how much of the additional computation is needed when is unknown and we have to learn it.
- comparison of NGN to previous works (AWNN, STE, ) should contain a clear table with hard MSE of each of the methods under the same setting (not under the same deployed count as in Figure 4) so that we can assess methods for the tasks of MSE minimization.
- parameter is important design choice (or another method hyper-parameter) that determines the final choice of (see Figure 5). Figure 5 suggests that when is large enough, the method basically stays near to the initial value of . How do you determine the value for a unseen problem?
4 comments
(2/2)
Q3: The learned t depends on λ; at what scale should λ be set? Figure 2 is this ablation: it sweeps λ at the baseline setting. λ is a price on capacity, so it sets the size–performance trade-off. Three observations make the choice easy:
- Plateaus. The learned count stays on the same integer over roughly a factor of two in λ (Figure 2), so λ needs only a coarse choice.
- A narrow default range. Apart from the λ sweeps of Figures 2 and 3, every NGN run in Sections 4 and 5 uses an initial λ between 1e-4 and 2e-3 (Section 3.2, Tables 12–13).
- One price, many counts. A single λ sets up to 64 per-module counts at once, for example the LoRA ranks of Table 4. The equivalent size search is combinatorial.
On real tasks, accuracy is nearly flat around the learned size (Figure 6), so there is no single optimal size to recover, and λ sets where on this plateau the count stops (Section 5.1). The price can also be changed after training without restarting (Appendix J.4).
Q4: How many extra epochs does the grown model need to reach the fixed baseline? The aim of NGN is not to beat a fixed model whose size is given, but to learn the size within the training run, without a separate size search, at comparable performance (Section 4). The paper compares NGN and fixed models only at equal epochs, so it does not measure how many extra epochs would close the gap. NGN gets no extra training in any comparison, and the captions of Figures 3 and 6 and Tables 3 and 4 now say so. At that equal budget, fixed prefixes in the 1D study reach lower MSE at matched count, by a median factor of 2.9 across counts (Figure 3), and Section 6 notes that annealing the soft boundary can need more steps. On the four ten-seed tasks, the deployed model is within 0.4 points of fixed retraining at the same size and number of epochs (Table 3).
Q5: A clear table for AWNN, STE and L0 under the same setting. Appendix Table 6 lists all four methods on the same task and candidate blocks, with the same 50 seeds and 25,000 epochs, and each method's own hyperparameters (Table 12). It reports the retained count, soft and truncated MSE as mean ± s.d., and the median truncated MSE: NGN 1.3e-5, STE 6.4e-5, L0 2.6e-4 and AWNN 5.3e-4. Figure 4 plots every seed at its retained count round(t)+1.
Q6: How should r be chosen for an unseen problem? r is a structural default, not a per-task hyperparameter. It is 0.5 in every growth experiment except the depth variants and the pruning application (Tables 12–13). The r = 1 panel of Figure 5 is an ablation: without ordering, the count grows past every initial value. It shows that ordering is needed, not that r needs tuning. AWNN makes a similar choice: it fixes the exponential importance family, which its authors report worked well without extensive tuning.
We hope these revisions address your concerns, and we would be glad to clarify anything further.
I keep my rating (2: weak rejection). Most notably, without a solution to Q4 the significance of the method is questionable.
Thank you for the reply. As in our response, every comparison in the paper trains NGN and the fixed model for the same number of epochs, and on the four ten-seed tasks of Table 3 the deployed model is already within 0.4 points of fixed retraining at that budget, so no extra epochs are needed there. To answer Q4 directly for the 1D task of Figure 3, we ran one additional experiment.
Setup (10 seeds, baseline λ and β schedules). After the standard 25,000-epoch NGN run, we truncate the model to its learned prefix and continue training it as an ordinary fixed-size network: the boundary is frozen, there is no penalty, and the cosine schedule restarts. We compare it with a fixed model of the same count trained from scratch for 25,000 epochs on the same seed.
- No extra epochs: 4 of 10 seeds are already at or below the fixed model's MSE; the paired median ratio is 1.20×.
- Less than a quarter of a budget: one more seed reaches it after 5,786 extra epochs (23% of the original 25,000), so half of the seeds match the fixed model with at most 23% extra training.
- The rest: one seed needs 19,547 extra epochs; the other 4 do not reach it within 25,000 extra epochs (1.03×, 1.36×, 1.88×, and one NGN run in a poor basin). Across all seeds, the median ratio falls to 0.78× after continuation.
The 2.9× median gap in Figure 3 pools eight values of λ; the comparison above is paired at one λ. We hope this answers Q4.
(1/2) Thank you for the detailed questions. Each one led to a concrete change in the paper. Below, W1–W4 are the weaknesses and Q1–Q6 the questions of your review, in the order you wrote them.
W1: The method replaces one unknown (the count) with another (λ), and performance depends on r. We agree that NGN still needs a setting, but it is a different kind of quantity. The count is a property of the solution: it can only be evaluated by training a model of that size, so choosing it requires a grid of runs, and a combinatorial one when every layer has its own count. λ is a price on capacity: t grows only while one more unit of t lowers the task loss by more than λ, so the data decide the count. With the same λ₀ = 1e-4 and λ schedule (Table 13), NGN learns 9 blocks on MNIST MLP, 5–6 on MNIST CNN, 8–9 on CIFAR-10 and 2–3 on Cora (Figure 6). Within one model, one λ gives the two TinyStories FFN layers 2 and 9 blocks (Table 3) and the 64 LoRA modules different ranks (Table 4). The snapping strength α defaults to λ₀ (every Section 4 study except the manifold study, and two Section 5 tasks) and otherwise scales with the task loss (Table 13). λ₀ stays between 1e-4 and 2e-3 on every task, and on the four tasks above one NGN run costs 0.20–0.48 of a fixed-size grid trained for the same number of epochs (Appendix Table 8). Details are under Q3 (λ) and Q6 (r).
W2: Some parts are not properly evaluated. These are your Q1, Q4 and Q5, answered below with new evidence.
W3: Vague sentences. We rewrote them; the sentence you cite in Q2 is now a bound, answered below.
W4: r is not defined. The submitted version gave the weights as w_k = 2^{−k} in Section 3, but the symbol r first appeared in Section 4.3 without being tied to them. The revision defines it where the weights first appear (Section 3.1): w_k = r^k, with r = 0.5 except where noted.
Q1: What does forcing t to an integer add? A deployed model keeps a whole number of blocks, and truncating to that prefix is faithful only when t sits at an integer. The relevant column of Table 1 is the truncated MSE, which measures the deployed model. With the integer term, soft and truncated MSE agree (baseline: 1.31e-5 and 1.31e-5). Without it (α = 0), soft MSE is 5.43e-5 but truncated MSE is 4.83e-4, which is 37× the baseline. The soft column alone hides this gap.
The truncation bound (Section 3.1, Eq. 5) explains why. At an integer boundary, every gate is within σ(−β/2) of 0 or 1, so the soft and hard models differ by at most 0.0025 Σ|w_k|‖f_k‖ at β = 12. A quarter step away from an integer, the nearest gate deviates by 0.047. Sharpening β alone therefore does not make truncation faithful; the periodic term brings t to an integer.
On a "wrong α": on the deployed model, even a constant α is far better than none. In Table 1, constant α = 1e-4 reaches truncated MSE 1.41e-5, against 4.83e-4 for α = 0. The real risk is applying α too strongly too early. In Figure 5 (right), with α at full strength from the start, runs initialized above about eight stay where they start. This is why α ramps up from zero in every experiment (Section 3.2).
Q2: What does "g_k(1−g_k) is concentrated at the boundary" mean? We agree that the sentence was vague, and we replaced it with a bound. Since σ(z)σ(−z) ≤ e^{−|z|},
|∂g_k/∂t| = β g_k(1−g_k) ≤ β e^{−β|t−k+δ|}.
Each unit of distance from the boundary therefore multiplies the gate's sensitivity by e^{−β}, so the movement of t is determined by the components adjacent to it. Appendix A gives the full gradient ∂J/∂t and the proof.