acceptodds
Under review as a conference paper at ICLR 2027

NGN: Learning Neural Network Size as a Differentiable Count

Abstract

Neural network size is usually chosen before training, separating architecture selection from weight optimization. We introduce the Neurogenesis Network (NGN), a differentiable parameterization for learning how many ordered structural components a model should use. For each ordered component group, one learnable boundary selects an active prefix while the model parameters are trained. The boundary can grow from a compact initialization and can be deployed by discarding components beyond the learned boundary. Controlled experiments examine convergence of the learned boundary, the performance of deployed prefixes, and comparisons with fixed-size models and alternative approaches to learning capacity. We then apply the same mechanism to MLPs, convolutional and graph networks, Transformers, state-space models, LoRA, and adapters. Across these settings, deploying only the learned prefix usually changes performance little, and the selected architectures perform similarly to fixed models trained at the same size. These results show that structural capacity can be optimized directly as a count.

Primary area:
optimization
Keywords:
Differentiable model sizing, neural network capacity, structural sparsity

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_vgLMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors provide a differentiable representation of how many ordered components a neural network should use. This differentiable representation allows to learn the component count together with the weights via a regularized loss minimization and therefore is usable in cases when both the weights and component count is unknown. Authors provide many controlled studies across different domains and ablations of hyper-parameters. They also show that a small adjustments to the method lead also to a method to learn a number of layers of a recursive network.

Critical strengths

The motivation is clear and method is well described. It is easy to implement and hyper-parameters of the method have a clear meaning and intuition. The method is also applicable in a wide range of domains.

Critical weaknesses

The method has a serious limitations

  • (see question 3). If the authors do not clarify this question, we may conclude that the method is just a replacement of one unknown quantity (the components count) with another hyper-parameter (λ\lambda)
  • (see question 6). Performance of the method seem to depend on the parameter rr which is not a priori known.

Some parts of the method are not properly evaluated (see questions 1, 4, 5).

The paper contains vague sentences ("The controlled study isolates the behavior of one boundary within a common function bank.", another example is in question 2 below).

Last but not least, the parameter rr (first mentioned in Section 4.3) seems not to be defined, this is a limitation to presentation.

Questions for authors
  1. what is the advantage of forcing tt to be a near integer in (5) when the formulas (3) and (4) seem to work with general t>0t>0? Table 1 indeed shows that dropping the sinus term is worse (MSE of 5.43e−055.43e-05) compared to keeping it (MSE of 1.84e−051.84e-05), but when we compare the numbers to Figure 2, it seems that the case α=0\alpha=0 is still better then wrong choice of α\alpha (that is influenced by the choice of λ\lambda). Can you provide more empirical evidence that would show the benefit of including the sinus term and/or some explanation?
  2. what do you mean by “gk(1−gk)g_k\left(1-g_k\right) being concentrated at the boundary” and how does it imply that “the task gradient is dominated by the current frontier”? The whole text around these formulations seems to be vague and also not supported by evidence
  3. Figure 2 shows that the learned tt depends substantially on λ\lambda (tt has transition points and even though small changes to λ\lambda typically lead to no changes in tt, we do not know a priori at which scale the λ\lambda should be varied). Can you explain what is the benefit of the your method beyond moving the decision from “what is the optimal model size” to “what is the optimal λ\lambda”? This limitation is somehow covered in the list of limitations, however, I consider this as a serious one and more attention should be put on it.
  4. as expected, Figure 3 indicates that the model performs worse with given tt when the tt was learned compared to when it was fixed manually. Do you have an empirical study that would show how far is the grown NGN model from the fixed-size baseline i.e. by looking at how many additional fine-tuning epochs do we need to get to the fixed-size baseline performance when we fix the learned tt (after it has been learned)? This study would help us to understand how much of the additional computation is needed when tt is unknown and we have to learn it.
  5. comparison of NGN to previous works (AWNN, STE, L0L_0) should contain a clear table with hard MSE of each of the methods under the same setting (not under the same deployed count as in Figure 4) so that we can assess methods for the tasks of MSE minimization.
  6. parameter rr is important design choice (or another method hyper-parameter) that determines the final choice of tt (see Figure 5). Figure 5 suggests that when rr is large enough, the method basically stays near to the initial value of tt. How do you determine the value rr for a unseen problem?
4 comments

(1/2) Thank you for the detailed questions. Each one led to a concrete change in the paper. Below, W1–W4 are the weaknesses and Q1–Q6 the questions of your review, in the order you wrote them.

W1: The method replaces one unknown (the count) with another (λ), and performance depends on r. We agree that NGN still needs a setting, but it is a different kind of quantity. The count is a property of the solution: it can only be evaluated by training a model of that size, so choosing it requires a grid of runs, and a combinatorial one when every layer has its own count. λ is a price on capacity: t grows only while one more unit of t lowers the task loss by more than λ, so the data decide the count. With the same λ₀ = 1e-4 and λ schedule (Table 13), NGN learns 9 blocks on MNIST MLP, 5–6 on MNIST CNN, 8–9 on CIFAR-10 and 2–3 on Cora (Figure 6). Within one model, one λ gives the two TinyStories FFN layers 2 and 9 blocks (Table 3) and the 64 LoRA modules different ranks (Table 4). The snapping strength α defaults to λ₀ (every Section 4 study except the manifold study, and two Section 5 tasks) and otherwise scales with the task loss (Table 13). λ₀ stays between 1e-4 and 2e-3 on every task, and on the four tasks above one NGN run costs 0.20–0.48 of a fixed-size grid trained for the same number of epochs (Appendix Table 8). Details are under Q3 (λ) and Q6 (r).

W2: Some parts are not properly evaluated. These are your Q1, Q4 and Q5, answered below with new evidence.

W3: Vague sentences. We rewrote them; the sentence you cite in Q2 is now a bound, answered below.

W4: r is not defined. The submitted version gave the weights as w_k = 2^{−k} in Section 3, but the symbol r first appeared in Section 4.3 without being tied to them. The revision defines it where the weights first appear (Section 3.1): w_k = r^k, with r = 0.5 except where noted.

Q1: What does forcing t to an integer add? A deployed model keeps a whole number of blocks, and truncating to that prefix is faithful only when t sits at an integer. The relevant column of Table 1 is the truncated MSE, which measures the deployed model. With the integer term, soft and truncated MSE agree (baseline: 1.31e-5 and 1.31e-5). Without it (α = 0), soft MSE is 5.43e-5 but truncated MSE is 4.83e-4, which is 37× the baseline. The soft column alone hides this gap.

The truncation bound (Section 3.1, Eq. 5) explains why. At an integer boundary, every gate is within σ(−β/2) of 0 or 1, so the soft and hard models differ by at most 0.0025 Σ|w_k|‖f_k‖ at β = 12. A quarter step away from an integer, the nearest gate deviates by 0.047. Sharpening β alone therefore does not make truncation faithful; the periodic term brings t to an integer.

On a "wrong α": on the deployed model, even a constant α is far better than none. In Table 1, constant α = 1e-4 reaches truncated MSE 1.41e-5, against 4.83e-4 for α = 0. The real risk is applying α too strongly too early. In Figure 5 (right), with α at full strength from the start, runs initialized above about eight stay where they start. This is why α ramps up from zero in every experiment (Section 3.2).

Q2: What does "g_k(1−g_k) is concentrated at the boundary" mean? We agree that the sentence was vague, and we replaced it with a bound. Since σ(z)σ(−z) ≤ e^{−|z|},

|∂g_k/∂t| = β g_k(1−g_k) ≤ β e^{−β|t−k+δ|}.

Each unit of distance from the boundary therefore multiplies the gate's sensitivity by e^{−β}, so the movement of t is determined by the components adjacent to it. Appendix A gives the full gradient ∂J/∂t and the proof.

(2/2)

Q3: The learned t depends on λ; at what scale should λ be set? Figure 2 is this ablation: it sweeps λ at the baseline setting. λ is a price on capacity, so it sets the size–performance trade-off. Three observations make the choice easy:

  • Plateaus. The learned count stays on the same integer over roughly a factor of two in λ (Figure 2), so λ needs only a coarse choice.
  • A narrow default range. Apart from the λ sweeps of Figures 2 and 3, every NGN run in Sections 4 and 5 uses an initial λ between 1e-4 and 2e-3 (Section 3.2, Tables 12–13).
  • One price, many counts. A single λ sets up to 64 per-module counts at once, for example the LoRA ranks of Table 4. The equivalent size search is combinatorial.

On real tasks, accuracy is nearly flat around the learned size (Figure 6), so there is no single optimal size to recover, and λ sets where on this plateau the count stops (Section 5.1). The price can also be changed after training without restarting (Appendix J.4).

Q4: How many extra epochs does the grown model need to reach the fixed baseline? The aim of NGN is not to beat a fixed model whose size is given, but to learn the size within the training run, without a separate size search, at comparable performance (Section 4). The paper compares NGN and fixed models only at equal epochs, so it does not measure how many extra epochs would close the gap. NGN gets no extra training in any comparison, and the captions of Figures 3 and 6 and Tables 3 and 4 now say so. At that equal budget, fixed prefixes in the 1D study reach lower MSE at matched count, by a median factor of 2.9 across counts (Figure 3), and Section 6 notes that annealing the soft boundary can need more steps. On the four ten-seed tasks, the deployed model is within 0.4 points of fixed retraining at the same size and number of epochs (Table 3).

Q5: A clear table for AWNN, STE and L0 under the same setting. Appendix Table 6 lists all four methods on the same task and candidate blocks, with the same 50 seeds and 25,000 epochs, and each method's own hyperparameters (Table 12). It reports the retained count, soft and truncated MSE as mean ± s.d., and the median truncated MSE: NGN 1.3e-5, STE 6.4e-5, L0 2.6e-4 and AWNN 5.3e-4. Figure 4 plots every seed at its retained count round(t)+1.

Q6: How should r be chosen for an unseen problem? r is a structural default, not a per-task hyperparameter. It is 0.5 in every growth experiment except the depth variants and the pruning application (Tables 12–13). The r = 1 panel of Figure 5 is an ablation: without ordering, the count grows past every initial value. It shows that ordering is needed, not that r needs tuning. AWNN makes a similar choice: it fixes the exponential importance family, which its authors report worked well without extensive tuning.

We hope these revisions address your concerns, and we would be glad to clarify anything further.

I keep my rating (2: weak rejection). Most notably, without a solution to Q4 the significance of the method is questionable.

Thank you for the reply. As in our response, every comparison in the paper trains NGN and the fixed model for the same number of epochs, and on the four ten-seed tasks of Table 3 the deployed model is already within 0.4 points of fixed retraining at that budget, so no extra epochs are needed there. To answer Q4 directly for the 1D task of Figure 3, we ran one additional experiment.

Setup (10 seeds, baseline λ and β schedules). After the standard 25,000-epoch NGN run, we truncate the model to its learned prefix and continue training it as an ordinary fixed-size network: the boundary is frozen, there is no penalty, and the cosine schedule restarts. We compare it with a fixed model of the same count trained from scratch for 25,000 epochs on the same seed.

  • No extra epochs: 4 of 10 seeds are already at or below the fixed model's MSE; the paired median ratio is 1.20×.
  • Less than a quarter of a budget: one more seed reaches it after 5,786 extra epochs (23% of the original 25,000), so half of the seeds match the fixed model with at most 23% extra training.
  • The rest: one seed needs 19,547 extra epochs; the other 4 do not reach it within 25,000 extra epochs (1.03×, 1.36×, 1.88×, and one NGN run in a poor basin). Across all seeds, the median ratio falls to 0.78× after continuation.

The 2.9× median gap in Figure 3 pools eight values of λ; the comparison above is paired at one λ. We hope this answers Q4.

Reviewer_sorPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose that the structural capacity of a neural network, e.g. graph networks, Transformers, SSMs, LoRA, can be optimized directly as a count. The proposed method is identical to structured pruning hence requires clarification.

Critical strengths

Selection of capacity based on boundary selection is a great idea. I do not recall this has been implemented anywhere hence novelty remains to be tested.

The detailed ablations are appreciated.

Critical weaknesses
  1. Proposed method structurally prunes an overparameterized model using saliency metrics to obtain a more efficient model. This idea is not novel and has been studied in prior literatures. My understanding could be wrong and the authors are welcome to challenge this.

  2. Continually finetuning a smaller model while adding more parameters is the harder task which the authors sort of motivated in the earlier sections but failed to show that it works. The authors mainly shows that their approach works when downsizing, i.e. structural pruning, using their proposed metric.

  3. The authors do not explicitly compare with existing approaches and all experiments are conducted based on their on ablations.

Minor: Figure 3: since the x and y axis are shared, why not combine the plots into one for better comparison?

Questions for authors

Questions raised in weaknesses.

1 comment

Thank you for the review and for appreciating the boundary idea and the ablations. W1–W3 are the three weaknesses of your review, in order; your questions refer to these weaknesses, so we answer them together.

W1: The method is identical to structured pruning. We see how the submitted version suggested this, since it led with prune-from-high results. The two work in opposite directions. Structured pruning starts from the full model and removes components, either after training (SparseGPT, Wanda) or during it (L0, SoRA), and the count emerges from many separate keep-or-remove decisions. NGN starts from t ≈ 0 and adds components during training. One boundary over an ordered sequence sets the count, and the deployed model is the prefix it has grown.

The revised related work states this, and says that the main goal of NGN is not compression but learning size during training. Prior methods that learn size during training (AWNN, L0) are discussed there and compared directly in Figure 4 and Appendix Table 6. Pruning a pretrained model appears only as an application in Appendix J, with Wanda as a baseline.

W2: Growth is motivated but not shown; the paper mainly shows downsizing. Every main result in the paper is grown from t ≈ 0; none is obtained by downsizing. The revision makes this explicit:

  • Figure 1 shows the boundary growing from t ≈ 0 to its final count during training.
  • Section 3.2 states that every experiment starts from t ≈ 0.
  • Every LoRA rank and adapter width in Table 4 grows from zero in each module.
  • The former "prune from high" analysis is now presented as insensitivity to the initial boundary (Section 4.3), a robustness check rather than the main use.

One point of scope: the formulation is unbounded, but our experiments preallocate a finite candidate set, and new functions could instead be allocated as t grows (Section 3.2).

W3: No explicit comparison with existing approaches. The paper compares NGN with seven baselines, five of them published methods, and with a fixed-size grid:

ComparisonMethodsWhere
Capacity learning in the controlled taskAWNN, L0 (hard concrete), STEFigure 4, Appendix Table 6
Adaptive LoRA rankSoRA, AdaLoRATable 4, Appendix Table 17
Pruning a pretrained modelWanda, randomAppendix J
Fixed-size grid over counts (new)ten seeds per count, same epochs as NGNFigure 6, Appendix Tables 8–9

Minor: Figure 3 axes are shared. Thank you; Figure 3 is now a single plot with both curves.

We hope these revisions address your concerns, and we would be glad to clarify anything further.

Reviewer_RrxLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce the Neurogenesis Network (NGN), a differentiable parameterization for learning how many ordered structural components a model should use. For each ordered component group, one learnable boundary selects an active prefix while the model parameters are trained. The boundary can grow from a compact initialization and can be deployed by discarding components beyond the learned boundary. These results show that structural capacity can be optimized directly as a count.

Critical strengths

1.The core idea of NGN is simple and neat. Instead of learning an independent gate for every component, NGN only learns a single shared boundary t over an ordered sequence. This allows the model to optimize structural count and model weights jointly during training.

2.The proposed mechanism is general and flexible. The authors apply the exact same boundary parameterization across diverse model families, including MLPs, CNNs, GNNs, Transformers, and Mamba. Moreover, it can control different structural dimensions, such as layer width, network depth, LoRA rank, and adapter width.

Critical weaknesses

1.The goal of this paper is to avoid manual model size selection. However, the proposed shared boundary introduces a new set of hyperparameters. Across different tasks, the authors still manually tune the ranges and schedules of these parameters (Table 10). Therefore, the hyperparameter tuning burden is not actually reduced.

2.NGN can safely discard components beyond the boundary only if t converges to an integer. However, this setting often fails on complex models (such as TinyStories in Table 3 and Llama-3-8B in Table 12). As a result, hard truncation causes a clear performance drop.

3.NGN only runs multiple random seeds on simple and small-scale experiments. The authors should also report multi-seed results on modern architectures.

Questions for authors

NONE

1 comment

Thank you for the positive assessment of the core idea and its generality. W1–W3 follow the numbering of your review.

W1: New hyperparameters replace size tuning (Table 13). We agree that NGN still has settings, but they are a different kind of quantity from the size. A size can only be evaluated by training a model of that size, so choosing it requires a grid of runs, and a combinatorial one when every layer has its own size. NGN's settings are a price and defaults that carry over across tasks:

  • λ (capacity price). t grows only while one more unit of t lowers the task loss by more than λ, so the data decide the count. With the same λ₀ = 1e-4, NGN learns 9 blocks on MNIST MLP, 5–6 on MNIST CNN, 8–9 on CIFAR-10 and 2–3 on Cora (Figure 6). Apart from the λ sweeps of Figures 2 and 3, every NGN run in Sections 4 and 5 uses an initial λ between 1e-4 and 2e-3, and the learned count stays on a plateau over about 2× in λ (Figure 2).
  • α (snapping strength). The default is α = λ₀, used in every Section 4 study except the manifold study (λ₀/2) and on MNIST MLP and TinyStories FFN. It ramps up from zero, so it does not hold back early growth, and it only needs to be large enough to snap t. It scales with the magnitude of the task loss, and on the other tasks it is 10–1000× λ₀ (Table 13).
  • Defaults shared across tasks. r = 0.5, the schedule shapes (decreasing λ, increasing α, increasing β), and an initial boundary t ≈ 0.

What this buys is shown by the new fixed-size grids (Section 5.1, Figure 6, Appendix Tables 8–9). On MNIST MLP, MNIST CNN, CIFAR-10 and Cora, NGN lands on the accuracy plateau of a ten-seed grid at 0.20–0.48 of the grid's cost, with every run trained for the same number of epochs. For per-module allocations, such as 64 LoRA ranks, a grid is not feasible at all.

W2: Truncation requires an integer t, which fails on complex models. We agree, and Section 6 reports this as a limitation. On TinyStories, truncation raises BPC by 0.3–2.6%. For PEFT adapters, it costs 2.5–3.1 points on four of five tasks, where only 12–47% of boundaries end near an integer. Boundaries between integers explain part of this, but not all: the TinyStories joint row has near-integer boundaries and still a 2.6% gap.

Where t reaches an integer, Section 3.1 bounds the soft-to-hard gap (Eq. 5). On the four ten-seed tasks, truncation leaves the median score unchanged, and the deployed model is within 0.4 points of fixed retraining at the same size and number of epochs (Table 3). LoRA is in between: on WikiSQL only 20–28% of boundaries end near an integer, yet soft and truncated scores are within 0.4 points, and on all five tasks no difference to the fixed LoRA model with the same allocation and epochs exceeds the 95% evaluation interval (Table 4, Appendix Table 19). Making snapping reliable on larger models is future work.

W3: Multiple seeds only on small experiments. We agree; this is a weakness of the paper. The TinyStories and PEFT rows are single runs because each language-model run is expensive, and the depth study is also a single run. Section 6 lists these as a limitation. The repeated evidence in the paper is:

  • 50 seeds for the capacity-matched comparison (Figure 3), the initial-boundary sweep (Figure 5) and the capacity-learning comparison (Figure 4);
  • 10 seeds per setting for the schedule ablation and for each manifold dimension;
  • 10 seeds for each of the four ten-seed rows of Table 3 and for every count of the new fixed-size grids.

For the single-run language-model rows, Appendix H gives the 95% binomial interval for each benchmark (±3.0 to ±4.4 points), and we do not interpret differences smaller than these intervals. The paper's central claims, learning the count in the training run and comparable deployed performance on the ten-seed tasks, do not rest on the single-run rows.

We hope these revisions address your concerns, and we would be glad to clarify anything further.

Reviewer_Un4aOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents NGN, a framework that learns the neural network size jointly with the model weights. Specifically, the framework replaces traditional layer structure with a multi-branch structure consisting of gated components (MLP/convolution/attention blocks). The number of active gates is learned as a separate parameter in combination with the weights. The loss function is modified to trade-off model size and performance. Results demonstrate that on several small-datasets and toy examples the model achieves competitive performance compared to fixed models of comparative sizes, moreover on synthetic data of dimension D the framework correctly picks the gating setup that correctly allows for learning to represent the D-dimensional manifold. The framework is tested using MLPs, CNNs, and transformer blocks on vision and language data.

Critical strengths
  • The framework is tested on a wide array of network types and tasks, including dense layers, convolution blocks, self-attention transformer blocks, and MAMBA blocks.
  • Good sanity checks such as the prune from high tests are included.
Critical weaknesses
  • The central claim of the paper is not supported. The author correctly notes in the introduction that usually "selecting these quantities (layer width) usually requires expertise or repeated training across candidate sizes". While the proposed framework eliminates the need for testing multiple runs, it takes longer to converge. It is unclear that simply running an initial test with several sizes would not find better fitting architectures faster (for instance, training for a single epoch could already provide intuition towards which architectures might be best suited for the task ).
  • The difference between reported baselines and the method seems relatively small. For instance for MNIST the difference between the fixed and soft accuracy is minimal with overlapping standard deviations (no statistical tests are reported). Moreover, for a number of rows (such as for GSM8K and WikiSQL) the truncated model scores higher than the soft model. Since there is no theoretical reason why this is should be the case, it stands to reason that the differences reflect dataset noise. This further highlights that the differences between the fixed and NGN models might be insignificant as they are similar size to the differences between truncated vs soft results. Considering the computational cost of NGN it is unclear what the benefit of the methods are considering there is no guarantee or much evidence that the method finds the optimal architecture setup (also see next point).
  • Only for the D-dimensional manifold toy example with synthetic data does the test that the framework finds the optimal D. But the example seems geared towards the method, as the optimum is very sharp: missing dimensions would cause large errors, and extra dimensions carry no information. Realistic tasks rarely have such a sharp optimum.
  • The base Lama-3-8b performance on GSM8K is 45.2% which is very low compared to the literature. This leads me to believe that the results reflect idiosyncrasies in the prompt / numerical extraction setup used. In this case, the improvement might be due to the model learning the expected answer format rather than learning real dataset related patterns. This is further supported by SoRA scoring well below the base model.
  • The paper needs to discuss Neural Architecture Search, as it more directly relevant to the presented method in comparison to literature in post-hoc pruning / sparsification. For instance, FBNetV2 (Wan et al., 2020) similarly replaces layers with multi-branch components and learns which "channels" to mask, DARTS (liu et al., 2019) learns architecture choices jointly with the weights very similarly to the proposed NGN method, MorphNet (Gordon et al., 2018) learns layer widths under a resource penalty, TAS (Dong el al, 2019) learns channel counts... the novelty of NGN needs to be reframed appropriately.
  • The paper is at times hard to follow, figure 5 for instance is difficult to understand. As I understand it anything above the x=y line represents growing runs while blow it are pruning runs? please add the reference line and/or use different colors, also add information about performance agreement and not just final size.
Questions for authors
  • How was the GMS8K evaluation carried out exactly? why does the base model performance appear to be relatively low?
2 comments

(1/2) Thank you for the thorough review and the concrete suggestions. Below, W1–W7 are the weaknesses of your review in the order you wrote them, and Q1–Q2 are your two questions. Your questions concern the same GSM8K issue as W5, so we answer them together under W5.

W1: A single run is slower, and a short test of several sizes might be faster. We now measure this.

  • Cost. On MNIST MLP, MNIST CNN, CIFAR-10 and Cora, at equal epochs one NGN run costs 1.0–2.3× a fixed run at the learned size, or 0.20–0.48 of a five- or six-size grid (Appendix Table 8); the overhead comes from evaluating every preallocated candidate during training. For LoRA and adapters at equal epochs, one NGN run takes 0.97–1.03× the training time of the matched fixed run on all ten runs of Table 4, and SoRA and AdaLoRA take 1.00× and 1.05–1.06× NGN's time (Appendix Table 18).
  • Quality. Accuracy is nearly flat around the learned size, and NGN lands on that plateau (Section 5.1, Figure 6).
  • Short-run search. This is a fair alternative, which we did not test; rankings after one epoch need not match converged rankings.
  • Combinatorics. Search cannot compete on per-module allocation: choosing a LoRA rank from 16 values for each of 64 projections gives 16^64 configurations, while NGN learns all 64 counts in one run.

W2: Small differences, overlapping standard deviations, no statistical tests. We agree that the differences are small. The aim of NGN is not to beat a fixed model, but to learn the size within the training run, without a separate size search, at comparable performance; Section 4 now states this purpose explicitly. To quantify the differences, Welch 95% intervals for NGN minus the grid-selected fixed model, from ten seeds per arm with all runs trained for the same number of epochs, are in the caption of Appendix Table 9:

TaskNGN − grid-selected (points)
MNIST MLP+0.12 [+0.04, +0.20]
MNIST CNN+0.05 [+0.01, +0.09]
CIFAR-10−0.28 [−0.54, −0.03]
Cora−0.52 [−1.99, +0.95]

NGN is slightly better on both MNIST tasks, slightly worse on CIFAR-10, and indistinguishable on Cora, where variance across seeds is high. The comparison also favors the baseline: the grid-selected fixed model is given the best of five or six sizes, chosen on test accuracy for MNIST and CIFAR-10, while NGN finds its size within a single run. Landing within a few tenths of a point of it is the intended result.

W3: Truncated scores higher than soft on GSM8K and WikiSQL. We agree with your reading: these differences are within evaluation noise. Truncation changes the predictor, so its score can move either way. The 95% binomial half-widths are ±4.4 points on GSM8K (500 problems) and ±3.1 on WikiSQL (1,000). The observed soft-to-truncated differences are 1.6 points on GSM8K and 0.4 on each WikiSQL task. Appendix H reports these intervals, and the paper does not interpret differences below them.

W4: The manifold optimum is sharp and unrealistic. The manifold study is a control experiment. It does not assume that real tasks have a known size. It asks whether the learned count follows the data rather than the settings, and that question needs a task where the right size is known. The task is built so that the answer is sharp: the data are a D-dimensional box mapped smoothly into 16 dimensions, so fewer than D bottleneck terms cannot reconstruct it, and a term beyond D has nothing left to encode (Appendix C).

  • Same settings for every D. λ, α, the schedules and the ten candidates are identical for D = 1, 2, 4, 6 and 8, and every run starts from t = 0. Only the data change, and the learned count follows them: it equals D in 49 of 50 runs (Table 2).
  • Failures are reported. One D = 8 seed selects 7 terms, and four seeds have visible soft-to-truncated gaps (Appendix Table 11).

On real tasks, accuracy is flat around the learned size (Figure 6), so there we claim only that NGN lands on that plateau.

(2/2)

W5, Q1 and Q2: How GSM8K was evaluated (Q1), and why the base score is low (Q2). The protocol is documented in Appendix H (Tables 19–20).

  • Setup. lm-eval GSM8K on 500 test problems, with plain zero-shot completions: no chat template, no in-context examples, greedy decoding.
  • Scoring. Table 4 reports flexible extraction (the last number in the generation); Table 20 gives both filters for every run.
  • Why the base score is low. The base model scores 45.2% under flexible extraction and 0.4% under strict matching, because it never writes the #### N answer line. This is far below published few-shot chain-of-thought results, which use a different protocol.
  • Comparisons we still make. Every adapted run is trained on the same GSM8K solutions, which end in the #### N line. Under strict matching, NGN is within 1.4 points of the fixed model at its size and epochs, for both LoRA and adapters.
  • Why SoRA scores below the base model. The records point to answer extraction, not a failed run (Appendix H):
    • SoRA learned the format: it scores 46.2% under strict matching, against the base model's 0.4%.
    • Under flexible extraction it scores 10.6 points lower, so in at least 53 of 500 problems its correct answer line is followed by a different final number. The other adapted runs except AdaLoRA show the same effect to a smaller degree (0.2–7.8 points). AdaLoRA shows the opposite: 3.8% strict and 56.8% flexible, so it rarely writes the answer line.
    • The model is not damaged: its Pile perplexity is 0.97 of the base model's, as for NGN and the fixed model, and its scores on the other four tasks are in the range of the other methods.
    • SoRA also stops at rank 11 against the target of 18.
  • What we no longer claim. We draw no conclusion about NGN against SoRA or AdaLoRA on GSM8K, and the Table 4 caption no longer ranks them.

W6: Discussion of NAS. Related work now has a NAS paragraph. Typical NAS trains and evaluates many candidates and retrains the selected one. Differentiable and one-shot methods (DARTS, FBNetV2) share weights in a supernetwork but still choose among a bounded, predefined set of options. The paragraph also cites the width-search methods you named: MorphNet alternates a resource-weighted sparsifying regularizer with uniform width expansion, and TAS learns a distribution over candidate channel counts per layer; both then train the selected network separately. These methods choose an architecture and then train it, while NGN learns how many ordered components a given architecture uses, in the run that trains the deployed weights. The two are complementary, and NGN could serve as a count primitive inside a search.

W7: Figure 5 is hard to follow.

  • The revised figure adds a dashed t = t_init reference line.
  • The NGN baseline is the left panel, and the two ablations are labelled.
  • The caption states what each panel shows.
  • Performance agreement. For the NGN baseline (50 seeds per initial value), the median truncated MSE stays between 9.8e-6 and 1.9e-5 for every initial boundary from 0 to 16, and soft and truncated MSE agree at every initialization. The final models therefore perform alike, not only their counts.
  • Real tasks. Appendix Table 10 reports the same check. A high initial boundary does not always return to the count learned from t₀ = 0, so all main results start from t₀ = 0.

We hope these revisions address your concerns, and we would be glad to clarify anything further.

Comments on the paper

1 comment

We thank all reviewers for their careful reading. We have uploaded a revised PDF and respond to each review separately.

Numbering. Our responses use the numbering of the revised PDF. Tables 1–3 and Figures 1–5 keep their numbers. The submitted Table 5 (PEFT) is now Table 4, Table 10 (settings) is Table 13, and Table 12 (PEFT diagnostics) is Table 16. The depth study (submitted Section 5.2 and Table 4) is now Appendix F and Table 15.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.