- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Residual-Aware Neural Spectral Capacity (ResNSC), an extension of NSC for scoring neural architectures from their specifications without training or instantiated weights. ResNSC maintains a coordinate-wise reference for accumulated residual variance and uses it to construct projection-level input and noise covariances. A history-aware allocator then selects residual and FFN widths under a resource budget. The paper evaluates the method through from-scratch pretraining, inherited-weight pruning, and architecture-ranking benchmarks.
Critical strengths
-
The paper demonstrates a meaningful limitation of NSC. The reversal experiments show that rearranging the same projection configurations can change trained performance despite preserving the NSC score.
-
The mathematical formulation is clearly scoped. The local identities and optimization proofs appear sound under the stated reference-model assumptions. The history-aware state representation provides a principled way to optimize the proposed numerical objective, with guarantees restricted to the specified finite candidate set.
-
The architecture-selection results are promising. Table 2 shows improved pretraining performance over NSC-selected architectures at comparable parameter budgets. Table 4 reports substantial pruning improvements across matched budgets using the same supernet and candidate menus.
Critical weaknesses
-
The architectural setting needs clearer explanation and separation across experiments. At pretraining, the full residual state has a fixed width within each model with only a prefix of its coordinates update at each block; at pruning and ranking experiments, constant active residual widths are used. This distinction is essential for understanding both the architecture and the residual ledger, but is easy to miss.
-
The pretraining comparisons do not establish matched training compute. Different residual-width profiles can incur different attention FLOPs. The paper does not report total training FLOPs for each pretraining configuration. It is therefore unclear whether the improvements persist under a matched training-FLOP budget.
-
The statement - "that architectural context can help guide resource allocation." - is loosely supported by the provided benchmarks in the ranking experiment. Table 5 shows nearly unchanged correlations on FlexiBERT and GPT-2, with an improvement on AutoFormer-Small. Increasing the number of distinct scores demonstrates greater sensitivity to architectural differences, but does not by itself establish that those distinctions are predictive.
Questions for authors
See the weaknesses.
5 comments
Response to Reviewer LSnS (2/3): W2
2. Comparison at matched training FLOPs (W2)
Thank you for highlighting the fact that comparable parameter counts and equal training tokens do not necessarily imply equal training FLOPs, because different residual widths can change attention cost. The two tables below show that ResNSC's performance benefits are not solely explained by higher training FLOPs. The first examines the comparisons reported in the submitted paper; the second adds matched-compute comparisons with VWT at both scales.
Existing comparisons. The following table pairs estimated training FLOPs with the mean WikiText-2 and LAMBADA perplexities reported in Table 2 of the submitted paper. At both scales, ResNSC achieves lower perplexities than NSC with 3.1% and 4.1% fewer estimated FLOPs, and lower perplexities than Reverse at equal estimated FLOPs. At 374M, it also achieves lower perplexities than Uniform and TLM with fewer estimated training FLOPs.
| Method | 121M EFLOPs | 121M WikiText-2 | 121M LAMBADA | 374M EFLOPs | 374M WikiText-2 | 374M LAMBADA |
|---|---|---|---|---|---|---|
| ResNSC | 3.137 | 51.13 | 202.8 | 26.463 | 31.20 | 53.0 |
| NSC | 3.237 | 53.40 | 221.4 | 27.604 | 32.40 | 66.9 |
| Reverse | 3.137 | 53.22 | 234.6 | 26.463 | 33.93 | 77.7 |
| Uniform (a) | 2.950 | 55.47 | 262.9 | 26.777 | 33.82 | 76.2 |
| TLM (a) | 2.950 | 55.43 | 252.1 | 26.777 | 34.19 | 78.1 |
| VWT | 2.711 | 52.33 | 252.6 | 25.017 | 31.99 | 70.6 |
(a) Projection-parameter totals are nearly equal within each scale, but ResNSC's mean active width is larger at 121M (730 versus 640) and smaller at 374M (992 versus 1,024). Attention-product cost follows the sum of active widths, explaining the higher/lower FLOPs relative to Uniform/TLM. Each scale is searched independently under a parameter ceiling.
Reduced-budget comparison with VWT. In the submitted paper, the architectures found by ResNSC at both scales use more training FLOPs than VWT. To test whether the gains persist without this additional compute, we reduce ResNSC's training-token budgets to approximately 86.4% and 94.5% of the submitted-paper budgets at 121M and 374M, respectively, matching VWT's 2.711 and 25.017 EFLOPs. Token accounting includes both main training and final decay. The table compares the reduced-budget runs with VWT's submitted-paper means.
| Method | 121M EFLOPs | 121M WikiText-2 | 121M LAMBADA | 374M EFLOPs | 374M WikiText-2 | 374M LAMBADA |
|---|---|---|---|---|---|---|
| ResNSC (reduced training-token) | 2.711 | 52.53 | 212.0 | 25.016 | 31.55 | 52.9 |
| VWT | 2.711 | 52.33 | 252.6 | 25.017 | 31.99 | 70.6 |
At matched training FLOPs, ResNSC retains lower LAMBADA perplexity at both scales and lower WikiText-2 perplexity at 374M. At 121M, WikiText-2 perplexity is 0.20 higher than VWT's (52.53 versus 52.33). The same reduced-budget 121M run also uses fewer estimated FLOPs than Uniform and TLM (2.711 versus 2.950 EFLOPs) and retains lower WikiText-2 and LAMBADA perplexities than both (52.53 and 212.0, versus 55.47 and 262.9 for Uniform and 55.43 and 252.1 for TLM).
Training FLOP estimation. We use the same dense matrix-multiplication estimate for every method:
Here and are the active residual and FFN hidden widths, is sequence length, is vocabulary size, and is the embedding/readout width (640 at 121M; 1024 at 374M). includes main-training and final-decay tokens. The estimate covers attention/FFN projections, attention products, and vocabulary readout, counting two FLOPs per multiply-add and the backward pass as twice the forward pass. One EFLOP is operations.
Changes in the revised paper. Sec. 4.1 will clarify the training-FLOP comparisons at both scales and report the reduced-budget comparisons with VWT: lower LAMBADA perplexity at both scales, lower WikiText-2 perplexity at 374M, and 0.20 higher WikiText-2 perplexity at 121M. The training-and-evaluation appendix will include both comparison tables, the dense FLOP-estimation formula, and the token-budget adjustments to approximately 86.4% and 94.5% of the submitted-paper budgets at 121M and 374M, counting both main training and final decay.
Response to Reviewer LSnS (3/3): W3
3. What the ranking experiments show (W3)
We agree with the reviewer that the greater architectural sensitivity obtained by incorporating residual context in this work does not establish better prediction in all cases. We distinguish two regimes:
-
Global ranking of diverse architectures with large variations in model size. Parameter count alone can already be a strong performance predictor in this regime (Kaplan et al., 2020; Serianni and Kalita, 2023; Wang, 2025; see also Appendix I). Residual context does not consistently improve prediction over NSC across the evaluated benchmarks: correlations are nearly unchanged on FlexiBERT and GPT-2 (Table 23), with mixed results on AutoFormer (Appendix I).
-
Substantial benefits for subtle variations that NSC cannot distinguish. At similar model sizes, NSC supplements parameter count by capturing how parameters are organized into projections (Zhu et al., 2026, Sec. 4.1). However, its position-independent aggregation cannot distinguish permutations of the same layer configurations. Resolving these differences is important for designing Transformers with heterogeneous layer settings and for pruning Transformers; our residual-context model enables ResNSC to distinguish such arrangements.
The submitted paper supports this indirectly through the improved performance of architectures selected by ResNSC-DP in variable-width Transformer design and pruning (Tables 2 and 4). To directly test the predictive value of the additional distinctions, the table below compares pairs with identical or similar NSC scores from 1,000 evaluated LoNAS subnetworks, using seven-task mean accuracy under inherited weights.
| NSC score difference | Pairs | Pairwise accuracy: ResNSC | Pairwise accuracy: NSC | Pairwise Kendall : ResNSC | Pairwise Kendall : NSC |
|---|---|---|---|---|---|
| Identical | 155 | 71.0% | -- | 0.42 | -- |
| Within 1% of range | 17,700 | 73.8% | 51.3% | 0.48 | 0.03 |
| Within 5% of range | 88,046 | 74.2% | 52.7% | 0.48 | 0.05 |
| Within 10% of range | 171,900 | 74.0% | 55.3% | 0.48 | 0.11 |
The identical-score pairs differ only in the ordering of their FFN widths across layers and have equal parameter counts; ResNSC orders 110 of these 155 pairs correctly. For pairs with similar NSC scores, its ranking accuracy is 73.8–74.2%, versus 51.3–55.3% for NSC. These results directly demonstrate useful performance distinctions among architectures that NSC cannot separate or scores similarly.
Changes in the revised paper. Sec. 4.3 will clarify that the additional architectural sensitivity from residual context does not establish better prediction in all cases. It will distinguish global ranking across diverse model sizes from resolving layer arrangements that NSC cannot distinguish, and cite prior work supporting parameter count as a strong predictor in the former regime. Appendix I will add the LoNAS pairwise results and evaluation protocol. The resource-allocation conclusion will be grounded in the architecture-selection results in Secs. 4.1–4.2 (Tables 2 and 4), with the new pairwise analysis providing direct prediction evidence.
Response to Reviewer LSnS (1/3): W1
We thank the reviewer for recognizing the limitation of NSC demonstrated by the reversal experiments, the clearly scoped mathematical guarantees, and the promising architecture-selection results. In response, we clarify the architectural settings across experiments (W1), report training-FLOP estimates and add comparisons with VWT at matched training FLOPs at both scales (W2), and refine our prediction claims with direct pairwise evidence for architectural distinctions that ResNSC enables us to resolve (W3).
1. Architectural setting and residual history of each experiment (W1)
Thank you for highlighting this distinction. We clarify below how the architectural settings and corresponding residual histories differ across experiments.
In the pretraining architectures, the full residual-state width is fixed within each model, while the active residual width may vary across blocks. Each block's attention and FFN branches read and update only the prefix of residual coordinates within that block's active width; the remaining coordinates are preserved. The FFN hidden width controls the expansion inside the branch, whose output returns to the active residual coordinates (Sec. 3.2; Appendix B.1).
The fixed-width pruning and ranking settings therefore retain position-dependent scoring, while variable-width pretraining also accounts for differences in residual history across coordinates.
Changes in the revised paper. Sec. 3.2 will distinguish the full residual-state width, active residual width, and FFN hidden width, and explicitly state that inactive coordinates are preserved. The comparison table above will be added at the beginning of Sec. 4. The setup paragraphs of Secs. 4.1–4.3 will identify each experiment's active-width setting and corresponding residual history.