acceptodds
Withdrawn from ICLR 2027

Lost in Aggregation: A General Perspective on Zero-Cost Neural Architecture Search

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selecting neural network architectures typically requires training and comparing many candidates. Even at the same parameter scale, different connectivity patterns and operations can lead to substantial performance differences. Zero-cost neural architecture search (NAS) evaluates candidates before training and thus offers a low-cost route to architecture screening. Existing zero-cost proxies, however, use different computational forms and their common structure remains insufficiently understood. We provide a general perspective grounded in the initialization state of a network. Although existing proxies use different information, they can be understood through the local effect of the network on output changes. We show that conventional global scores lose differences between layers when aggregating layer information, while other readouts retain only local responses or intermediate features. We therefore construct architecture scores that preserve layer-wise information and aggregate it after layer-wise readout. On a strictly parameter-matched candidate subset, the proposed method provides strong architecture screening performance across several evaluations. Experiments further show that both layer-wise information and its aggregation rule materially affect zero-cost proxy rankings.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Neural Architecture Search, Zero-Cost Proxies, Training-Free Architecture Evaluation, Layer-Wise Representations, Jacobian Response, Architecture Ranking

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_jcVXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the information loss caused by global aggregation in zero-cost neural architecture search. It provides a general perspective on existing zero-cost proxies and investigates the importance of preserving layer-wise information during architecture evaluation. Based on this analysis, the authors propose a layer-aware aggregation method that uses logarithmic aggregation to construct architecture scores. Experiments on a parameter-matched subset of NAS-Bench-101 demonstrate improvements in architecture ranking correlation and Top-K screening.

Critical strengths
  1. The motivation of this paper is relatively clear. Investigating the information loss caused by global aggregation is an interesting problem in zero-cost NAS.

  2. The paper provides theoretical analysis of layer-wise aggregation and evaluates different aggregation functions, which helps explain the motivation behind the proposed score.

Critical weaknesses
  1. The experimental validation does not sufficiently support one of the main motivations of this paper. The authors repeatedly argue that existing zero-cost proxies perform inconsistently across different tasks and search spaces. However, all experiments are conducted on only one parameter-matched subset of NAS-Bench-101 using CIFAR-10. It is unclear whether the proposed method can address this limitation across different tasks, datasets, or search spaces. I think more comprehensive experiments are necessary to support the generality of the proposed approach.

  2. The experimental settings are not sufficiently described. Although Table 1 lists many compared proxies, the paper does not clearly introduce these baselines or provide enough details about their implementation and evaluation settings. Moreover, the computational cost of the proposed method is not compared with existing zero-cost proxies. These details are important for evaluating the effectiveness and efficiency of the proposed method.

  3. The readability of this paper needs to be improved. Several symbols and mathematical expressions are not clearly explained, particularly in the Preliminaries and Section 4. For example, the construction and role of some intermediate representations and information extraction operators are difficult to understand. The authors should provide clearer definitions and explanations to make the proposed framework easier to follow.

  4. The Related Work section is not well organized. Many existing studies are directly listed together without sufficient classification or discussion of their limitations. The authors should systematically review different types of zero-cost proxies and clearly explain the differences between these methods and the proposed approach, especially existing methods that already consider layer-wise information.

  5. THe practical advantages of the proposed method are not sufficiently demonstrated. Although the proposed score achieves better ranking correlations, the improvements in Top-K screening are relatively small compared with some baselines. For example, Overlap@50 only increases from 0.140 for SNIP to 0.160 for the proposed method. It would be more convincing to evaluate whether these improvements can lead to better architectures in a complete NAS process.

Questions for authors

See weakness.1

Reviewer_sX7oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether aggregating initialization responses into a global representation discards information useful for zero-cost neural architecture search. It measures each convolutional or linear weight block through its full output-Jacobian energy, (El=∣Jl∣F2)(E_l=|J_l|_F^2), and examines power and logarithmic aggregation of these layer-wise responses. The theoretical analysis shows that the global kernel (JJ⊤)(JJ^\top) does not generally determine the allocation of response across layers, characterizes the weighting induced by different aggregation exponents, and decomposes the logarithmic score into overall response magnitude and a term describing the distribution of response across layers.

The empirical study considers 3,000 NAS-Bench-101 architectures within a fixed parameter-budget cohort, using reduced-width networks for proxy computation and benchmark validation accuracy as the target. The logarithmic score achieves a Spearman correlation of −0.7380, a Kendall correlation of −0.5420, and Overlap@50 and Overlap@100 of 0.160 and 0.270, respectively. The central empirical finding is that changing the aggregation rule while holding the underlying layer responses fixed can substantially affect architecture ranking.

Critical strengths

S1. The controlled aggregation comparison isolates a meaningful effect.

The most convincing evidence is the comparison of aggregation rules applied to the same layer-wise responses on the same architecture cohort. The absolute Spearman correlation increases from 0.6558 for summation to 0.7380 for logarithmic aggregation. All 3,000 architectures have 47 active weight blocks, so variation in the number of aggregated blocks does not explain this improvement. This experiment provides useful evidence that aggregation itself matters under the studied conditions.

S2. The theoretical analysis provides an interpretable account of the representation and scoring rule.

The non-identifiability result clarifies that a global kernel does not generally retain enough information to recover how response is allocated across parameter layers. The power-aggregation analysis and the decomposition of the logarithmic score further explain how different readouts depend on response magnitude and distribution. Together, these results provide a useful framework for interpreting the controlled empirical comparison.

S3. The supplementary material supports verification of the central numerical finding.

The supplied implementation computes the proposed full output-Jacobian energies, and the saved results allow the headline rank correlations and Top-K overlaps to be recomputed. The architecture identifiers and merged results are also consistent across the constituent cohorts. This makes the central aggregation result more assessable than it would be from the manuscript alone. My verification consisted of static code inspection and recomputation from saved outputs; it was not an independent end-to-end rerun.

Critical weaknesses

W1. The distinct scientific contribution beyond existing layer-wise proxies is not yet sufficiently established.

ZiCo [1] already uses logarithmic aggregation of layer-wise gradient statistics. InTrain [2] likewise includes a sum of logarithms of layer-wise participation ratios in its geometric-capacity component. Consequently, the general observation that nonlinear aggregation of layer-level quantities can be useful is already represented in prior work.

The present paper uses a different quantity—full output-Jacobian energy—and combines it with a non-identifiability analysis and a controlled aggregation study. These differences could constitute a meaningful contribution. However, the manuscript does not yet make sufficiently clear what this combination establishes beyond existing layer-wise proxies. In particular, the broad argument for preserving layer information before aggregation needs to be connected more precisely to the new understanding provided by the proposed representation and analysis. This matters to my assessment of the paper’s scientific significance, even though I find the controlled empirical result useful.

W2. The selection history of the aggregation rule and ranking direction is unclear, which limits the interpretation of the screening results.

For the Top-K evaluation, architectures are ranked in ascending order of the logarithmic score, consistent with its observed negative correlation with accuracy. It is unclear when logarithmic aggregation and the lower-is-better direction were selected relative to the evaluation of the initial 1,000 architectures and the additional 2,000 architectures.

This distinction affects the strength of the conclusions. A retrospective comparison can establish a useful association within the analyzed cohort, while an evaluation of a scoring rule fixed before the reported comparisons provides stronger evidence for its use in architecture screening. The uncertainty does not invalidate the reported correlations or the controlled aggregation comparison, and it is not evidence that the authors selected the rule post hoc. It does, however, leave the scope of the screening claim insufficiently specified. A brief account of the existing evaluation history could resolve much of this concern.

W3. The implementation provenance of some baseline results requires clarification.

Two supplied baseline runners differ materially from their corresponding reference implementations:

  • In the supplied ZiCo runner, zico_score clears gradients once before the minibatch loop, so subsequent backward passes accumulate gradients. The vendored reference implementation clears gradients within the loop and therefore measures a different sequence of gradient observations.

  • The supplied MeCo runner computes correlations across flattened samples in a batch, whereas the vendored reference implementation computes correlations across channels within an individual sample. This changes the statistic being evaluated.

It is unclear whether these supplied runners generated the reported baseline scores, whether another execution version was used, or whether these changes were intentional variants. These differences make the affected comparisons difficult to interpret without further clarification.

This concern is limited to those baseline comparisons. It does not invalidate the comparison of aggregation rules applied to the same layer energies, and it does not by itself explain the reported advantage over GradNorm or SNIP.

Questions for authors

Q1. What is the principal new insight relative to existing layer-wise proxies?

Relative to ZiCo and InTrain, what scientific conclusion is enabled specifically by using full output-Jacobian energies and by the non-identifiability analysis? Please explain how the controlled aggregation experiment supports that conclusion, beyond demonstrating another effective choice of layer statistic and readout.

A concise conceptual comparison would be sufficient: what information does the proposed representation retain, what does the analysis establish about that information, and what new understanding does the empirical result provide? This would help me reassess whether the contribution is sufficiently distinct and significant.

Q2. When and how were logarithmic aggregation and the lower-is-better ranking direction selected?

Were these choices fixed before analyzing the reported cohorts, selected using the initial 1,000 architectures and then applied unchanged to the additional 2,000, or selected with knowledge of results from both cohorts? Please clarify the role of each cohort and which existing comparisons informed these choices.

This would help distinguish an exploratory analysis of aggregation behavior from an evaluation of a previously specified screening rule. It would also allow the claims to be assessed at the appropriate scope. A description of the actual sequence of decisions is sufficient; an additional data split or experiment is not required.

Q3. Which implementations generated the reported ZiCo and MeCo results?

Please clarify whether the reported scores were produced by the supplied runners containing the gradient-accumulation and correlation-axis differences described above. If another version generated the results, please identify the relevant implementation difference or provide an existing code excerpt. If these modifications were intentional, please explain the intended variants and how they should be distinguished from the reference methods.

This clarification would determine how much weight I should place on the affected baseline comparisons. I am asking for implementation provenance and interpretation, rather than a rerun during rebuttal.

Reviewer_6uA7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper shows that the global scores lose differences between layers when aggregating layer information, while other readouts retain only local responses or intermediate features. Thus, they propose a new score that preserves layer information and aggregates it after layer-wise readout. Experiments show that this information affects the zero-cost ranking.

Critical strengths
  • Study layer-wise scoring is reasonable for better zero-cost NAS.
  • The experiment shows promising results.
Critical weaknesses
  • It is unclear from the abstract that the authors state, "their common structure remains insufficiently understood" and then "they can be understood through the local effect of the network on output changes". What is the point here?
  • Figure 1 is not well drawn. It takes a lot of effort to understand this figure.
  • The writing needs improvement. A lot of terms are used without proper definition. For example, in the abstract: aggregate it after layer-wise readout... What is readout, layer-wise readout? Different between "aggregate-before-readout" and "readout-before-aggregate"? In line 74, what is Global information? Why and how can global information compress away this distinction?
  • The presentation is poor. There is a lot of unused space, especially on page 7.
  • The experiment setup is unclear. The parameters are fixed at 5.7M and 3,000 nets without giving a reason.
Questions for authors

Besides the question in the Weaknesses section, I have additional questions.

  • Why do you select 5.7M parameters? Any reason? How are the networks selected? (Random selection?) If possible, provide the results with other numbers of parameters and the total number of networks will improve the experiment section.
  • Why is only NAS-Bench 101 tested? How about NAS-Bench-201, NAS-Bench-301, and NAS-Bench-Suite?
  • What is the benefit of this work? In NAS/zero-cost NAS, we search for a high-performing network given a search space and a task. Can we achieve better performance (test accuracy) using the proposed metric?
Reviewer_NCC6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how layer-wise information and aggregation choices affect zero-cost neural architecture search. It organizes existing proxies through a framework of initialization information extraction, representation, and statistical readout. The authors show that the global Jacobian Gram matrix does not uniquely determine the distribution of response magnitudes across layers. They define layer-wise responses as squared Frobenius norms of parameter-output Jacobian blocks and study power and logarithmic aggregation, including a decomposition of the logarithmic score into total-response and layer-imbalance terms. Experiments evaluate 3,000 strictly parameter-matched architectures from NAS-Bench-101 on CIFAR-10. Within this setting, the proposed logarithmic score achieves a Spearman correlation of −0.7380 and the highest reported Top-K overlap among the evaluated methods. The results suggest that aggregation choices can materially affect architecture rankings.

Critical strengths

The aggregation ablation is well controlled. Table 2 fixes the layer responses and varies the aggregation function, showing that the readout itself affects architecture rankings. The reported improvement in absolute Spearman correlation from 0.6558 to 0.7380 is worth investigating.

Critical weaknesses
  1. The contribution is insufficiently distinguished from existing layer-wise proxies. The motivation suggests that existing methods primarily characterize the network globally and pay limited attention to layer-wise information. However, ZiCo already applies a logarithm to layer-level gradient statistics before summing across layers, while NEAR performs layer-wise effective-rank readouts before aggregation. The proposed Jacobian energy differs from these statistics, but the paper does not sufficiently establish what new insight or capability follows from this difference. Acknowledging prior layer-wise aggregation in Section 2 does not fully resolve the broad framing in the introduction.
  2. The theoretical results characterize the aggregation function but provide limited explanation of its predictive benefit. Proposition 2 correctly establishes a monotonic change in the ratio of two layer contributions. It does not establish that increasing the relative contribution of weak-response layers improves architecture ranking, or that logarithmic aggregation should perform best. More importantly, Equation (11) gives Slog⁡=Llog⁡(ME/L)−LDKL(quni∥q).S_{\log}=L\log(M_E/L)-L D_{\mathrm{KL}}(q^{\mathrm{uni}}\|q). Since the experiments select lower scores, this rule favors greater imbalance at fixed total response and layer count. Meanwhile, Table 3 reports positive correlations between accuracy and statistics indicating more balanced responses. These marginal correlations are not mathematically contradictory, but the paper does not provide the conditional analysis needed to reconcile them or substantiate the claimed independent predictive role of layer allocation.
  3. The empirical evidence is too limited to establish a robust advantage. All evaluations use the same 3,000-architecture NAS-Bench-101/CIFAR-10 subset, with one initialization seed and a fixed probe. Averaging three benchmark training runs reduces noise in the target accuracy but does not establish stability of the initialization-based proxy. Some screening improvements are small: Overlap@50 corresponds to eight hits versus seven for SNIP, and Overlap@100 corresponds to 27 hits versus 26 for the α=1/4\alpha=1/4 variant. Without repeated proxy measurements or uncertainty estimates, the reliability of these differences remains unclear. The selection of the parameter group and the calibration of aggregation choices and ranking directions also require clarification.
  4. Important implementation and comparison details remain unclear. Table 1 labels the NEAR baseline as “ER-pre-mean,” whereas the original method includes both pre- and post-activation statistics. Its correspondence to the complete baseline is therefore uncertain. The PDF also does not adequately specify whether ElE_l is computed exactly or estimated using output-direction probes, nor the associated computational cost. Although supplementary code is mentioned, it was not available for this assessment, so these concerns concern the documented protocol rather than demonstrated implementation errors.
Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.