acceptodds
Withdrawn from ICLR 2027

Bayesian Optimization in Sequence-to-Architecture Latent Space for Zero-Shot NAS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-shot Neural Architecture Search removes the prohibitive cost of traditional NAS, but its search process is typically based on the evolutionary algorithm (EA); lacking an explicit model of the objective, it often resorts to a near-random search through mutation. Bayesian Optimization offers a principled alternative by modeling the objective and aggregating information across iterations, but scales poorly to the high-dimensional, discrete, graph-structured spaces of modern NAS, restricting its use to only small networks. In this paper, we bring Bayesian Optimization to zero-shot NAS for large-scale architectures by learning a latent space via a Variational Autoencoder trained to reconstruct a novel prefix encoding of architectures and propose a proxy scalarization that combines several zero-shot proxies into a single Bayesian Optimization objective. After only 10,000 iterations of the proposed search algorithm (8 hours on a single GPU), our method found a network architecture which under the given model parameter count constraints achieves state-of-the-art results on three separate tasks – image classification, object detection and semantic segmentation. The search algorithm code, the latent space model as well as the discovered architecture is available at https://github.com/HiddenForReview.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
neural architecture search, network encoding, training-free, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eLRGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper applies Bayesian optimization to zero-shot NAS using a VAE-learned architecture space. It represents architecture blocks as prefix sequences and combines zero-shot proxies into a single search score. The selected model reaches 81.51% top-1 accuracy on ImageNet-1K and is also evaluated on COCO and ADE20K.

Critical strengths

1、The problem is worth studying, and the proposed representation offers a practical way to search a complex architecture space with Bayesian optimization. 2、The paper reports results on ImageNet-1K and two downstream datasets rather than relying only on small NAS benchmarks.

Critical weaknesses

1、The paper mainly brings together existing ideas. It is not clear what is genuinely new compared with earlier work on latent-space NAS and Bayesian optimization. 2、The results do not show that Bayesian optimization is the reason for the improvement. There is no fair comparison with random or evolutionary search under the same budget, and the 0.36-point gain over UniNAS-A is reported without repeated runs. 3、The presentation is weak. Figure 1 is the only method figure and explains little beyond the general workflow. Table 3 extends beyond the normal text width, and equation references are inconsistent, using “Eq.”, “equation”, and “Equation”. 4、Key review materials are missing. The manuscript cites Appendix Table 5, Section A.4, Equation (12), and Appendix Figure 2, but none appears in the supplied file. The GitHub link also does not provide accessible code at review time.

Questions for authors

1、State what is new compared with existing latent-space NAS and Bayesian optimization methods, and explain why the difference matters. 2、Compare Bayesian optimization with random and evolutionary search under the same budget. Report results from repeated runs to support the claimed improvement. 3、Improve Figure 1 so that it shows the actual method, fix the width of Table 3, and use a consistent style for equation references. 4、Provide the missing appendix content, including Table 5, Section A.4, Equation (12), and Figure 2. Supply a working anonymous code repository that reviewers can access.

Reviewer_Bi3MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets zero-shot neural architecture search. It argues that evolutionary search is close to random mutation and that Bayesian optimization offers a model-based alternative but scales poorly to large discrete graph spaces. The proposed method learns a 32-dimensional latent space with a variational autoencoder over serialized UniNAS blocks and runs Gaussian-process Bayesian optimization in that space. A multi-dimensional training-free proxy is transformed into a scalar objective by summing whitened proxy components, with a soft penalty and a hard mask to enforce a parameter budget. Experiments on the ImageNet and COCO dataset show slightly better performance than UniNAS.

Critical strengths
  1. The argument for scalarization that allows pointwise evaluations is reasonable. Rank aggregation is undefined for unseen candidates, and a continuous objective is required for any surrogate-based search.

  2. The ablation study covers the three claimed components and reports the failure mode where a graph-based VAE yields an architecture that diverges during training, rather than hiding it.

  3. The evaluations are extended to detection, instance segmentation, and semantic segmentation under a fixed protocol for a meaningful check that the block is not overfitted to classification.

Critical weaknesses
  1. All reported numbers come from a single search run and a single final training run. The margins supporting the paper are only 0.36 points on ImageNet, 0.5 box AP on COCO, and 0.5 mIoU on ADE20K. No seed variance, repeated searches, or confidence intervals are reported.

  2. BO with learned architecture encodings has already been explored with VGAEs and GCN encoders (Sun & Shen, 2025; Hemmi et al., 2024, 2025; Shi et al., 2020 as cited by the authors). The paper’s distinction is to use a prefix-string VAE over UniNAS blocks and to scale to ImageNet-level architectures. This is a useful engineering extension, but prefix/grammar serialization, autoregressive decoding, and grammar-constrained generation are standard techniques. The paper does not show that prefix serialization is conceptually new or fundamentally better than graph-based encodings. The VGAE comparison in Table 4 is confounding, and a weak VGAE baseline cannot demonstrate the superiority of the representation itself.

  3. The ablation study supporting the central claim changes two factors at once. Removing the latent space replaces Bayesian optimization in a 32-dimensional continuous space with evolutionary search over graphs, so the 1.5-point drop cannot be attributed to the search algorithm rather than the encoding. A budget-matched random or evolutionary search inside the same latent space, and Bayesian optimization over prefix strings without the autoencoder, could separate them. However, neither of these results are reported.

  4. Table 2 compares across incompatible protocols. The proposed method is credited with 81.51 percent at 0.33 GPU days against GP-NAS at 73.4 and BONAS at 75.4, but those numbers come from different spaces, macro structures, and training recipes, and the method's own baseline is trained with the same strong recipe as the discovered model. It is not reliable to claim the superiority in the NAS comparisons without a shared protocol.

Questions for authors
  1. What does budget-matched random search in the same latent space achieve, and what does Bayesian optimization over prefix strings without the autoencoder achieve, so that search algorithm and encoding can be separated?

  2. What are the Kendall and Spearman correlations between the scalarized proxy and true accuracy on trained architectures, and which proxies enter p with what whitened weights?

Reviewer_LnDfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a zero-shot NAS framework that performs Bayesian Optimization in a learned architecture latent space. The authors use a VAE with prefix-based architecture encoding to construct a continuous latent space, and introduce a proxy scalarization method to combine multiple zero-shot proxies into a single BO objective. Experiments show that the discovered architecture achieves strong performance on ImageNet-1K, object detection/instance segmentation, and semantic segmentation under a similar parameter budget.

Critical strengths
  1. The paper applies Bayesian Optimization to large and heterogeneous architecture search spaces by learning a low-dimensional architecture representation. The prefix representation provides a compact encoding of architecture blocks and naturally supports sequential decoding with grammar constraints.
  2. The proposed proxy scalarization provides a simple way to combine multiple zero-shot proxies into a pointwise objective that can be directly used by BO.
  3. The experimental results are strong. The discovered architecture reaches 81.51% ImageNet-1K top-1 accuracy and also performs well on the reported downstream detection and segmentation tasks.
  4. The ablation results support the importance of the latent space, prefix encoding, and proxy scalarization.
Critical weaknesses
  1. The main components, including VAE-based architecture representations, Bayesian Optimization, and zero-shot proxies, are individually well established. The novelty mainly comes from combining these components with the proposed prefix representation and proxy scalarization.
  2. It is unclear how much of the improvement actually comes from Bayesian Optimization. The current ablation changes both the search algorithm and the architecture representation when comparing BO with evolutionary search, making it difficult to isolate the contribution of BO itself. A controlled comparison between BO and other search strategies in the same learned latent space would be more convincing.
  3. The effectiveness of BO strongly depends on whether the learned latent space provides a meaningful and sufficiently smooth landscape with respect to architecture quality. While the paper evaluates reconstruction and search performance, a more direct analysis of the relationship between latent-space distance and architecture performance would better justify performing BO in this space.
  4. The search objective is based on a scalarization of zero-shot proxies rather than the actual trained performance of candidate architectures. It would be useful to analyze how well this scalarized proxy correlates with the final performance, especially for architectures proposed by BO during later stages of the search.
  5. The method mainly provides a better search framework, but the discovered architectures do not lead to clear architectural insights or design principles. An analysis of what structural patterns BO consistently prefers could make the results more useful beyond finding a single high-performing architecture.
  6. The experiments are conducted within the UniNAS search space, so it remains unclear how well the learned representation and optimization framework generalize to substantially different architecture spaces.
Questions for authors

Listed in Weakness part

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.