acceptodds
Withdrawn from ICLR 2027

Neural Architecture Distributions: A New Paradigm for Stochastic Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Stochastic segmentation seeks to represent multiple plausible masks for a single image, a requirement in safety- and quality-critical applications such as medical imaging and building defect inspection. Most existing methods introduce stochasticity by injecting continuous latent variables or by following iterative denoising trajectories, making their stochastic sources difficult to search or audit. We study architecture distributions as a stochastic source for segmentation: instead of sampling a latent variable or noise, we sample a discrete architecture from a learned distribution over operator choices at multiple searchable positions in a segmentation backbone. Each sampled architecture yields one mask through the selected active path, so inference depends on the executed subnet rather than the complete candidate bank. This approach supports architectural provenance, since each output corresponds to a specific architecture configuration. To reduce collapse toward averaged masks, we train with set-level supervision by matching architecture-sampled predictions to the annotation set using an IoU-based energy-distance surrogate. We construct the candidate bank with evolutionary search, making the support of the stochastic source optimizable before distribution learning. NADS improves distribution matching and hypothesis coverage on LIDC-IDRI, and generalizes to additional stochastic benchmarks (Hippocampus, QUBIQ Prostate) as well as deterministic segmentation tasks (Cityscapes, Crack500). The resulting formulation learns an input-conditioned architecture distribution and produces multi-annotator output diversity through architecture sampling with set-level supervision.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Stochastic Segmentation, computer vision, Neural Architecture Distribution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_R9omOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NADS, a stochastic segmentation framework that generates diverse predictions through input-conditioned architecture sampling. It combines evolutionary search, shared-weight subnetworks, and a GED-based surrogate loss. Experiments on multiple benchmarks demonstrate promising performance.

Critical strengths

1.The paper explores an interesting alternative to conventional latent-variable-based stochastic segmentation by introducing input-conditioned architecture sampling. 2.The combination of architecture sampling and set-level supervision is well-motivated, aiming to balance prediction diversity and segmentation quality.

Critical weaknesses

1.The theoretical properties of the modified GED loss are insufficiently justified, particularly regarding the soft-IoU distance and its role in encouraging prediction diversity. 2.The method incurs substantial computational costs, with 387M parameters and approximately 35 hours of search and training, limiting its practical scalability. 3.The comparisons with recent methods are insufficient, particularly in Table 2, which lacks representative and state-of-the-art stochastic segmentation methods. Therefore, it remains unclear whether the proposed method consistently outperforms the latest approaches.

Questions for authors
  1. How does NADS fundamentally differ from conditional MoE methods, and what specific advantages does architecture-level sampling offer over conventional conditional routing?
  2. How is GED computed during evolutionary search when the architecture is fixed to a specific genotype? 3.Given the limitations of the soft-IoU distance, how does the proposed GED surrogate ensure effective distribution matching and prevent mode collapse?
Reviewer_qpsyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper focuses on stochastic medical (and beyond) image segmentation. In contrast to approaches building stochasticity through latent variable models or diffusion processes, they introduce stochasticity through input-conditioned discrete sampling of neural architectures. Specifically, they introduce stochasticity by allowing for several operator choices at pre-specified positions in the model. The operators are chosen to all keep the output tensor shape unchanged at any given position. The paper uses a first warmup and a stage of evolutionary search to construct a candidate bank of operators. In the actual training phase, Gumbel-softmax relaxation is used to make the training differentiable with respect to operator sampling.

One interesting message of the paper that can generate discussion is their finding that performance degrades unless training with set-level supervision using an IoU-based Generalized Energy Distance.

Critical strengths

The main strengths of the paper are compelling results and clarity.

Critical weaknesses

There are a few potential weaknesses that justify an overall mixed rating.

The first weakness is that by nature, this work would be fairly difficult to reproduce despite a reasonable attempt by the authors to accurately describe their approach. Indeed the training proceeds in several stages, including a warmup, an evolutionary search, and a final training, that are by necessity described at a high-level; plus the candidate pool creation and search most likely leaves technical elements unspecified despite the appendix.

Therefore in this specific case, I will point out the lack of anonymous github / shared public codebase / lack of declaration of intent to share code, as a strong weakness of the paper. I would be willing to reconsider this point if a link to an anonymous github is provided during the rebuttal phase.


A second concern for which I am curious of author feedback, is the potential for metric "hacking". The approach is somewhat "brute-force", whereby it results in a huge candidate ensemble of diverse architectures specifically trained to minimize an IoU-based GED. It is then evaluated in terms of GED and HM-IoU on in-distribution data, which it was specifically trained to optimize. Of course I would agree that these are standard metrics for stochastic segmentation, but in this instance isn't the good performance reported simply a consequence of metric "hacking"?

In particular, while the ablation study of Table 3 is interesting, it is concerning that the reported GED degrades below baselines specifically when ablating GED from the training loss. I understand the author's argument that using set-level supervision is essential for stochastic segmentation, but this precise point is open to debate from the existing literature, which reports decent (although inferior) results from single annotation training.


Finally:

  • the related work on neural architecture search stops at 2021 if I am not mistaken, is there no relevant work after then?

  • Although the paper is aimed at stochastic segmentation, an nnU-Net baseline, or a standard ensemble of nnU-Nets (even maybe reporting just HM-IoU) would be very welcome for an ICLR segmentation paper. Similarly, an ensemble of various architectures (possibly trained from annotator-specific annotations) would be a reasonable naive baseline given the topic of the paper (even if in the worst case it can only produce limited candidate segmentations).

  • Many figures are missing in the results table. If a GED32 and a GED100 can be computed, why is GED50 left empty?

  • Finally, I am concerned about the practicality of the approach. As a user, I would be concerned by (1) the relative complexity of the multi-stage training; (2) most likely much longer training times; and (3) the necessity to collect multi-annotator ground-truth segmentations for my use-case. Even collecting a sizeable single-annotator ground-truth dataset is a significant challenge in medical image segmentation.

Questions for authors

See the "weaknesses" box above.

Reviewer_7pezOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes NADS, a stochastic segmentation framework that models prediction diversity through input-conditioned neural architecture distributions. It combines weight-sharing, evolutionary search, and GED-based set supervision. Experiments on multiple benchmarks show promising results.

Critical strengths
  1. Interesting idea of modeling segmentation uncertainty through architecture sampling.
  2. Combines architecture search and set-level supervision in a coherent framework.
  3. Extensive experiments and ablation studies with competitive results.
Critical weaknesses
  • The theoretical justification for the proposed GED surrogate is somewhat insufficient. It is not fully clear whether architectural diversity can genuinely capture annotator disagreement.
  • The risk of validation-set overfitting introduced during the evolutionary search procedure requires further discussion and clarification.
  • The substantial size of the supernetwork together with the non-trivial overhead of offline search largely undermines the authors’ efficiency claims.
Questions for authors
  1. What theoretical properties does the proposed GED surrogate retain?
  2. How do the authors verify that architectural diversity represents meaningful annotation uncertainty?
  3. How is validation overfitting prevented during evolutionary search?
Reviewer_5unCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a system for learning the most likely discrete architecture from a architecture distribution to do stochastic segmentation. This amounts optimizing for a categorical architecture distribution over candidate operators. The most likely architecture for the task is optimized via evolutionary construction of the candidate bank. This appears to be effectively an elegant statistical approach of building a backbone and to do NAS for stochastic segmentation.

Critical strengths
  • In a way, the authors successfully combine the strengths of NAS with stochastic segmentation.
  • The work address an important problem in medical image segmentation
  • The work is well presented and easy to read, it provides state-of-the-art results for stochastic segmentation and has suitable experiments and ablations.
  • The survey of the literature is good and also covers NAS and its differences between it and stochastic segmentation.
Critical weaknesses
  • The work however has no mention of training times in the main body (briefly mentioned in appendix in Table 10). Although Inference time is comparable to other stochastic methods, it is not clear that the evolutionary nature of the training could be a significant disadvantage in practice.
  • All compared baselines are fixed, single-model architectures with limited capacities (30M–45M parameters). In contrast, the proposed method maintains 387M total parameters. Even with a inference footprint of 26M, the model benefits from a vastly larger shared parameter pool during optimization. Without comparing against search-based methods or capacity matching, it is unclear if the performance gains are from the architecture distribution or from a capacity advantage.
  • This also points to a lack of limitations and future directions that need to be addressed going forward. The limitations are very important to consider and should be in the main body too.
  • The schematic in Figure 1 needs improving as the most important parts are missing the novelty of the approach, the form of the architecture distribution (a Gaussian isn’t very helpful), all the architectures look the same despite having different a_i, and the evolutionary optimization approach.
  • There are no qualitative results given in the main text. It would be great to see Figure 2 from Appendix D1. In the main body. That is the fact that Figure 2 is in the fourth appendix doesn’t seem right. Figure 4 would be nice in the main body too if it fits in the extra page that is allowed after review.
  • It would be good to cite some of the fabric networks (e.g. from NIPS) as part of the NAS section.

Minor:

  • Second last sentence in abstract, are these names datasets, leaderboards or challenges, ICLR readers may not be familiar with all of them so explicitly sating it would be helpful
  • The tables need to be fit into the paper better, see oversize Table 1.
Questions for authors
  • While the small annotator pool cross datasets is a known benchmark constraint, its a challenge for discrete architecture distributions compared to continuous latent spaces (e.g., cVAEs) that support smooth interpolation. Sampling discrete operator subgraphs across L layers to generate K≥M hypotheses risks partitioning the combinatorial search space into arbitrary sub-networks that overfit. How do the authors verify that the surplus discrete configurations represent valid anatomical variations rather than artifacts?
  • Table 10 shows that the proposed method is significantly slower training time and memory usage, could the authors comment on where this is largely due from, I’m guessing the evolutionary algorithm? Are there alternative optimization strategies that might help in this regard?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.