acceptodds
Under review as a conference paper at ICLR 2027

Search2Adapt: Training-Free Adapter Architecture Search for Segmentation Foundation Models

Abstract

Recent foundation models show remarkable success across general-domain vision tasks but often struggle in specialized domains with distinct visual characteristics. Existing adaptation methods typically rely on fixed adapter architectures that cannot account for domain-specific characteristics. We propose Search2Adapt, a framework that automatically discovers effective adapter architectures for segmentation foundation models through training-free search. Search2Adapt parameterizes each adapter by its operation type, configuration, and layer-wise insertion, and perturbs these variables to construct candidate adapters. Each candidate is scored by a zero-shot proxy based on gradient statistics with a sparsity penalty on layer-wise insertion, and the perturbation variables are iteratively updated toward high-scoring candidates through score-weighted aggregation. This enables domain-specific adapter search without training individual candidates. To evaluate Search2Adapt across specialized segmentation tasks, we conduct extensive experiments on camouflage, polyp, and nuclei segmentation datasets. Search2Adapt achieves performance competitive with or superior to fixed adapter designs across diverse domains, while sensitivity analysis confirms that adapter configuration substantially affects segmentation performance. These results demonstrate effective domain-specific adapter architecture discovery for segmentation foundation models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Neural Architecture Search, Parameter-efficient Fine-tuning, Foundation Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fYiuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Search2Adapt, a training-free search framework for adapter architectures in segmentation foundation models. It searches over adapter operations, operation-specific configurations, and layer-wise insertion strengths. Candidate architectures are evaluated using a zero-cost gradient proxy, ZiCo, together with a sparsity penalty, and the search variables are updated through score-weighted perturbations. The selected architecture is then trained normally. Experiments on camouflage, polyp, and nuclei segmentation show that the searched adapters are generally competitive with fixed adapter designs.

Critical strengths
  • The problem is relevant: a fixed adapter architecture may not be equally suitable for different domains, and automatically searching adapter structures is a reasonable direction.

  • The search space is more flexible than simply choosing between a few fixed adapter types. It jointly considers adapter operations, their configurations, and layer-wise insertion patterns. The method is evaluated across multiple datasets, foundation models, and zero-cost proxies.

Critical weaknesses
  • The main assumption behind the method is not sufficiently validated. The framework relies on ZiCo scores to rank untrained adapter candidates, but the paper does not clearly establish the correlation between the proxy score and the final trained performance. It also does not provide a direct downstream comparison with random search or evolutionary search under the same candidate-evaluation budget. Therefore, it is unclear whether the reported gains come from the proposed perturbation search, the proxy, or simply selecting a good candidate from a larger search space.

  • Some of the main claims are overstated. The search still uses labeled data and forward/backward computation, and the selected adapter is trained normally, so “training-free” only applies to candidate evaluation rather than the full adaptation process. In addition, continuous layer-wise coefficients do not necessarily remove adapter parameters or computation, so the claimed sparsity and efficiency benefits are not fully demonstrated. The paper also does not isolate the contributions of operation search, configuration search, layer search, and the sparsity penalty.

Questions for authors
  • Can the authors report the relationship between zero-cost proxy scores and the final performance of the corresponding trained candidates? This would clarify whether the proxy provides a reliable ranking signal.

  • Under the same search space, proxy, and number of candidate evaluations, how does perturbation-based search compare with random search and evolutionary search?

  • How much computation is required for the complete pipeline, including candidate scoring, search, and final training, compared with training a fixed adapter? Do the continuous layer coefficients actually reduce parameter count, FLOPs, or memory?

  • Can the authors provide ablations that separately evaluate operation search, configuration search, layer-wise search, and the sparsity penalty?

These are central issues for assessing whether Search2Adapt is a reliable architecture-search method rather than a heuristic that happens to find competitive adapters on the evaluated tasks. If the authors address these concerns with convincing evidence, I would reconsider my current rating and potentially raise it to 3.

2 comments
  1. Thank you for your valuable comment. We examined the correlation between ZiCo scores and final fine-tuned performance on the TNBC dataset. Six different adapter configurations were each trained with three random seeds, and validation mDice was averaged across the seeds. The results show a Spearman rank correlation of 0.49 and a Pearson correlation of 0.76. We report these results and the correlation trajectories over training epochs in Figure 4 of Section 4.3.1. Together, the trajectories and final results suggest that ZiCo scores are associated with adapter trainability and provide a useful, though imperfect, ranking signal. These findings also highlight the need to develop a downstream-aware proxy. Our future work will therefore explore a new zero-cost proxy that better reflects downstream performance.
  2. We compared the proposed PZAS framework with random search and regularized evolutionary search (RE) in Table 5 of the supplementary material. All three methods use the same search space and number of candidate evaluation, enabling a controlled comparison of the search strategies. PZAS achieves competitive downstream performance compared with random search while yielding more parameter-efficient adapter architectures. This difference is consistent with the inclusion of a sparsity term in PZAS, which is absent from the random search. Compared with RE, PZAS improves downstream performance while reducing the number of adapter parameters by 40.5%. These results demonstrate that PZAS achieves competitive or improved downstream performance compared with existing search methods under the same budget. It also yields more compact adapter architectures.
  3. The proposed architecture contains 0.4879M encoder adapter parameters, compared with 0.0633M for SAM-Adapter and 0.1407M for sam2-adapter. In terms of the total number of trainable parameters, SAM-Adapter, SAM2-Adapter, and the proposed method contains 4.122M, 3.944M, and 4.547M parameters, respectively. This difference arises from the shared projections used in these existing methods across the 12 Transformer layers. In contrast, our method uses separate weights for each layer-specific adapter architecture, which can increase the parameter count. Nevertheless, as shown in Table 5 of the supplementary material, the sparsity term in our scoring metric helps identify more compact architectures than search methods that do not account for computational cost.

Regarding the continuous layer coefficients, Table 9 of the supplementary material compares continuous and hard insertion on two datasets. On Kvasir-SEG, hard insertion places adapters in only two of the 12 Transformer layers, whereas continuous insertion uses all 12 layers. Despite this, the architecture obtained with continuous insertion requires only 0.295M adapter parameters and 2.416 GFLOPs per image, compared with 2.370M parameters and 3.222 GFLOPs for hard insertion. These results show that continuous insertion can yield a more compact architecture with lower computational cost. This difference reflects the combinations of the layer-wise insertion pattern and the selected adapter operations.

  1. In our hierarchical search space, configuration choices are conditioned on the selected operation, and layer-wise search is performed jointly with these choices. This coupling makes it challenging to isolate their individual contributions through separate ablations. We provide an ablation of the sparsity penalty in Table 6. Removing the penalty slightly improves segmentation performance but increases the adapter parameter count and GFLOPs. Conversely, including the penalty reduces parameters by 25.6% and GFLOPs by 26.9%, with only a 0.002 decrease in S-measure. These results support the role of the sparsity penalty in balancing segmentation performance and adapter computational cost.

The authors have not adequately addressed my concerns. And, too late, I therefore maintain my original rating of Weak Reject.

Reviewer_tJKCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Search2Adapt, a framework for training-free adapter architecture search for segmentation foundation models. The method explores adapter operations, operation-specific configurations, and layer-wise modulation coefficients by generating candidate architectures through Gaussian perturbations. It evaluates these candidates using a zero-cost proxy with a layer-wise sparsity penalty and updates the search variables through score-weighted perturbation aggregation. Candidate architectures are evaluated without parameter updates during search, while the selected architecture is subsequently trained for the downstream task.

Critical strengths
  1. Segmentation domains exhibit different characteristics, so a fixed adapter architecture may not suit all tasks. Exploring adapter architecture as an optimization dimension is a promising direction for efficient domain adaptation.

  2. The paper evaluates six datasets and explores multiple method combinations on applicable tasks. These experiments demonstrate that the framework can accommodate different settings rather than being tailored to a single model or dataset.

  3. The configuration sensitivity analysis varies individual configurations while keeping the adapter operation fixed and uses three training seeds. The main results also report the mean and standard deviation across independent search runs, strengthening the credibility of the experiments.

Critical weaknesses
  1. Stronger empirical evidence is needed, particularly for the main contribution. The main experiments use SAM2 with ZiCo, making SAM2-Adapter a more direct baseline. In Table 1, Search2Adapt underperforms SAM2-Adapter on both metrics for CAMO and COD-10K. The two methods perform similar on MoNuSeg. Therefore, the current results do not establish a consistent advantage of automated search over directly using a fixed adapter architecture.

  2. The method relies on gradient statistics of untrained candidate adapters to estimate their final performance, but the paper does not report the rank correlation between proxy scores and post-training metrics such as Dice and mIoU within its search space.

  3. “Training-free” means that candidate network weights are not updated during search, but evaluating each candidate still requires forward and backward propagation. Although the paper reports GPU hours for selected settings, it does not compare runtime, memory usage, or total adaptation cost against other search methods.

Questions for authors

More implementation and evaluation details are needed. The main text do not fully specify the candidate configurations and their possible values for each operation. They also provide insufficient detail about the data and number of samples used for proxy evaluation, as well as the data splits used for architecture search and model training.

2 comments
  1. Thank you for pointing this out. The proposed framework includes adapter operations designed for specific objectives, such as capturing edge information and morphological characteristics. Each operation has configurations tailored to its purpose. These operation-conditioned configurations include the bottleneck dimensions, activation function type, and other operation-specific parameters. The layer-wise modulation coefficient is incorporated into each candidate Adapter operations, and its possible values vary depending on the backbone model. We provide more detailed descriptions of the search space, including the candidate configurations and their possible values for each operation, in Table 2 of the supplementary material.
  2. Architecture search uses data from the training split, which is also used for subsequent fine-tuning, We do not use a separate split for architecture search, neither validation nor test segmentation performance is used to guide the search. Candidate adapter configurations are evaluated using ZiCo scores computed on samples from the training split. We clarify these data splits and provide the proxy evaluation details, including the number of samples and how they are selected, in Section 4.1.2.

The authors have not adequately addressed my concerns. I also note that other reviewers share my concern about the reliability of ZiCo. Specifically, the response does not directly explain or address the underperformance relative to SAM2-Adapter on CAMO and COD-10K. I therefore maintain my original rating of Weak Reject.

Reviewer_qvFKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Search2Adapt, a training-free adapter architecture search framework for adapting segmentation foundation models to specialized domains. The method searches over predefined adapter operations, their configurations, and layer-wise insertion strengths using a perturbation-based strategy guided by zero-cost proxies, reducing the need to train individual candidate architectures during search. The authors evaluate the approach across six datasets covering camouflaged object detection, polyp segmentation, and nuclei segmentation, using different foundation models and scoring proxies. Experimental results show that Search2Adapt can discover task-specific adapter configurations and achieve competitive performance compared with existing adaptation methods, although it does not consistently outperform all baselines.

Critical strengths
  1. This paper proposed the novel perturbation-based search algorithm and applied it to adapter architecture search. I believe the algorithm has border impact not only to the PEFT community in computer vision.
  2. The paper evaluates the approach across six datasets covering camouflaged object detection, polyp segmentation, and nuclei segmentation, and investigates different foundation models and zero-cost scoring proxies. The additional configuration-sensitivity and search-dynamics analyses provide some evidence that adapter design choices matter and that the proposed search strategy can identify useful task-specific configurations. Although the method does not consistently outperform all baselines, the breadth of the evaluation supports the practical relevance of the approach and provides evidence beyond a single-dataset demonstration.
Critical weaknesses
  1. This paper relays on a hand-crafted, hard-coded set of NN architecture / parameter selection to begin search with. This will limit the long-term benefit of the proposed approach as it can not come up with new adapter architecture that might be beneficial to the given task, as well as might not be able to go out of the current well-known technique / use a combination of existing technique to get better results. The approach will also need continuously to be updated with the latest techniques.
Questions for authors
  1. The proposed Search2Adapt framework uses ZiCo as a zero-cost proxy to guide Adapter architecture search. However, I was wondering whether the authors have quantitatively investigated the correlation between the ZiCo scores of candidate Adapters and their final downstream segmentation performance (e.g., Dice or mIoU after training). Although the original ZiCo paper studied the correlation between ZiCo score and final performance, it's from the whole NN perspective, not from the adapter perspective.
  2. How does this work determine stopping condition (i.e. how the system knows when to stop search when further incremental architecture improvement doesn't worth the additional search time)?

I'm happy to consider raising the score once those two questions are addressed.

1 comment
  1. Thank you for the question. We examined the correlation between ZiCo scores and final downstream segmentation performance on the TNBC dataset using six searched adapter configurations. Each configuration was trained for 120 epochs, and its validation mDice was averaged over three random seeds. The ZiCo scores achieved a Spearman rank correlation of 0.49 and a Pearson correlation of 0.76 with the final validation mDice. We include the correlation trajectories over training epochs in Figure 4. Taken together, the correlation trajectories and final results suggest that ZiCo scores are also strongly associated with adapter trainability.
  2. The proposed method adopts a fixed-budget search strategy, where the stopping condition is determined by the predefined number of iterations and perturbation samples. We do not employ an adaptive early stopping criterion, ensuring a consistent evaluate budget throughout the search process. As shown in Table 4, increasing the number of search iterations does not necessarily improve downstream performance, while requiring additional computational cost. Based on this trade-off, we selected 20 iterations as the default search budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.