Search2Adapt: Training-Free Adapter Architecture Search for Segmentation Foundation Models

Recent foundation models show remarkable success across general-domain vision tasks but often struggle in specialized domains with distinct visual characteristics. Existing adaptation methods typically rely on fixed adapter architectures that cannot account for domain-specific characteristics. We propose Search2Adapt, a framework that automatically discovers effective adapter architectures for segmentation foundation models through training-free search. Search2Adapt parameterizes each adapter by its operation type, configuration, and layer-wise insertion, and perturbs these variables to construct candidate adapters. Each candidate is scored by a zero-shot proxy based on gradient statistics with a sparsity penalty on layer-wise insertion, and the perturbation variables are iteratively updated toward high-scoring candidates through score-weighted aggregation. This enables domain-specific adapter search without training individual candidates. To evaluate Search2Adapt across specialized segmentation tasks, we conduct extensive experiments on camouflage, polyp, and nuclei segmentation datasets. Search2Adapt achieves performance competitive with or superior to fixed adapter designs across diverse domains, while sensitivity analysis confirms that adapter configuration substantially affects segmentation performance. These results demonstrate effective domain-specific adapter architecture discovery for segmentation foundation models.