acceptodds
Under review as a conference paper at ICLR 2027

Routes Interfere, Representations Transfer: Parameter-Efficient Adaptation of Promptable Segmentation Models

Abstract

Promptable segmentation models increasingly expose several prompt interfaces over one shared backbone: a text-grounded concept route, a click-driven geometric route, and a visual-exemplar route. Adapting such a model to a specialist domain raises a question that cannot arise for single-interface models — should the routes share one adaptation? We answer it on SAM3 in a setting that needs all three routes at once, model-assisted annotation of printed circuit boards, where the pretrained model is not merely weak but silent, returning no confident detection at all for two of its seven concepts. First, routes interfere: folding geometric prompts into concept training costs 20.5 classification-gated F1 at an identical parameter budget, because the objectives ask the shared features for opposing properties — category discrimination against prompt-conditioned figure–ground separation — and the damage falls on recognition, not localisation, exactly as that account predicts. One small adapter set per route removes the conflict: 0.10% of the parameters raise classification-gated F1 from 38.0 to 55.9 and mask AP from 0.52 to 0.75 ( 22.3 → 51.2 on a board-disjoint held-out set), and a disjoint 0.011% in the mask decoder lifts one-click IoU from 0.79 to 0.89 while calibrating its own quality estimate well enough to triage which masks an annotator must review. Second, representations transfer across routes: adapters trained on text alone, never exposed to an exemplar, improve the untrained exemplar route by 45% relative, with the largest gains on the fine-grained concepts. Low-rank adaptation thus changes what the backbone represents rather than how language is matched to it — which also explains why placement dominates capacity: quadrupling the trainable budget drops the model below zero-shot. We release the adapters, code, and a 330-image, 6,968-mask benchmark.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Parameter-efficient fine-tuning, LoRA, promptable segmentation, prompt-interface interference, cross-route transfer, open-vocabulary instance segmentation, interactive segmentation, model-assisted annotation, Printed Circuit Boards

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yUK6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work argues that for specialist domains, one should use a new adapter for each segmentation prompt modality, rather than using a shared adapter. The authors show on a benchmark that they collected from other datasets (of circuit parts) that their proposed method performs better than baselines.

Critical strengths

The figures are actually relatively simple and easy to understand. I got most of the point of the method from the figures alone.

Also, the results appear to be stronger than baseline. This suggests that using per-modality adapters may be useful.

Critical weaknesses

In contrast to the figures, the text appears AI generated and is really hard to digest. There were several sentences and captions in the paper where I spent several minutes trying to figure out what they said, and still was not clear what they did. I want to emphasize this is despite being familiar with segmentation literature, and most of my understanding ended up drawing from prior knowledge rather than actually understanding what they wrote. Thus it is hard to judge the details of their results.

In my opinion until the paper is rewritten no amount results or SOTA improvement will make it worthy of acceptance.

Also, I am not really sure what the core novelty is. It looks like they argue to use a different set of LoRA weights for every type of condition. But I feel it is not clear what the insight from this is, then.

Questions for authors

It seems extremely counterintuitive that finetuning with LoRA has such strong results but with DoRA it is so weak. Is it possible the DoRA baseline did not have good hyperparameters or there was some other issue?

Reviewer_NHD5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to adapt a segmentation model with several prompt interfaces to a specialist domain. On SAM3, the authors train text-concept LoRA adapters in the later vision blocks and detector attention layers, then freeze them and train separate adapters in the interactive mask decoder. The exemplar route receives no dedicated training. Together, the adapters contain 978,944 parameters, approximately 0.114% of the model.

The paper reports that mixing text and geometric prompts reduces concept-route cgF1 by 20.5 points relative to text-only training at the same adapter budget. It also reports transfer to visual-exemplar inference without exemplar training: cross-image pmF1 improves from 0.337 to 0.490. Evaluation uses a newly annotated PCB dataset of 330 images and 6,968 masks across seven categories, with an additional disjoint test set of 44 industrial boards. The authors interpret the results as evidence for route interference and changes in shared visual representations.

Critical strengths
  1. The paper targets a practically relevant adaptation problem where concept retrieval, interactive correction and exemplar propagation coexist in annotation pipelines, and reports substantial performance gains across multiple key metrics including cgF1, mask AP, one-click IoU and review AUROC for domain-specific annotation. It provides informative diagnostic analysis disentangling localization and recognition behaviors and observes cross-route transfer without direct exemplar training, while maintaining transparent reporting by clearly stating limitations such as private test data and a single training seed.

  2. This work addresses a valuable multi-interface adaptation problem for annotation workflows and achieves clear, meaningful improvements on validation and out-of-distribution boards. The analysis decomposes model behavior into localization and presence recognition and reveals interesting transfer effects to the exemplar route; the authors also openly document caveats including uncontrolled baseline setups and limited test conditions, supporting reliable interpretation of the results.

Critical weaknesses
  1. The core interference experiment cannot clearly disentangle route-objective conflict. The routing of geometric prompts, loss formulation and gradient flow are underspecified. Given only 60% text samples are used in the mixed setting, the observed concept performance drop could arise from reduced text supervision or calibration shift instead of true cross-route interference. The absence of geometric-route metrics for the shared baseline further complicates interpretation. Additionally, exemplar transfer gains cannot be solely credited to the visual backbone, as adapted detection modules are reused during exemplar inference; improvements may originate from candidate generation, prompt matching or rescoring. The DINOv3 rescoring ablation is limited, and exemplar evaluation omits negative queries to test absent-concept recognition.
  2. Adapter comparisons are confounded by co-varying factors including adapter type, insertion position, tunable capacity and auxiliary modules. Current ablations only characterize the tested configurations and do not establish general principles for adapter placement or relative superiority. Moreover, key results lack robust uncertainty quantification: experiments rely on a single training seed, bootstrap analysis is not applied to the critical interference comparison, within-image and within-board correlations are ignored, and the disjoint test set validates final adaptation instead of the core interference and transfer claims.
Questions for authors
  1. What forward path do the geometric samples in the 60/40 baseline use: the interactive mask decoder or the concept detector’s visual-prompt interface? Which losses and presence labels apply, and which adapter parameters receive gradients? Please also report the shared model’s geometric-route performance, if available, to clarify the claimed trade-off.

  2. Does the interference gap persist when text-only and mixed training are matched for effective text-supervised updates and checkpoint selection, with presence supervision explicitly specified? Please report mask AP for the mixed configuration and a focused threshold or presence-gate diagnostic to distinguish discrimination changes from calibration changes.

  3. What are the exemplar results when only the vision adapters or only the detector adapters are active? Can the authors also clarify performance on negative exemplar queries? These results would help distinguish backbone changes from shared-detector adaptation and establish the scope of recognition transfer.

  4. How many independent physical boards are represented in training and validation? For the central text-only and 60/40 comparison, can the authors provide repeated-seed results and uncertainty on the difference, using image- or board-level resampling where appropriate? A large ratio sweep is less essential than a reliable, supervision-matched core comparison.

  5. Does the LoRA–DoRA gap remain under matched adaptation sites, head-freezing policies, and comparable tuning budgets? For other adapter families, please use comparable configurations appropriate to their architectures. Can the segmentation-head effect be separated from the auxiliary adapter, or can the placement/capacity claims be narrowed to the configurations actually tested?

Reviewer_KWsnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether a promptable segmentation model with multiple interfaces should use one shared adaptation or separate adapters per route. On SAM3, the authors adapt the text-grounded concept route and the click-driven geometric route separately for fine-grained PCB component segmentation, while using the untrained exemplar route to test cross-route transfer. They report that mixing text and geometric prompts into a shared adapter reduces classification-gated F1 by 20.5 points relative to text-only training at the same parameter budget, whereas route-specific LoRA adapters improve concept-route cgF1 from 38.0 to 55.9 on validation and from 22.3 to 51.2 on a board-disjoint test set. The text-trained adapters also improve exemplar-route performance, which the authors interpret as evidence of representation transfer. The evaluation includes a newly annotated 330-image, 6,968-mask PCB benchmark, adapter-family comparisons, route-specific ablations, and an industrial test set. The results are promising, but the central interference claim and the mechanistic interpretation of cross-route transfer are supported by limited controls.

Critical strengths
  1. The paper identifies a consequential adaptation question for multi-route models.

    • Rather than treating parameter-efficient adaptation as a single placement or capacity problem, it asks whether different prompt interfaces should share an adaptation.
    • This framing is practically relevant to models whose workflows combine text prompts, clicks, and visual exemplars.
  2. The evaluation targets a meaningful domain-shift failure and uses a board-disjoint test set.

    • The PCB task includes fine-grained, visually similar categories on which zero-shot SAM3 fails to produce confident detections.
    • The industrial test set contains different boards and is not used for model selection or threshold tuning, providing useful evidence beyond the image-wise validation split.
  3. The paper reports a useful decomposition of performance.

    • Separating recognition from localization helps show where adaptation changes performance: the main reported gain is in category presence and discrimination, rather than simply improving mask boundaries.
    • The per-category results further make the fine-grained gains visible.
  4. The method is parameter-efficient and technically well specified.

    • The authors describe adapter placement, objectives, matching, training, and evaluation in detail.
    • The two trained adapter sets together use 0.114% of SAM3’s parameters, and the paper reports inference costs as well as segmentation metrics.
  5. The experiments include informative negative results and controls.

    • The paper reports failures from union-mask supervision, higher-capacity configurations, and alternative adapter families.
    • These results help delimit the proposed recipe and make the study more useful than a report of only its best configuration.
  6. The untrained exemplar route is an interesting test of cross-route transfer.

    • The authors evaluate a route that receives no route-specific training objective or adapter, and report improvements when the text-trained adapters are active.
    • This is a useful empirical observation, even if the precise mechanism behind it is not fully isolated.
Critical weaknesses
  1. The central route-interference comparison does not establish that objective conflict, rather than reduced text supervision, causes the performance drop.

    • The mixed condition trains on text for only 60% of samples, while the comparison condition trains on text alone.
    • Because evaluation of the concept route is text-prompted, fewer text-prompt updates could explain at least part of the recognition loss.
    • The paper itself describes the mixture as diluting the text-conditioned signal.
    • Requested improvements:
      • Include a text-only control with equivalent text exposure and optimization budget, or a more controlled factorial comparison.
      • Provide evidence distinguishing objective conflict from reduced text supervision.
    • This concern bears directly on the paper’s main claim that the routes interfere through opposing demands on shared features.
  2. The cross-route result supports transfer of performance, but does not uniquely establish the proposed representational mechanism.

    • The exemplar route shares the vision backbone, fusion encoder, and detector with the text route, and the text adapters are placed in both the vision encoder and detector.
    • Thus, the exemplar improvement could arise from changes to the shared detector or its processing of visual prompts, rather than specifically from a change in the backbone representation.
    • The DINOv3 rescoring control is informative but does not isolate these possibilities.
    • Requested improvements:
      • Moderate the mechanistic conclusion to reflect the evidence currently available, or provide additional experiments that distinguish backbone representation changes from shared-detector effects.
Questions for authors

Clarification requests:

  1. Matched text supervision in the route-interference experiment: In the 60% text / 40% geometric mixed condition, how many text-prompt training updates does the model receive compared with the text-only condition? Do the authors have a text-only control with the same number of text updates and the same total optimization budget? Such a result could materially change the interpretation of the interference experiment.

  2. Source of cross-route transfer: Can the authors separate the contribution of the vision-encoder adapters from that of the fusion-encoder/DETR adapters to exemplar-route gains? If not, could they clarify the extent to which the reported result establishes transfer through the shared visual representation, as opposed to adaptation of components directly used by both routes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.