Routes Interfere, Representations Transfer: Parameter-Efficient Adaptation of Promptable Segmentation Models

Promptable segmentation models increasingly expose several prompt interfaces over one shared backbone: a text-grounded concept route, a click-driven geometric route, and a visual-exemplar route. Adapting such a model to a specialist domain raises a question that cannot arise for single-interface models — should the routes share one adaptation? We answer it on SAM3 in a setting that needs all three routes at once, model-assisted annotation of printed circuit boards, where the pretrained model is not merely weak but silent, returning no confident detection at all for two of its seven concepts. First, routes interfere: folding geometric prompts into concept training costs 20.5 classification-gated F1 at an identical parameter budget, because the objectives ask the shared features for opposing properties — category discrimination against prompt-conditioned figure–ground separation — and the damage falls on recognition, not localisation, exactly as that account predicts. One small adapter set per route removes the conflict: 0.10% of the parameters raise classification-gated F1 from 38.0 to 55.9 and mask AP from 0.52 to 0.75 ( 22.3 → 51.2 on a board-disjoint held-out set), and a disjoint 0.011% in the mask decoder lifts one-click IoU from 0.79 to 0.89 while calibrating its own quality estimate well enough to triage which masks an annotator must review. Second, representations transfer across routes: adapters trained on text alone, never exposed to an exemplar, improve the untrained exemplar route by 45% relative, with the largest gains on the fine-grained concepts. Low-rank adaptation thus changes what the backbone represents rather than how language is matched to it — which also explains why placement dominates capacity: quadrupling the trainable budget drops the model below zero-shot. We release the adapters, code, and a 330-image, 6,968-mask benchmark.