acceptodds
Under review as a conference paper at ICLR 2027

Your Model Already Knows, Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models

Abstract

Few-shot adaptation of vision-language models (VLMs) is difficult for specialized aerial, industrial, medical, and other imagery when only ten annotated images are available. Discrete prompt optimizers such as GEPA and DetPO keep the model frozen but search text without numerical gradients; low-rank adapters use gradient descent but learn millions of task-specific parameters. We revisit soft prompting, which optimizes a few continuous input tokens by gradient descent against a frozen backbone, and find that two defaults inherited from NLP hold it back: prefix placement and semantic initialization. Placing the tokens at the cross-modal boundary, between image and text, and initializing them from a semantically empty space token unlock its performance. On Roboflow20-VL, the resulting prompt uses 7,168 trainable parameters on average, outperforms GEPA and DetPO under a shared multi-class protocol, and matches the best LORA configuration at 14.2 mAP@50:95 with roughly 24,000× fewer trainable parameters. The learned tokens behave like prompts rather than weights: they transfer to a newer model version without retraining, and SOFT2HARD verbalizes them into an editable instruction that matches DetPO. The placement principle also carries beyond detection: on RoboCasa, a prompt that reaches the action expert of a frozen robot policy raises two near-floor tasks from 5.0% to 23.3% and 31.7% success. Our results suggest that modern VLMs already encode much of what specialized domains require; the task is not to teach them, but to learn how to ask.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Soft Prompting, Vision Language Models, Object Detection, Prefix Tuning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ZJPiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Objective: Adapt generative Vision-Language Models (VLMs) like Qwen3-VL-8B and Gemma-4-12B for specialized downstream tasks (e.g., 10-shot object detection and robot policy control) without tuning heavy LoRA adapters or relying on discrete text search.
  • Methodology: The study revisits soft prompting under several core principles:
    1. Cross-Modal Boundary Placement: Inserts 1 to 3 continuous soft prompt tokens directly between visual patch embeddings and text query tokens in sequence position.
    2. Meaning-Preserving Initialization: Initialized with a space character embedding, acting as a residual baseline that leaves text queries initially unchanged.
    3. SOFT2HARD Verbalization & Extensions: Distills continuous prompt vectors into human-readable text instructions via VLM descriptions and LLM summarization, and extends soft tokens to the action expert of frozen robot policies.
  • Key Findings & Empirical Validation: On Roboflow20-VL (10-shot, 20 domains), SOFTPROMPT averages 7,168 parameters (29.6 KB disk footprint) and reaches 14.2 mAP@50:95. It matches rank-64 LoRA (174.6M parameters, 666 MB) with fewer parameters and zero catastrophic task-forgetting on NaturalBench VQA (where LoRA degrades accuracy by 35%-56%).
Critical strengths
  • The paper re-evaluates multimodal soft prompting by identifying cross-modal placement (I; S; T]) and space-token initialization as essential design principles. Demonstrates that shallow input soft prompts can reach the Pareto efficiency frontier of massive LoRA adapters while keeping backbone weights 100% frozen.
  • Exceptional empirical scope across 20 domains, LVIS Rare 50, Gemma-4-12B, and RoboCasa robotics. Provides rigorous storage metrics, NaturalBench forgetting sweeps, vLLM serving export strategies, and transparent disclosures regarding training seed variance.
Critical weaknesses
  • Headline results rely on test-informed selection of prompt lengths per domain, rather than a strict holdout support validation set.
Questions for authors
  1. Can you report mean mAP@50:95 and 95% confidence intervals across 5 independent seeds for all 20 Roboflow domains in Table 1?
  2. Was prompt length L \in {1, 2, 3} chosen strictly on holdout support splits, or directly on test mAP for the main Qwen3-VL table?
  3. In Table 1, on medical domains, both SOFTPROMPT and LoRA hit a performance floor near 0.0 mAP (vs. 35.3 mAP for fine-tuned Grounding DINO). How can practitioners predict whether a domain requires soft prompt steering versus full adapter fine-tuning?
1 comment

We thank the reviewer for the positive evaluation and the specific questions.

Q1. Seeds and confidence intervals. We retrained one fixed SoftPrompt configuration (IST, L = 2, space initialization) and LoRA r = 64 with three seeds (42, 123, 2024) on every domain. Over the 19 domains where all six runs completed, SoftPrompt averages 11.74 mAP and LoRA 14.06. The paired difference is −2.33, with a 95% bootstrap CI over domains of [−5.32, +0.65]. We will report per-domain mean ± standard deviation and this interval in Table 1. Five seeds were not feasible within the discussion period; we will extend to five for the final version.

Q2. How L was chosen. In the submitted Table 1, L was chosen on test mAP, as disclosed in §4.1. We now select configurations by the lowest language-model loss on the RF20 validation split, with no access to test. On the nine domains where every signal is available, choosing hyper-params this way gives 16.41 mAP, against 16.79 for the test oracle; it matches the test choice on 8 of 9 domains. Validation mAP reaches only 14.79 (6 of 9), because validation splits hold just 3 to 30 images. We are extending the same rule to the length choice on all 20 domains, and it becomes the main-table protocol in the revision.

Q3. When to use a soft prompt versus an adapter. The frozen model's zero-shot mAP on the target domain is a practical predictor. On the medical domains the base model is near zero (dentalai 0.0, x-ray-id 0.1). There, no prompt helps, soft or discrete: SoftPrompt changes mAP by at most 0.1. LoRA reaches only 0.4 to 1.3, and only a fine-tuned specialist reaches 35.3. A soft prompt steers a recognition ability the frozen model already has; it cannot create one that is absent. Our recommendation to practitioners is to evaluate the frozen model zero-shot first and, if mAP is near zero, to use weight adaptation or a specialist detector. We will add this guidance with a per-domain plot of SoftPrompt gain against base mAP.

Presentation. We will tighten the method and results sections in the revision.

Reviewer_Z4fNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper revisits soft prompting for few-shot adaptation of generative vision-language models to specialized object-detection domains. The authors identify two design choices: placing learnable tokens at the image–text boundary (IST) and initializing them with a whitespace-token embedding.

On Roboflow20-VL, the method reports 14.2 mAP@50:95 using an average of 7,168 trainable parameters, matching the best class-name-only LoRA rank in the reported sweep while outperforming multi-class GEPA and DetPO. The paper further investigates soft-to-hard prompt verbalization, cross-version transfer to Qwen3.5-9B, and prompt placement in a frozen robotic policy. Extensive appendices provide implementation details, ablations, and analyses of training-seed variability.

Critical strengths

S1. Strong parameter efficiency with competitive reported performance. SoftPrompt substantially improves detection performance in several specialized domains using only a few thousand trainable parameters. The comparisons with multiple LoRA ranks and discrete prompt optimizers provide useful empirical context.

S2. Extensive analysis and transparent reporting. The paper examines prompt placement, initialization, length, verbalization, transfer, and robotic adaptation. The authors also explicitly report failure cases, training-seed variability, and limitations of their model-selection procedures.

Critical weaknesses

W1. Test-informed selection and substantial variability weaken the central performance claim. The Qwen3-VL headline selects prompt length separately for each domain using test mAP, while LoRA selects a single globally best rank from its test-evaluated sweep. This asymmetric selection, combined with predominantly single-run results, makes the reported 14.2-mAP parity difficult to interpret. Appendix R reports a mean training-seed CV of 28% on eight ablation datasets, and the corresponding multi-seed averages do not establish parity on that subset. A genuinely held-out, multi-seed comparison is needed.

W2. Secondary claims about transfer and practical advantages require stronger support. Appendix N reports 10.7 mAP for direct transfer to Qwen3.5-9B, whereas Appendix O reports 11.6, without explaining the apparent protocol difference. Soft2Hard improves aggregate mAP by only 0.3 over its no-soft-token control, with gains and losses split equally across domains. Furthermore, zero forgetting when task-specific soft tokens are omitted is not unique to soft prompting, since deactivating a LoRA adapter also restores the original model. These findings support narrower claims than the overall framing suggests.

Questions for authors

Q1. Independent model selection. Can the authors reproduce the main 20-domain results with prompt lengths and checkpoints selected exclusively using support-set validation? How were the dataset-specific training budgets and early-stopping criteria determined? Can LoRA ranks be selected under the same protocol?

Q2. Statistical reliability. Does SoftPrompt still match LoRA when both methods are evaluated across independent training seeds under the same selection procedure? Can the authors provide the full 20-domain multi-seed means and uncertainty estimates?

Q3. Transfer-result consistency. What explains the 10.7 versus 11.6 mAP results in Appendices N and O? Are different source prompts, checkpoint selections, or evaluation configurations involved?

Q4. Practical advantages. How does cross-task performance compare when both adaptation methods are activated or deactivated under matched conditions? What deployment advantage remains over removable LoRA adapters?

Q5. Training efficiency. What are the end-to-end training and model-selection costs of SoftPrompt relative to LoRA and discrete prompt optimization? Does the parameter-efficiency advantage translate into practical optimization efficiency?

1 comment

We thank the reviewer for a detailed and constructive review and answer each question in turn.

Q1. Independent model selection. RF20 validation splits hold 3 to 30 images, so validation mAP is too noisy to rank configurations. The language-model loss on the same images is far more stable. On the nine domains where every signal is available, we chose each domain's hyper-params by each rule:

Hyper-params chosen by (9 domains)Mean test mAPSame choice as test mAP
Test mAP (as submitted; an oracle)16.799 / 9
Validation loss (no test access)16.418 / 9
Always IST16.147 / 9
Validation mAP14.796 / 9

Validation loss costs 0.38 mAP relative to the oracle. Within each run, the checkpoint is also chosen by lowest validation loss. We are extending loss-based selection to length on all 20 domains, and it will become the main-table protocol in the revision.

Training budgets. The per-domain epoch budgets in the submission were not chosen on the support set, and we will state this. The three-seed comparison below uses one budget (5 epochs) for every domain.

Asymmetry with LoRA. You are right that LoRA used one global rank while L was chosen per domain. The fixed-configuration comparison under Q2 removes per-domain choice from both methods.

Q2. Statistical reliability. For both methods we trained three seeds (42, 123, 2024) per domain: one fixed SoftPrompt configuration (IST, L = 2, space initialization, 5 epochs) and LoRA r = 64. Over the 19 domains where all six runs completed:

3 seeds, no per-domain selectionMean mAP
SoftPrompt, fixed configuration11.74
LoRA r = 6414.06

The paired difference is −2.33 mAP (95% bootstrap CI over domains [−5.32, +0.65]), and SoftPrompt wins on 7 of 19 domains. With no per-domain choice, SoftPrompt does not match LoRA, and it varies more across seeds. We will add per-domain mean ± standard deviation for both methods. We will also restate the parity claim as conditional on choosing the configuration per domain.

Q3. Transfer: 10.7 versus 11.6. {{TRANSFER_CAUSE}}. The discrepancy in the scores was due to different seed runs and all other hyper-params are same. We will reconcile the two appendices to report same seed results.

W2. Soft2Hard. We agree that +0.3 mAP, with gains and losses split evenly across domains, does not demonstrate an accuracy benefit. What Soft2Hard provides is a readable, editable instruction derived from the learned prompt, which runs on the unmodified base model with no soft tokens. We will present it as an interpretability tool and remove any implication of an accuracy gain.

Q4. Forgetting with the adapters active. We agree that turning LoRA off restores the base model, just as omitting the prompt does. The matched comparison is therefore with both adapters active. We ran it on NaturalBench for all 20 domain adapters. The table shows the change in strict group accuracy from the base model's 37.1:

Adapter, activeMean Δ G-Acc (points)Worst domain
SoftPrompt−2.8−7.2
LoRA r = 64 (accuracy-matched rank)−13.0−36.5

With both adapters left on, LoRA at the accuracy-matched rank forgets 4.6 times more than SoftPrompt. In the paper we will replace "zero forgetting" with this active-versus-active comparison.

The deployment advantage is in serving. Every domain shares one unchanged set of model weights plus a few-kilobyte prompt, so the inference engine needs no support for switching adapters. The prompt can also stay on for mixed workloads with much less damage to other tasks.

Q5. Training efficiency. A SoftPrompt training run takes a median of about 2 hours on one GPU (243 runs; 10th to 90th percentile 7 minutes to 5.2 hours, driven mainly by image resolution). LoRA takes {{LORA_TIME}}. The backward pass still runs through the full frozen VLM, so the roughly 24,000-fold reduction in trainable parameters does not become a similar reduction in compute. The practical gains are storage (about 30 KB per domain versus 666 MB) and serving. Selecting by validation loss adds one forward pass per candidate and no extra training. We will add a cost table and state these points.

Reviewer_sKMLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates parameter-efficient few-shot adaptation of vision-language models (VLMs) through soft prompting. The authors propose placing learnable prompt tokens at the cross-modal boundary and initializing them with a space token to improve adaptation performance. Experiments on Roboflow20-VL demonstrate that the method achieves performance comparable to LoRA with substantially fewer trainable parameters. Additional experiments examine prompt interpretability, cross-model transfer, and adaptation to robotic policies.

Critical strengths
  1. High Parameter Efficiency: The proposed method achieves performance comparable to LoRA with substantially fewer trainable parameters, demonstrating its potential for efficient few-shot adaptation of VLMs.
  2. Comprehensive Experimental Evaluation: The paper provides extensive comparisons and ablation studies, along with evaluations of cross-model transfer and robotic policy adaptation, supporting the practical relevance of the proposed approach.
Critical weaknesses
  1. Limited Methodological Novelty: The proposed approach mainly builds on existing soft prompting techniques, with improvements focused on token placement and initialization. The methodological contribution is relatively incremental.
  2. Concerns About Experimental Reliability: The use of test-set performance for prompt length selection and the considerable variance across random seeds weaken the reliability of the main results, particularly the claimed performance parity with LoRA.
Questions for authors
  1. Experimental Reliability: How would the main results change if prompt lengths were selected using a validation set rather than test-set performance?
  2. Training Stability: Given the substantial variance across random seeds, can the authors clarify whether SoftPrompt consistently achieves performance comparable to LoRA across multiple runs?
1 comment

We thank the reviewer for the positive assessment and the focused questions on reliability.

Q1. Selection with a validation set instead of the test set. RF20 validation splits hold 3 to 30 images, so validation mAP is a noisy signal. The language-model loss on the same images is far more stable. On the nine domains where every signal is available, we chose each domain's hyper-params by each rule:

Hyper-params chosen by (9 domains)Mean test mAPSame choice as test mAP
Test mAP (as submitted; an oracle)16.799 / 9
Validation loss (no test access)16.418 / 9
Validation mAP14.796 / 9

Validation loss gives up 0.38 mAP relative to the oracle, while validation mAP gives up 2.00. In the revision, validation-loss selection will be the main-table protocol, and the test-selected row will appear only as an upper bound.

Q2. Stability across seeds. We retrained a single fixed configuration (IST, L = 2, space initialization) and LoRA r = 64 with three seeds on every domain. With no per-domain choice, SoftPrompt averages 11.74 mAP against LoRA's 14.06. The paired difference is −2.33 (95% CI [−5.32, +0.65]), and SoftPrompt wins on 7 of 19 domains. SoftPrompt also varies more across seeds than LoRA does. We will report per-domain mean ± standard deviation for both methods. We will also state that SoftPrompt's comparability with LoRA depends on choosing the configuration per domain.

Novelty. We agree that the method builds on existing soft prompting. We position the paper as an empirical study of what makes soft prompts work on current generative VLMs (placement relative to the image, initialization, length, and selection), together with Soft2Hard verbalization. The introduction will say this directly.

Reviewer_64ZpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors research a simple few-shot adaptation method for VLM, which is freezing the model and training only a small number of continuous prompt vectors which initialized as space tokens and inserted between the image and the text. The original weights of the VLM are frozen. Under causal attention, the soft tokens S can incorporate information from the preceding image tokens and subsequently influence the text queries and detection outputs. The paper evaluates six different sequence orderings, rather than comparing against a single prefix-based approach. The authors propose "SOFT2HARD" which use soft prompts to describe support images and aggregates them into a textual detection instruction, and finally performs evaluation using the base model without any soft tokens. The authors also informed a control experiment following the same pipeline excluding the soft tokens.

Critical strengths

The method is simple and practical. Its advantage in terms of storage, stemming from the use of extremely short input prompts. It is clear, making it able to be used across lots of domains. The experiments cover a broad range, including prompt searching, six different LoRA ranks, and detection tasks. The appendix provides a clear discussion on the method's limitations, including hyperparameter selection on test sets, training instability, performance loss during cross-model transfer, and the effects of distillation.

Critical weaknesses

1)The test set outputs were used in the parameter selection process—the appendix explicitly states that the prompt length L for each domain was chosen based on the test set's mAP which undermines the credibility of the evaluation. In the §4.1, it mentions that "chooses Lper domain from the reported short length sweep, just as the LORA row takes the best rank of its sweep (Appendix G)." In Appendix G, it also mentions that "For the reported Qwen3-VL headline, L is selected per domain by test mAP from the short length sweep; this test-informed selection is disclosed in §4.1."

2)This paper lacks innovation. Soft prompting is not a new concept; the authors' primary contribution lies in investigating aspects such as positioning, initialization, and the practical efficacy of extremely short prompts within generative multimodal sequences, rather than proposing an entirely novel adaptation paradigm.

Questions for authors

Does RF20 retain its core advantages when hyperparameter selection (L) is not performed using the test set? Please provide the full results for cases where L is fixed or where hyperparameter selection is restricted to the validation set only, and explain the rationale behind the training process and the criteria used for checkpoint selection.

What are the mean performance values for RF20 (across multiple seeds) under these fixed configurations, and what are the paired differences relative to LoRA/DetPO? The current table of random seeds covers ablation settings but does not directly validate the primary results. Please also clarify whether the performance advantage of IST/SIT over space/Xavier remains consistent.

1 comment

We thank the reviewer for the careful reading. The test-informed selection of L is the most important issue raised. We answer it with two new experiments: a selection rule that does not read test, and a three-seed comparison with no per-domain selection.

W1 / Q1. Selection without the test set. Validation mAP is a poor selection signal at this scale, because RF20 validation splits hold 3 to 30 images. The token-level language-model loss on the same images averages over every output token and is far more stable. We compared hyperparameter selection rules on the nine domains where every signal is available:

Hyper-params chosen by (9 domains)Mean test mAPSame choice as test mAP
Test mAP (as submitted; an oracle)16.799 / 9
Validation loss (no test access)16.418 / 9
Validation mAP14.796 / 9

(Means are higher than the 20-domain 14.2 because these nine domains include high-mAP ones such as trail-camera and aquarium-combined.)

Selecting by validation loss costs 0.38 mAP relative to the test oracle; selecting by validation mAP costs 2.00. In the revision, validation-loss selection will be the main-table protocol, and the test-selected row will appear only as an upper bound.

Training and checkpoint selection. Each run trains for a fixed epoch budget on the 10-shot support set. The checkpoint saved from each run is the one with the lowest validation loss. Test is never used for training or for choosing a checkpoint.

Q2. Multi-seed means and paired differences. We retrained one fixed configuration on every domain (IST, L = 2, space initialization, 5 epochs) with three seeds (42, 123, 2024), and retrained LoRA r = 64 with the same seeds. Over the 19 domains where all six runs completed:

3 seeds, no per-domain selectionMean mAP
SoftPrompt, fixed configuration11.74
LoRA r = 6414.06

The paired difference is −2.33 mAP (95% bootstrap CI over domains [−5.32, +0.65]); SoftPrompt wins on 7 of 19 domains. A single fixed configuration does not reach LoRA. SoftPrompt's accuracy depends on choosing the configuration per domain, which is why a selection rule without test access matters, and we will state this dependence explicitly. Against multi-class DetPO (12.5), the fixed configuration is 0.8 mAP lower; DetPO is a discrete optimizer and has no training seeds.

Design choices. In a search that uses only the support set, over format, length, initialization, and learning rate, space initialization is chosen on 12 of the 14 domains where a prompt is selected (the exceptions are wb-prova, which picks Xavier, and paper-parts, which picks mean). Placement splits between IST and TSI, so we will narrow the placement claim to "soft tokens should follow the image", which IST and TSI both do, and no longer rank IST above TSI.

W2. Novelty. We agree that soft prompting is not new and will say so directly in the introduction. The contribution is empirical:

  • which design choices determine whether soft prompts work in a current generative VLM: placement relative to the image, initialization, and length;
  • how to select them without test data;
  • Soft2Hard, which turns the learned prompt into a readable instruction that runs on the unmodified base model.

To our knowledge, soft prompting has not been studied in this form on current generative VLMs for detection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.