- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- Objective: Adapt generative Vision-Language Models (VLMs) like Qwen3-VL-8B and Gemma-4-12B for specialized downstream tasks (e.g., 10-shot object detection and robot policy control) without tuning heavy LoRA adapters or relying on discrete text search.
- Methodology: The study revisits soft prompting under several core principles:
- Cross-Modal Boundary Placement: Inserts 1 to 3 continuous soft prompt tokens directly between visual patch embeddings and text query tokens in sequence position.
- Meaning-Preserving Initialization: Initialized with a space character embedding, acting as a residual baseline that leaves text queries initially unchanged.
- SOFT2HARD Verbalization & Extensions: Distills continuous prompt vectors into human-readable text instructions via VLM descriptions and LLM summarization, and extends soft tokens to the action expert of frozen robot policies.
- Key Findings & Empirical Validation: On Roboflow20-VL (10-shot, 20 domains), SOFTPROMPT averages 7,168 parameters (29.6 KB disk footprint) and reaches 14.2 mAP@50:95. It matches rank-64 LoRA (174.6M parameters, 666 MB) with fewer parameters and zero catastrophic task-forgetting on NaturalBench VQA (where LoRA degrades accuracy by 35%-56%).
Critical strengths
- The paper re-evaluates multimodal soft prompting by identifying cross-modal placement (I; S; T]) and space-token initialization as essential design principles. Demonstrates that shallow input soft prompts can reach the Pareto efficiency frontier of massive LoRA adapters while keeping backbone weights 100% frozen.
- Exceptional empirical scope across 20 domains, LVIS Rare 50, Gemma-4-12B, and RoboCasa robotics. Provides rigorous storage metrics, NaturalBench forgetting sweeps, vLLM serving export strategies, and transparent disclosures regarding training seed variance.
Critical weaknesses
- Headline results rely on test-informed selection of prompt lengths per domain, rather than a strict holdout support validation set.
Questions for authors
- Can you report mean mAP@50:95 and 95% confidence intervals across 5 independent seeds for all 20 Roboflow domains in Table 1?
- Was prompt length L \in {1, 2, 3} chosen strictly on holdout support splits, or directly on test mAP for the main Qwen3-VL table?
- In Table 1, on medical domains, both SOFTPROMPT and LoRA hit a performance floor near 0.0 mAP (vs. 35.3 mAP for fine-tuned Grounding DINO). How can practitioners predict whether a domain requires soft prompt steering versus full adapter fine-tuning?
We thank the reviewer for the positive evaluation and the specific questions.
Q1. Seeds and confidence intervals. We retrained one fixed SoftPrompt configuration (IST, L = 2, space initialization) and LoRA r = 64 with three seeds (42, 123, 2024) on every domain. Over the 19 domains where all six runs completed, SoftPrompt averages 11.74 mAP and LoRA 14.06. The paired difference is −2.33, with a 95% bootstrap CI over domains of [−5.32, +0.65]. We will report per-domain mean ± standard deviation and this interval in Table 1. Five seeds were not feasible within the discussion period; we will extend to five for the final version.
Q2. How L was chosen. In the submitted Table 1, L was chosen on test mAP, as disclosed in §4.1. We now select configurations by the lowest language-model loss on the RF20 validation split, with no access to test. On the nine domains where every signal is available, choosing hyper-params this way gives 16.41 mAP, against 16.79 for the test oracle; it matches the test choice on 8 of 9 domains. Validation mAP reaches only 14.79 (6 of 9), because validation splits hold just 3 to 30 images. We are extending the same rule to the length choice on all 20 domains, and it becomes the main-table protocol in the revision.
Q3. When to use a soft prompt versus an adapter. The frozen model's zero-shot mAP on the target domain is a practical predictor. On the medical domains the base model is near zero (dentalai 0.0, x-ray-id 0.1). There, no prompt helps, soft or discrete: SoftPrompt changes mAP by at most 0.1. LoRA reaches only 0.4 to 1.3, and only a fine-tuned specialist reaches 35.3. A soft prompt steers a recognition ability the frozen model already has; it cannot create one that is absent. Our recommendation to practitioners is to evaluate the frozen model zero-shot first and, if mAP is near zero, to use weight adaptation or a specialist detector. We will add this guidance with a per-domain plot of SoftPrompt gain against base mAP.
Presentation. We will tighten the method and results sections in the revision.