- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VAPOR optimizes prompts for frozen VLMs by inspecting failures, verifying corrective patches, contrasting unsuccessful and corrected reasoning traces, and retaining reusable rules. It organizes these rules through visual and semantic clustering, producing 1.) routed expert prompts or 2.) a globally fused prompt. Experiments cover aesthetic classification (ArtiMuse-10K), safety classification (SafeTag-VL-3K), and a MMMU evaluation with 2-4 VLM backbones, two VAPOR variants (route/fuse), and three optimization baselines (IPO, TextGrad, and ERM) where two have been customized to the multimodal setup. On ArtiMuse-10K and SafeTag-VL-3K is also compares against LoRA fine-tuning and different training dataset sizes. Component ablations are available.
Critical strengths
- On Qwen3-VL-32B aesthetic classification, the fused variant reaches 65.77%, compared with 59.66% for IPO and 38.14% for zero-shot prompting. These are useful improvements.
Critical weaknesses
- Misses important related work. For example MIPROv2, OPRO and Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs (ICLR 2026) are not discussed. Only IPO, TextGrad, and ERM are used for baseline comparison.
- Experiments dont compare image-conditioned reflection against reflection using only textual descriptions or reasoning traces. Its unclear if and how much the use of visual evidence explains VAPORs gains.
Questions for authors
- You mention the optimization baselines TextGrad and ERM being customized for multimodal optimization. How did you do that?
4 comments
Thanks. The MIPROv2 comparison, description-based reflection control, and adapter details do address my original concerns. But from my perspective, it still remains not clear if these optimization budgets are comparable? I would still keep my initial rating for now.
We thank the reviewer for asking whether the optimization budgets are comparable. We first report the measured consumption of the additional SafeTag runs, then clarify how stage-wise performance can be compared along the same optimization process.
1. Measured consumption and final performance.
The additional runs use Qwen3-VL-32B as the target model and the same SafeTag training, validation, and test splits. We audited the saved API usage records and final predictions.
| Method | Successful optimization calls | Input tokens | Output tokens | optimization cost | Test accuracy |
|---|---|---|---|---|---|
| MIPROv2 | 1,126 | 1,940,330 | 287,089 | $0.429 | 86.57% |
| VAPOR-Route | 2,702 | 1,944,363 | 744,436 | $0.684 | 87.14% |
Optimization accounting includes recorded generation, reflection, verification, construction, and candidate-evaluation requests. Final test inference is excluded. Costs use the API provider’s listed prices, equivalent to approximately $0.1389 per million input tokens and $0.5556 per million output tokens.
VAPOR used 20.7% more optimization tokens. Its larger output-token consumption explains the 59.4% higher estimated cost, an absolute difference of $0.255. Final test accuracy was 0.57 percentage points higher in this single-run comparison.
2. Stage-wise comparison along the optimization trajectory.
Final totals alone do not show how performance develops with search consumption. Appendix C.1 and Figure 4 address this question for IPO, TextGrad, and ERM by reporting best-so-far validation accuracy at 200-call intervals, averaged over five runs.
The corresponding VAPOR checkpoint is constructed from the rules available at that stage. Because rules are obtained through sample-level optimization, they are consolidated into their respective expert prompts before evaluation. The expert library and routing mechanism are preserved.
MIPROv2 instead produces executable instruction-and-demonstration configurations. At each corresponding checkpoint, its available complete candidate is selected using the predefined validation-based rule and evaluated on the same full validation split.
This compares the useful results accumulated during search without equating a VAPOR expert-construction stage with a MIPROv2 search trial.
3. Search calls versus checkpoint-construction overhead.
VAPOR’s additional consolidation into temporary checkpoint experts is excluded from this search-only axis and recorded separately, together with evaluations performed solely to monitor progress. Such temporary construction just operate on a copy of the state and must not provide uncounted feedback to subsequent optimization. Any construction used to guide continued search belongs to the search budget. The following numerical illustration shows the reporting format.
The following table reports cumulative optimization consumption at common call checkpoints. Input and output tokens include successful optimization requests only. Test accuracy is measured after selecting the checkpoint according to validation accuracy; test labels are not used for checkpoint selection.
| Optimization calls | MIPROv2 input | MIPROv2 output | MIPROv2 test accuracy | VAPOR input | VAPOR output | VAPOR test accuracy |
|---|---|---|---|---|---|---|
| 200 | 280,516 | 53,461 | 85.71% | 105,223 | 48,889 | — |
| 400 | 683,587 | 105,666 | 85.71% | 212,718 | 96,509 | 85.29% |
| 600 | 966,745 | 159,119 | 85.86% | 323,804 | 147,976 | — |
| 800 | 1,333,843 | 208,074 | 86.57% | 481,793 | 205,752 | 86.29% |
| 1,000 | 1,706,176 | 257,087 | 86.57% | 651,154 | 262,578 | — |
| 1,126 | 1,940,330 | 287,089 | 86.57% | 732,707 | 298,893 | — |
| 1,200 | — | — | — | 795,698 | 322,905 | 86.43% |
| 1,600 | — | — | — | 1,094,032 | 443,058 | 86.71% |
| 2,000 | — | — | — | 1,411,739 | 554,931 | 86.86% |
| 2,400 | — | — | — | 1,701,151 | 661,955 | 87.14% |
| 2,702 | — | — | — | 1,944,363 | 744,436 | 87.14% |
The two views answer different questions. The measured totals quantify the resources consumed in the recorded runs. The stage-wise curve measures progress at matched search-call checkpoints, while separately recorded expert-construction and monitoring costs allow a total-consumption comparison.
Dear Reviewer #VDEV,
We would like to kindly follow up on our rebuttal. We would greatly appreciate it if you could let us know whether our responses have adequately addressed your new concerns or if any issues remain unresolved. Thank you for your time and consideration!
We thank the reviewer for recognizing the improvements and for the constructive comments.
1. Related work and baseline selection.
OPRO, “Large Language Models as Optimizers,” is cited as Yang et al. in our Introduction and references. OPRO generates instructions from historical instruction–score pairs. IPO applies a related search procedure to vision-language classification, and its official implementation builds on OPRO and CoOp.
Our IPO adapter retains this history-based search and replaces CLIP-based classification with evaluation by the target generative VLM. This allows candidate instructions to be scored using our SELECTED/DISCARD interface. The current adapter uses the instructions-only branch without individual image descriptions; IPO’s original paper also includes a configuration without descriptions.
MIPROv2 jointly optimizes instructions and few-shot demonstrations. VAPOR instead extracts reusable rules from verified failed-to-corrected trajectories and organizes them into expert prompts. The comparison below specifies Qwen3-VL-32B, 100 training examples, common validation/test splits, and retention of MIPROv2’s demonstration-selection mechanism.
MPO jointly optimizes textual and non-textual prompts, including generated or edited auxiliary images. VAPOR uses images as evidence during reflection but optimizes textual instructions without generating auxiliary visual prompts. Full MPO changes the optimization space and is therefore not a directly matched baseline for our textual prompt-optimization setting. Its contribution concerns optimizing multimodal prompt components, whereas ours concerns converting visual errors into verified textual rules.
2. Image-conditioned versus description-based reflection.
Our existing comparisons include text-based optimization through IPO and TextGrad and image-conditioned feedback through ERM. Since these methods differ in several components, we additionally compare reflection inputs within VAPOR-Route.
Description-VAPOR-Route replaces the reflector’s image with a fixed textual description generated without access to the correct label. The current prompt, prediction, training label, original reasoning, and memory remain available. The reflector, trajectory comparison, verification procedure, and search budget are unchanged. Task prediction and patch verification still receive images; only the reflection input changes.
The comparison uses Qwen3-VL-32B with 100 training examples. Description-VAPOR-Route scores below the reported VAPOR-Route mean by 4.63 percentage points on ArtiMuse and 2.46 points on SafeTag. This provides additional evidence that direct image access during reflection benefits prompt optimization beyond descriptions and existing reasoning. The larger difference on ArtiMuse is consistent with descriptions omitting fine-grained aesthetic details.
3. Multimodal adaptation of TextGrad and ERM.
Appendix B.4 describes the adapters. Both evaluate multimodal inputs, but their optimization feedback differs.
TextGrad. The system prompt is the optimizable text variable. The target VLM evaluates images and, for SafeTag, accompanying user instructions. Textual loss feedback contains training metrics, failed-example identifiers, predictions, correct labels, and SafeTag instructions. Qwen3-32B generates textual gradients and updates the prompt. The current backward engine does not receive images, descriptions, or reasoning traces. Revised prompts are evaluated with the VLM and selected by validation accuracy.
ERM. The target VLM receives the failed image, current prompt, prediction, training label, original reasoning, and retrieved feedback and exemplar context. It produces structured feedback describing the error, visual evidence, and suggested correction. Qwen3-32B aggregates this feedback into a prompt patch. Feedback memory, exemplar storage, and selective forgetting are retained.
ERM therefore already inspects images during reflection. Its comparison with VAPOR evaluates complete optimization procedures, while the description-based control targets direct visual access. The adapters share task splits, seed prompts, verdict format, and validation-based selection.