- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
STAVE learns two vectors that are added to the existing prompt embeddings of a frozen model: one changes the tokens that produce the answer, and the other changes the remaining context tokens. The vectors are trained on labeled examples with and without demonstrations, then used without demonstrations at test time. Experiments cover six multimodal and five language models, with ablations examining the token split and training objective.
Critical strengths
- Storing a task in just two vectors is useful because it avoids repeating demonstrations at inference, and Table 6 shows the same prompt length and measured cost as zero-shot inference.
- The gains also appear on held-out data across several backbones, and the token-split and training ablations help explain why the method works (Tables 2, 3, and 10; Appendix H).
Critical weaknesses
- Choosing the best epoch on the evaluation set makes Table 1 an optimistic estimate, and evaluating 16/32-shot ICL on fewer queries makes those comparisons less direct, although the epoch check partly addresses the first concern (Appendices F.2 and F.5).
- PT is trained for 500 steps using only the first answer token, while STAVE can train for 4,000 steps using the full answer, so the gap may partly reflect how they were trained (Appendix F.3).
Questions for authors
- How do the Table 1 results change when checkpoints are chosen without using the evaluation set and all methods are tested on the same queries?
- Does STAVE still outperform PT when PT is trained on the full answer and given a comparable training and tuning budget?
2 comments
Response to Reviewer Yu4N (2/2)
A2 (continued).
(2) New: PT retrained with the full-answer loss. We retrained all three PT modes with the full-answer cross-entropy, and set PT's training to 4,000 steps on VizWiz, OK-VQA and Flowers. STAVE itself trains for 4,000 steps only on Flowers and for only 250 to 1,000 steps on the other datasets. On DTD and CUB, its steps are no more than PT's 500 in Table 2, so there PT keeps its 500 steps and only its loss changes. Accuracy (%):
| Cell | STAVE | PT-Pre | PT-App | PT-BL |
|---|---|---|---|---|
| Idefics2 VizWiz | 69.63 ± 0.37 | 69.12 ± 0.55 | 68.00 ± 0.61 | 64.63 ± 7.07 |
| Idefics2 OK-VQA | 59.59 ± 0.50 | 59.00 ± 0.45 | 58.92 ± 0.51 | 59.23 ± 0.33 |
| Idefics2 DTD | 95.36 ± 0.47 | 95.04 ± 0.20 | 94.30 ± 0.38 | 94.66 ± 0.49 |
| Idefics2 Flowers | 95.67 ± 0.42 | 92.53 ± 5.97 | 93.20 ± 1.00 | 93.40 ± 0.28 |
| Idefics2 CUB | 93.50 ± 0.72 | 78.88 ± 25.33 | 95.00 ± 0.39 | 95.35 ± 0.31 |
| Qwen-VL VizWiz | 68.60 ± 0.55 | 67.48 ± 0.68 | 66.58 ± 0.45 | 66.92 ± 0.86 |
| Qwen-VL OK-VQA | 62.39 ± 0.12 | 59.83 ± 0.35 | 58.92 ± 0.53 | 59.58 ± 0.62 |
| Qwen-VL DTD | 95.20 ± 0.32 | 95.09 ± 0.32 | 93.40 ± 0.92 | 79.96 ± 24.53 |
| Qwen-VL Flowers | 96.30 ± 0.23 | 96.59 ± 0.54 | 95.42 ± 0.31 | 95.73 ± 0.15 |
| Qwen-VL CUB | 95.56 ± 0.26 | 95.39 ± 0.41 | 94.07 ± 0.70 | 94.68 ± 0.10 |
Even with many more training steps than STAVE, PT remains below it in all 12 VizWiz and OK-VQA comparisons. Over the five datasets, STAVE has the best average on both backbones (82.75 and 83.61 vs at most 81.88 and 82.88 for PT), and no PT mode is above STAVE in more than one of the 10 cells: PT-Pre only on Qwen-VL Flowers, and PT-App and PT-BL only on Idefics2 CUB, where PT-Pre falls to 78.88. Moreover, PT is less practical than STAVE:
- Its score depends on where the virtual tokens are placed. PT-Pre is the best mode in seven cells and PT-BL in three, and the three modes differ by up to 16.47 points within a cell. Each task therefore needs a search over placements, which STAVE does not need.
- Its training is unstable across runs. With the same data and number of steps, two Idefics2 VizWiz PT-BL runs reach 68.63 and 68.80, but the third ends at 56.47. One run of Idefics2 CUB PT-Pre and one of Qwen-VL DTD PT-BL end near 50% (49.63 and 51.63), while the other two runs are above 93.
- It costs more at inference. PT adds 20 virtual tokens to every prompt and has 10 times as many parameters, while STAVE keeps the zero-shot prompt length and cost (Table 7).
(3) Tuning budget: PT's scores are optimistic. PT is very sensitive to its learning rate, one more major drawback: a rate that trains well in one run can collapse in another on the same model and dataset (Appendix F.2). So in Table 2, we retrained every failed PT run at a retuned learning rate. No other method received such a per-run adjustment, so PT's scores are optimistic compared with those of STAVE and the task vector methods. Examples on Idefics2 (Table 2 setting):
| PT entry | Learning rate | Accuracy (%) |
|---|---|---|
| VizWiz PT-BL | 3×10⁻³ | 66.25 in one run, 0.01 in another (66.59 after changing to 5×10⁻³) |
| OK-VQA PT-BL | 10⁻² | 55.50 in one run, 0.11 in another (56.84 after changing to 10⁻³) |
| VizWiz PT-Pre, one run | 10⁻² / 3×10⁻³ / 10⁻³ / 3×10⁻⁴ | 43.39 / 0.00 / 0.00 / 64.74 |
A collapsed run outputs a repeated word (for example, "un un un"). In Table 2, a rate that was stable in one run failed in another run in 8 of the 15 Idefics2 PT entries, while no STAVE run on these cells diverged.
(4) Beyond the mean score. STAVE is also more stable than PT. Across repeated runs with the same data and configuration, whether a test answer is correct changes on 14 to 21% of the items for PT but on 9 to 12% for STAVE, in every cell and for every PT mode.
| Items whose correctness changes across runs (%) | Idefics2 VizWiz | Idefics2 OK-VQA | Qwen-VL VizWiz | Qwen-VL OK-VQA |
|---|---|---|---|---|
| STAVE | 9.24 | 12.05 | 10.44 | 12.09 |
| Best PT mode of Pre, App, BL | 14.42 | 16.53 | 17.57 | 18.95 |
(5) PT makes little use of visual demos. On TrueMICL [1], whose tasks are built to require the demo images, we trained PT with STAVE's data, steps and learning rate, and no demos in its training prompt. Results on Qwen2-VL-7B-Instruct with 4 demos at test (Avg7 is TrueMICL's 7-task average):
| Method (+ 4 demos at test) | Avg7 | Task-level Avg. | Episode-level Avg. | Avg7 change from adding the 4 demos |
|---|---|---|---|---|
| PT-Pre | 78.19 ± 2.93 | 90.33 ± 2.90 | 52.19 ± 6.16 | +7.57 |
| PT-App | 58.48 ± 9.45 | 74.58 ± 9.32 | 38.96 ± 11.13 | -7.90 |
| PT-BL | 67.10 ± 2.19 | 81.67 ± 4.86 | 46.89 ± 1.15 | -1.14 |
| STAVE | 86.38 ± 1.15 | 95.58 ± 1.51 | 66.88 ± 0.55 | +16.57 |
The 4 demos raise STAVE by 16.57 Avg7 points but PT-Pre by only 7.57, and lower PT-App and PT-BL. STAVE + 4 demos is 8.19 to 27.90 Avg7 points above the three PT modes.
We hope these results resolve both concerns and welcome any further questions.
[1] Chen et al. True Multimodal In-Context Learning Needs Attention to the Visual Context. COLM 2025.
Response to Reviewer Yu4N (1/2)
We sincerely thank the reviewer for the positive assessment and the constructive questions. We answer both below.
W1/Q1. Choosing the best epoch on the evaluation set makes Table 1 optimistic, and 16/32-shot ICL uses fewer queries. How do the results change when checkpoints are chosen without the evaluation set and all methods use the same queries?
A1. STAVE changes by at most 0.34 points when its checkpoints are chosen on a separate validation split, and 16/32-shot ICL scored on the full evaluation sets stays far below STAVE in every column.
(1) Already in the paper. Appendix F.2 states: "MimIC and HiFICL report results from the best-performing epoch on the evaluation split of these benchmarks, and we follow this protocol", and "Tables 2 and 10 instead select the checkpoint on validation samples". The protocol of Table 1 sets aside no validation split. Tables 2 and 10 score only the selected checkpoint on test, and Table 3 makes every per-task choice on a development split, never on the test split (Appendix F.4). There, STAVE has the best average on both Table 2 backbones, the best mean on five of the six Table 10 columns, and the highest Table 3 average.
(2) New: Table 1 with validation-chosen checkpoints. For each dataset we built a validation split from the training split that shares no image with the training samples or the evaluation set (1,000 items for VQAv2 and OK-VQA, and 2,000 images for COCO). For every run, we scored all saved STAVE epochs on this split, kept the epoch with the highest validation score, and reported only that epoch on the full evaluation set. VQA accuracy in %, COCO CIDEr:
The selection rule changes STAVE by at most 0.34 points. Even with STAVE chosen on the validation split and other methods still chosen on the evaluation set, STAVE outperforms or matches the strongest methods of Table 1 in every column.
(3) New: 16/32-shot ICL on the full evaluation sets. The paper scored these two rows on the first 1,000 items because of their prompt length. We now score them on the same queries as every other row (10,000 VQAv2, 5,046 OK-VQA and 5,000 COCO items) with the demos of the same three runs.
STAVE is above both ICL rows in every column, by 7.49 to 19.72 points over 32-shot ICL, with no demos in its prompt. In every column 32 demos score lower than 16, so adding demos does not close the gap. This is a weakness of ICL itself: beyond 16 demos, more demos lower its score instead of raising it, by 1.46 to 13.18 points across the six columns.
W2/Q2. PT trains for 500 steps on the first answer token, while STAVE can train for 4,000 steps on the full answer. Does STAVE still outperform PT with the full answer and a comparable training and tuning budget?
A2. Yes. Even with the full-answer loss and 4,000 steps, 4 times as many as STAVE uses on VizWiz and OK-VQA, all three PT modes stay below STAVE in every VQA cell. On Flowers, the only dataset where STAVE itself trains for 4,000 steps, STAVE is above every PT mode on Idefics2 and comparable to PT on Qwen-VL. Across all five Table 2 datasets, STAVE keeps the best average on both backbones. We also did more than tune PT's hyperparameters: PT is the only method whose learning rate was adjusted for individual runs, so its scores are in fact optimistic (see (3)). Points (2) to (5) continue in 2/2.
(1) Scope. The difference applies to Table 2 only. In Table 1, PT and STAVE already train with the same full-answer loss, for the same number of epochs, on the same 1,000 items, and PT's learning rate is tuned (Appendix F.2). There, STAVE is above all three PT modes in all six columns.