- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SIMIT constructs synthetic visual practice examples for test-time adaptation of unified multimodal models. The model generates questions, intended answers, and image descriptions, then produces images through native generation or a structured rendering skill library. Image verification, adaptive budget allocation, and confidence-based difficulty filtering determine the examples used for in-context learning or fine-tuning. Experiments cover 17 benchmarks in five domains, primarily with BAGEL-7B, plus ICL experiments with two additional models. The paper reports improvements over the base model and several inference-time and label-free adaptation baselines, and analyzes supervision quality and changes in prediction correctness.
Critical strengths
- A useful integration of visual synthesis and test-time adaptation. Combining native image generation with structured rendering extends self-generated practice to diverse visual inputs. Using the same model for synthesis, verification, and adaptation is a meaningful design constraint. The resulting system has practical value even where its individual components have precedents.
- Broad evaluation and useful diagnostic analyses. The experiments cover multiple adaptation modes and 17 benchmarks, with additional cross-model ICL results. Analyses of supervision correctness and the correction or disruption of predictions provide evidence about how the generated examples affect performance.
Critical weaknesses
- The experiments do not sufficiently isolate the value of the full pipeline. SIMIT combines several mechanisms, so establishing its contribution requires showing what they add over simpler self-generated practice. Table 1 lacks competitive same-backbone adaptations of Self-ICL or Analogical Prompting, while Table 3 tests only a restricted fixed-image alternative. The component ablations do not fully resolve this gap: diversity prompting, rendering skills, and verification are tested with adaptive budgeting and difficulty filtering disabled. This controls the synthesis budget, but leaves their effects within the complete system uncertain. The results establish that the pipeline can help, while providing weaker evidence for the necessity of its individual design choices.
- The concentration of gains leaves the proposed explanation incomplete. NoCaps, OK-VQA, and VizWiz account for most of BAGEL-ICL's gains, according to the reported values from Table 8; Appendix K also shows substantial variation across models. These patterns make it important to explain when and why synthetic practice helps. The supervision audit and error-correction analysis provide useful evidence of improvement, but do not explain this concentration or isolate answer-first synthesis as its cause. Consequently, the paper does not adequately distinguish gains in visual capabilities from adaptation to other factors, such as answer format, response length, or benchmark-specific response tendencies.
Questions for authors
- How does SIMIT compare with a simpler self-generated-demonstration approach under comparable compute? Within the full pipeline, what happens when only difficulty filtering is removed, given its negative effect when adaptive budgeting is disabled?
- What explains the concentration of improvements and their variation across models? Can the authors identify which capabilities or other response characteristics account for these changes?
- Existing methods such as VisionFoundry [1] construct questions and answers before image synthesis and verification. A brief discussion could clarify the relationship to SIMIT's query-conditioned, single-model adaptation setting.
[1] Guanyu Zhou et al. VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images. arXiv preprint arXiv:2604.09531v1. 2026.
5 comments
Weakness 1: Value of the full pipeline over simpler self-generated practice (R2.1, cont'd)
Full-system ablations (Q1, Table Q2). Tab. 2 removes DP, SL and V with ABA and DF disabled, which keeps the synthesis budget identical across variants. With ABA and DF enabled, a fair comparison would need a separate hyperparameter search for every variant, and each SIMIT-FT trial requires new synthesis and finetuning. In SIMIT-ICL, cached candidate pools make the full-system ablation of ABA and DF exact. DP and SL change the candidate pool itself, so ablating them in the full system requires a new candidate pool and a new hyperparameter search for every benchmark. Removing only DF, with the tuned ABA kept, turns the gain on the 14 benchmarks outside NoCaps, OK-VQA and VizWiz from +1.40% into −1.95%. Removing only ABA gives −1.07%, and removing both −5.29%. Both components are needed. Deciding where not to practice is the purpose of ABA. Practicing on every query hurts benchmarks where BAGEL is already confident, and ABA turns this loss into a gain while practicing on only 30.2% of queries. Where SIMIT does practice, the practice itself helps. On the 129 adapted queries of GQA, VQA-v2, AI2D, DocVQA and SEED-Bench, SIMIT-ICL gains +9.5 points (95% CI [+1.3, +17.7]).
Verification is active in the full system. The critic rejects at least 7% (ChartQA) to 46% (VizWiz) of triplet attempts, about 30k over the two candidate pools, and removing it lowers the finetuning gain from +4.35% to +3.08% and audited data quality from 81.8% to 75.4% (Tab. 2).
Table Q2. Full-system ablations of SIMIT-ICL (configuration selected per benchmark).
| ABA | DF | Mean | Mean excl. top-3 | Doc & Info | |
|---|---|---|---|---|---|
| Full SIMIT-ICL | ✓ | ✓ | +6.98% | +1.40% | +1.1% |
| Without DF | ✓ | ✗ | +4.22% | −1.95% | +0.6% |
| Without ABA (K = Kmax) | ✗ | ✓ | +4.95% | −1.07% | −2.0% |
| Without both (fixed K) | ✗ | ✗ | +1.48% | −5.29% | −4.3% |
NoCaps, OK-VQA and VizWiz select K = 4 with ABA and DF off, so their scores are identical in every row, and the mean excluding them is the informative column.
The negative DF effect in Tab. 2 came from a missing re-tuning (Table Q3). In Tab. 2, the row without ABA reused the DF band from joint ABA and DF tuning (the default band on 9 of 17 benchmarks) without re-tuning, which explains its drop. Only this row needed re-tuning. The full system was tuned jointly, and the other rows disable both ABA and DF, so they have no DF band to tune. Re-tuning only this row's DF band raises it from +3.27% to +5.14%, between the row without DF (+4.35%) and the full system (+6.39%). We re-tuned the five Image & Captioning benchmarks, where the drop occurred (5 to 8 bands each), and kept the other bands. At the final checkpoint the row gives +4.66%, so the ordering does not depend on checkpoint selection.
Table Q3. Revised rows of Tab. 2 (SIMIT-FT (TS), DP, SL and V enabled, best checkpoint).
| ABA | DF | Image (CIDEr) | Av. Gain |
|---|---|---|---|
| ✓ | ✓ | 1.14 (+5.8%) | +6.39% |
| ✗ | ✓, re-tuned | 1.13 (+4.8%) | +5.14% |
| ✗ | ✓, as in the paper | 1.07 (−0.6%) | +3.27% |
| ✗ | ✗ | 1.11 (+2.8%) | +4.35% |
Only the Image & Captioning benchmarks received new DF bands, so the other domains are unchanged from Tab. 2.
Answer-first construction is isolated on identical inputs in R2.2.
Weakness 2: What explains the concentration of gains (R2.2)
"The concentration of gains leaves the proposed explanation incomplete. NoCaps, OK-VQA, and VizWiz account for most of BAGEL-ICL's gains, according to the reported values from Table 8; Appendix K also shows substantial variation across models. These patterns make it important to explain when and why synthetic practice helps. The supervision audit and error-correction analysis provide useful evidence of improvement, but do not explain this concentration or isolate answer-first synthesis as its cause. Consequently, the paper does not adequately distinguish gains in visual capabilities from adaptation to other factors, such as answer format, response length, or benchmark-specific response tendencies."
Related Question 2
"What explains the concentration of improvements and their variation across models? Can the authors identify which capabilities or other response characteristics account for these changes?"
We agree, and we added App. O, which explains when synthetic practice helps. The following results answer the question.
-
Output-behavior calibration explains part of the concentration. Under the default lmms-eval prompts, which ask for "Unanswerable" when information is insufficient, zero-shot BAGEL abstains on 66% of answerable VizWiz questions and on 48% of OK-VQA questions. App. H, which reports the per-benchmark results, now states this calibration effect.
- VizWiz. SIMIT-ICL lowers abstention on answerable questions to 27% but keeps it at 76% on truly unanswerable ones, raising accuracy from 47.0 to 54.6 (+7.6, 95% CI [+3.8, +11.4]). Fixed rules cannot do this selectively. Banning abstention at decoding (37.7) or deleting the abstention sentence (42.2) both fall below zero-shot.
- OK-VQA. The abstention sentence is part of lmms-eval's default OK-VQA prompt, although OK-VQA has no unanswerable questions. SIMIT-ICL halves abstention to 20% without labels or prompt changes (37.6 to 48.5). Deleting the sentence reaches 61.6 but needs the oracle knowledge that no question is unanswerable, and the same deletion lowers VizWiz to 42.2.
- NoCaps. Response length does not explain the gain. SIMIT-ICL shortens captions from 14.0 to 9.8 words (references 11.4), while length prompts gain at most +0.035 CIDEr, against +0.407 for SIMIT-ICL. On COCO and Flickr30k, SIMIT does not change caption length.
This fits prior work, where demonstrations act largely through answer format and label space (Min et al., 2022) and multimodal ICL relies mostly on demo text (Baldassini et al., 2024).
-
SIMIT's gains go beyond calibration. A format, length or style shortcut would be carried by the demo text alone, and the 17 benchmarks reward different styles under lmms-eval's default prompts. Yet removing the images from SIMIT's demos loses 87% of the gain outside the three benchmarks (+1.40% to +0.18%), and text descriptions of the same images do not replace them (+0.06%; on NoCaps +35.9% instead of +53.9%). This gain is stable over three synthesis seeds (+1.34% ± 0.15). As a robustness check, we also recompute it without RefCOCO, the most variable benchmark. Images then still roughly double the gain (+1.27% vs. +0.67%) and help on 10 of 11 non-tied benchmarks. Self-generated text demos do not reproduce the gains even on the three benchmarks. Self-ICL changes NoCaps, OK-VQA and VizWiz by −3.8%, −3.8% and +2.8%, against +46.6%, +26.7% and +8.8% for SIMIT's own Q/A pairs without images and +53.9%, +28.9% and +16.1% with images. The gains are therefore not generic priming.
Weakness 1: Value of the full pipeline over simpler self-generated practice (R2.1)
Thank you for your constructive feedback.
Related Question 1
Simpler self-generated practice (Table Q1). We ran Self-ICL and Analogical Prompting with BAGEL-7B. To avoid a strawman, we used each paper's original prompts verbatim and changed only what multimodal input requires (attaching the image and extracting the answer). We evaluate each method on all queries, as published, and on exactly the queries and demo counts that SIMIT-ICL uses, which isolates the demonstration source.
Table Q1. Self-generated demonstrations for ICL (BAGEL-7B, 17 benchmarks, 500 queries each).
Top-3 = NoCaps, OK-VQA, VizWiz. Both SIMIT Q/A rows reuse SIMIT-ICL's retained demos and change only their content.
The multimodal extensions of these methods form a ladder that ends at SIMIT. Text exemplars (Analogical Prompting, Self-ICL) do not help. New Q/A pairs about the test image, without new images, lower the finetuning gain (−2.14% vs. +4.35%, Tab. 3). New images without selection help little in ICL (+1.48%, Table Q2). A one-call version that also draws its exemplars is not possible with current UMMs, because image generation runs in a separate decoding mode.