acceptodds
Under review as a conference paper at ICLR 2027

SIMIT: Self-Improving Vision-Language Models via Imagination at Test-Time

176 views

Abstract

We introduce SIMIT, a test-time self-improvement framework in which a vision-language model creates its own query-specific training data before answering an unlabeled test query. Our goal is to improve performance on a given target test set without annotations or external models. To pursue this goal, some test-time methods improve answers through longer reasoning or repeated sampling but leave the model unchanged. Others update the model using rewards based on majority agreement among multiple candidate answers or the model’s own judgments of correctness, but such rewards can reinforce existing mistakes. SIMIT instead imagines similar problems whose solutions it controls: a model need not correctly solve the target query to construct useful practice examples. It synthesizes diverse (question, answer, image) triplets by specifying each answer before realizing its image. To support diverse multimodal tasks, SIMIT combines native image generation with an extensible skill library for structured visuals, including documents, charts, and diagrams. Within this pipeline, multi-stage verification checks that images support their assigned answers, while difficulty filtering retains informative examples. Adaptive budgeting further targets this practice by allocating more samples to queries the model is less confident in answering. The resulting data can improve the model in-context (SIMIT-ICL) or in-weight (SIMIT-FT). Across 17 diverse benchmarks, SIMIT outperforms existing self-improvement methods. Its best configuration achieves a +7.20% mean relative gain over BAGEL-7B, while training-free SIMIT-ICL achieves +6.98%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLMs, test-time compute, test-time scaling, test-time adaptation, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 2 reviews, above 99% of the others.
Reviewer_QtEhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SIMIT constructs synthetic visual practice examples for test-time adaptation of unified multimodal models. The model generates questions, intended answers, and image descriptions, then produces images through native generation or a structured rendering skill library. Image verification, adaptive budget allocation, and confidence-based difficulty filtering determine the examples used for in-context learning or fine-tuning. Experiments cover 17 benchmarks in five domains, primarily with BAGEL-7B, plus ICL experiments with two additional models. The paper reports improvements over the base model and several inference-time and label-free adaptation baselines, and analyzes supervision quality and changes in prediction correctness.

Critical strengths
  1. A useful integration of visual synthesis and test-time adaptation. Combining native image generation with structured rendering extends self-generated practice to diverse visual inputs. Using the same model for synthesis, verification, and adaptation is a meaningful design constraint. The resulting system has practical value even where its individual components have precedents.
  2. Broad evaluation and useful diagnostic analyses. The experiments cover multiple adaptation modes and 17 benchmarks, with additional cross-model ICL results. Analyses of supervision correctness and the correction or disruption of predictions provide evidence about how the generated examples affect performance.
Critical weaknesses
  1. The experiments do not sufficiently isolate the value of the full pipeline. SIMIT combines several mechanisms, so establishing its contribution requires showing what they add over simpler self-generated practice. Table 1 lacks competitive same-backbone adaptations of Self-ICL or Analogical Prompting, while Table 3 tests only a restricted fixed-image alternative. The component ablations do not fully resolve this gap: diversity prompting, rendering skills, and verification are tested with adaptive budgeting and difficulty filtering disabled. This controls the synthesis budget, but leaves their effects within the complete system uncertain. The results establish that the pipeline can help, while providing weaker evidence for the necessity of its individual design choices.
  2. The concentration of gains leaves the proposed explanation incomplete. NoCaps, OK-VQA, and VizWiz account for most of BAGEL-ICL's gains, according to the reported values from Table 8; Appendix K also shows substantial variation across models. These patterns make it important to explain when and why synthetic practice helps. The supervision audit and error-correction analysis provide useful evidence of improvement, but do not explain this concentration or isolate answer-first synthesis as its cause. Consequently, the paper does not adequately distinguish gains in visual capabilities from adaptation to other factors, such as answer format, response length, or benchmark-specific response tendencies.
Questions for authors
  1. How does SIMIT compare with a simpler self-generated-demonstration approach under comparable compute? Within the full pipeline, what happens when only difficulty filtering is removed, given its negative effect when adaptive budgeting is disabled?
  2. What explains the concentration of improvements and their variation across models? Can the authors identify which capabilities or other response characteristics account for these changes?
  3. Existing methods such as VisionFoundry [1] construct questions and answers before image synthesis and verification. A brief discussion could clarify the relationship to SIMIT's query-conditioned, single-model adaptation setting.

[1] Guanyu Zhou et al. VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images. arXiv preprint arXiv:2604.09531v1. 2026.

5 comments
AuthorsOpenReview ↗

Weakness 1: Value of the full pipeline over simpler self-generated practice (R2.1)

Thank you for your constructive feedback.

"The experiments do not sufficiently isolate the value of the full pipeline. SIMIT combines several mechanisms, so establishing its contribution requires showing what they add over simpler self-generated practice. Table 1 lacks competitive same-backbone adaptations of Self-ICL or Analogical Prompting, while Table 3 tests only a restricted fixed-image alternative. The component ablations do not fully resolve this gap: diversity prompting, rendering skills, and verification are tested with adaptive budgeting and difficulty filtering disabled. This controls the synthesis budget, but leaves their effects within the complete system uncertain. The results establish that the pipeline can help, while providing weaker evidence for the necessity of its individual design choices."

Related Question 1

"How does SIMIT compare with a simpler self-generated-demonstration approach under comparable compute? Within the full pipeline, what happens when only difficulty filtering is removed, given its negative effect when adaptive budgeting is disabled?"

Simpler self-generated practice (Table Q1). We ran Self-ICL and Analogical Prompting with BAGEL-7B. To avoid a strawman, we used each paper's original prompts verbatim and changed only what multimodal input requires (attaching the image and extracting the answer). We evaluate each method on all queries, as published, and on exactly the queries and demo counts that SIMIT-ICL uses, which isolates the demonstration source.

  • Both fall below zero-shot. Analogical Prompting changes the mean by −22.0% and Self-ICL by −1.3%. Verbatim Self-ICL stays close to zero-shot because BAGEL often fails its first step, and it falls back to zero-shot on 21% to 100% of queries per benchmark. Forcing that step to produce pseudo-inputs cuts the fallback to at most 7.4%, yet the mean drops to −8.1% at 14.4× zero-shot. BAGEL recalls an exemplar in at most 5.2% of Analogical Prompting queries. Both papers note that their methods need strong instruction-following models (Yasunaga et al., 2024; Chen et al., 2023).
  • SIMIT's construction matters even without images. On the same queries, SIMIT's own Q/A pairs without images give +4.98%, against −0.2% for Self-ICL and −7.4% for Analogical Prompting. Adding the rendered images raises this to +6.98%, while text descriptions of the same images do not (+4.15%).
  • Compute does not explain the gap. The baselines cost 2.7× to 14.4× zero-shot, and Self-Consistency at a budget similar to SIMIT-ICL saturates (+0.17% at K = 16, +0.41% at K = 32 and 19×, Tab. 1). More compute does not help these baselines either. Forcing Self-ICL to produce pseudo-inputs raises its cost from 5.0× to 14.4× and lowers its mean from −1.3% to −8.1%.

Table Q1. Self-generated demonstrations for ICL (BAGEL-7B, 17 benchmarks, 500 queries each).

DemonstrationsMeanMedianMean excl. top-3Runtime
Zero-shot0.00%0.00%0.00%1×
All queries, original prompts
Analogical Prompting−21.99%−14.62%−21.92%9.3×
Self-ICL−1.30%−0.45%−1.23%5.0×
Self-ICL, forced pseudo-inputs−8.06%−3.29%−8.47%14.4×
Same queries and demo counts as SIMIT-ICL
Analogical Prompting−7.40%−0.85%−4.21%3.5×
Self-ICL−0.20%+0.00%+0.10%2.7×
Self-ICL, forced pseudo-inputs−1.42%+0.01%−0.42%5.6×
SIMIT Q/A, descriptions instead of images+4.15%+0.31%+0.06%
SIMIT Q/A, no images+4.98%+0.90%+0.18%
SIMIT-ICL+6.98%+1.30%+1.40%25×

Top-3 = NoCaps, OK-VQA, VizWiz. Both SIMIT Q/A rows reuse SIMIT-ICL's retained demos and change only their content.

The multimodal extensions of these methods form a ladder that ends at SIMIT. Text exemplars (Analogical Prompting, Self-ICL) do not help. New Q/A pairs about the test image, without new images, lower the finetuning gain (−2.14% vs. +4.35%, Tab. 3). New images without selection help little in ICL (+1.48%, Table Q2). A one-call version that also draws its exemplars is not possible with current UMMs, because image generation runs in a separate decoding mode.

AuthorsOpenReview ↗

Weakness 1: Value of the full pipeline over simpler self-generated practice (R2.1, cont'd)

Full-system ablations (Q1, Table Q2). Tab. 2 removes DP, SL and V with ABA and DF disabled, which keeps the synthesis budget identical across variants. With ABA and DF enabled, a fair comparison would need a separate hyperparameter search for every variant, and each SIMIT-FT trial requires new synthesis and finetuning. In SIMIT-ICL, cached candidate pools make the full-system ablation of ABA and DF exact. DP and SL change the candidate pool itself, so ablating them in the full system requires a new candidate pool and a new hyperparameter search for every benchmark. Removing only DF, with the tuned ABA kept, turns the gain on the 14 benchmarks outside NoCaps, OK-VQA and VizWiz from +1.40% into −1.95%. Removing only ABA gives −1.07%, and removing both −5.29%. Both components are needed. Deciding where not to practice is the purpose of ABA. Practicing on every query hurts benchmarks where BAGEL is already confident, and ABA turns this loss into a gain while practicing on only 30.2% of queries. Where SIMIT does practice, the practice itself helps. On the 129 adapted queries of GQA, VQA-v2, AI2D, DocVQA and SEED-Bench, SIMIT-ICL gains +9.5 points (95% CI [+1.3, +17.7]).

Verification is active in the full system. The critic rejects at least 7% (ChartQA) to 46% (VizWiz) of triplet attempts, about 30k over the two candidate pools, and removing it lowers the finetuning gain from +4.35% to +3.08% and audited data quality from 81.8% to 75.4% (Tab. 2).

Table Q2. Full-system ablations of SIMIT-ICL (configuration selected per benchmark).

ABADFMeanMean excl. top-3Doc & Info
Full SIMIT-ICL✓✓+6.98%+1.40%+1.1%
Without DF✓✗+4.22%−1.95%+0.6%
Without ABA (K = Kmax)✗✓+4.95%−1.07%−2.0%
Without both (fixed K)✗✗+1.48%−5.29%−4.3%

NoCaps, OK-VQA and VizWiz select K = 4 with ABA and DF off, so their scores are identical in every row, and the mean excluding them is the informative column.

The negative DF effect in Tab. 2 came from a missing re-tuning (Table Q3). In Tab. 2, the row without ABA reused the DF band from joint ABA and DF tuning (the default band on 9 of 17 benchmarks) without re-tuning, which explains its drop. Only this row needed re-tuning. The full system was tuned jointly, and the other rows disable both ABA and DF, so they have no DF band to tune. Re-tuning only this row's DF band raises it from +3.27% to +5.14%, between the row without DF (+4.35%) and the full system (+6.39%). We re-tuned the five Image & Captioning benchmarks, where the drop occurred (5 to 8 bands each), and kept the other bands. At the final checkpoint the row gives +4.66%, so the ordering does not depend on checkpoint selection.

Table Q3. Revised rows of Tab. 2 (SIMIT-FT (TS), DP, SL and V enabled, best checkpoint).

ABADFImage (CIDEr)Av. Gain
✓✓1.14 (+5.8%)+6.39%
✗✓, re-tuned1.13 (+4.8%)+5.14%
✗✓, as in the paper1.07 (−0.6%)+3.27%
✗✗1.11 (+2.8%)+4.35%

Only the Image & Captioning benchmarks received new DF bands, so the other domains are unchanged from Tab. 2.

Answer-first construction is isolated on identical inputs in R2.2.

AuthorsOpenReview ↗

Weakness 2: What explains the concentration of gains (R2.2)

"The concentration of gains leaves the proposed explanation incomplete. NoCaps, OK-VQA, and VizWiz account for most of BAGEL-ICL's gains, according to the reported values from Table 8; Appendix K also shows substantial variation across models. These patterns make it important to explain when and why synthetic practice helps. The supervision audit and error-correction analysis provide useful evidence of improvement, but do not explain this concentration or isolate answer-first synthesis as its cause. Consequently, the paper does not adequately distinguish gains in visual capabilities from adaptation to other factors, such as answer format, response length, or benchmark-specific response tendencies."

Related Question 2

"What explains the concentration of improvements and their variation across models? Can the authors identify which capabilities or other response characteristics account for these changes?"

We agree, and we added App. O, which explains when synthetic practice helps. The following results answer the question.

  1. Output-behavior calibration explains part of the concentration. Under the default lmms-eval prompts, which ask for "Unanswerable" when information is insufficient, zero-shot BAGEL abstains on 66% of answerable VizWiz questions and on 48% of OK-VQA questions. App. H, which reports the per-benchmark results, now states this calibration effect.

    • VizWiz. SIMIT-ICL lowers abstention on answerable questions to 27% but keeps it at 76% on truly unanswerable ones, raising accuracy from 47.0 to 54.6 (+7.6, 95% CI [+3.8, +11.4]). Fixed rules cannot do this selectively. Banning abstention at decoding (37.7) or deleting the abstention sentence (42.2) both fall below zero-shot.
    • OK-VQA. The abstention sentence is part of lmms-eval's default OK-VQA prompt, although OK-VQA has no unanswerable questions. SIMIT-ICL halves abstention to 20% without labels or prompt changes (37.6 to 48.5). Deleting the sentence reaches 61.6 but needs the oracle knowledge that no question is unanswerable, and the same deletion lowers VizWiz to 42.2.
    • NoCaps. Response length does not explain the gain. SIMIT-ICL shortens captions from 14.0 to 9.8 words (references 11.4), while length prompts gain at most +0.035 CIDEr, against +0.407 for SIMIT-ICL. On COCO and Flickr30k, SIMIT does not change caption length.

    This fits prior work, where demonstrations act largely through answer format and label space (Min et al., 2022) and multimodal ICL relies mostly on demo text (Baldassini et al., 2024).

  2. SIMIT's gains go beyond calibration. A format, length or style shortcut would be carried by the demo text alone, and the 17 benchmarks reward different styles under lmms-eval's default prompts. Yet removing the images from SIMIT's demos loses 87% of the gain outside the three benchmarks (+1.40% to +0.18%), and text descriptions of the same images do not replace them (+0.06%; on NoCaps +35.9% instead of +53.9%). This gain is stable over three synthesis seeds (+1.34% ± 0.15). As a robustness check, we also recompute it without RefCOCO, the most variable benchmark. Images then still roughly double the gain (+1.27% vs. +0.67%) and help on 10 of 11 non-tied benchmarks. Self-generated text demos do not reproduce the gains even on the three benchmarks. Self-ICL changes NoCaps, OK-VQA and VizWiz by −3.8%, −3.8% and +2.8%, against +46.6%, +26.7% and +8.8% for SIMIT's own Q/A pairs without images and +53.9%, +28.9% and +16.1% with images. The gains are therefore not generic priming.

Reviewer_LZEEOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SIMIT is a test-time self-improvement method for VLMs that uses no labels or external models. For each unlabeled test query, a unified multimodal model (BAGEL-7B) writes new (question, answer, image description) triplets, with the answer fixed first. It then renders each image, either with its own image generator or through 16 deterministic rendering "skills" for structured visuals (charts, diagrams, documents, geometry). The model verifies the images itself, keeps samples within a confidence band, and sets the number of samples from its zero-shot confidence. The samples serve as in-context demos (SIMIT-ICL) or as LoRA data, either per query (FT-SQ) or for the whole test set (FT-TS). On 17 LMMs-Eval-Lite benchmarks, method gains over zero-shot.The paper also provides ablations, comparisons with real labeled data, transfer to other benchmarks and to two other UMMs, and a theoretical argument that consensus rewards cannot fix a wrong initial majority. Overall, this is a good paper so I will recommand it.

Critical strengths
  • Paper introduces a interesting source of label-free supervision, realized as a working system across visual types.
  • The transparent empirical study is board and fair.
Critical weaknesses

These are why my score is not higher because the experiments part but neither is fatal.

  • The mechanism behind the gains is not isolated, and parts of the paper's own evidence point to simpler explanations than practising on imagined similar problems.
    • About 83% of SIMIT-ICL's summed per-benchmark gain comes from NoCaps, OK-VQA and VizWiz, and the median gain is about 1.3% (Tab. 8). On these benchmarks the base model seems limited by its output behaviour, e.g., over-abstention on VizWiz. On Doc & Infographic, where rendering should matter most, SIMIT ties TTRV (Tab. 1).
    • Many demos are generic. In Fig. 17, two identical counting questions flip "Unanswerable" to a correct answer, which suggests format or anti-abstention priming. Other demos restate the test question with the model's own answer (Figs. 18, 20, 24), which contradicts the claim that the model need not solve the target query.
    • Key controls are missing: demos from another query, one fixed set of demos for all queries, and text-only demos (Self-ICL style). All ablations use FT-TS, so for SIMIT-ICL there is no evidence that image generation is needed.
    • Sec. 4.4 compares labels for self-chosen synthetic questions with pseudo-labels for real test queries, so SIMIT's low error rate holds largely by construction.
  • The evaluation protocol weakens the label-free and consistent-improvement claims.
    • Seven synthesis hyperparameters plus LoRA settings are tuned per benchmark and mode on 50 labeled examples (App. G). This tuning can switch the method off where it does not help (TextCaps is exactly unchanged), so "without annotations" needs qualifying. A baseline that uses those 50 labels directly is missing.
    • All results are single runs on 500-example subsets, with no seeds or confidence intervals, and many per-benchmark gains are only 0–2 points. Per-benchmark numbers for the baselines are not reported.
Questions for authors
  1. Results for SIMIT-ICL, per benchmark, with (a) demos synthesized for another query, (b) one fixed set of demos for all queries, and (c) the same Q/A pairs without images.
  2. How often do retained demos restate the test question, or share SIMIT's final answer? What happens if they are removed?
  3. For VizWiz, OK-VQA and NoCaps: the "Unanswerable" rate before and after, VizWiz accuracy on answerable vs. unanswerable questions, and caption lengths.
  4. Results with one configuration shared by all benchmarks, the number of tuning trials per study, and a baseline that uses the 50 validation labels as demos or LoRA data.
  5. Mean ± std over at least three seeds, and per-benchmark results for all baselines.
9 comments
AuthorsOpenReview ↗

Weakness 1: Concentration of gains, output behavior, and Doc & Infographic (R1.1)

Thank you for your valuable feedback, and for proposing concrete control experiments.

"The mechanism behind the gains is not isolated, and parts of the paper's own evidence point to simpler explanations than practising on imagined similar problems. About 83% of SIMIT-ICL's summed per-benchmark gain comes from NoCaps, OK-VQA and VizWiz, and the median gain is about 1.3% (Tab. 8). On these benchmarks the base model seems limited by its output behaviour, e.g., over-abstention on VizWiz. On Doc & Infographic, where rendering should matter most, SIMIT ties TTRV (Tab. 1)."

Related Question 3

"For VizWiz, OK-VQA and NoCaps: the "Unanswerable" rate before and after, VizWiz accuracy on answerable vs. unanswerable questions, and caption lengths."

We agree that the mean hides this concentration. Robust aggregates and output-behavior measurements explain it. We added this statement to the per-benchmark results in App. H.

"On NoCaps, OK-VQA and VizWiz, much of the gain reflects calibration of output behavior such as abstention, which SIMIT learns without labels (App. O.1)."

Robust aggregates (Table L1). The mean relative gain weights low baselines heavily (NoCaps CIDEr 0.755). On the other 14 benchmarks, SIMIT-ICL improves 13 and ties 1 (mean +1.40%, median +1.15%; +1.34% ± 0.15 over three synthesis seeds), and SIMIT-FT (TS) reaches +2.40%, twice the best LFRL baseline on the same 14 benchmarks (TTRV, +1.20%). Across all 17 benchmarks, every SIMIT variant has a higher median than every baseline (+0.93% to +2.11%, against at most +0.78%), and every baseline falls below zero-shot on at least 3 benchmarks. The small gains outside the three benchmarks reflect selectivity rather than weak practice. SIMIT-ICL shows demos on only 30.2% of all queries (2,569 of 8,500), and on the 129 adapted queries of GQA, VQA-v2, AI2D, DocVQA and SEED-Bench it gains +9.5 points (95% CI [+1.3, +17.7], paired bootstrap stratified by benchmark).

Table L1. Aggregates for every method (paper runs).

MethodMeanMedianMean excl. top-3Improved / tied / worse
Thinking (1,000 tokens)−21.59%−9.79%−23.78%1 / 0 / 16
Self-Refine (1 iteration)−2.90%−2.23%−2.45%5 / 1 / 11
Self-Consistency (K = 32)+0.41%+0.25%+0.57%10 / 0 / 7
TTRL+0.59%+0.23%+0.63%10 / 1 / 6
EMPO+1.82%+0.59%+0.99%9 / 1 / 7
TTRV+2.07%+0.78%+1.20%13 / 1 / 3
SRLM+0.87%−0.12%−0.29%6 / 2 / 9
SIMIT-ICL+6.98%+1.30%+1.40%16 / 1 / 0
SIMIT-FT (SQ)+1.69%+0.93%+1.11%16 / 1 / 0
SIMIT-FT (TS)+6.39%+1.81%+2.40%16 / 1 / 0
SIMIT-FT (TS)→ICL+7.20%+2.11%+2.80%16 / 1 / 0

Top-3 = NoCaps, OK-VQA, VizWiz.

Output behavior (Q3, Table L2). The prompts are the lmms-eval defaults and identical for every method. On VizWiz and OK-VQA they ask the model to answer "Unanswerable" when information is insufficient. To test whether a fixed rule achieves the same effect, we also banned "Unanswerable" at decoding and deleted the abstention sentence from the prompt.

VizWiz. SIMIT corrects over-abstention selectively. Zero-shot BAGEL abstains on 70% of questions, including 66% of answerable ones. SIMIT-ICL lowers abstention on answerable questions to 27% but still abstains on 76% of truly unanswerable ones, so answerable accuracy rises from 36.7 to 48.7 while unanswerable accuracy stays high. Both fixed rules remove abstention almost everywhere and fall below zero-shot, whereas SIMIT-ICL reaches 54.6 (+7.6, 95% CI [+3.8, +11.4]). SIMIT learns when to answer from unlabeled data.

Table L2a. VizWiz (500 queries, 18% of them unanswerable).

MethodAccuracyAbstains, allAbstains, answerableAbstains, unanswerableAccuracy, answerableAccuracy, unanswerable
Zero-shot47.070.4%66.1%90.0%36.793.8
Ban "Unanswerable" at decoding37.70.0%0.0%0.0%43.710.3
Delete the abstention sentence42.24.4%2.7%12.2%46.821.2
SIMIT-ICL54.635.4%26.6%75.6%48.781.4

An ideal model abstains on 0% of answerable and 100% of unanswerable questions. Unanswerable questions are defined by the dataset's answer type.

AuthorsOpenReview ↗

Weakness 1: Concentration of gains, output behavior, and Doc & Infographic (R1.1, cont'd)

OK-VQA. SIMIT halves a prompt-induced abstention without labels. The abstention sentence is part of lmms-eval's default OK-VQA prompt, which we kept unchanged for every method, so this headroom comes from the standard benchmark prompt rather than from our setup. OK-VQA has no unanswerable questions, so every abstention is wrong, yet zero-shot BAGEL abstains on 48% of them. SIMIT-ICL halves this to 20% (+10.9, CI [+7.3, +14.5]) without any change to the prompt. The fixed rules score higher here only because they use the oracle knowledge that no question is unanswerable. Where abstention is sometimes correct, the same rules hurt. Deleting the sentence lowers VizWiz from 47.0 to 42.2, and banning abstention lowers it to 37.7, whereas SIMIT-ICL improves both benchmarks with the same prompt.

Table L2b. OK-VQA (500 queries, none unanswerable).

MethodAccuracyAbstains
Zero-shot37.648.2%
Ban "Unanswerable" at decoding (oracle)58.40.0%
Delete the abstention sentence (oracle)61.60.0%
SIMIT-ICL48.520.4%

NoCaps. The gain is not a length effect. NoCaps is a captioning task, so its prompt has no abstention option, and the relevant output behavior is caption length. SIMIT-ICL shortens captions from 14.0 to 9.8 words, closer to the 11.4-word references, but prompting for shorter captions gains at most +0.035 CIDEr, against +0.407 for SIMIT-ICL. On COCO and Flickr30k, SIMIT barely changes caption length.

Table L2c. Caption length and CIDEr gain over zero-shot.

BenchmarkWords, zero-shotWords, SIMIT-ICLWords, referencesCIDEr gain, best length promptCIDEr gain, SIMIT-ICL
NoCaps14.09.811.4+0.035+0.407
COCO10.110.110.6−0.049+0.005
Flickr30k10.19.812.8+0.064+0.009

Length prompts are "Provide a short one-sentence caption" and "Provide a one-sentence caption of about N words", with N the mean reference length on validation (oracle information). The better of the two is shown, which on Flickr30k is the oracle-length prompt.

These benchmarks are style-sensitive, and SIMIT's gains go beyond style. We evaluate all 17 benchmarks with lmms-eval's default prompts and metrics, identical for every method, and these benchmarks reward different output styles. In LMMs-Eval's own results, LLaVA-NeXT-34B scores 7 to 31 points below LLaVA-1.5-7B on NoCaps, OK-VQA and TextCaps while outperforming it on DocVQA, GQA and TextVQA (Zhang et al., 2025a), so large gains on the former are expected from any method that adapts output behavior. A style shortcut, however, would be carried by the demo text alone. Yet the same Q/A pairs without images lose most of the gain outside NoCaps, OK-VQA and VizWiz (+1.40% to +0.18%, R1.3), and self-generated text demos lower the mean (Self-ICL −1.3%). Across benchmarks with such different preferred styles, SIMIT-ICL is significantly above zero-shot on 7 and never significantly below it.

Doc & Infographic. This is where query-specific, rendered practice matters most. We agree that SIMIT's margin is smallest here. Yet demos of another query (−0.4%), one fixed demo set (+0.2%) and the same Q/A pairs without images (+0.4%) all fall below SIMIT-ICL (+1.1%; Table L3 in R1.3). Without the skill library, finetuning on synthetic data lowers Doc & Info by 2.2% (Tab. 2), the largest drop of any domain. The remaining gap reflects headroom (DocVQA is at 93.2 ANLS zero-shot). TTRV also uses more access, training on the whole test set with 32 rollouts per query, whereas SIMIT-ICL adapts each query in isolation. In the same test-set setting, SIMIT-FT→ICL reaches 79.7 versus 79.6 for TTRV.

AuthorsOpenReview ↗

Weakness 2: Generic and restated demonstrations (R1.2)

"Many demos are generic. In Fig. 17, two identical counting questions flip "Unanswerable" to a correct answer, which suggests format or anti-abstention priming. Other demos restate the test question with the model's own answer (Figs. 18, 20, 24), which contradicts the claim that the model need not solve the target query."

Related Question 2

"How often do retained demos restate the test question, or share SIMIT's final answer? What happens if they are removed?"

Restated demos are rare and do not drive the gains. Figs. 15 to 25 show selected examples of a compact variant (ABA and DF disabled, K = 2; App. N), not the demos that the full pipeline retains. We count a demo as restating the test question if the two questions match exactly or have a question-embedding cosine similarity of at least 0.85. Over all 7,176 retained demos, at most 1.3% restate the test question on any benchmark (the maximum is on VizWiz), affecting at most 3.8% of queries. The demo answer equals SIMIT's final answer for at most 11.2% of queries (VizWiz), and at most 4.8% of demos duplicate another demo of the same query. Removing all restated demos leaves the gain unchanged (+6.90% vs. +6.98%, median +1.24% vs. +1.30%; per benchmark in Table L3, column e).

Generic demos. A demo that answers instead of abstaining can flip an "Unanswerable", and we quantify this as abstention calibration in R1.1. Generic content does not explain the rest of the gain, since the same Q/A pairs without images lose 87% of it outside NoCaps, OK-VQA and VizWiz (R1.3).

Similar demos are expected by design. Useful practice is correct, relevant to the test query, and not redundant. Relevance pulls demos toward the test query while diversity prompting pushes them apart, so demos that resemble the test question are expected rather than a failure. For "What breed is this dog?", useful practice contrasts breeds the model may confuse, each labeled correctly by construction. Such a set is informative whichever breed the test image shows, so the model does not need to solve the test query to build it. This agrees with work on demonstration selection, where demos help most when they are similar to the test input but not redundant with each other (Liu et al., 2022; Levy et al., 2023; Ye et al., 2023), and on test-time finetuning, where nearest neighbours alone are redundant (Hübotter et al., 2025). Restated demos are the rare case where relevance overrides diversity, and identical demos are failures of diversity.

"Need not solve the target query." The claim concerns the source of supervision. Answers are fixed for constructed questions rather than inferred for the test query. The restated cases in Figs. 18, 20 and 24 are informative. BAGEL answers "Unanswerable" as a solver, yet names the angel, the dog and the French bulldog when writing demos. It holds knowledge that it withholds under the abstention instruction, and the demos bring it back.

References

  • Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of LLMs. In ICLR, 2025.
  • Itay Levy, Ben Bogin, and Jonathan Berant. Diverse demonstrations improve in-context compositional generalization. In ACL, 2023.
  • Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In DeeLIO Workshop at ACL, 2022.
  • Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exemplars for in-context learning. In ICML, 2023.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of New Results and Revisions

We thank the reviewers and the AC. Both reviewers found SIMIT's label-free supervision interesting and its evaluation broad, and both asked the same central question. What drives SIMIT's gains, and would simpler self-generated practice do as well? We answered with new controls and baselines on all 17 benchmarks, under the same lmms-eval protocol as the paper.

New experiments.

  • Controls for SIMIT-ICL (LZEE). Demos synthesized for another query, one fixed demo set for all queries, the same Q/A pairs without images, text descriptions instead of images, and restated demos removed.
  • Simpler self-generated baselines (QtEh). Analogical Prompting and Self-ICL with their original prompts on the same backbone.
  • Answer-first vs. answer-after labels (LZEE, QtEh). SIMIT's synthetic inputs relabeled with BAGEL's own answers, with a blind audit of both label sources.
  • Output behavior (LZEE, QtEh). Abstention rates, answerable vs. unanswerable accuracy, caption lengths, and prompt and decoding controls.
  • Tuning dependence (LZEE). One configuration shared by all benchmarks, trial counts, and baselines that use the 50 validation labels as demonstrations or finetuning data.
  • Full-system ablations (QtEh). DF and ABA removed inside the full SIMIT-ICL, and the finetuning ablation row without ABA with its DF band re-tuned.
  • Statistics (LZEE). Three synthesis seeds, paired bootstrap confidence intervals, and median and trimmed means for every method.

Main findings.

  1. Rendered images matter. Removing the images from SIMIT's demos nearly halves the VizWiz gain (+16.1% to +8.8%) and removes most of the gain outside NoCaps, OK-VQA and VizWiz. Replacing them with text descriptions lowers the NoCaps gain from +53.9% to +35.9%.
  2. Simpler self-generated practice does not work. With their original prompts, Analogical Prompting and Self-ICL both perform worse than zero-shot BAGEL, by 22.0% and 1.3% on average across the 17 benchmarks. Both generate text-only demos. Used the same way, as text-only demos without their images, SIMIT's own Q/A pairs give +4.98% on average, and the full method with rendered images +6.98%.
  3. Answer-first labels help most where the model's knowledge is weakest. Relabeling the same synthetic inputs with BAGEL's own answers lowers the mean gain from +6.98% to +5.75% and the OK-VQA gain from +28.9% to +9.5%.
  4. The concentration of gains is explained. On VizWiz, SIMIT corrects BAGEL's over-abstention selectively and without labels, which no fixed prompt or decoding rule achieves. The NoCaps gain is not a length effect. Restated demos are rare (at most 1.3%), and removing them leaves the gain unchanged.
  5. Each component helps in the full system. Removing DF or ABA from the full SIMIT-ICL turns the gain outside NoCaps, OK-VQA and VizWiz negative. Re-tuning the DF band of the finetuning ablation row without ABA raises it from +3.27% to +5.14%, between the full system (+6.39%) and the row without DF (+4.35%).
  6. SIMIT's label-free practice is more reliable than the 50 calibration labels. Used directly as demonstrations, these labels improve only 6 to 9 of 17 benchmarks in every draw and lower AI2D and Doc & Info, whereas SIMIT-ICL improves 16. Finetuning on them lowers 5 benchmarks, whereas SIMIT-FT (TS) lowers none.
  7. The gains are robust. Over three synthesis seeds, SIMIT-ICL's mean gain is +6.87% ± 0.20, and +1.34% ± 0.15 outside NoCaps, OK-VQA and VizWiz. SIMIT-ICL is significantly above zero-shot on 7 benchmarks, and no SIMIT variant is significantly below zero-shot on any benchmark. Even without per-benchmark synthesis hyperparameters, SIMIT remains the best method with single-query and with test-set access.

Revisions in the updated PDF. Table, figure and appendix numbers in our responses refer to this version.

  • Added App. O with the new analyses of output behavior, controls, simpler self-generated baselines, answer-first labels, full-system ablations and calibration.
  • Added a paragraph to Sec. 4.4 that separates what SIMIT practices on from how it labels (App. O.3).
  • Stated in App. H, with the per-benchmark results, that much of the gain on NoCaps, OK-VQA and VizWiz reflects calibration of output behavior.
  • Updated the finetuning ablation (Tab. 2) with the re-tuned row.
  • Moved cross-dataset and cross-model generalization from the appendix into the main text (Sec. 4.5 and 4.6).
  • Added hyperparameter-free self-improvement as future work in App. P, and discussed VisionFoundry in App. K.
AuthorsOpenReview ↗

Summary of Discussions

We thank the reviewers for their constructive feedback, which has substantially strengthened our work. Our paper received two reviews, and both recommend acceptance. They recognized SIMIT's core idea as "an interesting source of label-free supervision, realized as a working system across visual types" (LZEE) and as "a useful integration of visual synthesis and test-time adaptation" with practical value (QtEh). For each test query, the model imagines similar practice problems and fixes each answer before generating the image, so its supervision does not depend on solving the query correctly. Reviewers highlighted that using one model for synthesis, verification and adaptation is "a meaningful design constraint" (QtEh), and praised the broad, transparent and fair evaluation on 17 benchmarks with useful diagnostic analyses (LZEE, QtEh). Both found the paper clear and easy to follow.

After the rebuttal, Reviewer LZEE stated that all their concerns were addressed, called SIMIT "an excellent paper", and raised their score from Weak to Clear acceptance (3->4). Reviewer QtEh, who also recommends acceptance (3), raised questions that closely overlap with LZEE's. We ran every experiment QtEh requested and hope the new results address their questions as well.

Both reviewers mainly asked whether simpler explanations could account for the gains. Our new experiments on all 17 benchmarks show that simpler self-generated practice performs worse than the base model, while SIMIT's gains rely on its generated images and are robust across seeds.

Overall, the discussion confirms SIMIT as a novel and practical approach to test-time self-improvement, with a 7.2% mean relative gain across 17 benchmarks and improvements on all three tested models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.