acceptodds
Under review as a conference paper at ICLR 2027

Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks

12 views

Abstract

Visual in-context learning has been proposed as a pathway towards dynamic models that can generate predictions based on a provided context and thereby can adapt to new vision tasks at test-time. Yet, the evaluation of the adaptation capabilities of these models has been limited to narrow setups that mainly mirror tasks or image domains from pre-training for which real adaptation is not required. We address this gap by constructing a broad Visual In-Context BEnchmark (VIBE) with a focus on diverse imaging domains and a wide range of tasks. With this, we are able to get a much clearer picture of the adaptive capabilities of visual in-context models when faced with new image- and task distributions. We stress test seven models on datasets and tasks (in total, we explore dataset-task combinations) and compare them under a unified, reproducible evaluation protocol, in an one-shot setting. Our evaluation uncovers key insights on the state of visual in-context learning, including limitations, systematic failure modes and promising directions. To foster broader evaluation, we openly release our VIBE toolkit upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Computer Vision, Benchmarking, Visual in-Context Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TzYMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents VIBE, a Visual In-Context BEnchmark and toolkit with a unified, reproducible evaluation protocol for visual in-context learning (VICL). Every task is posed as a one-shot visual analogy: given a context pair (Cin, Cout) and a query image Q, the model has to predict the output for Q. The benchmark covers 12 tasks in three groups (Perception & Localization, Restoration & Enhancement, Manipulation & Transformation), 14 datasets and 6 imaging domains, giving 106 dataset-task combinations. The protocol fixes three things:

  1. Task encodings: RGB renderings of segmentation masks, depth maps, edges, keypoints and boxes.
  2. Context selection: a class-aware greedy policy for tasks with semantic classes, and fixed-seed random selection otherwise.
  3. Post-processing: a shared decoding step before metrics, such as color-aware mIoU for segmentation.

Seven VICL models (MAE-VQGAN, Painter, LVM, DeLVM, PromptDiffusion, SD-VICL and Analogist) are stress tested against a Copy target baseline and combined into a rank-aggregated score. The main findings are:

  1. Most models are multi-task models instead of in-context learners. They learn a switch that steers them to a pre-training task and keep their pre-training color maps.
  2. SD-VICL and MAE-VQGAN predict more contextually, which the authors link to correspondence.
  3. No model handles rotation or flipping.
  4. Contextual tasks are inherently ill-posed, which calls for context-sensitive metrics.
Critical strengths
  1. The paper gives a unified protocol across many models, tasks and domains, putting 7 VICL models on the same 106 dataset-task combinations with fixed context sets, the same task encodings and the same post-processing. In my opinion, it gives the field a reasonable comparison since each model was so far evaluated on its own tasks and mostly on images close to its training data. The Copy target baseline is a simple addition, because several models fall to or below it, and PromptDiffusion ranks below it overall. This is the main reason I think the benchmark is worth publishing in some form.

  2. The paper shows convincingly that most models fall back to tasks and color maps from pre-training instead of following the context. The Painter example in Figure 7 is especially clear i.e., when prompted with colorization on NYU-Depth V2, it still produces its gray depth maps. The discussion of inherently contextual vision tasks, with the wolf and bison colorization example in Table 5, makes a good case that ground-truth metrics are ill-posed in VICL, and the field could take this on.

Critical weaknesses
  1. All results use a single context pair per query in the one-shot setting. VICL performance is known to depend strongly on which context pair is shown (Zhang et al., NeurIPS 2023; Sun et al., 2023), but the paper reports no variation over context pairs. So the gaps between models, the ranking in Table 4, and the claim that most models only learn a switch might change if different context pairs were used. They already store a list of context samples per query for few-shot prompting, so this seems feasible to report.

  2. The main confounds are argued in Appendix A.1 but they are not measured. The authors acknowledge that models would likely score better with their own task encodings, and that some evaluation datasets were part of pre-training. They then state that "in practice, this effect does not take shape", but give no numbers/ reasons why this happens. The claim of most models being multi-task models, and not in-context learners, builds on this point. Two experiments would establish it: A] an ablation with each model's native task encodings, or the mIoU with Hungarian matching that A.1 mentions; B] results on the dataset-task combinations not seen by any models in pre-training. C] Also, the model set is incomplete. IMProv and Visual Token Matching are discussed in Section 2 but not evaluated, and SegGPT is not mentioned at all. That weakens the claim to stress test the most prominent VICL models.

  3. The related work could briefly discuss multitask dense prediction transformers and adapter-based methods for generalising to new tasks and domains, as an alternative to in-context adaptation. Currently, it is missing from the paper.

Questions for authors

Questions whose answers could change my assessment:

  1. How much do the scores and the ranking in Table 4 change if each query is evaluated with several different context pairs, for example five draws from the stored context lists, or with context pairs retrieved by similarity to the query?
  2. Can you report results with each model's native task encodings, at least for segmentation and depth, and the segmentation scores with Hungarian matching that Appendix A.1 refers to? This would tell me whether the failures come from the task or from the encoding.
  3. Which of the 14 datasets appear in each model's pre-training data, and do the conclusions hold on the dataset-task combinations that are unseen for all models?
  4. How was the SD-VICL re-implementation validated against the original method? Does it reproduce the numbers reported by Oorloff et al. (2025)?
  5. Why were IMProv, Visual Token Matching and SegGPT not evaluated? For Analogist, does replacing the original VLM with Qwen3-VL-8B-Instruct change its results?
  6. Section 4.1.3 says SD-VICL is one of the only models that start to address object removal. In Table 3, however, it has the second-lowest PSNR (12.19) and SSIM (0.36) among the models, and four models have better LPIPS. Which one is correct?
  7. Section 4.1.2 explains SD-VICL's denoising by "the use of a quantized codebook (Rombach et al., 2022)", but Table 8 lists the Stable Diffusion models as VAE-KL, which is a continuous autoencoder, and not a codebook as far as I know. Please correct or justify this.
  8. Minor: Tables 37 to 39 are titled "Style Transfer" but appear to report modality transfer.

As suggestions for future work, which do not affect my score, I would like to see results in the few-shot setting. I would also like to see the context-sensitive metrics and human evaluation protocols that the authors propose in Section 5.

11 comments
AuthorsOpenReview ↗

Response to Reviewer TzYM (1/6)

We thank the reviewer for the detailed and constructive review, and for finding the benchmark worth publishing and the evidence on pretraining fallback convincing.

W1: Single context pair per query

All results use a single context pair per query in the one-shot setting. VICL performance is known to depend strongly on which context pair is shown (Zhang et al., NeurIPS 2023; Sun et al., 2023), but the paper reports no variation over context pairs. So the gaps between models, the ranking in Table 4, and the claim that most models only learn a switch might change if different context pairs were used. They already store a list of context samples per query for few-shot prompting, so this seems feasible to report.

Response: We agree that context choice matters and added an evaluation with context pairs retrieved by visual similarity to the query (Zhang et al., NeurIPS 2023) on 23 task-dataset combinations covering all task groups, for all models (Appendix A.13, Table 60). The results support our conclusions:

  1. Better context helps only models that use it. Retrieved contexts improve SD-VICL and Analogist on all 23 combinations, MAE-VQGAN on 21 and IMProv on 20, often by large margins. For LVM, DeLVM and Painter, they help on only 12–13 of 23 combinations, i.e., close to chance, and the changes are small in either direction. That a more informative context barely affects these models directly supports our claim that they learn a task switch rather than adapting to the context.
  2. Retrieval partly makes the task easier. The Copy-target baseline also improves on all combinations, as retrieved context outputs resemble the query's target. Gains under retrieval should therefore be read relative to this baseline.
  3. The ranking is stable. We computed the model ranking as in Table 5, but restricted to the 23 task-dataset combinations, once with VIBE's standard context selection and once with the retrieval method of Zhang et al. (2023), and compared the two. SD-VICL and Painter remain first and second under both context selections, and the ranking of the eight models (excluding the Copy-target baseline) agrees with Kendall's τ = 0.86 (p < 0.01; 0.72 including the baseline). The only changes among models are two swaps of neighbouring ranks (LVM/MAE-VQGAN and IMProv/Analogist), while the Copy-target baseline moves up from eighth to fifth place, consistent with point 2.

W2: Confounds argued but not measured

The main confounds are argued in Appendix A.1 but they are not measured. The authors acknowledge that models would likely score better with their own task encodings, and that some evaluation datasets were part of pre-training. They then state that "in practice, this effect does not take shape", but give no numbers/ reasons why this happens. The claim of most models being multi-task models, and not in-context learners, builds on this point. Two experiments would establish it: A] an ablation with each model's native task encodings, or the mIoU with Hungarian matching that A.1 mentions; B] results on the dataset-task combinations not seen by any models in pre-training. C] Also, the model set is incomplete. IMProv and Visual Token Matching are discussed in Section 2 but not evaluated, and SegGPT is not mentioned at all. That weakens the claim to stress test the most prominent VICL models.

AuthorsOpenReview ↗

Response to Reviewer TzYM (2/6)

A] Native task encodings / Hungarian matching

Response: We added both experiments (Appendix A.16, Tables 64 and 65).

What the two evaluations measure. VIBE evaluates in-context learning, i.e., whether a model adapts to a task as defined by the context. The output encoding shown in the context (e.g., the class palette or the depth color map) is part of that task definition. A model that solves a task only in the encoding it was trained with reproduces a task learned in its weights, as any supervised multi-task model would, but does not adapt to the context. VIBE's standard metrics therefore require the demonstrated encoding. The encoding-invariant evaluation answers a different question: does the model solve the task in any encoding? We use it as a diagnostic to separate the two, not as a replacement, and keep the standard evaluation for all rankings.

Encoding-invariant evaluation (segmentation, edges, keypoints). The encoding-invariant metrics replace the palette- or color-based post-processing of the standard metrics with a clustering of the predicted colors. For segmentation, we cluster the predicted pixel colors into k groups with k-means, where k is the number of classes in the query's ground truth, and match clusters to classes one-to-one by Hungarian matching on IoU (Hungarian mIoU). For edges and keypoints, we cluster into two groups and evaluate the one with the higher F1 as foreground, which makes the metric invariant to the color of edges and keypoints and to whether they are darker or lighter than the background. Selected results (mIoU / F1, Std. → Inv.; all models in Appendix A.16, Table 64):

ModelSUIM seg.DRAM seg.DeepCrack edgesOTP keypoints
SD-VICL0.292 → 0.4140.415 → 0.4630.625 → 0.6740.073 → 0.108
Painter0.171 → 0.4650.171 → 0.6620.019 → 0.1150.019 → 0.119
LVM0.156 → 0.4380.247 → 0.4880.082 → 0.1510.087 → 0.122
DeLVM0.213 → 0.4450.253 → 0.4110.007 → 0.0710.031 → 0.098
MAE-VQGAN0.225 → 0.3590.411 → 0.4670.289 → 0.3120.026 → 0.041
Copy target0.203 → 0.2970.337 → 0.3640.041 → 0.0770.012 → 0.019

The results confirm the reviewer's intuition and make our claim measurable. Painter falls below the Copy-target baseline on segmentation under the standard evaluation, yet achieves the best encoding-invariant scores of all models: it solves segmentation, but in its own training encoding rather than the one demonstrated. LVM and DeLVM show the same pattern to a lesser degree. SD-VICL, MAE-VQGAN and IMProv already follow the demonstrated encoding, so their scores change less, and SD-VICL remains the strongest model on edge detection under both evaluations. The Copy-target baseline gains only moderately, so the large gains are not an artifact of the more lenient post-processing. The encoding-invariant evaluation is not uniformly easier either: PromptDiffusion drops on DeepCrack (0.195 → 0.097).

Native encodings (depth). Metric depth cannot be recovered from an arbitrary encoding, so we instead vary the encoding in the context: VIBE's Turbo color map, and grayscale with near points dark or light (256 bins over the depth range, 0–10 m for NYUv2 and 0–26.1 m for SeaThru) (RMSE ↓; all models in Appendix A.16, Table 65):

ModelNYUv2 Turbo / near dark / near lightSeaThru Turbo / near dark / near light
SD-VICL1.478 / 1.867 / 1.6352.283 / 2.326 / 2.220
Painter2.704 / 0.407 / 5.0023.583 / 8.098 / 11.986
LVM2.727 / 3.333 / 4.3824.341 / 3.187 / 14.242
Analogist3.150 / 3.698 / 3.9023.167 / 4.144 / 21.683

Painter, trained on NYUv2 depth with grayscale near-dark encoding, reaches 0.407 in exactly that setting, far ahead of all other models. With the reversed encoding its error rises to 5.002, and on unseen SeaThru data it fails under both grayscale encodings. This is in-weight learning in its plainest form. SD-VICL performs similarly under all three encodings. Most other models degrade with the near-light encoding, on SeaThru often collapsing to uniformly black or white outputs.

Together with the mismatched-context experiment (see our response to Reviewer b5ZA, W3), these results replace the argument in A.1 with measurements: some models resort to in-weight knowledge of a task, while others predict from the transformation shown in the context.

AuthorsOpenReview ↗

Response to Reviewer TzYM (3/6)

B] Combinations unseen in pretraining

Response: We added a systematic overlap analysis based on the training data reported in each model's paper (Table 1; details in Tables 57 and 58 and Appendix A.11).

Datasets. Only one of the 14 datasets appears in pretraining: NYU Depth V2, used by Painter for depth estimation. Hence, 105 of the 106 dataset-task combinations were not seen as task-dataset pairs by any model, and 99 of them use no image seen in training (excluding the seven NYU Depth V2 combinations). The conclusions hold on these combinations unchanged. Overlap with web-scale image data (e.g., LAION) cannot be ruled out. We list these sources in Table 57.

Tasks. At the task level, overlap is substantial for LVM, Painter and PromptDiffusion. It does not, however, explain our findings. Models trained on a task with encoded outputs (e.g., segmentation, depth) still fail when the demonstrated encoding differs from their pretraining encoding, while the three models without task overlap (SD-VICL, MAE-VQGAN, IMProv) all rank above the task-trained LVM and Painter on segmentation and edge detection. Some VIBE tasks were part of no model's training (modality transfer, flipping, rotation, object removal), so they can only be solved by adapting to the context. Such adaptation is rare: SD-VICL solves modality transfer on IXI, and MAE-VQGAN partly handles flips. Object removal is beginning to be addressed by SD-VICL, IMProv and MAE-VQGAN, which shows qualitatively but not in the pixel-based metrics, which are misaligned with this task (as discussed for colorization in Section 5; see also Q6). No model solves rotation.

C] Model set (IMProv, Visual Token Matching, SegGPT)

Response: We agree that the model set needs an explicit motivation, and we now state our inclusion criteria in Section 4, before the model descriptions. VIBE includes models that (i) solve multiple image-to-image tasks (ii) from the visual context alone, without test-time parameter updates (L156).

  • IMProv meets both criteria if its manually written task prompts are replaced by a text description derived from the visual context, as Analogist does: we use Qwen3-VL-8B-Instruct to describe the context pair and query, so that no manual task description enters. We added IMProv (S2CV+LAION) to the results in the main paper, covering all 106 combinations. It ranks seventh of the eight models overall, ahead of PromptDiffusion and the Copy-target baseline, with its best results on segmentation and edge detection (fourth on both).
  • VTM fails (ii): it fine-tunes task-specific bias parameters on the support set. In addition, it was trained only on the narrower domain of indoor scenes (Taskonomy).
  • SegGPT fails (i): it learns in context within segmentation but does not address the other 11 tasks. As it is exclusive to segmentation, we do not consider it.

In the updated manuscript, you will find VTM and SegGPT discussed in Section 2, and we thank the reviewer for pointing out the missing SegGPT reference.

W3: Related work on multitask dense prediction and adapters

The related work could briefly discuss multitask dense prediction transformers and adapter-based methods for generalising to new tasks and domains, as an alternative to in-context adaptation. Currently, it is missing from the paper.

Response: Thank you for the suggestion. We added a paragraph to Section 2 that discusses multi-task dense prediction transformers (MTI-Net, InvPT, MulT, TaskPrompter) and parameter-efficient adaptation (VPT, AdaptFormer, LoRA, Polyhistor), as well as few-shot dense predictors that combine meta-training with test-time tuning (VTM, Chameleon). We position them as the weight-update alternative to VICL: they adapt through (a few) trained parameters, whereas VICL adapts from a demonstration alone. This is the setting VIBE evaluates.

Q1: Variation over context pairs

How much do the scores and the ranking in Table 4 change if each query is evaluated with several different context pairs, for example five draws from the stored context lists, or with context pairs retrieved by similarity to the query?

Response: In short, we evaluated all models with context pairs retrieved by similarity to the query (Zhang et al., 2023) on 23 task-dataset combinations (Appendix A.13, Table 60). Retrieval improves the models that use the context on nearly all combinations, while LVM, DeLVM and Painter improve on only about half of them, i.e., close to chance. The ranking remains stable (Kendall's τ = 0.86 among the eight models). Details are in our response to W1 above. Note that in the updated manuscript, the ranking table referred to in your question as Table 4 is now Table 5.

Reviewer_ZNjqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VIBE, a unified and reproducible benchmark for visual in-context learning (VICL), covering 12 image-to-image tasks across 14 datasets and six visual domains, comprising 106 task-dataset combinations. It establishes a standardized evaluation protocol with unified task encodings, data preprocessing, deterministic context sampling, and task-specific post-processing and evaluation. Using this framework, the paper evaluates seven VICL models in a one-shot setting and finds that most models remain strongly biased toward their pretraining tasks rather than genuinely adapting to the provided visual context.

Critical strengths
  1. Broad task and domain coverage. The benchmark covers substantially different task families, including perception, image restoration, and image manipulation, as well as diverse image distributions such as medical, underwater, and thermal imagery. This breadth is a valuable contribution, enabling a more comprehensive assessment of VICL models and revealing limitations that may not be apparent in narrower evaluation settings.
Critical weaknesses
  1. Unclear pretraining overlap. A central motivation of the paper is that existing evaluations largely mirror tasks or image domains seen during pretraining, without requiring genuine adaptation. However, it remains unclear to what extent VIBE itself addresses this issue, as some evaluated tasks, datasets, or domains may also have been encountered during the models' pretraining. Without a systematic analysis of such overlaps, it is difficult to assess whether VIBE truly evaluates adaptation to unseen tasks and domains.

  2. Evaluation metrics are sensitive to output formats. Some evaluation metrics depend heavily on predefined output format, potentially penalizing models that correctly perform a task but use different visual representations. For example, in object localization, a model may correctly identify object locations but receive a low score for not following the required color. While adherence to demonstrated formats is relevant to VICL, I think successfully completing the task is more important than strictly following a predefined output format.

Questions for authors
  1. Please provide domain-wise aggregate results to better assess cross-domain generalization.

  2. How do task performance and adherence to the specified output format change under N-shot settings beyond one-shot? I'm curious whether additional demonstrations can mitigate the observed output-formatting issues.

  3. Given the limitations of metrics like PSNR and LPIPS acknowledged in Section 5 and Appendix A.1, how robust are the reported model rankings and conclusions? Is there any alternative metrics that can better reflect actual task completion and avoid rewarding such shortcuts?

  4. Some metrics (e.g. keypoint detection) may underestimate task competence when models produce meaningful predictions but fail to follow the specified output encoding. Could the authors define and report format-invariant metrics alongside the current metrics to disentangle task competence from output-format adherence?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZNjq (1/3)

We thank the reviewer for their feedback and for highlighting the benchmark's broad task and domain coverage.

W1: Unclear pretraining overlap

A central motivation of the paper is that existing evaluations largely mirror tasks or image domains seen during pretraining, without requiring genuine adaptation. However, it remains unclear to what extent VIBE itself addresses this issue, as some evaluated tasks, datasets, or domains may also have been encountered during the models' pretraining. Without a systematic analysis of such overlaps, it is difficult to assess whether VIBE truly evaluates adaptation to unseen tasks and domains.

Response: We agree and added a systematic overlap analysis, based on the training data reported in each model's paper: Table 1 gives an overview, and Tables 57 and 58 (appendix) give details per dataset and per task.

Datasets. Overlap is minimal: of the 14 VIBE datasets, only NYU Depth V2 was used in training, by Painter for depth estimation. Overlap at the image level with web-scale training data (e.g., LAION for the Stable Diffusion-based models) cannot be ruled out. We list these sources explicitly in Table 57.

Tasks. Overlap is substantial for some models (LVM, Painter, PromptDiffusion) and absent for others (MAE-VQGAN, SD-VICL, IMProv). It does not, however, explain the ranking:

  • The three models without task overlap (SD-VICL, MAE-VQGAN, IMProv) all rank above the task-trained LVM and Painter on segmentation and edge detection, and SD-VICL ranks first overall.
  • Overlap pays off only where the output is a natural image. Painter and LVM rank among the best on restoration tasks they were trained on, but rank poorly on trained perception tasks with encoded outputs, such as segmentation and depth. There, they recognize the task but reproduce their pretraining encoding instead of the demonstrated one. For example, LVM fails to switch from black edges on white to white edges on black.
  • Some VIBE tasks were part of no model's training (modality transfer, flipping, rotation, object removal), so they can only be solved by adapting to the context. Such adaptation is rare, and where it occurs, it again comes from models without task overlap: SD-VICL solves modality transfer on IXI, and MAE-VQGAN partly handles flips. Object removal is beginning to be addressed by SD-VICL, IMProv and MAE-VQGAN, which shows qualitatively but not in the pixel-based metrics, which are misaligned with this task (as discussed for colorization in Section 5). No model solves rotation.

We would like to emphasize the design principle behind this: in VICL, the context defines the task, including its output encoding. A model that solves a seen task only in its pretraining encoding is performing multi-task inference, not in-context learning. VIBE's encodings therefore differ from each model's native encoding in at least one aspect, so that even tasks seen during training can only be solved by adapting to the demonstrated form. To separate this from task competence as such, we additionally report encoding-invariant metrics (see our responses to W2 and Q4).

Thus, prior exposure helps a model recognize a task, but not adapt to its demonstrated form, which supports our hypothesis that these models learn a task switch rather than adapting in context. We added this analysis to Appendix A.11.

W2: Evaluation metrics are sensitive to output formats

Some evaluation metrics depend heavily on predefined output format, potentially penalizing models that correctly perform a task but use different visual representations. For example, in object localization, a model may correctly identify object locations but receive a low score for not following the required color. While adherence to demonstrated formats is relevant to VICL, I think successfully completing the task is more important than strictly following a predefined output format.

Response: We agree that it matters whether a model solves the task at all, and we now report this separately (see Q4). We do, however, see adherence to the demonstrated format as part of the task in in-context learning, not as an add-on to it. The context is the only place where the task is defined. If a model produces a correct segmentation in its own training palette instead of the demonstrated one, it has recognized a task it knows from training, but it has not adapted to the task as specified. Rewarding this would favor models whose training covers the evaluated tasks over models that adapt to new task definitions, and adaptation is the capability VIBE is designed to measure. We therefore keep the standard evaluation for the rankings and add encoding-invariant evaluation as a diagnostic that makes the distinction visible (Appendix A.16).

AuthorsOpenReview ↗

Response to Reviewer ZNjq (2/3)

Q1: Domain-wise aggregate results

Please provide domain-wise aggregate results to better assess cross-domain generalization.

Response: Thank you for the suggestion. We added domain-wise rankings (Table 8, appendix), aggregated over all task-dataset combinations within each domain. Three observations stand out:

  • No model generalizes uniformly across domains. Ranks vary strongly by domain: Painter ranks first on Arts but eighth on Underwater, and LVM ranks first on Object/Human and Wildlife but sixth on Medical.
  • Specialized domains are hardest for models trained on everyday images. On Medical and Underwater, the Copy-target baseline outperforms several models (4th place in both). SD-VICL ranks first on Medical and MAE-VQGAN on Underwater, both without task overlap.
  • Aggregation matters. By domain, LVM and SD-VICL swap places. Tasks applied to every dataset (inpainting, denoising, colorization, rotation, flip) appear in every domain and thus get more weight, and LVM was trained on three of these tasks, which gives it an edge in the domain-wise aggregation. We consider the task-wise ranking (Table 5) the better fit for VIBE's goal, since it rewards adaptation across all tasks rather than strength on a few task types that occur everywhere.

Q2: N-shot settings

How do task performance and adherence to the specified output format change under N-shot settings beyond one-shot? I'm curious whether additional demonstrations can mitigate the observed output-formatting issues.

Response: We evaluated N ∈ {1, 2, 3, 7} context pairs for the three models that natively accept multiple context pairs: SD-VICL, LVM and DeLVM, the latter up to N = 3, the maximum number of context pairs it was trained with. The evaluation covers 16 task-dataset combinations spanning 11 of the 12 tasks. We omitted rotation and flip, which no model solves even in the one-shot setting. MAE-VQGAN is excluded, because adding pairs to its fixed image grid shrinks each image to very low resolution. The results (Appendix A.12, Table 59) show a clear split:

  • SD-VICL improves consistently on almost all tasks and metrics, e.g., segmentation mIoU on SUIM from 0.29 to 0.50, keypoint F1 on OTP from 0.07 to 0.23, and detection mAP on LLVIP from 0.29 to 0.47 (1 → 7 shots).
  • LVM and DeLVM benefit little. Gains are small and saturate after 2–3 shots. On tasks whose output encoding must be adopted from the context, LVM even degrades with more demonstrations, e.g., segmentation mIoU on SUIM from 0.16 to 0.10 and keypoint F1 on OTP from 0.09 to 0.02.

Additional demonstrations therefore do not mitigate the formatting issues: for models that fall back to their pretraining encoding, more examples of the demonstrated encoding do not make them adopt it. This is consistent with our hypothesis that these models learn a task switch. Only SD-VICL, which already follows the demonstrated encoding in the one-shot setting, turns additional context into better predictions.

AuthorsOpenReview ↗

Response to Reviewer ZNjq (3/3)

Q3: Robustness of rankings to metric limitations

Given the limitations of metrics like PSNR and LPIPS acknowledged in Section 5 and Appendix A.1, how robust are the reported model rankings and conclusions? Is there any alternative metrics that can better reflect actual task completion and avoid rewarding such shortcuts?

Response: We thank the reviewer for picking up this point. That standard metrics such as PSNR and LPIPS can reward shortcuts in VICL is itself one of the insights of our evaluation: it only becomes visible when many models are compared on many tasks under one protocol, as in VIBE. Accordingly, we highlight it in Section 5 as an open problem and an important direction for future work, i.e., metrics that reflect task completion aligned with the context. Designing such metrics for all tasks goes beyond the scope of a benchmark paper, but we took three steps to make the current rankings robust and their limitations transparent.

  1. Several metrics per task. For image outputs, the ranking aggregates PSNR, SSIM and LPIPS per task (A.5), so no single metric decides a rank.
  2. Shortcut baseline. The Copy-target baseline exposes where a metric rewards shortcuts rather than task completion. For example, on depth estimation, copying the context target already outperforms all models except SD-VICL on NYUv2 (RMSE 1.596), so a model needs to clearly exceed it to count as solving the task.
  3. Encoding-invariant evaluation for the perception tasks (Appendix A.16, Table 64; see Q4). The main conclusions hold: SD-VICL remains the strongest model on edge detection under both evaluations, and the per-dataset rankings of edge detection agree strongly (Kendall's τ 0.71–0.93 across the eight models). Segmentation rankings change more (τ 0.25–0.29), because models like Painter solve segmentation in their own encoding. This is the distinction between in-weight and in-context prediction that VIBE is designed to capture (see W2).

Q4: Format-invariant metrics

Some metrics (e.g. keypoint detection) may underestimate task competence when models produce meaningful predictions but fail to follow the specified output encoding. Could the authors define and report format-invariant metrics alongside the current metrics to disentangle task competence from output-format adherence?

Response: Yes, we added encoding-invariant evaluation for segmentation (k-means clustering of the predicted colors with Hungarian matching to the classes) and for edge and keypoint detection (two-way clustering, invariant to the color of edges and keypoints and to whether they are darker or lighter than the background), and alternative encodings for depth (Appendix A.16, Tables 64 and 65). Segmentation is the clearest example: Painter's mIoU on DRAM rises from 0.171, below the Copy-target baseline, to 0.662, the best score of all models, while SD-VICL, which follows the demonstrated encoding, changes little (0.415 → 0.463). The encoding-invariant evaluation thus disentangles the two aspects as suggested: Painter, LVM and DeLVM often solve the task but in their training encoding, while SD-VICL, MAE-VQGAN and IMProv adapt to the encoding provided in the context. Full results and discussion are in our response to Reviewer TzYM (W2 A).

Reviewer_b5ZAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a new benchmark for visual in-context learning (VICL), covering 12 tasks and using existing datasets. The authors evaluate seven image-to-image models. The findings suggest that many existing VICL models rely mostly on pretrained knowledge rather than genuinely adapting to the provided visual demos.

Critical strengths

The aims to provide a comprehensive visual in-context learning benchmark is timely and important. The paper provide a unified and reproducible evaluation protocol and some potentially useful analysis of failure modes.

Critical weaknesses

Insufficient and unclear motivation

  • The paper does not convincingly establish the limitations of existing VICL benchmarks or explain why a new benchmark is necessary. The motivation is largely restricted to the claim that prior evaluations use narrow, pretraining-aligned task and image distributions, without a systematic and clear comparison demonstrating the deficiencies of existing benchmarks.

  • The benchmark primarily measures performance across heterogeneous image-to-image tasks rather than isolating the in-context adaptation capability that motivates its introduction.

Outdated model selection

The benchmark evaluates mostly older models (most of them are before 2025). Also, it focus on image-to-image models rather than the current SOTA visual in-context learning models, like GPT image 2.5, Qwen-image-2.1, gemini image etc.

Some claims are not supported by experimental results

The paper said that most VICL models are merely multi-task models rather than genuine in-context learners. However, no experiments directly measure whether predictions depend on the provided context. They should do some simple controls like removing parts of the context, or replacing it with an unrelated example, or changing the demonstrated transformation but keeping the query fixed etc.

Questions for authors

See above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer b5ZA (1/3)

We thank the reviewer for their feedback and for recognizing that a comprehensive VICL benchmark is "timely and important".

W1: Insufficient and unclear motivation

The paper does not convincingly establish the limitations of existing VICL benchmarks or explain why a new benchmark is necessary. The motivation is largely restricted to the claim that prior evaluations use narrow, pretraining-aligned task and image distributions, without a systematic and clear comparison demonstrating the deficiencies of existing benchmarks.

The benchmark primarily measures performance across heterogeneous image-to-image tasks rather than isolating the in-context adaptation capability that motivates its introduction.

Response: We agree that the motivation should rest on a systematic comparison, and we added one to the introduction (Table 1): it shows which tasks and domains each evaluated VICL model was originally evaluated on, and how. It reveals three deficiencies:

  1. Narrow coverage. No model was evaluated on more than 5 of VIBE's 12 tasks. No prior evaluation covers any manipulation task, and none uses medical, underwater, art or wildlife data.
  2. Overlap with training tasks. Painter, PromptDiffusion and LVM were evaluated mostly on tasks they were trained on (marked † in Table 1). Painter and PromptDiffusion were even evaluated on test splits of their own training datasets.
  3. No shared protocol. Each model was evaluated on its own selection of tasks, datasets, encodings and metrics, some of them non-standard. Five of eight reuse parts of the narrow protocol of Bar et al. (2022), which covers only foreground segmentation, single-object detection and colorization on natural images. Results are therefore not comparable across models. VIBE provides the first comparison of all models under one protocol.

On isolating in-context adaptation: we agree that aggregate performance alone does not isolate it. We therefore added analyses that do: context controls that break the correspondence between context input and output, reverse the demonstrated encoding, or vary which context pair is shown (see W3). In addition, an encoding-invariant evaluation separates solving a task from adopting the demonstrated encoding (Appendix A.16), and an analysis of overlap with pretraining tasks and datasets (Appendix A.11) shows which tasks and datasets each model has seen. Together, they show which models adapt to the context and which rely on tasks learned during training.

AuthorsOpenReview ↗

Response to Reviewer b5ZA (2/3)

W2: Outdated model selection

The benchmark evaluates mostly older models (most of them are before 2025). Also, it focus on image-to-image models rather than the current SOTA visual in-context learning models, like GPT image 2.5, Qwen-image-2.1, gemini image etc.

Response: Scope. VIBE evaluates visual in-context learning: models that infer an image-to-image task from a visual example pair. The seven models cover the VICL approaches proposed so far, from inpainting-based models to autoregressive and diffusion-based models. The most recent, SD-VICL, appeared in 2025 (ICCV 2025). We have additionally added IMProv to the results in the main paper, covering all 106 combinations (see our response to Reviewer TzYM).

Recent image generation models. GPT Image 2.5 and Qwen-Image-2.1 were released on 8 and 20 September 2026, around the submission deadline. The ICLR 2027 reviewer guidelines consider work from the two months before the deadline contemporaneous, and state that missing comparisons to it "cannot be a basis for rejection".

We nevertheless agree that it is an interesting question whether recent multi-image generation and editing models act as visual in-context learners. We therefore evaluated the proposed open-weight model Qwen-Image-2.1. It receives the context pair and the query as images, with a task-agnostic instruction, so that the task has to be inferred from the visual context alone. This keeps the comparison fair to the other models and follows our setup for the text-conditioned VICL models Analogist and IMProv, which likewise receive no manual task description.

We ran it on 18 task-dataset combinations covering all task groups. It ranks in the lower middle of the field, with an average rank of 5.6 among the nine models. It performs best only on RGB→IR modality transfer and stays far behind the best VICL models on the perception tasks (e.g., edge F1 on DeepCrack 0.059 vs. 0.625, mIoU on DRAM 0.200 vs. 0.415).

TaskDatasetMetricBest VICL modelQwen-Image-2.1Rank (of 9)
DepthNYUv2RMSE ↓1.478 (SD-VICL)3.3269
DepthSeaThruRMSE ↓2.157 (PromptDiff.)3.7078
Edge det.DeepCrackF1 ↑0.625 (SD-VICL)0.0596
Edge det.DukeF1 ↑0.538 (SD-VICL)0.0562
Keyp. det.OTPF1 ↑0.087 (LVM)0.0038
Obj. det.LLVIP[email protected] ↑0.290 (SD-VICL)0.0903
Sem. seg.SUIMmIoU ↑0.292 (SD-VICL)0.1815
Sem. seg.DRAMmIoU ↑0.415 (SD-VICL)0.2006
Artif. rem.CDDPSNR ↑15.31 (Painter)12.167
ColorizeDRAMPSNR ↑19.68 (Painter)14.276
ColorizeSUIMPSNR ↑16.11 (LVM)13.666
DenoiseOTPPSNR ↑24.94 (LVM)20.202
InpaintDukePSNR ↑23.14 (LVM)13.428
Obj. rem.RORDPSNR ↑15.84 (LVM)13.774
Mod. trans.IXI (axial)PSNR ↑21.04 (SD-VICL)12.506
Mod. trans.IXI (coronal)PSNR ↑19.26 (SD-VICL)12.445
Mod. trans.IXI (sagittal)PSNR ↑20.50 (SD-VICL)10.968
Mod. trans.LLVIPPSNR ↑11.98 (SD-VICL)12.531

A recent, large-scale image generation model thus does not solve VIBE either. It is competitive where the task resembles image editing, but struggles where the context defines an unfamiliar output format. This mirrors the in-weight behavior we observe for the specialized VICL models.

We did not include closed APIs (GPT Image, Gemini): their models change without versioning or are retired altogether (e.g., the VICL model Analogist relies on the GPT-4V API, which has since been shut down, see our response to Reviewer TzYM, Q5), which conflicts with VIBE's goal of reproducible evaluation. However, VIBE's toolkit supports adding them through a simple model interface.

AuthorsOpenReview ↗

Response to Reviewer b5ZA (3/3)

W3: Some claims are not supported by experimental results

The paper said that most VICL models are merely multi-task models rather than genuine in-context learners. However, no experiments directly measure whether predictions depend on the provided context. They should do some simple controls like removing parts of the context, or replacing it with an unrelated example, or changing the demonstrated transformation but keeping the query fixed etc.

Response: We agree that this claim needs direct measurements, and we added three controls of the kind the reviewer suggests.

1. Breaking the correspondence within the context pair (Appendix A.15, Table 63). We combine the input of one context example with the output of another example from the same task-dataset combination, keeping the query fixed, on 18 task-dataset combinations covering all task groups. The mismatched pair still shows the task and its output format, but no longer the transformation from input to output. A model that only recognizes the task degrades little, whereas a model that infers the mapping from the context pair should degrade. The count gives the number of combinations in which the main metric drops:

SD-VICLIMProvMAE-VQGANAnalogistDeLVMLVMPainterPromptDiff.
18/1816/1816/1816/1814/1813/1813/186/18

These counts have to be read against how well a task is solved in the first place: if a model already fails a task with the matched context, a mismatched context has little room to make it worse. More telling are tasks a model does solve whose scores barely change under a mismatched context. Painter, for example, keeps its scores on depth (NYUv2 RMSE 2.704 vs. 2.719), detection ([email protected] 0.246 vs. 0.242) and colorization (DRAM PSNR 19.68 vs. 19.61), i.e., its prediction is driven by what it learned in its weights rather than by the context. SD-VICL, in contrast, collapses on the perception tasks (e.g., edge F1 on DeepCrack 0.625 → 0.036). For the full results and discussion, see Appendix A.15, Table 63.

2. Changing the demonstrated transformation with the query fixed (Appendix A.16, Table 65). For depth estimation, we reverse the demonstrated encoding (near points dark vs. light). SD-VICL follows the reversal with similar errors under both encodings (NYUv2 RMSE 1.867 vs. 1.635). Painter reaches 0.407 with the encoding it was trained on, but 5.002 with the reversed one. Most other models degrade strongly and, on SeaThru, often collapse to uniformly black or white outputs.

3. Varying which context is shown (Appendix A.13, Table 60). With context pairs retrieved by visual similarity to the query (procedure of Zhang et al. "What makes good examples for visual in-context learning?." NeurIPS 2023), SD-VICL and Analogist improve on all 23 combinations, while LVM, DeLVM and Painter improve on only 12–13, close to chance.

Together with the encoding-invariant evaluation (Appendix A.16, Table 64; see our response to Reviewer TzYM, W2 A), these controls point consistently in the same direction. The picture is a gradual tendency rather than a strict split: SD-VICL, MAE-VQGAN, IMProv and Analogist tend towards contextual prediction, while LVM, DeLVM, Painter and PromptDiffusion rely more on knowledge learned in their weights. We have rephrased the claim in the paper accordingly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their detailed and constructive feedback. The reviews converged on three questions, which we addressed with new experiments and revisions to the manuscript. The detailed answers are in each reviewer's thread:

  • Does pretraining overlap explain the findings? See our responses to Reviewer ZNjq (W1) and Reviewer TzYM (W2 B).
  • Does VIBE measure in-context adaptation or only task competence? See our responses to Reviewer b5ZA (W3) and Reviewer TzYM (W1, W2 A) for the context controls and the encoding-invariant evaluation, and to Reviewer ZNjq (W2, Q3, Q4) for the role of the metrics.
  • Are the model set and protocol complete? See our responses to Reviewer b5ZA (W1, W2) and Reviewer TzYM (W2 C, Q5).

Changes to the manuscript

Main paper

  • Table 1 (Introduction), new: prior evaluations of each model (tasks, datasets, metrics), showing that no shared protocol existed before VIBE.
  • Section 2: discussion of Visual Token Matching, SegGPT and multitask dense prediction.
  • Section 4: explicit model inclusion criteria, IMProv added as eighth model, and a note on Analogist's VLM replacement (GPT-4V has been shut down).
  • Results: IMProv is included on all 106 combinations in all result tables, the rankings (Tables 5 and 8), the spider plots, and the compute and model overviews (Tables 9 and 10). The ranking table formerly numbered Table 4 is now Table 5.
  • Sections 4.1.2, 4.1.3 and 5: corrected the description of Stable Diffusion's latent space, clarified the object-removal results, and rephrased the claim that most VICL models act as multi-task learners to a tendency, in line with the new measurements.

Appendix (new sections)

  • A.11 Pretraining overlap: task and dataset overlap per model (Tables 57 and 58).
  • A.12 Few-shot evaluation: 1 to 7 context pairs on 16 combinations (Table 59).
  • A.13 Context selection: retrieval-based vs. standard context pairs on 23 combinations, including the resulting ranking (Table 60).
  • A.14 Analogist reproduction: GPT-4V vs. Qwen3-VL-8B prompts on Analogist's released data (Table 62).
  • A.15 Mismatched context: context input and output taken from different examples (Table 63).
  • A.16 Encoding-invariant evaluation: clustering-based post-processing for segmentation, edge and keypoint detection (Table 64), and alternative depth encodings (Table 65).

Reported in the responses only: Qwen-Image-2.1 as a recent image generation model (Reviewer b5ZA, W2) and our validation of the SD-VICL re-implementation (Reviewer TzYM, Q4).

AuthorsOpenReview ↗

Second revision of the manuscript

Following the second round of discussion, we uploaded a further revision. The changes:

Main paper

  • Line 225: VIBE's encodings are designed to largely differ from those used to train the evaluated models.
  • Line 451: pointer to the encoding-invariant results (Appendix A.16).

Appendix

  • A.11: new Table 58b, showing per model and task-dataset combination whether the task or the dataset is part of the reported training data.
  • A.12, A.13, A.15: all experiments now use the same 18 task-dataset combinations, and the median relative gain or drop of the main metric is reported (Tables 59, 60 and 63). The context-selection ranking (Table 61) is updated accordingly.
  • A.17 (new): validation of our SD-VICL re-implementation (Table 66), previously reported in the responses only.
  • A.18 (new): five random draws of the context pair under the standard context selection, with standard deviations (Table 67) and the ranking per draw (Table 68).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.