- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents VIBE, a Visual In-Context BEnchmark and toolkit with a unified, reproducible evaluation protocol for visual in-context learning (VICL). Every task is posed as a one-shot visual analogy: given a context pair (Cin, Cout) and a query image Q, the model has to predict the output for Q. The benchmark covers 12 tasks in three groups (Perception & Localization, Restoration & Enhancement, Manipulation & Transformation), 14 datasets and 6 imaging domains, giving 106 dataset-task combinations. The protocol fixes three things:
- Task encodings: RGB renderings of segmentation masks, depth maps, edges, keypoints and boxes.
- Context selection: a class-aware greedy policy for tasks with semantic classes, and fixed-seed random selection otherwise.
- Post-processing: a shared decoding step before metrics, such as color-aware mIoU for segmentation.
Seven VICL models (MAE-VQGAN, Painter, LVM, DeLVM, PromptDiffusion, SD-VICL and Analogist) are stress tested against a Copy target baseline and combined into a rank-aggregated score. The main findings are:
- Most models are multi-task models instead of in-context learners. They learn a switch that steers them to a pre-training task and keep their pre-training color maps.
- SD-VICL and MAE-VQGAN predict more contextually, which the authors link to correspondence.
- No model handles rotation or flipping.
- Contextual tasks are inherently ill-posed, which calls for context-sensitive metrics.
Critical strengths
-
The paper gives a unified protocol across many models, tasks and domains, putting 7 VICL models on the same 106 dataset-task combinations with fixed context sets, the same task encodings and the same post-processing. In my opinion, it gives the field a reasonable comparison since each model was so far evaluated on its own tasks and mostly on images close to its training data. The Copy target baseline is a simple addition, because several models fall to or below it, and PromptDiffusion ranks below it overall. This is the main reason I think the benchmark is worth publishing in some form.
-
The paper shows convincingly that most models fall back to tasks and color maps from pre-training instead of following the context. The Painter example in Figure 7 is especially clear i.e., when prompted with colorization on NYU-Depth V2, it still produces its gray depth maps. The discussion of inherently contextual vision tasks, with the wolf and bison colorization example in Table 5, makes a good case that ground-truth metrics are ill-posed in VICL, and the field could take this on.
Critical weaknesses
-
All results use a single context pair per query in the one-shot setting. VICL performance is known to depend strongly on which context pair is shown (Zhang et al., NeurIPS 2023; Sun et al., 2023), but the paper reports no variation over context pairs. So the gaps between models, the ranking in Table 4, and the claim that most models only learn a switch might change if different context pairs were used. They already store a list of context samples per query for few-shot prompting, so this seems feasible to report.
-
The main confounds are argued in Appendix A.1 but they are not measured. The authors acknowledge that models would likely score better with their own task encodings, and that some evaluation datasets were part of pre-training. They then state that "in practice, this effect does not take shape", but give no numbers/ reasons why this happens. The claim of most models being multi-task models, and not in-context learners, builds on this point. Two experiments would establish it: A] an ablation with each model's native task encodings, or the mIoU with Hungarian matching that A.1 mentions; B] results on the dataset-task combinations not seen by any models in pre-training. C] Also, the model set is incomplete. IMProv and Visual Token Matching are discussed in Section 2 but not evaluated, and SegGPT is not mentioned at all. That weakens the claim to stress test the most prominent VICL models.
-
The related work could briefly discuss multitask dense prediction transformers and adapter-based methods for generalising to new tasks and domains, as an alternative to in-context adaptation. Currently, it is missing from the paper.
Questions for authors
Questions whose answers could change my assessment:
- How much do the scores and the ranking in Table 4 change if each query is evaluated with several different context pairs, for example five draws from the stored context lists, or with context pairs retrieved by similarity to the query?
- Can you report results with each model's native task encodings, at least for segmentation and depth, and the segmentation scores with Hungarian matching that Appendix A.1 refers to? This would tell me whether the failures come from the task or from the encoding.
- Which of the 14 datasets appear in each model's pre-training data, and do the conclusions hold on the dataset-task combinations that are unseen for all models?
- How was the SD-VICL re-implementation validated against the original method? Does it reproduce the numbers reported by Oorloff et al. (2025)?
- Why were IMProv, Visual Token Matching and SegGPT not evaluated? For Analogist, does replacing the original VLM with Qwen3-VL-8B-Instruct change its results?
- Section 4.1.3 says SD-VICL is one of the only models that start to address object removal. In Table 3, however, it has the second-lowest PSNR (12.19) and SSIM (0.36) among the models, and four models have better LPIPS. Which one is correct?
- Section 4.1.2 explains SD-VICL's denoising by "the use of a quantized codebook (Rombach et al., 2022)", but Table 8 lists the Stable Diffusion models as VAE-KL, which is a continuous autoencoder, and not a codebook as far as I know. Please correct or justify this.
- Minor: Tables 37 to 39 are titled "Style Transfer" but appear to report modality transfer.
As suggestions for future work, which do not affect my score, I would like to see results in the few-shot setting. I would also like to see the context-sensitive metrics and human evaluation protocols that the authors propose in Section 5.
11 comments
Response to Reviewer TzYM (2/6)
A] Native task encodings / Hungarian matching
Response: We added both experiments (Appendix A.16, Tables 64 and 65).
What the two evaluations measure. VIBE evaluates in-context learning, i.e., whether a model adapts to a task as defined by the context. The output encoding shown in the context (e.g., the class palette or the depth color map) is part of that task definition. A model that solves a task only in the encoding it was trained with reproduces a task learned in its weights, as any supervised multi-task model would, but does not adapt to the context. VIBE's standard metrics therefore require the demonstrated encoding. The encoding-invariant evaluation answers a different question: does the model solve the task in any encoding? We use it as a diagnostic to separate the two, not as a replacement, and keep the standard evaluation for all rankings.
Encoding-invariant evaluation (segmentation, edges, keypoints). The encoding-invariant metrics replace the palette- or color-based post-processing of the standard metrics with a clustering of the predicted colors. For segmentation, we cluster the predicted pixel colors into k groups with k-means, where k is the number of classes in the query's ground truth, and match clusters to classes one-to-one by Hungarian matching on IoU (Hungarian mIoU). For edges and keypoints, we cluster into two groups and evaluate the one with the higher F1 as foreground, which makes the metric invariant to the color of edges and keypoints and to whether they are darker or lighter than the background. Selected results (mIoU / F1, Std. → Inv.; all models in Appendix A.16, Table 64):
| Model | SUIM seg. | DRAM seg. | DeepCrack edges | OTP keypoints |
|---|---|---|---|---|
| SD-VICL | 0.292 → 0.414 | 0.415 → 0.463 | 0.625 → 0.674 | 0.073 → 0.108 |
| Painter | 0.171 → 0.465 | 0.171 → 0.662 | 0.019 → 0.115 | 0.019 → 0.119 |
| LVM | 0.156 → 0.438 | 0.247 → 0.488 | 0.082 → 0.151 | 0.087 → 0.122 |
| DeLVM | 0.213 → 0.445 | 0.253 → 0.411 | 0.007 → 0.071 | 0.031 → 0.098 |
| MAE-VQGAN | 0.225 → 0.359 | 0.411 → 0.467 | 0.289 → 0.312 | 0.026 → 0.041 |
| Copy target | 0.203 → 0.297 | 0.337 → 0.364 | 0.041 → 0.077 | 0.012 → 0.019 |
The results confirm the reviewer's intuition and make our claim measurable. Painter falls below the Copy-target baseline on segmentation under the standard evaluation, yet achieves the best encoding-invariant scores of all models: it solves segmentation, but in its own training encoding rather than the one demonstrated. LVM and DeLVM show the same pattern to a lesser degree. SD-VICL, MAE-VQGAN and IMProv already follow the demonstrated encoding, so their scores change less, and SD-VICL remains the strongest model on edge detection under both evaluations. The Copy-target baseline gains only moderately, so the large gains are not an artifact of the more lenient post-processing. The encoding-invariant evaluation is not uniformly easier either: PromptDiffusion drops on DeepCrack (0.195 → 0.097).
Native encodings (depth). Metric depth cannot be recovered from an arbitrary encoding, so we instead vary the encoding in the context: VIBE's Turbo color map, and grayscale with near points dark or light (256 bins over the depth range, 0–10 m for NYUv2 and 0–26.1 m for SeaThru) (RMSE ↓; all models in Appendix A.16, Table 65):
| Model | NYUv2 Turbo / near dark / near light | SeaThru Turbo / near dark / near light |
|---|---|---|
| SD-VICL | 1.478 / 1.867 / 1.635 | 2.283 / 2.326 / 2.220 |
| Painter | 2.704 / 0.407 / 5.002 | 3.583 / 8.098 / 11.986 |
| LVM | 2.727 / 3.333 / 4.382 | 4.341 / 3.187 / 14.242 |
| Analogist | 3.150 / 3.698 / 3.902 | 3.167 / 4.144 / 21.683 |
Painter, trained on NYUv2 depth with grayscale near-dark encoding, reaches 0.407 in exactly that setting, far ahead of all other models. With the reversed encoding its error rises to 5.002, and on unseen SeaThru data it fails under both grayscale encodings. This is in-weight learning in its plainest form. SD-VICL performs similarly under all three encodings. Most other models degrade with the near-light encoding, on SeaThru often collapsing to uniformly black or white outputs.
Together with the mismatched-context experiment (see our response to Reviewer b5ZA, W3), these results replace the argument in A.1 with measurements: some models resort to in-weight knowledge of a task, while others predict from the transformation shown in the context.
Response to Reviewer TzYM (3/6)
B] Combinations unseen in pretraining
Response: We added a systematic overlap analysis based on the training data reported in each model's paper (Table 1; details in Tables 57 and 58 and Appendix A.11).
Datasets. Only one of the 14 datasets appears in pretraining: NYU Depth V2, used by Painter for depth estimation. Hence, 105 of the 106 dataset-task combinations were not seen as task-dataset pairs by any model, and 99 of them use no image seen in training (excluding the seven NYU Depth V2 combinations). The conclusions hold on these combinations unchanged. Overlap with web-scale image data (e.g., LAION) cannot be ruled out. We list these sources in Table 57.
Tasks. At the task level, overlap is substantial for LVM, Painter and PromptDiffusion. It does not, however, explain our findings. Models trained on a task with encoded outputs (e.g., segmentation, depth) still fail when the demonstrated encoding differs from their pretraining encoding, while the three models without task overlap (SD-VICL, MAE-VQGAN, IMProv) all rank above the task-trained LVM and Painter on segmentation and edge detection. Some VIBE tasks were part of no model's training (modality transfer, flipping, rotation, object removal), so they can only be solved by adapting to the context. Such adaptation is rare: SD-VICL solves modality transfer on IXI, and MAE-VQGAN partly handles flips. Object removal is beginning to be addressed by SD-VICL, IMProv and MAE-VQGAN, which shows qualitatively but not in the pixel-based metrics, which are misaligned with this task (as discussed for colorization in Section 5; see also Q6). No model solves rotation.
C] Model set (IMProv, Visual Token Matching, SegGPT)
Response: We agree that the model set needs an explicit motivation, and we now state our inclusion criteria in Section 4, before the model descriptions. VIBE includes models that (i) solve multiple image-to-image tasks (ii) from the visual context alone, without test-time parameter updates (L156).
- IMProv meets both criteria if its manually written task prompts are replaced by a text description derived from the visual context, as Analogist does: we use Qwen3-VL-8B-Instruct to describe the context pair and query, so that no manual task description enters. We added IMProv (S2CV+LAION) to the results in the main paper, covering all 106 combinations. It ranks seventh of the eight models overall, ahead of PromptDiffusion and the Copy-target baseline, with its best results on segmentation and edge detection (fourth on both).
- VTM fails (ii): it fine-tunes task-specific bias parameters on the support set. In addition, it was trained only on the narrower domain of indoor scenes (Taskonomy).
- SegGPT fails (i): it learns in context within segmentation but does not address the other 11 tasks. As it is exclusive to segmentation, we do not consider it.
In the updated manuscript, you will find VTM and SegGPT discussed in Section 2, and we thank the reviewer for pointing out the missing SegGPT reference.
W3: Related work on multitask dense prediction and adapters
The related work could briefly discuss multitask dense prediction transformers and adapter-based methods for generalising to new tasks and domains, as an alternative to in-context adaptation. Currently, it is missing from the paper.
Response: Thank you for the suggestion. We added a paragraph to Section 2 that discusses multi-task dense prediction transformers (MTI-Net, InvPT, MulT, TaskPrompter) and parameter-efficient adaptation (VPT, AdaptFormer, LoRA, Polyhistor), as well as few-shot dense predictors that combine meta-training with test-time tuning (VTM, Chameleon). We position them as the weight-update alternative to VICL: they adapt through (a few) trained parameters, whereas VICL adapts from a demonstration alone. This is the setting VIBE evaluates.
Q1: Variation over context pairs
How much do the scores and the ranking in Table 4 change if each query is evaluated with several different context pairs, for example five draws from the stored context lists, or with context pairs retrieved by similarity to the query?
Response: In short, we evaluated all models with context pairs retrieved by similarity to the query (Zhang et al., 2023) on 23 task-dataset combinations (Appendix A.13, Table 60). Retrieval improves the models that use the context on nearly all combinations, while LVM, DeLVM and Painter improve on only about half of them, i.e., close to chance. The ranking remains stable (Kendall's τ = 0.86 among the eight models). Details are in our response to W1 above. Note that in the updated manuscript, the ranking table referred to in your question as Table 4 is now Table 5.
Response to Reviewer TzYM (1/6)
We thank the reviewer for the detailed and constructive review, and for finding the benchmark worth publishing and the evidence on pretraining fallback convincing.
W1: Single context pair per query
Response: We agree that context choice matters and added an evaluation with context pairs retrieved by visual similarity to the query (Zhang et al., NeurIPS 2023) on 23 task-dataset combinations covering all task groups, for all models (Appendix A.13, Table 60). The results support our conclusions:
W2: Confounds argued but not measured