acceptodds
Under review as a conference paper at ICLR 2027

Not Just What It Is, But What It Isn't: Comparative Meta In-Context Learning for Fine-Grained Categorization

Abstract

Visual in-context learning enables pretrained multimodal large language models to perform downstream categorization from labeled demonstrations without further parameter updates. Fine-grained categorization requires recognizing not only what a query is, but also what it is not among visually similar alternatives. While substantial effort has focused on demonstration retrieval and context construction, strengthening this comparative use of context through limited-data meta-training remains less explored. Standard cross-entropy (CE) supervises the correct label but does not explicitly penalize incorrect continuations once generation departs from the target sequence. Our diagnostic analysis shows that CE tuning raises both target and distractor scores, partially offsetting gains in the discriminative margin. We propose CLOUD (Contrastive Likelihood Only Upon Divergence), which complements target-label learning with explicit negative supervision on in-context distractor labels. Across three backbone configurations, CLOUD improves accuracy over CE-based Meta-ICL by 1.74–2.21% on six held-out natural fine-grained benchmarks and 2.14–6.11% on three specialized benchmarks. With CLOUD, Qwen3-VL-2B further outperforms Qwen3-VL-32B with vanilla ICL by 4.63% on average across all nine benchmarks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multi-modal, In-context Learning, Fine-Grained Categorization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_J55EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Brief Summary: The paper tackles the task of fine-grained classification. The authors find naive cross-entropy loss isn't enough, instead propose their method "CLOUD" which gives additional negative signal to distractor labels, and only give the negative to the non-shared suffix tokens. The high-level idea when answering is to use CLIP to get initial set of candidate labels. Then for each label, ask the model the model the question and get the probability scores for the output labels to answer. Experiments on multiple fine-grained datasets (flowers, food, aircraft, cub-200, cars etc) show consistent improvement over competitive Meta-ICL baselines.

Critical strengths

Pros:

  1. The authors study an important problem of fine-grained classification and show consistent improvements over Meta-ICL across datasets showing that negative signal on close classes can be helpful.

  2. The authors provide a number of visualizations, and detailed ablation study.

  3. The authors show that training on just Oxford Pets and DTD, they can generalize to a bunch of other fine-grained datasets with huge improvements over vanilla!

Critical weaknesses

Cons:

  1. My main concern is that for training, the authors are using CLIP for centroid. So if CLIP has some wrong adjacent words, they would give additional negative signal which may not be intended. The paper heavily relies on CLIP. The authors should show results with other models like Siglip2.

  2. I think the authors are missing an important ablation in Table 7 of InfoNCE + CE. The main idea of the paper is to give additional negative signals, but that is already covered by InfoNCE, but the separate signal for correct from CE. Right now it is unclear if we really need the token-wise negative signal.

  3. Overall improvements on many datasets over Meta-ICL are somewhat incremental though overall positive.

Questions for authors

see weakness.

Q1. (Minor) it is unclear how generalizable this method to other work streams outside of fine-grained image classifications.

6 comments
AuthorsOpenReview ↗

Response to Reviewer J55E — Part 1: Retrieval, Negative Supervision, and SigLIP2

Thank you for your careful review and positive assessment of our results. We appreciate your recognition of the consistent improvements, diagnostic analyses, and transfer from Oxford Pets and DTD to unseen datasets. We address your concerns about retrieval-dependent supervision, the CE + InfoNCE comparison, the magnitude of the gains, and applicability beyond fine-grained classification below.

 

W1: Retrieval, Negative Supervision, and SigLIP2

Thank you for raising this concern. Retrieval selects contextual competitors, while ground-truth labels determine whether they receive positive or negative supervision. Let Yctx be the retrieved context labels and y∗y^{\ast} the query’s ground-truth label. Positive CE supervises y∗y^{\ast}, including when retrieval misses it, and the negative set is N=Yctx∖y∗.\mathcal{N}=Y_{\mathrm{ctx}}\setminus\\{y^*\\}.

The true class is therefore explicitly excluded from negative supervision. A visually similar but different class is a valid distractor under the dataset’s labels; learning to reject such a competitor is the intended fine-grained discrimination task. Retrieval errors can change the relevance and difficulty of the context, but they do not turn the ground-truth class into a negative training target.

To test whether the gains depend on CLIP, we replace it with SigLIP2 ViT-L/16 for both meta-training context construction and downstream retrieval, and independently retrain Meta-ICL and CLOUD. We retain the same source data, Qwen3-VL-2B backbone, and optimization settings, and use identical evaluation contexts for the matched methods. This evaluates the complete training-and-evaluation pipeline with the new retriever. 6 natural FGVC benchmarks — Accuracy / Hit rate (%)

Retriever / MethodFlowersFoodAircraftCUB-200CarsNABirdsNatural mean
Hit rate (CLIP)99.6498.5885.1596.7698.6892.6695.25
Hit rate (SigLIP2)99.7699.3298.3597.4899.8495.7098.41
Vanilla ICL (CLIP)72.1376.1342.0330.4179.1638.1056.33
Meta-ICL (CLIP)96.4488.5654.3760.8284.4257.3173.65
CLOUD (CLIP)96.9490.2256.2065.0787.1459.5875.86
Vanilla ICL (SigLIP2)72.6076.3243.2331.1278.6238.1156.67
Meta-ICL (SigLIP2)96.2689.5955.1860.5585.2056.8773.94
CLOUD (SigLIP2)96.8390.4255.7263.4386.8957.8875.19

3 specialized benchmarks — Accuracy / Hit rate (%)

Retriever / MethodCropDiseaseHyper-KvasirMVTecSpecialized mean
Hit rate (CLIP)96.5695.1296.6696.12
Hit rate (SigLIP2)97.0296.5996.8296.81
Vanilla ICL (CLIP)42.4630.1839.1137.25
Meta-ICL (CLIP)54.3743.0947.6948.39
CLOUD (CLIP)62.8947.8151.9954.23
Vanilla ICL (SigLIP2)43.1531.2440.8638.42
Meta-ICL (SigLIP2)55.8247.8149.9251.18
CLOUD (SigLIP2)63.0250.9753.1055.70

All entries are percentages. Under SigLIP2, CLOUD exceeds matched Meta-ICL by 1.25 percentage points on the 6 natural FGVC benchmarks and 4.52 points on the 3 specialized benchmarks. It improves over Meta-ICL on every dataset under both retrievers. These matched results demonstrate that CLOUD’s contribution extends beyond CLIP-specific retrieval: it improves the use of context supplied by either retriever. We have added the retraining setup and results to Appendix C.5 and Table 18 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer J55E — Part 2: CE + InfoNCE and Token-Level Negative Supervision

W2: CE + InfoNCE and Token-Level Negative Supervision

Thank you for this suggestion. We have added CE + InfoNCE under the same training and evaluation settings as Table 8, retaining positive CE supervision alongside whole-label contrastive learning.

ObjectiveFlowersFoodAircraftCUB-200CarsNABirdsMean (%)
Meta-ICL96.4488.5654.3760.8284.4257.3173.65
Whole-Label InfoNCE92.2384.6547.7053.4083.0053.0369.00
CE + InfoNCE96.6889.4855.1862.0185.5158.8074.61
CLOUD96.9490.2256.2065.0787.1459.5875.86

CE + InfoNCE improves mean accuracy to 74.61%, exceeding Whole-Label InfoNCE (69.00%) and CE-based Meta-ICL (73.65%). CLOUD further reaches 75.86% and outperforms CE + InfoNCE on all 6 natural FGVC benchmarks. Its advantage therefore persists when the contrastive baseline also includes positive CE supervision. We have added this comparison and the corresponding analysis to Section 5.6 and Table 8 of the revised manuscript. To explain the difference between the two negative objectives, we examine their local gradient contributions at supervised tokens in the unclipped region. For a negative candidate, define:

pc,t=Pθ(ct∣X,c<t),sc=1Tc∑tlog⁡pc,t,wc=esc/τ∑j∈y∪Nesj/τ.p_{c,t}=P_\theta(c_t\mid X,c_{<t}),\quad s_c=\frac{1}{T_c}\sum_t\log p_{c,t},\quad w_c=\frac{e^{s_c/\tau}}{\sum_{j\in\\{y\\}\cup\mathcal{N}}e^{s_j/\tau}}.

InfoNCE and CLOUD use the following objectives. The sum in CLOUD runs over the supervised positions:

LNCE=−log⁡wy,Uc=−1mc∑t∈Sclog⁡(1−pc,t),mc=∣Sc∣.\mathcal{L}_{\mathrm{NCE}}=-\log w_y,\qquad \mathcal{U}_c=-\frac{1}{m_c}\sum_{t\in S_c}\log(1-p_{c,t}),\qquad m_c=|S_c|.

Omitting outer loss and candidate-averaging coefficients, the local negative-branch contributions to the negative token’s own logit are

gc,tNCE=wcτTc(1−pc,t),gc,tCLOUD=pc,tmc.g^{\mathrm{NCE}}_{c,t}=\frac{w_c}{\tau T_c}(1-p_{c,t}),\qquad g^{\mathrm{CLOUD}}_{c,t}=\frac{p_{c,t}}{m_c}.

First, CLOUD applies stronger direct suppression to high-probability negative tokens, whereas InfoNCE assigns larger own-logit suppression gradients to low-probability tokens within the same negative candidate. For two supervised positions with probabilities 0.9 and 0.1, the gradient ratios are 9:1 for CLOUD and 1:9 for InfoNCE. CLOUD therefore directs stronger token-level suppression toward positions where the model still strongly supports an incorrect candidate.

Second, CLOUD maintains direct supervision at other positions when one token already has low probability. Holding other token probabilities and candidate scores fixed, lowering one token probability reduces the sequence score and its InfoNCE weight, weakening InfoNCE’s negative gradients at the candidate’s remaining positions. CLOUD’s local gradient at each supervised position depends on that position’s own probability, so a low probability elsewhere does not attenuate its direct suppression. These expressions describe local negative-branch contributions; parameter sharing and positive CE supervision jointly determine the full parameter update.

In fine-grained classification, closely related categories often have linguistically plausible multi-token names. A few low-probability tokens can lower an incorrect label’s overall sequence score without adequately constraining its remaining high-probability continuations. CLOUD avoids relying solely on such local probability bottlenecks and maintains direct suppression at positions within confusable candidates that the model still strongly supports. Its gains over CE + InfoNCE reach 3.06 percentage points on CUB-200 and 1.63 points on Cars. The matched comparison and gradient analysis jointly support the value of token-level negative supervision beyond adding sequence-level contrast.

AuthorsOpenReview ↗

Response to Reviewer J55E — Part 3: Magnitude and Consistency of the Gains

W3: Magnitude and Consistency of the Gains

Thank you for your comments. The gains vary across datasets, and their practical significance is clearest when considered alongside baseline accuracy, domain transfer, and the matched computational setting.

1. Smaller absolute gains on near-saturated datasets coexist with larger gains on harder benchmarks. On Flowers, Qwen3-VL-2B improves from 96.44% to 96.94%, reducing the remaining error by 14.0%; Qwen3-VL-4B improves from 96.10% to 97.43%, a 34.1% relative error reduction. On less saturated benchmarks, the absolute gains are larger: Qwen3-VL-2B improves by 4.25 percentage points on CUB-200 and 2.72 points on Cars, while Qwen3-VL-4B improves by 3.90 points on Aircraft (Table 1). The small gains on some datasets therefore do not characterize the method’s benefit on the more challenging benchmarks.

2. Gains increase under domain transfer and conflicting contextual bindings. Across Qwen3-VL-2B, Qwen3-VL-4B, and Gemma3-4B, mean gains over Meta-ICL rise from 2.21/1.74/1.76 percentage points on the 6 natural FGVC benchmarks to 5.84/6.11/2.14 points on the 3 specialized benchmarks (Tables 1–2). Under Counterfactual Binding, CLOUD reaches 43.62% versus 31.07% for Meta-ICL, a 12.55-point gain (Table 7). This deliberately conflicting setting tests whether the model follows demonstration-defined visual bindings despite familiar category semantics. These results show that the benefit extends to cross-domain discrimination and strengthens when evaluating fidelity to conflicting contextual bindings.

3. The gains are consistent under matched resources and across training seeds. CLOUD and Meta-ICL use the same backbone, 5,549 source training pairs, and evaluation contexts, with no additional inference procedure or annotations. CLOUD improves on every dataset across all three trained backbones in Tables 1–2. The three-training-seed comparison in Section 5.1 and Table 3 further gives 76.08% versus 73.69% on the natural benchmarks and 54.10% versus 48.52% on the specialized benchmarks. The advantage also persists under Free Generation: 74.06% versus 73.06% on the 6 natural FGVC benchmarks (Table 6). These results establish consistent gains from the training objective across domains, training runs, and prediction protocols, without increasing inference cost. Per-seed results are available in Appendix C.1 and Table 14 of the revised manuscript.

Reviewer_2anDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a meta-training objective named CLOUD for fine-grained in-context classification. This method combines cross-entropy supervision for correct labels and non-likelihood penalty for incorrect context labels, and excludes shared token prefixes to avoid gradient conflicts. The authors conduct meta-training on 5,549 query-context pairs in two source datasets and evaluate it on nine retained benchmark tests using three backbone configurations. CLOUD consistently outperforms Meta-ICL based on cross-entropy and shows more significant improvements in professional domains.

Critical strengths
  1. The paper focuses on improving the discriminative use of visual demonstrations rather than merely optimizing demonstration selection. Its candidate-score analysis provides a concrete motivation for explicit negative supervision in fine-grained classification.
  2. CLOUD is evaluated on three backbone configurations and nine held-out datasets, with matched training and evaluation settings relative to CE-based Meta-ICL. The improvements across datasets, retrieval strategies, and prediction protocols support the effectiveness of the proposed objective.
Critical weaknesses
  1. The method is of limited innovation, and the actual contribution of the prefix gating mechanism is not yet clear. The proposed negative supervision is closely related to the existing Unlikelihood Training, while the shared prefix gating mechanism is a relatively direct extension. Moreover, the prefix gating ablation in Table 7 only uses synthetic hierarchical labels and lacks validation in the original label space, as well as comparison with stronger contrastive learning methods combined with CE.
  2. The experiments are not sufficient to fully prove the improvement in the ability to utilize comparative visual context. CLIP retrieval has included the correct category in the context of most test samples, but the paper lacks a control experiment to distinguish the retrieval prior, label information, and the effect of visual context. Additionally, the attention analysis did not use the same error candidate categories across different models. Therefore, it is difficult to determine whether CLOUD truly enhances the visual comparative ability or mainly reduces the scores of incorrect categories.
Questions for authors
  1. The proposed negative supervision is closely related to existing unlikelihood training. Could the authors further clarify the fundamental methodological differences and the specific novelty of CLOUD beyond prefix-aware masking?
  2. How frequently do target and distractor labels share token prefixes during standard meta-training? What proportion of negative samples actually activates the proposed prefix-gating mechanism?
  3. Table 7 compares CLOUD with a standalone whole-label InfoNCE objective. Could the authors also evaluate CE combined with InfoNCE or other standard contrastive objectives under matched training settings?
  4. CLOUD penalizes all in-context distractor labels equally. Have the authors investigated whether focusing on the hardest negatives or using score-dependent negative weighting could achieve comparable or better results?
13 comments
AuthorsOpenReview ↗

Response to Reviewer 2anD — Part 1: Methodological Novelty and Diagnostic Evidence

We sincerely thank you for your constructive feedback and for recognizing our focus on the discriminative use of visual demonstrations, the candidate-score motivation, and the consistent improvements across three backbones and nine held-out benchmarks under matched settings. Below, we address your concerns and questions in detail with additional theoretical analyses and control experiments—covering methodological novelty, prefix gating in the original label space, the CE + InfoNCE control, disentangling visual context from retrieval priors, and negative weighting strategies.

 

W1 & Q1: Methodological Distinctions from UL and Core Contributions Beyond Prefix Gating

Thank you for your constructive comments. Although CLOUD and unlikelihood training (UL) locally use the same −log⁡(1−p)-\log(1-p) operator, this specific functional form (along with prefix gating, which we ablate in W1 & Q2 below) is chosen to serve our broader modeling objective. The two methods address fundamentally different problems with distinct modeling principles. Below we clarify our methodological distinctions beyond prefix gating and summarize our main contributions:

1. Methodological distinction from standard UL (Multimodal Multi-Candidate Competition vs. Text-Only Single-Sequence Repetition Penalty)

a. Different task domains, modeling goals, and supervision paths: Standard UL targets open-ended text generation, with negative supervision primarily addressing repetition and other degenerate generation behaviors. CLOUD instead targets multimodal fine-grained visual Meta-ICL: its negative candidates come from competing classes in the current support set, and its supervision aims to resolve visual confusion between the query image and these contextual alternatives. It is built on the idea that distinguishing visually confusable categories requires recognizing not just what a query image is, but what it isn't among in-context alternatives. Accordingly, CLOUD evaluates multiple competing candidate labels along their own respective autoregressive histories (c<tc_{<t}) in parallel, directly suppressing the distractor trajectories that compete with the target at test time.

b. Leveraging the dual role of multimodal demonstrations: Without extra data augmentation, CoT annotations, architectural changes, or additional inference stages, CLOUD directly uses the retrieved multimodal demonstrations as both positive visual references and the negative candidate set to rule out. Within this multi-trajectory setup, prefix gating simply prevents positive–negative gradient conflicts on shared prefixes among multi-token class names (Eq. 4).

2. Diagnostic contribution: Why does standard CE meta-training struggle with fine-grained discrimination?

CLOUD is motivated by two empirical findings on why standard CE Meta-ICL falls short in fine-grained ranking (Fig. 1, Appendix Tables 9–10):

a. Target–distractor score co-inflation: Supervising only the ground-truth label raises target log-scores by 0.561 nats on average, but simultaneously raises the strongest in-context distractor by 0.247 nats, narrowing the net gain in discriminative margin (Fig. 1(A)).

b. Identifying and reversing candidate-biased attention: When each model scores its own hardest incorrect in-context candidate, Meta-ICL allocates 41.0% of its demonstration attention to that candidate’s own example, whereas CLOUD allocates 46.7% to the GT example (Table 9). In a separate blocking analysis using fixed in-context negatives—the first two non-GT labels in demonstration order—blocking the scored candidate’s own demonstration lowers its score by 1.72 nats for Meta-ICL but only 0.30 nats for CLOUD (Table 10). These complementary analyses show greater attention to GT evidence and weaker dependence on the incorrect candidate’s matching example. The matched-candidate controls in Tables 9–10 retain both patterns when the same incorrect labels are scored across models, supporting CLOUD’s comparative use of evidence across demonstrations. Consistent with this pattern, the discriminative margin increases from 0.978 to 2.213 nats, reaching 2.26 times the Meta-ICL margin (Fig. 1(A)).

Continued in Part 2: training-side contributions and cross-domain generalization.

AuthorsOpenReview ↗

Response to Reviewer 2anD — Part 2: Training-Side Contributions and Cross-Domain Generalization

W1 & Q1: Methodological Distinctions and Core Contributions (continued)

3. Distinction from prior fine-grained ICL work (Training-Side Comparison & Cross-Domain Meta-Generalization):

a. Training models to compare demonstrations: Most fine-grained ICL studies focus on retrieval strategies to select better examples, yet providing similar examples does not guarantee that the model can tell subtle differences apart. We address the complementary training-side question: how to train models to compare multiple demonstrations jointly rather than matching each candidate in isolation when support–query alignment is imperfect.

b. Cross-dataset and cross-domain meta-generalization without extra overhead: Recent training-based fine-grained MLLM methods (e.g., those combining CoT supervision with RL rollouts) require extra annotations and longer decoding, and mainly test base-to-new generalization within the training datasets. Using no CoT annotations or extra inference steps, we evaluate cross-dataset and cross-domain meta-generalization: models are meta-trained on only two small source datasets (pets and textures) and tested with frozen weights across nine unseen datasets and specialized domains.

4. Cross-domain generalization with high data efficiency and zero inference overhead (2B Outperforming 32B Using 5,549 Pairs):

CLOUD keeps the exact baseline inference procedure with zero extra cost. Meta-trained on 5,549 query–context pairs from two source datasets, CLOUD consistently improves three backbones across nine unseen benchmarks—gaining +1.74–2.21 percentage points on six natural FGVC datasets and +2.14–6.11 percentage points across three specialized domains with large distribution shifts (plant pathology, medical endoscopy, and industrial defects). Overall, CLOUD-2B outperforms Vanilla ICL-32B by 4.63 percentage points on average across all nine benchmarks, while showing stronger context reliance under scrambled labels (+5.49 percentage points) and counterfactual binding (+12.55 percentage points).

Overall, our work provides a clean, non-engineering-heavy solution for fine-grained multimodal ICL: incorporating competition among contextual candidates into a simple training objective improves cross-demonstration discrimination, supported by attention and intervention analyses, while enabling data-efficient cross-domain generalization without additional inference overhead.

AuthorsOpenReview ↗

Response to Reviewer 2anD — Part 3: Prefix-Gating Frequency and Original-Label Validation

W1 & Q2: Prefix-Gating Frequency and Original-Label Validation

Thank you for your thoughtful question. In autoregressive generation, a distractor label and the ground-truth label share the same conditional token probabilities at positions within their common prefix. Without prefix gating, positive CE supervision and the negative penalty act on the same shared-prefix probability ptp_t within one training episode. Their loss-gradient contributions are, respectively,

$ -a\nabla_{\theta}\log p_t \qquad\text{and}\qquad -b\nabla_{\theta}\log(1-p_t). $

Here, a,b>0a,b>0 include the corresponding weights and normalization factors. These contributions oppose each other in the unclipped region, creating a direct optimization conflict, as formalized in Section 3.3, Eq. 4. Prefix gating excludes these shared-prefix positions from negative supervision while retaining their positive CE supervision.

Regarding the trigger frequency, we compute the exact statistics on our standard meta-training set (Pets + DTD: 5,549 episodes, 16,875 in-context negative instances, and 50,740 negative tokens). In the original label space, 2.3% of negative instances (387 / 16,875, affecting 7.0% of training queries) share a token prefix with the target label and trigger the prefix gate, masking 0.8% of all negative tokens (387 / 50,740). Shared-prefix conflicts therefore occur infrequently under the standard labels. We use a controlled synthetic shared-prefix stress test in Table 8 to increase the frequency of these conflicts during training and more directly examine the role of gating, while keeping the source images and downstream evaluation protocol unchanged. The additional original-label experiments establish its practical contribution under the standard setting.

Following your suggestion, we provide direct validation in the original label space, including a single run with training seed 42 and three runs with training seeds 41, 42, and 43, and evaluate synthetic prefix-conflict levels of 25%–100% on the six natural and three specialized benchmarks. We have added the motivation for the stress test to Section 5.6 and the trigger statistics, original-label validation, and comparisons across conflict rates to Appendix C.8 (Tables 20–22) of the revised manuscript.

Original-label validation: accuracy (%). Single run uses training seed 42; three-seed results use training seeds 41, 42, and 43.

BenchmarksTraining runsCLOUD w/ gateCLOUD w/o gateGain (pp)
6 naturalSingle run75.8675.54+0.32
6 natural3 seeds76.08 ± 0.4975.66 ± 0.60+0.42
3 specializedSingle run54.2353.47+0.76
3 specialized3 seeds54.10 ± 0.7853.13 ± 1.53+0.96

Lower accuracy and greater cross-seed variability without gating. Under the original labels, removing prefix gating lowers mean accuracy for every training seed on both benchmark groups. The ungated variant trails gated CLOUD by 0.42 percentage points on the 6 natural benchmarks and 0.96 points on the 3 specialized benchmarks. On the specialized benchmarks, its cross-seed standard deviation is nearly twice as large (1.53 vs. 0.78), showing less stable performance across training runs even when shared-prefix conflicts are infrequent.

Continued in Part 4: controlled shared-prefix stress tests and trigger statistics.

Reviewer_vRv7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The model proposed in the paper builds upon the cross-entropy objective by imposing a negative constraint on the generation probabilities of incorrect candidate labels within the context. The proposed method yields significant performance gains across specialized domains, including agriculture, medicine, and industry.

Critical strengths
  1. The model proposed in the paper builds upon the cross-entropy objective by imposing a negative constraint on the generation probabilities of incorrect candidate labels within the context.
  2. The proposed method yields significant performance gains across specialized domains, including agriculture, medicine, and industry.
  3. The ablation study is comprehensive, effectively isolating the individual contributions of the bounded penalty and the prefix gating mechanism.
  4. The 2B-parameter Qwen3-VL CLOUD model outperforms the 32B-parameter Qwen3-VL (using vanilla ICL) on average across nine datasets.
Critical weaknesses
  1. Related approaches—such as CIRCLES, ContextNav, Coreset Optimization, DARA, and Fine-R1—are direct competitors; however, the paper does not provide comparisons under identical experimental settings.
  2. The primary results (Tables 1 and 2) are based on single-run experiments, with no variance reported from multi-seed evaluations.
  3. Table 3 indicates that the performance advantage of CLOUD over Meta-ICL diminishes as the number of examples per class increases, suggesting that CLOUD’s benefits may be concentrated in low-shot scenarios.
  4. Incorporating supervision for incorrect candidate labels may increase the computational cost of the forward and backward passes; the paper should explicitly report the actual training time, VRAM usage, and additional computational overhead.
  5. While the 2B-parameter Qwen3-VL CLOUD outperforms the 32B-parameter Qwen3-VL (vanilla ICL) across nine datasets, the paper does not report results for a 32B-parameter version of Qwen3-VL CLOUD.
  6. Since the primary tasks rely on a predefined set of categories for scoring, the method should also be evaluated in scenarios involving unknown categories or open-set settings.
  7. The paper should provide clear data on actual training time, inference time, VRAM usage, and additional computational overhead, and include further comparisons with other mainstream methods.
Questions for authors

For specific details, please refer to points 1, 2, 4, 5, and 7 under "Critical Weaknesses."

6 comments
AuthorsOpenReview ↗

Response to Reviewer vRv7 — Part 1: Related Work and Baselines

We sincerely thank you for your time, constructive feedback, and thoughtful review. We are very encouraged by your positive recognition of our core training formulation (imposing negative constraints on in-context distractor labels), our significant performance gains across specialized domains (agriculture, medicine, and industry), our comprehensive ablation study isolating the bounded penalty and prefix gating mechanisms, and the strong empirical result that our 2B Qwen3-VL CLOUD model outperforms the 32B Qwen3-VL model (using Vanilla ICL) on average across nine datasets.

Below, we address your points under Critical Weaknesses and Questions in detail—providing clear positioning against related works, statistical stability analyses, high-shot scaling explanations, training-time and memory profiling, clarification of inference overhead, and discussions on model scaling and open-set scenarios. The revised manuscript includes the corresponding clarifications and results.

 

W1: Comparison with Recent Multimodal ICL and Fine-Grained Adaptation Works

Thank you for your thoughtful comments. We discuss these related works in Sections 2.1–2.2. To ensure strict single-variable control isolating the effect of the training objective, we summarize how these methods differ from CLOUD in pipeline stage and evaluation regime below:

Method familyTarget stageExtra supervision / architecture & inference overheadGeneralization regimeRelation to CLOUD
CIRCLES, ContextNav, Coreset Opt.Demonstration retrieval & context constructionAttribute retrieval, agentic refinement, or coreset searchTraining-free ICLComplementary focus; retrieval-protocol results in Table 5
DARA, Fine-R1Attention reallocation (DARA); CoT and RL training (Fine-R1)Learnable attention-reallocation parameters (DARA); synthesized CoT rationales and reasoning training (Fine-R1)DARA includes transfer to untuned tasks; Fine-R1 evaluates base-to-new generalizationAttention adjustment or reasoning supervision; CLOUD focuses on a competitive objective over class labels
Meta-ICL, Whole-Label InfoNCE, CLOUDMeta-training with class-name supervisionZero inference overhead; original architecture and readoutCross-dataset / cross-domain transfer: Pets & DTD to 9 unseen benchmarksMatched comparisons in Tables 1–8

Complementary roles of context construction and training objectives. CIRCLES, ContextNav, and Coreset Opt. focus on constructing the demonstration context, while CLOUD improves the model’s discriminative use of the supplied context. Table 5 evaluates CLOUD under three retrieval protocols—C-R, C-N, and I-kNN—with identical contexts for CLOUD and Meta-ICL within each protocol. CLOUD improves over Meta-ICL by 2.21, 2.63, and 2.61 percentage points, respectively, showing that its gains persist across different context-construction strategies.

DARA and Fine-R1 improve model adaptation through attention reallocation and reasoning-oriented training, respectively. DARA introduces learnable attention-reallocation parameters to improve visual context utilization, while Fine-R1 uses synthesized CoT rationales and reinforcement learning for fine-grained recognition. CLOUD focuses on the training objective: it introduces explicit negative supervision over competing contextual labels, using class-name supervision without adding attention-reallocation parameters or CoT annotations, and retains the original inference procedure. We therefore compare against CE-based Meta-ICL and alternative objectives under matched backbones, source data, contexts, and inference protocols to isolate the contribution of distractor supervision. Section 4.1 of the revised manuscript clarifies this baseline selection.

AuthorsOpenReview ↗

Response to Reviewer vRv7 — Part 2: Multi-Seed Results and Analysis of More Demonstrations

W2: Multi-Seed Evaluation and Statistical Stability

Thank you for this valuable suggestion. We train Meta-ICL and CLOUD (Qwen3-VL-2B) using three training seeds—41, 42, and 43—with training contexts and evaluation demonstrations held fixed across runs. CLOUD outperforms Meta-ICL under all three training seeds on both the natural and specialized benchmark averages. Its mean accuracy is 76.08 ± 0.49% on the six natural benchmarks and 54.10 ± 0.78% on the three specialized benchmarks, compared with 73.69 ± 0.13% and 48.52 ± 1.06% for Meta-ICL. These correspond to mean gains of 2.39 and 5.57 percentage points, respectively.

We report the individual runs and mean ± sample standard deviation below. We have added the summary to Section 5.1 and Table 3 of the revised manuscript, with per-dataset results in Appendix C.1 and Table 14.

6 natural FGVC benchmarks — accuracy (%).

MethodTraining seedFlowersFoodAircraftCUB-200CarsNABirdsMean
Meta-ICLseed 4196.2888.6055.0660.9884.5957.5673.84
Meta-ICLseed 4296.4488.5654.3760.8284.4257.3173.65
Meta-ICLseed 4396.4488.2154.4060.7984.3357.3573.59
Meta-ICLMean ± Std96.38±0.0988.46±0.2254.61±0.3960.86±0.1084.45±0.1357.41±0.1373.69±0.13
CLOUDseed 4197.0490.0856.9864.0586.9959.3475.74
CLOUDseed 4296.9490.2256.2065.0787.1459.5875.86
CLOUDseed 4397.1790.5757.5565.8387.8260.9376.64
CLOUDMean ± Std97.05±0.1190.29±0.2556.91±0.6864.98±0.8987.32±0.4459.95±0.8676.08±0.49

3 specialized benchmarks — accuracy (%).

MethodTraining seedCropDiseaseHyper-KvasirMVTecMean
Meta-ICLseed 4156.1543.0349.7649.65
Meta-ICLseed 4254.3743.0947.6948.39
Meta-ICLseed 4355.0641.7845.7947.54
Meta-ICLMean ± Std55.19 ± 0.9042.63 ± 0.7447.75 ± 1.9948.52 ± 1.06
CLOUDseed 4163.3145.4351.0353.26
CLOUDseed 4262.8947.8151.9954.23
CLOUDseed 4364.5249.1650.7254.80
CLOUDMean ± Std63.57 ± 0.8447.47 ± 1.8851.25 ± 0.6654.10 ± 0.78

 

W3: Performance with More Demonstrations

The smaller average gain at higher shots does not mean that CLOUD is effective only with fewer demonstrations. To examine this trend, we add detailed per-dataset results for all 6 context configurations in Appendix C.4, Table 17, of the revised manuscript. These results distinguish two factors:

1. Remaining headroom varies across datasets. At fixed K = 4, increasing demonstrations per class reduces the average gain, but some datasets approach saturation: Meta-ICL reaches 98.34% on Flowers at 3-shot. On more challenging datasets, CLOUD continues to deliver clear improvements. At 2-shot and 3-shot, it improves CUB-200 by 1.86 and 1.07 percentage points and NABirds by 1.80 and 1.92 points, respectively. Thus, the smaller average gain coexists with substantial gains on harder datasets.

2. Stronger class competition increases CLOUD’s advantage. More demonstrations do not necessarily lead to smaller gains. At fixed M = 1, increasing K adds more competing classes, and CLOUD’s average advantage grows from 1.59 percentage points at 2-way to 3.80 points at 8-way. At 8-way, the gains reach 4.95 points on Aircraft and 6.04 points on CUB-200. This directly supports the value of explicit distractor supervision when the context presents stronger class competition, rather than a benefit restricted to small contexts.

3. More effective use of demonstrations under resource constraints. Adding image demonstrations increases visual token counts and requires more computation and memory to process longer contexts. CLOUD achieves 75.86% with 4-way 1-shot contexts, approaching the 76.61% of Meta-ICL with 4-way 2-shot contexts while using half as many demonstrations. This highlights CLOUD’s practical value: stronger fine-grained discrimination with fewer demonstrations and a lower context-processing burden.

AuthorsOpenReview ↗

Response to Reviewer vRv7 — Part 3: Computational Cost and the 32B Comparison

W4: Training Time, Memory, and Computational Overhead

Thank you for your question. Compared with CE-based Meta-ICL, CLOUD increases total training time by only approximately 1 minute (3.5%). The measured per-step peak allocated memory increases by an average of 5.32 MB per GPU, with the p99 of the paired differences equal to 10.35 MB. We have added the computational-cost summary to Section 3.4 and report detailed timing and memory measurements in Appendix C.2 (Table 15) of the revised manuscript.

The table below reports profiling results on eight H20 GPUs, using the default 4-way 1-shot setup and 3 training epochs on 5,549 source query–context pairs.

Efficiency & memory metricMeta-ICLCLOUD
Training scale (epochs / effective samples / steps per rank)3 / 16,632 / 2,0793 / 16,632 / 2,079
Total training wall-clock time≈ 28.50 min (3.8 GPU·h)≈ 29.50 min (3.9 GPU·h)
Wall-clock / step compute time (s/step, p50)0.82 / 0.640.85 / 0.67
Stage breakdown p50 (data / prefill / replay / backward)0.20 / 0.13 / 0.08 / 0.22 s0.23 / 0.13 / 0.08 / 0.22 s
Per-GPU peak allocated memory, mean (MB)17,232.0917,237.41
Per-GPU peak allocated memory, p99 (MB)24,873.2124,876.87

The table reports absolute per-GPU, per-step peak allocated memory in MB. The differences above are paired by GPU and training step (CLOUD − Meta-ICL), covering the forward and backward passes but excluding the optimizer update.

CLOUD reuses shared-context KV states across candidate labels, avoiding repeated encoding of the multimodal context. It introduces no additional model parameters or inference steps and retains the same inference procedure as Meta-ICL.

 

W5: Role of the 32B Model Comparison

Thank you for your suggestion. We include Qwen3-VL-32B as a larger-model reference to examine whether improving context utilization through meta-training can enable compact models to outperform a much larger model used directly for ICL. Across the 9 benchmarks in Tables 1–2, CLOUD with Qwen3-VL-2B exceeds 32B Vanilla ICL by 4.63 percentage points on average, using only 1/16 as many parameters. This comparison highlights the practical value of targeted meta-training for compact models.

To isolate the contribution of CLOUD’s training objective, we conduct matched comparisons against CE-based Meta-ICL on Qwen3-VL-2B, Qwen3-VL-4B, and Gemma3-4B. Across these backbones, CLOUD improves mean accuracy by 1.74–2.21 percentage points on the 6 natural FGVC benchmarks and 2.14–6.11 points on the 3 specialized benchmarks, supporting its effectiveness across the evaluated model families and scales.

Given this experimental focus and the resource demands of multi-image meta-training at the 32B scale, we retain Qwen3-VL-32B as a reference without task-specific meta-training and focus the controlled training comparisons on the three compact backbones.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary

Dear Area Chairs and Reviewers,

As the rebuttal period draws to a close, we sincerely thank you for your time, careful consideration, and constructive feedback. We appreciate the opportunity to clarify CLOUD’s core ideas and provide additional experiments and analyses addressing your concerns. We hope this discussion brings into clearer focus the distinctive contribution of our work and its practical value for multimodal in-context learning research.

1. From selecting useful demonstrations to learning how to use them more effectively. Much of multimodal ICL research focuses on selecting better demonstrations, while how to train models to make better use of the supplied context remains comparatively underexplored. CLOUD addresses this question by strengthening discrimination among visually confusable alternatives within the same context. Our diagnostics reveal an interesting phenomenon: positive-only meta-training raises both target and distractor scores, offsetting part of the discriminative gain. CLOUD turns the context itself into comparative supervision: demonstrations serve as both positive visual references and competing classes to reject. This brings “not just what it is, but what it isn’t” into the training objective.

2. An objective designed to realize contextual competition. CLOUD and unlikelihood training share only the mathematical form of the local penalty operator; their task settings, negative-candidate construction, and supervision paths differ. Our contribution centers on dynamic, support-conditioned competition among autoregressive class-label trajectories. The final loss formulation serves this modeling goal from several complementary perspectives: token-level supervision directly suppresses highly supported incorrect continuations; its local gradients avoid letting a single low-probability token weaken suppression elsewhere in the candidate; and prefix gating preserves positive supervision while preventing conflicting gradients on shared prefixes. The token-dependent penalty also permits simple candidate averaging while emphasizing high-probability negative tokens. The added CE + InfoNCE comparison, aggregation studies, and original-label gating ablations support these choices as a coherent realization of the core idea.

3. A new diagnostic perspective on multimodal context utilization. Beyond reporting accuracy, our work examines how training changes candidate scores, attention allocation, and dependence on contextual evidence. Label-only controls, matched-candidate comparisons, attention blocking, and counterfactual bindings jointly support more effective cross-demonstration evidence use, beyond retrieved-label priors alone. Together, these analyses provide a new diagnostic perspective and controlled analysis approach for studying whether multimodal models compare contextual evidence or rely on isolated candidate matching.

4. Broad empirical evidence with a simple, practical method. CLOUD improves three backbones across nine unseen benchmarks after meta-training on only 5,549 pairs from two source datasets. Additional experiments support robustness across training seeds and independently retrained SigLIP2 settings. We further examine absolute accuracy levels and relative improvements across datasets, clarifying CLOUD’s greater advantages in more challenging settings. These evaluations contribute evidence about when comparative supervision matters and how it transfers across domains. The method’s appeal is its simplicity: it uses the demonstrations and class labels already available, requires no extra annotations or architectural changes, and adds only approximately 1 minute (3.5%) to training time, with no additional inference overhead. CLOUD offers a simple, lightweight training objective rather than a stack of engineering additions, complementing advances in retrieval and context construction.

The revised manuscript incorporates the corresponding results and clarifications, with changes highlighted in blue. We sincerely thank all reviewers for your constructive feedback and thoughtful engagement throughout this discussion. We appreciate your recognition of our focus on improving visual context utilization, the consistent gains across datasets and specialized domains, and the value of the diagnostic analyses and ablation studies. Your questions and suggestions help us strengthen the evidence and present our contributions more clearly.

Thank you again for your time and thoughtful consideration. If any questions or concerns remain, we would welcome the opportunity to address them as thoroughly as possible during the remaining discussion period. We sincerely appreciate your efforts throughout this process and wish you all the best.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.