- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Brief Summary: The paper tackles the task of fine-grained classification. The authors find naive cross-entropy loss isn't enough, instead propose their method "CLOUD" which gives additional negative signal to distractor labels, and only give the negative to the non-shared suffix tokens. The high-level idea when answering is to use CLIP to get initial set of candidate labels. Then for each label, ask the model the model the question and get the probability scores for the output labels to answer. Experiments on multiple fine-grained datasets (flowers, food, aircraft, cub-200, cars etc) show consistent improvement over competitive Meta-ICL baselines.
Critical strengths
Pros:
-
The authors study an important problem of fine-grained classification and show consistent improvements over Meta-ICL across datasets showing that negative signal on close classes can be helpful.
-
The authors provide a number of visualizations, and detailed ablation study.
-
The authors show that training on just Oxford Pets and DTD, they can generalize to a bunch of other fine-grained datasets with huge improvements over vanilla!
Critical weaknesses
Cons:
-
My main concern is that for training, the authors are using CLIP for centroid. So if CLIP has some wrong adjacent words, they would give additional negative signal which may not be intended. The paper heavily relies on CLIP. The authors should show results with other models like Siglip2.
-
I think the authors are missing an important ablation in Table 7 of InfoNCE + CE. The main idea of the paper is to give additional negative signals, but that is already covered by InfoNCE, but the separate signal for correct from CE. Right now it is unclear if we really need the token-wise negative signal.
-
Overall improvements on many datasets over Meta-ICL are somewhat incremental though overall positive.
Questions for authors
see weakness.
Q1. (Minor) it is unclear how generalizable this method to other work streams outside of fine-grained image classifications.
6 comments
Response to Reviewer J55E — Part 2: CE + InfoNCE and Token-Level Negative Supervision
W2: CE + InfoNCE and Token-Level Negative Supervision
Thank you for this suggestion. We have added CE + InfoNCE under the same training and evaluation settings as Table 8, retaining positive CE supervision alongside whole-label contrastive learning.
| Objective | Flowers | Food | Aircraft | CUB-200 | Cars | NABirds | Mean (%) |
|---|---|---|---|---|---|---|---|
| Meta-ICL | 96.44 | 88.56 | 54.37 | 60.82 | 84.42 | 57.31 | 73.65 |
| Whole-Label InfoNCE | 92.23 | 84.65 | 47.70 | 53.40 | 83.00 | 53.03 | 69.00 |
| CE + InfoNCE | 96.68 | 89.48 | 55.18 | 62.01 | 85.51 | 58.80 | 74.61 |
| CLOUD | 96.94 | 90.22 | 56.20 | 65.07 | 87.14 | 59.58 | 75.86 |
CE + InfoNCE improves mean accuracy to 74.61%, exceeding Whole-Label InfoNCE (69.00%) and CE-based Meta-ICL (73.65%). CLOUD further reaches 75.86% and outperforms CE + InfoNCE on all 6 natural FGVC benchmarks. Its advantage therefore persists when the contrastive baseline also includes positive CE supervision. We have added this comparison and the corresponding analysis to Section 5.6 and Table 8 of the revised manuscript. To explain the difference between the two negative objectives, we examine their local gradient contributions at supervised tokens in the unclipped region. For a negative candidate, define:
InfoNCE and CLOUD use the following objectives. The sum in CLOUD runs over the supervised positions:
Omitting outer loss and candidate-averaging coefficients, the local negative-branch contributions to the negative token’s own logit are
First, CLOUD applies stronger direct suppression to high-probability negative tokens, whereas InfoNCE assigns larger own-logit suppression gradients to low-probability tokens within the same negative candidate. For two supervised positions with probabilities 0.9 and 0.1, the gradient ratios are 9:1 for CLOUD and 1:9 for InfoNCE. CLOUD therefore directs stronger token-level suppression toward positions where the model still strongly supports an incorrect candidate.
Second, CLOUD maintains direct supervision at other positions when one token already has low probability. Holding other token probabilities and candidate scores fixed, lowering one token probability reduces the sequence score and its InfoNCE weight, weakening InfoNCE’s negative gradients at the candidate’s remaining positions. CLOUD’s local gradient at each supervised position depends on that position’s own probability, so a low probability elsewhere does not attenuate its direct suppression. These expressions describe local negative-branch contributions; parameter sharing and positive CE supervision jointly determine the full parameter update.
In fine-grained classification, closely related categories often have linguistically plausible multi-token names. A few low-probability tokens can lower an incorrect label’s overall sequence score without adequately constraining its remaining high-probability continuations. CLOUD avoids relying solely on such local probability bottlenecks and maintains direct suppression at positions within confusable candidates that the model still strongly supports. Its gains over CE + InfoNCE reach 3.06 percentage points on CUB-200 and 1.63 points on Cars. The matched comparison and gradient analysis jointly support the value of token-level negative supervision beyond adding sequence-level contrast.
Response to Reviewer J55E — Part 3: Magnitude and Consistency of the Gains
W3: Magnitude and Consistency of the Gains
Thank you for your comments. The gains vary across datasets, and their practical significance is clearest when considered alongside baseline accuracy, domain transfer, and the matched computational setting.
1. Smaller absolute gains on near-saturated datasets coexist with larger gains on harder benchmarks. On Flowers, Qwen3-VL-2B improves from 96.44% to 96.94%, reducing the remaining error by 14.0%; Qwen3-VL-4B improves from 96.10% to 97.43%, a 34.1% relative error reduction. On less saturated benchmarks, the absolute gains are larger: Qwen3-VL-2B improves by 4.25 percentage points on CUB-200 and 2.72 points on Cars, while Qwen3-VL-4B improves by 3.90 points on Aircraft (Table 1). The small gains on some datasets therefore do not characterize the method’s benefit on the more challenging benchmarks.
2. Gains increase under domain transfer and conflicting contextual bindings. Across Qwen3-VL-2B, Qwen3-VL-4B, and Gemma3-4B, mean gains over Meta-ICL rise from 2.21/1.74/1.76 percentage points on the 6 natural FGVC benchmarks to 5.84/6.11/2.14 points on the 3 specialized benchmarks (Tables 1–2). Under Counterfactual Binding, CLOUD reaches 43.62% versus 31.07% for Meta-ICL, a 12.55-point gain (Table 7). This deliberately conflicting setting tests whether the model follows demonstration-defined visual bindings despite familiar category semantics. These results show that the benefit extends to cross-domain discrimination and strengthens when evaluating fidelity to conflicting contextual bindings.
3. The gains are consistent under matched resources and across training seeds. CLOUD and Meta-ICL use the same backbone, 5,549 source training pairs, and evaluation contexts, with no additional inference procedure or annotations. CLOUD improves on every dataset across all three trained backbones in Tables 1–2. The three-training-seed comparison in Section 5.1 and Table 3 further gives 76.08% versus 73.69% on the natural benchmarks and 54.10% versus 48.52% on the specialized benchmarks. The advantage also persists under Free Generation: 74.06% versus 73.06% on the 6 natural FGVC benchmarks (Table 6). These results establish consistent gains from the training objective across domains, training runs, and prediction protocols, without increasing inference cost. Per-seed results are available in Appendix C.1 and Table 14 of the revised manuscript.
Response to Reviewer J55E — Part 1: Retrieval, Negative Supervision, and SigLIP2
Thank you for your careful review and positive assessment of our results. We appreciate your recognition of the consistent improvements, diagnostic analyses, and transfer from Oxford Pets and DTD to unseen datasets. We address your concerns about retrieval-dependent supervision, the CE + InfoNCE comparison, the magnitude of the gains, and applicability beyond fine-grained classification below.
W1: Retrieval, Negative Supervision, and SigLIP2
Thank you for raising this concern. Retrieval selects contextual competitors, while ground-truth labels determine whether they receive positive or negative supervision. Let Yctx be the retrieved context labels and y∗ the query’s ground-truth label. Positive CE supervises y∗, including when retrieval misses it, and the negative set is N=Yctx∖y∗.
The true class is therefore explicitly excluded from negative supervision. A visually similar but different class is a valid distractor under the dataset’s labels; learning to reject such a competitor is the intended fine-grained discrimination task. Retrieval errors can change the relevance and difficulty of the context, but they do not turn the ground-truth class into a negative training target.
To test whether the gains depend on CLIP, we replace it with SigLIP2 ViT-L/16 for both meta-training context construction and downstream retrieval, and independently retrain Meta-ICL and CLOUD. We retain the same source data, Qwen3-VL-2B backbone, and optimization settings, and use identical evaluation contexts for the matched methods. This evaluates the complete training-and-evaluation pipeline with the new retriever. 6 natural FGVC benchmarks — Accuracy / Hit rate (%)
3 specialized benchmarks — Accuracy / Hit rate (%)
All entries are percentages. Under SigLIP2, CLOUD exceeds matched Meta-ICL by 1.25 percentage points on the 6 natural FGVC benchmarks and 4.52 points on the 3 specialized benchmarks. It improves over Meta-ICL on every dataset under both retrievers. These matched results demonstrate that CLOUD’s contribution extends beyond CLIP-specific retrieval: it improves the use of context supplied by either retriever. We have added the retraining setup and results to Appendix C.5 and Table 18 of the revised manuscript.