acceptodds
Under review as a conference paper at ICLR 2027

UVA: A UNIFIED VISION-LANGUAGE ALIGNMENT FRAMEWORK FOR FINE-GRAINED ZERO-SHOT AND FEW-SHOT CLASSIFICATION

Abstract

Vision-language models such as Contrastive Language-Image Pre-training (CLIP) achieve strong zero-shot recognition, yet they remain unreliable on fine-grained visual classification, where visually similar classes differ only in small, localised parts. Recent training-free methods address this by matching selected image regions against detailed class descriptions generated by large language models (LLMs). We show that this is insufficient: on CUB-200-2011, zero-shot region matching reaches only 61.17% accuracy, and accuracy falls to 0% on the images whose regions are least well grounded in their class descriptions. We introduce Unified Vision-Language Alignment (UVA), a framework built on a frozen CLIP ViT-B/16 that combines Attention-guided Bounding-box Selection (ABS), Bi-directional Fine-grained Text Alignment Description Refinement (BiFTA-DR), lightweight visual and textual residual adapters, and Weighted Cross-Alignment (WCA). The adapters are trained with a region-to-class-prototype contrastive objective that aligns local image regions with refined class descriptions. With 16 labelled images per class, UVA improves CUB-200-2011 accuracy from 56.02% (CLIP) to 80.29 ± 0.31% while training only 1.05M parameters, approximately 0.7% of the backbone. Across Oxford-IIIT Pet, ImageNet-100, and CUB-200- 2011, the benefit of adaptation increases with the difficulty of the zero-shot task. Finally, we show that agreement between WCA and Localized Zero-Shot Learning (LaZSL), an independent optimal-transport matcher, provides an unsupervised confidence signal: predictions on which the two agree are 28–41 percentage points more accurate than those on which they disagree.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Vision-Language Models, Fine-Grained Visual Classification, Zero-Shot Learning, Cross-Modal Alignment, Residual Adapters, Optimal Transport Verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yyBvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes UVA, a fine-grained vision-language classification framework. The method combines combines Attention-guided Bounding-box Selection (ABS), Bi-directional Fine-grained Text Alignment Description Refinement (BiFTA-DR), lightweight visual and textual residual adapters, and Weighted Cross-Alignment (WCA). The adapters are trained to improve the alignment between local visual regions and fine-grained textual descriptions. Experiments on Oxford-IIIT Pet, ImageNet-100, and CUB-200-2011 show the largest improvement on CUB, where UVA reaches 80.29% accuracy in the 16-shot setting compared with 61.17% for the zero-shot regional baseline. The paper further introduces a “Difficulty-Scaling Rule” based on the observation that adaptation gains increase as the zero-shot task becomes more difficult, and uses agreement between WCA and LaZSL as an unsupervised confidence signal.

Critical strengths
  1. The paper addresses an important problem in fine-grained vision-language recognition, where subtle local visual differences are difficult to align with textual descriptions.
  2. The method achieves a substantial accuracy improvement on CUB-200-2011, clearly outperforming the compared few-shot baselines.
Critical weaknesses
  1. Limited novelty: the method is largely a combination of existing components (ABS, BiFTA-DR, WCA) with standard residual adapters.
  2. Overclaimed “Difficulty-Scaling Rule”: the evidence is limited to three datasets, with a substantial gain only on CUB.
  3. Weak statistical reliability: the reported mean ± std is based on only two runs, which is insufficient for a robust estimate of variance.
  4. This article is overly reliant on AI and has a lot of hallucinations. For instance, the author cites the ABS method, which should stand for Attention Based Selection, but mistakenly uses attention guided bounding box selection.
Questions for authors
  1. Could you provide an ablation that more clearly isolates the contribution of the dual residual adapters from ABS, BiFTA-DR, and WCA?
  2. Could you provide additional evidence to support the proposed “Difficulty-Scaling Rule” beyond the three evaluated datasets?
  3. How many independent runs were used to compute the reported mean ± std? Could you report results over more runs?
2 comments
AuthorsOpenReview ↗

ABS expansion corrected, unsupported results removed and named in the AI Use Statement, novelty narrowed to residual calibration within ABS+WCA pending the global-feature control, “Difficulty-Scaling Rule” replaced by the observed gains, “± std” replaced by means with ranges (three draws per dataset planned), and component ablations ready to post if they finish.

Thank you for the review. We agree with several of your criticisms and have revised the paper to address them. Please also see our general response.

W4 (over-reliance on AI, e.g. the ABS expansion). You are right. ABS stands for Attention-Based Selection, not "Attention-guided Bounding-box Selection". The general response lists every error we found, including results that had no experiment record, and all of them have been removed. We checked each method description against the original papers and the public code, and the AI Use Statement now names these errors.

W1 (limited novelty). We agree that each component has a close precedent, and the contributions section now says so explicitly, citing CLIP-Adapter, Tip-Adapter and CLIP's contrastive loss. What remains is a narrower claim, a residual calibration of region and description embeddings inside an ABS+WCA pipeline. As Reviewer BvN7 points out, whether that is distinctive depends on the global-feature control. We have implemented it and will report it if it finishes during the discussion.

W2 / Q2 (the "Difficulty-Scaling Rule"). We agree that this was overclaimed and have removed it. The paper now only reports what we observed on three benchmarks, namely that the adapters add 0.33, 0.99 and 19.12 points over zero-shot ABS+WCA, and states clearly that three data points cannot establish a relationship (Section 4.2). We have no results on further datasets.

W3 / Q3 (only two runs). That is correct. Pets and CUB each have two runs, which differ in the random draw of training images, and ImageNet-100 has one. We have removed the "±\pm std" and now report means with ranges (CUB 79.98 to 80.60, Pets 93.00 to 93.54), and Appendix C explains how the draws were made. More runs require retraining on our cached features, and we plan at least three draws per dataset for the final version.

Q1 (an ablation separating the adapters from ABS, BiFTA-DR and WCA). Here is what we can show today. Region matching adds 5.15 points zero-shot on CUB (ABS+WCA vs CLIP), and BiFTA-DR adds 2.80 points on global features. The adapter step also changes the descriptions, but 96.3% of the CUB descriptions it uses are already in the zero-shot set. Running ABS+WCA with BiFTA-DR and no adapters, and running with only AvA_v or only AtA_t switched on at test time, needs only inference with the saved checkpoints. That code is ready, and we will post the results here if they complete.

This paper contains a large amount of AI-generated content and AI hallucinations, so I do not believe it meets ICLR’s submission standards. The authors’ response merely removed everything I had raised concerns about, which does not address my concern. I have therefore decided to maintain my score.

Reviewer_BvN7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that region-based and description-based CLIP classifiers are limited by misalignment between local region embeddings and LLM-generated descriptions. Building on a frozen CLIP with an ABS + WCA pipeline, UVA adds two residual adapters Av,AtA_v, A_t, blended as r~i=λAv(ri)+(1−λ)ri∣λAv(ri)+(1−λ)ri∣2\tilde{r}_i = \frac{\lambda A_v(r_i) + (1-\lambda) r_i}{\\| \lambda A_v(r_i) + (1-\lambda) r_i \\|_2} (and analogously for descriptions, with λ=0.2\lambda = 0.2), and trains them with a region-to-class-prototype InfoNCE loss on 16 shots per class. It reports 93.27 / 80.29 / 83.71 on Pets / CUB / ImageNet-100 versus 92.94 / 61.17 / 82.72 for the training-free base, with the largest gain on CUB.

Critical strengths

The CUB result is the main reason to take this paper seriously. With a frozen backbone and only ≈1.05\approx 1.05M trainable parameters (≈0.7\approx 0.7\\% of CLIP), accuracy rises from 61.1761.17 (ABS+WCA) to 80.29±0.3180.29 \pm 0.31 with 1616 shots, in roughly 2020 minutes of training on one A100. Even the 11-shot result (68.4068.40) exceeds the 1616-shot CLIP-Adapter (66.3066.30) and Tip-Adapter (65.5065.50) numbers in Table 1. If this holds under a matched protocol, it would show that region/description pipelines are limited mainly by alignment, rather than by the quality of the regions or descriptions themselves.

Critical weaknesses
  1. The technical novelty is limited, and the one element that could constitute a new finding is not tested. As I read the method, each component has close precedents: dual residual adapters blended back into frozen visual and text features follow CLIP-Adapter (including the text-side branch), matching features to cached class prototypes relates to Tip-Adapter, and an InfoNCE pull toward class targets is a natural continuation of CLIP's own contrastive objective. Applying these ingredients within an ABS+WCA pipeline is a reasonable engineering transfer, but I do not see a new mechanism beyond it — the paper's only substantive claim to novelty is the thesis that region/description pipelines are bottlenecked by alignment rather than by region or description quality. That claim is currently hard to separate from a simpler explanation, since the adapters are trained against attribute descriptions — a supervision target standard CLIP-Adapter never receives. The decisive control is the same adapters and prototype InfoNCE on the global CLS feature without ABS regions: if it approaches 80.29 on CUB, the gain reflects the new supervision target, not region alignment. This is not an optional ablation, because without it the headline conclusion is consistent with the data but not established, and the paper's novelty rests entirely on an untested alternative. I would raise my score if this control shows a clear gap; a simple diagnostic (region–description similarity or retrieval before vs. after adaptation) would further support the mechanism.

  2. Hyperparameters appear to be selected on CUB test accuracy. The sweeps in Sec. 5.2 peak at λ=0.2\lambda = 0.2 and K=30K = 30 with 80.6080.60, which the limitations section identifies as the best random draw (range 79.9879.98–80.6080.60). These values are then used on all datasets. A 44-shot validation split exists but is used only for checkpoint selection. The effect is probably large enough to survive proper selection, but the paper should redo the selection on validation data.

Questions for authors
  1. What accuracy does the global-feature variant (adapters Av,AtA_v, A_t plus description-prototype InfoNCE, without ABS regions) achieve on CUB?
  2. Were the Table 1 few-shot baselines re-run on your 1616-shot splits and ViT-B/16 checkpoint, or copied from prior work? Could you add MaPLe or PLOT under the same protocol?
  3. If λ\lambda and KK are selected on the 44-shot validation split instead of CUB test accuracy, which values are chosen and what CUB accuracy results?
  4. What are the agreement-gate statistics (agreement rate, and accuracy of agreeing vs. disagreeing predictions) on CUB? Please also report the standard deviation on ImageNet-100.
1 comment
AuthorsOpenReview ↗

Brief summary: The global-feature control is accepted as decisive, so the region-level claims have been narrowed and the control implemented, with results to be posted if it finishes. Unsupported sweeps and baseline rows have been removed, and λ = 0.2 and K = 30 are stated as untuned defaults. Published Pets baselines are added as context only, the missing ImageNet-100 standard deviation is explained, and the Pets gate’s error-detection metrics are reported, with CUB agreement ready to run.

Thank you for a careful and precise review. Your ethics flag was justified, and our general response explains what happened and what we removed.

W1 / Q1 (the global-feature control). We agree that this control is decisive and that, without it, the paper cannot claim the gain comes from region-level alignment. The revision now says this in Sections 1, 2 and 5.3. We have also dropped the claim that frozen representations "remain misaligned even when features are localised".

The only evidence we have right now is zero-shot. On CUB, the global CLIP embedding gives 56.02 and ABS regions with WCA give 61.17. ABS's own ablation reports that attention-guided crops beat random crops on CUB (61.01 vs 53.58), so it matters which regions are chosen. Neither result speaks to your point about the supervision target, though.

We have implemented the control exactly as you describe. It uses the same adapters and the same region-to-prototype InfoNCE objective,

L=−log⁡exp⁡(z~⊤μ^y/τ)∑cexp⁡(z~⊤μ^c/τ),τ=0.07\mathcal{L} = -\log \frac{\exp(\tilde{z}^{\top} \hat{\mu}_{y} / \tau)}{\sum_{c} \exp(\tilde{z}^{\top} \hat{\mu}_{c} / \tau)}, \quad \tau = 0.07

where z~\tilde{z} is the adapted global CLS embedding rather than N=100N = 100 regions, with the same training settings and the same 16-shot draw. It needs no image encoding and trains on roughly 3,200 cached vectors. If the result comes in during the discussion, we will post it here. Should it come close to UVA, we will say that the gain reflects the supervision target rather than region-level alignment.

W2 / Q3 (were the hyperparameters chosen on CUB test accuracy?). No. The sweeps that peaked at λ=0.2\lambda = 0.2 and K=30K = 30 had no experiment record and have been removed. Both values are simply the defaults in our scripts. No other value was ever trained or evaluated, so they were not picked on test accuracy, but they were not picked on the validation split either. We therefore cannot answer Q3 yet, and Section 5.2 now says so.

Q2 (were the few-shot baselines re-run or copied?). Neither. Those rows had no record and have been removed. We also could not find published 16-shot ViT-B/16 baselines on CUB, because CoOp, CLIP-Adapter, Tip-Adapter, CLIP-LoRA and GDA all use the 11-dataset CoOp benchmark, which leaves CUB out. For Pets, Appendix D.7 now lists published 16-shot ViT-B/16 results as context only, kept apart from our own measurements: Tip-Adapter 91.54, CoOp 92.0 to 92.53, MaPLe 93.2, PLOT++ 93.6 and GDA 93.73. Our two Pets runs give 93.00 and 93.54, but the splits and seeds are different. A training-free Tip-Adapter on our exact draws, computed from cached features, is implemented. MaPLe and PLOT need full training, and we do not plan to run them during the discussion.

Q4 (agreement on CUB and the ImageNet-100 standard deviation). We have not evaluated agreement on CUB yet. It only needs inference with the saved checkpoint, and the code is ready. ImageNet-100 was run once, so there is no standard deviation to report, and its adapter checkpoint was not kept; the paper now states both. For Pets we now report how well the gate A=\unicodex1D7D9[y^WCA=y^LaZSL]A = \unicode{x1D7D9}[\hat{y}_{\mathrm{WCA}} = \hat{y}_{\mathrm{LaZSL}}] detects errors: precision 40.5%, recall 12.7% and a false-alarm rate of 1.3%.

Reviewer_TziHOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UVA, a framework for fine-grained classification with a frozen CLIP backbone. It diagnoses that zero-shot region-description matching fails mainly because the two modalities' embeddings are misaligned, not because regions are poorly selected. UVA combines attention-based region selection, description deduplication, lightweight residual adapters (1.05M parameters), and weighted cross-modal scoring, with a parallel optimal-transport matcher providing a label-free agreement gate. On CUB-200-2011, UVA reaches 80.29% with 16 shots per class, substantially outperforming baseline methods.

Critical strengths

The paper's diagnosis reframes the failure of fine-grained zero-shot matching from poor region selection to misalignment between region and description embeddings (Figure 1). This is the useful contribution: it places the fix on the two embeddings that the matcher compares. UVA follows that diagnosis with two small residual adapters on a frozen CLIP (1.05M parameters). With 16 labelled images per class, CUB accuracy rises from 61.17% for unadapted ABS+WCA to 80.29% (Table 1).

Critical weaknesses
  1. The image and text residual adapters are turned on and off together. Equations (6) and (7) share one λ\lambda: at λ=0\lambda=0 both are off, and at λ=0.2\lambda=0.2 both are on (Table 9). This paper should also explore turning one modality off (λ=0\lambda=0) while keeping the other on, to see how much each modality contributes on its own, and whether the two modalities need different λ\lambda.
  2. The grounding failure in Section 1 (Line 70) is measured only for zero-shot WCA. Figure 8 in Appendix D.4 rank CUB images by how well unadapted regions match the class descriptions: the highest decile is 86.0%, and the lowest is 0.0%. After adaptation, the paper should also recompute the same deciles with the adapted embeddings, to see whether the images that were completely wrong are the ones that get fixed.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Brief summary: the adapters have separate weights and share only a fixed λ = 0.2, show from weight norms that both adapters are trained, and have the per-modality ablations and the split-half grounding-decile analysis set up to post if they finish. We also correct the record by removing the unsupported λ sweep and Figure 8 deciles, fixing the parameter count to 262,144, limiting the early-stopping claim to logged runs, and recasting the “Difficulty-Scaling Rule” as an observed trend.

Thank you for a constructive review, and for seeing the diagnosis as the useful part of the paper. Please also see our general response.

W1 (one shared λ\lambda, switching a modality off, separate λ\lambda per modality). This is the right ablation and we have not run it yet. To clarify the current setup, the visual and textual adapters have their own weights and only the blend factor is shared:

r~i=λAv(ri)+(1−λ) ri∥λAv(ri)+(1−λ) ri∥2,λ=0.2,\tilde{r}_i = \frac{\lambda A_v(r_i) + (1 - \lambda)\, r_i}{\left\lVert \lambda A_v(r_i) + (1 - \lambda)\, r_i \right\rVert_2}, \qquad \lambda = 0.2,

with the same form for AtA_t on the descriptions. We fixed λ=0.2\lambda = 0.2 in advance and never trained with another value. The λ\lambda sweep in the original submission had no experiment record behind it, so we removed it (see the general response).

What we can say from the released checkpoints is that both adapters are clearly trained. The Frobenius norm of W1W_1 is 3.2 to 3.5 times its expected value under PyTorch's default initialisation on CUB, and 1.8 to 2.0 times on Pets (new Table 5, Appendix B). This shows that both adapters are used. It does not tell us how much each one contributes to accuracy.

Switching one trained adapter off at test time needs no training, and the code for it is ready. Training each adapter on its own, and trying λv≠λt\lambda_v \neq \lambda_t, takes nine short runs per dataset on cached features. We will post these results here if they finish during the discussion. If not, they remain a stated limitation (Section 5.2).

W2 (recompute the grounding deciles with adapted embeddings). First we need to correct the record. The image-level deciles in the original Figure 8 (86.0% in the top decile, 0.0% in the bottom) have no experiment record, and we removed them. What we actually ran was a class-level check with a GOAL fine-tuned CLIP, which showed that the least-grounded CUB classes had 0% zero-shot accuracy. We did not keep the per-class values, so Appendix D.3 now presents this only as qualitative motivation.

We have set up the analysis you suggest on the saved checkpoints. For each class it uses one random half of the test images to rank difficulty and the other half to measure accuracy, averaged over many random splits. Using the same images for both would inflate the gains in the hardest deciles through regression to the mean. We will post the result if it completes.

Other corrections you may notice. The adapters have 262,144 parameters, not 1.05M. The early-stopping statement now refers only to runs whose logs we kept. The "Difficulty-Scaling Rule" has been replaced with an observed trend across three benchmarks (Section 4.2).

Thanks for the responses. As the authors acknowledge, the original submission contains a lot of AI-hallucinated results without supporting experimental records, which I consider to violate ICLR’s submission policy. And my concerns posted in Weaknesses remain unresolved, as the related experiments are still in progress. Moreover, after removing the unsupported 16-shot results in Table 1, UVA is the only 16-shot method, leaving no baseline evaluated under the same supervision regime.

Based on the above considerations, I decide to lower my overall rating to 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Unsupported results from LLM-assisted drafting were found and removed, and method expansions, parameter count and the two-run “± std” were corrected, with an apology. Every reported number now traces to the log or released files, and zero-shot results match published values within 0.30 points. New no-training analyses are added, and the remaining requested controls are coded and will be posted if they finish.

General response

We thank all three reviewers for their time and for the care they put into the reviews. We have uploaded a revised PDF in which changes to the submitted version are shown in blue.

A problem with the initial submission. When we went back and checked every number in the paper against our experiment log, we found results that have no experiment record behind them: the CoOp, CLIP-Adapter and Tip-Adapter rows of Table 1, the λ\lambda and KK sweeps, the shot-count curve (including the 1-shot number), the grounding deciles in Figure 8, a qualitative figure, and the training-time and latency numbers. They entered the paper during LLM-assisted drafting, and we did not catch them. Some descriptions were also wrong. We gave incorrect expansions for ABS and BiFTA, reported 1.05M adapter parameters when the true count is 262,144 (about 0.18% of CLIP, not 0.7%), and quoted a "±0.31\pm 0.31" standard deviation from only two runs. All of this has been removed or corrected, every method description has been checked against the original papers and our code, and the AI Use Statement now says plainly what went wrong. We apologise to the reviewers and the AC. The responsibility is ours.

What the revised paper reports. Every number can now be traced to our log or to files we release. That covers the zero-shot results, two 16-shot draws on Pets (93.00, 93.54) and on CUB (79.98, 80.60), a single ImageNet-100 run (83.71), and the agreement statistics. As a sanity check, our zero-shot CLIP and ABS+WCA numbers are within 0.30 points of what WCA and ABS publish for the same backbone (new Appendix D.7). For example, CLIP on CUB gives 56.02 against 56.01 published, and ABS+WCA gives 61.17 against 61.01.

New analyses that needed no training.

  1. The BiFTA-DR descriptions largely overlap with the zero-shot ones. On CUB, 96.3% of the K=30K = 30 descriptions per class already appear among the 50 CuPL descriptions that zero-shot ABS+WCA uses. So the description change that comes with the adapters mostly removes descriptions; it adds very few.
  2. Our lower CuPL number on CUB (54.59 vs 56.42 published) comes from a different description file. The file shipped with the ABS code shares no sentence with the one WCA used.
  3. Treating LaZSL disagreement as an error detector on Pets, it flags 12.7% of all errors while raising a false alarm on 1.3% of correct predictions; 40.5% of flagged predictions are wrong, compared with 6.5% overall. We reconstructed these numbers from the logged rates, since image-level predictions were not saved.
  4. Both trained adapters moved well away from their initial scale (new Table 5). This also made us correct an overstatement in the paper: the anchoring bound needs ∥A(r)∥2≤B\lVert A(r) \rVert_2 \le B with λ<11+B\lambda < \frac{1}{1 + B}, i.e. B<4B < 4 at λ=0.2\lambda = 0.2, and the trained weights do not guarantee that.

Experiments still in progress. Several of the controls the reviewers asked for can run on our cached features without encoding any images again. These are the global-feature control, a training-free Tip-Adapter on our exact 16-shot draws, a test-time single-adapter ablation, ABS+WCA with BiFTA-DR and no adapters, agreement statistics on CUB, and difficulty deciles after adaptation. We have written and tested the code for all of them. If they finish during the discussion period we will post the results in this thread. Until then, we are not drawing any conclusions from them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.