- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LAGO, a training-free framework for zero-shot visual-text alignment that aims to improve fine-grained recognition through language-guided region discovery. The method first uses FastSAM to generate class-agnostic object proposals and performs visual-only region search. It then constructs a semantic prototype from intermediate predictions to guide a second-stage region refinement process, with confidence-dependent weighting. Finally, local object features, contextual features, and global image representations are aggregated for classification. Experiments on multiple zero-shot recognition and distribution-shift benchmarks demonstrate improvements over existing CLIP-based methods.
Critical strengths
- Reasonable and practically relevant design.
The proposed framework addresses the limitations of global image-text matching by incorporating localized visual evidence. The two-stage strategy provides an intuitive mechanism for combining class-agnostic visual proposals with semantic guidance, while maintaining a training-free inference setting.
- Relatively comprehensive experimental evaluation.
The paper evaluates the method across multiple datasets and CLIP backbones, including distribution-shift benchmarks. It also provides component ablations, candidate-budget analyses, statistical tests, and controlled comparisons between directed discovery and post-hoc reranking, which help assess the contributions of individual components.
Critical weaknesses
- The motivation is based on a well-explored research problem, with insufficient justification of the proposed conceptual distinction.
The central motivation, namely that global visual representations may overlook discriminative local details and that localized visual features should be aligned with fine-grained textual descriptions, has been extensively investigated in previous literature. Representative examples include WCA (Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models, ICML 2024), BiFTA (Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models, TMLR 2026), and RegionCLIP (RegionCLIP: Region-Based Language-Image Pretraining, CVPR 2022). These studies have already established the importance of regional visual-text alignment and explored different mechanisms for local evidence extraction, selection, and matching. Although this paper emphasizes directed region discovery rather than post-hoc region scoring, the manuscript does not sufficiently establish why this distinction represents a substantial conceptual advance over existing localized alignment approaches. The motivation would be more convincing if the authors could clearly identify a fundamental limitation of prior approaches that cannot be adequately addressed through simpler region selection or refinement strategies.
- Limited methodological novelty; the framework resembles an engineering-oriented integration of existing techniques.
The proposed method primarily combines several established components, including off-the-shelf FastSAM proposals, frozen CLIP embeddings, two-stage region search, prediction-derived semantic prototypes, confidence-weighted scoring, and feature aggregation. Most of these operations rely on conventional similarity computations, weighted combinations, and heuristic selection procedures. While the integration is technically reasonable, the individual components do not introduce substantial algorithmic innovations. In particular, the central semantic-guidance mechanism is implemented through a weighted combination of visual and textual similarity scores, without a sufficiently distinctive formulation or deeper methodological insight. Consequently, the approach appears closer to a carefully engineered inference pipeline than a fundamentally new framework. The empirical improvements alone do not fully compensate for the limited methodological novelty expected at ICLR.
Questions for authors
-
Limited evidence for the central mechanism. The controlled comparison between directed discovery and post-hoc reranking reports only a 0.80 percentage-point improvement on a 1,000-image CUB subset with a single backbone. Although statistically significant, broader validation is needed to establish the generality of this contribution.
-
Evaluation-protocol concerns. The main benchmark results in Tables 2 and 3 use dataset-specific coefficient selection based on evaluation labels. Although the authors separately report target-label-free experiments, the strongest reported results do not represent a strictly label-free deployment setting.
-
Fairness of matched-budget comparisons. In Table 1, WCA evaluates 120 crops, whereas LAGO evaluates up to 120 candidate regions plus an additional full image. Furthermore, the controlled search-policy comparison does not use exactly the same number of valid evaluated neighbors. These differences should be considered when interpreting the improvements.
-
Limited evidence for confidence adaptation. The improvement over fixed semantic guidance is relatively small, and the fixed-setting audit does not establish statistical significance after multiple-comparison correction. This weakens the justification for confidence adaptation as an essential component.
6 comments
Q1. Is there evidence for directed discovery beyond the 1,000-image CUB subset?
Yes. We extended the intervention to all five full evaluation sets, using frozen CLIP ViT-B/16 on CUB, DTD, ImageNet-A, and ImageNet-V2 and ViT-B/32 on Places365. The two arms share FastSAM proposals, Stage-1 features, top-5 semantic prototype, confidence, context processing, and identical final aggregation. We also match the allowance of new unique-coordinate encodings, counting evaluated candidates on both sides. Only the search policy differs: language directs future crop locations, versus post-hoc semantic ranking after a visual acquisition trajectory.
New full-set matched-acquisition intervention:
| Dataset / backbone | Directed (%) | Post-hoc (%) | Δ (pp) | Paired 95% CI | Reported Holm p |
|---|---|---|---|---|---|
| CUB / B/16 | 60.661 | 59.458 | +1.203 | [+1.065, +1.341] | 1.000 |
| DTD / B/16 | 54.103 | 52.766 | +1.337 | [+1.124, +1.550] | 1.000 |
| ImageNet-A / B/16 | 58.864 | 57.693 | +1.171 | [+0.824, +1.518] | 1.000 |
| ImageNet-V2 / B/16 | 65.759 | 65.310 | +0.449 | [+0.329, +0.569] | 1.000 |
| Places365 / B/32 | 41.581 | 40.551 | +1.030 | [+0.970, +1.091] | 1.000 |
The new paired results favor language-directed acquisition on every full dataset (+0.449 to +1.337 pp), including +1.171 pp on ImageNet-A and +1.030 pp on Places365. Since proposals, features already acquired, and final aggregation are shared, this directly tests the additional value of changing where the model looks rather than only changing region weights. The CUB/DTD B/16 runs also add a backbone setting beyond the paper's L/14 evaluations.
Manuscript changes. We will promote the full-set intervention to the principal directed-discovery ablation beside Table 10, expand Appendix B.2 with the five paired results and exact matching protocol, and position the original CUB-subset test as initial evidence rather than the sole basis for the mechanism claim. We will include the global-grid control in the adjacent search-policy comparison.
Q2. Do Tables 2–3 use target evaluation labels to select coefficients?
Yes. Tables 2–3 use evaluation-label-selected coefficients and will be presented explicitly as calibrated/oracle diagnostics, not target-label-free deployment results. The appropriate deployment evidence is the fixed unified and source-specified transfer protocol already reported separately in Tables 1 and 6. We have additionally tested how much coefficient selection changes predictions in a new ImageNet-V2 audit.
| New ImageNet-V2 protocol check | Coefficient selection | Report images | Accuracy (%) |
|---|---|---|---|
| Fixed unified | Prespecified tuple | All 10,000 | 65.45 |
| Held-out calibration | Separate labeled 2,000 | Disjoint 8,000 | 65.13 |
| Fixed, same report images | Prespecified tuple | Same 8,000 | 65.11 |
| Full-test oracle | Labels from all 10,000 | Same 10,000 | 65.69 |
On the identical 8,000 images, held-out calibration changes accuracy by only +0.02 pp relative to the fixed tuple (65.13 vs. 65.11). On all 10,000 images, the evaluation-label oracle is +0.24 pp above fixed inference (65.69 vs. 65.45). These new measurements make the size of calibration's effect concrete in the audited setting.
Manuscript changes. We will clearly separate target-label-free fixed/source transfer from evaluation-label-selected calibration in the experimental setup, Tables 1–3 and 6, and captions, explicitly identify the labels used for selection, and add the ImageNet-V2 held-out/fixed/oracle check to the protocol appendix. Target-label-free claims will reference the fixed/source-specified results, not the oracle table.
Thank you for the thoughtful assessment of novelty and evaluation fairness. We will make the methodological distinction more explicit and have added experiments that test it against post-hoc selection and simpler proposal generation, with separately controlled encoding and inference costs.
W1–W2. What is the conceptual advance beyond existing localized visual–text alignment?
The contribution is semantic feedback during region acquisition, not the general idea of local image–text alignment. WCA, BiFTA, and RegionCLIP have already established the value of region-level evidence. The specific decision studied in LAGO is which spatial regions a frozen visual encoder should evaluate next after intermediate class evidence becomes available. A post-hoc method can change the weights assigned to its existing crop pool; it cannot use that feedback to acquire a previously unseen region without another acquisition step. LAGO separates class-agnostic initialization from this later, language-directed allocation.
We agree that the manuscript did not make candidate acquisition vs. post-acquisition scoring sufficiently explicit. The best way to substantiate the distinction is to hold the existing components fixed and change only the acquisition decision. We have now done so on five full datasets (Q1 below). We also added a non-iterative multiscale global-grid + semantic-reranking control: local allocation performs better in observed accuracy on CUB (+0.932 pp), DTD (+0.366), ImageNet-A (+0.553), ImageNet-V2 (+0.380), and Places365 (+0.222) under that study's matched proposal-acquisition protocol. These tests address whether adaptive acquisition contributes beyond simply applying familiar modules.
Manuscript changes. We will revise the Introduction and Related Work to place WCA, BiFTA, and RegionCLIP explicitly in the regional-alignment literature, distinguish what determines the crop pool from how that pool is scored, and state the proposed acquisition policy as the central contribution. We will make this distinction visible in the Method description and connect it directly to the new controlled ablations rather than claiming novelty for the individual pretrained or fusion components.