acceptodds
Under review as a conference paper at ICLR 2027

LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment

Abstract

Zero-shot visual recognition often depends on localized parts, attributes, and textures that are diluted in global image representations. Existing training-free localized methods alleviate this limitation by evaluating multiple image regions, but their candidate locations are typically generated before class semantics determine where to look, restricting language to post-hoc scoring of already sampled evidence. We introduce LAGO (LAnguage-Guided adaptive Object-region focus), which reframes localized recognition as language-guided directed region discovery. LAGO first establishes a compact, class-agnostic object-centric visual initialization, then constructs a soft semantic prototype from the intermediate prediction to actively steer subsequent search toward class-relevant evidence during inference. This object-first, language-second design addresses the circular dependence between recognizing a class and locating its supporting evidence, while preserving complementary local, contextual, and global cues. Across standard and distribution-shift benchmarks, LAGO transfers with a single fixed configuration and uses candidate regions more effectively under matched budgets. Controlled experiments further isolate the proposed mechanism, showing that allowing language to alter the search trajectory retrieves more relevant regions and improves recognition over applying the same semantic signal only after candidate generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
zero-shot recognition, vision-language models, visual-text alignment, fine-grained recognition, language-guided region discovery, distribution shift robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MnXvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LAGO, a training-free framework for zero-shot visual-text alignment that aims to improve fine-grained recognition through language-guided region discovery. The method first uses FastSAM to generate class-agnostic object proposals and performs visual-only region search. It then constructs a semantic prototype from intermediate predictions to guide a second-stage region refinement process, with confidence-dependent weighting. Finally, local object features, contextual features, and global image representations are aggregated for classification. Experiments on multiple zero-shot recognition and distribution-shift benchmarks demonstrate improvements over existing CLIP-based methods.

Critical strengths
  1. Reasonable and practically relevant design.

The proposed framework addresses the limitations of global image-text matching by incorporating localized visual evidence. The two-stage strategy provides an intuitive mechanism for combining class-agnostic visual proposals with semantic guidance, while maintaining a training-free inference setting.

  1. Relatively comprehensive experimental evaluation.

The paper evaluates the method across multiple datasets and CLIP backbones, including distribution-shift benchmarks. It also provides component ablations, candidate-budget analyses, statistical tests, and controlled comparisons between directed discovery and post-hoc reranking, which help assess the contributions of individual components.

Critical weaknesses
  1. The motivation is based on a well-explored research problem, with insufficient justification of the proposed conceptual distinction.

The central motivation, namely that global visual representations may overlook discriminative local details and that localized visual features should be aligned with fine-grained textual descriptions, has been extensively investigated in previous literature. Representative examples include WCA (Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models, ICML 2024), BiFTA (Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models, TMLR 2026), and RegionCLIP (RegionCLIP: Region-Based Language-Image Pretraining, CVPR 2022). These studies have already established the importance of regional visual-text alignment and explored different mechanisms for local evidence extraction, selection, and matching. Although this paper emphasizes directed region discovery rather than post-hoc region scoring, the manuscript does not sufficiently establish why this distinction represents a substantial conceptual advance over existing localized alignment approaches. The motivation would be more convincing if the authors could clearly identify a fundamental limitation of prior approaches that cannot be adequately addressed through simpler region selection or refinement strategies.

  1. Limited methodological novelty; the framework resembles an engineering-oriented integration of existing techniques.

The proposed method primarily combines several established components, including off-the-shelf FastSAM proposals, frozen CLIP embeddings, two-stage region search, prediction-derived semantic prototypes, confidence-weighted scoring, and feature aggregation. Most of these operations rely on conventional similarity computations, weighted combinations, and heuristic selection procedures. While the integration is technically reasonable, the individual components do not introduce substantial algorithmic innovations. In particular, the central semantic-guidance mechanism is implemented through a weighted combination of visual and textual similarity scores, without a sufficiently distinctive formulation or deeper methodological insight. Consequently, the approach appears closer to a carefully engineered inference pipeline than a fundamentally new framework. The empirical improvements alone do not fully compensate for the limited methodological novelty expected at ICLR.

Questions for authors
  1. Limited evidence for the central mechanism. The controlled comparison between directed discovery and post-hoc reranking reports only a 0.80 percentage-point improvement on a 1,000-image CUB subset with a single backbone. Although statistically significant, broader validation is needed to establish the generality of this contribution.

  2. Evaluation-protocol concerns. The main benchmark results in Tables 2 and 3 use dataset-specific coefficient selection based on evaluation labels. Although the authors separately report target-label-free experiments, the strongest reported results do not represent a strictly label-free deployment setting.

  3. Fairness of matched-budget comparisons. In Table 1, WCA evaluates 120 crops, whereas LAGO evaluates up to 120 candidate regions plus an additional full image. Furthermore, the controlled search-policy comparison does not use exactly the same number of valid evaluated neighbors. These differences should be considered when interpreting the improvements.

  4. Limited evidence for confidence adaptation. The improvement over fixed semantic guidance is relatively small, and the fixed-setting audit does not establish statistical significance after multiple-comparison correction. This weakens the justification for confidence adaptation as an essential component.

6 comments

Thank you for the thoughtful assessment of novelty and evaluation fairness. We will make the methodological distinction more explicit and have added experiments that test it against post-hoc selection and simpler proposal generation, with separately controlled encoding and inference costs.

W1–W2. What is the conceptual advance beyond existing localized visual–text alignment?

The contribution is semantic feedback during region acquisition, not the general idea of local image–text alignment. WCA, BiFTA, and RegionCLIP have already established the value of region-level evidence. The specific decision studied in LAGO is which spatial regions a frozen visual encoder should evaluate next after intermediate class evidence becomes available. A post-hoc method can change the weights assigned to its existing crop pool; it cannot use that feedback to acquire a previously unseen region without another acquisition step. LAGO separates class-agnostic initialization from this later, language-directed allocation.

We agree that the manuscript did not make candidate acquisition vs. post-acquisition scoring sufficiently explicit. The best way to substantiate the distinction is to hold the existing components fixed and change only the acquisition decision. We have now done so on five full datasets (Q1 below). We also added a non-iterative multiscale global-grid + semantic-reranking control: local allocation performs better in observed accuracy on CUB (+0.932 pp), DTD (+0.366), ImageNet-A (+0.553), ImageNet-V2 (+0.380), and Places365 (+0.222) under that study's matched proposal-acquisition protocol. These tests address whether adaptive acquisition contributes beyond simply applying familiar modules.

Manuscript changes. We will revise the Introduction and Related Work to place WCA, BiFTA, and RegionCLIP explicitly in the regional-alignment literature, distinguish what determines the crop pool from how that pool is scored, and state the proposed acquisition policy as the central contribution. We will make this distinction visible in the Method description and connect it directly to the new controlled ablations rather than claiming novelty for the individual pretrained or fusion components.

Q1. Is there evidence for directed discovery beyond the 1,000-image CUB subset?

Yes. We extended the intervention to all five full evaluation sets, using frozen CLIP ViT-B/16 on CUB, DTD, ImageNet-A, and ImageNet-V2 and ViT-B/32 on Places365. The two arms share FastSAM proposals, Stage-1 features, top-5 semantic prototype, confidence, context processing, and identical final aggregation. We also match the allowance of new unique-coordinate encodings, counting evaluated candidates on both sides. Only the search policy differs: language directs future crop locations, versus post-hoc semantic ranking after a visual acquisition trajectory.

New full-set matched-acquisition intervention:

Dataset / backboneDirected (%)Post-hoc (%)Δ (pp)Paired 95% CIReported Holm p
CUB / B/1660.66159.458+1.203[+1.065, +1.341]1.000
DTD / B/1654.10352.766+1.337[+1.124, +1.550]1.000
ImageNet-A / B/1658.86457.693+1.171[+0.824, +1.518]1.000
ImageNet-V2 / B/1665.75965.310+0.449[+0.329, +0.569]1.000
Places365 / B/3241.58140.551+1.030[+0.970, +1.091]1.000

The new paired results favor language-directed acquisition on every full dataset (+0.449 to +1.337 pp), including +1.171 pp on ImageNet-A and +1.030 pp on Places365. Since proposals, features already acquired, and final aggregation are shared, this directly tests the additional value of changing where the model looks rather than only changing region weights. The CUB/DTD B/16 runs also add a backbone setting beyond the paper's L/14 evaluations.

Manuscript changes. We will promote the full-set intervention to the principal directed-discovery ablation beside Table 10, expand Appendix B.2 with the five paired results and exact matching protocol, and position the original CUB-subset test as initial evidence rather than the sole basis for the mechanism claim. We will include the global-grid control in the adjacent search-policy comparison.

Q2. Do Tables 2–3 use target evaluation labels to select coefficients?

Yes. Tables 2–3 use evaluation-label-selected coefficients and will be presented explicitly as calibrated/oracle diagnostics, not target-label-free deployment results. The appropriate deployment evidence is the fixed unified and source-specified transfer protocol already reported separately in Tables 1 and 6. We have additionally tested how much coefficient selection changes predictions in a new ImageNet-V2 audit.

New ImageNet-V2 protocol checkCoefficient selectionReport imagesAccuracy (%)
Fixed unifiedPrespecified tupleAll 10,00065.45
Held-out calibrationSeparate labeled 2,000Disjoint 8,00065.13
Fixed, same report imagesPrespecified tupleSame 8,00065.11
Full-test oracleLabels from all 10,000Same 10,00065.69

On the identical 8,000 images, held-out calibration changes accuracy by only +0.02 pp relative to the fixed tuple (65.13 vs. 65.11). On all 10,000 images, the evaluation-label oracle is +0.24 pp above fixed inference (65.69 vs. 65.45). These new measurements make the size of calibration's effect concrete in the audited setting.

Manuscript changes. We will clearly separate target-label-free fixed/source transfer from evaluation-label-selected calibration in the experimental setup, Tables 1–3 and 6, and captions, explicitly identify the labels used for selection, and add the ImageNet-V2 held-out/fixed/oracle check to the protocol appendix. Target-label-free claims will reference the fixed/source-specified results, not the oracle table.

Reviewer_xmrUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LAGO (Language-Guided Adaptive Object-Region Focus), a training-free method for zero-shot image recognition. Traditional localized image-text alignment methods rely on passive sampling where candidate regions are cropped before text semantics can influence where to look. To overcome this limitation, LAGO replaces pre-sampled crops with a language-guided, two-stage search strategy: it first establishes a class-agnostic visual initialization to obtain initial crops, and then dynamically decides where to look next based on the intermediate class semantics and prediction confidence. Finally, an object-context dual-channel aggregation module fuses regions, context, and global image representations for final classification. Extensive experiments on standard zero-shot and distribution-shift benchmarks show that LAGO consistently outperforms full-image inference and existing localized baselines under matched region budgets without requiring any model training or target-label tuning.

Critical strengths
  1. The paper addresses a meaningful and important limitation of existing localized image-text alignment methods: language is typically used to score pre-generated crops rather than actively guide crops discovery.

  2. The proposed object-first, language-second strategy provides a reasonable way to mitigate unreliable early semantic predictions. The method is compatible with frozen CLIP backbones and does not require additional model training.

  3. Comprehensive experiments across datasets and backbones demonstrate that language-guided active search convincingly outperforms post-hoc semantic reranking. The inclusion of statistical tests and fixed-transfer evaluations further solidifies these claims.

Critical weaknesses
  1. The performance gains from combined modules are minor and statistically insignificant post-Holm correction; stricter ablations are required to validate the directed search mechanism independently.
  2. Maximizing similarity to a shared text prototype risks capturing broad visual features rather than fine-grained discriminative details, lacking proof that retrieved regions prioritize key evidence.
  3. The efficiency protocol ignores candidate evaluation costs during iterative search, and LAGO's cold latency is 4.21× slower than WCA, necessitating an end-to-end equal-compute comparison.
  4. Localized greedy search is prone to local optima, ignores feature complementarity, and lacks comparison against simpler baselines like non-iterative dense proposal generation with semantic ranking.
  5. Hyperparameter sensitivity analysis only covers aggregation terms, leaving the robustness of search trajectories and the standalone benefit of confidence-adaptive guidance unproven.
Questions for authors
  1. Can the author provide a controlled ablation that holds total feature extractions and aggregation fixed to isolate the performance gain of the search trajectory?
  2. How does the method prevent the soft text prototype from favoring shared background or generic class features over fine-grained discriminative attributes?
  3. How does LAGO perform when compared against baselines under a strict equal-runtime or equal-FLOP budget during cold inference?
  4. Why is local iterative search preferable to non-iterative global proposal generation followed by post-hoc semantic ranking?
  5. How sensitive is recognition accuracy to core search parameters like perturbation step sizes (Δx,Δy,ρ\Delta x, \Delta y, \rho) and the IoU diversity threshold (τsearch\tau_{search})?
6 comments

Thank you for the concrete requests to isolate the search mechanism, inspect the acquired evidence, and compare against simpler methods at matched cost. We have conducted new controlled experiments for each point and will incorporate the results into the manuscript as described below.

Q1. Does the language-guided search trajectory help when feature extractions and aggregation are fixed?

Yes. We added a full-dataset, paired directed-discovery vs. post-hoc-reranking intervention on CUB, DTD, ImageNet-A, ImageNet-V2 (frozen CLIP ViT-B/16), and Places365 (ViT-B/32). Both arms use the same FastSAM proposals, Stage-1 features, top-5 prototype, confidence, context processing, final aggregation, and allowance of newly encoded unique coordinates. Every candidate actually evaluated counts toward that allowance. The only change is whether language affects which regions are encoded next, or merely reranks regions acquired by a visual trajectory.

New full-set trajectory ablation (matched encodings and fusion):

Dataset / backboneDirected (%)Post-hoc (%)Δ (pp)Paired 95% CIReported Holm p
CUB / B/1660.66159.458+1.203[+1.065, +1.341]1.000
DTD / B/1654.10352.766+1.337[+1.124, +1.550]1.000
ImageNet-A / B/1658.86457.693+1.171[+0.824, +1.518]1.000
ImageNet-V2 / B/1665.75965.310+0.449[+0.329, +0.569]1.000
Places365 / B/3241.58140.551+1.030[+0.970, +1.091]1.000

Directed acquisition improves observed accuracy on all five full sets at matched encoding allowance, extending the original 1,000-image CUB intervention across fine-grained objects, textures, shifted ImageNet images, and scenes. Because final aggregation and the existing initialization are identical between arms, this new contrast specifically tests the benefit of semantic steering during acquisition, rather than additional crops or a different classifier.

Manuscript changes. We will add this controlled experiment next to the component results in Table 10, state precisely which modules and per-image encodings are matched, and extend Appendix B.2 with the full-set paired comparisons and experimental protocol. We will identify the earlier CUB-subset result as the preliminary version of this test.

Q2. Does the soft text prototype prefer discriminative object regions rather than generic/background content?

The two-stage design addresses this risk by first anchoring search to class-agnostic object proposals and then using a bounded, confidence-dependent semantic term rather than an unconstrained text-only objective. To test what the guide actually retrieves, we added CUB region-content diagnostics using ground-truth object boxes, part annotations, and class-separation scores for the top three encoded crops.

New diagnostic (directed − post-hoc)ΔPaired 95% CI
Background fraction (lower is better)−0.002661[−0.004490, −0.000863]
Ground-truth box coverage+0.002247[+0.000336, +0.004187]
IoU with ground-truth object box+0.002231[+0.000852, +0.003602]
Mean visible-part coverage+0.001604[−0.000661, +0.003832]
Union visible-part coverage+0.001070[−0.000486, +0.002592]
True-class minus strongest-confuser margin+0.000092[−0.000033, +0.000218]
Fraction of crops with positive class margin+0.000633[−0.002646, +0.003797]

The new measurements show less background and higher object-box coverage and IoU under directed discovery, with paired intervals excluding zero for these geometric measures. The class-margin and part-coverage gains are smaller, so our supported interpretation is improved object focus, rather than a guarantee that every selected crop contains the most discriminative part.

Manuscript changes. We will add a new quantitative retrieved-region content analysis alongside the localization examples, define each geometric and class-margin measure, and discuss the distinction between object localization and fine-grained discrimination when explaining the soft prototype.

Q3. Does LAGO retain an advantage under equal cold runtime or equal neural-forward FLOPs?

We ran both requested cost-matched comparisons, charging iterative candidate evaluations rather than only the final selected crops.

New end-to-end cold audit (2,000 ImageNet-V2 images). Timing covers image preparation, FastSAM, all search/candidate encodings, and final fusion; weights/text embeddings remain resident, while per-image state is uncached.

MethodAccuracy (%)Mean encoder inputsCold ms/imageWithin ±5% of LAGO latency?
LAGO65.00929.3851412.58Reference
WCA147264.8014731380.87Yes
Dense144064.7015111395.10Yes
WCA12064.05121360.29No

At matched measured latency, LAGO is +0.20 pp over WCA1472 and +0.30 pp over Dense1440. The WCA120 row retains the lower-cost operating point for interpreting the trade-off, rather than treating a 120-crop comparison as equal runtime.

New neural-forward FLOPs audit (a separate 1,000-image ImageNet-V2 sample). We count both CLIP and FastSAM forwards, using two FLOPs per multiply–accumulate.

MethodAccuracy (%)TFLOPs/image
LAGO67.3033.131
WCA66.9033.119
Dense66.6033.119

At approximately equal neural FLOPs, the LAGO differences are +0.40 pp vs. WCA (paired 95% CI [−0.90, +1.70], exact p=0.878) and +0.70 pp vs. Dense (CI [−0.20, +1.60], exact p=0.405). These numbers provide the requested end-to-end and forward-compute accounting; the audits have distinct samples and cost definitions.

Manuscript changes. We will extend the efficiency analysis around Table 17 with both new audits, specify the cold-inference boundary and all counted encoder calls, and present accuracy and computation together. We will retain the original 4.21× WCA@120 latency comparison, while distinguishing it from matched-cost comparisons.

Reviewer_4m11OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LAGO is a training-free inference procedure for zero-shot recognition with a frozen CLIP model. It targets the setting in which localized methods generate candidate regions before class semantics can affect where evidence is sought. FastSAM supplies class-agnostic boxes; Stage 1 searches around those boxes using similarity to the full-image feature; Stage 2 builds a top-5 text prototype from the intermediate prediction and uses confidence to weight semantic similarity during a second local search. The final classifier mixes object crops, context crops, and the full image. The paper reports dataset-specific score calibration separately from a single ImageNet-V2-specified coefficient tuple, compares the full pipeline with WCA under a stated 120-region budget, and includes a controlled Stage-2 search intervention on a 1,000-image CUB subset.

Critical strengths

S1. The paper separates evaluation-label coefficient search from target-label-free transfer and applies paired bootstrap intervals, exact McNemar tests, and Holm correction to the comparisons it treats as primary. Section 4.1, Table 1, Table 6, and Appendices A.6–A.7 make the scope of each number explicit. This matters because the submission can be judged on the fixed protocol rather than on the strongest calibrated table.

S2. Under that fixed protocol, the full pipeline is more accurate than the stated WCA@120 baseline on all four reported targets, and the CUB, Places365, and ImageNet-A differences remain significant after Holm correction (Table 1: +0.76, +3.66, and +2.80). On ImageNet-A the same frozen tuple is also +7.01 over matched full-image inference (Table 6). These are the results that most reasonably support the system as a whole.

Critical weaknesses

W1. The distinctive claim—that language-guided search, as opposed to class-agnostic proposals and multi-crop aggregation, improves recognition—is not isolated. In the frozen ImageNet-V2 audit that the authors treat as primary for component effects (Table 10), complete LAGO reaches 65.45%, Stage 1 only 65.20%, Stage 2 only 65.13%, and fixed guidance 65.20%; no contrast remains significant after Holm correction. Matched full-image CuPL inference is 63.30% (Table 19), so most of that gap is already present before semantic search. The direct intervention uses 1,000 CUB images (Appendix B.2): accuracy rises by 0.80 points (95% CI [+0.20, +1.50], p=0.0215), while Query Top-3 cosine rises by 0.00043, directed search evaluates more unique boxes (382 versus 355), and an object-only decision rule reverses the accuracy gap (58.80% versus 59.10%). This affects the central contribution directly. A full-set comparison that matches encoder calls and reports Stage-1-only against WCA would address it; a sentence of clarification would not.

W2. Under the source-specified tuple, the gains over matched full-image inference are small outside ImageNet-A, and cold inference is much more expensive. Table 6 reports +1.01, +0.54, +1.06, +0.13, +0.14, +0.78, +7.01, and +0.20 on the eight transfer sets; after Holm correction only ImageNet, Places365, and ImageNet-A are significant. On Places365, WCA@120 at 40.66 lies below the matched full image at 43.54, while LAGO reaches 44.32, so the +3.66 in Table 1 is a much larger contrast than the +0.78 over the global baseline. Cold LAGO takes 1702 ms/image against 404 ms for WCA@120 (Table 17), a 4.21× increase, with two-stage search alone at 1107 ms. Tables 2–3, where every cell is highest, select fusion weights on evaluation labels and reproduce published baseline numbers, including WCA’s default setting of N=60 rather than the 120-view protocol. The authors label this distinction. The effect sizes and search cost remain properties of the reported method.

Questions for authors

Q1. On the full Table 1 datasets and on ImageNet-V2, what is the paired accuracy of Stage-1-only LAGO versus WCA, and of directed Stage-2 search versus post-hoc reranking, when both sides encode the same number of boxes? (W1.) A reliable encode-matched gain on the full sets, large enough to explain the Places365 and ImageNet-A margins, would move the paper toward weak acceptance. If Stage-1-only matches the full pipeline against WCA, the directed-discovery claim would not support acceptance.

Q2. What system is the “visual initialization” reference in Table 15, and which prediction is −0.52 on ImageNet-R? The ImageNet-V2 entry of +0.74 does not match the 0.25-point Stage-1 gap in Table 10. If semantic refinement reduces accuracy on renditions relative to the class-agnostic stage, the transfer claim needs that scope. If the reference is a different system, this concern can be removed.

Q3. For Table 1, were the WCA predictions recomputed with 120 random views, the same text bank, and the same decision to include or exclude the original image? Places365 WCA@120 is 40.66, identical to the published ViT-B/32 WCA number, whose paper sets the default crop count to N=60 (Li et al., ICML 2024, arXiv:2406.02915). Image-level discordant counts would resolve whether that equality is a matched rerun.

5 comments

Thank you for identifying the experiments that would determine whether language-guided discovery contributes beyond object-centric initialization and crop aggregation. We have run new full-set, paired comparisons with explicit encoder accounting, as well as cost-matched inference audits. We address your three questions and the related transfer/cost concern below.

Q1 / W1. Stage-1-only vs. WCA, and directed discovery vs. post-hoc reranking, at matched encoder budgets

We performed both requested full-set comparisons. The new evaluations cover CUB, DTD, ImageNet-A, and ImageNet-V2 with frozen CLIP ViT-B/16, and Places365 with ViT-B/32. Each paired comparison uses fixed coefficients and the same backbone/text bank within a dataset. The CUB/DTD B/16 runs supplement the L/14 results in the submission.

New experiment A — Stage 1 vs. WCA with equal total encoder inputs. We independently resample WCA crops so that, for each image, both arms make the same total number of image-encoder calls, including full-image and context inputs.

Dataset / backboneStage 1 (%)Matched WCA (%)Δ (pp)Paired 95% CI
CUB / B/1660.16459.528+0.636[+0.049, +1.223]
DTD / B/1653.13852.785+0.353[−0.445, +1.151]
ImageNet-A / B/1656.96056.213+0.747[+0.093, +1.413]
ImageNet-V2 / B/1665.14064.760+0.380[−0.070, +0.830]
Places365 / B/3240.70740.499+0.208[−0.002, +0.432]

Stage 1 alone provides a relatively small improvement over matched WCA, notably +0.747 pp on ImageNet-A and +0.208 pp on Places365. This establishes the appropriate baseline for testing what semantic refinement adds.

New experiment B — directed Stage-2 discovery vs. post-hoc semantic reranking, with equal new encodings. Both arms share FastSAM proposals, Stage-1 features, the top-5 text prototype, confidence, context processing, and final object/context/global aggregation. We hold the allowance of new unique-coordinate encodings identical and charge every evaluated candidate; because initialization is shared, the total encoder-input budget is also matched between these two arms. The intervention is whether semantics determine subsequent crop coordinates or only rerank crops acquired by a visual trajectory.

Dataset / backboneDirected (%)Post-hoc (%)Δ (pp)Paired 95% CIReported Holm p
CUB / B/1660.66159.458+1.203[+1.065, +1.341]1.000
DTD / B/1654.10352.766+1.337[+1.124, +1.550]1.000
ImageNet-A / B/1658.86457.693+1.171[+0.824, +1.518]1.000
ImageNet-V2 / B/1665.75965.310+0.449[+0.329, +0.569]1.000
Places365 / B/3241.58140.551+1.030[+0.970, +1.091]1.000

Under this controlled intervention, directed discovery yields higher paired accuracy on all five full datasets, including the two benchmarks highlighted in your acceptance criterion: ImageNet-A (+1.171 pp) and Places365 (+1.030 pp). This extends the original 1,000-image CUB result to full datasets while isolating the choice of where to acquire features from downstream fusion.

New experiment C — is Stage 1 already as accurate as full LAGO? In the same new evaluation flow, adding Stage 2 increases ImageNet-A from 56.960% to 58.864% (+1.904 pp) and Places365 from 40.707% to 41.581% (+0.874 pp); the gains on CUB, DTD, and ImageNet-V2 are +0.497, +0.965, and +0.619 pp. Thus, Stage 1 does not reach the full pipeline's observed accuracy. This comparison measures the complete Stage-2 increment, including its extra computation; experiment B is the equal-budget test of directed search itself. The larger original Table 1 WCA@120 margins refer to a different baseline/protocol and should not be equated with these controlled increments.

Manuscript changes. We will add experiments A–C to the component ablations beside Table 10, report the per-image encoder-budget rule and paired predictions, and expand Appendix B.2 from a CUB-only intervention into a five-dataset directed-vs.-post-hoc analysis. We will revise the Table 1 discussion to distinguish the aggregate WCA comparison from the directly isolated Stage-2 contribution, particularly for Places365 and ImageNet-A.

Q2. What does “visual initialization” mean in Table 15, and why do its V2 numbers differ from Table 10?

Table 15 does not compare the full method with a separately executed Stage-1-only system. Its reference is visual-score aggregation on the same final crop cache: crop weights are determined by similarity to the full-image feature, while the other prediction uses the final semantic reweighting/fusion on those same cached crops. By contrast, Table 10 compares two separately executed pipelines: full LAGO and Stage 1 alone.

Reported entryActual comparisonΔ (pp)
Table 15, ImageNet-V2Final scoring/fusion − visual-score aggregation, same final cache+0.74
Table 10, ImageNet-V2Full pipeline − separately run Stage 1+0.25
Table 15, ImageNet-RFinal scoring/fusion − visual-score aggregation, same final cache−0.52

Therefore, +0.74 and +0.25 measure different interventions, resolving the apparent inconsistency. The −0.52 pp on ImageNet-R is a decrease from semantic/final fusion conditional on the final crop cache; it is not an isolated failure of the Stage-2 search trajectory.

Manuscript changes. We will rename Table 15's reference “visual-score aggregation on the final crop cache,” specify the two predictions and subtraction direction in its caption, cross-reference Table 10, and explicitly state the ImageNet-R scoring/fusion exception in the distribution-shift discussion.

Q3. Was Table 1's WCA@120 a matched rerun rather than the published N=60 result?

Yes: our Table 1 WCA setting uses 120 newly sampled random crop views, the same CLIP backbone and class-text bank as the corresponding LAGO run, plus a separately encoded full image for WCA crop weighting. WCA@120 therefore entails 121 image-encoder inputs. WCA's use of the global feature to weight crops is different from directly adding a global classification logit, which is why simply counting the original image is insufficient to standardize the decision rule.

We have also run new full-image-fusion controls with the same global-logit weight (0.025) on both sides, charging the full-image encoding to each method. The new per-image Stage-1/WCA experiment in Q1 independently matches all image-encoder calls rather than assuming that 120 selected regions means equal total compute. The equal rounded Places365 accuracy (40.66%) does not identify the crop count or verify a rerun by itself; the run settings and paired predictions are the appropriate audit.

Manuscript changes. We will make “120 random crops + 1 global-image encoding”, shared backbone/text bank, random-view protocol, and global-image use explicit in the Table 1 caption and evaluation protocol. We will add image-level WCA-correct/LAGO-incorrect and LAGO-correct/WCA-incorrect counts, alongside the prediction manifest/seed, to the supplement for the original Table 1 run. These counts are not in the current summary and must be extracted from saved predictions rather than inferred from rounded accuracy.

Comments on the paper

1 comment

Dear reviewers, with the discussion period nearing its end, could you please share your thoughts on the authors’ rebuttal and whether your concerns have been addressed? Please also update your scores if warranted. Thank you for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.