acceptodds
Under review as a conference paper at ICLR 2027

Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometry

Abstract

Visual distinctions are often finer than those reflected in linguistic conceptualization. Vision-language models exhibit a similar asymmetry: a distinction can remain discriminable in frozen image geometry while being weakly addressable through the native text interface. We study this gap by separating visual discriminability from linguistic addressability in text-to-image retrieval. Using FactorAtlas, a fully crossed testbed of 23,040 images spanning shape, hue, pattern, and nuisance variation, we compare both readouts on held-out images of the same distinctions. We then derive image-side contrasts that separate each value from its alternatives for matched visual grounding, and test whether this reduces the native-text access gap across factors and models. Direction-specific and visual-absence controls tie these gains to the relevant visual contrast; the gains persist after global alignment and extend to compositional retrieval and natural images. Together, these results show that visual discriminability and linguistic addressability need not coincide, and that matched visual grounding can probe and reduce the resulting access gap.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision-language models, cross-modal alignment, text-to-image retrieval, visual discriminability, linguistic addressability, matched visual grounding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kA9jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the gap between "visual representations containing certain attribute information" and "text queries being able to effectively access that information" in frozen visual language models. The authors constructed a FactorAtlas containing 23,040 images to calibrate the visual orientation of the target attribute relative to other attributes and incorporated it into text queries. The paper validates its effectiveness on seven models, various hold-out protocols, combined retrieval, and natural image data. The main contributions are controlled attribute access auditing and experimental evidence of cross-modal retrieval mismatch.

Critical strengths

The core diagnostics are supported by comprehensive control experiments. FactorAtlas cross-controls semantic factors and separates interference scenarios for calibration, validation, and testing. The paper examines alternative explanations such as prototype attraction, mismatch orientation, cue word variation, visual signal cancellation, and purely visual endpoints. In particular, Appendix C.5 examines whether the gains simply come from replacing text queries with visual classifiers, making the main conclusions more credible than simply showing retrieval improvements.

The gains are broad and have direct cross-domain evidence. The average mAP of the seven models improves from .660/.619/.567 to .803/.786/.862 for shape, color, and texture, respectively. Even more convincingly, without fitting the Fashionpedia target domain, directly transferring the four texture orientations from FactorAtlas still improves the average mAP from .583 to .719. This supports the finding that the orientations have value beyond the calibration distribution.

Critical weaknesses

The methodological novelty is limited, and the added knowledge needs to be more clearly defined. The update of "original query + difference in positive and negative visual centroids" is similar to the classic Rocchio relevance feedback; its target direction relative to other categories is merely a scaling of the central category prototype. The observation that "information exists in a single-modal representation, but cross-modal access fails" is closely related to the cited LABCLIP. Therefore, the main added value of this paper lies in audit design and empirical findings, where the current explanation of the relationship between these existing concepts is still insufficient.

The key data construction details for audit contributions are insufficient for verification. The appendix does not clearly define the generation rules for the ten textures and the specific definitions for the twelve colors; the natural image experiment also does not list the four clothing categories reserved by Fashionpedia, the fixed footwear subset of UT-Zappos, or the complete sample partitioning. No accessible code or data entry points are found in the PDF. These choices directly affect the task difficulty and potential category relevance, which is particularly important for papers that primarily contribute to controlled auditing.

Questions for authors

Clarification of Materials: Could you provide the FactorAtlas rendering script, natural image sample IDs, attribute mappings, and partitioning files, especially the specific category selections for Fashionpedia and UT-Zappos?

Supplementary Evidence Affecting Judgment: Could Table 22 report the AP and gain fluctuations for each partition and attribute? Especially for COCO-Facet, which only contains 306 images, and UT-Zappos, which has 96 products, it is currently difficult to determine the stability of migration gains.

4 comments
AuthorsOpenReview ↗

Response to Reviewer kA9j - Part1

We thank the reviewer for the thoughtful assessment, particularly for recognizing the value of our controlled audit and diagnostic experiments. We appreciate the specific suggestions regarding reproducibility and empirical stability.

We address three points:

  • Relation to prior work (W1): Distinguishing our audit from Rocchio-style updates and LABCLIP.
  • Reproducibility (W2/Q1): Clarifying cohort construction and providing the corresponding materials.
  • Stability (Q2): Reporting partition- and attribute-level variability, including boundary cases.

W1. Relationship to prior work

We thank the reviewer for pointing out the connection to classical Rocchio relevance feedback. We agree that our query update shares a similar centroid-difference operation. As the reviewer recognizes, our primary contribution lies in the controlled audit design and empirical findings: we derive distinction-specific contrasts from frozen image embeddings and use them to ground native text queries, testing how much linguistic access can be improved through matched intervention.

In addition, our distinction-level analysis complements prior work on cross-modal mismatch, including LABCLIP. While LABCLIP improves attribute–object correspondence through a shared alignment transformation, we examine linguistic access at the level of individual named visual distinctions and test whether matched visual contrasts can reduce access gaps even after broader alignment (§3.4).

We will revise the Introduction to make our contribution clearer and bring the relevant Related Work discussion forward to better explain how our distinction-level audit and matched intervention relate to established relevance-feedback and cross-modal alignment approaches.


W2/Q1. Construction and reproducibility

We appreciate the reviewer's request for more explicit cohort-construction details and reproducibility materials. To clarify the protocols described in Appendices B and H, we summarize the key construction and partition choices below.

  • FactorAtlas: 8 shapes × 12 hues × 10 patterns × 24 nuisance realizations (12 contexts × 2 renderer replicas). Twelve fixed linear-RGB hues and 10 patterns defined by procedural Blender shaders (Wave, Checker, Voronoi, Noise, Brick) plus a uniform surface. Contexts 0–3/4–5/6–11 define calibration/validation/test.

  • Fashionpedia: Six patterns across four garment categories (dress, shirt/blouse, skirt, top/T-shirt/sweatshirt), holding out each category in turn. For dress length, the 447 dress crops of the official validation split serve as a fixed, untouched test gallery.

  • COCO-Facet: 306 images (172 metal, 29 stone, 105 wood), evaluated with five image-disjoint rotations.

  • UT-Zappos: Women/Shoes/Heels/Slip-On, with 96 products across four materials and three toe-style × heel-height cells (11, 8, 5 products per material), evaluated with product-disjoint rotations.

To address the reviewer's reproducibility concerns, we provide additional supplementary materials for this rebuttal, including the rendering script, FactorAtlas specification, per-cell render seeds, source-image IDs, attribute mappings, exact partition assignments, and scripts that rebuild and verify the natural-image cohorts from raw annotations. These materials enable independent verification of the evaluation cohorts.

AuthorsOpenReview ↗

Response to Reviewer kA9j - Part2

Q2. Partition- and attribute-level AP and gain variability

To directly address the reviewer's request for partition- and attribute-level AP and gain fluctuations, we report native AP, matched AP, and their differences for every partition ×\times attribute combination across all five settings. Complete results are provided in the supplementary material (results/partition_tables.md). The following table summarizes macro-mAP gains and variability across seven frozen backbones:

SettingGain ± SDPositive evaluations
DTD+.233 ± .00870/70
Fashionpedia pattern+.181 ± .02828/28
Fashionpedia length+.108 ± .00535/35
COCO-Facet+.237 ± .04035/35
UT-Zappos+.057 ± .06326/35

SD is the mean of the across-partition gain SDs computed separately for each backbone. Fashionpedia length uses calibration-support resamples evaluated on a fixed test gallery.

All 168 backbone ×\times partition evaluations across the first four settings show positive macro-mAP gains. At the attribute level, Fashionpedia pattern improves across all 24 category–attribute means, while DTD improves for 45/47 texture classes. For Fashionpedia length, matched AP remains stable across support resamples on the fixed test gallery (SD ≤.022\leq .022 across attributes), although the magnitude of improvement varies across length attributes.

Given the reviewer's particular concern about the smaller cohorts, we further examine COCO-Facet (306 images) and UT-Zappos (96 products) at the attribute and partition levels. The following table reports per-rotation AP gains alongside attribute-level visual–native AP gaps, comparing image-derived visual retrieval (visual discriminability) with native linguistic access (linguistic addressability):

AttributeR1R2R3R4R5Visual–native gapMatched gain
COCO: Metal+.22+.19+.19+.18+.11+.193+.177
COCO: Stone+.41+.22+.34+.45+.40+.322+.367
COCO: Wood+.14+.22+.18+.13+.16+.170+.167
UT: Leather+.01-.00+.07-.12+.02-.066-.004
UT: Suede-.02+.00+.01-.06+.04-.026-.003
UT: Patent Leather+.04+.12+.08+.02+.29+.098+.109
UT: Synthetic+.35+.10+.00+.16+.03+.101+.126

Seven-backbone means. R1–R5: per-rotation AP gains. Visual–native gap: image-derived visual retrieval minus native text retrieval AP; matched gain: matched grounding minus native text AP, averaged across rotations.

COCO-Facet shows positive gains for every attribute and rotation at the seven-backbone mean, although Stone exhibits substantial native AP variability (.40–.61) with only 4–9 test images per rotation. In contrast, UT-Zappos shows greater rotation-level variability, with positive macro-mAP gains in 26/35 backbone–rotation evaluations. These results highlight the variability associated with evaluating relatively small natural-image cohorts, which we acknowledge as a limitation.

Looking at the attribute-level results, matched access gains track the magnitude of the visual–linguistic discrepancy: in COCO-Facet, Stone has the largest gap (+.322) and gain (+.367), while Metal and Wood show smaller gaps and gains. In UT-Zappos, Leather and Suede have negative gaps and negligible gains, whereas Patent Leather and Synthetic have positive gaps and correspondingly larger gains.

Together, these results acknowledge partition-level variability in smaller natural-image cohorts while providing additional attribute-level evidence consistent with the intended role of matched grounding in improving linguistic access where a gap exists.


Overall, these clarifications aim to make our distinction-level audit and its empirical contributions more transparent and reproducible. We hope that the additional materials and partition-resolved results address the reviewer's questions and concerns, and provide a clearer basis for assessing both the strengths and limitations of our findings.

AuthorsOpenReview ↗

Manuscript Revision Update

To address your concerns, we have incorporated the clarifications and additional analyses from our rebuttal into the revised manuscript.

  • Relationship to prior work (W1): We revised the Introduction (§1, L42–44, L50–52) and Related Work (§2, L91–96) to clarify our contribution relative to Rocchio relevance feedback and LABCLIP. These revisions more clearly establish the controlled distinction-level audit and matched intervention as tools for examining value-specific linguistic access gaps, including after broader alignment.
  • Construction and reproducibility (W2/Q1): We expanded Appendix B (Tables 5–6, L756–786) and Appendix H.3 (L1469–1489, L1509–1514) with explicit FactorAtlas specifications and natural-image cohort protocols. These details make the construction of the evaluated visual distinctions and held-out conditions more transparent and verifiable. We also updated the Reproducibility Statement (L558–565) to summarize the supporting appendix materials and state our plan to publicly release the code and reproducibility materials upon de-anonymization.
  • Empirical stability and access gaps (Q2): We added Appendix H.4 (Tables 26–27, L1517–1565) to report partition-level variability and attribute-level visual–native gaps alongside matched gains. These analyses help establish where access gains are consistent and how their magnitude relates to the underlying linguistic access gap, while documenting variability in smaller natural-image cohorts.

The accompanying supplementary material provides the detailed construction resources and partition-by-attribute results described in our rebuttal.

We hope these revisions strengthen the transparency and empirical foundation of our distinction-level audit. We welcome any further questions or discussion regarding the revised manuscript and our responses.

We thank the reviewer again for their constructive feedback. We hope that our rebuttal responses and revised manuscript have adequately addressed the concerns raised in the review. If so, we would greatly appreciate reconsideration of the overall assessment and score.

We welcome any remaining questions or concerns and would be happy to address them during the discussion period.

Reviewer_FxV6OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the discrepancy between visual discriminability and linguistic addressability in frozen vision-language models. The authors introduce FactorAtlas, a controlled dataset of 23,040 rendered images varying in shape, hue, pattern, and nuisance factors. They compare image-derived visual attribute representations with native text queries to examine differences in retrieval performance. To improve linguistic access, the paper proposes matched visual grounding, which adjusts text embeddings using image-derived target-versus-rest contrast directions. Experiments across seven CLIP and SigLIP variants report improvements in attribute-level and compositional text-to-image retrieval. Additional analyses examine direction specificity, prompt robustness, global alignment, and generalization to natural-image datasets.

Critical strengths
  1. The paper provides a reasonably systematic empirical evaluation across multiple vision-language backbones, controlled synthetic settings, and natural-image datasets, supported by several relevant control experiments.

  2. The figures are visually appealing and well organized, making the experimental results easier to follow.

Critical weaknesses
  1. Limited novelty and insight. The central observation that visual discriminability and linguistic addressability are not fully consistent seems rather unsurprising. If they were fully consistent, most of the fundamental challenges in cross-modal representation learning would already be resolved. I therefore find it difficult to see what new scientific insight is gained simply by demonstrating this discrepancy. Moreover, the paper does not explore why such a discrepancy exists or provide deeper insights into its underlying mechanisms. The research question seems too broad compared with what the paper actually answers.

  2. Limited significance of the proposed improvement. The proposed method incorporates visual discriminative directions derived from labeled images into text embeddings, which are then used for image retrieval. For example, to retrieve images with a "striped" pattern, the method modifies the text embedding using a visual direction computed from image features extracted from labeled examples of striped and other patterns. These labels are explicitly used to construct the visual discriminative direction. In a sense, this turns a cross-modal retrieval problem into a partially within-modal one. Since within-modal retrieval is generally easier, especially when additional labeled information from the target modality is available, the reported improvements are largely expected. I am therefore not convinced that these results demonstrate a meaningful advance in cross-modal representation learning.

This concern is further supported by Appendix C.5, where purely visual directions achieve nearly the same retrieval performance as the proposed method when sufficient visual examples are available. For instance, on SigLIP2 Base, the proposed method achieves 0.878 mAP for pattern retrieval, compared with 0.870 using purely visual directions. Similar results are observed for shape (0.813 vs. 0.812) and hue (0.854 vs. 0.833). These results make it difficult to identify the actual contribution of the linguistic component.

Questions for authors

Please see Critical Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer FxV6 - Part1

We thank the reviewer for the detailed assessment and the opportunity to clarify our scientific contribution and the role of native text.

Our response focuses on two questions:

  • Novelty and scientific insight (W1): What distinction-level auditing reveals beyond known cross-modal mismatch and broader alignment approaches.
  • Significance and linguistic contribution (W2): Why matched grounding is not simply visual-only retrieval, and what retaining native text contributes.

W1. Novelty and scientific insight.

We appreciate the reviewer's concern that our research question may appear broader than what the experiments actually address. We agree that cross-modal mismatch is already well established. Prior work has examined discrepancies between within-modal structure and cross-modal correspondence, as well as approaches to improving alignment at broader representational and compositional scales [1–3]. These findings are also a central motivation for our work. Building on this foundation, we specifically take individual named visual distinctions as the unit of analysis, examining their accessibility through native text queries and testing to what extent corresponding image-side contrasts can reduce access gaps in frozen representations.

The broader evaluations that follow suggest that distinction-level auditing and matched visual intervention allow us to examine not only where linguistic access falls short of visual discriminability, but also to what extent the corresponding image-side structure can reduce these gaps and support retrieval beyond the original audit setting.

As these distinctions and their implications may have remained somewhat implicit in our presentation, we will make them clearer in the Introduction and bring the relevant Related Work discussion forward to better position our contribution relative to prior work.


W2. Significance of matched grounding and native text.

As the reviewer notes, visual-only directions approach matched grounding with abundant labeled examples, raising an important question about the contribution of native text. To further examine this question, we revisited our existing factor-pair holdout experiments and found that retaining native text can remain functionally useful even with full calibration support. In these experiments, individual attribute values remain observed during calibration, but specific combinations are excluded from both calibration and validation. For example, the visual contrast for "red" is estimated without any red cubes, yet retrieval is evaluated on red cubes at test time.

Held-out pairFactorNativeVisual-onlyMatchedMedian α\alpha
Hue × ShapeHue.638.492.675.10
Pattern × HueHue.664.515.689.05
Pattern × ShapePattern.583.692.747.25
Pattern × ShapeShape.684.694.748.20

Full calibration support; seven-backbone mean factor-level macro-mAP. Median α\alpha is computed across backbones.

These results suggest that native text plays an important role in generalization beyond the calibration conditions. Although matched grounding incorporates labeled within-modal visual evidence, retaining the native text query provides a useful anchor when the visual direction alone generalizes poorly.

The benefit of retaining native text is further observed in cross-domain transfer. We compared matched grounding with pure visual directions in the FactorAtlas-to-Fashionpedia setting (Appendix H.4), without using Fashionpedia images for calibration or hyperparameter selection. While visual-only retrieval improves mAP from .583 to .667, matched grounding reaches .719, outperforming visual-only across all seven backbones and 28 backbone–partition combinations. These results suggest that the generalization benefits of retaining native text observed under controlled synthetic conditions can also extend to natural images.

We would also like to clarify the interpretation of Appendix C.5. The analysis examines the role of native text by comparing finite grounding (validation-selected 0≤α<∞0 \leq \alpha < \infty) with the pure visual endpoint (α→∞\alpha \rightarrow \infty) across different levels of calibration support. While their performance becomes similar with abundant support, matched grounding shows substantial advantages under limited support.

These comparisons suggest that matched grounding and visual-only retrieval need not be functionally interchangeable, differing both in their performance under limited labeled support and in how well they transfer beyond the calibration conditions.

AuthorsOpenReview ↗

Response to Reviewer FxV6 - Part2

Together, our findings connect distinction-level diagnosis with matched visual intervention. Value-specific visual contrasts reduce residual linguistic access gaps even after broader text–image alignment, while native text provides a useful pretrained anchor when visual-only directions generalize poorly. These improvements extend to compositional retrieval and natural images without retraining the encoders or recomputing the gallery image embeddings.

We hope these clarifications and supporting evidence encourage the reviewer to reconsider the scientific contribution and significance of our work.


[1] Liang et al. (2022). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeurIPS 2022.

[2] Moayeri et al. (2023). Text-to-Concept (and Back) via Cross-Model Alignment. ICML 2023.

[3] Koishigarina et al. (2026). CLIP Behaves Like a Bag-of-Words Model Cross-modally but Not Uni-modally. ICLR 2026.

AuthorsOpenReview ↗

Manuscript Revision Update

To address your concerns regarding scientific novelty and the role of native text, we have incorporated the relevant clarifications and additional analyses from our rebuttal into the revised manuscript.

  • Scientific contribution (W1): We refined the Introduction (§1, L42–44, L50–52) and moved Related Work directly after it (§2) to more clearly articulate the scientific contribution and empirical scope of our distinction-level audit and matched intervention, and their relationship to established cross-modal mismatch and alignment research.

  • Functional role of native text (W2): We added Appendix D.1 (Table 14, L1104–1125) and expanded Appendix H.5 (Table 28, L1587–1606) with comparisons against pure visual retrieval under unseen semantic combinations and cross-domain transfer. These additions provide further evidence for the functional benefit of retaining native text beyond the calibration conditions.

We hope these revisions help clarify the scientific contribution and empirical significance of our work. We welcome any further questions or discussion regarding the revised manuscript and our responses.

We thank the reviewer again for their constructive feedback. We hope that our rebuttal responses and revised manuscript have adequately addressed the concerns raised in the review. If so, we would greatly appreciate reconsideration of the overall assessment and score.

We welcome any remaining questions or concerns and would be happy to address them during the discussion period.

Reviewer_wvz1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the asymmetry in vision-language models where fine visual distinctions are encoded in image embeddings but remain difficult to access via text queries. Using a controlled testbed called FactorAtlas, the authors show that visual discriminability often exceeds linguistic addressability. They propose "Matched Visual Grounding," a method that uses visual contrast directions to ground text queries, which improves retrieval performance for specific visual attributes.

Critical strengths

N/A

Critical weaknesses

1.Clarity and Presentation: The paper is currently difficult to follow. The narrative flow and organization could be improved to better guide the reader through the core concepts and contributions.

2.Limited Scope of Evaluation: The primary experiments rely heavily on a narrow, synthetic dataset (FactorAtlas). While the authors include experiments on real images, these are restricted to specific, low-level attributes (e.g., patterns, materials), which may not fully reflect the complexity of real-world scenarios.

3.Scalability of the Method: The proposed "target-versus-rest visual contrast" appears computationally expensive. Constructing these contrasts for large-scale, real-world databases seems impractical and introduces significant overhead.

4.Real-World Applicability: Given the reliance on synthetic data and the computational demands of the proposed method, there are significant concerns regarding its practical utility and scalability in large-scale, real-world image retrieval settings.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Response to Reviewer wvz1 - Part1

We thank the reviewer for the constructive feedback and positive assessment of soundness. We appreciate the opportunity to clarify the paper's motivation and practical implications.

Our responses address four related concerns:

  • Presentation (W1): Clarifying the progression from audit to intervention and consequences.
  • Evaluation scope (W2): Explaining why fine-grained distinctions matter and how broadly the calibrated visual contrasts generalize.
  • Scalability (W3): Clarifying the reuse and computational efficiency of value-specific grounding.
  • Practical utility (W4): Showing how targeted grounding supports compositional retrieval and cross-dataset reuse.

W1. Presentation and narrative

We recognize that the central motivation could be made more explicit in the Introduction to help readers follow the subsequent progression. We therefore clarify it here.

Contrastive vision-language models reliably match text and images for high-level semantic concepts, such as object categories like "dog" or "person", which underlies their strong zero-shot recognition. Matching is considerably less reliable for finer-grained visual properties such as surface pattern, hue, or shape [1], particularly when a query combines several of them [2]. Prior work also shows that attribute-object binding information present within each modality can remain poorly accessed cross-modally [3]. We therefore ask, for each such property, whether its distinction is encoded in frozen image geometry and, if so, how well the native text query can access it and whether that access can be improved.

Accordingly, the paper follows audit → intervention → consequences: diagnosing distinction-specific access gaps, testing under held-out conditions whether the corresponding visual contrasts can reduce them, and examining what the improved access enables, including compositional retrieval and transfer to natural images.

To make this progression clearer, we will refine the Introduction to open with this motivation and present the core concepts and contributions more explicitly, move Related Work (currently Sec. 5) directly after the Introduction to clarify our positioning relative to prior work, and connect the main findings to the held-out evaluations and their consequences.


W2. Evaluation scope and generalization

Fine-grained visual distinctions are well-established subjects of investigation in vision–language research [1, 2], rather than merely simplified proxies for broader scene understanding. These distinctions also arise naturally in practical settings, such as fashion and product search, where queries specify visual properties beyond object categories (e.g., a striped blue shirt, a midi-length white dress, or suede shoes) that native text matching may not reliably address. To study these distinctions systematically, we use FactorAtlas for controlled diagnosis, ruling out cross-factor shortcuts, and extend our analysis to real-world natural images to examine whether the same findings persist beyond the controlled setting.

Our natural-image evaluations cover diverse visual attributes and generalization conditions: textures in real photographs (DTD), garment patterns and lengths (Fashionpedia), materials in natural-image crops (COCO-Facet), and footwear materials across distinct products (UT-Zappos). These evaluations include held-out garment categories and product-disjoint testing, extending the analysis beyond the controlled synthetic testbed. Notably, FactorAtlas-derived visual contrasts transfer directly to Fashionpedia without target-domain fitting or validation (mAP .583 → .719; Appendix H.4).

These results show that distinction-specific access improvements extend beyond synthetic images to natural-image datasets, while remaining within the fine-grained scope examined in this work.

AuthorsOpenReview ↗

Response to Reviewer wvz1 - Part2

W3. Scalability and computational cost

We understand the reviewer's concern about the computational cost of constructing value-specific visual contrasts for large-scale retrieval. We would like to clarify that each contrast and its grounded query are computed once per named visual value (e.g., "striped") from a small labeled calibration set, not from the entire retrieval gallery. Once estimated, they can be stored and reused across all subsequent retrievals, regardless of the gallery size.

The main practical requirement is therefore labeled calibration support for each value, which need not be extensive. In our support-scaling analysis (Appendix G.2), 32 labeled images per value recover approximately 81% of the full-support access gain on SigLIP2 Base, averaged over three factors and two held-out protocols (unseen nuisance contexts and unseen hue-shape combinations). Moreover, contrasts estimated in one dataset can be reused in another without refitting, as shown by the FactorAtlas-to-Fashionpedia transfer in W2.


W4. Practical implications

We believe the responses to W1–W3 also address this concern. Fine-grained visual distinctions are themselves retrieval targets in practice (W1), our analysis extends beyond synthetic data to natural images (W2), and calibrated visual contrasts can be reused within existing frozen VLMs without retraining (W3).

Beyond these points, our experiments in Section 4 show that improved factor-level access supports compositional retrieval (§4.1) and that validation-guided grounding enables selective correction where beneficial (§4.2). These findings support practical use for predefined fine-grained visual distinctions, while extending this approach to broader open-vocabulary retrieval remains a direction for future work.


We hope these explanations and supporting evidence address the reviewer's concerns regarding evaluation scope, scalability, and practical applicability. We will incorporate the clarifications provided throughout this response into the revised manuscript.


[1] Zhao et al. (2022). VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations.

[2] Kamath et al. (2023). Text Encoders Bottleneck Compositionality in Contrastive Vision-Language Models. EMNLP 2023.

[3] Koishigarina et al. (2026). CLIP Behaves Like a Bag-of-Words Model Cross-modally but Not Uni-modally. ICLR 2026.

AuthorsOpenReview ↗

Manuscript Revision Update

To address your concerns regarding presentation and evaluation scope, we have incorporated the relevant clarifications from our rebuttal into the revised manuscript.

  • Motivation and presentation (W1): We refined the Introduction (§1, L42–44, L50–52, L59–60) and moved Related Work directly after it (§2) to clarify our positioning relative to prior work and make the progression from distinction-level audit to matched intervention and its retrieval consequences clearer.

  • Evaluation scope and generalization (W2): We expanded Appendix H.3 (L1469–1489, L1509–1514) to clarify the Fashionpedia and UT-Zappos cohort construction and held-out protocols. These revisions make the evaluation conditions and scope of the natural-image evidence more explicit, providing a clearer basis for assessing the generalization results discussed in our rebuttal.

We hope these revisions help clarify the motivation, empirical scope, and practical relevance of our work. We welcome any further questions or discussion regarding the revised manuscript and our responses.

We thank the reviewer again for their constructive feedback. We hope that our rebuttal responses and revised manuscript have adequately addressed the concerns raised in the review. If so, we would greatly appreciate reconsideration of the overall assessment and score.

We welcome any remaining questions or concerns and would be happy to address them during the discussion period.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of our controlled evaluations and diagnostic analyses, and the opportunity to clarify our scientific contribution, practical significance, and evaluation reliability.

Building on prior work on cross-modal mismatch, we take individual named visual distinctions as the unit of analysis and examine how their linguistic access gaps can be probed and reduced using corresponding image-side visual contrasts, including after broader text–image alignment.

Our distinction-level audit separates visual discriminability from native linguistic addressability, while matched visual grounding provides a controlled intervention supported by direction-specific and visual-evidence controls. Held-out evaluations further examine generalization beyond calibration conditions and the implications of improved access for retrieval.

Our individual responses address three complementary aspects:

  • Scope and practical relevance (wvz1): The relevance of fine-grained distinctions, generalization beyond synthetic images, and practical utility.
  • Novelty and linguistic contribution (FxV6): The distinction-level contribution beyond known mismatch and the functional role of native text.
  • Reproducibility and stability (kA9j): Evaluation construction, reproducibility materials, and partition- and attribute-level analyses of gain variability and access gaps.

We will incorporate these clarifications into the revised manuscript, and hope that they, together with the supporting evidence in our individual responses, provide a clearer account of our contribution and its empirical scope.

AuthorsOpenReview ↗

Manuscript Revision Update

Following our rebuttal, we have revised the manuscript to address the reviewers' concerns regarding scientific contribution and presentation, the role of native text, and evaluation scope and reliability.

We refined the Introduction and Related Work (§§1–2) to clarify the positioning, motivation, and progression of our distinction-level audit and matched intervention. We also incorporated additional analyses, experimental comparisons, and methodological details (Appendices B, D.1, and H.3–H.5) to provide further empirical support for our findings and directly address the questions raised during the review.

The corresponding revisions are detailed in our reviewer-specific updates. We sincerely thank the reviewers for their constructive feedback and welcome further questions or discussion regarding the revised manuscript and our responses.

Area_Chair_Np56OpenReview ↗

Reviewer FxV6 & wvz1] Urgent Reminder: Discussion Period Ends November 18

Dear Reviewers FxV6 & wvz1,

The discussion period will end on November 18, 2026. Please carefully review the authors' responses and actively engage in discussions.

If you have not yet initiated a discussion, please do so immediately. Reviewers are expected to address the authors' rebuttals, provide constructive follow-up comments, and update their evaluations when appropriate.

Please also ensure that your reviews reflect independent human judgment and that any AI assistance is properly documented in accordance with the review guidelines.

Active participation is an essential reviewer responsibility and is critical to ensuring a rigorous and fair review process.

Thank you for your cooperation.

AuthorsOpenReview ↗

Consolidated Discussion Update — Reviewer Concerns and Supporting Evidence

We sincerely thank the Area Chair and all reviewers for their constructive feedback. As the discussion period approaches its conclusion, we summarize how our rebuttal and November 16 revision further clarify and substantiate our original scientific contribution.

1. Reviewer wvz1 — Scope, Scalability, and Presentation

  • Synthetic evaluation / generalization → FactorAtlas was intentionally designed as a fully controlled testbed to isolate distinction-level linguistic access gaps. Our existing evaluations on DTD, Fashionpedia, COCO-Facet, and UT-Zappos demonstrate relevance beyond synthetic settings, including FactorAtlas-to-Fashionpedia transfer without target-domain fitting (mAP 0.583 → 0.719) (Appendix H).
  • Scalability → Visual contrasts are computed once per attribute value and reused across galleries; 32 labeled images per value recover ~81% of the full-support gain (Appendix G.2).
  • Presentation → We clarified the original progression from audit → matched intervention → retrieval consequences (§§1–2).

2. Reviewer FxV6 — Scientific Contribution and Native Text

  • Novelty / scientific insight → We clarified our central contribution as a controlled, distinction-level audit of visual discriminability versus linguistic addressability, together with matched interventions that probe and reduce the resulting access gaps. We also made its relationship to prior cross-modal mismatch and retrieval approaches more explicit (§§1–2).
  • Visual-only alternative → Further analysis of existing held-out and cross-domain evaluations shows that native text remains useful beyond calibration conditions. On Fashionpedia, matched grounding outperforms visual-only retrieval (0.719 vs. 0.667 mAP), supporting native text's role as a generalization anchor (Appendices D.1 and H.5).

3. Reviewer kA9j — Reproducibility and Stability

  • Recognized contribution / prior work → We appreciate the reviewer's recognition of our controlled audit and cross-domain evidence, and clarified their relationship to Rocchio and LABCLIP (§§1–2).
  • Reproducibility → We provided the requested rendering resources, factor specifications, sample IDs, mappings, and partitions to support verification (Appendices B and H.3; supplementary materials).
  • Stability → Partition- and attribute-level analyses show positive macro-mAP gains in 35/35 COCO-Facet and 26/35 UT-Zappos evaluations, while documenting small-cohort variability (Appendix H.4).

Together, these clarifications and analyses strengthen the evidence for our original central claim: visual distinctions can be discriminable in frozen image geometry yet insufficiently addressable through native text, and matched visual grounding can reduce these access gaps while retaining text supports generalization.

We would greatly appreciate the Area Chair's careful consideration of these clarifications and supporting analyses in the final assessment. We would also welcome the reviewers' perspectives on this evidence and any remaining concerns during the discussion period, and would be happy to provide further clarification.

We sincerely thank the Area Chair and reviewers for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.