- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the gap between "visual representations containing certain attribute information" and "text queries being able to effectively access that information" in frozen visual language models. The authors constructed a FactorAtlas containing 23,040 images to calibrate the visual orientation of the target attribute relative to other attributes and incorporated it into text queries. The paper validates its effectiveness on seven models, various hold-out protocols, combined retrieval, and natural image data. The main contributions are controlled attribute access auditing and experimental evidence of cross-modal retrieval mismatch.
Critical strengths
The core diagnostics are supported by comprehensive control experiments. FactorAtlas cross-controls semantic factors and separates interference scenarios for calibration, validation, and testing. The paper examines alternative explanations such as prototype attraction, mismatch orientation, cue word variation, visual signal cancellation, and purely visual endpoints. In particular, Appendix C.5 examines whether the gains simply come from replacing text queries with visual classifiers, making the main conclusions more credible than simply showing retrieval improvements.
The gains are broad and have direct cross-domain evidence. The average mAP of the seven models improves from .660/.619/.567 to .803/.786/.862 for shape, color, and texture, respectively. Even more convincingly, without fitting the Fashionpedia target domain, directly transferring the four texture orientations from FactorAtlas still improves the average mAP from .583 to .719. This supports the finding that the orientations have value beyond the calibration distribution.
Critical weaknesses
The methodological novelty is limited, and the added knowledge needs to be more clearly defined. The update of "original query + difference in positive and negative visual centroids" is similar to the classic Rocchio relevance feedback; its target direction relative to other categories is merely a scaling of the central category prototype. The observation that "information exists in a single-modal representation, but cross-modal access fails" is closely related to the cited LABCLIP. Therefore, the main added value of this paper lies in audit design and empirical findings, where the current explanation of the relationship between these existing concepts is still insufficient.
The key data construction details for audit contributions are insufficient for verification. The appendix does not clearly define the generation rules for the ten textures and the specific definitions for the twelve colors; the natural image experiment also does not list the four clothing categories reserved by Fashionpedia, the fixed footwear subset of UT-Zappos, or the complete sample partitioning. No accessible code or data entry points are found in the PDF. These choices directly affect the task difficulty and potential category relevance, which is particularly important for papers that primarily contribute to controlled auditing.
Questions for authors
Clarification of Materials: Could you provide the FactorAtlas rendering script, natural image sample IDs, attribute mappings, and partitioning files, especially the specific category selections for Fashionpedia and UT-Zappos?
Supplementary Evidence Affecting Judgment: Could Table 22 report the AP and gain fluctuations for each partition and attribute? Especially for COCO-Facet, which only contains 306 images, and UT-Zappos, which has 96 products, it is currently difficult to determine the stability of migration gains.
4 comments
Response to Reviewer kA9j - Part2
Q2. Partition- and attribute-level AP and gain variability
To directly address the reviewer's request for partition- and attribute-level AP and gain fluctuations, we report native AP, matched AP, and their differences for every partition attribute combination across all five settings. Complete results are provided in the supplementary material (results/partition_tables.md). The following table summarizes macro-mAP gains and variability across seven frozen backbones:
| Setting | Gain ± SD | Positive evaluations |
|---|---|---|
| DTD | +.233 ± .008 | 70/70 |
| Fashionpedia pattern | +.181 ± .028 | 28/28 |
| Fashionpedia length | +.108 ± .005 | 35/35 |
| COCO-Facet | +.237 ± .040 | 35/35 |
| UT-Zappos | +.057 ± .063 | 26/35 |
SD is the mean of the across-partition gain SDs computed separately for each backbone. Fashionpedia length uses calibration-support resamples evaluated on a fixed test gallery.
All 168 backbone partition evaluations across the first four settings show positive macro-mAP gains. At the attribute level, Fashionpedia pattern improves across all 24 category–attribute means, while DTD improves for 45/47 texture classes. For Fashionpedia length, matched AP remains stable across support resamples on the fixed test gallery (SD across attributes), although the magnitude of improvement varies across length attributes.
Given the reviewer's particular concern about the smaller cohorts, we further examine COCO-Facet (306 images) and UT-Zappos (96 products) at the attribute and partition levels. The following table reports per-rotation AP gains alongside attribute-level visual–native AP gaps, comparing image-derived visual retrieval (visual discriminability) with native linguistic access (linguistic addressability):
| Attribute | R1 | R2 | R3 | R4 | R5 | Visual–native gap | Matched gain |
|---|---|---|---|---|---|---|---|
| COCO: Metal | +.22 | +.19 | +.19 | +.18 | +.11 | +.193 | +.177 |
| COCO: Stone | +.41 | +.22 | +.34 | +.45 | +.40 | +.322 | +.367 |
| COCO: Wood | +.14 | +.22 | +.18 | +.13 | +.16 | +.170 | +.167 |
| UT: Leather | +.01 | -.00 | +.07 | -.12 | +.02 | -.066 | -.004 |
| UT: Suede | -.02 | +.00 | +.01 | -.06 | +.04 | -.026 | -.003 |
| UT: Patent Leather | +.04 | +.12 | +.08 | +.02 | +.29 | +.098 | +.109 |
| UT: Synthetic | +.35 | +.10 | +.00 | +.16 | +.03 | +.101 | +.126 |
Seven-backbone means. R1–R5: per-rotation AP gains. Visual–native gap: image-derived visual retrieval minus native text retrieval AP; matched gain: matched grounding minus native text AP, averaged across rotations.
COCO-Facet shows positive gains for every attribute and rotation at the seven-backbone mean, although Stone exhibits substantial native AP variability (.40–.61) with only 4–9 test images per rotation. In contrast, UT-Zappos shows greater rotation-level variability, with positive macro-mAP gains in 26/35 backbone–rotation evaluations. These results highlight the variability associated with evaluating relatively small natural-image cohorts, which we acknowledge as a limitation.
Looking at the attribute-level results, matched access gains track the magnitude of the visual–linguistic discrepancy: in COCO-Facet, Stone has the largest gap (+.322) and gain (+.367), while Metal and Wood show smaller gaps and gains. In UT-Zappos, Leather and Suede have negative gaps and negligible gains, whereas Patent Leather and Synthetic have positive gaps and correspondingly larger gains.
Together, these results acknowledge partition-level variability in smaller natural-image cohorts while providing additional attribute-level evidence consistent with the intended role of matched grounding in improving linguistic access where a gap exists.
Overall, these clarifications aim to make our distinction-level audit and its empirical contributions more transparent and reproducible. We hope that the additional materials and partition-resolved results address the reviewer's questions and concerns, and provide a clearer basis for assessing both the strengths and limitations of our findings.
Manuscript Revision Update
To address your concerns, we have incorporated the clarifications and additional analyses from our rebuttal into the revised manuscript.
- Relationship to prior work (W1): We revised the Introduction (§1, L42–44, L50–52) and Related Work (§2, L91–96) to clarify our contribution relative to Rocchio relevance feedback and LABCLIP. These revisions more clearly establish the controlled distinction-level audit and matched intervention as tools for examining value-specific linguistic access gaps, including after broader alignment.
- Construction and reproducibility (W2/Q1): We expanded Appendix B (Tables 5–6, L756–786) and Appendix H.3 (L1469–1489, L1509–1514) with explicit FactorAtlas specifications and natural-image cohort protocols. These details make the construction of the evaluated visual distinctions and held-out conditions more transparent and verifiable. We also updated the Reproducibility Statement (L558–565) to summarize the supporting appendix materials and state our plan to publicly release the code and reproducibility materials upon de-anonymization.
- Empirical stability and access gaps (Q2): We added Appendix H.4 (Tables 26–27, L1517–1565) to report partition-level variability and attribute-level visual–native gaps alongside matched gains. These analyses help establish where access gains are consistent and how their magnitude relates to the underlying linguistic access gap, while documenting variability in smaller natural-image cohorts.
The accompanying supplementary material provides the detailed construction resources and partition-by-attribute results described in our rebuttal.
We hope these revisions strengthen the transparency and empirical foundation of our distinction-level audit. We welcome any further questions or discussion regarding the revised manuscript and our responses.
We thank the reviewer again for their constructive feedback. We hope that our rebuttal responses and revised manuscript have adequately addressed the concerns raised in the review. If so, we would greatly appreciate reconsideration of the overall assessment and score.
We welcome any remaining questions or concerns and would be happy to address them during the discussion period.
Response to Reviewer kA9j - Part1
We thank the reviewer for the thoughtful assessment, particularly for recognizing the value of our controlled audit and diagnostic experiments. We appreciate the specific suggestions regarding reproducibility and empirical stability.
We address three points:
W1. Relationship to prior work
We thank the reviewer for pointing out the connection to classical Rocchio relevance feedback. We agree that our query update shares a similar centroid-difference operation. As the reviewer recognizes, our primary contribution lies in the controlled audit design and empirical findings: we derive distinction-specific contrasts from frozen image embeddings and use them to ground native text queries, testing how much linguistic access can be improved through matched intervention.
In addition, our distinction-level analysis complements prior work on cross-modal mismatch, including LABCLIP. While LABCLIP improves attribute–object correspondence through a shared alignment transformation, we examine linguistic access at the level of individual named visual distinctions and test whether matched visual contrasts can reduce access gaps even after broader alignment (§3.4).
We will revise the Introduction to make our contribution clearer and bring the relevant Related Work discussion forward to better explain how our distinction-level audit and matched intervention relate to established relevance-feedback and cross-modal alignment approaches.
W2/Q1. Construction and reproducibility
We appreciate the reviewer's request for more explicit cohort-construction details and reproducibility materials. To clarify the protocols described in Appendices B and H, we summarize the key construction and partition choices below.
FactorAtlas: 8 shapes × 12 hues × 10 patterns × 24 nuisance realizations (12 contexts × 2 renderer replicas). Twelve fixed linear-RGB hues and 10 patterns defined by procedural Blender shaders (Wave, Checker, Voronoi, Noise, Brick) plus a uniform surface. Contexts 0–3/4–5/6–11 define calibration/validation/test.
Fashionpedia: Six patterns across four garment categories (dress, shirt/blouse, skirt, top/T-shirt/sweatshirt), holding out each category in turn. For dress length, the 447 dress crops of the official validation split serve as a fixed, untouched test gallery.
COCO-Facet: 306 images (172 metal, 29 stone, 105 wood), evaluated with five image-disjoint rotations.
UT-Zappos: Women/Shoes/Heels/Slip-On, with 96 products across four materials and three toe-style × heel-height cells (11, 8, 5 products per material), evaluated with product-disjoint rotations.
To address the reviewer's reproducibility concerns, we provide additional supplementary materials for this rebuttal, including the rendering script, FactorAtlas specification, per-cell render seeds, source-image IDs, attribute mappings, exact partition assignments, and scripts that rebuild and verify the natural-image cohorts from raw annotations. These materials enable independent verification of the evaluation cohorts.