acceptodds
Under review as a conference paper at ICLR 2027

Can Language Represent Semantic Variation? A Semantic Basis for VLM Robustness Audits

Abstract

Vision-Language Models (VLMs) underpin a broad range of real-world AI applications. Yet understanding their robustness under semantic variation remains difficult, since such variations are hard to formalize directly in image space. We therefore ask whether language can provide a reliable representation of semantic variation for robustness auditing. We construct semantic directions from semantic-contrast text pairs and aggregate them across textual contexts for consistency. Cross-modal correction further calibrates these directions to the visual embedding space. Controlled visual validation shows that language-derived semantic representations align well with corresponding visual semantic variations. Building on this semantic representation, we establish a semantic robustness auditing framework for VLMs that quantifies prediction stability along semantic variation with closed-form boundary margins. The framework supports auditing across data, prompt, and model contexts, while enabling fine-grained diagnosis of where robustness is preserved or degraded. Our audits reveal systematic variation in semantic robustness across contexts alongside stable semantic patterns. Domain-specific applications further show that the audit localizes fragile patterns aligned with task priors and guides model selection under target-domain robustness requirements. Overall, our framework provides a principled basis for evaluating and diagnosing VLM robustness under semantic variation.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models, Semantic Robustness, Robustness Auditing, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CcKiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a semantic robustness auditing framework for Vision-Language Models, aiming to evaluate prediction stability under semantic variations. The authors first construct semantic directions from contrastive text pairs and aggregate them across different textual contexts to improve consistency. A cross-modal correction mechanism is then introduced to align these directions with the visual embedding space. Based on the resulting semantic representations, the framework derives closed-form boundary margins to quantify prediction stability along different semantic directions. Experiments validate the alignment between language-derived directions and controlled visual variations, and demonstrate the framework's application to robustness analysis across datasets, prompt learning methods, and VLM architectures.

Critical strengths
  1. Meaningful research problem. The paper investigates the robustness of VLMs under different semantic variations, aiming to go beyond conventional accuracy-based evaluations by analyzing prediction stability at the attribute and class levels. This fine-grained auditing perspective provides useful insights into model behavior across different datasets, prompts, and model settings.

  2. Clear and interpretable method design. The paper constructs semantic directions from contrastive text pairs, combines aggregation across textual contexts with cross-modal correction, and derives closed-form decision boundary margins along these directions. The overall framework is clearly structured and enables the analysis of prediction stability without explicitly generating semantically modified images.

  3. Relatively comprehensive experimental evaluation. The authors first validate the alignment between language-derived semantic directions and controlled visual variations. They then conduct audits across multiple datasets, prompt learning methods, and VLM architectures, providing further analyses at both the attribute and class levels. These experiments demonstrate the potential applicability of the proposed framework across different evaluation settings.

Critical weaknesses
  1. Insufficient validation of the robustness metric. Although the paper demonstrates alignment between language-derived and visual semantic directions, it remains unclear whether the proposed Semantic Boundary Margin reliably reflects robustness under actual visual semantic variations.

  2. Ambiguity in the definition of semantic robustness. The framework equates prediction invariance with robustness, without sufficiently considering that the importance of semantic attributes varies across object categories. For example, the colors and textures of certain animals are closely related to their biological characteristics and can be important for distinguishing species. In contrast, for man-made objects such as teapots, structural features like the spout and handle are often more important for identifying the object, while surface colors and decorative patterns can vary considerably without changing its identity. Therefore, sensitivity to certain semantic variations may reflect appropriate category discrimination rather than poor robustness. The paper does not sufficiently distinguish between these cases when interpreting semantic boundary margins.

  3. Incomplete experimental details. Appendix A.2 states “We set δmax = X,” which appears to be an unreplaced placeholder. Since this parameter affects the computed margins, its missing value raises reproducibility concerns.

Questions for authors

Please refer to Weaknesses 1 and 2. I would particularly appreciate clarification on the validity of the proposed robustness metric and the distinction between semantic robustness and legitimate sensitivity to class-discriminative attributes.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CcKi

We sincerely appreciate the reviewer's insightful comments and helpful suggestions. We provide our point-by-point responses below and hope they satisfactorily address the concerns raised.

W1. Validity and interpretation of the semantic boundary margin.

To assess whether semantic boundary margins reliably reflect prediction robustness under visual semantic variations, we conducted additional experiments on PUG using CLIP ViT-B/16 across 12 controlled variations. Within each class, we examined whether attributes with larger average semantic margins exhibit smaller drops in prediction accuracy. We report robustness agreement (Agreement), measured as the fraction of attribute comparisons for which semantic margins correctly predict the ordering of accuracy drops. Individual compares the 12 variations, whereas Family compares the four families using their average margins and accuracy drops. As shown below, semantic margins agree well with observed robustness differences at both levels, supporting their reliability in assessing VLM robustness under semantic transformations. We have expanded Section 5.1 with these robustness agreement results.

MethodAgreement ↑ (Individual)Agreement ↑ (Family)
Ours73.2%81.7%
− Template aggregation71.4%76.7%
− Affine correction53.3%64.7%

W2. Semantic robustness versus class-discriminative sensitivity.

We thank the reviewer for highlighting this important distinction. We agree that sensitivity to attributes used to distinguish classes, such as color differences between animal species, can be appropriate. Larger margins are therefore not universally preferable. However, our framework provides quantitative support for examining this distinction. It provides exact margins along the audited semantic directions and identifies the class predicted after the first prediction flip. These outputs quantify the semantic extent required for a prediction change and identify the competing class, supporting task-specific assessment of whether the transition reflects a robustness failure or appropriate category discrimination. We have further clarified in Section 6 that margin values should be interpreted in light of the task and the role of the audited attribute. For fine-grained classification, this includes considering the meaning of the semantic change together with the class predicted after the flip. We have also outlined future work on robustness criteria that account for semantic variation and expected class transitions.

W3. Missing value of δmax⁡\delta_{\max}.

Our updated manuscript now clarifies in Appendix A.2 that δmax⁡=1.0\delta_{\max}=1.0 represents the maximum audited semantic extent.

AuthorsOpenReview ↗

Looking forward to further feedback

We thank the reviewer again for the careful review and constructive feedback. With the discussion period nearing its end, we would be grateful if the reviewer could take a look at our responses and let us know whether the concerns have been addressed. This would allow us to address any remaining concerns before the discussion closes.

For convenience, we briefly summarize the main additions reported in our responses and incorporated into the revised manuscript.

  • Metric validation (W1). We conducted new controlled experiments on PUG to directly test whether larger semantic margins correspond to smaller drops in prediction accuracy under visual semantic variations. The results show strong agreement at both the individual-attribute and attribute-family levels, while component ablations demonstrate the benefits of aggregation and affine correction.
    Manuscript updates: The new results and analyses have been added to Section 5.1 (Table 1).

  • Class-discriminative sensitivity (W2). We clarified how semantic margins quantify prediction stability and how their robustness interpretation depends on the task and the attribute’s discriminative role. By providing both the margin and the class predicted after the first flip, the framework offers a basis for examining whether a transition reflects a robustness failure or appropriate category discrimination. We also outlined future work on robustness criteria that account for attribute roles and expected class transitions.
    Manuscript updates: These discussions and future directions have been added to Section 6.

  • Experimental details (W3). We corrected the typo in App. A.2 and specified the maximum audited semantic extent as δmax⁡=1.0\delta_{\max}=1.0.

Our earlier responses and the revised manuscript provide the supporting results and detailed explanations. We hope these additions and clarifications address the main concerns, and we would be very grateful if the reviewer would consider updating the rating in light of these additions.

Thank you for the thoughtful response and additional experiments, which have addressed my main concerns. I appreciate the authors' efforts and have decided to increase my rating.

We sincerely thank the reviewer for the positive feedback and for carefully considering our additional experiments and clarifications. We are pleased that our responses have addressed the main concerns.

Reviewer_mh4EOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether language can represent semantic variation in VLMs for robustness auditing. The method constructs semantic directions from semantic-contrast text pairs, aggregates them across templates and class contexts, and calibrates them to the visual embedding space through affine cross-modal correction. Experiments validate the visual alignment of these directions on PUG: ImageNet and conduct audits across datasets, prompt-learning methods, and VLMs, examining attribute-level and class-level robustness differences as well as non-uniform robustness scaling.

Critical strengths

S1. The research question is interesting and practically meaningful. Different modalities vary in their expressive capabilities and the information they preserve, and language often provides a highly compressed description of visual content. Investigating whether language can adequately represent semantic variation is therefore a valuable research direction.

S2. The method is clearly structured. It addresses both directional variation caused by textual contexts and cross-modal mismatch, with controlled visual validation and ablation studies examining the contributions of aggregation and affine correction.

S3. The experiments cover diverse auditing contexts and demonstrate the framework’s potential for diagnosing fine-grained robustness differences.

Critical weaknesses

W1. The alignment of semantic directions is insufficient to establish the practical validity of semantic boundary margins. The experiments primarily assess agreement between aggregated language-derived and visual directions, without directly testing whether the estimated margins predict prediction flips or robustness rankings under actual visual transformations.

W2. The method normalizes margins using the average geodesic distance between corrected text embeddings, but it remains unclear whether the same normalized margin corresponds to comparable visual semantic variation across different VLMs. An ablation study on the normalization would help clarify its contribution and its effect on cross-model comparisons.

Questions for authors

Please see the weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer mh4E

We sincerely thank the reviewer for the careful assessment and valuable feedback. Below, we respond to each point in turn and hope our clarifications address the reviewer's concerns.

W1. Practical validity of semantic boundary margins.

To directly assess whether semantic margins reflect prediction robustness under visual transformations, we conducted additional experiments on PUG using CLIP ViT-B/16 across 12 semantic variations. Within each class, we tested whether attributes with larger semantic margins are associated with smaller drops in prediction accuracy. The table below reports robustness agreement (Agreement), calculated as the fraction of attribute comparisons for which semantic margins correctly predict the ordering of accuracy drops. Individual compares the 12 variations, while Family compares the four families using their average margins and accuracy drops. The agreement with observed visual robustness at both levels supports the practical validity of semantic margins for assessing VLM robustness. Affine correction provides the largest gain, while template aggregation further improves agreement. We have revised Section 5.1 to include this validation and the corresponding component ablations.

MethodAgreement ↑ (Individual)Agreement ↑ (Family)
Ours73.2%81.7%
− Template aggregation71.4%76.7%
− Affine correction53.3%64.7%

W2. Semantic scale normalization and cross-model comparability.

We conducted an additional ablation and found that semantic-scale normalization improves cross-model comparability. We evaluated four VLMs (CLIP ViT-B/16, OpenCLIP ViT-B/16, EVA02-B/16, and SigLIP/14) under controlled semantic transformations. For each class and attribute setting, we tested whether raw and normalized mean margins correctly indicate which models experience smaller drops in prediction accuracy. The table below reports robustness agreement for both individual semantic variations and attribute families. Normalization improves robustness agreement at both levels, supporting normalized margins as a more reliable basis for cross-model comparisons. We have incorporated these cross-model results into Section 6, together with an explanation of how normalization supports reliable comparisons.

MethodAgreement ↑ (Individual)Agreement ↑ (Family)
Ours75.4%80.9%
− Scale normalization49.0%46.4%
Reviewer_mh4EOpenReview ↗

Response to authors

The authors addressed most of my concerns. Therefore, I will keep my positive score. Thank you.

We sincerely thank the reviewer for the positive feedback and for carefully considering the additional results and clarifications. We are pleased that the metric validation and normalization analysis address the remaining concerns.

Reviewer_tjc6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper uses language-derived semantic directions to analyze prediction stability in vision-language models. It combines aggregation across templates and categories with affine calibration to reduce contextual variation and cross-modal mismatch. The method computes the first classification-boundary crossing along each direction and compares the resulting margins across data, prompts, and models.
Experiments include controlled visual validation and analyses at the attribute, class, and application levels.

Critical strengths
  1. The paper makes robustness comparisons more specific by examining individual attributes and classes. The finding that overall improvements can accompany declines in particular attributes or categories illustrates what this analysis adds beyond aggregate metrics.

  2. Aggregation and cross-modal calibration have clear motivations. The authors examine textual consistency and test correspondence with controlled visual changes in PUG. The substantial improvement from affine calibration in Figure 5 provides empirical support for this design choice.

  3. Once a semantic direction is given, the boundary margin can be computed directly without an iterative search. The appendix also examines how directional errors affect the margin, allowing the method to assess sensitivity rather than merely report a score.

Critical weaknesses
  1. The validation establishes directional alignment more directly than margin validity. The PUG experiments measure average directional similarity and attribute retrieval. The more consequential test is whether images that flip earlier under actual visual changes also receive smaller semantic margins. This would directly test the usefulness of the audit score.

  2. The contribution over closely related work needs a more specific comparison. Existing methods [1] also use language-defined transformations and closed-form prediction-invariant intervals. The paper should distinguish the benefits of aggregation, affine calibration, and cross-context auditing, ideally by comparing audit performance under matched settings rather than directional consistency alone.

  3. The dependence of cross-model rankings on scale normalization is unclear. Dividing the margin by the average spherical distance between paired prompts directly affects model comparisons. Reporting rankings before and after normalization would help establish whether the main conclusions are sensitive to this choice.

[1] Semantic Robustness Certification for Vision-Language Models, 2026. https://arxiv.org/abs/2606.18839

Questions for authors
  1. Can the authors directly test the relationship between semantic margins and observed classification flips under controlled PUG variations? For which attributes does the relationship hold best or break down?

  2. Compared with existing language-driven semantic robustness methods, do aggregation and calibration improve the prediction of actual classification changes?

  3. How much do cross-model rankings change without scale normalization? Which conclusions about model robustness depend on this choice?

3 comments
AuthorsOpenReview ↗

Response to Reviewer tjc6

We sincerely thank the reviewer for the thoughtful comments and constructive suggestions. We address each concern below and hope our responses help clarify and strengthen our work.

W1&Q1. Practical validity of semantic boundary margins.

To directly test whether semantic margins reflect robustness under visual transformations, we conducted additional experiments on PUG using CLIP ViT-B/16 across 12 semantic variations. Within each class, we assessed whether attributes with larger semantic margins exhibit smaller drops in prediction accuracy. The table below reports robustness agreement (Agreement), calculated as the fraction of attribute comparisons for which semantic margins correctly predict the ordering of accuracy drops. Individual compares the 12 variations, while Family compares the four families using their average margins and accuracy drops. The results show that semantic margins agree well with prediction robustness, supporting their practical validity for measuring VLM robustness. Affine correction provides the largest gain, while template aggregation further improves agreement. We have revised Section 5.1 to include this direct validation.

MethodAgreement ↑ (Individual)Agreement ↑ (Family)
Ours73.2%81.7%
− Template aggregation71.4%76.7%
− Affine correction53.3%64.7%

We also examined which attributes show stronger or weaker agreement with observed prediction robustness. Agreement is highest within the Scale family (86.7%) and lowest within Background (65.2%). Our analysis points to two contributing factors. First, background variations involve diverse scene content that is difficult to capture with a simple text prompt such as “in a restaurant”. Second, we observe less realistic scenes in PUG images under background visual variations, which also contributes to the gap.

W2&Q2. Comparison with existing methods.

To assess whether our design improves audit performance, we conducted additional matched comparisons with existing methods. Our work targets systematic audits of semantic robustness across data, prompt, and model contexts, whereas existing methods use semantic directions for sample-level interpretation or certification. Existing language-driven methods, including CounTEX (Kim et al., CVPR'23), SpLiCE (Bhalla et al., NeurIPS'24), and SemCert (Yang et al., ICML'26), focus on counterfactual explanations, interpretable concept decomposition, and semantic robustness certification, respectively, without systematically investigating the reliability of semantic directions for cross-context comparisons. To support reliable cross-context comparisons, our framework uses aggregation to reduce dependence on textual context, affine cross-modal correction to improve text–image alignment, and semantic-scale normalization to make margins comparable. The results below show that our method achieves higher robustness agreement than existing methods. Together with the ablations in W1/Q1, they demonstrate that our framework makes margin comparisons more reliable for cross-context auditing. We have revised Section 5.1 to include these matched comparisons.

MethodAgreement ↑ (Individual)Agreement ↑ (Family)
CounTEX55.1%62.7%
SpLiCE54.7%60.7%
SemCert58.6%63.0%
Ours73.2%81.7%

W3&Q3. Effect of semantic scale normalization on cross-model comparisons.

To assess how semantic normalization affects cross-model comparisons, we conducted an additional ablation comparing raw and normalized margins. Semantic-scale normalization can change the ordering of models by their margins because the normalization factor is model-specific. We estimate these factors from the same text-defined semantic contrasts. The table below reports results for four VLMs (CLIP ViT-B/16, OpenCLIP ViT-B/16, EVA02-B/16, and SigLIP/14) under controlled semantic variations. For each class and attribute, we tested whether raw and normalized mean margins correctly identify which models experience smaller drops in prediction accuracy under the semantic variation. The results show that normalized margins more accurately identify which models are more robust at both levels of semantic granularity, supporting the role of normalization in cross-model comparisons. We have further clarified this interpretation and included the normalization ablation in Section 6.

MethodAgreement ↑ (Individual)Agreement ↑ (Family)
Ours75.4%80.9%
− Scale normalization49.0%46.4%

The authors explained their details in the rebuttal. They also added more results. Most of my concerns have been addressed. I will keep my positive score. Thank you.

We sincerely thank the reviewer for the positive assessment and for taking the time to review the additional results and clarifications. We are glad that the metric validation, matched comparisons, and normalization analysis address the remaining concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Key Contributions

We sincerely thank the reviewers for their constructive feedback and the Area Chair for their time and consideration. Our work investigates whether language can reliably represent visual semantic variation and thereby provide a principled basis for auditing VLM robustness. The additional experiments and discussions further substantiate three key contributions of our work.

1. Reliable language-based representation of visual semantic variation (all reviewers). Our framework enables semantic robustness auditing without explicitly generating semantically varied images. Through new controlled experiments on PUG, we demonstrate that language-derived semantic margins reliably reflect relative robustness under actual visual variations. These results establish the practical value of language as a basis for semantic robustness auditing.

  • Manuscript updates: New results and analyses are included in Section 5.1 (Table 1).

2. Reliable semantic robustness comparisons across contexts (tjc6 and mh4E). Our framework enables systematic robustness comparisons across data, prompt, and model contexts. In particular, our new experiments across four VLMs demonstrate that semantic normalization substantially improves agreement with observed visual robustness by accounting for scale differences in how models represent the same semantic variation. These results substantiate the effectiveness of our framework in enabling reliable cross-model auditing.

  • Manuscript updates: New results and analysis are included in Section 6 (Table 2).

3. Fine-grained robustness diagnosis and task-specific insights (CcKi). Our audits can reveal robustness disparities that aggregate performance metrics overlook, including attribute-specific and class-specific degradation despite overall robustness improvements. By quantifying when predictions change and identifying the resulting predicted classes, our framework further provides a quantitative basis for distinguishing robustness failures from appropriate category discrimination and assessing robustness under task-specific requirements.

  • Manuscript updates: Further discussion on task-specific robustness assessment is included in Section 6.

Following our rebuttal, all three reviewers explicitly acknowledged that their major concerns had been addressed.

Our work establishes language as a practical basis for semantic robustness auditing, advancing VLM evaluation beyond aggregate performance measurement toward fine-grained robustness diagnosis across diverse contexts. We believe that our contributions and new insights into VLM robustness will be of broad interest to the ICLR community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.