- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a semantic robustness auditing framework for Vision-Language Models, aiming to evaluate prediction stability under semantic variations. The authors first construct semantic directions from contrastive text pairs and aggregate them across different textual contexts to improve consistency. A cross-modal correction mechanism is then introduced to align these directions with the visual embedding space. Based on the resulting semantic representations, the framework derives closed-form boundary margins to quantify prediction stability along different semantic directions. Experiments validate the alignment between language-derived directions and controlled visual variations, and demonstrate the framework's application to robustness analysis across datasets, prompt learning methods, and VLM architectures.
Critical strengths
-
Meaningful research problem. The paper investigates the robustness of VLMs under different semantic variations, aiming to go beyond conventional accuracy-based evaluations by analyzing prediction stability at the attribute and class levels. This fine-grained auditing perspective provides useful insights into model behavior across different datasets, prompts, and model settings.
-
Clear and interpretable method design. The paper constructs semantic directions from contrastive text pairs, combines aggregation across textual contexts with cross-modal correction, and derives closed-form decision boundary margins along these directions. The overall framework is clearly structured and enables the analysis of prediction stability without explicitly generating semantically modified images.
-
Relatively comprehensive experimental evaluation. The authors first validate the alignment between language-derived semantic directions and controlled visual variations. They then conduct audits across multiple datasets, prompt learning methods, and VLM architectures, providing further analyses at both the attribute and class levels. These experiments demonstrate the potential applicability of the proposed framework across different evaluation settings.
Critical weaknesses
-
Insufficient validation of the robustness metric. Although the paper demonstrates alignment between language-derived and visual semantic directions, it remains unclear whether the proposed Semantic Boundary Margin reliably reflects robustness under actual visual semantic variations.
-
Ambiguity in the definition of semantic robustness. The framework equates prediction invariance with robustness, without sufficiently considering that the importance of semantic attributes varies across object categories. For example, the colors and textures of certain animals are closely related to their biological characteristics and can be important for distinguishing species. In contrast, for man-made objects such as teapots, structural features like the spout and handle are often more important for identifying the object, while surface colors and decorative patterns can vary considerably without changing its identity. Therefore, sensitivity to certain semantic variations may reflect appropriate category discrimination rather than poor robustness. The paper does not sufficiently distinguish between these cases when interpreting semantic boundary margins.
-
Incomplete experimental details. Appendix A.2 states “We set δmax = X,” which appears to be an unreplaced placeholder. Since this parameter affects the computed margins, its missing value raises reproducibility concerns.
Questions for authors
Please refer to Weaknesses 1 and 2. I would particularly appreciate clarification on the validity of the proposed robustness metric and the distinction between semantic robustness and legitimate sensitivity to class-discriminative attributes.
4 comments
Looking forward to further feedback
We thank the reviewer again for the careful review and constructive feedback. With the discussion period nearing its end, we would be grateful if the reviewer could take a look at our responses and let us know whether the concerns have been addressed. This would allow us to address any remaining concerns before the discussion closes.
For convenience, we briefly summarize the main additions reported in our responses and incorporated into the revised manuscript.
-
Metric validation (W1). We conducted new controlled experiments on PUG to directly test whether larger semantic margins correspond to smaller drops in prediction accuracy under visual semantic variations. The results show strong agreement at both the individual-attribute and attribute-family levels, while component ablations demonstrate the benefits of aggregation and affine correction.
Manuscript updates: The new results and analyses have been added to Section 5.1 (Table 1). -
Class-discriminative sensitivity (W2). We clarified how semantic margins quantify prediction stability and how their robustness interpretation depends on the task and the attribute’s discriminative role. By providing both the margin and the class predicted after the first flip, the framework offers a basis for examining whether a transition reflects a robustness failure or appropriate category discrimination. We also outlined future work on robustness criteria that account for attribute roles and expected class transitions.
Manuscript updates: These discussions and future directions have been added to Section 6. -
Experimental details (W3). We corrected the typo in App. A.2 and specified the maximum audited semantic extent as .
Our earlier responses and the revised manuscript provide the supporting results and detailed explanations. We hope these additions and clarifications address the main concerns, and we would be very grateful if the reviewer would consider updating the rating in light of these additions.
Thank you for the thoughtful response and additional experiments, which have addressed my main concerns. I appreciate the authors' efforts and have decided to increase my rating.
We sincerely thank the reviewer for the positive feedback and for carefully considering our additional experiments and clarifications. We are pleased that our responses have addressed the main concerns.
Response to Reviewer CcKi
We sincerely appreciate the reviewer's insightful comments and helpful suggestions. We provide our point-by-point responses below and hope they satisfactorily address the concerns raised.
W1. Validity and interpretation of the semantic boundary margin.
To assess whether semantic boundary margins reliably reflect prediction robustness under visual semantic variations, we conducted additional experiments on PUG using CLIP ViT-B/16 across 12 controlled variations. Within each class, we examined whether attributes with larger average semantic margins exhibit smaller drops in prediction accuracy. We report robustness agreement (Agreement), measured as the fraction of attribute comparisons for which semantic margins correctly predict the ordering of accuracy drops. Individual compares the 12 variations, whereas Family compares the four families using their average margins and accuracy drops. As shown below, semantic margins agree well with observed robustness differences at both levels, supporting their reliability in assessing VLM robustness under semantic transformations. We have expanded Section 5.1 with these robustness agreement results.
W2. Semantic robustness versus class-discriminative sensitivity.
We thank the reviewer for highlighting this important distinction. We agree that sensitivity to attributes used to distinguish classes, such as color differences between animal species, can be appropriate. Larger margins are therefore not universally preferable. However, our framework provides quantitative support for examining this distinction. It provides exact margins along the audited semantic directions and identifies the class predicted after the first prediction flip. These outputs quantify the semantic extent required for a prediction change and identify the competing class, supporting task-specific assessment of whether the transition reflects a robustness failure or appropriate category discrimination. We have further clarified in Section 6 that margin values should be interpreted in light of the task and the role of the audited attribute. For fine-grained classification, this includes considering the meaning of the semantic change together with the class predicted after the flip. We have also outlined future work on robustness criteria that account for semantic variation and expected class transitions.
W3. Missing value of δmax.
Our updated manuscript now clarifies in Appendix A.2 that δmax=1.0 represents the maximum audited semantic extent.