acceptodds
Under review as a conference paper at ICLR 2027

Causally Motivated Logic Alignment for Prompt Tuning in Vision-Language Models

Abstract

Recent studies have shown that pre-trained vision-language models, such as CLIP, can effectively adapt to diverse downstream tasks through parameter-efficient prompt tuning. Unfortunately, the tuned models can exploit spurious correlations during prediction, resulting in a failure to generalize to out-of-distribution test data, especially when the tuning dataset exhibits bias. How to achieve cross-modal mitigation of spurious correlations during prompt tuning of vision-language models remains an open question. In this paper, the challenging problem is tackled by leveraging the stable relationship between necessary and sufficient causal features and the corresponding label. On the one hand, we constrain the learning process of prompt by reinforcing the necessary and sufficient connection between the textual labels and textual features. On the other hand, the probability of necessity and sufficiency between the textual features and the filtered visual features is measured and maximized to enhance cross-modal feature alignment. By simultaneously optimizing these two objectives, we can achieve cross-modal mitigation of spurious correlations because the logic equivalence between textual labels and visual features is bolstered. Theoretical analysis on generalization error indicates that our method achieves a tighter generalization error bound than existing approaches. We evaluate the proposed method, CaMLAP, on several widely used out-of-distribution datasets, and the empirical results demonstrate the superiority of our method over the state-of-the-art competitors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Models, Prompt Tuning, Out-of-Distribution Generalization, Causality, Probability of Necessity and Sufficiency, Spurious Correlations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kmreOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CaMLAP, an out-of-distribution generalization method that uses probability of necessity and sufficiency to reduce reliance on spurious correlations. Through cross-modal logic alignment and textual logic alignment, the method improves the robustness of vision-language models. Experiments on several benchmarks show promising results, with CaMLAP outperforming multiple prompt-tuning baselines in various benchmark.

Critical strengths
  • The proposed cross-modal logic alignment and textual logic alignment seems interesting and promising.

  • The proposed approach outperform various prompt tuning methods on the selected benchmarks.

Critical weaknesses
  • The experimental evidence is missing comparisons with many related works, such as [1, 2, 3], which makes the results less convincing.

  • The evaluation benchmarks used in the paper are limited compared with other related work [1, 2, 3]

Reference:

[1] CLIPood: Generalizing CLIP to Out-of-Distributions, ICML 23

[2] LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models, CVPR 23

[3] Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition, ICLR 25

Questions for authors

Provide more benchmarks results as outline in [1, 2, 3], and compare the performance.

Reference:

[1] CLIPood: Generalizing CLIP to Out-of-Distributions, ICML 23

[2] LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models, CVPR 23

[3] Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition, ICLR 25

1 comment
AuthorsOpenReview ↗

Response to Reviewer kmre

We thank the reviewer for recognizing the promise of textual and cross-modal logic alignment and the improvements over prompt-tuning baselines on the evaluated benchmarks. We address the requested comparisons and benchmark coverage below.

1. Additional comparisons with related methods

We have added comparisons with CLIPood, LASP, and PromptSRC†+SAS\mathrm{PromptSRC}^{\dagger}+\mathrm{SAS}. The last baseline incorporates SAS from Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition (ICLR 2025), addressing the reviewer's third reference. All reported values are means over three random seeds. The tables report Worst and Avg accuracy (%). Bold denotes the best result and underlining denotes the second-best result in each column. The ImageNet subset contains two classes.

ResNet-50

MethodWaterbirds WorstWaterbirds AvgCelebA WorstCelebA AvgImageNet subset WorstImageNet subset AvgPACS WorstPACS Avg
CLIPood58.983.464.983.860.489.979.791.6
LASP52.184.563.182.558.791.380.291.5
PromptSRC†+SAS\mathrm{PromptSRC}^{\dagger}+\mathrm{SAS}61.6‾\underline{61.6}85.8‾\underline{85.8}68.6‾\underline{68.6}86.3‾\underline{86.3}84.5‾\underline{84.5}93.2‾\underline{93.2}81.1‾\underline{81.1}92.0‾\underline{92.0}
CaMLAP67.586.269.987.390.295.182.493.7

ViT-B/32

MethodWaterbirds WorstWaterbirds AvgCelebA WorstCelebA AvgImageNet subset WorstImageNet subset AvgPACS WorstPACS Avg
CLIPood49.568.268.184.867.589.482.593.2
LASP47.375.667.785.280.791.581.993.6
PromptSRC†+SAS\mathrm{PromptSRC}^{\dagger}+\mathrm{SAS}54.3‾\underline{54.3}80.571.4‾\underline{71.4}86.1‾\underline{86.1}88.9‾\underline{88.9}93.8‾\underline{93.8}83.9‾\underline{83.9}94.7‾\underline{94.7}
CaMLAP61.280.3‾\underline{80.3}73.186.991.895.484.395.2

CaMLAP achieves the highest worst-group/domain accuracy in all eight settings among these methods. Compared with the strongest added baseline, the gains range from 0.4 to 6.9 percentage points, averaging 3.26 points across the eight settings. Average accuracy is higher in seven settings; on Waterbirds with ViT-B/32, it is slightly lower (80.3% versus 80.5%). These results strengthen the empirical comparison while preserving the main emphasis on performance in the worst group or domain.

2. Benchmark coverage and generalization scope

The four benchmarks cover distinct sources of distribution shift: background association in Waterbirds, gender association in CelebA, object co-occurrence in the two-class ImageNet subset, and visual style in PACS. Using both ResNet-50 and ViT-B/32 further examines whether the observed gains persist across different encoder architectures.

We agree that this evaluation does not cover the full benchmark suites or all generalization protocols considered in the cited works. Our conclusions are restricted to the evaluated group and domain shifts. Extending CaMLAP to more settings is a valuable direction for future work. We will incorporate the added comparisons and make the evaluation scope explicit in a future version.

We thank the reviewer again for recognizing the dual-alignment design and benchmark improvements, and for the constructive suggestions. We will incorporate the additional results and these clarifications into a future version, and use the broader benchmark recommendations to guide future improvements.

Reviewer_BEMqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Causally Motivated Logic Alignment for Prompting (CaMLAP) to mitigate spurious correlations in vision-language models under distribution shifts. The method introduces two objectives based on the probability of necessity and sufficiency (PNS): textual logic alignment between class labels and textual representations, and cross-modal logic alignment between textual representations and filtered visual representations. Learnable prompts and visual projection layers are optimized through a minimax objective combining these terms with classification loss. The authors also present a mutual-information-based generalization bound. Experiments on Waterbirds, CelebA, an ImageNet subset, and PACS report improved worst-group or worst-domain accuracy across two CLIP backbones.

Critical strengths
  1. The manuscript is well-written and easy to follow.
  2. This paper targets the important generalization issue in prompt learning. The experiments show the effectiveness and the theoretical analysis is solid.
Critical weaknesses
  1. The main comparison experiments are very insufficient. The compared approaches are old-fashioned, mainly concentrated on 23/24 published papers. I am also curious about why the comparison does not follow the standard evaluation protocol (e.g., base-to-novel generalization, ImageNet-based cross-dataset generalization) that are applied in PromptSRC, DePT, etc.
  2. The causal interpretation of the proposed objectives is insufficiently justified. Identifying PNS requires exogeneity and monotonicity, but Appendix D mainly states these assumptions without establishing their validity for learned CLIP representations. In particular, replacing prompts or visual projection layers does not automatically constitute a valid intervention on the underlying semantic causes. The selected Grad-CAM visualizations are also insufficient to establish that the method identifies necessary and sufficient features. Controlled counterfactual experiments, such as changing backgrounds while preserving foreground objects, would provide more direct evidence.
  3. Important implementation details of the PNS objectives are missing. Equation (4) involves indicator functions, representation distributions, KL regularization, and a separation constraint, but the manuscript does not specify the differentiable surrogates, distribution parameterizations, prior estimation, or constraint enforcement used in practice. The binary sign-based formulation is also not explicitly extended to multiclass classification on PACS. Appendix E provides standard training settings but does not adequately describe the alternating minimization/maximization procedure or the construction and updating of visual class prototypes. These omissions substantially hinder reproducibility.
  4. The experimental reporting leaves uncertainty about robustness and evaluation scope. No standard deviations or confidence intervals are provided, making small improvements on CelebA and PACS difficult to assess. The ImageNet experiment uses only two classes and a custom train/test split, whose sample counts and construction should be specified. Hyperparameter selection and validation protocols are also unclear.
Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer BEMq -Part 1

We thank the reviewer for recognizing the clarity of the manuscript, the importance of generalization in prompt learning, the experimental effectiveness, and the theoretical analysis. We appreciate the detailed concerns and respond to each below.

1. Baseline coverage and choice of evaluation protocol

We have added comparisons with CLIPood [1], LASP [2], and PromptSRC†+SAS[3]\mathrm{PromptSRC}^{\dagger}+\mathrm{SAS [3]}. All reported values are means over three random seeds. The tables report Worst and Avg accuracy (%). Bold denotes the best result and underlining denotes the second-best result in each column. The ImageNet subset contains two classes.

ResNet-50

MethodWaterbirds WorstWaterbirds AvgCelebA WorstCelebA AvgImageNet WorstImageNet AvgPACS WorstPACS Avg
CLIPood58.983.464.983.860.489.979.791.6
LASP52.184.563.182.558.791.380.291.5
PromptSRC†+SAS\mathrm{PromptSRC}^{\dagger}+\mathrm{SAS}61.6‾\underline{61.6}85.8‾\underline{85.8}68.6‾\underline{68.6}86.3‾\underline{86.3}84.5‾\underline{84.5}93.2‾\underline{93.2}81.1‾\underline{81.1}92.0‾\underline{92.0}
CaMLAP67.586.269.987.390.295.182.493.7

ViT-B/32

MethodWaterbirds WorstWaterbirds AvgCelebA WorstCelebA AvgImageNet WorstImageNet AvgPACS WorstPACS Avg
CLIPood49.568.268.184.867.589.482.593.2
LASP47.375.667.785.280.791.581.993.6
PromptSRC†+SAS\mathrm{PromptSRC}^{\dagger}+\mathrm{SAS}54.3‾\underline{54.3}80.571.4‾\underline{71.4}86.1‾\underline{86.1}88.9‾\underline{88.9}93.8‾\underline{93.8}83.9‾\underline{83.9}94.7‾\underline{94.7}
CaMLAP61.280.3‾\underline{80.3}73.186.991.895.484.395.2

CaMLAP achieves the highest worst-group/domain accuracy in all eight settings among these methods. Compared with the strongest added baseline, the gains range from 0.4 to 6.9 percentage points, averaging 3.26 points across the eight settings. Average accuracy is higher in seven settings; on Waterbirds with ViT-B/32, it is slightly lower (80.3% versus 80.5%). These results strengthen the empirical comparison while preserving the main emphasis on performance in the worst group or domain.

Our evaluation targets changes in class-context associations and visual domains within a fixed label space. Base-to-novel evaluation instead tests transfer to unseen classes, while ImageNet-based cross-dataset evaluation tests transfer to different target datasets. These protocols address complementary aspects of generalization. The selected benchmarks allow us to examine background, gender, object co-occurrence, and style shifts that are directly relevant to our stated problem. They do not establish performance under unseen-class or broad cross-dataset transfer. We will make this scope explicit and incorporate the additional baseline results in a future version; broader transfer benchmarks remain a future research direction.

[1] CLIPood: Generalizing CLIP to Out-of-Distributions, ICML 23

[2] LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models, CVPR 23

[3] Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition, ICLR 25

AuthorsOpenReview ↗

Response to Reviewer BEMq -Part 2

2. Causal interpretation and the PNS assumptions

(1) Empirical evidence supporting spurious-correlation mitigation

Our quantitative results, component ablations, and Grad-CAM visualizations provide complementary evidence that CaMLAP mitigates reliance on spurious correlations under the evaluated distribution shifts. In the additional comparisons, CaMLAP achieves higher worst-group/domain accuracy in all eight dataset-backbone settings than the three added baselines, and higher average accuracy in seven settings relative to the strongest added baseline. These improvements cover shifts involving background, gender, object co-occurrence, and visual style.

The component ablations further support the contribution of the proposed objectives. In Table 2, removing cross-modal logic alignment reduces worst-group/domain accuracy by 15.9, 39.7, 11.6, and 1.7 percentage points on Waterbirds, CelebA, the two-class ImageNet subset, and PACS, respectively; removing textual logic alignment reduces the corresponding accuracies by 2.0, 2.4, 1.6, and 1.6 points. The Grad-CAM visualizations in Figures 2 and 3 also show more object-focused activation and less emphasis on spuriously correlated context, including backgrounds in Waterbirds and baby faces in pacifier classification. Together, these results support the effectiveness of the dual-alignment design in mitigating spurious-correlation reliance on the evaluated benchmarks.

(2) PNS assumptions and future semantic causal identification

We agree that exogeneity and monotonicity are assumptions supporting the PNS identification formula, as described in Appendix D. The auxiliary prompt and projection branches construct alternatives in representation space for a PNS-motivated objective; replacing these components does not automatically certify a valid intervention on the underlying semantic causes. Our current experiments support the mitigation effect but do not establish that all learned representations satisfy these assumptions.

We also agree that Grad-CAM cannot establish identification of necessary and sufficient semantic causes. Strict semantic causal identification and quantitative assessment of semantic necessity and sufficiency are themselves interesting and challenging topics in causal discovery and causal representation learning. Integrating CaMLAP with explicit causal-identification methods, and developing dedicated quantitative criteria and interventional or counterfactual evaluation protocols for this purpose, are valuable directions for our future research.

We will incorporate these explanations into a future version, clearly distinguishing the empirical evidence for spurious-correlation mitigation, the assumptions underlying the PNS formulation, and the stronger requirements for semantic causal identification.

3. Implementation details and reproducibility

We clarify the architectural roles in Eqs. (5)-(7). Both CLIP encoders remain frozen. The trainable components are two textual prompts, QQ and Qˉ\bar Q, and two linear visual filters, hh and hˉ\bar h.

  • Cross-modal term: textual class embeddings act as classifiers for the filtered visual representations, comparing the principal and auxiliary visual branches.
  • Textual term: class prototypes derived from the filtered visual space act as classifiers for the principal and auxiliary textual representations through cosine-similarity logits.
  • Optimization and inference: We use alternating gradient updates for the min-max objective in Eq. (7): the minimization step updates Q,hQ,h with the auxiliary branch held fixed, and the maximization step updates Qˉ,hˉ\bar Q,\bar h with the principal branch held fixed. The CLIP encoders remain frozen throughout. Inference uses only Q,hQ,h.

For PACS, the prediction rule uses the K-class cosine-softmax classifier, with seven classes. However, this prediction rule alone does not specify the multiclass extension of the PNS training loss.

We agree that these structural definitions and the standard optimizer settings in Appendix E are insufficient for complete reproduction. The submission lacks an explicit account of the differentiable surrogates, representation distributions and priors, KL computation, separation-constraint handling, multiclass PNS formulation, update frequency, and prototype construction/update rules. We acknowledge this documentation limitation. A future version will expand the implementation specification and algorithm description to make these choices explicit, alongside the code release stated in Appendix E.

AuthorsOpenReview ↗

Response to Reviewer BEMq -Part 3

4. Statistical reporting, ImageNet scope, and hyperparameters

The reviewer is correct that the ImageNet experiment is a two-class subset consisting of water bottles and baby pacifiers, organized into three groups: water bottles, pacifiers without babies, and pacifiers with babies. It evaluates a specific object-co-occurrence shortcut, rather than full ImageNet-1K classification. Appendix E describes transferring part of the original training data to the test set, but does not provide sufficient sample counts or split-construction details. We will label the subset consistently and document the split more precisely in a future version.

All reported experimental metrics are averages over three random seeds. We agree that the manuscript should have stated this explicitly. The results therefore summarize multiple runs, although the absence of standard deviations or confidence intervals still limits the interpretation of small margins, particularly on PACS. We report the observed mean-accuracy differences without claiming statistical significance. The additional comparisons strengthen baseline coverage but do not replace uncertainty estimates.

Regarding loss weights, Table 3 shows that α∈{10,20,30}\alpha\in\{10,20,30\} gives 86.1%, 90.2%, and 88.7% worst-group accuracy versus 78.6% at zero, and β∈{0.1,1,10}\beta\in\{0.1,1,10\} gives 89.7%, 90.2%, and 89.3%. The PNS weights are selected using the validation set. Together, the sensitivity results and three-seed averages support the reported performance, while leaving uncertainty around small margins insufficiently quantified. We will incorporate the validation-selection procedure and three-seed reporting protocol into a future version; broader statistical evaluation and adaptive weighting remain future improvement directions.

We thank the reviewer again for recognizing the importance and promise of the work and for the constructive, detailed feedback. We will incorporate these explanations, the additional baseline results, and the identified reporting improvements into a future version.

Reviewer_BVQZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of spurious correlations in pretrained vision-language models (VLMs), such as CLIP, which may rely on features that are predictive in the training distribution but fail to generalize under distribution shifts. To address this issue, the authors propose CaMLAP, a prompt-tuning method that incorporates objectives motivated by the Probability of Necessity and Sufficiency (PNS) to improve out-of-distribution (OOD) generalization. The main idea is to reduce reliance on unstable or spurious features while encouraging more robust and reliable alignment between visual and textual representations.

Critical strengths

The use of Probability of Necessity and Sufficiency (PNS)-motivated objectives for both textual and cross-modal (visual–textual) alignment to improve out-of-distribution (OOD) generalization is interesting.

The paper evaluates CaMLAP across multiple datasets and two backbone architectures, demonstrating improvements in worst-group accuracy.

The paper provides useful ablation studies that help assess the contributions of the proposed components.

Critical weaknesses
  1. Insufficient evidence of reduced reliance on spurious correlations: Improvements in worst-group accuracy and Grad-CAM visualizations do not directly establish that CaMLAP reduces reliance on spurious correlations. More targeted analyses are needed to validate this central claim.

  2. Sensitivity to loss hyperparameters: The proposed method appears sensitive to the weighting of the PNS objectives, particularly the hyperparameter (ϕ\phi). This sensitivity may make hyperparameter tuning challenging and limit the method's robustness and applicability across different datasets.

Questions for authors
  1. Hyperparameter selection and robustness: Could the authors clarify the validation and hyperparameter selection protocols, particularly for the PNS loss weights? It would also be valuable to report performance stability across datasets and multiple random seeds to assess the robustness of the proposed method.

  2. Quantitative analysis of spurious correlations: Could the authors quantitatively evaluate how model predictions change when known spurious attributes are systematically modified while preserving the class-relevant features? Such an experiment would provide more direct evidence that CaMLAP reduces reliance on spurious correlations, beyond improvements in worst-group accuracy and qualitative Grad-CAM visualizations.

1 comment
AuthorsOpenReview ↗

Response to Reviewer BVQZ

We thank the reviewer for recognizing the interest of using PNS-motivated objectives for both textual and cross-modal alignment, the improvements in worst-group accuracy across datasets and backbones, and the usefulness of our ablation studies. We address the two main concerns below.

1. Evidence concerning reliance on spurious correlations

(1) Quantitative improvements and Grad-CAM evidence of mitigation

Our quantitative results and Grad-CAM visualizations provide complementary evidence that CaMLAP mitigates spurious correlations on the evaluated benchmarks. The group-based evaluations specifically examine settings where dominant training associations become unreliable, including bird species versus background in Waterbirds and pacifier presence versus baby-face co-occurrence in the ImageNet subset.

In the additional comparisons, CaMLAP outperforms all three added baselines in worst-group/domain accuracy across all eight dataset-backbone settings, while achieving higher average accuracy in seven of the eight settings relative to the strongest added baseline. For example, with ResNet-50, CaMLAP improves Worst/Avg accuracy on Waterbirds from 61.6%/85.8% to 67.5%/86.2%, and on the ImageNet subset from 84.5%/93.2% to 90.2%/95.1%, compared with PromptSRC†+SAS\mathrm{PromptSRC}^{\dagger}+\mathrm{SAS}. These results demonstrate improvements on the groups most affected by the training shortcuts while maintaining competitive average performance.

The Grad-CAM visualizations in Figures 2 and 3 further support this interpretation: CaMLAP shows more object-focused activation and less emphasis on spuriously correlated context, such as backgrounds in Waterbirds and baby faces in pacifier classification. The comparison with the variant without textual logic alignment also illustrates the contribution of the textual objective to reducing attention to distracting features. Together, the Worst/Avg improvements, component ablations, and visualization results support the effectiveness of CaMLAP in mitigating reliance on spurious correlations under the evaluated distribution shifts.

(2) Identification of necessary and sufficient semantic causes

We agree with the reviewer that Grad-CAM cannot establish identification of necessary and sufficient semantic causes. Quantitatively identifying such causes is itself an interesting and challenging topic in causal discovery and causal representation learning, requiring explicit interventional or counterfactual criteria beyond activation localization. We therefore distinguish the empirical evidence of spurious-correlation mitigation from the stronger claim of identifying semantic causes.

Integrating CaMLAP with causal identification, and developing quantitative criteria and evaluation protocols for identifying necessary and sufficient semantic causes, are valuable directions for our future research. We will incorporate these explanations into a future version and clarify both the supporting role of the current evidence and the scope of the causal-identification claims.

2. Loss weights, hyperparameter selection, and stability

We agree that the weight of cross-modal alignment requires care. Table 3 also shows that the improvement is not confined to a single isolated setting. On the ImageNet subset, with β=1\beta=1, choosing α∈{10,20,30}\alpha\in\{10,20,30\} yields worst-group accuracies of 86.1%, 90.2%, and 88.7%, compared with 78.6% when α=0\alpha=0. With α=20\alpha=20, choosing β∈{0.1,1,10}\beta\in\{0.1,1,10\} yields 89.7%, 90.2%, and 89.3%, a spread of 0.9 percentage points. This supports an effective interval of regularization strengths, particularly for the textual term. Larger weights degrade performance, so we do not claim insensitivity to arbitrary choices.

The PNS weights are selected using the validation set, and all reported experimental metrics are averages over three random seeds. Appendix E reports dataset-specific (α,β)(\alpha,\beta) settings of (3, 2) for Waterbirds, (10, 1) for CelebA, and (20, 1) for the ImageNet subset. Thus, the reported results summarize multiple runs, and the weights are chosen according to validation performance. We agree that these details should have been stated explicitly. Although three-seed averaging reduces dependence on a single run, the absence of standard deviations or confidence intervals limits claims about statistical stability. We will incorporate more clarification on the validation-selection procedure and three-seed reporting protocol into a future version.

We thank the reviewer again for recognizing the dual-alignment design, empirical improvements, and component analyses, and for the constructive suggestions. We will incorporate these clarifications and address the identified reporting limitations in a future version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.