Causally Motivated Logic Alignment for Prompt Tuning in Vision-Language Models

Recent studies have shown that pre-trained vision-language models, such as CLIP, can effectively adapt to diverse downstream tasks through parameter-efficient prompt tuning. Unfortunately, the tuned models can exploit spurious correlations during prediction, resulting in a failure to generalize to out-of-distribution test data, especially when the tuning dataset exhibits bias. How to achieve cross-modal mitigation of spurious correlations during prompt tuning of vision-language models remains an open question. In this paper, the challenging problem is tackled by leveraging the stable relationship between necessary and sufficient causal features and the corresponding label. On the one hand, we constrain the learning process of prompt by reinforcing the necessary and sufficient connection between the textual labels and textual features. On the other hand, the probability of necessity and sufficiency between the textual features and the filtered visual features is measured and maximized to enhance cross-modal feature alignment. By simultaneously optimizing these two objectives, we can achieve cross-modal mitigation of spurious correlations because the logic equivalence between textual labels and visual features is bolstered. Theoretical analysis on generalization error indicates that our method achieves a tighter generalization error bound than existing approaches. We evaluate the proposed method, CaMLAP, on several widely used out-of-distribution datasets, and the empirical results demonstrate the superiority of our method over the state-of-the-art competitors.