acceptodds
Under review as a conference paper at ICLR 2027

Beyond Best-Prompt AUROC: Textual Reachability of Distribution Shifts in VLM

Abstract

Models that connect images and text are increasingly used to describe visual shifts between image collections. A common approach is based on testing many textual descriptions and selecting the one that most strongly separates the two collections, as measured by Prompt AUROC. Yet, strong separation does not necessarily imply a meaningful explanation of the shift. In our experiments, the prompt "necessity", unrelated to blur, almost perfectly separates original images from blurred versions of them (Prompt AUROC 0.999). Our analysis explains how a large visual shift and a search over many descriptions can produce this result. Under a Gaussian model, we derive that Prompt AUROC depends on both the magnitude of the visual shift and how well text aligns with it. We show that we can largely predict the highest Prompt AUROC from the magnitude of the visual shift and the directional coverage of the candidate text embeddings, independently of the prompts' meanings. We thus introduce textual reachability, a label-free measure of the separation achievable through text relative to the best linear separation available from image embeddings. Across 113 diverse shifts, it distinguishes those captured by text from those where arbitrary words achieve similarly high AUROC. It also predicts 1) whether discovered words keep their alignment for other VLM backbones and 2) the performance gains from using text to adapt models to new image collections. Best-prompt AUROC correlates negatively with both.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models, Distribution shift, Interpretability, Domain adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gLGdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a failure mode in VLM-based shift descriptions: selecting the prompt with the highest Prompt AUROC can yield semantically irrelevant explanations, since large visual shifts and broad prompt searches can produce accidental separations. The authors explain this geometrically and introduce textual reachability, a label-free metric comparing text-aligned separation with the best possible visual linear separation, calibrated by a permutation baseline. Across 113 shifts, textual reachability better distinguishes meaningful descriptions from spurious prompts and correlates more strongly with prompt transfer and text-guided adaptation than best-prompt AUROC.

Critical strengths
  1. The decomposition of Prompt AUROC into visual shift magnitude and text alignment is elegant and helps explain the empirical phenomenon.
  2. The evaluation is comprehensive. It evaluates 113 shifts across synthetic transformations, real domain shifts, generated-image shifts, and acquisition shifts.
  3. The paper connects the metric to cross-encoder transfer and text-guided adaptation gains, showing that textual reachability is not only conceptually cleaner but also more predictive of downstream utility.
Critical weaknesses
  1. The paper’s strongest failure cases stem from exhaustive searches over large vocabularies of readable single-token words, creating a multiple-comparisons issue that can surface irrelevant terms like “necessity” or “rages.” Yet most methods for describing image-set differences use image-conditioned captioning, VLM queries, or LLM-guided proposals, which constrain candidates to visually grounded descriptions. Thus, while the paper identifies a real statistical artifact, its practical significance may be overstated without evidence that it appears in realistic image-conditioned pipelines.
  2. The paper argues that some discovered prompts are meaningful while others are unrelated, but it does not fully formalize the semantic evaluation. A more systematic human study or automated semantic audit would strengthen this claim.
  3. Textual reachability relies on covariance whitening, Mahalanobis shift magnitude, candidate-bank baselines, and normalization. Its high computational complexity and lack of intuitive interpretation may hinder its wider adoption.
Questions for authors
  1. A generic prompt such as “low quality” or “different style” may align with many shifts and achieve high reachability, while a precise prompt may have lower coverage but better explanatory value. Does the ranking favor broad descriptors?
  2. Since the method relies on estimating covariance and whitening in high-dimensional embedding spaces, how sensitive is R to sample size, covariance regularization, and class imbalance?
  3. How expensive is this step in practice for large datasets or large candidate banks? Are there approximate or analytic alternatives that preserve the benefits of calibration while reducing computation?
  4. The current downstream results appear largely correlational. If prompts are selected by reachability instead of AUROC, does this lead to better text-guided adaptation performance across tasks?
  5. How sensitive are the reported reachability scores to the size and semantic coverage of the candidate set? Could low reachability sometimes reflect missing or insufficiently expressive prompts rather than an inherent lack of linguistic describability of the visual shift?
1 comment

I thank the authors for the detailed response and additional results. The sensitivity studies and throuput analysis can greatly strengthen the submission. However, I remain skeptical towards the practical significance of this paper. The paper does not directly evaluate a realistic image-conditioned pipeline where descriptions are generated from the images and then selected. So it remains unclear how often the highlighted semantically irrelevant high-AUROC prompts would occur in practical workflows. The downstream evidence also remains diagnostic rather than showing actual improved adaptation decisions. I would prefer to retain my current score.

Reviewer_wvKuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that best-prompt AUROC can be misleading when evaluating whether language meaningfully describes a visual distribution shift. A prompt may achieve very high AUROC simply because two image collections are strongly separated in a vision-language model’s embedding space, even when the selected word has little semantic relationship to the actual difference between them. This problem becomes worse when searching thousands of prompts, because some unrelated word can align with the shift by chance. To address this, the paper introduces textual reachability, which measures how much of the underlying visual difference can genuinely be captured by text while accounting for both the magnitude of the visual shift and the alignment expected simply from searching many prompts. Across 113 distribution shifts, textual reachability better distinguishes meaningful descriptions from arbitrary high-AUROC prompts and better predicts whether discovered descriptions transfer across different vision-language models and whether text-guided adaptation improves performance.

Critical strengths

The paper presents that best-prompt AUROC is not a reliable measure of whether a text description actually captures a visual distribution shift. High AUROC can arise simply because the source and target image distributions are far apart in the VLM embedding space, allowing even weakly related or arbitrary words to separate them. Searching over many prompts further amplifies this problem because some unrelated prompt will align with the shift by chance. The paper theoretically decomposes this effect and empirically shows that visual shift magnitude and prompt-search geometry alone can largely predict best-prompt AUROC, without considering the meanings of the prompts.

The paper then introduces textual reachability, measure designed to determine whether a visual distribution shift is genuinely captured by pool of language. Instead of measuring only how well the best prompt separates two image sets, textual reachability measures how much of the available visual separation can be recovered through text, while correcting for alignment that would arise from searching many prompts by chance.

Across 113 distribution shifts, reachability better distinguishes meaningful descriptions from spurious high-AUROC prompts and positively predicts cross-VLM transfer and gains from text-guided adaptation, suggesting that it better reflects when language provides a useful representation of a visual shift.

Critical weaknesses
  • The paper proposes a separability measure of two groups of images given VML and pool of text, whereas Best-Prompt AUROC is proposing a method to separate images given VLM. I.E. the paper proposes a measure of how well a visual distribution shift between two image groups can be captured by a VLM and a pool of candidate texts, whereas Best-Prompt AUROC measures how well a selected text prompt can distinguish the two image groups. These metrics address related but somewhat different questions and may therefore be complementary. The reviewer is not fully convinced that the extensive analysis of Best-Prompt AUROC is necessary to motivate the proposed metric, and this analysis makes the paper more difficult to follow.

  • The proposed textual reachability measure appears relatively simple and heuristic, largely adapting standard image-space separability and alignment concepts to evaluate text-based representations of visual shifts. The paper could better clarify what is technically novel about the measure beyond this combination.

  • The paper would benefit from a more intuitive explanation of both textual reachability and the limitations of Best-Prompt AUROC before introducing the mathematical formulation. Although the proposed method itself is relatively simple, its current presentation makes the core idea unnecessarily difficult to understand.

  • A most of the evaluation relies on synthetic image transformations (Reviewer's this assertion may not incorrect. the reviewer understood this based on the result table in the paper). While these experiments provide useful controlled settings, it remains unclear how well the proposed measure reflects textual reachability for complex, naturally occurring distribution shifts. Additional validation or discussion of realistic scenarios would strengthen the empirical contribution.

Questions for authors

.

Reviewer_d4FjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how vision-language models describe visual shifts between image collections and questions whether Prompt AUROC is a reliable measure of description quality. Existing methods such as VisDiff search over candidate text descriptions and select the one that best separates the two collections based on CLIP scores. The authors argue that high AUROC does not necessarily indicate a meaningful description of the visual shift. Using a Gaussian model, they show that AUROC depends on both the magnitude of the shift and the alignment of the text with the shift direction. They further demonstrate that searching over many candidate descriptions can produce high AUROC even when the descriptions are not semantically relevant. To address this limitation, they propose textual reachability, a metric that measures how much of the optimal linear separability can be recovered through text while accounting for chance performance. Experiments across synthetic, generative, medical, and stylistic shifts suggest that textual reachability better captures whether a shift can be meaningfully described in language. The authors also report positive correlations between textual reachability and downstream cross-encoder transfer and text-guided adaptation gains, in contrast to the negative correlations observed with best-prompt AUROC.

Critical strengths

1- The paper identifies an important limitation of best-prompt AUROC, showing that high scores do not necessarily indicate meaningful descriptions of visual shifts. The examples of unrelated descriptions achieving high AUROC are convincing, and the Gaussian analysis provides a clear theoretical explanation of how shift magnitude and text alignment affect the metric.

2- The experimental design is controlled, with positive and negative controls, separate data splits, and a permutation baseline that accounts for the full search process. The cross-encoder transfer and text-guided adaptation experiments also explore the practical relevance of the proposed metric.

Critical weaknesses

1- The generality of the claims is not fully supported by the experiments. Although the paper evaluates 113 shifts, most involve image corruptions, generator differences, hospital acquisition differences, or style changes, while content-level shifts are largely absent. The natural-domain pairs also use shared classes and class-stratified sampling, limiting semantic variation. Furthermore, Appendix L reports substantially larger prediction errors for semantic-domain shifts (0.135–0.165) than for other shifts (approximately 0.06–0.07), raising questions about the generality of the meaning-independent AUROC prediction. The authors should evaluate more content-level shifts or narrow their claims accordingly.

2- The validation of textual reachability is insufficient. The proposed metric is compared only against best-prompt AUROC on the transfer and adaptation outcomes, which the paper already identifies as a weak measure of semantic alignment. Simpler alternatives, such as uncorrected alignment (κmax), the AUROC-scale variant (ΔA), and raw cosine excess, are not evaluated on these outcomes, leaving the necessity of the costly permutation correction unclear. Moreover, despite introducing reference descriptions for semantic assessment, the paper provides only selected qualitative examples rather than a quantitative evaluation of how well reachability distinguishes meaningful from irrelevant descriptions across the registry.

Questions for authors

1- How does textual reachability perform on content-level shifts, for example collections that differ in the frequency of certain objects, scenes, or activities? Does the geometric prediction of best-prompt AUROC still hold in that setting, given the larger errors reported for semantic-domain shifts in Appendix L? 2- How does R compare with simpler alternatives, such as uncorrected alignment κmax, the AUROC-scale variant ΔA, or the raw cosine excess of Appendix J, on the transfer and adaptation outcomes in Table 2? This would show whether the permutation baseline is necessary. 3- Can you report the results of the semantic assessment mentioned in Section 4 quantitatively, for example how well R separates correct from incorrect descriptions across all shifts with reference descriptions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.