- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies a failure mode in VLM-based shift descriptions: selecting the prompt with the highest Prompt AUROC can yield semantically irrelevant explanations, since large visual shifts and broad prompt searches can produce accidental separations. The authors explain this geometrically and introduce textual reachability, a label-free metric comparing text-aligned separation with the best possible visual linear separation, calibrated by a permutation baseline. Across 113 shifts, textual reachability better distinguishes meaningful descriptions from spurious prompts and correlates more strongly with prompt transfer and text-guided adaptation than best-prompt AUROC.
Critical strengths
- The decomposition of Prompt AUROC into visual shift magnitude and text alignment is elegant and helps explain the empirical phenomenon.
- The evaluation is comprehensive. It evaluates 113 shifts across synthetic transformations, real domain shifts, generated-image shifts, and acquisition shifts.
- The paper connects the metric to cross-encoder transfer and text-guided adaptation gains, showing that textual reachability is not only conceptually cleaner but also more predictive of downstream utility.
Critical weaknesses
- The paper’s strongest failure cases stem from exhaustive searches over large vocabularies of readable single-token words, creating a multiple-comparisons issue that can surface irrelevant terms like “necessity” or “rages.” Yet most methods for describing image-set differences use image-conditioned captioning, VLM queries, or LLM-guided proposals, which constrain candidates to visually grounded descriptions. Thus, while the paper identifies a real statistical artifact, its practical significance may be overstated without evidence that it appears in realistic image-conditioned pipelines.
- The paper argues that some discovered prompts are meaningful while others are unrelated, but it does not fully formalize the semantic evaluation. A more systematic human study or automated semantic audit would strengthen this claim.
- Textual reachability relies on covariance whitening, Mahalanobis shift magnitude, candidate-bank baselines, and normalization. Its high computational complexity and lack of intuitive interpretation may hinder its wider adoption.
Questions for authors
- A generic prompt such as “low quality” or “different style” may align with many shifts and achieve high reachability, while a precise prompt may have lower coverage but better explanatory value. Does the ranking favor broad descriptors?
- Since the method relies on estimating covariance and whitening in high-dimensional embedding spaces, how sensitive is R to sample size, covariance regularization, and class imbalance?
- How expensive is this step in practice for large datasets or large candidate banks? Are there approximate or analytic alternatives that preserve the benefits of calibration while reducing computation?
- The current downstream results appear largely correlational. If prompts are selected by reachability instead of AUROC, does this lead to better text-guided adaptation performance across tasks?
- How sensitive are the reported reachability scores to the size and semantic coverage of the candidate set? Could low reachability sometimes reflect missing or insufficiently expressive prompts rather than an inherent lack of linguistic describability of the visual shift?
I thank the authors for the detailed response and additional results. The sensitivity studies and throuput analysis can greatly strengthen the submission. However, I remain skeptical towards the practical significance of this paper. The paper does not directly evaluate a realistic image-conditioned pipeline where descriptions are generated from the images and then selected. So it remains unclear how often the highlighted semantically irrelevant high-AUROC prompts would occur in practical workflows. The downstream evidence also remains diagnostic rather than showing actual improved adaptation decisions. I would prefer to retain my current score.