acceptodds
Under review as a conference paper at ICLR 2027

Auditing VLM Repair: Variance Geometry and Guarded Candidate Selection

Abstract

Vision-language models (VLMs) are often repaired at deployment by changing the geometry of frozen image and text embeddings. Whitening, prompt adaptation, and nuisance removal can improve average accuracy, yet the same operation can amplify a spurious factor and damage the least represented groups. We study this problem as an audit and selection task: a repair method proposes candidates, while a deployment layer decides which candidate is safe to apply. We introduce a group-free guarded selector built from task-conditioned variance diagnostics, validation-average preservation, and prompt stability. The diagnostics separate semantic alignment from nuisance alignment in the observed covariance spectrum and expose when covariance corrections should be used, ignored, or filtered. Across 156 scenarios spanning Waterbirds, CelebA, synthetic CIFAR shifts, PACS, and two VLM backbones, naive validation-average selection yields a worst-group/domain delta of -0.006, whereas the guarded policy yields +0.037 with a 95% bootstrap interval of [+0.003, +0.078] and reduces harmful selections from 30 to 9. The combined risk signal identifies harmful observed-covariance candidates with AUROC 0.956. Adding TIE-style candidates to the pool raises robust delta to +0.073 and gives a paired gain of +0.041 over our candidates alone. The results establish deployment-time auditing as a complementary layer for VLM repair, while identifying a clear boundary where strong nuisance shift remains difficult.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Vision-Language Models, Test-Time Adaptation, Robustness under Distribution Shift, Representation Learning, Spurious Correlations, Selective Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qN6SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of selecting appropriate repair methods for frozen Vision-Language Models, aiming to improve robustness without relying on group labels during deployment. The authors propose Guarded-Repair, a candidate-selection framework that restricts the available repair methods to selected families and chooses candidates based on validation-average accuracy. The paper also introduces variance geometry and complementary risk signals, including validation accuracy preservation and prompt stability, to identify potentially harmful repairs.

Experiments across 156 scenarios involving Waterbirds, CelebA, PACS, and synthetic CIFAR shifts show that the proposed selection policy improves worst-group/domain accuracy while reducing harmful selections. The authors further demonstrate that incorporating TIE-style repair candidates into an audited candidate pool can provide additional robustness improvements.

Critical strengths
  1. Practical relevance of the research problem. The paper addresses an important issue in VLM deployment: improving average accuracy does not necessarily lead to better worst-group robustness. Separating repair generation from deployment selection provides a useful perspective on evaluating and applying existing repair methods.

  2. Evaluation across multiple settings. The authors conduct experiments across several datasets, distribution shifts, and VLM backbones. The evaluation considers both average accuracy and worst-group/domain performance, providing a more comprehensive assessment than relying solely on overall accuracy.

  3. Empirical analysis of repair risks and candidate selection. The paper examines how different repair methods affect average accuracy and worst-group/domain robustness, and evaluates several signals for identifying potentially harmful repairs. It also compares different candidate pools under a consistent selection protocol, providing useful empirical insights into the benefits and limitations of candidate selection.

Critical weaknesses
  1. Limited methodological novelty and insufficient analysis of complementarity.

The paper demonstrates empirical benefits from combining different repair candidate pools. However, this improvement appears to arise primarily from expanding the candidate space rather than from a principled mechanism that exploits the complementary properties of different repair methods. Although the combined pool outperforms the individual pools, the paper provides limited analysis of why these methods are complementary, under which conditions each method is preferable, and whether the improvement can be attributed to their complementary strengths rather than simply having more candidates to choose from.

  1. Insufficient exploration of adaptive repair strategies.

The paper demonstrates that different repair methods perform differently across tasks and distribution shifts. However, the proposed framework primarily selects a single repair candidate based on validation performance rather than adaptively combining multiple candidates according to the characteristics of individual inputs. It remains unclear whether such candidate-level selection fully exploits the complementary strengths of different repair methods. For example, input-dependent weighting or adaptive fusion strategies could provide useful alternative approaches. A discussion or comparison of these alternatives would help clarify the advantages and limitations of the proposed candidate-selection framework.

  1. Limited evaluation on complex, real-world scenarios.

Although the paper evaluates multiple datasets, the experiments still rely heavily on relatively controlled classification settings, such as CIFAR-100, Waterbirds, CelebA, and synthetic distribution shifts. Given the paper's emphasis on deployment-time VLM repair, it is unclear whether the proposed framework can generalize to more complex and realistic scenarios. Evaluations on more diverse real-world distribution shift benchmarks, such as DomainNet or WILDS, would provide stronger evidence of its practical applicability.

Questions for authors

Please refer to the weaknesses above.

Reviewer_HcoPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors consider repair mechanisms for VLM systems, and outline protocol limitations in past works.

Critical strengths

Appreciate the clear hypothesis statement and the broad set of experimental datasets considered, although I am left wondering about statistical significance given the 156 tested scnearios.

The quality of the writing initially looks quite promising. However, as will be discussed further below, the prose is strongest when it's describing things, but it often leaves the "why" a mystery. It's mechanically quite readable, but it does not necessarily fully convey the points required to support the nature of the contribution.

Critical weaknesses

As described above, the prose, while competent, is fundamentally lacking in its descriptions of why choices were made. However the why, to me, is the important thing for understanding the nature of the contribution.

The discussion is a key example of this. The first two paragraphs just list things - it's fundamentally not a discussion at all, just a blast of details. Take "The cross-backbone results and TIE stress tests show how selection and diagnosis work alongside existing repair mechanisms." To break this sentence down - it describes that the cross-backbone results and stress tests show something, but it doesn't actually describe what it shows, nor does it equip the reader with the context of what they should be looking at in these results. And even if the reader was equipped to read these results, the sentence doesn't link itself to the actual experimental data (as table/figure refs). Finally, there's some ambiguity regarding the structure - is this "how selection and diagnosis work alongside existing repair mechanisms [to do XYZ?]" or is it "how selection and diagnosis work - alongside existing repair mechanisms - [to do XYZ?]"? The nature of the presentation, and the lack of focus on explaining what is actually happening, and why it's actually important both invites this questioning regarding the structure of the sentence, and more broadly leaves the reader unequipped to understand the implications of the work. And if this is to be a discussion, what are the implications of the observations regarding these results?

I stress that this is a breakdown of one particular sentence in the Discussion section, but I would argue that it is indicative of an anti-pattern that runs through the meat of the document, that is most prevalent within the Discussions.

Another way of examining this limitation in the presentation is to ask who the audience is for this work, and how they would take something from it? Knowing that repair mechanisms are useful is definitely useful. But why do they work? How well do they generalize? How much of performance is test specific, and how much of it is general? How confident can we be about the confidence intervals o the experimental set? These are all questions that could be discussed, and would have clear implications for a readers ability to take more from this work than "Base + Tie, Fixed gate produces the best results for the tested experiments".

Now, I will note that as the discussion goes on some questions about implications are discussed - in particular lines 446 - 452, and again in the paragraph starting from 453. But I would argue that this leaves some of the most interesting implications for future work. "A single test accuracy for the selected candidate is insufficient because it does not reveal which candidates were considered, which validation signal selected the winner, or whether a candidate was rejected for instability. Our protocol records these decisions at the scenario level and then aggregates them with the same units used for uncertainty estimation." - this is interesting, but I'm left wondering if this could be experimentally demonstrated, by constructing comparisons between performance using prior protocols and this approach, and then delving into the details.

Smaller notes

  • "To control small eigenvalues, we use shrinkage" - statements like this provide the what, but not the why. Similarly when introducing the filtered score, it's introduced without a justificaiton as to why a form like this will be impactful.
  • Bibliography is missing details. For example, Farina (2024) should not be cited for arXiv, this was a paper that was accepted at NeurIPS and should be cited as such.
  • AI Use Statement is highly perfunctory, and could use additional detail.
Questions for authors

Is it possible to augment the 95% CI's with a hypothesis test to confirm that the leading robust delta is actually a statistically significant observation?

Reviewer_yH2BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates frozen-VLM repair into candidate generation and deployment selection. Its main policy restricts eligible candidate families before maximizing validation-average accuracy. A separate risk diagnostic combines variance geometry, accuracy preservation, and prompt stability.

Critical strengths
  1. Table 2 reports average gains, worst-group gains, and harmful selections, addressing practical deployment needs.
  2. Tables 3–4 openly report negative findings, including tint failures and weak geometry-only detection.
  3. Table 5 holds the gate fixed while expanding the candidate pool, supporting complementarity between repair mechanisms.
Critical weaknesses
  1. Algorithm 1 relies on a predefined family whitelist, but Sections 4.3 and 6.6 do not independently validate its design across evaluation tasks.
  2. The authors should clarify when the rule was fixed and whether existing results distinguish family exclusion from general auditing capability.
  3. Table 4 omits validation-plus-prompt scoring without VAI, leaving geometry’s incremental value unresolved and requiring that comparison.
Questions for authors
  1. Before observing results from which tasks was the family whitelist fixed?
  2. Can existing records report selection performance within each candidate family separately?
  3. Do existing risk scores include validation-plus-prompt scoring with VAI removed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.