- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the problem of selecting appropriate repair methods for frozen Vision-Language Models, aiming to improve robustness without relying on group labels during deployment. The authors propose Guarded-Repair, a candidate-selection framework that restricts the available repair methods to selected families and chooses candidates based on validation-average accuracy. The paper also introduces variance geometry and complementary risk signals, including validation accuracy preservation and prompt stability, to identify potentially harmful repairs.
Experiments across 156 scenarios involving Waterbirds, CelebA, PACS, and synthetic CIFAR shifts show that the proposed selection policy improves worst-group/domain accuracy while reducing harmful selections. The authors further demonstrate that incorporating TIE-style repair candidates into an audited candidate pool can provide additional robustness improvements.
Critical strengths
-
Practical relevance of the research problem. The paper addresses an important issue in VLM deployment: improving average accuracy does not necessarily lead to better worst-group robustness. Separating repair generation from deployment selection provides a useful perspective on evaluating and applying existing repair methods.
-
Evaluation across multiple settings. The authors conduct experiments across several datasets, distribution shifts, and VLM backbones. The evaluation considers both average accuracy and worst-group/domain performance, providing a more comprehensive assessment than relying solely on overall accuracy.
-
Empirical analysis of repair risks and candidate selection. The paper examines how different repair methods affect average accuracy and worst-group/domain robustness, and evaluates several signals for identifying potentially harmful repairs. It also compares different candidate pools under a consistent selection protocol, providing useful empirical insights into the benefits and limitations of candidate selection.
Critical weaknesses
- Limited methodological novelty and insufficient analysis of complementarity.
The paper demonstrates empirical benefits from combining different repair candidate pools. However, this improvement appears to arise primarily from expanding the candidate space rather than from a principled mechanism that exploits the complementary properties of different repair methods. Although the combined pool outperforms the individual pools, the paper provides limited analysis of why these methods are complementary, under which conditions each method is preferable, and whether the improvement can be attributed to their complementary strengths rather than simply having more candidates to choose from.
- Insufficient exploration of adaptive repair strategies.
The paper demonstrates that different repair methods perform differently across tasks and distribution shifts. However, the proposed framework primarily selects a single repair candidate based on validation performance rather than adaptively combining multiple candidates according to the characteristics of individual inputs. It remains unclear whether such candidate-level selection fully exploits the complementary strengths of different repair methods. For example, input-dependent weighting or adaptive fusion strategies could provide useful alternative approaches. A discussion or comparison of these alternatives would help clarify the advantages and limitations of the proposed candidate-selection framework.
- Limited evaluation on complex, real-world scenarios.
Although the paper evaluates multiple datasets, the experiments still rely heavily on relatively controlled classification settings, such as CIFAR-100, Waterbirds, CelebA, and synthetic distribution shifts. Given the paper's emphasis on deployment-time VLM repair, it is unclear whether the proposed framework can generalize to more complex and realistic scenarios. Evaluations on more diverse real-world distribution shift benchmarks, such as DomainNet or WILDS, would provide stronger evidence of its practical applicability.
Questions for authors
Please refer to the weaknesses above.