Intervene or Preserve? Selective Test-Time Intervention for Vision-Language Models

Target supervision at test time is usually treated as an instruction to adapt. We show that even correct target labels can be unsafe adaptation signals. Under a fixed sparse-label budget, semantic intervention yields large gains when queried labels identify a substantial fraction of the target semantic state, but its benefit rapidly diminishes and can become negative as the same supervision is diluted across increasingly complex target populations. This exposes a fundamental distinction between having target labels and having sufficient semantic authority to justify changing a frozen model's predictions. We introduce Selective Test-Time Intervention (STTI), a test-time inference framework that treats sparse target labels as optional semantic evidence rather than mandatory adaptation signals. STTI first constructs a strong preserved hypothesis from complementary frozen vision–language views, then actively queries a tiny number of target labels, uses them to identify shared target semantic correspondences, and constructs a competing intervention hypothesis. A regime-blind, risk-constrained controller chooses between these hypotheses, intervening only when observable evidence indicates that the expected benefit is sufficient to justify the associated risk. Across 11 benchmarks spanning diverse semantic complexities and annotation budgets, STTI retains the large gains available when sparse supervision is informative while progressively reverting to the frozen hypothesis as its semantic coverage diminishes. Mechanistic analyses show that queried labels resolve semantic populations rather than isolated examples, while causal diagnostics identify semantic coverage as a principal bottleneck governing intervention utility. These findings support a broader principle for test-time learning: preserve the frozen semantic prior by default, and intervene only when sparse supervision identifies enough of the target state to warrant changing it.