acceptodds
Under review as a conference paper at ICLR 2027

Intervene or Preserve? Selective Test-Time Intervention for Vision-Language Models

Abstract

Target supervision at test time is usually treated as an instruction to adapt. We show that even correct target labels can be unsafe adaptation signals. Under a fixed sparse-label budget, semantic intervention yields large gains when queried labels identify a substantial fraction of the target semantic state, but its benefit rapidly diminishes and can become negative as the same supervision is diluted across increasingly complex target populations. This exposes a fundamental distinction between having target labels and having sufficient semantic authority to justify changing a frozen model's predictions. We introduce Selective Test-Time Intervention (STTI), a test-time inference framework that treats sparse target labels as optional semantic evidence rather than mandatory adaptation signals. STTI first constructs a strong preserved hypothesis from complementary frozen vision–language views, then actively queries a tiny number of target labels, uses them to identify shared target semantic correspondences, and constructs a competing intervention hypothesis. A regime-blind, risk-constrained controller chooses between these hypotheses, intervening only when observable evidence indicates that the expected benefit is sufficient to justify the associated risk. Across 11 benchmarks spanning diverse semantic complexities and annotation budgets, STTI retains the large gains available when sparse supervision is informative while progressively reverting to the frozen hypothesis as its semantic coverage diminishes. Mechanistic analyses show that queried labels resolve semantic populations rather than isolated examples, while causal diagnostics identify semantic coverage as a principal bottleneck governing intervention utility. These findings support a broader principle for test-time learning: preserve the frozen semantic prior by default, and intervene only when sparse supervision identifies enough of the target state to warrant changing it.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-Time Adaptation, Test-Time Intervention, Sparse Supervision, Semantic Coverage, Vision-Language Models, Risk-Constrained Inference, Selective Prediction, Domain Adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PF8TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses that correct sparse target labels can still be unsafe for VLM test-time adaptation due to insufficient semantic coverage. It proposes Selective Test-Time Intervention (STTI), a preserve-or-intervene framework treating sparse labels as optional semantic evidence. STTI uses complementary frozen VLM views, active querying, a permutation-equivariant transducer, and a risk-constrained controller to intervene only with sufficient semantic authority. On 11 VLM benchmarks, STTI retains gains under high coverage and reverts to preservation as coverage drops.

Critical strengths
  1. The STTI design is interesting and novel, especially compared with existing methods that default to adaptation whenever labeled information is available.

  2. It achieves strong results on 11 VLM benchmarks, and the experimental results are reasonably convincing.

Critical weaknesses
  1. Experimental performance actually degrades in simple semantic scenarios, which limits the applicability of the method.

  2. The method adopts a dual-VLM backbone consisting of OpenAI CLIP and SigLIP2, whereas most baselines use only CLIP. This makes it difficult to establish whether the observed gains stem from the STTI design rather than from the auxiliary visual information provided by SigLIP2.

Questions for authors
  1. As shown in Table 26, why does performance drop significantly when SigLIP2 is used as the primary backbone? Is the method's contribution more attributable to the STTI design or to the choice of primary backbone?

  2. What is the computational overhead of the dual-VLM architecture adopted by the method compared with the baselines?

3 comments
AuthorsOpenReview ↗

Sparse-Supervision Trade-offs, Cross-Backbone Generalization, and Computational Efficiency

We thank the reviewer for constructive feedback on the preserve-or-intervene formulation and semantic-complexity evaluation. We agree that the key empirical questions are: (i) why simpler or unconditional methods can beat selective intervention in low-complexity episodes, (ii) how much gain comes from the second VLM vs. STTI (including the SigLIP2-primary result), and (iii) the practical cost of the entire method. We present matched experiments and runtime measurements below.

These additional experimental results are provided here and have not yet been included in the submitted manuscript. We will incorporate relevant findings and clarifications into a revised submission during the discussion period, as appropriate. We welcome further questions or suggestions.

Q1. Why can simpler methods perform better in some sparse semantic regimes?

Semantic complexity measures diversity, not annotation scarcity. In each M=4 experiment, every query episode reveals four target labels. With a semantically concentrated B=64 batch, those labels may cover much of the target population, so aggressive adaptation helps; in high-complexity or ALL-class settings, the same budget covers only a small slice of target semantics. The risk-constrained controller exploits supported interventions without assuming that any correct queried label justifies updating the entire batch.

Table R1a. Matched preservation, intervention, and source-trained gated-cache controls at M=4. Top-1 accuracy (%) averaged equally over ten non-ImageNet datasets per regime. All methods share the same frozen preservation state (P0); adapted methods use the same queried supports and four-label budgets. The gated cache uses a separate source-trained six-feature preserve/update controller.

Semantic regimePreserve P0 (M=0)Always interveneGated matched cacheSTTI (frozen policy)
B64 Very-Low78.9889.6180.9585.28
B64 Low77.8981.8278.4980.43
B64 Medium76.1670.4376.0376.10
B1000 Medium78.7869.7278.5078.78
B1000 High79.6968.0579.3779.66
B1000 Very-High78.6667.2678.5378.66
ALL80.3667.5479.6880.36
Seven-regime mean78.6573.4978.7979.90

Interpretation. The reviewer is correct that selectivity is not always maximized accuracy within a single regime. In Very-Low, always intervening reaches 89.61%, above STTI’s 85.28%, although STTI still improves P0 by +6.30 pp. Thus, the selector forgoes some concentrated-regime accuracy instead of always exploiting all four labels. Consistently, unconditional intervention is 5.73 pp below P0 in B64 Medium and 12.82 pp below P0 in ALL, while STTI is −0.06 pp and equal to P0, respectively. At B1000 High, STTI is −0.03 pp below preservation, so protection is not absolute. A more aggressive source-mean-matched cache achieves a higher overall mean (80.66% vs 79.90%) via larger sparse-regime gains, but trails STTI by 1.45 pp in Very-High and 2.24 pp in ALL. The aim is a measured utility–preservation trade-off, not regime-by-regime dominance.

To test whether gains come solely from adding a trained preserve/update gate to a conventional cache, we evaluate a separate source-trained gated-cache control with identical P0, queried images/labels, budgets, and evaluation folds.

Table R1b. Paired comparison of learned gated cache across annotation budgets. Mean accuracy (%) over seven regimes with equal datasets; last column reports STTI − gated cache (percentage points) with paired dataset-bootstrap 95% CIs, conditional on fixed source policies and this benchmark panel.

Label budgetPreserve P0Learned gated cacheSTTISTTI − gated cache (pp), 95% CI
M=178.6578.4878.63+0.15 [+0.02, +0.64]
M=278.6578.6479.19+0.55 [+0.21, +0.98]
M=478.6578.7979.90+1.10 [+0.59, +1.66]

Interpretation. Table R1b tests whether STTI’s gains can be reproduced by adding a source-trained six-feature preserve/update controller to a conventional cache. With identical preservation predictions (P⁰ = 78.65%), queried samples, budgets, and folds, the gated cache reaches 78.48%, 78.64%, and 78.79% at M=1, 2, and 4. Its best gain over P⁰ is +0.14 pp (M=4), versus +1.25 pp for STTI. STTI outperforms the gated cache by +0.15, +0.55, and +1.10 pp across budgets; all paired 95% CIs exclude zero, and all ten dataset means favor STTI at M=4. The widening gap indicates STTI uses added sparse supervision more effectively. Overall, the results attribute improvements to the full semantic intervention-and-selection pipeline beyond a conventional cache with a learned gate, rather than attributing the improvement merely to selective authorization.

AuthorsOpenReview ↗

Cross-Backbone Generalization

Q2. Is the improvement mainly due to SigLIP2, and what explains the earlier SigLIP2-primary result?

To separate gains from stronger frozen representations vs selective semantic intervention, we compare primary-only, auxiliary-only, frozen two-view preservation (P0), and STTI. P0 and STTI use the same frozen encoders and preservation predictions, so STTI − P0 isolates the added benefit of the full intervention-and-selection pipeline, independent of adding the auxiliary encoder. The source-trained intervention operator, controller, and thresholds are fixed in all cross-backbone transfer experiments.

Table R2. Component attribution and frozen cross-backbone transfer (M=4). Top-1 accuracy (%) averaged equally across 10 datasets and 7 semantic-complexity regimes. Primary StatA, auxiliary-only, and P0 are label-free (M=0); STTI queries 4 labels. Confidence intervals are paired, dataset-bootstrap 95% intervals with fixed learned parameters.

Primary / auxiliaryPrimary StatAAuxiliary onlyP0 (two views)STTISTTI − P0 (pp), 95% CI
CLIP / SigLIP269.1080.7078.6579.90+1.250 [0.644, 1.931]
PE-Core / SigLIP285.0680.7085.4385.97+0.537 [0.266, 0.865]
EVA02 / SigLIP279.5380.7081.2582.47+1.220 [0.400, 2.558]
SigLIP2 / OpenCLIP81.4778.9482.5982.87+0.283 [0.124, 0.507]
CLIP / OpenCLIP69.1078.9476.9478.24+1.300 [0.734, 2.013]
PE-Core / OpenCLIP85.0678.9485.2585.80+0.546 [0.280, 0.878]
EVA02 / OpenCLIP79.5378.9480.6581.62+0.978 [0.387, 1.872]

The seven-regime average mixes settings where sparse supervision is highly informative with ones where intervention is less reliable. We therefore also report gains separately for sparse and dense semantic regimes.

Table R2a. Sparse vs. dense improvement over frozen preservation P0 at M=4 (pp). Sparse mean averages B64 Very-Low/Low/Medium; dense mean averages B1000 High/Very-High/ALL. B1000 Medium is still included in the overall seven-regime mean.

Primary / auxiliaryB64 Very-LowB64 LowSparse meanDense meanOverall mean
CLIP / SigLIP2+6.30+2.54+2.93−0.01+1.250
PE-Core / SigLIP2+3.07+0.66+1.250.00+0.537
EVA02 / SigLIP2+6.57+1.83+2.840.00+1.220
SigLIP2 / OpenCLIP+1.48+0.53+0.660.00+0.283

Table R2a reports descriptive subgroup means from the same evaluation panel. The paired confidence intervals in Table R2 reflect overall seven-regime effects, not separately tested subgroup effects.

Interpretation: Incremental contribution and semantic-regime dependence. Table R2 shows STTI adds predictive value beyond the frozen dual-view baseline. For CLIP/SigLIP2, STTI raises P0 from 78.65% to 79.90% (+1.250 pp [0.644, 1.931]), improving all ten datasets at M=4. Gains also hold for stronger backbones: PE-Core (+0.537 pp) and EVA02 (+1.220 pp) and remain when OpenCLIP replaces SigLIP2 as the auxiliary (+1.300, +0.546, +0.978 pp for CLIP, PE-Core, EVA02).

Table R2a shows the source of these gains. Across the three B64 regimes, STTI improves over the matched P0 by +2.93 pp (CLIP), +1.25 pp (PE-Core), +2.84 pp (EVA02), and +0.66 pp (SigLIP2-primary) on average. Gains are largest in semantically concentrated settings, peaking in Very-Low at +6.30 pp (CLIP) and +6.57 pp (EVA02). In the High, Very-High, and ALL regimes, STTI largely matches frozen preservation predictions, with mean differences of −0.01 to 0.00 pp.

These consistent findings show STTI’s gain is not just adding a stronger auxiliary VLM. Its fixed-parameter intervention-and-selection pipeline uses sparse target supervision when informative, but largely preserves the frozen prediction when semantic coverage is insufficient. Thus, the seven-regime mean reflects helpful intervention in concentrated regimes and conservative preservation in dense regimes, not uniformly small gains.

SigLIP2-primary generalization. We also evaluated STTI with SigLIP2 as the primary backbone and OpenCLIP as the auxiliary, reusing the learned intervention operator and controller without backbone-specific refitting. At M=4, STTI improves the frozen preservation baseline from 82.59% to 82.87% (+0.283 pp [0.124, 0.507]), with nine dataset wins and one tie. As Table R2a shows, gains are concentrated in sparse regimes (+1.48 pp in Very-Low; +0.53 pp in Low), while preservation remains stable in B1000 and ALL. Along with PE-Core and EVA02, these results support fixed-parameter cross-backbone applicability across the evaluated settings. Full regime-level results will appear in Appendix F.

AuthorsOpenReview ↗

Computational Efficiency

Q3. What is the computational cost of the dual-VLM system?

We agree that assessing STTI requires end-to-end computational efficiency, not just cached-feature processing. We therefore expand the comparison of original vs. optimized implementations on the same hardware using SUN397 (397 classes) with batch sizes B=64 and B=1,000.

Table R3. Original and optimized GPU runtime (seconds per transductive batch) and peak allocated memory.

MethodOriginal B64Optimized B64Original B1000Optimized B1000Peak GPU (GiB)
Dirichlet2.102—2.655—2.175
TransCLIP0.131—1.768—2.381
StatA0.054—1.742—2.383
MOON0.305—0.622—2.381
Two-encoder preservation P01.0500.7159.1151.9085.139*
Full STTI1.1740.7599.8222.1335.139

Memory figures assume B=1,000 and report peak allocated GPU memory. Original values come from the server benchmark; optimized P0/STTI timings come from later runs on the same hardware. P0 memory is unchanged from the original benchmark. A dash means no separate optimized baseline was reported.

Optimization and measured efficiency. Our optimized inference uses BF16 SigLIP2 weights, PyTorch scaled dot-product attention (attn_implementation="sdpa"), and torch.inference_mode() with CUDA autocast. CUDA profiling confirmed the fused FlashAttention kernel (aten::_scaled_dot_product_flash_attention) during SigLIP2 inference. We report these details without attributing the speedup to any single component.

The update cuts total STTI runtime from 1.174 to 0.759 s at B=64 (1.55×; −35.3%) and from 9.822 to 2.133 s at B=1,000 (4.60×; −78.3%).

Incremental cost beyond the dual-VLM baseline. P0 already includes both frozen encoders. Compared to this optimized preservation pathway, the full intervention-and-selection process adds 0.044 s (6.2%) at B=64, rising to 0.225 s (11.8%) at B=1,000. At B=1,000, full STTI costs 2.133 ms per transductive input when amortized over the batch, not 2.133 ms per independent single-image prediction.

Interpretation. These results show substantial implementation improvements and separate the cost of two independent semantic views from the added cost of sparse-label intervention. However, larger-batch overhead remains meaningful, and STTI uses more GPU memory than lightweight single-VLM baselines. The method therefore yields an accuracy–computation trade-off, not negligible inference cost. We will add the optimized results, implementation details, and remaining efficiency limits to the revised paper.

Revision and discussion. We will revise the manuscript to include matched baseline comparisons, corrected cross-backbone evaluation, and optimized computational analysis, with a clearer discussion of accuracy–preservation–efficiency trade-offs. We thank the reviewer for the constructive suggestions, which have strengthened the evaluation.

Reviewer_eVaFOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the following problem: when only a few target labels are available at test time, should those labels be used to modify predictions for the entire batch?The method first constructs a preserved prediction from two frozen vision-language models. It then uses a small number of labels to construct an intervention prediction. Finally, a controller decides whether to keep the original prediction or use the intervention prediction.The downstream task is image classification. Each target batch provides only 1, 2, or 4 queried labels. The paper evaluates the method on multiple classification datasets, batch sizes, and levels of semantic complexity.

Critical strengths
  • The paper clearly distinguishes between a label being correct and a label having enough coverage to support broad intervention. The different behavior under high and low coverage is supported by experiments.
  • Table 27 compares methods using the same queried samples and preservation state. The results suggest that deciding whether to update may indeed be useful.
Critical weaknesses
  • The need for a VLM is not sufficiently justified. The downstream task is image classification, but the paper does not explain why a practical system would have only 1-4 labels per batch, or why an existing pure-vision classifier or lightweight calibration method would not be sufficient. VLMs are useful for zero-shot and cross-category transfer, but that alone does not establish the necessity of the full STTI system.
  • The gain over simpler alternatives is not isolated. The matched cache is forced to use the queried labels, while STTI may choose not to use them. Thus, the current comparison supports the value of selective updating, but does not establish that the specialized semantic transducer is necessary. The two-encoder design also adds substantial cost, and the accuracy-cost trade-off is not sufficiently positioned against simpler methods.
Questions for authors
  1. Please describe one realistic downstream scenario in which a target batch has only 1-4 labels, a frozen VLM is available, and an existing classifier cannot directly use the available information. What is the relationship between the target classes and the source-side training classes?
  2. Under comparable target-label and class-prior information, could you compare a pretrained pure-vision CNN/ViT with lightweight calibration against the full STTI system? Please state clearly what pretraining, source supervision, and class-alignment information each method receives. If the pure-vision method is not applicable, please explain what information is missing and what the VLM provides.
  3. Could you add the same preserve/update switch to the matched cache, using the same preservation state, queried samples, and source data as STTI? This would separate the value of selective decision-making from the value of the more complex intervention module.
3 comments
AuthorsOpenReview ↗

Vision-Only Comparisons, Selective Intervention, and Computational Efficiency

We thank the reviewer for noting the difference between label correctness and semantic coverage and for raising two key questions: whether a frozen VLM is appropriate for the sparse-label setting, and whether STTI offers benefits beyond a simpler selectively gated cache. We address all three questions with clarified assumptions and two additional controlled evaluations.

These additional experimental results are provided here and have not yet been included in the submitted manuscript. We will incorporate relevant findings and clarifications into a revised submission during the discussion period, as appropriate. We welcome further questions or suggestions.

Q1. Practical setting, role of VLMs, and source–target classes

For land-use recognition following deployment in a new geographic region: the target region and category names are known, but local annotations are expensive. A frozen VLM can score these categories using text descriptions without training a new target-specific classification head. If an expert labels only a few images in a batch, the key question is whether those labels cover enough of the batch’s semantics to justify updating other predictions. Our M = 1, 2, 4 labels per episode are controlled tests of this limited-coverage scenario, not an assumption about real deployment budgets.

The target class names and the unlabeled batch are available; only M queried target labels are revealed. The intervention transducer, controller, normalization, and risk threshold are trained on source episodes and frozen at evaluation. For each of the ten non-ImageNet target datasets, we use strict leave-one-dataset-out fitting: the entire dataset is excluded from model selection and risk calibration. ImageNet is an additional unseen target and is never used as a source dataset (Appendix A). This ensures dataset-level separation, not zero semantic overlap between source and target class names. A supervised vision-only classifier is a valid alternative if it already has an appropriate target-class head; our point is that a VLM can use a provided category vocabulary without learning such a head.

AuthorsOpenReview ↗

Selective Intervention

Q2. Do pretrained image-only classifiers with lightweight corrections suffice?

We added supervised ResNet-50 (torchvision ImageNet-1K V2) and ViT-B/16 (ImageNet-1K V1) using their existing 11,000-class supervised heads. Thus, neither classifier head is penalized for missing target classes. We compare frozen predictions, class-bias calibration, cosine-cache correction, and their combination. All adaptive methods use the same queried images and labels, with the same 1,000-class vocabulary, and evaluation folds across 152 shared ImageNet-Sketch/ImageNet-V2 tasks (1,824 budget–fold states). The Lightweight-correction settings are fixed.

Table R1. Seven-regime mean top-1 accuracy (%) for image-only and VLM systems. Each column equally weights the seven semantic-complexity regimes; M is the number of queried labels per episode.

MethodSketch M=1Sketch M=2Sketch M=4V2 M=1V2 M=2V2 M=4
ResNet-50 (frozen)30.3530.3530.3571.3371.3371.33
ResNet-50 + bias31.1531.2831.3472.1172.1572.08
ResNet-50 + cache30.8631.5032.7071.8272.3073.26
ResNet-50 + bias/cache31.7632.4933.7372.4473.1374.01
ViT-B/16 (frozen)29.9429.9429.9470.3770.3770.37
ViT-B/16 + bias30.6030.5430.5670.8670.9370.91
ViT-B/16 + cache30.3130.7231.5170.9071.2971.86
ViT-B/16 + bias/cache30.9631.4432.2471.2471.5972.30
Frozen VLM preservation (P⁰)70.7270.7270.7275.9775.9775.97
STTI70.7471.4972.1976.4676.8277.28

At M=4, the best corrected image-only setup (ResNet-50 with bias/cache correction) reaches 33.73%/74.01% on ImageNet-Sketch/V2 versus 72.19%/77.28% for STTI, an absolute +38.46/+3.27 pp gap. This gap is not simply due to missing classifier heads, since both supervised heads cover all 1,000 classes. Lightweight image-only adaptation still helps: ResNet-50 improves +3.38/+2.68 pp over its frozen predictions, whereas STTI adds +1.46/+1.31 pp over a stronger frozen baseline. Overall, the main gain comes from pretrained semantic representations, with STTI offering additional targeted improvements. Moreover, these models have different pretraining supervision, objectives, and computational costs, so Table R1 highlights the benefit of the pretrained image–text prior under these shifts, not a pretraining-matched claim that all pure-vision systems are inadequate.

We agree that computational efficiency matters. On the same GPU hardware, we further optimized STTI, reducing runtime from 1.174 to 0.759 s at B=64 (1.55×) and from 9.822 to 2.133 s at B=1,000 (4.60×). The optimized two-encoder preservation baseline takes 0.715 and 1.208 s, implying STTI overhead of 0.044 s at B=64 and 0.925 s at B=1,000. These results show substantially improved efficiency, while noting the remaining cost of the auxiliary VLM and intervention processing. We will add the updated measurements and implementation details to the revised manuscript.

AuthorsOpenReview ↗

Computational Efficiency

Q3. Does a source-trained preserve/update gate on the matched cache remove STTI's advantage?

This is the key attribution concern, and we agree that Table 27's unconditional matched cache alone cannot answer it. We therefore constructed a gated matched-cache baseline that can also preserve P⁰. It has its own six-feature source-trained preserve/update controller, fitted using cache predictions, with cache strength selected within source folds. It shares with STTI the identical frozen P⁰, target episodes, queried images and labels, M, and evaluation folds. No target evaluation labels are used to fit either decision rule. STTI retains its frozen learned intervention operator and controller under the corrected target-input evaluation.

Table R2. Gated cache versus STTI (ten non-ImageNet datasets; seven-regime, equal-dataset means). The final column gives paired dataset-bootstrap 95% CIs, conditional on the fitted source policies.

BudgetPreserve P⁰ (%)Gated cache (%)STTI (%)STTI − gated cache (pp), 95% CI
M = 178.6578.4878.63+0.15 [+0.02, +0.64]
M = 278.6578.6479.19+0.55 [+0.21, +0.98]
M = 478.6578.7979.90+1.10 [+0.59, +1.66]

Table R2 shows that conventional cache correction offers little benefit compared to a semantic intervention. With the same preservation hypothesis (P⁰ = 78.65%) and queried labels, gated caching achieves only 78.48%, 78.64%, and 78.79% at M=1, 2, and 4—nearly matching P⁰ at M=4 (+0.14 pp) and slightly worse at smaller budgets. Instead, STTI rises from 78.63% to 79.90%, reaching +1.25 pp over P⁰ at M=4. STTI beats gated caching by +0.15, +0.55, and +1.10 pp, with paired 95% CIs excluding zero at all budgets; at M=4, all ten dataset means favor STTI. The widening gap indicates that STTI uses added target supervision more effectively than simply abstention. Although the M=1 gain is modest, the comparison supports the value of the complete semantic intervention-and-selection pipeline, rather than isolating the contribution of each component individually.

Table R3. Where the M = 4 difference arises (top-1 accuracy, %). Regime-level differences are descriptive; no per-regime significance is claimed.

Semantic complexityPreserve P⁰Gated cacheSTTISTTI − gated cache (pp)
B=64 Very-Low78.9880.9585.28+4.33
B=64 Low77.8978.4980.43+1.94
B=64 Medium76.1676.0376.10+0.07
B=1,000 Medium78.7878.5078.78+0.28
B=1,000 High79.6979.3779.66+0.29
B=1,000 Very-High78.6678.5378.66+0.13
ALL80.3679.6880.36+0.68

Relative to the same P⁰, STTI improves by +2.93 pp across the three B=64 regimes (vs +0.81 pp for gated caching). Over High/Very-High/ALL, the mean change is −0.01 pp (vs −0.38 pp). The mean of the worst 10% of dataset–regime changes is −0.37 pp for STTI (vs −2.23 pp; descriptive, not a safety guarantee). Overall, STTI yields more gain in concentrated semantic populations while better preserving performance when coverage is limited.

Conclusion. These new controls establish a more precise claim than the original unmatched comparisons: the full semantic intervention-and-selection pipeline provides incremental benefit beyond a source-trained preserve/update gate on a conventional cache, given identical frozen preservation and target supervision. They do not prove that each transducer component is necessary or that a VLM always outperforms a similarly pretrained image-only model. We will add the controlled results and related scope/compute caveats to the revised paper.

Reviewer_GLcNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Selective Test-Time Intervention (STTI), a framework for vision-language models that determines whether sparse target-label supervision should modify or preserve frozen predictions. STTI combines complementary frozen VLM predictions, actively queries a small number of target labels, constructs a semantic intervention hypothesis, and employs a risk-constrained controller to decide whether intervention is beneficial. Experiments across 11 classification benchmarks and different semantic complexity regimes demonstrate improved robustness over sparse-supervision baselines, particularly when queried labels provide limited semantic coverage. Additional analyses investigate the relationship between semantic coverage, intervention utility, and prediction preservation.

Critical strengths
  1. Interesting problem formulation: The paper identifies an important limitation of sparse-supervision-based test-time adaptation: even correct target labels can degrade predictions when their semantic coverage is insufficient. The preserve-or-intervene formulation provides a useful perspective beyond unconditional adaptation.

  2. Comprehensive experimental and mechanistic analysis: Evaluation across 11 benchmarks, multiple annotation budgets, and varying semantic complexity demonstrates the effectiveness of selective intervention. The matched-preservation comparisons and semantic coverage diagnostics further strengthen the empirical findings.

Critical weaknesses
  1. Limited generalization across VLM backbones: Although the semantic coverage phenomenon is demonstrated across multiple VLMs, the complete STTI intervention mechanism does not transfer consistently. In particular, the SigLIP2-primary configuration fails to retain the benefits of intervention, raising concerns about backbone dependence and broader applicability.

  2. Limited gains attributable specifically to selective intervention: A substantial portion of the improvement over standard baselines comes from the stronger frozen preservation hypothesis. Under matched preservation, the incremental benefits of STTI are smaller, with statistically inconclusive differences in several regimes. Moreover, the conservative controller rarely intervenes in semantically complex settings, making its advantages over simply retaining the frozen predictor less convincing.

Questions for authors
  1. How can STTI improve its cross-backbone generalization, particularly for SigLIP2, where the learned intervention mechanism fails to transfer effectively?

  2. Given that STTI rarely intervenes in high-complexity regimes, can the authors further quantify its incremental benefit over always preserving the frozen hypothesis, including statistical significance across datasets?

  3. How sensitive is the intervention controller to the source-domain composition and the selected risk threshold? Does its effectiveness remain consistent under larger domain shifts?

5 comments
AuthorsOpenReview ↗

Cross-Backbone Generalization

Thank you for noting our preserve-or-intervene formulation and the matched-preservation and semantic-coverage analyses. We agree that backbone transfer, gain attribution, and source-calibrated risk require more direct evidence, so we ran matched cross-backbone evaluations, a stronger gated-cache control, and targeted source/risk/shift sensitivity studies. We address the three questions separately.

These additional experimental results are provided here and have not yet been included in the submitted manuscript. We will incorporate relevant findings and clarifications into a revised submission during the discussion period, as appropriate. We welcome further questions or suggestions.

Q1. How well does the complete STTI mechanism transfer beyond CLIP, particularly to SigLIP2?

We revisited the cross-backbone evaluation, aligning text preprocessing with each pretrained backbone's requirements. We repeated the matched experiments while keeping the learned intervention operator, controller, thresholds, and annotation budgets unchanged.

Table R1. Fixed-parameter transfer: STTI gain over its identical preserved predictor (P⁰), in percentage points [paired 95% CI]. Ten non-ImageNet datasets; seven equally weighted complexity regimes per dataset; same evaluation episodes and M queried labels per fold.

Primary / auxiliary VLMM=1: Δ [95% CI]M=2: Δ [95% CI]M=4: Δ [95% CI]M=4 datasets (W/T/L)
CLIP ViT-B/16 / SigLIP2−0.015 [−0.200, 0.142]+0.541 [0.220, 0.932]+1.250 [0.644, 1.931]10/0/0
PE-Core-L14-336 / SigLIP2+0.018 [0.002, 0.037]+0.224 [0.070, 0.428]+0.537 [0.266, 0.865]10/0/0
EVA02-CLIP-L/14-336 / SigLIP2+0.071 [0.006, 0.183]+0.567 [0.126, 1.342]+1.220 [0.400, 2.558]10/0/0
SigLIP2 / DataComp OpenCLIP ViT-L/14+0.025 [0.000, 0.064]+0.150 [0.025, 0.316]+0.283 [0.124, 0.507]9/1/0

Inference: Paired dataset bootstrap, conditional on the original fixed source fits and this ten-dataset benchmark panel. PE-Core/EVA02 were the two prespecified added primary-backbone contrasts (Holm-adjusted exact-sign p=0.0039 at M=4); the remaining contrasts are supporting analyses, not independent replications.

Interpretation. Table R1 shows stronger cross-backbone generalization with fixed learned parameters. At M=4, STTI outperforms the same frozen-preservation baseline by +1.250 pp [0.644, 1.931] (CLIP), +0.537 pp [0.266, 0.865] (PE-Core), +1.220 pp [0.400, 2.558] (EVA02), and +0.283 pp [0.124, 0.507] (SigLIP2) (paired 95% CIs). PE-Core and EVA02 improve on all ten datasets without backbone-specific refitting, demonstrating that the gains are not driven by a small number of favorable datasets. Results also hold when swapping SigLIP2 for OpenCLIP (+0.546 pp for PE-Core; +0.978 pp for EVA02), providing further evidence that STTI's effectiveness is not tied to a specific auxiliary backbone. These results support the cross-backbone applicability of the tested fixed-parameter intervention-and-selection pipeline, while acknowledging that the magnitude of improvement varies across backbones.

AuthorsOpenReview ↗

Selective Intervention

Q2. What is gained beyond preserving P⁰, especially when intervention is rare?

We agree that end-to-end comparisons against CLIP-start baselines overstate what can be attributed specifically to selection. Accordingly, we evaluate the original CLIP/SigLIP2 configuration against its identical P⁰, unconditional semantic intervention Pᴵ, and a stronger source-trained gated cache. The latter has its own six-feature preserve/update controller and source-selected cache strength; it receives the same P⁰, U+D queries, labels, evaluation folds, and target budget as STTI. This extends the original unconditional matched-cache analysis (Appendix F.3, Table 27).

Table R2. Where intervention helps and where preservation matters (M=4; mean top-1 accuracy, %). Same ten-dataset, seven-regime corrected-input panel; the last row weights all seven regimes equally.

Semantic complexityPreserve P⁰Always intervene PᴵSource-trained gated cacheSTTI
B=64 Very-Low78.9889.6180.9585.28
B=64 Low77.8981.8278.4980.43
B=64 Medium76.1670.4376.0376.10
B=1,000 Medium78.7869.7278.5078.78
B=1,000 High79.6968.0579.3779.66
B=1,000 Very-High78.6667.2678.5378.66
ALL80.3667.5479.6880.36
Seven-regime mean78.6573.4978.7979.90

Interpretation. The benefit is selective utility, not a claim of uniformly best accuracy. With four labels, STTI improves over the same P⁰ by +6.30 pp (Very-Low) and +2.54 pp (Low); its, with an aggregate improvement is gain of +1.250 pp [0.644, 1.931] (Table R1). Unconditional Pᴵ is better in Very-Low (89.61% vs. 85.28%), showing a conservatism cost, but becomes harmful as coverage drops: in ALL it scores 67.54% vs. P⁰’s 80.36% (−12.82 pp), while STTI preserves 80.36%. The controller intervenes in 43.35% of Very-Low and 21.35% of Low episodes, and rarely in Very-High/ALL, preventing harmful counterfactuals; we do not count matching P⁰ as an improvement. Labels are queried even when the policy abstains.

Importantly, a simpler cache with a source-trained gate does not close the gap: STTI exceeds it by +0.15 pp [0.02, 0.64], +0.55 pp [0.21, 0.98], and +1.10 pp [0.59, 1.66] at M=1/2/4 (paired dataset-bootstrap CIs); at M=4, all ten dataset means favor STTI. At M=4 in the three sparse B=64 regimes, STTI gains +2.93 pp over P⁰ vs +0.81 pp for gated caching, while High/Very-High/ALL means are −0.01 vs. −0.38 pp. Thus, the full semantic-intervention-plus-selection design improves use of sparse labels beyond adding an abstention switch, though it does not disentangle each transducer component’s effect. For PE-Core, EVA02, and SigLIP2, unconditional Pᴵ can exceed selective STTI in aggregate accuracy, consistent with targeting source-calibrated risk-sensitive behavior rather than maximizing target mean accuracy.

AuthorsOpenReview ↗

Distribution-Shift Robustness

Q3. How sensitive are source composition and risk thresholds, and what happens under larger shifts?

Source composition and calibration. We varied prespecified source pools and source-side risk floors while keeping the upstream intervention operator fixed. Performance is not invariant: across three fixed 3-source draws, the M=4 gain over P⁰ ranges from −0.204 to +1.229 pp (clean CLIP-source) and +0.754 to +1.972 pp (clean SigLIP2-source). With six sources, ranges are +0.177 to +1.214 pp and +0.281 to +1.305 pp. These are ranges over fixed draws, not confidence intervals, and do not justify selecting the best draw on the target.

Table R3. Source-selected risk tolerance versus held-out target behavior (separate clean SigLIP2-source refitting study, M=4). All values are percentage points relative to the same P⁰. Tail = mean of the worst 10% of target dataset–regime changes.

Permitted source-side risk floorSeven-regime mean ΔTarget tail ΔWorst target cell Δ
−0.25 pp+0.039−2.962−8.469
−0.50 pp (prespecified)+0.034−3.070−8.781
−1.00 pp+0.326−3.108−8.875
−2.00 pp+0.818−3.108−8.875

Interpretation. Relaxing the source-calibrated constraint may increase average target utility without removing harmful target cells. With the prespecified −0.50 pp floor, the separately source-refitted SigLIP2-primary controller achieves only +0.034 pp [0.000, 0.088] gain over P⁰ (exact dataset sign p=0.25) and fails the dense-regime 0.5 pp noninferiority criterion. We therefore do not claim that the fixed-parameter transfer in Table R1 ensures successful independent refitting or target-domain risk control. These sensitivity results use a different clean source-training protocol and are not numerically comparable to Table R1.

Larger domain shifts. We also reran the fixed-parameter CLIP-primary pipeline on 304 matched clean/shift task pairs, keeping the 1,000 candidate classes and the same sampled budgets/folds. Because the panels are ImageNet-derived, they test substantial visual shift within a shared class ontology, not new unseen classes.

Table R4. Corrected M=4 shift utility: STTI minus matched P⁰ (pp). Seven-regime means are descriptive; Very-Low and Low are reported to reveal where limited query coverage still succeeds or fails.

Shift panelSeven-regime ΔB=64 Very-Low ΔB=64 Low Δ
ImageNet-A+0.113+1.271−0.481
ImageNet-R+0.316+1.901+0.703
ImageNet-Sketch+1.464+5.250+5.234
ImageNet-V2+1.314+7.959+1.236

Interpretation. Table R4 shows positive seven-regime mean gains over matched P⁰ on all four ImageNet shifts, largest on ImageNet-Sketch (+1.464 pp) and ImageNet-V2 (+1.314 pp). Gains are strongest when sparse annotations provide good semantic coverage. At B=64 Very-Low, STTI improves preservation by +5.250 pp (Sketch) and +7.959 pp (V2); at B=64 Low, by +5.234 pp and +1.236 pp. Paired task-bootstrap 95% intervals for Sketch/V2 are positive in both regimes.

Selective preservation under shift. ImageNet-R improves, while ImageNet-A shows a smaller gain (+0.113 pp): up in Very-Low (+1.271 pp) but down in Low (−0.481 pp). The A-Low result is also worse than the matched clean episode (−1.442 pp [−2.611, −0.469]), indicating sensitivity. In the tested B=1,000 and ALL regimes, STTI preserves the matched P⁰ predictions. Overall, STTI delivers meaningful gains under some distribution shifts when sparse supervision is informative, while conservatively preserving behavior in higher-complexity regimes. Because the evaluation uses the shared ImageNet ontology and capped A/R are not standard restricted 200-class scores, these results support selective transfer across the tested shifts, not general distribution-shift robustness.

Revision and discussion. We provide additional experiments and numerical results for review. During the discussion period, we will update the paper to include the latest cross-backbone evaluation, matched controls, and calibration/shift qualifications. We welcome further questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up before the discussion deadline

We sincerely thank reviewers PF8T, eVaF, and GLcN for their constructive feedback and the Area Chair for overseeing the review process.

We have provided additional experiments, controlled comparisons, and analyses addressing the key concerns raised. We appreciate the reviewers' time and consideration of these results.

As the author–reviewer discussion period approaches its conclusion, we would welcome any final feedback on whether our responses have adequately addressed the concerns or whether any important issues remain unresolved.

We remain happy to provide further clarification before the discussion deadline. Thank you again for your valuable feedback and consideration.

Dear Reviewers and Area Chair,

We have uploaded the revised manuscript, incorporating the additional experiments and clarifications presented in our rebuttal, including matched controls, cross-backbone and distribution-shift evaluations, and computational analysis.

Thank you for your time, constructive feedback, and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.