- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses that correct sparse target labels can still be unsafe for VLM test-time adaptation due to insufficient semantic coverage. It proposes Selective Test-Time Intervention (STTI), a preserve-or-intervene framework treating sparse labels as optional semantic evidence. STTI uses complementary frozen VLM views, active querying, a permutation-equivariant transducer, and a risk-constrained controller to intervene only with sufficient semantic authority. On 11 VLM benchmarks, STTI retains gains under high coverage and reverts to preservation as coverage drops.
Critical strengths
-
The STTI design is interesting and novel, especially compared with existing methods that default to adaptation whenever labeled information is available.
-
It achieves strong results on 11 VLM benchmarks, and the experimental results are reasonably convincing.
Critical weaknesses
-
Experimental performance actually degrades in simple semantic scenarios, which limits the applicability of the method.
-
The method adopts a dual-VLM backbone consisting of OpenAI CLIP and SigLIP2, whereas most baselines use only CLIP. This makes it difficult to establish whether the observed gains stem from the STTI design rather than from the auxiliary visual information provided by SigLIP2.
Questions for authors
-
As shown in Table 26, why does performance drop significantly when SigLIP2 is used as the primary backbone? Is the method's contribution more attributable to the STTI design or to the choice of primary backbone?
-
What is the computational overhead of the dual-VLM architecture adopted by the method compared with the baselines?
3 comments
Cross-Backbone Generalization
Q2. Is the improvement mainly due to SigLIP2, and what explains the earlier SigLIP2-primary result?
To separate gains from stronger frozen representations vs selective semantic intervention, we compare primary-only, auxiliary-only, frozen two-view preservation (P0), and STTI. P0 and STTI use the same frozen encoders and preservation predictions, so STTI − P0 isolates the added benefit of the full intervention-and-selection pipeline, independent of adding the auxiliary encoder. The source-trained intervention operator, controller, and thresholds are fixed in all cross-backbone transfer experiments.
Table R2. Component attribution and frozen cross-backbone transfer (M=4). Top-1 accuracy (%) averaged equally across 10 datasets and 7 semantic-complexity regimes. Primary StatA, auxiliary-only, and P0 are label-free (M=0); STTI queries 4 labels. Confidence intervals are paired, dataset-bootstrap 95% intervals with fixed learned parameters.
| Primary / auxiliary | Primary StatA | Auxiliary only | P0 (two views) | STTI | STTI − P0 (pp), 95% CI |
|---|---|---|---|---|---|
| CLIP / SigLIP2 | 69.10 | 80.70 | 78.65 | 79.90 | +1.250 [0.644, 1.931] |
| PE-Core / SigLIP2 | 85.06 | 80.70 | 85.43 | 85.97 | +0.537 [0.266, 0.865] |
| EVA02 / SigLIP2 | 79.53 | 80.70 | 81.25 | 82.47 | +1.220 [0.400, 2.558] |
| SigLIP2 / OpenCLIP | 81.47 | 78.94 | 82.59 | 82.87 | +0.283 [0.124, 0.507] |
| CLIP / OpenCLIP | 69.10 | 78.94 | 76.94 | 78.24 | +1.300 [0.734, 2.013] |
| PE-Core / OpenCLIP | 85.06 | 78.94 | 85.25 | 85.80 | +0.546 [0.280, 0.878] |
| EVA02 / OpenCLIP | 79.53 | 78.94 | 80.65 | 81.62 | +0.978 [0.387, 1.872] |
The seven-regime average mixes settings where sparse supervision is highly informative with ones where intervention is less reliable. We therefore also report gains separately for sparse and dense semantic regimes.
Table R2a. Sparse vs. dense improvement over frozen preservation P0 at M=4 (pp). Sparse mean averages B64 Very-Low/Low/Medium; dense mean averages B1000 High/Very-High/ALL. B1000 Medium is still included in the overall seven-regime mean.
| Primary / auxiliary | B64 Very-Low | B64 Low | Sparse mean | Dense mean | Overall mean |
|---|---|---|---|---|---|
| CLIP / SigLIP2 | +6.30 | +2.54 | +2.93 | −0.01 | +1.250 |
| PE-Core / SigLIP2 | +3.07 | +0.66 | +1.25 | 0.00 | +0.537 |
| EVA02 / SigLIP2 | +6.57 | +1.83 | +2.84 | 0.00 | +1.220 |
| SigLIP2 / OpenCLIP | +1.48 | +0.53 | +0.66 | 0.00 | +0.283 |
Table R2a reports descriptive subgroup means from the same evaluation panel. The paired confidence intervals in Table R2 reflect overall seven-regime effects, not separately tested subgroup effects.
Interpretation: Incremental contribution and semantic-regime dependence. Table R2 shows STTI adds predictive value beyond the frozen dual-view baseline. For CLIP/SigLIP2, STTI raises P0 from 78.65% to 79.90% (+1.250 pp [0.644, 1.931]), improving all ten datasets at M=4. Gains also hold for stronger backbones: PE-Core (+0.537 pp) and EVA02 (+1.220 pp) and remain when OpenCLIP replaces SigLIP2 as the auxiliary (+1.300, +0.546, +0.978 pp for CLIP, PE-Core, EVA02).
Table R2a shows the source of these gains. Across the three B64 regimes, STTI improves over the matched P0 by +2.93 pp (CLIP), +1.25 pp (PE-Core), +2.84 pp (EVA02), and +0.66 pp (SigLIP2-primary) on average. Gains are largest in semantically concentrated settings, peaking in Very-Low at +6.30 pp (CLIP) and +6.57 pp (EVA02). In the High, Very-High, and ALL regimes, STTI largely matches frozen preservation predictions, with mean differences of −0.01 to 0.00 pp.
These consistent findings show STTI’s gain is not just adding a stronger auxiliary VLM. Its fixed-parameter intervention-and-selection pipeline uses sparse target supervision when informative, but largely preserves the frozen prediction when semantic coverage is insufficient. Thus, the seven-regime mean reflects helpful intervention in concentrated regimes and conservative preservation in dense regimes, not uniformly small gains.
SigLIP2-primary generalization. We also evaluated STTI with SigLIP2 as the primary backbone and OpenCLIP as the auxiliary, reusing the learned intervention operator and controller without backbone-specific refitting. At M=4, STTI improves the frozen preservation baseline from 82.59% to 82.87% (+0.283 pp [0.124, 0.507]), with nine dataset wins and one tie. As Table R2a shows, gains are concentrated in sparse regimes (+1.48 pp in Very-Low; +0.53 pp in Low), while preservation remains stable in B1000 and ALL. Along with PE-Core and EVA02, these results support fixed-parameter cross-backbone applicability across the evaluated settings. Full regime-level results will appear in Appendix F.
Computational Efficiency
Q3. What is the computational cost of the dual-VLM system?
We agree that assessing STTI requires end-to-end computational efficiency, not just cached-feature processing. We therefore expand the comparison of original vs. optimized implementations on the same hardware using SUN397 (397 classes) with batch sizes B=64 and B=1,000.
Table R3. Original and optimized GPU runtime (seconds per transductive batch) and peak allocated memory.
| Method | Original B64 | Optimized B64 | Original B1000 | Optimized B1000 | Peak GPU (GiB) |
|---|---|---|---|---|---|
| Dirichlet | 2.102 | — | 2.655 | — | 2.175 |
| TransCLIP | 0.131 | — | 1.768 | — | 2.381 |
| StatA | 0.054 | — | 1.742 | — | 2.383 |
| MOON | 0.305 | — | 0.622 | — | 2.381 |
| Two-encoder preservation P0 | 1.050 | 0.715 | 9.115 | 1.908 | 5.139* |
| Full STTI | 1.174 | 0.759 | 9.822 | 2.133 | 5.139 |
Memory figures assume B=1,000 and report peak allocated GPU memory. Original values come from the server benchmark; optimized P0/STTI timings come from later runs on the same hardware. P0 memory is unchanged from the original benchmark. A dash means no separate optimized baseline was reported.
Optimization and measured efficiency. Our optimized inference uses BF16 SigLIP2 weights, PyTorch scaled dot-product attention (attn_implementation="sdpa"), and torch.inference_mode() with CUDA autocast. CUDA profiling confirmed the fused FlashAttention kernel (aten::_scaled_dot_product_flash_attention) during SigLIP2 inference. We report these details without attributing the speedup to any single component.
The update cuts total STTI runtime from 1.174 to 0.759 s at B=64 (1.55×; −35.3%) and from 9.822 to 2.133 s at B=1,000 (4.60×; −78.3%).
Incremental cost beyond the dual-VLM baseline. P0 already includes both frozen encoders. Compared to this optimized preservation pathway, the full intervention-and-selection process adds 0.044 s (6.2%) at B=64, rising to 0.225 s (11.8%) at B=1,000. At B=1,000, full STTI costs 2.133 ms per transductive input when amortized over the batch, not 2.133 ms per independent single-image prediction.
Interpretation. These results show substantial implementation improvements and separate the cost of two independent semantic views from the added cost of sparse-label intervention. However, larger-batch overhead remains meaningful, and STTI uses more GPU memory than lightweight single-VLM baselines. The method therefore yields an accuracy–computation trade-off, not negligible inference cost. We will add the optimized results, implementation details, and remaining efficiency limits to the revised paper.
Revision and discussion. We will revise the manuscript to include matched baseline comparisons, corrected cross-backbone evaluation, and optimized computational analysis, with a clearer discussion of accuracy–preservation–efficiency trade-offs. We thank the reviewer for the constructive suggestions, which have strengthened the evaluation.
Sparse-Supervision Trade-offs, Cross-Backbone Generalization, and Computational Efficiency
We thank the reviewer for constructive feedback on the preserve-or-intervene formulation and semantic-complexity evaluation. We agree that the key empirical questions are: (i) why simpler or unconditional methods can beat selective intervention in low-complexity episodes, (ii) how much gain comes from the second VLM vs. STTI (including the SigLIP2-primary result), and (iii) the practical cost of the entire method. We present matched experiments and runtime measurements below.
These additional experimental results are provided here and have not yet been included in the submitted manuscript. We will incorporate relevant findings and clarifications into a revised submission during the discussion period, as appropriate. We welcome further questions or suggestions.
Q1. Why can simpler methods perform better in some sparse semantic regimes?
Semantic complexity measures diversity, not annotation scarcity. In each M=4 experiment, every query episode reveals four target labels. With a semantically concentrated B=64 batch, those labels may cover much of the target population, so aggressive adaptation helps; in high-complexity or ALL-class settings, the same budget covers only a small slice of target semantics. The risk-constrained controller exploits supported interventions without assuming that any correct queried label justifies updating the entire batch.
Table R1a. Matched preservation, intervention, and source-trained gated-cache controls at M=4. Top-1 accuracy (%) averaged equally over ten non-ImageNet datasets per regime. All methods share the same frozen preservation state (P0); adapted methods use the same queried supports and four-label budgets. The gated cache uses a separate source-trained six-feature preserve/update controller.
Interpretation. The reviewer is correct that selectivity is not always maximized accuracy within a single regime. In Very-Low, always intervening reaches 89.61%, above STTI’s 85.28%, although STTI still improves P0 by +6.30 pp. Thus, the selector forgoes some concentrated-regime accuracy instead of always exploiting all four labels. Consistently, unconditional intervention is 5.73 pp below P0 in B64 Medium and 12.82 pp below P0 in ALL, while STTI is −0.06 pp and equal to P0, respectively. At B1000 High, STTI is −0.03 pp below preservation, so protection is not absolute. A more aggressive source-mean-matched cache achieves a higher overall mean (80.66% vs 79.90%) via larger sparse-regime gains, but trails STTI by 1.45 pp in Very-High and 2.24 pp in ALL. The aim is a measured utility–preservation trade-off, not regime-by-regime dominance.
To test whether gains come solely from adding a trained preserve/update gate to a conventional cache, we evaluate a separate source-trained gated-cache control with identical P0, queried images/labels, budgets, and evaluation folds.
Table R1b. Paired comparison of learned gated cache across annotation budgets. Mean accuracy (%) over seven regimes with equal datasets; last column reports STTI − gated cache (percentage points) with paired dataset-bootstrap 95% CIs, conditional on fixed source policies and this benchmark panel.
Interpretation. Table R1b tests whether STTI’s gains can be reproduced by adding a source-trained six-feature preserve/update controller to a conventional cache. With identical preservation predictions (P⁰ = 78.65%), queried samples, budgets, and folds, the gated cache reaches 78.48%, 78.64%, and 78.79% at M=1, 2, and 4. Its best gain over P⁰ is +0.14 pp (M=4), versus +1.25 pp for STTI. STTI outperforms the gated cache by +0.15, +0.55, and +1.10 pp across budgets; all paired 95% CIs exclude zero, and all ten dataset means favor STTI at M=4. The widening gap indicates STTI uses added sparse supervision more effectively. Overall, the results attribute improvements to the full semantic intervention-and-selection pipeline beyond a conventional cache with a learned gate, rather than attributing the improvement merely to selective authorization.