acceptodds
Under review as a conference paper at ICLR 2027

When Should We Intervene at Test Time?

11 views

Abstract

Test-time adaptation helps a deployed model respond to changing inputs, but the same intervention can rescue one batch and damage the next. Updating normalization statistics, for example, can improve corrupted images yet fail when a batch contains mostly one class. Average accuracy hides these failures and cannot tell a model when to use an intervention. Target labels can guide this decision, but they are scarce and some choices are already clear. We introduce PORTAL, a controller that chooses when to deploy an intervention, which inputs to label, and when to stop querying. Each available prediction rule is a candidate, and its predictions are fixed before current feedback arrives. Every acquired label therefore evaluates all candidates on the same input. PORTAL combines these shared comparisons and directs queries toward candidates that can still overtake the current leader. It stops when the remaining labels cannot change the choice or when the possible accuracy gain no longer justifies another query. Our analysis links the required feedback to accuracy gaps and shared mistakes, explaining why some choices need much less evidence. Experiments show that this control turns an intervention that loses accuracy overall into a useful choice on selected batches. It prevents severe failures caused by changes in arrival order and preserves gains across complete corruption streams. When a fixed rule already works well, PORTAL uses little feedback to retain its performance. Periodic label checks reveal when past evidence needs refreshing and help the controller recover as conditions change. The acquired labels also train stronger candidates for future batches, and selection brings nearly all of their accuracy gains into deployment. The same feedback therefore improves both the choice made now and the predictions available to later batches.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, active model selection, label-efficient adaptation, test-time intervention, active learning, domain shift

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zzdbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper treats the various test-time adaptation interventions (updating BN statistics, flip averaging, etc.) as a fixed set of candidate prediction rules. For each batch it then answers three questions: which candidate to deploy, which inputs to label, and when to stop buying labels. The method is called PORTAL. It groups inputs by the joint prediction pattern of all candidates, so that one label can score every candidate at once. It also maintains a shared Dirichlet posterior over the candidates' accuracies. Labels go to rivals that could still overtake the current leader within the remaining budget. Querying stops when no rival is reachable, or when an upper bound on the value of information falls below the label price. Four theorems relate the number of labels required to accuracy gaps and shared mistakes. Experiments on PACS, VLCS, OfficeHome, CIFAR-C and ImageNet-C show the following. The selection mechanism avoids the collapse of BN+flip under class-block arrival (22.58 to 51.28). On one PACS replay it turns a harmful challenger into a small gain. On CIFAR-C it uses 91-95% fewer labels than fixed-budget selectors. These labels can also be used afterwards to train stronger candidates.

Critical strengths
  1. The paper identifies a problem that is easy to overlook: the same test-time intervention can help on one batch and hurt on the next, and average accuracy hides this. The problem is clearly defined (fixed candidates, a query set and a deployment set per batch), and its existence is demonstrated: under class-block arrival BN+flip drops from 50.9 to 22.58 (Fig. 2, Table 49). The method is a decision-level algorithmic contribution rather than a new model. It tries to get better behaviour out of existing candidates by improving how they are orchestrated at the algorithm level, instead of by training on more data or compute. At this venue that is an uncommon and welcome direction, and I value it.
  2. The work is careful and honest. The experiments are extensive, the theoretical proofs are detailed, and the authors report results that go against their method as well as those in its favour.
Critical weaknesses
  1. PORTAL is not more accurate than simply using the best fixed rule with no labels at all. On CIFAR-10-C and CIFAR-100-C it is 0.39 and 0.45 points below fixed BN+flip, respectively (Table 2). On the main PACS stream it reaches 59.31 with about 131 labels. Fixed BN+flip and mean logits reach 59.68 and 61.63 with zero labels (Table 62). On ImageNet-C it retains about 73% of the gain obtained by always applying the intervention (Sec. 4.2). Across the eleven PACS arrival conditions in Table 49, it beats BN+flip in only two. The only clear win is class-block arrival. There BN+flip collapses to 22.58 and PORTAL gets 51.28. But under this condition the plain flip rule gets 50.90 and a simple Max-KL selector gets 54.11. So what PORTAL buys is insurance against a wrong choice, not higher accuracy. The abstract says it "preserves gains" and "retains performance". This reads as more than the results above show.

  2. The part of the method that the paper rests on is never tested on its own. The core idea is to group inputs by the joint prediction pattern of all candidates and to maintain a shared posterior with covariance across candidates. The paper ablates the acquisition rule, the stopping rule, the prior, the label price, history, pruning and tie-breaking. But it never replaces the shared posterior with independent per-candidate posteriors while keeping everything else fixed. In Table 62, what actually matters is history. With it, PORTAL beats the ERM selectors by 0.3 to 0.6 points using only one third of the labels. Without it, PORTAL drops to the same level as those selectors (58.98 vs 58.97) and uses the same number of labels. History assumes that class proportions stay stable over time. This is exactly what class-block arrival violates (decay 53.60 vs cumulative 51.74, Table 66).

  3. The theory does not describe the algorithm that is actually run. The stopping rule from Theorem 1 "does not fire at price 0.002" (p71). In 654 of 655 batches, querying stops because no rival is reachable. Theorems 3 and 4 analyze a separate pilot-then-confirm procedure in Appendix B and C. The authors themselves say that PORTAL's "finite-batch decisions are evaluated directly in the experiments" (p26). Theorem 2 is a lower bound for delta-correct strategies, and the paper never shows that PORTAL is one. So the abstract's sentence "our analysis links the required feedback to accuracy gaps and shared mistakes" describes a different procedure, not the one run in the tables.

Questions for authors
  1. Which procedure is actually run in each experiment (Sec. 4.1, 4.2, ImageNet-C, 4.6)? Can the shared-posterior controller of Sec. 2 be run on the PACS/VLCS replays and on ImageNet-C, with scoring only on the evaluation set?
  2. Please add an ablation that replaces the shared posterior with independent per-candidate posteriors, keeping reachability and history unchanged.
  3. In which deployment settings should one choose PORTAL over the best fixed rule? Please report in the main text the gap to the best fixed rule in hindsight under each condition, alongside fixed BN+flip and mean logits.
  4. How does PORTAL relate to knowledge-gradient / EVPI stopping rules in ranking and selection, and to active model selection and active testing? The paper does not discuss these lines of work.
5 comments

We thank Reviewer zzdb again for the review. Our rebuttal provides point-by-point responses and additional controlled experiments addressing the concerns,and we would appreciate the feedback and are happy to clarify any remaining points. In light of the clarification provided, we respectfully hope you to reconsider your score.

I thank the authors for the detailed response. It resolves two of my three concerns in large part. The new control replaces the shared posterior with independent Beta models. With four candidates the shared posterior uses 23 to 41 percent fewer labels at the same accuracy, and with five candidates it is slightly more accurate with fewer labels. This is the ablation I asked for. On the theory, I accept the correction: Theorem 1 does contain the reachability clause, and my statement on this point was incomplete. Theorems 2 to 4 still analyse procedures other than the controller that is run. My first concern is unchanged. In the authors' own summary of Table 66, PORTAL is below the best fixed rule in six of ten conditions, with all gaps within one point, and in the new direct runs it is clearly ahead only under class-block arrival. The value of the method is protection against a bad fixed choice at a low label cost, and the abstract should say this plainly. I raise my rating from 2 to 3 on two conditions. First, the new evidence should be in the paper. The new controls and comparisons exist only in the comments, and I could not find a revised PDF. Second, the main text should carry the evidence for its main claims. I share the concern of another reviewer about length: the paper has 78 pages, and the response answers most of my questions by pointing to appendix tables. If a revision resolves these two points in a reasonable way, my rating is 3. If not, I will return to 2.

We thank Reviewer zzdb for considering raising the score. We have revised the PDF and provided the new content directly in the main text. Specifically, we updated the abstract on page 1, added the gaps to the best fixed rule in Table 4 on page 8 and the independent Beta ablation in Table 5 on page 8, and included the clarifications on procedures, theorems, and related work in Sections 1, 2.3, and 3.1.

Reviewer_Sg99OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces PORTAL, a controller that chooses when to deploy a test-time intervention, which inputs to label, and when to stop querying. Motivated by the observation that the same intervention can rescue one batch and damage the next, PORTAL uses limited target labels to evaluate candidates through shared comparisons. It directs queries toward candidates that can still overtake the current leader and stops when further queries cannot change the choice or justify their cost. The analysis links required feedback to accuracy gaps and shared mistakes. Experiments show that PORTAL prevents severe failures caused by changes in arrival order, preserves gains across complete corruption streams, and uses little feedback when a fixed rule already works well. The acquired labels also train stronger candidates for future batches.

Critical strengths
  • The paper addresses an interesting problem in test-time adaptation, where the same intervention can improve accuracy on one batch but cause substantial degradation on another. Selecting interventions for individual batches provides a useful approach to avoiding harmful interventions.
  • Portal makes use of shared comparisons, where every acquired label evaluates all candidates on the same input. This allows the controller to compare candidate performance using limited target labels and reduce labeling costs.
  • PORTAL combines Bayesian posterior estimation with active querying and stopping rules. It directs queries toward candidates that can still overtake the current leader and stops when the remaining labels cannot change the choice or when the possible accuracy gain no longer justifies another query.
  • The paper provides theoretical analysis linking the required feedback to accuracy gaps and shared mistakes.
  • The paper evaluates PORTAL across multiple datasets and distribution-shift settings. The experiments show that PORTAL prevents severe failures caused by changes in arrival order, preserves gains across complete corruption streams, and uses little feedback when a fixed rule already works well.
Critical weaknesses
  • One of my major concerns is selecting a candidate using limited target feedback that has already been studied in active model selection. Specifically, CODA (ICCV 2025), Online active model selection (AISTATS 2021), and HILTTA (TMLR 2024) should be compared with the proposed method. The paper does not clearly describe which points have novelty compared with the previous works.

  • The main experiment does not provide a direct comparison between the proposed method and the previous works such as CODA. Although Section 4.1 mentions a HILTTA-style controller, this limited comparison does not establish superiority over an established TTA selection method across the main benchmarks. A more systematic evaluation under the same candidate pool, query budget, and deployment protocol is needed.

  • While Table 3 provides a useful acquisition ablation, the results do not establish the necessity of the full framework. In particular, PORTAL achieves 59.32% accuracy with 85.6 labels, compared with 59.31% and 96.8 labels for disagreement querying. Since these methods share the same posterior, history, budget, and stopping rule, additional ablations are needed to isolate the contributions of covariance modeling, reachability stopping, and EVPI stopping. The computational overhead of these components should also be reported.

  • In Table 2, on CIFAR-10-C and CIFAR-100-C, PORTAL performs slightly worse than fixed BN+flip despite requiring target annotations. Although avoiding harmful interventions is a valid objective, the paper should better establish when and how often this benefit justifies the additional labeling and computational costs.

  • PORTAL requires label feedback before finalizing the current batch’s deployment decision, but annotation delays and inference overhead receive limited evaluation. Larger candidate pools may also produce sparse prediction-pattern statistics. Scalability and robustness to delayed or noisy feedback remain unclear.

Questions for authors
  • Could the authors provide a more systematic comparison with existing active model selection methods under the same candidate pool, query budget, and deployment protocol? In particular, how much improvement comes from PORTAL's selection mechanism rather than the benefits of supervised candidate learning?
1 comment

We would like to thank the reviewer again for the thorough review. To further address any remaining concerns, we have made the code used in this study available for replication on anonymous repository. You can use this code to verify the rigor of our work: https://anonymous.4open.science/r/PORTAL-8322/experiments/coda_pool/README.md

Thank you again for your time.

Reviewer_Wpr5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to choose among test-time prediction rules using a limited target-label budget. PORTAL compares candidates through shared labels, directs queries toward comparisons that may change the decision, and stops when further querying is unlikely to be worthwhile. The paper provides theoretical analysis of feedback requirements and stopping decisions, alongside experiments under distribution shifts and different arrival orders.

Critical strengths
  1. The paper addresses a practical concern: an adaptation rule that helps on average can still substantially harm individual batches.

  2. Each queried label evaluates all candidates on the same input, making efficient use of feedback and accounting for shared errors.

  3. The evaluation examines accuracy, intervention-induced harm, and label consumption across multiple datasets and arrival conditions.

Critical weaknesses
  1. I have concerns about the practical feasibility of this setting. PORTAL requires access to target-domain labels during deployment and assumes that query and deployment samples share the same true-label distribution within each prediction-pattern stratum. The former may be acceptable in active TTA or continual learning with an available replay buffer, but the latter is difficult to ensure in practice. Under severe corruption, such as severity level 5 in ImageNet-C, even human annotators may be unable to reliably determine the ground-truth labels. I believe these requirements substantially limit PORTAL’s practical applicability.

  2. The paper provides theoretical analysis, but the results mainly support candidate comparisons and stopping decisions for the current batch under specific assumptions. They do not directly establish the overall optimality of the practical querying strategy or its deployment benefits. Ideas such as querying disagreements and dropping clearly inferior candidates are also fairly intuitive. The authors should clarify how much consistent improvement PORTAL offers over simple selectors with the same label budget, and which improvements the theory actually explains.

  3. In practice, PORTAL looks more like a candidate-policy selector using a small number of target-domain labels. “No intervention” is simply one option obtained by including the original model in the candidate set; the method does not directly address how to design interventions or determine their strength. The paper should define its contribution more precisely and explain what new problem or method it introduces beyond active model selection or best-candidate identification.

Questions for authors

Please check the weaknesses. If the author can address my questions, I will raise my score.

Comments on the paper

4 comments

Dear Reviewers,

The authors have submitted their responses to your comments. If you have not already done so, please review their rebuttals and engage in further discussion, particularly regarding any remaining concerns, unresolved issues, or points requiring additional clarification.

Thank you for your time and continued contributions to the review process.

Regards,

AC

Dear Reviewers,

We thank all reviewers again for their time and thoughtful feedback. We summarize the main evidence from our rebuttal below and share additional results addressing the fairness of model-selection comparisons. We have now completed comparisons on CODA’s original tasks. Every method selects one candidate using 100 labels. The table reports cumulative regret; lower is better.

MethodPACSCIFAR-10 lowCIFAR-10 high
PORTAL4.8437.1862.05
CODA, published57.958.774.1
CODA, our rerun57.9858.6874.15
Random sampling, published216.6410.6346.2
Uncertainty sampling, published6.9629.7281.7
Active Testing, published101.8399.9154.9
VMA, published116.5476.5383.7
ModelSelector, published, AISTATS 202597.9567.290.9
HILTTA selector, adapted inputs100.96466.69195.33

PORTAL achieves the lowest cumulative regret among all listed methods on all three tasks and reaches the highest available terminal accuracy. Published values come from CODA, ICCV 2025, Table 1. Our CODA rerun closely reproduces those results. The adapted HILTTA selector uses prediction probabilities as query features and their mean as its anchor.

Label feedback under severe corruption follows established practice: SimATTA, ICLR 2024 evaluates Tiny-ImageNet-C at severity 5 (Section 5.2, Table 3); HILTTA, TMLR 2024 evaluates ImageNet-C at severity 5 with human-label feedback (Appendix A.1); EATTA, CVPR 2025 also evaluates ImageNet-C at severity 5 with target-label feedback (Section 4, Table 1). PORTAL studies how to use this feedback more effectively.

We hope that you will consider our rebuttal. We are happy to clarify any remaining issues and respectfully ask the reviewers to reconsider their comments in light of the clarifications provided.

best,

authors

We thank reviewer Sg99 for the time and constructive feedback.Please review our latest response and the revised manuscript. They include a comparison under CODA’s original single-candidate protocol. and please let us know if you still have any questions or if there is anything that needs improvement.We will do our best to solve your problem.

AuthorsOpenReview ↗

Rebuttal Summary

We thank all reviewers for their constructive comments. Below, we summarize the main content used to address the reviewers’ concerns. All additional experimental results, annotation data, and reproduction scripts are available for inspection and reproduction in our anonymous repository.

  1. Direct comparison on CODA’s original tasks. All methods select one unchanged model from the same supplied candidate pool, use 100 labels, and score the complete original population. The table below reports cumulative regret at 100 labels; lower is better.
MethodSourcePACSCIFAR-10 lowCIFAR-10 high
PORTALThis work4.8437.1862.05
CODA, publishedICCV 202557.958.774.1
CODA, our rerunICCV 202557.9858.6874.15
Random sampling, publishedStandard baseline216.6410.6346.2
Uncertainty sampling, publishedStandard baseline6.9629.7281.7
Active Testing, publishedICML 2021101.8399.9154.9
VMA, publishedMachine Learning 2024116.5476.5383.7
ModelSelector, publishedAISTATS 202597.9567.290.9
HILTTA selector, adapted inputsTMLR 2024100.96466.69195.33
  1. Controlled evidence for shared feedback and stopping. Under matched priors, feedback access, and single-candidate deployment, replacing the shared posterior with independent Beta models reduces label efficiency. With four candidates, the shared model uses 23–41% fewer labels, with a mean accuracy gap within 0.08 percentage points. With five candidates, the shared model achieves 61.96% using 130.8 labels, versus 61.45% using 150 labels for independent estimation. A separate stopping comparison achieves the same 59.32% accuracy with 85.6 labels, whereas confidence stopping requires 392.8 labels.

  2. Annotation availability and practical benefits of annotation under severe corruption. Two independent annotators evaluated 200 ImageNet-C severity-5 images without clean references, model predictions, or reference answers. Explicit-answer coverage was 74.5% / 83.5%, and agreement with benchmark labels was 76.51% / 88.62%, respectively. The deployment accuracy of the learned combination supported by the two annotation sets was 50.12% / 50.17%, respectively, compared with 45.68% for fixed averaging.

  3. Deployment gains. Direct fixed-candidate selection on class-block PACS reaches 67.46%, compared with 54.90% for fixed averaging, using 86.3 labels.

Together, these results demonstrate PORTAL’s practical utility and its performance gap relative to existing methods. We thank the reviewers and the chairs again for their active engagement and time during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.