- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper treats the various test-time adaptation interventions (updating BN statistics, flip averaging, etc.) as a fixed set of candidate prediction rules. For each batch it then answers three questions: which candidate to deploy, which inputs to label, and when to stop buying labels. The method is called PORTAL. It groups inputs by the joint prediction pattern of all candidates, so that one label can score every candidate at once. It also maintains a shared Dirichlet posterior over the candidates' accuracies. Labels go to rivals that could still overtake the current leader within the remaining budget. Querying stops when no rival is reachable, or when an upper bound on the value of information falls below the label price. Four theorems relate the number of labels required to accuracy gaps and shared mistakes. Experiments on PACS, VLCS, OfficeHome, CIFAR-C and ImageNet-C show the following. The selection mechanism avoids the collapse of BN+flip under class-block arrival (22.58 to 51.28). On one PACS replay it turns a harmful challenger into a small gain. On CIFAR-C it uses 91-95% fewer labels than fixed-budget selectors. These labels can also be used afterwards to train stronger candidates.
Critical strengths
- The paper identifies a problem that is easy to overlook: the same test-time intervention can help on one batch and hurt on the next, and average accuracy hides this. The problem is clearly defined (fixed candidates, a query set and a deployment set per batch), and its existence is demonstrated: under class-block arrival BN+flip drops from 50.9 to 22.58 (Fig. 2, Table 49). The method is a decision-level algorithmic contribution rather than a new model. It tries to get better behaviour out of existing candidates by improving how they are orchestrated at the algorithm level, instead of by training on more data or compute. At this venue that is an uncommon and welcome direction, and I value it.
- The work is careful and honest. The experiments are extensive, the theoretical proofs are detailed, and the authors report results that go against their method as well as those in its favour.
Critical weaknesses
-
PORTAL is not more accurate than simply using the best fixed rule with no labels at all. On CIFAR-10-C and CIFAR-100-C it is 0.39 and 0.45 points below fixed BN+flip, respectively (Table 2). On the main PACS stream it reaches 59.31 with about 131 labels. Fixed BN+flip and mean logits reach 59.68 and 61.63 with zero labels (Table 62). On ImageNet-C it retains about 73% of the gain obtained by always applying the intervention (Sec. 4.2). Across the eleven PACS arrival conditions in Table 49, it beats BN+flip in only two. The only clear win is class-block arrival. There BN+flip collapses to 22.58 and PORTAL gets 51.28. But under this condition the plain flip rule gets 50.90 and a simple Max-KL selector gets 54.11. So what PORTAL buys is insurance against a wrong choice, not higher accuracy. The abstract says it "preserves gains" and "retains performance". This reads as more than the results above show.
-
The part of the method that the paper rests on is never tested on its own. The core idea is to group inputs by the joint prediction pattern of all candidates and to maintain a shared posterior with covariance across candidates. The paper ablates the acquisition rule, the stopping rule, the prior, the label price, history, pruning and tie-breaking. But it never replaces the shared posterior with independent per-candidate posteriors while keeping everything else fixed. In Table 62, what actually matters is history. With it, PORTAL beats the ERM selectors by 0.3 to 0.6 points using only one third of the labels. Without it, PORTAL drops to the same level as those selectors (58.98 vs 58.97) and uses the same number of labels. History assumes that class proportions stay stable over time. This is exactly what class-block arrival violates (decay 53.60 vs cumulative 51.74, Table 66).
-
The theory does not describe the algorithm that is actually run. The stopping rule from Theorem 1 "does not fire at price 0.002" (p71). In 654 of 655 batches, querying stops because no rival is reachable. Theorems 3 and 4 analyze a separate pilot-then-confirm procedure in Appendix B and C. The authors themselves say that PORTAL's "finite-batch decisions are evaluated directly in the experiments" (p26). Theorem 2 is a lower bound for delta-correct strategies, and the paper never shows that PORTAL is one. So the abstract's sentence "our analysis links the required feedback to accuracy gaps and shared mistakes" describes a different procedure, not the one run in the tables.
Questions for authors
- Which procedure is actually run in each experiment (Sec. 4.1, 4.2, ImageNet-C, 4.6)? Can the shared-posterior controller of Sec. 2 be run on the PACS/VLCS replays and on ImageNet-C, with scoring only on the evaluation set?
- Please add an ablation that replaces the shared posterior with independent per-candidate posteriors, keeping reachability and history unchanged.
- In which deployment settings should one choose PORTAL over the best fixed rule? Please report in the main text the gap to the best fixed rule in hindsight under each condition, alongside fixed BN+flip and mean logits.
- How does PORTAL relate to knowledge-gradient / EVPI stopping rules in ranking and selection, and to active model selection and active testing? The paper does not discuss these lines of work.
5 comments
I thank the authors for the detailed response. It resolves two of my three concerns in large part. The new control replaces the shared posterior with independent Beta models. With four candidates the shared posterior uses 23 to 41 percent fewer labels at the same accuracy, and with five candidates it is slightly more accurate with fewer labels. This is the ablation I asked for. On the theory, I accept the correction: Theorem 1 does contain the reachability clause, and my statement on this point was incomplete. Theorems 2 to 4 still analyse procedures other than the controller that is run. My first concern is unchanged. In the authors' own summary of Table 66, PORTAL is below the best fixed rule in six of ten conditions, with all gaps within one point, and in the new direct runs it is clearly ahead only under class-block arrival. The value of the method is protection against a bad fixed choice at a low label cost, and the abstract should say this plainly. I raise my rating from 2 to 3 on two conditions. First, the new evidence should be in the paper. The new controls and comparisons exist only in the comments, and I could not find a revised PDF. Second, the main text should carry the evidence for its main claims. I share the concern of another reviewer about length: the paper has 78 pages, and the response answers most of my questions by pointing to appendix tables. If a revision resolves these two points in a reasonable way, my rating is 3. If not, I will return to 2.
We thank Reviewer zzdb for considering raising the score. We have revised the PDF and provided the new content directly in the main text. Specifically, we updated the abstract on page 1, added the gaps to the best fixed rule in Table 4 on page 8 and the independent Beta ablation in Table 5 on page 8, and included the clarifications on procedures, theorems, and related work in Sections 1, 2.3, and 3.1.
We thank Reviewer zzdb again for the review. Our rebuttal provides point-by-point responses and additional controlled experiments addressing the concerns,and we would appreciate the feedback and are happy to clarify any remaining points. In light of the clarification provided, we respectfully hope you to reconsider your score.