- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to decide whether to deploy test-time adaptation in a search system using a small labeled sample. It compares no adaptation, adaptation on every query, and selective adaptation using a score-entropy threshold, replacing the incumbent only when statistical evidence supports an improvement in average performance. It also analyzes selection risk and the potential gains from routing. Offline replay shows fewer harmful deployments, although finite-sample guarantees are conservative and the tested entropy-threshold routing rules generally offer limited gains over the best fixed action.
Critical strengths
With limited labels, CPS reduces harmful deployments in offline experiments while retaining gains over the incumbent. Reporting both harmful-deployment rates and regret makes the tradeoff between safety and performance clear. The per-query caches also facilitate verification of the results.
Critical weaknesses
-
The claimed upper bound on the routing value of the full entropy signal is unsupported. Section 5.5 uses cross-fitted and in-sample estimates over ten quantile bins to bracket the optimal routing gain achievable from entropy. These estimates underpin the claim that entropy captures at most about one fifth of the available per-query routing headroom. However, even if every binwise conditional mean were known exactly, the resulting routing value could be strictly below the optimum attainable from the full signal. The optimistic bias of in-sample fitting does not control this information loss. The empirical analysis fails to distinguish limited information in entropy from limited expressiveness of the routing rules. Consequently, the gains achieved by restricted rules do not justify the claimed ceiling on the value of the full signal. This undermines a central empirical conclusion.
-
The contribution beyond existing safe-selection methods is not sufficiently established. Table 2 evaluates complete selection procedures containing both fixed and routed actions. The value added by entropy-based routing within these procedures remains unclear. A matched ablation should hold the fixed-action configurations, labeling budget, and certification procedure constant while comparing fixed-action-only selection, selection with entropy-based routing, and selection with shuffled-entropy routing. The best threshold fitted on the full data pool in Section 5.5 cannot substitute for this comparison. Given the reliance on existing statistical tools and the unsupported claim about signal value, the evidence does not establish a substantial contribution beyond those tools.
The presentation also overstates parts of the contribution and contains jumps in reasoning. The abstract prominently lists supporting analyses, including a standard sample-size formula, alongside the main results. Section 4 describes Proposition 2 as showing that routing gains are "often small", although the proposition provides no such conclusion. These statements blur the distinctions between existing tools, new results, and empirical observations. It is the authors’ responsibility to make these distinctions clear and ensure that their claims match the supporting evidence. Readers should not have to reconstruct them by repeatedly cross-checking the exposition, theorems, and experiments.
The central empirical conclusion lacks adequate support, and the remaining contribution is insufficient to justify acceptance on its own. I would reconsider my rating if the authors address these concerns with sufficient evidence or make substantive revisions that still establish a significant contribution. Changes limited to wording or other minor issues would not change my recommendation.
Questions for authors
-
How does Section 5.5 control the routing value lost through binning, and thereby support the claim that the full entropy signal captures at most about one fifth of the available per-query routing headroom?
-
Under the same fixed-action configurations, labeling budget, and certification procedure, how do harmful-deployment rates and regret differ among fixed-action-only selection, selection with entropy-based routing, and selection with shuffled-entropy routing? What additional value do these comparisons establish?