- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies risk certification for two-stage model routing, where selection-conditioned error need not vary monotonically with routing thresholds. FS-Cert-U freezes a candidate family and screening order, applies exact binomial tests with fixed-sequence stopping, and optimizes utility within the certified set. Experiments across 33 QA configurations show utility gains over matched Bonferroni and Holm procedures.
Critical strengths
Clear risk formulation. Extensive matched experiments.
Critical weaknesses
Thanks for submitting this paper. The distinction between certifying a policy set and selecting a policy within that set is useful, and the experimental reporting is careful. However, I still have several major concerns that support rejection.
First, the methodological novelty appears limited. The construction combines a conditional binomial test, independently ordered fixed-sequence testing, and empirical utility maximization. Theorem 1 follows by considering the first unsafe hypothesis, while Theorem 2 follows from Hoeffding’s inequality and a union bound. Appendix F.5 acknowledges that fixed-sequence testing itself is not new, but the listed routing-specific features do not clearly require a new statistical argument: shared samples are already allowed by the procedure, and conditioning handles the random denominator. What technical obstacle do you resolve beyond instantiating Learn-then-Test with standard selective-risk tests and fixed-sequence FWER control?
Second, the main comparison does not establish the value of the proposed ordering strategy against equally informed alternatives. FS-Cert-U uses screening data to prioritize candidates, whereas Bonferroni and Holm do not exploit that order. This is a valid comparison of those procedures, but it leaves unclear whether the gains require stopping at the first failure. An independently screened single-candidate test or screen-weighted Bonferroni procedure would also provide valid protection while using the screening information. Can you compare against these alternatives under the same data budget to show what fixed-sequence certification specifically contributes?
Third, the utility guarantee excludes the main source of potential utility loss. Theorem 2 compares the selected policy only with the best policy already inside the certified prefix. It does not measure losses from candidate discretization or early stopping. These losses appear substantial: Table 7 reports utility of 0.349 under p-value-first ordering but −0.244 under utility-first ordering, with nonempty rates of 91.5% and 0.5%, respectively. Although the paper explicitly acknowledges this guarantee boundary, can you quantify the gap to the best safe policy in the frozen family and separate screening, certification, and final-selection losses? Without this analysis, the regret theorem offers limited guidance about routing quality.
Questions for authors
How does FS-Cert-U compare with screening-aware alternatives, such as testing only the best screened candidate or using screen-weighted Bonferroni correction, under the same data budget?
Can you quantify the utility gap to the best population-safe policy in the frozen family, separating losses from screening order, certification stopping, and final utility selection?
Response to Reviewer zxSg
We thank the reviewer for the constructive feedback. We clarify our methodological contribution and present new screening-aware comparisons and safe-oracle analyses.
1. Contribution Beyond Standard LTT
FS-Cert-U contributes a routing-specific certification-to-deployment framework, not a new multiple-testing rule. Although LTT/FST provides general testing guarantees, it does not specify how to integrate non-monotonic, selection-conditioned routing risk with downstream cost-sensitive policy selection. Our framework addresses:
Our contribution is the integrated safety–utility guarantee for the full routing-to-deployment procedure, rather than a new testing inequality.
2. Screening-Aware Comparisons Under Matched Budgets
To address unequal access to screening information, we added screening-best single-candidate testing, screening-weighted Bonferroni, and random-order FST. All methods share frozen candidates, screening information, certification samples, and held-out evaluations across 33 QA configurations × 30 matched splits (α=0.10, δ=0.05, λ=0.25). Empty certificates trigger zero-cost direct abstention.
For screening-weighted Bonferroni, screening p-values determine softmax weights wi with temperature T=0.05; candidate i is certified when picert≤δwi.
CIs are pointwise bootstrap intervals over configuration-level mean paired differences.
The gains cannot be explained merely by access to screening information. The single-candidate baseline highlights the value of retaining multiple certified options; weighted Bonferroni provides another screening-aware comparison; and random-order FST illustrates the importance of informative ordering.
FS-Cert-U is a routing-optimized instantiation of fixed-sequence certification. Its advantage arises from integrated candidate management, certification, and deployment—not a new FST rule or universally optimal ordering.
3. Safe-Oracle Gap and Utility-Loss Decomposition
We conducted experiments on five controlled populations with known risks and utilities, with 1,000 independent calibration trials each. Conditional on safe, nonempty certification, we decomposed the gap to the best population-safe policy in the frozen family (four populations shown):
The dominant losses arise from limited test power and early-stopping exclusion, not within-set utility selection. This quantifies the safe-oracle gap beyond Theorem 2's certified-set guarantee.
Paired counterfactuals holding candidate families and certification p-values fixed show that screening-guided ordering reduces the diagnostic stopping penalty in Disconnected from 0.4700 to 0.1381 (gain +0.3319, 95% CI [+0.3214, +0.3426]); the ordering benefit is +0.6399 in Irregular. The oracle-based no-stopping counterfactual is diagnostic, not a deployable FWER-controlled procedure.
In Narrow-corridor, 94% of trials yield empty certificates, producing an all-trial safe-oracle gap of 0.8846. This exposes finite-sample certification power as a limitation in constrained regimes.
Overall, FS-Cert-U improves utility over the evaluated equally informed alternatives, while the decomposition identifies the remaining losses and clarifies the scope of its theoretical guarantees.