acceptodds
Under review as a conference paper at ICLR 2027

FS-Cert-U: Risk Certification Before Utility Optimization for Non-Monotonic Two-Stage Model Routing

Abstract

Two-stage model routing trades answer coverage against inference cost, but the system-level selection-conditioned error rate (SCER) need not vary monotoni- cally with either routing threshold. Moreover, searching and certifying policies on the same evidence can invalidate fixed-policy guarantees. We introduce FS- Cert-U, which separates policy construction, ordering, certification, and utility selection: design data freeze a finite policy family, screening data fix its order, and independent certification applies exact candidate-wise tests with fixed-sequence stopping. Utility is optimized only within the resulting certified set. We prove finite-sample family-wise safety of the certified policy set without requiring in- dependence across candidate tests, together with a utility-regret bound relative to the best certified policy. Across 33 QA routing configurations, FS-Cert-U shows a systematic certification-power advantage over matched Bonferroni correction: dataset-level utility gains persist across all six evaluated risk targets, while all 33 paired gains are positive at the primary operating point. The advantage also persists across four Guardian-cost weights. Controlled-population experiments directly probe false certification and confirm the predicted decrease in post-certification utility regret with increasing certification data

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
selective question answering, uncertainty quantification, risk calibration, abstention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_zxSgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies risk certification for two-stage model routing, where selection-conditioned error need not vary monotonically with routing thresholds. FS-Cert-U freezes a candidate family and screening order, applies exact binomial tests with fixed-sequence stopping, and optimizes utility within the certified set. Experiments across 33 QA configurations show utility gains over matched Bonferroni and Holm procedures.

Critical strengths

Clear risk formulation. Extensive matched experiments.

Critical weaknesses

Thanks for submitting this paper. The distinction between certifying a policy set and selecting a policy within that set is useful, and the experimental reporting is careful. However, I still have several major concerns that support rejection.

First, the methodological novelty appears limited. The construction combines a conditional binomial test, independently ordered fixed-sequence testing, and empirical utility maximization. Theorem 1 follows by considering the first unsafe hypothesis, while Theorem 2 follows from Hoeffding’s inequality and a union bound. Appendix F.5 acknowledges that fixed-sequence testing itself is not new, but the listed routing-specific features do not clearly require a new statistical argument: shared samples are already allowed by the procedure, and conditioning handles the random denominator. What technical obstacle do you resolve beyond instantiating Learn-then-Test with standard selective-risk tests and fixed-sequence FWER control?

Second, the main comparison does not establish the value of the proposed ordering strategy against equally informed alternatives. FS-Cert-U uses screening data to prioritize candidates, whereas Bonferroni and Holm do not exploit that order. This is a valid comparison of those procedures, but it leaves unclear whether the gains require stopping at the first failure. An independently screened single-candidate test or screen-weighted Bonferroni procedure would also provide valid protection while using the screening information. Can you compare against these alternatives under the same data budget to show what fixed-sequence certification specifically contributes?

Third, the utility guarantee excludes the main source of potential utility loss. Theorem 2 compares the selected policy only with the best policy already inside the certified prefix. It does not measure losses from candidate discretization or early stopping. These losses appear substantial: Table 7 reports utility of 0.349 under p-value-first ordering but −0.244 under utility-first ordering, with nonempty rates of 91.5% and 0.5%, respectively. Although the paper explicitly acknowledges this guarantee boundary, can you quantify the gap to the best safe policy in the frozen family and separate screening, certification, and final-selection losses? Without this analysis, the regret theorem offers limited guidance about routing quality.

Questions for authors

How does FS-Cert-U compare with screening-aware alternatives, such as testing only the best screened candidate or using screen-weighted Bonferroni correction, under the same data budget?

Can you quantify the utility gap to the best population-safe policy in the frozen family, separating losses from screening order, certification stopping, and final utility selection?

1 comment

Response to Reviewer zxSg

We thank the reviewer for the constructive feedback. We clarify our methodological contribution and present new screening-aware comparisons and safe-oracle analyses.

1. Contribution Beyond Standard LTT

FS-Cert-U contributes a routing-specific certification-to-deployment framework, not a new multiple-testing rule. Although LTT/FST provides general testing guarantees, it does not specify how to integrate non-monotonic, selection-conditioned routing risk with downstream cost-sensitive policy selection. Our framework addresses:

  • Routing-aware certification: Changing the Primary threshold can replace a Guardian answer without changing coverage. Exact conditional binomial SCER tests handle policy-dependent random answer counts, including zero-answer cases.
  • Simultaneous safety: Independently determined candidate prioritization and FST provide family-wise safety for the entire certified set despite dependent tests (Theorem 1).
  • Safe utility selection: Data-dependent selection within the certified set preserves safety without further multiplicity correction. Theorem 2 bounds utility-selection regret even when certification data are reused.

Our contribution is the integrated safety–utility guarantee for the full routing-to-deployment procedure, rather than a new testing inequality.

2. Screening-Aware Comparisons Under Matched Budgets

To address unequal access to screening information, we added screening-best single-candidate testing, screening-weighted Bonferroni, and random-order FST. All methods share frozen candidates, screening information, certification samples, and held-out evaluations across 33 QA configurations × 30 matched splits (α=0.10\alpha=0.10, δ=0.05\delta=0.05, λ=0.25\lambda=0.25). Empty certificates trigger zero-cost direct abstention.

For screening-weighted Bonferroni, screening p-values determine softmax weights wiw_i with temperature T=0.05T=0.05; candidate ii is certified when picert≤δwip_i^{\mathrm{cert}} \leq \delta w_i.

BaselineFS-Cert-U utility gain95% bootstrap CI
Screening-best single candidate+0.1045[+0.0795, +0.1299]
Screening-weighted Bonferroni (T=0.05T=0.05)+0.0511[+0.0339, +0.0740]
Random-order FST+0.2920[+0.2320, +0.3499]

CIs are pointwise bootstrap intervals over configuration-level mean paired differences.

The gains cannot be explained merely by access to screening information. The single-candidate baseline highlights the value of retaining multiple certified options; weighted Bonferroni provides another screening-aware comparison; and random-order FST illustrates the importance of informative ordering.

FS-Cert-U is a routing-optimized instantiation of fixed-sequence certification. Its advantage arises from integrated candidate management, certification, and deployment—not a new FST rule or universally optimal ordering.

3. Safe-Oracle Gap and Utility-Loss Decomposition

We conducted experiments on five controlled populations with known risks and utilities, with 1,000 independent calibration trials each. Conditional on safe, nonempty certification, we decomposed the gap to the best population-safe policy in the frozen family (four populations shown):

PopulationTest-power lossOrder/stopping lossSelection lossTotal gap
Monotone0.00010.00290.00000.0030
Irregular0.13910.07280.00000.2119
Disconnected0.43640.13390.00000.5703
Conflict S40.12230.06850.00020.1910

The dominant losses arise from limited test power and early-stopping exclusion, not within-set utility selection. This quantifies the safe-oracle gap beyond Theorem 2's certified-set guarantee.

Paired counterfactuals holding candidate families and certification p-values fixed show that screening-guided ordering reduces the diagnostic stopping penalty in Disconnected from 0.4700 to 0.1381 (gain +0.3319, 95% CI [+0.3214, +0.3426]); the ordering benefit is +0.6399 in Irregular. The oracle-based no-stopping counterfactual is diagnostic, not a deployable FWER-controlled procedure.

In Narrow-corridor, 94% of trials yield empty certificates, producing an all-trial safe-oracle gap of 0.8846. This exposes finite-sample certification power as a limitation in constrained regimes.

Overall, FS-Cert-U improves utility over the evaluated equally informed alternatives, while the decomposition identifies the remaining losses and clarifies the scope of its theoretical guarantees.

Reviewer_fEcMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors consider the setting of two model routing, with a primary model and guardian model that both use thresholds in order to make safe and reliable predictions including the possibility of abstaining. The authors provide a metric, SCER, that measures the prediction error given that either model responds and that the system doesn't abstain. The authors then investigate how to use the primary and guardian model thresholds to manage SCER risk, whilst maintaining coverage. They highlight an issue that changing thresholds can change which model answers responds to a question which impacts the SCER risk in a non-monotonic manner. To begin to address this, they provide FS-Cert-U , a framework that certifies that the probability that a finite subset of candidate thresholds contains an unsafe model (a model with a higher than wanted SCER risk) is sufficiently low. They then can choose a model based on their utility formulation, balancing guardian model usage, within this certified set of candidate thresholds They also give a regret bound within the finite set chosen within their certification framework between the final and best overall model within the finite candidate set. They demonstrate the their framework across several QA datasets.

Critical strengths
  • Interesting problem setting when needing to deal with two models, an answerer and an additional model, that guards the answer's predictions, both involving thresholded decisions based on uncertainty and the problem of the non-monotonically changing SCER risk as threshold choices vary.
  • The novel introduction of the new metrics for evaluating system performance in this setting.
  • They introduce the FS-Cert-U framework that certifies that the probability that a subset of candidate of thresholds contains a model with a high SCER risk is below a given tolerance
Critical weaknesses
  • The paper is rather poorly written and overly difficult to read. The motivation for the problem is rather absent and intuition needs to be provided throughout the paper for all of the new contributions and the story needs reshaping in the abstract and introduction to motivate why a reader should care about this two model setup and give more practical applications of where one would see such as setup in practice.
  • Further to this, several terms and variables are not defined before their introduction in the text. Small clarifications on what each new variable is would be helpful to a reader.
  • Experiments are limited to QA which is a rather simple setup which neglects a more motivated settings such as LLM guardrails where certification and guarantees would be more meaningful. Evaluation on such applications would greatly strengthen the paper in my opinion.
  • Their certification framework requires 4 dataset splits. This is rather complex especially if data were limited. This is even more of a problem as the model sacrifices utility here over say the naive method in order to allow certification.
  • Over claiming in several areas. For example in Section 3.3, the authors claim "FS-Cert-U operates empirically closer to the target than the more conservative Bonferroni and Holm baselines while retaining the same frozen candidate family. " This does not look like the case in Figures 3 and 4, with differences between methods looking insignificant. Claims therefore should be toned down in the text. This also poses more of a problem as the Bonferroni method for example, as far as I can tell, does not necessarily require as complicated machinery nor 4 held out splits, raising the question why given the lack of meaningful differences we would use the FS-Cert-U methodology over this simple baseline?
  • The paper uses as thresholded uncertainty score the sequence level entropy. This is a rather poor uncertainty metric for sequence level generation, not taking into account the semantics of sentences. This raises the question on how the framework handles different token-level uncertainty measures such as the log-likelihood of generations for example, and moreover if the framework works well for semantic entropy, a more natural uncertainty measure for QA especially on tasks like TriviaQA?
Questions for authors
  • What exactly are the advantages over a Bonferoni correction here given the lack of significant differences between this simple method and your more complex method?
  • How does this framework translate to situations where guarantees are more important i.e. in LLM guardrails where an inference cost-safety tradeoff is more motivated and less forced than in QA.
  • How does your framework handle different uncertainty scores, specifically semantic uncertainty.
5 comments

Response to Reviewer fEcM

We thank the reviewer for the constructive feedback. We have revised the manuscript to strengthen the practical motivation, clarify notation and intuition, and temper claims. We also added experiments on LLM guardrails, semantic uncertainty, and risk–cost sensitivity.

1. Practical Motivation: LLM Guardrails

We extend FS-Cert-U beyond QA to safety-sensitive LLM guardrails. On WildGuardTest, Qwen2.5-1.5B-Instruct serves as Primary and Qwen2.5-7B-Instruct as Guardian. Responses are evaluated by an automatic WildGuard judge; SCER is the harmful-response rate conditional on delivering a substantive answer.

Across 1,692 eligible prompts and 30 paired splits (α=0.10\alpha=0.10, δ=0.05\delta=0.05, λ=0.25\lambda=0.25):

UncertaintyMethodUtilityCoverageNonempty
EntropyFS-Cert-U0.11760.267618/30
EntropyBonferroni/Holm0.03580.119210/30
Normalized NLLFS-Cert-U0.10370.220414/30
Normalized NLLBonferroni/Holm0.00770.02442/30

FS-Cert-U gains +0.0818 utility with entropy and +0.0960 with NLL over both baselines; entropy-based coverage more than doubles. Mean held-out harmful-response rates among answering splits are 0.0606 (FS-Cert-U) and 0.0547 (baselines), both below the 0.10 target on average. This illustrates practical utility in safety-sensitive routing.

2. Semantic Entropy and Uncertainty Measures

FS-Cert-U is uncertainty-score agnostic: certification does not require calibrated error probabilities. On 3,000 TriviaQA questions with Qwen2.5-1.5B/7B, we compare token predictive entropy, length-normalized NLL, and semantic entropy (bidirectional NLI clustering of eight sampled responses).

Semantic entropy yields error-detection AUROC ≈0.85, yet no nonempty certificates at α=0.10\alpha=0.10. Thus, good error discrimination need not imply certifiable conditional risk. At the exploratory target α=0.30\alpha=0.30, semantic entropy yields nonempty FS-Cert-U certificates in 30/30 splits, with utility 0.2943 vs. 0.2053 for Bonferroni/Holm (+0.0889) and mean held-out SCER 0.2424. This shows compatibility with semantic uncertainty when the risk target is feasible.

3. Why FS-Cert-U Over Bonferroni?

The advantage is greater certification power under family-wise safety control. Bonferroni tests KK candidates at δ/K\delta/K, whereas independently ordered fixed-sequence testing can test each reached candidate at the full level δ\delta.

We examined six risk targets (α∈0.05,0.10,0.15,0.20,0.25,0.30\alpha\in\\{0.05,0.10,0.15,0.20,0.25,0.30\\}) and four cost weights (λ∈0,0.10,0.25,0.50\lambda\in\\{0,0.10,0.25,0.50\\}). Across 33 QA configurations × 50 matched splits, FS-Cert-U has positive mean utility gains over both Bonferroni and Holm in all 24 settings; pointwise configuration-bootstrap 95% CIs exclude zero in 23/24 settings for each baseline.

At λ=0.25\lambda=0.25, gains over Bonferroni are:

α\alpha0.050.100.150.200.250.30
Utility gain+0.0180+0.0493+0.0230+0.0292+0.0115+0.0081

The gains persist across risk targets and Guardian costs, rather than arising at only one operating point.

4. Data Splits and Deployment

FS-Cert-U does not require four data splits at deployment. Design and screening may be merged if candidate construction and ordering remain independent of certification data. Certification observations are reused for utility optimization; the test set is used only for experimental evaluation. Deployment therefore needs two functional calibration pools, not four.

Together, these new analyses strengthen the practical motivation, clarify where uncertainty measures help or fail, and demonstrate the empirical advantage over Bonferroni/Holm while preserving the intended scope of the safety guarantee.

Thank you for your responses and for the additional experiments which have helped to clarify most of my concerns. A few comments / questions I would like to make:

  • I still encourage the authors to refine an improve the writing and go over this carefully. It's still not the most clear paper and the writing makes it harder for a reader to get the clear contributions and takeaways that you would like the paper to convey. I would like to see this improved further in the final version.
  • For your method not requiring 4 data splits: I get your point but could you explain if and when combining the two design and screening splits would make sense with a practical realistic example. In your the experiments that you perform, can you do this merging to simplify your method? If not why?
  • What is your intuition on why Naive in Table 1 seems to beat your method consistently across most of the metrics apart from Guardian calls. This seems a like quite a bit of a tradeoff to use your certified tradeoff

I'm open to changing increasing my score, provided these final few things are answered and I appreciate the authors taking the time to address them!

Thank you very much for your thoughtful follow-up and continued engagement. We appreciate your recognition of our additional experiments and the opportunity to address your remaining questions.

1. Writing and Clarity

We appreciate this feedback. We have further revised the Abstract and Introduction to clarify the motivation, central contributions, certification-first principle, and scope of our theoretical guarantees. We would be happy to further refine the presentation if you have any additional concerns or suggestions.

2. Merging Design and Screening

Yes, merging Design and Screening is theoretically valid and practically feasible, and we have now evaluated this simplification on our full QA benchmark.

For example, a customer-support system could use the same historical labeled queries to construct candidate routing thresholds and prioritize them for certification. The essential requirement is that the candidate family and testing order are fixed independently of the certification data.

Experimental setup. We merged the original 10% Design and 20% Screening partitions into a shared 30% pre-certification pool, keeping the 35% Certification and 35% Test partitions unchanged. We reconstructed both the candidate families and testing orders using the merged pool, with the same risk target (α=0.10), significance level (δ=0.05), and Guardian-call cost (λ=0.25). We evaluated all 33 QA configurations across CommonsenseQA, TriviaQA, and ScienceQA, using 500 matched splits per configuration and direct abstention when no policy was certified.

DatasetSeparate UtilityMerged Utility
CommonsenseQA0.441350.44419
TriviaQA0.094250.09869
ScienceQA0.575750.57761
Overall0.370450.37350

The certified-set nonempty rate also increased from 91.53% to 92.48%.

These results show that the merged implementation preserves the same finite-sample safety guarantee and achieves slightly higher average utility in this benchmark. The Design and Screening stages can therefore use either separate or shared data, providing flexibility across deployment settings. The Test partition is needed only for experimental evaluation, not deployment.

3. Understanding the Naive Baseline

Thank you for highlighting this trade-off. The key distinction is that Naive and FS-Cert-U do not operate under the same safety requirement. Naive maximizes empirical utility under an empirical risk constraint, whereas FS-Cert-U selects only statistically certified policies.

Our original QA experiments already show the practical consequence. The frequency of test SCER exceeding the risk target was:

DatasetNaiveFS-Cert-U
CommonsenseQA12.76%1.76%
TriviaQA15.35%2.50%
ScienceQA33.14%2.79%

To further quantify this trade-off, we evaluated three synthetic populations with known true risks and utilities, representing different risk structures: Monotone, where risk changes monotonically across candidate policies; Irregular, where risk varies non-monotonically; and Disconnected, where safe policies form separated regions. These settings allow us to examine how risk structure affects utility and unsafe policy selection.

We used 1,000 paired trials per population, with 1,000 screening samples and 2,000 certification samples per trial. Both methods used the same candidate families and certification data.

PopulationNaive UtilityFS-Cert-U UtilityNaive Unsafe SelectionFS-Cert-U Unsafe Selection
Irregular0.93720.714948.5%0.2%
Disconnected0.89950.360063.7%1.0%
Monotone0.64720.64390.5%0.0%

These results explain the trade-off: Naive can achieve higher utility by selecting policies that frequently violate the population risk constraint, particularly in irregular and disconnected risk settings. In contrast, FS-Cert-U sacrifices some utility to provide statistically valid safety protection. When unsafe selection is rare for both methods, as in the Monotone setting, their utilities are nearly identical.

Importantly, our objective is not to maximize utility without a safety guarantee, but to optimize utility among statistically certified policies. This is why Bonferroni and Holm are the more appropriate safety-matched baselines for evaluating FS-Cert-U.

Reviewer_3hZSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes FS-Cert-U for risk certification and utility optimization in two-stage model routing. The method uses independent data splits to construct candidate policies and determine the testing order, then applies exact binomial tests and fixed-sequence testing to obtain a certified set, and finally selects a policy within the set that balances answer coverage and Guardian call cost.

Critical strengths
  1. The motivation makes sense, controlling risk in non-monotonic two-stage routing is a practical problem.

  2. The method provides finite-sample set-level risk guarantees, and analyzes the effects of ordering, certification sample size, and call cost across multiple datasets, model pairs, and repeated splits.

Critical weaknesses
  1. The novelty is limited. The core pipeline in this paper heavily overlaps with existing designs in Learn-then-Test [1]. The utility regret bound follows directly from standard Hoeffding's inequality and a union bound.
  2. Poorly written. The manuscript, captions, and Appendix repeatedly emphasize the limitations of the proofs and experiments. But who asked these questions? The authors should more clearly tell readers what this paper actually adds.
  3. The main experiments primarily compare against Bonferroni and Holm, while omitting methods like BalanceRAG / SGT [2] that also provide set-level risk certification for cascaded systems. In Appendix E.8, LEC-Routing achieves higher coverage and utility across all three datasets on the representative pair averages.
  4. Appendix E.1 specifies that when the certified set is empty (Acert=∅\mathcal{A}_{\text{cert}} = \emptyset), the fallback policy still invokes the Guardian on every query before abstaining, incurring a utility of −λ-\lambda. However, certification failure is already known prior to deployment, so the system can abstain directly without invoking the Guardian. This accounting convention may unfairly penalize baselines that return empty certified sets more frequently.

[1] Learn then test: Calibrating predictive algorithms to achieve risk control [2] BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generation

Questions for authors

see weaknesses

3 comments

Response to Reviewer 3hZS

We thank the reviewer for recognizing the practical motivation and finite-sample guarantees of FS-Cert-U. We have revised the Abstract, Introduction, figure captions, and Appendix to clarify our contributions and temper claims. We also added matched-budget experiments addressing the empirical concerns.

1. Methodological Contribution Beyond LTT

FS-Cert-U provides an integrated certification-to-deployment framework for non-monotonic, selection-conditioned routing risk. LTT and Hoeffding bounds are general statistical tools; they do not by themselves specify how to certify a routing-policy set and then optimize cost-sensitive utility using the same certification data. Our framework combines:

  • Exact SCER certification: Conditional binomial tests account for policy-dependent random answer counts without assuming risk monotonicity.
  • Simultaneous safety: Independently determined candidate prioritization and fixed-sequence testing provide family-wise certification of a policy set despite dependence among tests (Theorem 1).
  • Safe utility selection: Cost-sensitive selection from the certified set reuses certification data without additional safety multiplicity correction, with finite-sample within-set regret guarantees (Theorem 2).

Our contribution is the joint safety–utility guarantee for the complete routing-to-deployment procedure, not a new concentration inequality.

2. Comparisons with Risk-Certification Methods

We expanded comparisons to BalanceRAG-SGT (DWD/UGD) and Pareto Testing (PT2). All methods share frozen model outputs, the initial 10×10 threshold grid, the total calibration budget, and held-out evaluations across 33 QA configurations × 30 paired splits (α=0.10\alpha=0.10, δ=0.05\delta=0.05, λ=0.25\lambda=0.25). We evaluate both cost-bearing fallback (U=−λU=-\lambda) and zero-cost direct abstention (U=0U=0) on matched supplementary runs.

Mean utility gain (FS-Cert-U minus baseline):

BaselineCost-bearing fallbackDirect abstention95% CI (direct)
BalanceRAG-SGT-DWD+0.0571+0.0306[+0.0191, +0.0426]
BalanceRAG-SGT-UGD+0.0587+0.0322[+0.0204, +0.0443]
Bonferroni+0.0567+0.0458[+0.0218, +0.0771]
Holm+0.0541+0.0432[+0.0189, +0.0749]
Pareto Testing (PT2)+0.0087+0.0082[+0.0055, +0.0114]

CIs are pointwise configuration-level bootstrap intervals.

FS-Cert-U has positive mean utility gains over all evaluated baselines under both fallback conventions, and every reported direct-abstention CI excludes zero. Against PT2, FS-Cert-U achieves higher mean utility in 28/33 configurations, with five ties and no losses.

Regarding LEC-Routing, its higher empirical coverage and utility on the reported representative pairs reflect a different objective and guarantee. LEC targets risk-constrained retention with marginal-risk control. In contrast, FS-Cert-U simultaneously certifies a policy set with family-wise safety control for downstream cost-sensitive selection, with bounded within-set utility regret. These methods therefore address different certification objectives.

3. Fallback Accounting and Robustness

The advantages persist when empty certificates use zero-cost direct abstention rather than cost-bearing fallback. Under direct abstention, empty certificates make no Guardian calls. As the matched results show, FS-Cert-U maintains positive mean utility advantages against every compared baseline, with all reported direct-abstention confidence intervals excluding zero.

In summary, the new comparisons extend beyond standard multiplicity corrections, include closely related risk-certification methods, and show that the utility gains are robust to fallback accounting. They support the practical value of our certification-first routing design while clarifying its distinct theoretical guarantees.

Thank you for the rebuttal, most of my concerns have been addressed. I am willing to raise my score. However, I still think the presentation need to be strengthened.

Thank you very much for your positive feedback and for acknowledging our additional experiments and clarifications. We greatly appreciate your willingness to reconsider the score.

We also take your comments on the presentation very seriously. We have further revised the manuscript to improve its clarity, organization, and overall readability, and have submitted the updated version.

We would be grateful if you could review these revisions, and we sincerely welcome any further suggestions for improvement.

Reviewer_fiXJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the LLM routing problem, where prompts are routed to either a cheap model (primary model) or escalated to a guardian model (expensive), or else abstained from answering. There are two objectives in this problem: utility and safety. That is, the error rate of the answer should be lower (safety), with fewer calls to the guardian model (utility).

The authors propose FS-Cert-U, which is basically a threshold-based routing algorithm. Usually, this type of algorithm maintains 2 thresholds: the error rate of the primary model (τP\tau_{P}) and the error rate of the guardian model (τG\tau_G) in increasing order. Prompts with an error rate lower than τP\tau_{P} routed to primary model, less than τG\tau_G are routed to the guardian model and rests are abstained from answering. Utility and safety have a non-monotonic relationship with these thresholds.

The proposed algorithm works in 4 steps. The data set is divided into 4 sets. The design dataset constructs a finite family of policies; the screening dataset fixes a permutation of policies; the certification dataset certifies the best policy in the permutation; finally, the test dataset evaluates the performance.

Experimental results are given comparing the proposed method with Bonferroni correction and Holm correction across different datasets.

Critical strengths

For the most part, it is easy to follow. Balanced theory and experiments.

Critical weaknesses
  1. This method cannot be used in a test-time routing environment. Most of the work in the literature treats real-time routing as active learning.

  2. It is simply a classification problem (since it is not real-time routing); hence, the complexities introduced in the paper seem to be unnecessary.

  3. The solution seems overly complicated, yet incremental.

  4. Experiments do not compare it with any of the related state-of-the-art routing algorithms, only with Bonferroni correction and Holm correction.

  5. The difference between the design step and the screen step is not clearly discussed in the main section.

  6. Theoretical results are not compared with the existing methods.

I think this paper needs a significant work to be done to compare the theoretical and empirical results with other algorithms in the literature.

Questions for authors
  1. I do not really understand what the difference design and screen step ?

  2. How do the theoretical results compare with a normal threshold classification or any other existing methods?

I might adjust the score based on the answers.

4 comments

Response to Reviewer fiXJ

We thank the reviewer for the constructive feedback. We clarify real-time applicability, the design/screening distinction, and theoretical contributions. We also report matched-budget comparisons with related risk-certification methods.

1. Real-Time Routing vs. Active Learning

FS-Cert-U supports real-time, per-query routing. Risk certification runs once offline; the deployed thresholds then route each query to the Primary model, the Guardian, or abstention. No test-time labels, online updates, or repeated certification are needed.

The problem differs from ordinary classification. Changing the Primary threshold can replace a Guardian answer with a Primary answer without changing coverage, so system-level selection-conditioned error (SCER) need not be monotonic. Empirical threshold tuning alone does not guarantee finite-sample SCER control after policy selection.

2. Design vs. Screening

  • Design determines what to test: Construct a finite threshold-policy family from uncertainty-score quantiles and remove redundant candidates.
  • Screening determines the test order: Rank this frozen family using independent data to improve fixed-sequence certification power.

We have clarified these distinct roles in the revised manuscript.

3. Theoretical Comparison with Existing Methods

FS-Cert-U provides finite-sample safety and utility guarantees for two-stage routing, beyond empirical threshold tuning or fixed-policy calibration.

  • Vs. threshold classification: Selection over threshold pairs induces statistical selection bias. FS-Cert-U certifies system-level conditional risk qθ=E[Zθ]/E[Sθ]q_\theta=\mathbb{E}[Z_\theta]/\mathbb{E}[S_\theta] using exact conditional binomial tests that handle random, policy-dependent answer counts without assuming monotonicity.
  • Vs. LTT/multiple testing: LTT supplies general testing tools, but not routing-specific conditional-risk tests or a downstream utility-selection guarantee. Independently ordered fixed-sequence testing gives family-wise safety for the entire certified set despite dependent tests (Theorem 1). Unlike Bonferroni's δ/K\delta/K, each reached candidate is tested at level δ\delta.
  • Vs. risk-certified routing: Pareto Testing uses empirical frontier screening; BalanceRAG-SGT uses graphical significance propagation; LEC-Routing targets marginal-risk-controlled retention. FS-Cert-U instead combines a frozen routing-policy family, independent prioritization, and simultaneous set-level safety for cost-sensitive selection.

Theorems 1–2 cover the final data-dependent deployment decision. With probability at least 1−δ−η1-\delta-\eta, FS-Cert-U either fails closed (empty certified set) or deploys an α\alpha-safe policy whose utility regret relative to the best policy within the certified set is at most

2(1+λ)log⁡(2K/η)2m.2(1+\lambda)\sqrt{\frac{\log(2K/\eta)}{2m}}.

Certification data can be reused for utility selection without additional safety multiplicity correction. This regret bound does not claim optimality against all population-safe policies.

4. Matched-Budget Comparisons

We additionally evaluated BalanceRAG-SGT (DWD/UGD) and Pareto Testing (PT2) on the same two-stage routing task. Methods share frozen model outputs, an initial 10×1010\times10 threshold grid, equal total calibration budgets, and held-out evaluations across 33 QA configurations × 30 paired splits (α=0.10\alpha=0.10, δ=0.05\delta=0.05, λ=0.25\lambda=0.25). Empty certificates use zero-cost direct abstention.

Mean utility gain (FS-Cert-U minus baseline):

BaselineUtility gain95% bootstrap CI
BalanceRAG-SGT-DWD+0.0306[+0.0191, +0.0426]
BalanceRAG-SGT-UGD+0.0322[+0.0204, +0.0443]
Pareto Testing (PT2)+0.0082[+0.0055, +0.0114]

CIs are pointwise configuration-level bootstrap intervals.

FS-Cert-U has higher mean utility than all three baselines, with all reported intervals excluding zero. Against PT2, it has 28 wins, 5 ties, and 0 losses across 33 configurations.

Together, these results support the value of FS-Cert-U for real-time routing: non-monotonic SCER certification, simultaneous certified-set safety, and cost-sensitive deployment with bounded within-set utility regret.

Thank you for the revisions and clarifications.

  1. Regarding the first weakness, the model has to be trained offline and deployed online right ? It is not a test-time adaptive algorithm. In the literature some works formulates routing as partial monitoring or active learning where, they adapt to the routing rate based on uncertainty, or guardian model agreement on the test time itself.

  2. Even if it not monotonic can we just use a simple neural network for the classification ?

  3. I do not see a regret bound comparison with other baselines if there are any ? If not are there any matching lower bound ? How can we see how could a bound is without a baseline or lower bound to compare ?

I am not still convinced on the significance of this work. Furthermore lack of theoretical and empricial comparison is still my concern. Therefore I keep the score as it is. That is being said, I do not have direct expertise in this field. I might be missing something too. However this is my genuine final thoughts on this paper.

Clarification: Online Adaptation, Neural Routing, and Regret Comparisons

We thank the reviewer for the thoughtful follow-up. We appreciate your acknowledgment that this research area is outside your primary expertise. Nevertheless, we would like to clarify several important points about our theoretical and empirical contributions.

1. Offline certification versus test-time adaptive routing

Yes, the reviewer is correct. FS-Cert-U is an offline-calibrated, online-deployed framework, not a test-time adaptive learning algorithm. The LLMs are fixed, and thresholds are calibrated and certified before deployment. At inference, decisions depend on query uncertainty scores, without online parameter updates.

Partial-monitoring, contextual-bandit, and active-learning approaches instead learn or adapt policies from sequential feedback, often targeting cumulative regret. Our objective is providing finite-sample conditional-risk guarantees for the policy selected before deployment.

We validated deployment through sequential routing replay on 70,972 held-out requests across all 33 configurations. FS-Cert-U achieved 100% agreement with batch routing decisions, without test-time correctness labels or online recalibration. CPU routing-decision overhead was 0.351 μs (p50) and 0.412 μs (p95), excluding model inference. This demonstrates efficient per-query deployment with fixed, offline-certified thresholds.

2. Can a neural network replace the threshold-based router?

Neural networks can certainly make routing decisions. However, this does not diminish the significance of threshold-based routing and its statistical guarantees.

Indeed, influential prior work in selective prediction and risk control has demonstrated the importance of threshold-based decision rules and post-hoc calibration without requiring a separately trained neural routing classifier. For example, Geifman and El-Yaniv (NeurIPS 2017) developed confidence-threshold-based selective classification with high-probability risk guarantees; Conformal Risk Control (ICLR 2024) and Learn-then-Test (2025) established general statistical calibration frameworks for pretrained predictors; and, more directly related to our setting, LEC-Routing (ICML 2026) investigated calibrated threshold policies for two-model routing with selection-conditioned risk control.

These works illustrate an important distinction: the expressiveness of the decision function and the statistical reliability of the resulting decisions are separate research questions. The availability of neural classifiers does not make threshold-based calibration or risk certification unnecessary.

Our work addresses this question in two-stage routing. Even with simple uncertainty thresholds, maximizing utility does not automatically guarantee a prescribed population-level SCER constraint.

Our experiments show why two-stage routing merits explicit risk certification. We analyzed the Primary–Guardian threshold grids across 33 routing configurations (11 model pairs each on CommonsenseQA, TriviaQA, and ScienceQA). Among these, 29/33 configurations exhibit SCER direction reversals along a threshold axis, while 30/33 contain adjacent policies where coverage remains unchanged but SCER increases. These findings show that non-monotonic risk is widespread across the evaluated model pairs, not isolated examples.

We further compared FS-Cert-U against simpler routing and certification alternatives using known finite populations constructed from real QA model-output caches. We conducted 1,000 independent calibration trials, each drawing 300 Design, 600 Screening, and 1,000 Certification samples, with (α=0.10), (δ=0.05), and (λ=0.25). All methods were evaluated against the same finite-population utility objective. FS-Cert-U achieved a mean utility of 0.3514, substantially outperforming single-policy Holdout (0.1426), Primary-only certification (0.2370), Guardian-only certification (0.2654), and Single-cascade (0.1037).

Importantly, these experiments support the practical value of certifying and selecting among two-stage routing policies, rather than claiming that threshold-based routers universally outperform neural classifiers. Our contribution lies in providing finite-sample risk certification and cost-sensitive utility selection, which remain meaningful regardless of whether the underlying routing policies are threshold-based or neural.

Therefore, our contribution is not predicated on threshold rules being more expressive than neural networks. Rather, it lies in providing a statistically principled and practically effective certification-to-deployment framework for non-monotonic, cost-sensitive routing—a research objective that remains meaningful regardless of the underlying policy architecture.

3. Regret-Bound Comparisons and the Role of Certification

We thank the reviewer. We distinguish within-set selection regret from certification-induced utility loss using theory and safe-oracle experiments.

A. Theoretical Comparison with Existing Methods

FS-Cert-U combines finite-sample risk certification and cost-sensitive policy selection in non-monotonic two-stage routing. Its contribution is a certification-to-deployment framework with joint safety–utility guarantees, not a new testing inequality.

  • Vs. neural/threshold routing: Learning a router does not certify population-level conditional error. Exact conditional-binomial SCER tests handle random, policy-dependent answer counts without monotonicity. The same principle accommodates independently trained, frozen neural policies.

  • Vs. LTT/multiple testing: LTT supplies general testing tools. FS-Cert-U applies them to non-monotonic routing with downstream utility selection. An order fixed independently of certification data enables fixed-sequence family-wise control despite dependent tests, testing each reached candidate at level δ\delta rather than Bonferroni's δ/K\delta/K. No additional safety correction is needed for selection within the certified set.

  • Vs. risk-certified routing: Pareto Testing combines empirical screening and fixed-sequence testing; BalanceRAG uses graphical significance propagation; LEC studies selection-conditioned risk in cascades. FS-Cert-U certifies frozen Primary–Guardian–Abstain policies and selects by explicit Guardian-call utility.

Regret guarantee. For KK frozen candidates and mm certification observations, Theorem 2 guarantees, with probability at least 1−η1-\eta, whenever Acert≠∅\mathcal A _{\mathrm{cert}}\neq\varnothing:

0≤max⁡θ∈AcertUλ(θ)−Uλ(θ^)≤2(1+λ)log⁡(2K/η)2m.0\leq \max _{\theta\in\mathcal A _{\mathrm{cert}}} U _{\lambda}(\theta)-U _{\lambda}(\hat\theta)\leq 2(1+\lambda)\sqrt{\frac{\log(2K/\eta)}{2m}}.

With Theorem 1, this yields joint safety–utility control with probability at least 1−δ−η1-\delta-\eta. Bonferroni and Holm have the same form of within-set regret bound with the same frozen family and empirical selector. Our advantage is retaining more useful certified policies under simultaneous safety, not a faster generic regret rate. We claim no matching minimax lower bound.

B. Beyond Within-Set Regret: Best Safe Policy

We decompose overall utility loss using three population values:

  • U_safe: Best utility among population-safe candidates in the frozen family.
  • U_cert: Best utility in the certified set.
  • U_deployed: Utility of the deployed policy.

When the certified set is nonempty and contains only safe policies:

U_safe − U_deployed = (U_safe − U_cert) + (U_cert − U_deployed).

U_safe − U_cert is the certification gap, reflecting excluded safe policies due to limited power, ordering, or stopping; Theorem 2 does not bound it. U_cert − U_deployed is the selection regret, bounded by Theorem 2. Thus, identical regret rates can yield different deployed utilities because different safe policies survive certification.

C. Empirical Evaluation Against a Common Safe Oracle

Using known finite populations from real QA model-output caches, we ran 1,000 independent calibration trials, each with 300 Design, 600 Screening, and 1,000 Certification observations (α=0.10\alpha=0.10, δ=0.05\delta=0.05, λ=0.25\lambda=0.25). The highest-utility population-safe candidate in each frozen family defines an exactly evaluable oracle.

MethodMean utility gap to safe oracle
FS-Cert-U0.1212
Bonferroni0.1749
Holm0.1723

Gaps are averaged over trials with at least one safe candidate and include certification and selection effects. FS-Cert-U reduces the mean gap by 30.7% vs. Bonferroni and 29.7% vs. Holm, highlighting the value of certification.

In the near-tie experiment (Figure 8), mean population selection regret decreases from 0.003465 (m=280m=280) to 0.0001525 (m=4480m=4480), consistent with Theorem 2.

We also tested 33 QA configurations, four risk targets, and three calibration budgets with 50 paired splits per setting. FS-Cert-U has higher aggregate mean utility than Bonferroni, Holm, and both BalanceRAG-SGT variants across all 12 risk–budget settings. At α=0.10\alpha=0.10 with full calibration budget, gains are +0.0801 over Bonferroni, +0.0729 over SGT-DWD, and +0.0052 over Pareto Testing (PT2).

Overall, FS-Cert-U combines simultaneous finite-sample SCER safety with cost-sensitive selection that safely reuses certification data. The safe-oracle and matched-budget results show higher retained deployment utility despite the same order of generic within-set regret bound.

Reviewer_zApWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Basic flow proposed in the paper is that a primary model answers the query when it is confident. Otherwise the query routes to a stronger Guardian model. Both Primary and Guardian can answer the query if uncertainty is below a set threshold, otherwise abstain. Two thresholds (τP, τG) set the policy. The goal is to certify the error rate among answered queries (SCER), which need not be monotone in either threshold, and to search over policies without invalidating the guarantee. Using the same evidence/data to certify as well as search or optimize over threshold pairs can invalidate fixed-policy guarantees. This motivates the main contribution of the paper - a four-stage design that first determines a finite policy family, the testing order, and finds a certified set; and then finally optimizes utility within the certified set. The experiments show that this method achieves higher utility than alternative methods it is compared against.

Critical strengths

The paper seems to be sound in formulation and approach. The experiments are reasonably well-constructed to confirm that FS-Cert-U operates within the intended target risk (held-out) and compare risk vs. coverage and utility comparisons with baselines and test across major QA benchmarks. FS-Cert-U does well at the primary operating point - (from Table 1) it seems that FS-Cert-U compared to Bonferroni/Holm primarily achieves higher coverage (~2-10pp) with fewer Guardian calls, although at a slightly higher SCER%.

Critical weaknesses

The empirical gains versus the baselines don’t seem to be significant across different alpha. From figure 4 it seems that the gains in utility at a fixed alpha are minimal other than the chosen operating point of alpha=0.10. Also it is not clear why lambda =0.25 was chosen for the utility.

Also it seems like much of the construction in paper was already present in Learn-then-Test and Laufer-Goldshtein et al., Efficiently Controlling Multiple Risks with Pareto Testing (ICLR 2023).

Questions for authors

None

1 comment

Response to Reviewer zApW

We thank the reviewer for recognizing the soundness of FS-Cert-U and its coverage–cost advantages. To address the concerns, we conducted joint risk–cost sensitivity analyses and new matched-budget comparisons with Pareto Testing and related risk-certification methods.

1. Utility Gains Across Risk Targets and Cost Weights

We evaluated six risk targets (α∈0.05,0.10,0.15,0.20,0.25,0.30\alpha\in\\{0.05,0.10,0.15,0.20,0.25,0.30\\}) and four Guardian-cost weights (λ∈0,0.10,0.25,0.50\lambda\in\\{0,0.10,0.25,0.50\\}), extending beyond the primary operating point α=0.10\alpha=0.10.

Across 33 QA configurations × 50 matched splits, FS-Cert-U achieves positive mean utility gains over both Bonferroni and Holm in all 24 settings. Pointwise configuration-bootstrap 95% confidence intervals exclude zero in 23/24 settings for each baseline.

At λ=0.25\lambda=0.25, the gains over Bonferroni are:

Risk target α\alpha0.050.100.150.200.250.30
Utility gain+0.0180+0.0493+0.0230+0.0292+0.0115+0.0081

Thus, the utility advantage persists across risk targets and cost preferences, rather than being restricted to one operating point.

2. Choice of Guardian-Cost Weight

λ=0.25\lambda=0.25 is an illustrative operational trade-off between answer coverage and Guardian-call cost, not a theoretically optimal setting. The family-wise safety guarantee (Theorem 1) does not depend on λ\lambda; the utility-regret bound (Theorem 2) explicitly accounts for its influence. Our sensitivity experiments confirm that the empirical gains persist across the evaluated cost weights.

3. Contribution Beyond LTT and Pareto Testing

FS-Cert-U provides a routing-specific certification-to-deployment framework with joint finite-sample safety–utility guarantees. LTT and Hoeffding bounds provide general testing tools, but do not by themselves specify how to obtain a family-wise safe set of routing policies and perform cost-sensitive selection using the same certification data.

FS-Cert-U independently prioritizes a frozen policy family, certifies candidates using exact conditional binomial SCER tests and fixed-sequence testing, and selects utility within the certified set. Theorems 1–2 jointly guarantee family-wise safety and bounded within-set utility regret, including when certification data are reused for utility selection. This differs from Pareto Testing's frontier-oriented candidate screening.

We additionally compared BalanceRAG-SGT (DWD/UGD), Pareto Testing (PT2), Bonferroni, and Holm under identical frozen model outputs, candidate grids, calibration budgets, and held-out evaluations across 33 QA configurations × 30 paired splits (α=0.10\alpha=0.10, δ=0.05\delta=0.05, λ=0.25\lambda=0.25). Empty certificates use zero-cost direct abstention.

Mean utility gain (FS-Cert-U minus baseline):

BaselineUtility gain95% bootstrap CI
BalanceRAG-SGT-DWD+0.0306[+0.0191, +0.0426]
BalanceRAG-SGT-UGD+0.0322[+0.0204, +0.0443]
Bonferroni+0.0458[+0.0218, +0.0771]
Holm+0.0432[+0.0189, +0.0749]
Pareto Testing (PT2)+0.0082[+0.0055, +0.0114]

CIs are pointwise configuration-level bootstrap intervals.

FS-Cert-U achieves higher mean utility than all five baselines, with all reported CIs excluding zero. Against PT2, it records 28 wins, 5 ties, and 0 losses across 33 configurations.

Together, the theoretical and empirical results support FS-Cert-U's practical value across risk–cost settings and relative to both conventional and related certification approaches.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.