acceptodds
Under review as a conference paper at ICLR 2027

Recalibrated Routing for LLM Judges: Optimizing Classification Metrics On A Budget

Abstract

Routing is important in LLM-as-a-judge research because it combines models with different comparative strengths in a cost-effective manner. In practice, performance of such routers is often measured using interpretable classification metrics such as precision, recall, and score; however, existing methods typically optimize objectives in the form of average proxy quality that need not align with these deployment metrics. In this paper, we develop a budget-aware framework that directly optimizes the chosen metric. In particular, we use a quasi-linear representation to formulate routing as a mixed-integer linear program with nuisance parameters. Furthermore, we apply a post-hoc calibration that guarantees to improve on the raw AI outputs with high probability. For theoretical guarantees, we characterize the population-optimal router and recalibration cutoff and derive regret bounds for our metrics in terms of nuisance regression estimation error. We apply the method on a real-world safety violation detection problem at a tech company and show that our routing method outperforms existing methods in the literature.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI routing, recalibration, classification metrics, statistical regret bounds

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PG2QOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ReCal-Route, which selects LLM judges under a cost budget and adjusts their predictions using confidence scores and other available information to improve classification metrics. It provides theoretical guarantees and shows empirical improvements on sentiment classification and safety screening.

Critical strengths

Strength #1: well-motivated practical formulation: the paper jointly optimizes judge selection and classification performance under a budget. such a setting is very useful and relevant in real-world deployment of LLMs. the post-call recalibration uses the returned prediction to improve decisions without additional model calls, which is also a tremendous practical advantage

Strength #2: substantial theoretical depth: the regret bounds accomodate imperfect probability estimates with aligned decisions, analysis connecting estimation error, margins, and routing regret, and matching lower bounds.

Critical weaknesses

Weakness #1: Insufficient comparisons against recalibrated baselines: the results show the effectiveness of the complete pipeline, but it's unclear how much joint routing adds beyond supervised post-processing. See questions below for ablations that could be run.

Questions for authors
  1. How much does the adaptive routing add beyond recalibration?
  2. What about when competing routers receive same post-processing?
  3. How well can a model predict the label using only the information available before calling a judge? and how much does adding the judges’ answers and confidence improve the results?
4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer PG2Q

Thank you so much for your helpful comments and support for our work.

1. How much does the adaptive routing add beyond recalibration?

We compare learned routing, where there are 8 context-based groups, each with its own call plan, against no routing (one call plan for all items), using either recalibrated or native decisions. Results are reported in the table below. We find that routing improves recalibrated decisions in two settings: with tag context, it adds +1.32pp on render + meshgrid and +0.93pp on multimodal; the other four settings show essentially no value change. On the other hand, without recalibration, routing improves performance in all six settings, adding +0.49 to +3.05pp, with every 95% interval above zero.

In this following table, for each context/panel, Routing, recalibrated means the difference in mean F1 (learned routing - no routing) where both policies have post-call recalibration. Routing, native means the difference in mean F1 (learned routing - no routing) where both policies do not have post-call recalibration. Lastly, Recalibration, learned routing means the difference in mean F1 between recalibrated and unrecalibrated policies, with learned routing in both.

Context, panelRouting, recalibratedRouting, nativeRecalibration, learned routing
Tags, render only−0.02 [−0.34, +0.30]+2.99 [+2.66, +3.31]+3.51 [+3.08, +3.94]
Tags, render + meshgrid+1.32 [+1.07, +1.58]+2.43 [+2.14, +2.73]+1.82 [+1.46, +2.19]
Tags, multimodal+0.93 [+0.73, +1.13]+3.05 [+2.70, +3.39]+1.04 [+0.78, +1.30]
CLIP, render only−0.13 [−0.42, +0.15]+2.70 [+2.40, +3.01]+4.13 [+3.65, +4.62]
CLIP, render + meshgrid−0.25 [−0.51, +0.02]+0.90 [+0.66, +1.14]+2.73 [+2.40, +3.06]
CLIP, multimodal−0.14 [−0.50, +0.21]+0.49 [+0.11, +0.87]+3.80 [+3.40, +4.19]

We will add this ablation study to the revised submission. The main takeaway is that post-call recalibration is responsible for most of the improvement in mean F1. When the context variable is sparse and interpretable, such as a binary vector of semantic tags, routing optimization can drive the improvement further in presence of recalibration as well.

AuthorsOpenReview ↗

Rebuttal for Reviewer PG2Q (continued)

2. What about when competing routers receive same post-processing?

Thank you for this insightful question. As requested, we give the competing methods post-call calibration on both the IMDb and asset datasets. In particular, we fix each competitor's routing/cascading policy and apply our recalibration probability model on top as a final layer. For each competing method, this recalibration model is trained beforehand using cross-fitting; furthermore, the best recalibration model and regularization penalty are chosen using training F1 score. We note that in all methods, recalibration does not change any call plans, so the costs remain the same.

3D-generated asset data: B=2

Below, we report mean held-out F1 (%) at B=2 over 100 random splits of the data with 70% training data. Here, + R denotes post-call recalibration, and we bold the best F1 result in each column. Recall that there are two context types: (i) binary vector of semantic tags indicating different properties of an item, and (ii) generic CLIP embeddings.

MethodTags: renderTags: render–mesh gridTags: multimodalCLIP: renderCLIP: render–mesh gridCLIP: multimodal
ReCal-Route85.1985.6377.1585.4784.9976.95
RouterBench-kNN + R83.4582.5970.8985.4085.0077.87
Avengers-Pro + R85.0683.8574.3085.8484.9177.13
Unified Router + R83.8283.1175.5685.5784.4078.37
WISERouter + R84.3085.3875.3186.1384.9576.79
Depth-2 cascade + R82.8781.9972.6386.3285.1978.33

We have the following main takeaways:

Utility of post-call recalibration. Recalibration improves 29 of the 30 competitor/context/input comparisons at B=2, often substantially: RouterBench-kNN rises from 75.06 to 83.45 with render-only tags, while Cascade rises from 80.58 to 86.32 with render-only CLIP. These gains demonstrate that our recalibration component can and should be applied broadly beyond our own router.

Context matters. ReCal-Route still achieves the highest mean F1 in all three semantic-tag settings at B=2. Sparse, discrete tags may make the relevant regional differences in judge reliability particularly suitable for our group-based routing. It could suggest that in isolation, our routing method may be more suitable for interpretable context. On the other hand, recalibrated competitors are stronger with dense CLIP embeddings, making better routing representations for generic embeddings a meaningful direction for future work. We have revised our appendix section to include this table, as well as full native/recalibrated budget curves.

IMDb: comparison at B=6

For completeness, we also extend the experiment above to the IMDb dataset, using 40 random data splits. Below, the unit for budgets is nominal cost unit per 10,000 reviews, and the arrows show mean F1 results from native to recalibrated competing methods.

Firstly, the results reinforce our previous point about post-call recalibration: recalibration improves all 18 competitor/context means. Secondly, regarding relative performance, ReCal-Route is first with tags and essentially tied in mean performance with the strongest recalibrated competitor in the other two contexts: its deficits to FrugalML are only 0.084 percentage points with LSA and 0.039 with combined context. It exceeds each of the other five recalibrated competitors in all three contexts. This supports the competitiveness of our joint routing-and-recalibration procedure after competing routers receive the same recalibration model class.

MethodTagsLSACombined
ReCal-Route87.6190.6390.67
RouterBench-kNN84.56 → 84.6087.51 → 90.1386.72 → 90.01
Avengers-Pro86.79 → 87.0387.54 → 90.4786.92 → 90.40
Unified Router86.87 → 86.9388.83 → 90.4888.89 → 90.46
WISERouter86.82 → 86.8787.30 → 90.4986.87 → 90.41
Depth-2 cascade87.15 → 87.3586.61 → 90.5486.64 → 90.54
FrugalML84.91 → 87.5884.91 → 90.7284.91 → 90.71
AuthorsOpenReview ↗

Rebuttal for Reviewer PG2Q (continued)

Stress test: restricting the quality–cost action library

To stress test routing performance when high-return actions are scarce and offer poorer F1 value, we retain/filter the 10, 20, or 30 call plans with the lowest development F1-to-cost ratios from the original 66-plan library. The retained collection of actions is fixed across budgets and shared by all considered methods.

We evaluate all three library sizes at B = 5.25, 5.5, 5.75, and 6, using 40 random splits in each of six settings. Competitors retain their original accuracy/correctness objectives in the respective work. In this stress test, +R_acc also accuracy-selected post-call recalibration (whereas the main VLM/IMDb recalibration experiments above use an F1-selected recalibration model and threshold).

Remark: Constrained Cascade has to commit to a valid plan before calling a judge. Due to our experiment setup, we have to minimally modify the cascade, which originally called judges sequentially: due to call plan pruning, certain judges may not have a feasible judge to form a valid pair. We mitigate this by scoring the plans beforehand using the Cascade's linearized score combining accuracy and cost.

At B=6, the table compares ReCal-Route with the highest mean among recalibrated competitors having all 40 successful splits in that setting. Here, the reported values are again mean F1 (%), and we bold the larger mean value in each pair.

Retained plansMethodTags: renderTags: render–mesh gridTags: multimodalCLIP: renderCLIP: render–mesh gridCLIP: multimodal
10ReCal-Route81.3883.1277.6585.1785.3576.03
10Best +R_acc (40 splits)80.5981.1669.0784.8585.5376.85
20ReCal-Route83.9084.0577.4685.6185.1477.33
20Best +R_acc (40 splits)81.6982.4972.3085.3185.6477.57
30ReCal-Route84.3185.4277.4285.4984.9576.95
30Best +R_acc (40 splits)84.2785.3976.5486.2985.8178.47

Among complete 40-split comparisons at B=6, ReCal-Route performs the best for all three semantic-tag settings for every library size. With 10 or 20 plans it also leads CLIP render-only; its largest deficit to a full-coverage recalibrated competitor is 0.82 and 0.50 points, respectively. With 30 plans, it exceeds every native competitor in all six settings; after competitor recalibration, it trails on CLIP by 0.80–1.52 points. These results support competitive routing under action restrictions, with stronger performance for semantic context.

We also count how many splits/settings that a method successfully returns a policy. In the following table, counts include all four budgets, including the tighter cases: 960 split/settings per library size. ReCal-Route matches the highest successful-evaluation count with 10 and 30 plans. With 20 plans, Constrained Cascade completes 930 cases versus 899 for ReCal-Route.

Retained plansReCal-RoutekNN +R_accAvengers +R_accUnified +R_accWISE +R_accConstrained Cascade +R_acc
10833/960746/960827/960778/960108/960833/960
20899/960894/960898/960898/960130/960930/960
30960/960960/960960/960960/960391/960960/960
AuthorsOpenReview ↗

Rebuttal for Reviewer PG2Q (continued)

3. How well does pre-call information alone predict the label, and what do judge outputs add?

A classifier on context only (tags or CLIP), trained and tuned like our method, compared with the same classifier plus judge answers and confidence:

Context, panelContext only (F1)+ ReCal-Route at budget 2+ all six judges (cost 13.3)
Tags, render only79.7+5.5 [+5.0, +6.0]+6.1 [+5.7, +6.6]
Tags, render + meshgrid79.7+5.9 [+5.5, +6.4]+4.8 [+4.3, +5.3]
Tags, multimodal66.7+10.4 [+9.9, +11.0]+9.7 [+9.2, +10.3]
CLIP, render only84.1+1.4 [+1.0, +1.8]+2.6 [+2.2, +3.0]
CLIP, render + meshgrid84.1+0.9 [+0.5, +1.3]+0.6 [+0.2, +1.1]
CLIP, multimodal71.2+5.7 [+5.2, +6.3]+6.8 [+6.2, +7.3]

With CLIP embeddings on the render panels, judges add only 0.6 to 2.6pp. With coarse context or the harder multimodal task, judges add 5 to 10pp. Thus, applying our method to VLM-judges could substantially improve the performance.

Reviewer_86ZJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a method for selecting the LLM judge to use (among a candidate set) based on contextual information about the object being assessed, while accounting for a budget constraint. The goal is to balance LLM evaluation costs with judgment quality. Compared with existing work, the key innovation lies in aligning the training target with the actual performance metric of interest (e.g., F1 score stead of average prediction accuracy).

Critical strengths
  • I like the idea of focusing on the actual performance metric of interest, as things like F1 score is way more important than overall accuracy in some application contexts, e.g., content moderation. The problem is relevant to practice.
Critical weaknesses
  • The theoretical results are difficult to follow due to the excessive use of notation. The authors also provide little interpretation of the proved statements. For example, what do we learn from the bound in Theorem 1? Is it intended to provide insights into the fundamental problem structure, or to serve as a performance guarantee? If the latter, how tight is the bound?
  • The paper is based on the assumption that there is no distribution shift between the training and test sets, which is not representative of the real-world scenarios. Given that the primary contribution of this paper is a practical framework, I think it is important to make sure the empirical assessment aligns better with the practice.
Questions for authors

See above

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 86ZJ

1. The theoretical results are difficult to follow due to the excessive use of notation. The authors also provide little interpretation of the proved statements. For example, what do we learn from the bound in Theorem 1? Is it intended to provide insights into the fundamental problem structure, or to serve as a performance guarantee? If the latter, how tight is the bound?

Thank you for highlighting this issue. We will simplify the notation in the main text and add an intuitive interpretation right after Theorem 1.

Purpose of Theorem 1. Theorem 1 provides a statistical guarantee for our ReCal-Route algorithm. Under the stated assumptions, the learned policy costs less than the budget with high probability and, in the meantime, it achieves nearly optimal FβF_\beta within a pre-specified router class under a slightly smaller budget B−2bnB-2b_n, where the slack term 2bn2b_{n} accounts for estimation errors.

Interpretation. The two terms identify difference sources of performance loss. In particular, the term ene_n captures sampling uncertainty when comparing routers and decision thresholds, and the term an(1+α)/νa_n^{(1+\alpha)/\nu} measures how errors in estimated recalibration translate into loss in the target metric. Here, α\alpha controls how much probability lies near the optimal decision boundary (aka the strength of the margin condition), while ν\nu controls how strongly the limiting scores are compressed toward their threshold.

An implication of this result is that it is unnecessary to get consistent probability estimation everywhere. We show that even a misspecified model can attain the optimal decision approximately if an appropriate threshold recovers the oracle decision and the margin conditions hold.

Tightness. We did not claim that Theorem 1 is tight in every regime, but it does require fewer assumptions. On the other hand, Theorem 2 yields faster rates under the additional Assumption 4. For example, with a finite router class, a root-nn consistent score estimator, and α=ν=1\alpha=\nu=1, Theorem 1 yields an Op(n−1/2)O_p(n^{-1/2}) bound, whereas Theorem 2 yields an Op(n−2/3)O_p(n^{-2/3}) bound when router selection does not dominate. Theorem 3 then gives a matching minimax lower bound for the sharper rate.

We will revise the presentation to distinguish the roles of these theorems explicitly: Theorem 1 gives the general guarantee and error decomposition, Theorem 2 establishes when faster convergence is possible, and Theorem 3 provides a matching lower bound.

AuthorsOpenReview ↗

Rebuttal for Reviewer 86ZJ (continued)

2. Distribution shift

We evaluate the original ReCal-Route under two controlled forms of distribution shift, using all six context/panel settings, the full 66-plan library, and 40 splits at B=2B=2. A tag combination is the complete configuration of six binary semantic tags. For example, a combination could be “person + text”. We keep combinations with at least 20 examples and construct 70/30 development/test splits using tags and their frequencies.

  • Common support: every combination appears in both sets, but its frequency changes. Approximately 60% of examples have at least two active tags enter test; we then add sampled items with 00 or 11 active tag to the test set to reach the 30%30\% test data size allocation.
  • New combinations: two combinations in the dataset, together representing 10–20% of the eligible population, are excluded from development and placed entirely in test. Each individual tag value remains represented in development. The test set includes both seen and unseen combinations.

The above sampling procedure applies to both the semantic tag and CLIP embedding settings.

Reweighting under common support

When all target combinations are represented in development, we can estimate the likelihood ratio

{\widehat P_{\mathrm{development}}(g)}$$ from unlabeled tag frequencies. The original recalibration model fit and candidate-partition construction remain unchanged. However, with this likelihood ratio, we havereplace development TP, FP, and FN counts with weighted counts (where the weight is precisely the likelihood ratio), and average cost with weighted average cost. Model/penalty selection, partition and call-plan selection, and the final cutoff then use these weighted quantities. For CLIP, this is **tag-assisted adaptation** in literature: it uses auxiliary unlabeled tags, and the reported costs do not include an additional charge for obtaining them. ### Results **Original ReCal-Route has higher mean F1 than the best competitor in 10 of 12 settings**, including all six tag settings. It is only 0.009 percentage points behind in common-support CLIP multimodal, and 0.286 points behind in the remaining setting. **Reweighting improves budget feasibility of the router:** due to better importance weighting of the items, the router can use cheaper VLMs for the less represented combinations. The results show that reweighted ReCal-Route reduces violations in every common-support setting; it completely eliminates observed violations in five settings while reducing number of violations from 27 to 6 in the CLIP multimodal setting. Each entry reports **mean F1 (%) [budget violations / successful splits]**. A violation means that a split’s mean test cost exceeds $B=2$. We also report “best native competitor” mean F1 separately for each row. Competitors retain their native routing objectives and predictions. **Common-support shift** | Context, panel | ReCal-Route | Reweighted ReCal-Route | Best native competitor | |---|---:|---:|---| | Tags, render only | 88.00 [2/40] | 88.24 [0/40] | Unified Router: 85.13 [0/40] | | Tags, render + meshgrid | 88.48 [10/40] | 88.34 [0/40] | WISE: 87.22 [0/40] | | Tags, multimodal | 82.31 [9/40] | 81.85 [0/40] | Unified Router: 81.21 [0/40] | | CLIP, render only | 88.85 [2/40] | 88.76 [0/40] | Avengers-Pro: 85.67 [0/40] | | CLIP, render + meshgrid | 88.82 [5/40] | 88.74 [0/40] | Avengers-Pro: 84.89 [0/40] | | CLIP, multimodal | 81.66 [27/40] | 81.17 [6/40] | Unified Router: 81.67 [0/40] | **New-combination shift** This table reports **whole-test performance, including both seen and unseen combinations**. In this case, we cannot simply use reweighting the new combinations do not appear in the development data by design. | Context, panel | ReCal-Route | Best native competitor | |---|---:|---| | Tags, render only | 83.49 [0/40] | WISE: 80.38 [0/40] | | Tags, render + meshgrid | 83.48 [4/40] | WISE: 82.06 [0/35]† | | Tags, multimodal | 80.78 [2/40] | Unified Router: 78.80 [0/40] | | CLIP, render only | 85.68 [0/40] | WISE: 81.23 [0/40] | | CLIP, render + meshgrid | 85.39 [2/40] | Avengers-Pro: 81.10 [0/40] | | CLIP, multimodal | 80.12 [18/40] | Unified Router: 80.40 [0/40] | *Remark*: WISE failed to return a valid policy on five splits; its statistics are computed using the remaining 35. ReCal-Route achieves mean F1 of 83.26 on these splits, for a direct comparison with WISE. Overall, these controlled studies illustrate the utility of ReCal-Route beyond completely random data splits. They also show why realized test cost must be evaluated explicitly when context distributions change, and reweighting could help bringing down the cost. In practical settings, we could detect unseen combinations by simply running a cheap VLM through a much smaller sample (of the possibly large testing set). Through human evaluation of this smaller sample, we can improve the coverage of our routing policy.

Hi Reviewer 86ZJ,

We have incorporated the distribution shift (into Appendix H) and explanations around the theoretical results (in Section 4) in our revised manuscript. In the remaining time, we would really appreciate it if you could indicate whether our new experiments and explanations are satisfactory. Thank you so much for your time and helpful comments.

Reviewer_ft22OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReCal-Route, which combines budget-constrained LLM judge routing with post-call recalibration to directly optimize FβF_\beta. It uses cross-fitted probability estimates and mixed-integer optimization, with regret bounds and a population-level no-harm guarantee under stated assumptions. Experiments on IMDb sentiment classification and proprietary VLM safety screening show improved F1 over raw routing and competing methods in most evaluated settings.

Critical strengths

Well-motivated objective and decision framework. Directly optimizing FβF_\beta addresses the mismatch between average judge accuracy and deployment objectives in imbalanced classification. Combining budgeted call selection with post-call recalibration makes effective use of information already obtained from judges.

Critical weaknesses
  1. Insufficient isolation of the source of improvement. Comparisons change both routing and the final decision rule. Without fixed-judge recalibration and baselines equipped with the same recalibration layer, it remains unclear how much benefit comes from the proposed routing optimization rather than post-processing. This limits the evidence for the routing contribution.

  2. The gaps between theory and methods. The analysis assumes independently fixed partitions and sufficiently accurate joint optimization, whereas experiments use label-dependent partitions and an iterative search without a global optimality certificate. These gaps limit how strongly the regret and minimax results substantiate the implemented method.

  3. Insufficient discussion of related work. The paper omits closely related studies, including [1] on robust budget-constrained judge routing and [2] on cost-aware conditional judge selection. [3] also establish the relationship between optimal F1 and probability thresholds. Discussing these connections would help clarify which components are established and what the proposed combination contributes beyond prior work.

[1] Zhang, Wenbo, et al. "Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge." ICML 2026.

[2] Zhu, Bin, Yi Xie, and Yanghui Rao. "Stopping and Routing LLM Judge Panels." arXiv preprint arXiv:2608.19802 (2026).

[3] Lipton, Zachary Chase, Charles Elkan, and Balakrishnan Narayanaswamy. "Thresholding classifiers to maximize F1 score." arXiv preprint arXiv:1402.1892 (2014).

Questions for authors
  1. How does ReCal-Route compare with a fixed inexpensive judge plus identical recalibration, and with existing routers augmented by the same recalibration layer? These controls would clarify the incremental value of joint routing optimization.

  2. Which guarantees apply to the experimental implementation? Can you bound or empirically quantify the optimization gap and explain how learning partitions on the same development data affects the analysis?

7 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer ft22

Thank you for your helpful comments. We answer the theoretical questions first.

2. Which guarantees apply to the experimental implementation? Can you bound or empirically quantify the optimization gap and explain how learning partitions on the same development data affects the analysis?

We revised the theory section in the paper to consider the best feasible policy (with respect to budget BB) within the class induced by the learned partition, or a finite collection of learned partitions. In particular, we do not have to assume that the learned partition is globally optimal for the final metric, as such a partition may be combinatorially hard to find.

We address the concern of data dependence between the partition and downstream procedure of ReCal-Route by bounding the estimation error uniformly over a fixed class containing every partition the learning procedure could produce. This is possible because our partition form is amenable to VC complexity generalization bound analysis: in particular, it is binning a linear score of the context-evidence features with a parameter vector (a slight generalization of a half-space function). For partitions formed by KK bins of a learned projection in dd dimensions, the additional complexity is Hχ=(d+K)log⁡(eK)H_\chi=(d+K)\log(eK). This term enters the uniform-convergence bound as a constant factor, and the estimation term of Theorem 1 remains O(n−1/2)O(n^{-1/2}). The argument therefore accounts for the partition’s data dependence with the procedure, the partitions do not have to be fixed.

As for numerical optimization, we do not claim that the optimizer found by the MiLP will be exact. For our theoretical results to be the same, we would need εopt=oP(n−1/2)\varepsilon_{\mathrm opt}=o_{P}(n^{-1/2}), where nn is the sample size. If nn is in the thousands, and the fact that our program is small in the experiments (hundreds of variables at most), this optimization gap is reasonable in our opinion. We can explicitly include this optimization error term in our theoretical results for clarity.

3. Discussion of related work

Lipton et al. [3] provided the population optimal thresholding for maximizing F1 score for generic classification problems. With the true conditional probabilities and β=1\beta=1, their characterization of the optimal cutoff coincides with our Eq. (7). We have added the reference in the revision. However, they do not establish statistical regret rates accounting for estimated scores and learned routing policies. Our analysis provides such guarantees, including faster rates under the additional margin and routing-gap conditions in Theorem 2 and the corresponding asymptotic minimax lower bound in Theorem 3. In particular, these theorems provide a sharp analysis of how the estimated conditional probability of the true labels and the routing policy interact under FβF_{\beta} optimization. The first nontrivial ingredient of our theoretical setup is that we do not require the fitted score to converge to the population probability of positive label given covariates, as long as Assumption 2 holds, which states that the limiting conditional probability preserves the true rank. In particular, Assumption 3 only requires a margin condition and the exponent does not have to be known. For the upper-bound guarantee of our procedure (Theorem 2), we employed a careful empirical process approach to take advantage of the margin condition and obtain a rate of qnq_n that can be strictly faster than n−1/2n^{-1/2}.

As for previous routing literature, [1] considers policies that route between reasoning and non-reasoning judges to improve accuracy under a budget and distribution shift. Its theory establishes uniqueness and convergence of the routing optimization. [2] builds calibrated judge panels using held-out squared-loss improvement relative to cost: discard redundant judges, add complementary judges, route specialists to relevant groups, and stop when further additions offer insufficient gain. Rather than making new routing pipelines, our paper formulates direct optimization of non-decomposable classification metrics (FβF_{\beta} specifically) under budget and fundamentally highlights the role of post-call recalibration. We also conduct a theoretical analysis that precisely attributes FβF_{\beta} optimization regret to decision-boundary uncertainty, score compression, and routing difficulty, with matching lower bounds. In all, we provide a coherent statistical framework for LLM routing with classification metrics, with a novel theory contribution.

AuthorsOpenReview ↗

Rebuttal for Reviewer ft22 (continued)

1a. Ablation of the recalibration layer against routing optimization

We compare learned routing, where there are 8 context-based groups, each with its own call plan, against no routing (one call plan for all items), using either recalibrated or native decisions. Results are reported in the table below. We find that routing improves recalibrated decisions in two settings: with tag context, it adds +1.32pp on render + meshgrid and +0.93pp on multimodal; the other four settings show essentially no value change. On the other hand, without recalibration, routing improves performance in all six settings, adding +0.49 to +3.05pp, with every 95% interval above zero.

In this following table, for each context/panel, Routing, recalibrated means the difference in mean F1 (learned routing - no routing) where both policies have post-call recalibration. Routing, native means the difference in mean F1 (learned routing - no routing) where both policies do not have post-call recalibration. Lastly, Recalibration, learned routing means the difference in mean F1 between recalibrated and unrecalibrated policies, with learned routing in both.

Context, panelRouting, recalibratedRouting, nativeRecalibration, learned routing
Tags, render only−0.02 [−0.34, +0.30]+2.99 [+2.66, +3.31]+3.51 [+3.08, +3.94]
Tags, render + meshgrid+1.32 [+1.07, +1.58]+2.43 [+2.14, +2.73]+1.82 [+1.46, +2.19]
Tags, multimodal+0.93 [+0.73, +1.13]+3.05 [+2.70, +3.39]+1.04 [+0.78, +1.30]
CLIP, render only−0.13 [−0.42, +0.15]+2.70 [+2.40, +3.01]+4.13 [+3.65, +4.62]
CLIP, render + meshgrid−0.25 [−0.51, +0.02]+0.90 [+0.66, +1.14]+2.73 [+2.40, +3.06]
CLIP, multimodal−0.14 [−0.50, +0.21]+0.49 [+0.11, +0.87]+3.80 [+3.40, +4.19]

We will add this ablation study to the revised submission. The main takeaway is that post-call recalibration is responsible for most of the improvement in mean F1. When the context variable is sparse and interpretable, such as a binary vector of semantic tags, routing optimization can drive the improvement further in presence of recalibration as well.

Q3: How well does pre-call information alone predict the label, and what do judge outputs add?

A classifier on context only (tags or CLIP), trained and tuned like our method, compared with the same classifier plus judge answers and confidence:

Context, panelContext only (F1)+ ReCal-Route at budget 2+ all six judges (cost 13.3)
Tags, render only79.7+5.5 [+5.0, +6.0]+6.1 [+5.7, +6.6]
Tags, render + meshgrid79.7+5.9 [+5.5, +6.4]+4.8 [+4.3, +5.3]
Tags, multimodal66.7+10.4 [+9.9, +11.0]+9.7 [+9.2, +10.3]
CLIP, render only84.1+1.4 [+1.0, +1.8]+2.6 [+2.2, +3.0]
CLIP, render + meshgrid84.1+0.9 [+0.5, +1.3]+0.6 [+0.2, +1.1]
CLIP, multimodal71.2+5.7 [+5.2, +6.3]+6.8 [+6.2, +7.3]

With CLIP embeddings on the render panels, judges add only 0.6 to 2.6pp. With coarse context or the harder multimodal task, judges add 5 to 10pp. Thus, applying our method to VLM-judges could substantially improve the performance.

AuthorsOpenReview ↗

Rebuttal for Reviewer ft22 (continued)

1b. Comparing with routers armed with post-call recalibration

Thank you for this insightful question. As requested, we give the competing methods post-call calibration on both the IMDb and asset datasets. In particular, we fix each competitor's routing/cascading policy and apply our recalibration probability model on top as a final layer. For each competing method, this recalibration model is trained beforehand using cross-fitting; furthermore, the best recalibration model and regularization penalty are chosen using training F1 score. We note that in all methods, recalibration does not change any call plans, so the costs remain the same.

3D-generated asset data: B=2

Below, we report mean held-out F1 (%) at B=2 over 100 random splits of the data with 70% training data. Here, + R denotes post-call recalibration, and we bold the best F1 result in each column. Recall that there are two context types: (i) binary vector of semantic tags indicating different properties of an item, and (ii) generic CLIP embeddings.

MethodTags: renderTags: render–mesh gridTags: multimodalCLIP: renderCLIP: render–mesh gridCLIP: multimodal
ReCal-Route85.1985.6377.1585.4784.9976.95
RouterBench-kNN + R83.4582.5970.8985.4085.0077.87
Avengers-Pro + R85.0683.8574.3085.8484.9177.13
Unified Router + R83.8283.1175.5685.5784.4078.37
WISERouter + R84.3085.3875.3186.1384.9576.79
Depth-2 cascade + R82.8781.9972.6386.3285.1978.33

We have the following main takeaways:

Utility of post-call recalibration. Recalibration improves 29 of the 30 competitor/context/input comparisons at B=2, often substantially: RouterBench-kNN rises from 75.06 to 83.45 with render-only tags, while Cascade rises from 80.58 to 86.32 with render-only CLIP. These gains demonstrate that our recalibration component can and should be applied broadly beyond our own router.

Context matters. ReCal-Route still achieves the highest mean F1 in all three semantic-tag settings at B=2. Sparse, discrete tags may make the relevant regional differences in judge reliability particularly suitable for our group-based routing. It could suggest that in isolation, our routing method may be more suitable for interpretable context. On the other hand, recalibrated competitors are stronger with dense CLIP embeddings, making better routing representations for generic embeddings a meaningful direction for future work. We have revised our appendix section to include this table, as well as full native/recalibrated budget curves.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttals

To the AC and Reviewers,

Thank you so much for your helpful comments in improving multiple aspects of our paper. We summarize the main takeaways of our rebuttals and extra experiments.

  1. Reconciliation of practical implementation and theoretical results: we revised the theory section in the paper (Section 4) to consider the best feasible policy (with respect to budget) within the class induced by the learned partition, or a finite collection of learned partitions. In particular, we do not have to assume that the learned partition is globally optimal for the final metric. We have also included the extra implementation detail about selecting recalibration models in a Remark in the theory section.

  2. Discussion of related work: Our paper has certain components (such as the LLM routing problem, or thresholds of F1 classifiers) that appeared in previous literature, but plays a unique role in constructing a direct optimization procedure for non-decomposable classification metrics, as well as bringing a nontrivial theoretical contribution (Section 1, Section A.2).

  3. Ablation of recalibration against routing optimization: The main takeaway is that post-call recalibration is responsible for most of the improvement in mean F1. When the context variable is sparse and interpretable, such as a binary vector of semantic tags, routing optimization can drive the improvement further in presence of recalibration as well (Appendix F).

  4. Comparing with competing methods armed with post-call recalibration: The results strongly demonstrate that our recalibration component can and should be applied broadly beyond our own router. As for the value of routing optimization in presence of recalibration, we find that our routing method may be more suitable for sparse, interpretable context (Appendices E.6, E.7).

  5. Distribution shift: We show that ReCal-Route still performs very well comparatively under covariate (context) shift, while a reweighted version of ReCal-Route can achieve nearly the same performance with less cost (Appendix H).

We have incorporated these useful ablation results in our manuscript, as indicated in the particular positions in brackets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.