- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes ReCal-Route, which selects LLM judges under a cost budget and adjusts their predictions using confidence scores and other available information to improve classification metrics. It provides theoretical guarantees and shows empirical improvements on sentiment classification and safety screening.
Critical strengths
Strength #1: well-motivated practical formulation: the paper jointly optimizes judge selection and classification performance under a budget. such a setting is very useful and relevant in real-world deployment of LLMs. the post-call recalibration uses the returned prediction to improve decisions without additional model calls, which is also a tremendous practical advantage
Strength #2: substantial theoretical depth: the regret bounds accomodate imperfect probability estimates with aligned decisions, analysis connecting estimation error, margins, and routing regret, and matching lower bounds.
Critical weaknesses
Weakness #1: Insufficient comparisons against recalibrated baselines: the results show the effectiveness of the complete pipeline, but it's unclear how much joint routing adds beyond supervised post-processing. See questions below for ablations that could be run.
Questions for authors
- How much does the adaptive routing add beyond recalibration?
- What about when competing routers receive same post-processing?
- How well can a model predict the label using only the information available before calling a judge? and how much does adding the judges’ answers and confidence improve the results?
4 comments
Rebuttal for Reviewer PG2Q (continued)
2. What about when competing routers receive same post-processing?
Thank you for this insightful question. As requested, we give the competing methods post-call calibration on both the IMDb and asset datasets. In particular, we fix each competitor's routing/cascading policy and apply our recalibration probability model on top as a final layer. For each competing method, this recalibration model is trained beforehand using cross-fitting; furthermore, the best recalibration model and regularization penalty are chosen using training F1 score. We note that in all methods, recalibration does not change any call plans, so the costs remain the same.
3D-generated asset data: B=2
Below, we report mean held-out F1 (%) at B=2 over 100 random splits of the data with 70% training data. Here, + R denotes post-call recalibration, and we bold the best F1 result in each column. Recall that there are two context types: (i) binary vector of semantic tags indicating different properties of an item, and (ii) generic CLIP embeddings.
| Method | Tags: render | Tags: render–mesh grid | Tags: multimodal | CLIP: render | CLIP: render–mesh grid | CLIP: multimodal |
|---|---|---|---|---|---|---|
| ReCal-Route | 85.19 | 85.63 | 77.15 | 85.47 | 84.99 | 76.95 |
| RouterBench-kNN + R | 83.45 | 82.59 | 70.89 | 85.40 | 85.00 | 77.87 |
| Avengers-Pro + R | 85.06 | 83.85 | 74.30 | 85.84 | 84.91 | 77.13 |
| Unified Router + R | 83.82 | 83.11 | 75.56 | 85.57 | 84.40 | 78.37 |
| WISERouter + R | 84.30 | 85.38 | 75.31 | 86.13 | 84.95 | 76.79 |
| Depth-2 cascade + R | 82.87 | 81.99 | 72.63 | 86.32 | 85.19 | 78.33 |
We have the following main takeaways:
Utility of post-call recalibration. Recalibration improves 29 of the 30 competitor/context/input comparisons at B=2, often substantially: RouterBench-kNN rises from 75.06 to 83.45 with render-only tags, while Cascade rises from 80.58 to 86.32 with render-only CLIP. These gains demonstrate that our recalibration component can and should be applied broadly beyond our own router.
Context matters. ReCal-Route still achieves the highest mean F1 in all three semantic-tag settings at B=2. Sparse, discrete tags may make the relevant regional differences in judge reliability particularly suitable for our group-based routing. It could suggest that in isolation, our routing method may be more suitable for interpretable context. On the other hand, recalibrated competitors are stronger with dense CLIP embeddings, making better routing representations for generic embeddings a meaningful direction for future work. We have revised our appendix section to include this table, as well as full native/recalibrated budget curves.
IMDb: comparison at B=6
For completeness, we also extend the experiment above to the IMDb dataset, using 40 random data splits. Below, the unit for budgets is nominal cost unit per 10,000 reviews, and the arrows show mean F1 results from native to recalibrated competing methods.
Firstly, the results reinforce our previous point about post-call recalibration: recalibration improves all 18 competitor/context means. Secondly, regarding relative performance, ReCal-Route is first with tags and essentially tied in mean performance with the strongest recalibrated competitor in the other two contexts: its deficits to FrugalML are only 0.084 percentage points with LSA and 0.039 with combined context. It exceeds each of the other five recalibrated competitors in all three contexts. This supports the competitiveness of our joint routing-and-recalibration procedure after competing routers receive the same recalibration model class.
| Method | Tags | LSA | Combined |
|---|---|---|---|
| ReCal-Route | 87.61 | 90.63 | 90.67 |
| RouterBench-kNN | 84.56 → 84.60 | 87.51 → 90.13 | 86.72 → 90.01 |
| Avengers-Pro | 86.79 → 87.03 | 87.54 → 90.47 | 86.92 → 90.40 |
| Unified Router | 86.87 → 86.93 | 88.83 → 90.48 | 88.89 → 90.46 |
| WISERouter | 86.82 → 86.87 | 87.30 → 90.49 | 86.87 → 90.41 |
| Depth-2 cascade | 87.15 → 87.35 | 86.61 → 90.54 | 86.64 → 90.54 |
| FrugalML | 84.91 → 87.58 | 84.91 → 90.72 | 84.91 → 90.71 |
Rebuttal for Reviewer PG2Q (continued)
Stress test: restricting the quality–cost action library
To stress test routing performance when high-return actions are scarce and offer poorer F1 value, we retain/filter the 10, 20, or 30 call plans with the lowest development F1-to-cost ratios from the original 66-plan library. The retained collection of actions is fixed across budgets and shared by all considered methods.
We evaluate all three library sizes at B = 5.25, 5.5, 5.75, and 6, using 40 random splits in each of six settings. Competitors retain their original accuracy/correctness objectives in the respective work. In this stress test, +R_acc also accuracy-selected post-call recalibration (whereas the main VLM/IMDb recalibration experiments above use an F1-selected recalibration model and threshold).
Remark: Constrained Cascade has to commit to a valid plan before calling a judge. Due to our experiment setup, we have to minimally modify the cascade, which originally called judges sequentially: due to call plan pruning, certain judges may not have a feasible judge to form a valid pair. We mitigate this by scoring the plans beforehand using the Cascade's linearized score combining accuracy and cost.
At B=6, the table compares ReCal-Route with the highest mean among recalibrated competitors having all 40 successful splits in that setting. Here, the reported values are again mean F1 (%), and we bold the larger mean value in each pair.
| Retained plans | Method | Tags: render | Tags: render–mesh grid | Tags: multimodal | CLIP: render | CLIP: render–mesh grid | CLIP: multimodal |
|---|---|---|---|---|---|---|---|
| 10 | ReCal-Route | 81.38 | 83.12 | 77.65 | 85.17 | 85.35 | 76.03 |
| 10 | Best +R_acc (40 splits) | 80.59 | 81.16 | 69.07 | 84.85 | 85.53 | 76.85 |
| 20 | ReCal-Route | 83.90 | 84.05 | 77.46 | 85.61 | 85.14 | 77.33 |
| 20 | Best +R_acc (40 splits) | 81.69 | 82.49 | 72.30 | 85.31 | 85.64 | 77.57 |
| 30 | ReCal-Route | 84.31 | 85.42 | 77.42 | 85.49 | 84.95 | 76.95 |
| 30 | Best +R_acc (40 splits) | 84.27 | 85.39 | 76.54 | 86.29 | 85.81 | 78.47 |
Among complete 40-split comparisons at B=6, ReCal-Route performs the best for all three semantic-tag settings for every library size. With 10 or 20 plans it also leads CLIP render-only; its largest deficit to a full-coverage recalibrated competitor is 0.82 and 0.50 points, respectively. With 30 plans, it exceeds every native competitor in all six settings; after competitor recalibration, it trails on CLIP by 0.80–1.52 points. These results support competitive routing under action restrictions, with stronger performance for semantic context.
We also count how many splits/settings that a method successfully returns a policy. In the following table, counts include all four budgets, including the tighter cases: 960 split/settings per library size. ReCal-Route matches the highest successful-evaluation count with 10 and 30 plans. With 20 plans, Constrained Cascade completes 930 cases versus 899 for ReCal-Route.
| Retained plans | ReCal-Route | kNN +R_acc | Avengers +R_acc | Unified +R_acc | WISE +R_acc | Constrained Cascade +R_acc |
|---|---|---|---|---|---|---|
| 10 | 833/960 | 746/960 | 827/960 | 778/960 | 108/960 | 833/960 |
| 20 | 899/960 | 894/960 | 898/960 | 898/960 | 130/960 | 930/960 |
| 30 | 960/960 | 960/960 | 960/960 | 960/960 | 391/960 | 960/960 |
Rebuttal for Reviewer PG2Q (continued)
3. How well does pre-call information alone predict the label, and what do judge outputs add?
A classifier on context only (tags or CLIP), trained and tuned like our method, compared with the same classifier plus judge answers and confidence:
| Context, panel | Context only (F1) | + ReCal-Route at budget 2 | + all six judges (cost 13.3) |
|---|---|---|---|
| Tags, render only | 79.7 | +5.5 [+5.0, +6.0] | +6.1 [+5.7, +6.6] |
| Tags, render + meshgrid | 79.7 | +5.9 [+5.5, +6.4] | +4.8 [+4.3, +5.3] |
| Tags, multimodal | 66.7 | +10.4 [+9.9, +11.0] | +9.7 [+9.2, +10.3] |
| CLIP, render only | 84.1 | +1.4 [+1.0, +1.8] | +2.6 [+2.2, +3.0] |
| CLIP, render + meshgrid | 84.1 | +0.9 [+0.5, +1.3] | +0.6 [+0.2, +1.1] |
| CLIP, multimodal | 71.2 | +5.7 [+5.2, +6.3] | +6.8 [+6.2, +7.3] |
With CLIP embeddings on the render panels, judges add only 0.6 to 2.6pp. With coarse context or the harder multimodal task, judges add 5 to 10pp. Thus, applying our method to VLM-judges could substantially improve the performance.
Rebuttal for Reviewer PG2Q
Thank you so much for your helpful comments and support for our work.
1. How much does the adaptive routing add beyond recalibration?
We compare learned routing, where there are 8 context-based groups, each with its own call plan, against no routing (one call plan for all items), using either recalibrated or native decisions. Results are reported in the table below. We find that routing improves recalibrated decisions in two settings: with tag context, it adds +1.32pp on render + meshgrid and +0.93pp on multimodal; the other four settings show essentially no value change. On the other hand, without recalibration, routing improves performance in all six settings, adding +0.49 to +3.05pp, with every 95% interval above zero.
In this following table, for each context/panel, Routing, recalibrated means the difference in mean F1 (learned routing - no routing) where both policies have post-call recalibration. Routing, native means the difference in mean F1 (learned routing - no routing) where both policies do not have post-call recalibration. Lastly, Recalibration, learned routing means the difference in mean F1 between recalibrated and unrecalibrated policies, with learned routing in both.
We will add this ablation study to the revised submission. The main takeaway is that post-call recalibration is responsible for most of the improvement in mean F1. When the context variable is sparse and interpretable, such as a binary vector of semantic tags, routing optimization can drive the improvement further in presence of recalibration as well.