acceptodds
Under review as a conference paper at ICLR 2027

ReCal: Reward Calibration for RL-based LLM Routing

Abstract

Large language model (LLM) routing has emerged as an effective paradigm for leveraging the complementary strengths of multiple LLMs through dynamic model and reasoning-strategy selection. Recent reinforcement learning (RL)-based routing methods further improve routing quality by optimizing routing policies from interaction feedback. However, they still struggle to provide informative and comparable learning signals under heterogeneous tasks with varying difficulty. In practice, multiple objectives (e.g., correctness, format behavior) are aggregated into a single scalar reward, leading to ambiguous credit assignment and conflicting optimization signals. Moreover, reward distributions vary substantially across instances and datasets, making their optimization contributions difficult to compare and potentially leading to imbalanced policy updates. To address these issues, we propose **ReCal**, a **Re**ward **Cal**ibration framework for RL-based LLM routing. We first introduce a hierarchical reward decomposition mechanism with component-wise advantage estimation. We further propose a distribution-aware optimization strategy that calibrates optimization contributions through variance-aware reweighting and per-dataset normalization. Experiments on seven QA benchmarks show that ReCal consistently improves routing performance and achieves more stable training dynamics compared with baselines. Additional experiments on mathematical reasoning further show that this calibration procedure can be applied beyond QA through task-specific reward instantiation.

Primary area:
reinforcement learning
Keywords:
LLM Routing, Reinforcement Learning, Reward Calibration, Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8gt1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReCal, a reward calibration method for RL-based LLM routing. The method separates routing rewards into multiple components and estimates advantages for each component before aggregation. It also introduces variance-aware reweighting and per-dataset normalization to adjust learning signals across different samples and datasets. Experiments on seven QA benchmarks show improvements over Router-R1. The authors also report results on mathematical reasoning tasks.

Critical strengths

Presentation: The paper is easy to follow. I like how Figure 2 presents the full method. The layout is clean, and the tables make it easy to check the results.

Experiments: The authors test on seven QA datasets, including five out-of-domain datasets. The improvement over Router-R1 is clear. The ablations are fairly detailed, with separate tests for variance-aware reweighting and per-dataset normalization. The additional math experiments are also useful.

Critical weaknesses

Limited Novelty: The main idea of normalizing each reward component before aggregation has already been explored in GDPO and MO-GRPO. ReCal follows a very similar formulation in Eqs. 4–5. Its additional mechanisms include correctness gating, variance-aware reweighting, and per-dataset normalization. These appear to be relatively incremental changes. The paper does not clearly distinguish its algorithm from GDPO and MO-GRPO. Direct comparisons are also missing. The current ablations show that the components are useful, but do not establish a substantial improvement over existing multi-reward optimization methods.

Missing Cost Analysis: The experiments focus on answer accuracy. The paper does not report the actual routing cost or inference latency. The routing reward also encourages multiple external model calls. Higher accuracy may therefore come at a higher inference cost. I would like to see whether ReCal still offers an advantage over simpler approaches, such as repeated sampling without routing, under the same time and API budget.

Questions for authors

Prior Methods: Eqs. 4 and 5 seem close to GDPO and MO-GRPO. What is different in ReCal? Could the authors compare these methods using the same reward components and training setup?

Routing Cost: How many external model calls does ReCal make per question? What are the average API cost and inference latency compared with Router-R1? Does ReCal outperform repeated sampling without routing under the same inference budget?

Variance Reweighting: Eq. 7 gives more weight to groups with higher reward variance. Why should these groups be more useful for training? With only four rollouts per query, the variance estimate may also be noisy. Have the authors examined this?

2 comments

Thank you for recognizing the clear presentation and consistent empirical improvements. We address the methodological differences, inference budget, and variance reweighting below.

Q1. Differences from GDPO and MO-GRPO.

GDPO, MO-GRPO, and ReCal share component-wise normalization before aggregation. MO-GRPO Eq. (5) sums standardized reward components equally, while our CAE weights them after standardization to express objective priorities. With unit weights and identical rewards, these advantage constructions coincide apart from numerical stabilizers. GDPO further applies batch-wise normalization after aggregation.

ReCal extends this foundation to calibrate learning signals across heterogeneous routing queries and datasets. Its correctness-first reward hierarchy links auxiliary routing behaviors to answer quality. Variance-aware reweighting then adjusts contributions across sampled routing groups using clipped dispersion-dependent factors, while per-dataset normalization computes statistics separately for each dataset in the current batch. Thus, ReCal jointly addresses objective-level learning-signal clarity and instance- and dataset-level optimization comparability.

To evaluate these additional calibration operations, we use GDPO as the matched comparator. The “w/o DAO” configuration retains the same reward components, correctness gating, objective weights, and training setup, obtaining 0.456 average EM versus 0.481 for ReCal. This directly measures the benefit of DAO under shared rewards and priorities.

This comparison also determines our experimental choice. GDPO preserves the weighted aggregation and post-aggregation normalization used in our training pipeline, allowing the comparison to focus on DAO. A separate run of MO-GRPO’s published equal-weight formulation would additionally change objective priorities. We therefore use the matched GDPO comparison to assess the contribution beyond the component-wise normalization shared by these methods.

Q2. Routing cost and repeated sampling.

How many external calls does ReCal make per question?

On the same 3,125 test questions, ReCal makes 3,129 external calls and Router-R1 makes 3,148, corresponding to 1.0013 and 1.0074 calls per question. ReCal’s accuracy improvement therefore occurs at nearly identical external-call usage.

How do average API cost and inference latency compare with Router-R1?

The similar external-call usage suggests broadly comparable API cost and latency. ReCal’s reward and advantage calibration occurs during training, adding no inference-time model passes or API calls.

Does ReCal outperform repeated sampling without routing?

We evaluated Llama-3.1-70B-Instruct answering complete questions directly, without a router or subquestion decomposition. Sampling uses temperature 0.7, top-p 1.0, and a 512-token output limit. K=1 takes the first sample per question; K=3 applies majority voting to three independently seeded samples. Answers are normalized before voting, ties select the earliest valid answer, and ground truth is used only for scoring.

MethodMacro EMMacro F1External calls/question
ReCal0.47800.56291.0013
Router-R10.43260.52031.0074
Non-routing, K=10.30340.40021
Non-routing, K=3 voting0.31630.41223

ReCal outperforms direct answering at comparable call counts and three-sample voting despite the latter using approximately three times as many external calls.

Q3. Why emphasize reward dispersion, and how reliable are small response groups?

Greater within-group reward dispersion indicates more differentiated feedback among sampled routing decisions, motivating greater optimization weight. We use it as an observable proxy for potentially informative routing differences. Clipping bounds the influence of this signal when estimates fluctuate.

At G=4, positive, uniform, and inverse weighting obtain average EM of 0.481, 0.458, and 0.451, respectively. The inverse control reciprocates the scaling factor and uses the same clipping bounds, [0.8, 1.25]. These results support the positive weighting direction.

We also evaluated G=2, obtaining 0.434 average EM. Although below 0.481 at G=4, it remains above Router-R1’s 0.414. ReCal therefore remains effective with only two samples per query, while benefiting from a larger response group. Together, these results support the empirical usefulness of bounded dispersion reweighting under small-group sampling.

Thank you for your response. I will maintain my original score.

Reviewer_ASiDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReCal, a reward calibration framework for reinforcement learning-based LLM routing. ReCal addresses two challenges in policy optimization: ambiguous learning signals induced by scalarized multi-objective rewards, as well as heterogeneous optimization contributions across routing instances and datasets. The framework integrates hierarchical reward decomposition, component-wise advantage estimation, variance-aware reweighting, and per-dataset normalization. Experiments across seven QA benchmarks demonstrate consistent improvements over Router-R1, lifting the average exact match from 0.414 to 0.481. The paper further includes ablation studies, training-dynamics analyses, evaluations with an expanded candidate model pool, and supplementary experiments on mathematical reasoning tasks.

Critical strengths

(1) Consistent empirical improvements across heterogeneous QA benchmarks. ReCal yields consistent gains over Router-R1 on all seven evaluated QA datasets, lifting the average EM from 0.414 to 0.481. Improvements generalize beyond the two training datasets to five out-of-domain benchmarks. Supplementary F1 results and ablation studies further validate the framework’s empirical effectiveness. This cross-set consistency constitutes our strongest evidence supporting acceptance.

(2) A practical and computationally lightweight approach for enhancing RL-based routing. This work targets a meaningful limitation of multi-objective reward optimization in LLM routing. The proposed calibration operations can be plugged into existing GRPO-style training pipelines, requiring no extra model forward passes or external API calls for reward calibration. The framework is thus straightforward to implement and holds practical value for existing RL-based routing systems.

Critical weaknesses

(1) Limited methodological novelty and insufficient isolation of the proposed calibration mechanisms.

(a) The methodological novelty of ReCal is not sufficiently established against prior literature. Its component-wise advantage estimation builds upon the group-relative normalization adopted in GRPO (Shao et al., 2024), and its dataset-level normalization is conceptually related to reward-scale normalization for multi-task reinforcement learning such as PopArt (Hessel et al., 2019). Adaptive reward weighting has also been explored in multi-objective LLM reinforcement learning (Lu et al., 2026). While ReCal assembles these ideas within the specialized setting of LLM routing, the paper fails to clearly identify a novel optimization principle or unique empirical advantage emerging from this combination that cannot be explained by the individual constituent mechanisms.

(b) More critically, the experimental evidence does not fully disentangle these contributions. For instance, the scalarized-same-reward control in Appendix D removes correctness gating and distribution-aware optimization alongside modifying the advantage estimation formulation. As a result, its performance gap compared with ReCal cannot be uniquely attributed to the proposed advantage estimation strategy. Despite the additional ablation studies provided, the standalone contribution of correctness gating remains under-validated. This weakens the evidential support for the paper’s core methodological claims.

(2) Insufficient justification and validation of variance-aware reweighting, together with missing cost-aware evaluation.

(c) ReCal assigns larger optimization weights to query groups exhibiting higher reward standard deviation, implicitly treating reward dispersion as an indicator of informative routing differences. Nevertheless, high variance can also stem from stochastic reward noise or unstable model outputs. With merely four rollouts per query, the reliability of this signal warrants further scrutiny. The subsequent per-dataset normalization may also partially cancel the absolute scaling introduced by variance-aware reweighting, leaving the interaction between these two operations ambiguous.

(d) Furthermore, the routing reward explicitly incentivizes additional external-model calls, yet the paper lacks a sufficiently detailed accuracy–cost trade-off analysis relative to Router-R1. Accordingly, it remains unclear whether the observed accuracy gains correspond to more effective routing decisions under matched inference budgets. These concerns undermine the mechanistic interpretability and practical significance of the reported improvements.

Questions for authors

(1) Can the authors provide a controlled ablation that preserves the same correctness-gated reward components and distribution-aware optimization, while replacing only component-wise advantage estimation with scalarized group-relative advantage estimation? Additionally, what fraction of the overall improvement can be attributed to correctness gating in isolation?

(2) What evidence justifies treating higher within-group reward variance as a signal of greater optimization priority? Specifically, have the authors compared positive variance weighting against uniform weighting or inverse-variance weighting, or investigated its sensitivity to rollout group size?

(3) How does per-dataset normalization interact with the preceding variance-aware scaling? Since standardization can remove uniform positive rescaling within a dataset, could the authors clarify which relative weighting effects survive normalization and supply empirical evidence for their contribution?

(4) Does ReCal retain its performance advantage over Router-R1 under matched inference costs or an identical budget of external model calls?

(5) Are the main reported results stable across multiple random seeds? In particular, can the authors report the variability of the average EM gain and clarify whether the claimed faster convergence persists under a fixed training-step budget?

2 comments

Thank you for recognizing ReCal’s consistent gains and practical value. We address the contribution and your five questions below.

Contribution and related work

ReCal jointly calibrates learning-signal clarity and optimization comparability. LSC preserves objective-specific improvements before aggregation; DAO calibrates contributions across heterogeneous routing instances and datasets.

GRPO normalizes scalarized rewards; CAE normalizes components before weighted aggregation. PopArt adaptively normalizes value targets while preserving unnormalized predictions; ReCal uses batch-local advantage statistics within each dataset. Lu et al. (2026) adapt objective weights to explore reward trade-offs; ReCal retains specified objective weights and calibrates routing-group contributions using reward dispersion and dataset statistics.

With the same rewards and training setup, our GDPO implementation (“w/o DAO”) obtains 0.456 average EM versus 0.481 for ReCal, supporting the additional distribution-aware calibration.

Q1. Isolating CAE and correctness gating

We conducted two controlled ablations. The CAE control retains the same gated rewards, reward weights, and DAO, replacing only component-wise estimation with scalarized group-relative estimation. The gating control retains CAE, DAO, and other settings, removing only the auxiliary correctness gate.

ConfigurationAvg. EMAvg. F1
Full ReCal0.48060.564
Replace only CAE0.42600.513
Remove only gating0.43460.529

Both ablations reduce EM/F1 on all seven datasets. Full EM/F1 is recomputed from the original per-dataset scores for consistent precision. These drops quantify each component’s conditional contribution with the other components retained; interactions mean they are not additive shares of the total gain.

Q2. Weighting direction and group size

Greater reward dispersion indicates more differentiated feedback among sampled routing decisions, motivating greater optimization weight. It serves as a proxy for potentially informative differences; clipping limits the influence of noisy estimates.

At G=4, positive, uniform, and inverse weighting obtain average EM of 0.481, 0.458, and 0.451. The inverse control reciprocates the scaling factor with the same clipping bounds, [0.8, 1.25]. This supports positive weighting.

At G=2, ReCal obtains 0.434 EM, compared with 0.481 at G=4 and 0.414 for Router-R1. It remains effective with two samples while benefiting from a larger group, supporting practical effectiveness under small-group sampling.

Q3. Interaction with dataset normalization

A common positive multiplier for an entire dataset is canceled by standardization, up to numerical stabilizers. ReCal’s implemented factors vary across query groups and reward components. After weighted aggregation, normalization uses a common mean and standard deviation within each dataset, computed with valid-token masks.

For pre-normalization advantages B_u and B_v, their normalized difference is (B_u-B_v)/(s_d+epsilon). Nonuniform factors change the numerator and generally survive the common normalization, while the overall dataset scale is removed.

Removing reweighting while retaining dataset normalization reduces EM from 0.481 to 0.458, supporting its contribution after normalization.

Q4. Comparable external-call usage

In an additional evaluation on the same 3,125 test questions, ReCal and Router-R1 obtain macro EM/F1 of 0.4780/0.5629 and 0.4326/0.5203. Their external-call totals are 3,129 and 3,148, or 1.0013 and 1.0074 per question. ReCal therefore achieves higher accuracy at nearly identical observed call usage, with slightly fewer calls.

Q5. Multiple seeds and a fixed training budget

Across two training seeds per method, seven-benchmark macro EM is 0.4760 ± 0.0065 for ReCal and 0.4234 ± 0.0129 for Router-R1 (mean ± sample standard deviation). ReCal improves average EM in both seeds by 6.63 and 3.89 percentage points, yielding a mean gain of 5.26 points. ReCal outperforms Router-R1 on all seven datasets in the additional seed.

Across two training runs per method, both ReCal runs outperform both Router-R1 runs in mean NQ/HotpotQA F1 at all eight shared validation checkpoints from steps 15 to 120. At step 120, the across-seed means are 0.5592 and 0.4688. This supports consistent test gains across seeds and better validation performance at the same training-step budget.

Thank you for the authors' rebuttal. However, significant concerns persist, and I stand by my original rating.

Reviewer_b45MOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

the paper is focusing on the llm training based on router r1. it trained the llm router with the HIERARCHICAL REWARD and VARIANCE-AWARE REWEIGHTING (they claim as distribution aware) mechanism and use the eperiments to show the performance is sota.

Critical strengths

The experiment are solid and the adv/reward manipulation is interesting and effective shown by the main experiments and abalation.

Critical weaknesses

my main concern is the novelty. I think the hierarchical reward decomposition and the variance aware weight maybe useful in practice, but it is more like an engineer improvement for me, not a novel method.

Questions for authors
  1. how the hierarchical stucture designed. any other stucture can work or only this can work. does this stucture indepedent of the task and the routed model?
  2. for figure 3b, any missing component will make training collapse, then why vanilla route R1 does not work (since route r1 does not have any of these components)?
2 comments

Thank you for recognizing the effectiveness of ReCal and the strength of its experiments. We clarify the contribution and address your questions below.

Novelty

ReCal introduces a learning-signal calibration framework for RL-based routing. It addresses two connected problems: scalar reward aggregation obscures objective-specific improvements, while heterogeneous reward distributions make optimization contributions difficult to compare across routing instances and datasets.

These problems motivate the two-stage design. Learning Signal Clarity Calibration (LSC) organizes rewards around answer correctness and estimates component-wise advantages before aggregation, preserving objective-specific relative improvements. Distribution-aware Optimization (DAO) then calibrates their optimization contributions through variance-aware reweighting and per-dataset normalization.

The contribution is therefore the formulation and design of routing learning-signal calibration. The lightweight statistical operations implement this design.

Questions 1

The hierarchy follows a correctness-first principle: auxiliary behaviors should support answer quality. The answer reward remains active, while information, format, routing, and balance rewards are activated only when the answer meets the correctness threshold. This avoids rewarding auxiliary behavior independently of adequate answer quality. The auxiliary rewards share one gate; they are not successive levels of gating.​

Other structures may work, and we do not claim this hierarchy is uniquely optimal. The reward definitions and gating criterion are task-specific, whereas component-wise advantage estimation and distribution-aware calibration can be retained. Our mathematical reasoning experiments illustrate this separation by adapting the reward instantiation while retaining the calibration procedure. The hierarchy does not require particular candidate-model identities.

Questions 2

Router-R1’s format and EM rewards define a simpler optimization objective. ReCal introduces additional objectives concerning information quality, routing calls, and model utilization. These objectives follow a unified design: improve the routing process while keeping answer quality primary. The information reward evaluates information returned by external models, the routing reward encourages multi-round routing, and the balance reward discourages excessive concentration on particular models.

The components therefore serve complementary purposes. Removing the balance reward, for example, retains the incentive for additional calls while removing the incentive for balanced model utilization. Such an ablation changes the coordination among objectives; it does not recover Router-R1’s simpler format-plus-EM objective.

Figure 3 shows less stable routing behavior and reward trajectories under the tested ablations than under full ReCal, supporting the value of the complete design. Router-R1 can nevertheless remain stable with its simpler objective. These observations are compatible: ReCal’s additional components serve complementary purposes within a richer routing objective, and the tested partial ablations weaken training stability under that objective.

Thanks for the author's reply. while i believe the soundness and novelty concern has not been solved. I maintain my score

Comments on the paper

1 comment

Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.