- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReCal, a reward calibration method for RL-based LLM routing. The method separates routing rewards into multiple components and estimates advantages for each component before aggregation. It also introduces variance-aware reweighting and per-dataset normalization to adjust learning signals across different samples and datasets. Experiments on seven QA benchmarks show improvements over Router-R1. The authors also report results on mathematical reasoning tasks.
Critical strengths
Presentation: The paper is easy to follow. I like how Figure 2 presents the full method. The layout is clean, and the tables make it easy to check the results.
Experiments: The authors test on seven QA datasets, including five out-of-domain datasets. The improvement over Router-R1 is clear. The ablations are fairly detailed, with separate tests for variance-aware reweighting and per-dataset normalization. The additional math experiments are also useful.
Critical weaknesses
Limited Novelty: The main idea of normalizing each reward component before aggregation has already been explored in GDPO and MO-GRPO. ReCal follows a very similar formulation in Eqs. 4–5. Its additional mechanisms include correctness gating, variance-aware reweighting, and per-dataset normalization. These appear to be relatively incremental changes. The paper does not clearly distinguish its algorithm from GDPO and MO-GRPO. Direct comparisons are also missing. The current ablations show that the components are useful, but do not establish a substantial improvement over existing multi-reward optimization methods.
Missing Cost Analysis: The experiments focus on answer accuracy. The paper does not report the actual routing cost or inference latency. The routing reward also encourages multiple external model calls. Higher accuracy may therefore come at a higher inference cost. I would like to see whether ReCal still offers an advantage over simpler approaches, such as repeated sampling without routing, under the same time and API budget.
Questions for authors
Prior Methods: Eqs. 4 and 5 seem close to GDPO and MO-GRPO. What is different in ReCal? Could the authors compare these methods using the same reward components and training setup?
Routing Cost: How many external model calls does ReCal make per question? What are the average API cost and inference latency compared with Router-R1? Does ReCal outperform repeated sampling without routing under the same inference budget?
Variance Reweighting: Eq. 7 gives more weight to groups with higher reward variance. Why should these groups be more useful for training? With only four rollouts per query, the variance estimate may also be noisy. Have the authors examined this?
2 comments
Thank you for your response. I will maintain my original score.
Thank you for recognizing the clear presentation and consistent empirical improvements. We address the methodological differences, inference budget, and variance reweighting below.
Q1. Differences from GDPO and MO-GRPO.
GDPO, MO-GRPO, and ReCal share component-wise normalization before aggregation. MO-GRPO Eq. (5) sums standardized reward components equally, while our CAE weights them after standardization to express objective priorities. With unit weights and identical rewards, these advantage constructions coincide apart from numerical stabilizers. GDPO further applies batch-wise normalization after aggregation.
ReCal extends this foundation to calibrate learning signals across heterogeneous routing queries and datasets. Its correctness-first reward hierarchy links auxiliary routing behaviors to answer quality. Variance-aware reweighting then adjusts contributions across sampled routing groups using clipped dispersion-dependent factors, while per-dataset normalization computes statistics separately for each dataset in the current batch. Thus, ReCal jointly addresses objective-level learning-signal clarity and instance- and dataset-level optimization comparability.
To evaluate these additional calibration operations, we use GDPO as the matched comparator. The “w/o DAO” configuration retains the same reward components, correctness gating, objective weights, and training setup, obtaining 0.456 average EM versus 0.481 for ReCal. This directly measures the benefit of DAO under shared rewards and priorities.
This comparison also determines our experimental choice. GDPO preserves the weighted aggregation and post-aggregation normalization used in our training pipeline, allowing the comparison to focus on DAO. A separate run of MO-GRPO’s published equal-weight formulation would additionally change objective priorities. We therefore use the matched GDPO comparison to assess the contribution beyond the component-wise normalization shared by these methods.
Q2. Routing cost and repeated sampling.
How many external calls does ReCal make per question?
On the same 3,125 test questions, ReCal makes 3,129 external calls and Router-R1 makes 3,148, corresponding to 1.0013 and 1.0074 calls per question. ReCal’s accuracy improvement therefore occurs at nearly identical external-call usage.
How do average API cost and inference latency compare with Router-R1?
The similar external-call usage suggests broadly comparable API cost and latency. ReCal’s reward and advantage calibration occurs during training, adding no inference-time model passes or API calls.
Does ReCal outperform repeated sampling without routing?
We evaluated Llama-3.1-70B-Instruct answering complete questions directly, without a router or subquestion decomposition. Sampling uses temperature 0.7, top-p 1.0, and a 512-token output limit. K=1 takes the first sample per question; K=3 applies majority voting to three independently seeded samples. Answers are normalized before voting, ties select the earliest valid answer, and ground truth is used only for scoring.
ReCal outperforms direct answering at comparable call counts and three-sample voting despite the latter using approximately three times as many external calls.
Q3. Why emphasize reward dispersion, and how reliable are small response groups?
Greater within-group reward dispersion indicates more differentiated feedback among sampled routing decisions, motivating greater optimization weight. We use it as an observable proxy for potentially informative routing differences. Clipping bounds the influence of this signal when estimates fluctuate.
At G=4, positive, uniform, and inverse weighting obtain average EM of 0.481, 0.458, and 0.451, respectively. The inverse control reciprocates the scaling factor and uses the same clipping bounds, [0.8, 1.25]. These results support the positive weighting direction.
We also evaluated G=2, obtaining 0.434 average EM. Although below 0.481 at G=4, it remains above Router-R1’s 0.414. ReCal therefore remains effective with only two samples per query, while benefiting from a larger response group. Together, these results support the empirical usefulness of bounded dispersion reweighting under small-group sampling.