ReCal: Reward Calibration for RL-based LLM Routing

Large language model (LLM) routing has emerged as an effective paradigm for leveraging the complementary strengths of multiple LLMs through dynamic model and reasoning-strategy selection. Recent reinforcement learning (RL)-based routing methods further improve routing quality by optimizing routing policies from interaction feedback. However, they still struggle to provide informative and comparable learning signals under heterogeneous tasks with varying difficulty. In practice, multiple objectives (e.g., correctness, format behavior) are aggregated into a single scalar reward, leading to ambiguous credit assignment and conflicting optimization signals. Moreover, reward distributions vary substantially across instances and datasets, making their optimization contributions difficult to compare and potentially leading to imbalanced policy updates. To address these issues, we propose **ReCal**, a **Re**ward **Cal**ibration framework for RL-based LLM routing. We first introduce a hierarchical reward decomposition mechanism with component-wise advantage estimation. We further propose a distribution-aware optimization strategy that calibrates optimization contributions through variance-aware reweighting and per-dataset normalization. Experiments on seven QA benchmarks show that ReCal consistently improves routing performance and achieves more stable training dynamics compared with baselines. Additional experiments on mathematical reasoning further show that this calibration procedure can be applied beyond QA through task-specific reward instantiation.