- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors identify flaws in existing agentic routing schemes: they either make incremental, reactive decisions without sustained global guidance—leading to shortsighted planning—or fix the task structure prior to execution, unable to re-evaluate based on feedback—resulting in rigid planning. Therefore, they propose AdaRouter, an adaptive planning framework for agentic routing designed to coordinate calls to a heterogeneous, fixed set of LLM "workers," thereby balancing task accuracy against inference costs. AdaRouter addresses this gap by maintaining an explicit solution plan and dynamically adjusting it in real-time as feedback from workers arrives.
The method was evaluated across 11 benchmarks covering mathematics (GSM8K, MATH), code (HumanEval, MBPP), general knowledge/common sense (MMLU, GPQA), multi-hop QA (HotpotQA, 2WikiMultiHopQA, MuSiQue), and world knowledge (NQ, TriviaQA), as well as on four held-out datasets for out-of-distribution (OOD) testing. AdaRouter achieved a macro-average accuracy of 65.4%, outperforming the strongest routing baseline by 9.5 percentage points (with gains of 12.5 and 8.7 percentage points in code and multi-hop tasks, respectively); meanwhile, it required fewer worker node invocations (averaging 1.5) than multi-turn interaction baselines and incurred lower total costs than GraphPlanner and Prompt-LLM. Across various \alpha settings, the method consistently maintained a position on the Pareto frontier regarding the trade-off between accuracy and cost.
Critical strengths
The proposed AdaRouter method employs a scientifically grounded, two-level training framework that effectively resolves the "credit-assignment" challenge. Because the "plan-adaptation" mechanism alters both current worker invocations and future execution trajectories—making it difficult to attribute final outcomes to specific individual decisions—AdaRouter introduces a complementary supervision strategy: trajectory-level GRPO (which holistically evaluates correctness, invocation costs, and adaptation quality) provides a global signal, while token-level OPSD offers dense, token-weight-based guidance on when and how to execute "REFINE" or "REVISE" operations. This design is significant as it enables the automated learning of adaptive behaviors rather than relying on manually defined rules; consequently, AdaRouter achieves a macro-averaged accuracy of 65.4%—surpassing the strongest baseline by 9.5 percentage points—with significantly fewer worker invocations than multi-round baselines. This simultaneous enhancement of performance and efficiency embodies the core value of the "routing" mechanism.
Critical weaknesses
-
This evaluation relies on a single, fixed router configuration and does not examine the system's scalability. All primary variants of AdaRouter utilize Qwen3-4B-Instruct as the trainable router, paired with a fixed pool of eight frozen worker models (ranging in size from 7B to 119B parameters). Although generalization experiments covered unseen tasks and unseen worker LLMs, the router strategy itself was not tested under varying conditions; there is currently no evidence regarding how this adaptive planning paradigm performs when employing a router backbone that is larger, more capable, or architecturally different. This is a crucial point, as the study's core claims—such as state-of-the-art accuracy, superior cost-accuracy trade-offs, and robust generalization—are derived from a specific 4B-parameter strategy; if these advantages are heavily dependent on the router's capacity or the specific configuration of the reward/evaluation mechanism, the paradigm's practical applicability could be far more limited than the paper suggests.
-
Determining the REFINE/REVISE reward involves nuanced qualitative judgments—such as whether a query is self-contained and free of unverified information, and whether the revised observation yields "useful evidence"—whereas trajectory analysis relies on aggregate counts derived from "single-model offline annotation." The report does not mention any validation regarding human inter-rater agreement or evaluator quality. This is critical because these rewards serve not merely as evaluation metrics but as training signals: if the automated evaluator harbors noise or systematic bias, then the REFINE/REVISE behaviors learned by the model—as well as key conclusions regarding the reduction of unnecessary plan revisions and the increased utility of the first worker invocation following a revision—might well be artifacts of the evaluation mechanism rather than genuine improvements in routing quality.
Questions for authors
The rewards and trajectory metrics within the REFINE/REVISE mechanism rely on nuanced automated judgments—such as determining whether a query is "self-contained" and "free of unverified information," whether the remaining plan is "inappropriate," and whether the revised observations provide "useful evidence." Did you validate these evaluators against human-annotated data (e.g., by examining agreement rates or conducting error analysis)? If the evaluators are noisy, to what extent might the learned revision behaviors—and the reported outcome of "reduced unnecessary revisions"—simply be artifacts of the evaluators' characteristics rather than genuine improvements in routing?
The reported total cost accounts solely for the invocation overhead of the worker nodes. Given that the Router itself is a 4-billion-parameter Large Language Model (LLM) responsible for generating plans, queries, and revisions across multiple rounds of interaction, why was the Router's own inference cost excluded? How would the Pareto analysis results regarding the trade-off between accuracy and cost (see Figure 3) change if this cost were included?
Follow-up on Our Rebuttal
Dear Reviewer HPCu,
Thank you again for your careful review and constructive feedback on our paper. We have submitted our point-by-point responses to the concerns raised in your review and would greatly appreciate it if you could take a moment to look through them.
We hope our clarifications address your questions and provide a better understanding of our proposed method and experimental results. If any concerns remain or further clarification would be helpful, please do not hesitate to let us know. We would be happy to continue the discussion.
Thank you again for your time and consideration.
Best regards,
The Authors