- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper addresses the problem that most existing LLM routers treat the question as a whole, without judging whether the question needs to be decomposed. To this end, the paper proposes Granule-R1, which trains a small model as the router with RL. Through the actions Think, Decompose, Search, Commit, and Answer in a multi-turn interaction, the router decides (i) whether to answer directly, send the whole question to a model, or decompose it into subquestions, (ii) which model to send each subquestion to, and (iii) how to commit subanswers and integrate them into the final answer. The paper trains the router with PPO, using a reward composed of answer correctness, format, decomposition quality, efficiency based on model prices, and adjustments after failures. The paper follows the setting of Router-R1, including its models, datasets, hyperparameters, and results, and compares on several general and multi-hop QA benchmarks. It claims to improve accuracy while reducing routing cost, and that a pool of only small models can match whole-question routing with larger models.
Critical strengths
None. I cannot identify any strength of this paper. Please see the Weaknesses below.
Critical weaknesses
-
The proposed method is not very novel. (i) The paper builds directly on Router-R1, with the same training data, models, and evaluation setting. Router-R1 can already send subquestions to different models during multi-turn interaction, and the authors acknowledge in the appendix that Router-R1 performs implicit decomposition. This paper only turns decomposition into an explicit action and adds some reward shaping. (ii) Decomposing a question into subtasks and assigning them to different models has already been done by R2-Reasoner and RouteGoT. R2-Reasoner is only compared in the appendix through a zero-shot transfer that the authors themselves acknowledge is not a fully matched comparison. (iii) The paper is framed with terms such as self-adaptive, introspection, and capability boundaries, but the actual mechanism is just a decompose tag and a few reward terms.
-
The credibility of the experimental results is questionable. (i) The EM and F1 of all baselines in Table 1, as well as the costs of all baselines in Table 7, are all the same as the numbers published in the Router-R1 paper. However, the appendix says that "GraphRouter is independently trained" and "RouterDC uses its public checkpoint", which is inconsistent with the numbers being exactly the same. (ii) Some conclusions are inaccurate and overstated. For example, the paper states that "it surpasses the strongest 27B-access multi-round router by 0.02 EM points (0.303 vs. 0.301)", but the difference is 0.002 rather than 0.02, and such a small difference does not support the word "surpasses". (iii) Apart from Appendix A.8, the main results, the small-pool comparison, and the ablations are all single runs without standard deviations or confidence intervals. Many improvements over the baselines are very small and may not be significant.
-
The introduction raises two questions, how to decompose the original question and where the capability boundaries of the available models lie for different subquestions, but the experiments do not decouple these two questions and analyze them separately. (i) If only decomposition is used without model routing, e.g., with a single model in the pool, could the performance gain come from decomposition itself? The paper has no such experiment and does not compare with single-model decomposition methods such as Least-to-Most and Self-Ask. (ii) If the question is decomposed randomly or by fixed rules before routing, would there also be a large gain? The paper does not test this either. (iii) The only support for capability boundaries is statistics such as call ratio, with no evidence that the router actually selects the right model. The ablation study cannot tell whether the gain comes from decomposition, from routing, or from their combination.
Questions for authors
- The process is a multi-turn loop in which the router generates reasoning and actions over an ever-growing context at each turn, but the cost only counts the output tokens of external models and ignores the router's own computation, the overhead of multiple external calls, and the end-to-end latency. What is the complete cost once the router's own computation is included? Please report the distribution of the router's input and output token lengths (maximum, mean, and median) and the end-to-end latency, and compare them with Router-R1.
- The router has a heavy reasoning load. It judges difficulty, decomposes the question, selects models, judges the returned results, commits subanswers, and integrates the final answer, all of which are reasoning-heavy, yet the router is only a 3B model. Table 15 shows that about 94% of the sampled errors come from the router itself and only 6% from the limited capability of the candidate models, and Table 10 shows that allowing replanning increases format violations and lowers accuracy, suggesting that the router gets confused as the task chain grows longer. Could the authors analyze the router's reasoning capability, e.g., the accuracy of its decomposition, model selection, and subanswer commitment, or the results with a larger router?
- The reward design is complex and sensitive. There are 5 reward terms and, together with the sub-terms of the decomposition reward, about 13 weights, all tuned manually in preliminary experiments without any sensitivity analysis. How do accuracy and cost change as the reward weights varies? Could the authors provide an accuracy–cost curve and a sensitivity analysis of the reward weights?
- Which numbers in Tables 1 and 7 are copied from the published Router-R1 results, and which are re-run by the authors? If GraphRouter was trained independently and RouterDC was re-evaluated with its public checkpoint, why are the results digit-for-digit identical to those published by Router-R1?
- How much does performance improve if the pool contains only one model and only decomposition is used? What are the results if the question is decomposed randomly or by fixed rules before routing?