- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the limitations of existing LLM Routers, which typically rely on specific task distributions and candidate model pools, making them difficult to adapt to dynamic routing environments. To overcome these challenges, the paper proposes RouteFM, a general-purpose routing model based on cross-environment pretraining. RouteFM constructs capability representations of anonymous candidate models from their historical behavior, enabling the pretrained Router to adapt to new routing environments without parameter updates. Experimental results demonstrate that RouteFM achieves promising performance in both In-Domain Routing and Cross-Modal Transfer settings.
Critical strengths
S1. This paper investigates cross-environment generalization from the perspective of reusable routing capabilities, offering clear research value and practical significance.
S2. The design of RouteFM is well-motivated and reasonable, and the proposed method achieves competitive experimental results.
Critical weaknesses
S1. Insufficient disclosure and analysis of the candidate model pools. The paper lacks detailed statistics on the candidate models used during pretraining and testing. This may weaken the paper's claims regarding generalization across different candidate model pools.
S2. Lack of ablation studies on key components. The paper lacks necessary ablation studies on several critical design choices, such as the Capability Profile Branch and the loss function.
Questions for authors
- Could the authors provide more details about the candidate model pools?
(a) Which candidate models are used during pretraining? What is the size and diversity of the overall model pool in terms of model families, parameter scales, and the composition of open-source and closed-source models?
(b) To what extent do the candidate models used for testing overlap with those used for pretraining? How do their distributions differ in terms of model families and parameter scales?
(c) How is the candidate model pool constructed for each pretraining episode? Are models randomly sampled, or is a specific sampling strategy employed based on model capabilities, families, or other factors?
- Could the authors provide additional ablation studies on the following key components?
(a) Capability Profile Branch: Table 2 includes an ablation of Target-to-Context Attention, but lacks a corresponding ablation of the Capability Profile Branch. The authors are encouraged to evaluate a variant that entirely removes capability profiling and retains only the Raw Context Branch, to assess the independent contribution and necessity of the Capability Profile.
(b) Training Objectives: RouteFM employs three training objectives: Point-wise Regression, Pairwise Ranking, and Routing-aware Regret. The authors are encouraged to ablate each loss term individually to assess its contribution to the final routing performance.
3 comments
Thanks for the authors' further clarification and ablation studies. This is an interesting attempt to bring foundation model pretraining to LLM routing. I therefore tend to keep my overall positive score.
Thank you for your positive feedback
We sincerely thank the reviewer for their time and thoughtful consideration. We greatly appreciate your positive feedback and continued support of our work.
Response to Reviewer AEru
We sincerely thank the reviewer for recognizing the research value, practical significance, and effectiveness of RouteFM, as well as for the constructive suggestions. In response, we provide detailed statistics on candidate model pools and training–test overlap, clarify our episodic pretraining procedure, and conduct additional ablation studies on key architectural components and training objectives. We address these concerns below.
R1: Could the authors provide more details about the candidate model pools?
We thank the reviewer for this valuable suggestion. We have added detailed candidate pool statistics and training–test overlap analysis in the revised Appendix B.1, along with further clarification of the episodic pretraining procedure in Appendix B.2.
(a) RouteFM is not trained on a single fixed model pool. Instead, it combines task-specific candidate pools from LLMRouterBench, RouterBench, RouterEval, and MixInstruct. Together, these sources contain 182 unique models across 15 major model families, including Llama, Qwen, Mistral, DeepSeek, Gemma, GPT, and Claude. The pool includes 170 open-weight models and 12 proprietary API models, covering a wide range of parameter scales.
(b) None of the nine candidate models in MMR-Bench appears in pretraining (0/9 exact model-ID overlap), although some model families overlap (5/6 families). Importantly, RouteFM treats all candidate models as anonymous, without access to their names, identities, or provider information. Instead, it infers candidate capabilities solely from behavioral observations, enabling adaptation to unseen models without relying on fixed model identities.
(c) As detailed in Appendix B.2, each pretraining episode is constructed from a single source and task, with 6–24 candidate models and varying behavioral context sizes. Candidate ordering is randomly permuted to prevent dependence on fixed model positions. Target queries are sampled from three regimes: Natural (65%), Opportunity (20%), and Boundary (15%), encouraging the router to learn target-conditioned model comparison.
R2: Could the authors provide additional ablation studies on the following key components?
We thank the reviewer for these valuable suggestions. Following the reviewer's suggestions, we conduct additional ablation studies on MMR-Bench to evaluate the contributions of the Capability Profile Branch and the three training objectives. The new results have been incorporated into Table 2 of the revised manuscript.
(a) Capability Profile BranchRemoving capability profiling slightly improves performance at K=8, but consistently reduces performance at larger context sizes, with a 0.94-point drop in the Large setting. This suggests that direct access to raw observations can be sufficient when context is extremely limited, while capability profiling becomes increasingly valuable as more behavioral evidence is available, providing a compact global summary that complements fine-grained context retrieval.
(b) Training ObjectivesThe three objectives make complementary contributions. Removing point-wise regression consistently degrades performance, particularly at K=8 (−1.66 points), highlighting its importance for accurate capability estimation. Pairwise ranking provides substantial gains at larger context sizes, especially K=32 (1.80 points) and Large (2.17 points), demonstrating the importance of explicitly learning relative candidate preferences. Routing-aware regret provides additional improvements at K=8 (0.56 points) and Large (0.85 points), although its contribution varies across context sizes.
Overall, these results clarify the distinct roles of capability profiling and the three training objectives, as well as how their contributions vary with the amount of available behavioral evidence.