- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RouterFactory provides a shared formulation and software stack for developing, evaluating, and serving LLM routers. Its xRouteBench spans generic, memory, caption-based vision, textual time-series, and personalized settings; the paper compares router families and demonstrates deployment and model selection within fixed multi-agent topologies.
Critical strengths
Common candidate outputs, metrics, and training-to-serving interfaces could make routing comparisons more reusable and reduce implementation confounds. The evaluation shows task-specific variation, includes a simple Elo baseline, and adds human preference data whose rankings differ from simulated-persona results. These comparisons are more useful than a claim that one router always wins. The integrated stack, multi-turn and personalized cases, and deployment examples are real work, even though their incremental research value needs stronger demonstration. A reusable infrastructure contribution can be valuable without a new routing algorithm.
Critical weaknesses
The main contribution is a unification scheme and SDK rather than a new routing algorithm, so its value depends on showing what the abstraction enables that existing routing infrastructure does not. LLMRouterBench already provides a collector, evaluator, and adapter workflow; RouterFactory's added multi-turn, personalized, and deployment breadth may be valuable, but Fig. 6 (p.14) is an illustrative subclass rather than a runnable comparison or evidence that the common interface improves router development or the validity of comparisons. Router performance gains in Table 2 could also be achieved by methods outside this abstraction, so they do not by themselves measure the SDK's benefit.
A versioned, runnable add-task or add-pool example with its prompts, parsers, splits, prices, and required code changes, plus a precise capability comparison, would make the infrastructure contribution assessable without demanding a new algorithm. The relevant standard is whether this scheme makes a useful new class of comparison or deployment reproducible with less work.
The common benchmark also imposes harmonization choices whose effects are not isolated. App. C/H already explains that images and video become shared captions, time series become textual/numeric descriptions (with long raw sequences truncated), and memory uses fixed top-five retrieval. These are defensible controls, but they test routing over common representations rather than native modality eligibility, full time-series inputs, or long-history selection. It would help to show which task rankings persist under the original benchmark representations and which depend on conversion.
Table 2's average equally weights seven columns even though Table 6 has 27 Video versus 3,729 Generic tests; GraphRouter's 45.46 versus Elo's 44.68 average is much closer than its Generic result. Multi-round/RL methods run at one point (§5.1), while single-turn methods enter the α/β sweep, so the claimed common frontier has a scope limit. Figs. 4–5 show frontier plots, but numerical points and the normalization of performance against dollar cost are needed to reproduce the ranking. Paired per-task uncertainty, a stated aggregation/workload target, and explicit cost scaling would show which conclusions survive the common representation.
Separately, App. H.1 (p.23) groups ten-choice MMLU-Pro with an A–D/four-option prompt, and p.27's multi-round template permits only A–E. If either template was executed, it could change labels and comparisons; if not, the documentation should say so. An executed serialized request, parser, and option-wise error counts would resolve this without assuming an evaluation error.
The human-preference study adds useful realism, but the best personalized router's 83.05% agreement exceeds the simple Elo router's 82.20% by only 0.85 points on eight held-out sessions from 15 users and 40 total sessions (Table 4). A session-clustered paired interval and same-architecture user-context control would show whether personalization drives that small difference. Likewise, the multi-turn methods share a Qwen2.5-3B controller (§5.1); a brief failure audit would help distinguish routing limits from controller or formatting limits before drawing broad multi-turn conclusions. These close results do not negate the resource, but they leave its claimed empirical improvement and the value of unification uncertain.
Questions for authors
-
Beyond Fig. 6's illustrative subclass, what reproducible task/pool change or comparison demonstrates a capability this unification provides beyond LLMRouterBench's collection/evaluation pipeline? Please identify the versioned artifact, configuration, and necessary code changes.
-
How do the documented caption, time-series, and retrieval transformations affect conclusions relative to the original benchmark semantics, and what aggregation target and cost normalization underlie the common α/β ranking? Can the numerical frontier points and paired per-task or session-clustered comparisons against Elo be provided?
-
Did the evaluated MMLU-Pro prompts and parser support all ten choices? Please show a serialized evaluated request and option-wise/format-error counts, clarifying whether App. H.1's A–D and the multi-round A–E templates were executed or abbreviated.
9 comments
Response 2
Continuation of our response to [W1, Q1] from the previous page:
These examples serve different purposes. The first demonstrates configuration-level task switching, whereas the second demonstrates the additional capability of transferring a trained routing policy into heterogeneous downstream workflows. The latter is the more direct evidence of RouterFactory's infrastructure contribution beyond conventional response collection and offline evaluation.
3. Required modifications and reproducibility details.
The benchmark artifact is xRouteBench (§3, Appendix C), with candidate models and per-token pricing documented in Appendix E (Table 12). Task transformations, evaluation metrics, and prompts are specified in Appendices C and H, while multi-agent topologies and their node-level prompts are provided in Appendices G and H.3.
Table 1 lists the required changes and the components reused in each workflow.
Table 1. Required changes and reusable components by operation.
| Operation | Required changes | Components reused |
|---|---|---|
| Switch between existing tasks | Change the dataset identifier (Listing B.1) | Router implementations, training pipeline, evaluator |
| Add a new task | Register its input transformation/prompt template and evaluation metric; provide the task data and splits | Data collection, router training, evaluation |
| Add a candidate model | Add its endpoint and input/output token prices to the pool configuration; collect the corresponding responses | Data engine, router interfaces, evaluation protocol |
| Add a new router | Implement route_single/route_batch in MetaRouter and, when needed, loss_func in BaseTrainer | Data construction, training execution, evaluation, deployment |
| Change multi-agent topology | Change the orchestration and node prompts (Listing B.2) | Trained router weights, routing algorithm, shared inference interface |
Importantly, adding a new candidate pool requires collecting the corresponding candidate responses and retraining policies when their learned parameters depend on that pool. Our claim is that the shared infrastructure and router interfaces do not need to be reimplemented, not that every extension requires zero computation or configuration changes.
4. Empirical evidence and scope of the infrastructure contribution.
The results in manuscript Table 8 demonstrate that the same trained routing implementations can be evaluated within five distinct multi-agent workflows, where the final workflow answer is assessed using the corresponding task metric. This is a comparison enabled by the shared infrastructure, rather than a claim that the infrastructure itself directly improves router accuracy.
Furthermore, §5.4 (manuscript Table 7) demonstrates a complementary deployment workflow using real-user supervision. Through Slack/OpenClaw, we collect 234 pairwise preference records from 15 users across 40 sessions, train routers on 32 sessions, and evaluate them on eight held-out sessions. PersonalizedRouter achieves 83.05% preference accuracy. This illustrates how the infrastructure supports a different supervision signal and interactive deployment setting within the same router development framework.
Together, these experiments demonstrate the practical value of the unification: heterogeneous routing policies can share experimental infrastructure and be reused across different supervision settings and downstream applications without rewriting their core routing implementations. We do not attribute the performance improvements of individual routers to the SDK itself, nor claim a measured reduction in development time.
5. Artifact release.
Upon acceptance, we will publicly release the complete RouterFactory code, xRouteBench data, trained router weights, and reproduction configurations.
W2: Harmonization choices and modality transformations are not isolated.
Response. We sincerely thank the reviewer for this valuable suggestion. Our benchmark adopts shared textual representations to ensure consistent candidate inputs and controlled comparisons across heterogeneous LLMs. To further investigate the impact of input representations, we provide additional analyses of multimodal routing and full-history memory access, examining how routing performance and rankings change under different input configurations.
(i) Sensitivity to multimodal representations.
We compare text-based routing with multimodal query embeddings using Qwen3-VL-Embedding-2B for MLPRouter, GraphRouter, and RouterDC. The multimodal setting pairs question text with original images (Geometry3K and MathVista), sampled video frames (Video), or time-series charts (TimeSeries). RouterDC additionally uses a trainable linear projection.
Response 3
Continuation of our response to W2 from the previous page:
Table 2. Router performance under different query representations (manuscript Table 6).
| Router | Representation | Geometry3K | MathVista | Video | TimeSeries |
|---|---|---|---|---|---|
| MLPRouter | Text | 27.87 | 34.00 | 29.63 | 56.69 |
| MLPRouter | Qwen3-VL | 29.51 | 35.00 | 29.63 | 57.48 |
| GraphRouter | Text | 42.62 | 50.00 | 22.22 | 62.99 |
| GraphRouter | Qwen3-VL | 27.87 | 28.00 | 22.22 | 63.78 |
| RouterDC | Text | 16.39 | 24.00 | 25.93 | 45.67 |
| RouterDC | Qwen3-VL | 26.23 | 35.00 | 25.93 | 59.06 |
The results reveal several important findings:
- Geometry3K and MathVista: Router rankings change with the input representation. MLPRouter ranks first on Geometry3K with multimodal inputs and ties RouterDC on MathVista. RouterDC improves by 9.84 and 11.00 points, respectively.
- Video: All three routers achieve identical scores under both representations, preserving their relative rankings.
- TimeSeries: GraphRouter remains the strongest method under both representations, while RouterDC improves by 13.39 points and overtakes MLPRouter under multimodal inputs.
These results demonstrate that routing representations can substantially influence performance and rankings, while some task-specific rankings remain stable across representations.
(ii) Sensitivity to full-history memory access.
We additionally conduct experiments on all 314 held-out LoCoMo questions, comparing fixed top-five retrieval with full-history access. The full-history setting preserves all 25–30 conversation sessions (509–568 utterances, approximately 19K–24K tokens) without retrieval or truncation.
Table 3. LoCoMo token-level F1 under different memory representations (Appendix C.2, manuscript Table C.2.1).
| Method | Paper Top-5 | Matched Top-5 | Full History | Δ (Full − Matched) |
|---|---|---|---|---|
| kNN-MultiRound | 24.70 | 28.09 | 36.02 | +7.93 |
| LLM-MultiRound | 24.60 | 28.41 | 36.99 | +8.58 |
| Router-R1 | 24.60 | 16.15 | 36.37 | +20.22 |
Compared with the matched Top-5 setting, full-history access consistently improves all three multi-turn routers, with gains of 7.93, 8.58, and 20.22 points, respectively. LLM-MultiRound remains the highest-performing method, while Router-R1 moves from third to second under full-history access. These results demonstrate that context availability influences both routing performance and relative rankings.
(iii) Benchmark harmonization and evaluation consistency.
Our benchmark standardizes heterogeneous tasks through task-specific transformations:
- Vision and Video: A fixed vision-language model generates shared textual descriptions, ensuring identical candidate inputs.
- TimeSeries: Generated pattern descriptions are combined with raw numerical values (up to 200 values), providing numerical evidence alongside higher-level temporal information. Our multimodal experiments further examine routing with time-series charts.
- Memory: A fixed Contriever-based top-five retrieval pipeline provides identical conversational evidence across candidate models, while our full-history experiments evaluate the effect of expanded context.
We have clarified these design choices in §3 ("Design principle"). Table 2 reports the multimodal comparison from §5.2 (manuscript Table 6). Table 3 reports the full-history comparison from Appendix C.2 (manuscript Table C.2.1).
Together, these experiments demonstrate that input representations and context availability play important roles in routing performance. The multimodal analysis identifies both stable and representation-sensitive rankings, while full-history access substantially improves memory-routing performance. These findings further strengthen the empirical analysis of RouterFactory under its unified evaluation framework.
Response 1
We sincerely thank the reviewer for the thoughtful and constructive evaluation of our work. We appreciate your recognition of the practical value of unifying diverse routing paradigms, particularly the support for multi-turn routing, personalized routing, and real-world deployment. We also appreciate your detailed suggestions regarding infrastructure reproducibility, benchmark validity, evaluation fairness, and empirical evidence, which have helped us further strengthen our manuscript. Below, we provide point-by-point responses to your comments and highlight the corresponding revisions and additional analyses.
Response. We sincerely thank the reviewer for the constructive feedback. We agree that the infrastructure contribution should be demonstrated through concrete, reproducible workflows rather than an illustrative interface or router performance gains alone. We have therefore expanded Appendix B.1 with two worked examples (Listings B.1–B.2) and clarify the distinction from LLMRouterBench below.
1. What does RouterFactory enable beyond LLMRouterBench?
We acknowledge that LLMRouterBench already provides configurable response collection, evaluation, and router adapters. Our contribution is not another collection/evaluation pipeline or a new routing algorithm, but a unified infrastructure that connects data construction, router training, evaluation, and deployment across single-turn, multi-turn, and personalized routing.
Specifically, RouterFactory provides three complementary capabilities:
MetaRouterandBaseTrainerinterfaces while retaining their own routing decisions and learning objectives (§4, Figure 6).The distinction is therefore not that these applications are impossible with existing tools, but that RouterFactory provides a common training-to-deployment abstraction for heterogeneous routing policies, rather than requiring separate algorithm-specific workflows.
2. Concrete reproducible examples added in Appendix B.1.
We have added two worked examples that specify the required configuration changes, routing calls, and corresponding experimental results.
Example 1: Switching tasks with a fixed candidate pool (Appendix B.1, Listing B.1). To reproduce the EloRouter versus Largest-LLM comparison on Video and Geometry3K, the user changes only
task = "video"totask = "multimodal_geometry3k"in Listing B.1, or equivalently changes the--datasetsargument in the command-line runner. Both experiments use the same 18 candidate models (Appendix E, Table 12), performance-only objective (alpha=1,beta=0), and evaluation procedure. The runner trains EloRouter on the selected task's training split and evaluates both methods on its corresponding test split. No modifications to either routing algorithm or the shared evaluator are required. Manuscript Table 3 reports EloRouter versus Largest-LLM scores of 25.93 vs. 22.22 on Video and 45.90 vs. 37.70 on Geometry3K. This example demonstrates that task-specific supervision and evaluation can be changed without rebuilding the routing pipeline.Example 2: Reusing a trained router across multi-agent workflows (Appendix B.1, Listing B.2). More importantly, §5.4 (manuscript Table 8) demonstrates the reuse of trained routing policies beyond offline benchmark evaluation. We train the routers on the Generic LLM Tasks training split and apply the same trained policies across five multi-agent topologies: Star, Tree, Graph, Chain, and Plan-Exec-Sum. Each agent node independently selects a model through
router.route_single({"query": node_prompt})["model_name"]. In Star, the planner assigns subtasks to actors, whereas in Chain, each subsequent agent receives and verifies the preceding agent's output. Switching between these topologies changes the orchestration and node prompts, but not the trained router weights, routing implementation, or shared routing interface. For example, MFRouter achieves 75.40 on Star and 78.60 on Chain, compared with 69.00 for Largest-LLM in both settings. Across all five topologies, six of the seven learned routers outperform Largest-LLM on average; MFRouter achieves 76.48 versus 71.48 for Largest-LLM.