acceptodds
Under review as a conference paper at ICLR 2027

RouterFactory: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

Abstract

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective LLM deployment. Existing routers span binary quality predictors, cost-aware cascades, graph-based routers, and agentic routers, yet their diverse formalisms and incompatible implementations, coupled with the absence of a standardized evaluation pipeline, hinder fair comparison and further extension. In this paper, we present a unified formulation of LLM routing as a sequential decision process. Under this formulation, a router can be characterized in terms of five types of components: context encoders, model encoders, scoring functions, decision rules, and learning signals. Existing methods can then be organized into three families: single-turn, multi-turn, and personalized routing. Building on this formulation, we develop an automated pipeline that constructs routing supervision by systematically running a pool of candidate models across benchmarks and evaluates routers in terms of both response quality and inference cost under a unified protocol. The resulting benchmark, xRouteBench, spans generic LLM tasks, memory-augmented, vision (image and video), time-series, and personalized routing scenarios. Grounded in the formulation and pipeline, we present RouterFactory, an open-source infrastructure for standardized and modular implementation of LLM routers, where users can add a new router by implementing only a routing method and a loss function and access built-in implementations of 16 representative routers spanning all three families. Using the library and benchmark, we conduct a systematic empirical study of LLM routing and find that learned routers achieve a 17.4% relative improvement over the largest fixed-model baseline, router rankings reverse in favor of lightweight designs under tighter cost constraints, and user-conditioned routing delivers consistent personalization gains.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Routing, Model Routing, RouteBench

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GBpJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

RouterFactory provides a shared formulation and software stack for developing, evaluating, and serving LLM routers. Its xRouteBench spans generic, memory, caption-based vision, textual time-series, and personalized settings; the paper compares router families and demonstrates deployment and model selection within fixed multi-agent topologies.

Critical strengths

Common candidate outputs, metrics, and training-to-serving interfaces could make routing comparisons more reusable and reduce implementation confounds. The evaluation shows task-specific variation, includes a simple Elo baseline, and adds human preference data whose rankings differ from simulated-persona results. These comparisons are more useful than a claim that one router always wins. The integrated stack, multi-turn and personalized cases, and deployment examples are real work, even though their incremental research value needs stronger demonstration. A reusable infrastructure contribution can be valuable without a new routing algorithm.

Critical weaknesses

The main contribution is a unification scheme and SDK rather than a new routing algorithm, so its value depends on showing what the abstraction enables that existing routing infrastructure does not. LLMRouterBench already provides a collector, evaluator, and adapter workflow; RouterFactory's added multi-turn, personalized, and deployment breadth may be valuable, but Fig. 6 (p.14) is an illustrative subclass rather than a runnable comparison or evidence that the common interface improves router development or the validity of comparisons. Router performance gains in Table 2 could also be achieved by methods outside this abstraction, so they do not by themselves measure the SDK's benefit.

A versioned, runnable add-task or add-pool example with its prompts, parsers, splits, prices, and required code changes, plus a precise capability comparison, would make the infrastructure contribution assessable without demanding a new algorithm. The relevant standard is whether this scheme makes a useful new class of comparison or deployment reproducible with less work.

The common benchmark also imposes harmonization choices whose effects are not isolated. App. C/H already explains that images and video become shared captions, time series become textual/numeric descriptions (with long raw sequences truncated), and memory uses fixed top-five retrieval. These are defensible controls, but they test routing over common representations rather than native modality eligibility, full time-series inputs, or long-history selection. It would help to show which task rankings persist under the original benchmark representations and which depend on conversion.

Table 2's average equally weights seven columns even though Table 6 has 27 Video versus 3,729 Generic tests; GraphRouter's 45.46 versus Elo's 44.68 average is much closer than its Generic result. Multi-round/RL methods run at one point (§5.1), while single-turn methods enter the α/β sweep, so the claimed common frontier has a scope limit. Figs. 4–5 show frontier plots, but numerical points and the normalization of performance against dollar cost are needed to reproduce the ranking. Paired per-task uncertainty, a stated aggregation/workload target, and explicit cost scaling would show which conclusions survive the common representation.

Separately, App. H.1 (p.23) groups ten-choice MMLU-Pro with an A–D/four-option prompt, and p.27's multi-round template permits only A–E. If either template was executed, it could change labels and comparisons; if not, the documentation should say so. An executed serialized request, parser, and option-wise error counts would resolve this without assuming an evaluation error.

The human-preference study adds useful realism, but the best personalized router's 83.05% agreement exceeds the simple Elo router's 82.20% by only 0.85 points on eight held-out sessions from 15 users and 40 total sessions (Table 4). A session-clustered paired interval and same-architecture user-context control would show whether personalization drives that small difference. Likewise, the multi-turn methods share a Qwen2.5-3B controller (§5.1); a brief failure audit would help distinguish routing limits from controller or formatting limits before drawing broad multi-turn conclusions. These close results do not negate the resource, but they leave its claimed empirical improvement and the value of unification uncertain.

Questions for authors
  1. Beyond Fig. 6's illustrative subclass, what reproducible task/pool change or comparison demonstrates a capability this unification provides beyond LLMRouterBench's collection/evaluation pipeline? Please identify the versioned artifact, configuration, and necessary code changes.

  2. How do the documented caption, time-series, and retrieval transformations affect conclusions relative to the original benchmark semantics, and what aggregation target and cost normalization underlie the common α/β ranking? Can the numerical frontier points and paired per-task or session-clustered comparisons against Elo be provided?

  3. Did the evaluated MMLU-Pro prompts and parser support all ten choices? Please show a serialized evaluated request and option-wise/format-error counts, clarifying whether App. H.1's A–D and the multi-round A–E templates were executed or abbreviated.

9 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the thoughtful and constructive evaluation of our work. We appreciate your recognition of the practical value of unifying diverse routing paradigms, particularly the support for multi-turn routing, personalized routing, and real-world deployment. We also appreciate your detailed suggestions regarding infrastructure reproducibility, benchmark validity, evaluation fairness, and empirical evidence, which have helped us further strengthen our manuscript. Below, we provide point-by-point responses to your comments and highlight the corresponding revisions and additional analyses.

[W1, Q1]: Infrastructure contribution and reproducible workflows beyond LLMRouterBench

Response. We sincerely thank the reviewer for the constructive feedback. We agree that the infrastructure contribution should be demonstrated through concrete, reproducible workflows rather than an illustrative interface or router performance gains alone. We have therefore expanded Appendix B.1 with two worked examples (Listings B.1–B.2) and clarify the distinction from LLMRouterBench below.

1. What does RouterFactory enable beyond LLMRouterBench?

We acknowledge that LLMRouterBench already provides configurable response collection, evaluation, and router adapters. Our contribution is not another collection/evaluation pipeline or a new routing algorithm, but a unified infrastructure that connects data construction, router training, evaluation, and deployment across single-turn, multi-turn, and personalized routing.

Specifically, RouterFactory provides three complementary capabilities:

  • Unified development and training: Different routing methods share the MetaRouter and BaseTrainer interfaces while retaining their own routing decisions and learning objectives (§4, Figure 6).
  • Cross-paradigm evaluation: Single-turn, multi-turn, and personalized routers can be evaluated through a common framework, with routing-specific supervision and trajectory costs handled accordingly (§§2–4).
  • Reusable deployment: Trained routing policies can be integrated into interactive applications and multi-agent workflows through the same routing interface, without reimplementing the underlying routing algorithms (§5.4).

The distinction is therefore not that these applications are impossible with existing tools, but that RouterFactory provides a common training-to-deployment abstraction for heterogeneous routing policies, rather than requiring separate algorithm-specific workflows.

2. Concrete reproducible examples added in Appendix B.1.

We have added two worked examples that specify the required configuration changes, routing calls, and corresponding experimental results.

  • Example 1: Switching tasks with a fixed candidate pool (Appendix B.1, Listing B.1). To reproduce the EloRouter versus Largest-LLM comparison on Video and Geometry3K, the user changes only task = "video" to task = "multimodal_geometry3k" in Listing B.1, or equivalently changes the --datasets argument in the command-line runner. Both experiments use the same 18 candidate models (Appendix E, Table 12), performance-only objective (alpha=1, beta=0), and evaluation procedure. The runner trains EloRouter on the selected task's training split and evaluates both methods on its corresponding test split. No modifications to either routing algorithm or the shared evaluator are required. Manuscript Table 3 reports EloRouter versus Largest-LLM scores of 25.93 vs. 22.22 on Video and 45.90 vs. 37.70 on Geometry3K. This example demonstrates that task-specific supervision and evaluation can be changed without rebuilding the routing pipeline.

  • Example 2: Reusing a trained router across multi-agent workflows (Appendix B.1, Listing B.2). More importantly, §5.4 (manuscript Table 8) demonstrates the reuse of trained routing policies beyond offline benchmark evaluation. We train the routers on the Generic LLM Tasks training split and apply the same trained policies across five multi-agent topologies: Star, Tree, Graph, Chain, and Plan-Exec-Sum. Each agent node independently selects a model through router.route_single({"query": node_prompt})["model_name"]. In Star, the planner assigns subtasks to actors, whereas in Chain, each subsequent agent receives and verifies the preceding agent's output. Switching between these topologies changes the orchestration and node prompts, but not the trained router weights, routing implementation, or shared routing interface. For example, MFRouter achieves 75.40 on Star and 78.60 on Chain, compared with 69.00 for Largest-LLM in both settings. Across all five topologies, six of the seven learned routers outperform Largest-LLM on average; MFRouter achieves 76.48 versus 71.48 for Largest-LLM.

AuthorsOpenReview ↗

Response 2

Continuation of our response to [W1, Q1] from the previous page:

These examples serve different purposes. The first demonstrates configuration-level task switching, whereas the second demonstrates the additional capability of transferring a trained routing policy into heterogeneous downstream workflows. The latter is the more direct evidence of RouterFactory's infrastructure contribution beyond conventional response collection and offline evaluation.

3. Required modifications and reproducibility details.

The benchmark artifact is xRouteBench (§3, Appendix C), with candidate models and per-token pricing documented in Appendix E (Table 12). Task transformations, evaluation metrics, and prompts are specified in Appendices C and H, while multi-agent topologies and their node-level prompts are provided in Appendices G and H.3.

Table 1 lists the required changes and the components reused in each workflow.

Table 1. Required changes and reusable components by operation.

OperationRequired changesComponents reused
Switch between existing tasksChange the dataset identifier (Listing B.1)Router implementations, training pipeline, evaluator
Add a new taskRegister its input transformation/prompt template and evaluation metric; provide the task data and splitsData collection, router training, evaluation
Add a candidate modelAdd its endpoint and input/output token prices to the pool configuration; collect the corresponding responsesData engine, router interfaces, evaluation protocol
Add a new routerImplement route_single/route_batch in MetaRouter and, when needed, loss_func in BaseTrainerData construction, training execution, evaluation, deployment
Change multi-agent topologyChange the orchestration and node prompts (Listing B.2)Trained router weights, routing algorithm, shared inference interface

Importantly, adding a new candidate pool requires collecting the corresponding candidate responses and retraining policies when their learned parameters depend on that pool. Our claim is that the shared infrastructure and router interfaces do not need to be reimplemented, not that every extension requires zero computation or configuration changes.

4. Empirical evidence and scope of the infrastructure contribution.

The results in manuscript Table 8 demonstrate that the same trained routing implementations can be evaluated within five distinct multi-agent workflows, where the final workflow answer is assessed using the corresponding task metric. This is a comparison enabled by the shared infrastructure, rather than a claim that the infrastructure itself directly improves router accuracy.

Furthermore, §5.4 (manuscript Table 7) demonstrates a complementary deployment workflow using real-user supervision. Through Slack/OpenClaw, we collect 234 pairwise preference records from 15 users across 40 sessions, train routers on 32 sessions, and evaluate them on eight held-out sessions. PersonalizedRouter achieves 83.05% preference accuracy. This illustrates how the infrastructure supports a different supervision signal and interactive deployment setting within the same router development framework.

Together, these experiments demonstrate the practical value of the unification: heterogeneous routing policies can share experimental infrastructure and be reused across different supervision settings and downstream applications without rewriting their core routing implementations. We do not attribute the performance improvements of individual routers to the SDK itself, nor claim a measured reduction in development time.

5. Artifact release.

Upon acceptance, we will publicly release the complete RouterFactory code, xRouteBench data, trained router weights, and reproduction configurations.


W2: Harmonization choices and modality transformations are not isolated.

Response. We sincerely thank the reviewer for this valuable suggestion. Our benchmark adopts shared textual representations to ensure consistent candidate inputs and controlled comparisons across heterogeneous LLMs. To further investigate the impact of input representations, we provide additional analyses of multimodal routing and full-history memory access, examining how routing performance and rankings change under different input configurations.

(i) Sensitivity to multimodal representations.

We compare text-based routing with multimodal query embeddings using Qwen3-VL-Embedding-2B for MLPRouter, GraphRouter, and RouterDC. The multimodal setting pairs question text with original images (Geometry3K and MathVista), sampled video frames (Video), or time-series charts (TimeSeries). RouterDC additionally uses a trainable linear projection.

AuthorsOpenReview ↗

Response 3

Continuation of our response to W2 from the previous page:

Table 2. Router performance under different query representations (manuscript Table 6).

RouterRepresentationGeometry3KMathVistaVideoTimeSeries
MLPRouterText27.8734.0029.6356.69
MLPRouterQwen3-VL29.5135.0029.6357.48
GraphRouterText42.6250.0022.2262.99
GraphRouterQwen3-VL27.8728.0022.2263.78
RouterDCText16.3924.0025.9345.67
RouterDCQwen3-VL26.2335.0025.9359.06

The results reveal several important findings:

  • Geometry3K and MathVista: Router rankings change with the input representation. MLPRouter ranks first on Geometry3K with multimodal inputs and ties RouterDC on MathVista. RouterDC improves by 9.84 and 11.00 points, respectively.
  • Video: All three routers achieve identical scores under both representations, preserving their relative rankings.
  • TimeSeries: GraphRouter remains the strongest method under both representations, while RouterDC improves by 13.39 points and overtakes MLPRouter under multimodal inputs.

These results demonstrate that routing representations can substantially influence performance and rankings, while some task-specific rankings remain stable across representations.

(ii) Sensitivity to full-history memory access.

We additionally conduct experiments on all 314 held-out LoCoMo questions, comparing fixed top-five retrieval with full-history access. The full-history setting preserves all 25–30 conversation sessions (509–568 utterances, approximately 19K–24K tokens) without retrieval or truncation.

Table 3. LoCoMo token-level F1 under different memory representations (Appendix C.2, manuscript Table C.2.1).

MethodPaper Top-5Matched Top-5Full HistoryΔ (Full − Matched)
kNN-MultiRound24.7028.0936.02+7.93
LLM-MultiRound24.6028.4136.99+8.58
Router-R124.6016.1536.37+20.22

Compared with the matched Top-5 setting, full-history access consistently improves all three multi-turn routers, with gains of 7.93, 8.58, and 20.22 points, respectively. LLM-MultiRound remains the highest-performing method, while Router-R1 moves from third to second under full-history access. These results demonstrate that context availability influences both routing performance and relative rankings.

(iii) Benchmark harmonization and evaluation consistency.

Our benchmark standardizes heterogeneous tasks through task-specific transformations:

  • Vision and Video: A fixed vision-language model generates shared textual descriptions, ensuring identical candidate inputs.
  • TimeSeries: Generated pattern descriptions are combined with raw numerical values (up to 200 values), providing numerical evidence alongside higher-level temporal information. Our multimodal experiments further examine routing with time-series charts.
  • Memory: A fixed Contriever-based top-five retrieval pipeline provides identical conversational evidence across candidate models, while our full-history experiments evaluate the effect of expanded context.

We have clarified these design choices in §3 ("Design principle"). Table 2 reports the multimodal comparison from §5.2 (manuscript Table 6). Table 3 reports the full-history comparison from Appendix C.2 (manuscript Table C.2.1).

Together, these experiments demonstrate that input representations and context availability play important roles in routing performance. The multimodal analysis identifies both stable and representation-sensitive rankings, while full-history access substantially improves memory-routing performance. These findings further strengthen the empirical analysis of RouterFactory under its unified evaluation framework.

Reviewer_ia53OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RouterFactory, an open-source library for developing, evaluating and deploying LLM routers. The authors describe routing as a sequential decision process, where each router is made of five components (context encoder, model encoder, scoring function, decision rule and learning signal), and use it to group routers into single-turn, multi-turn and personalized ones. They also build a benchmark, xRouteBench, evaluating 18 open-weight models on generic tasks, memory, vision, video, time series, and personalization. They show that learned routers are 17.4% better than always calling the largest model, that the ranking of the routers changes with cost, that multi-turn routing does not beat single-turn routing, and that personalized routers help. The routers are also deployed on Slack with real users and inside multi-agent systems.

Critical strengths

(1) The problem is important and practical. Every routing paper uses its own models, data and code, so it is hard to compare results across papers. A library that runs all routers on the same queries, models and costs could be a useful tool for the community, and adding a new task or model only needs a config change.

(2) The experiments cover many settings, and some of the findings are interesting. Fig. 4 shows that the best router changes a lot with the cost weight (RouterDC goes from 1st to 10th on the generic tasks), and Table 4 shows that GMTRouter, the best router with the persona judge, is only sixth with real users. I also like that the authors collected preferences from real users, which is rare in routing papers.

Critical weaknesses

(1) I find the paper really hard to follow. For example, I couldn't understand the decision rule sentence in Sec. 2.1 (“greedy arg max is the default is optimal for Eq. 1”). Also, the abstract says that rankings “reverse in favor of lightweight designs” without saying which designs are lightweight. I would like to see a careful proofreading and more precise claims, please.

(2) In my opinion, the analysis of the main result is weak. The 17.4% gain compares the best of nine routers (45.46) with Largest-LLM (38.72), which is barely above Smallest-LLM (37.94), and it is not clear to me which 671B model (DeepSeek or Cogito) Largest-LLM is. Also, EloRouter ignores the query and still gets 44.68, so most of the gain can be reached without routing per query? I would like to see the best single model and an oracle router as baselines, similar to what's done in RouterBench.

(3) I am not convinced that xRouteBench tests the settings it is built for. Sec. 3 motivates it with “image and video inputs that only a subset of candidate models can process”, but all images and videos are converted to text, and the memory track only uses up to five retrieved items. These tracks are also very small (video is 0.6% of the test queries in Fig. 2) but count as much as the generic track (78%) in the Table 2 average. I would like to see real multimodal inputs, the full memory history and confidence intervals.

(4) I think the paper misses closely related work. RouteLLM already offers an open framework for serving and evaluating routers but is cited only as a router, while (i) RouterArena which is an open platform for comparing routers, and (ii) MasRouter which routes LLMs in multi-agent systems, are not cited. Please discuss these prior works, and share an ablation of the five components to show that your formulation is useful.

(5) I also think the multi-turn conclusions are not well supported. The multi-turn routers use a small 3B model (Qwen2.5-3B-Instruct) to split the question and merge the answers, and they score only 13 to 36 on the generic tasks, far below simply always using the smallest model (57.55). To me, this points to the weak 3B model, not to multi-turn routing itself. I would like to see a stronger base model here.

I'm willing to increase my score if all of my concerns are addressed properly.

Questions for authors

(1) In Table 2, LLM-MultiRound gets 31.03 on MathVista! This is not possible with 100 test questions, and the same happens on Geometry3K and TimeSeries. Can you explain this?

(2) Was Router-R1 retrained on xRouteBench, or used as released? Since it was trained on QA data such as NQ and HotpotQA, I am not sure it is a fair test here.

8 comments
AuthorsOpenReview ↗

Response 1

Thank you for recognizing the practical importance of the problem and the value of the preference data collected from real users. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have.

W1: The paper is hard to follow.

Response. We sincerely thank the reviewer for pointing out these unclear statements. We agree that the original wording in §2.1 was confusing and that the abstract should more precisely identify the routing methods underlying our cost-sensitive findings. We have revised the abstract, the “Scoring function and decision rule” paragraph in §2.1, and the opening paragraph of §5.3. These changes are highlighted in blue.

(i) Regarding the decision rule in §2.1.

The original sentence contained a grammatical error and an imprecise explanation. Our intended distinction is between quality-only routing and cost-aware routing. The argmax decision rule applies in both cases.

Specifically, when the scoring function g(q,m)g(q,m) predicts only the response quality of candidate model mm, selecting arg⁡max⁡mg(q,m)\arg\max_m g(q,m) corresponds to the performance-only objective (α=1,β=0\alpha=1,\beta=0). When inference cost is considered (β>0\beta>0), the routing decision must additionally account for the candidate's cost. For example, a single-turn router may select the model maximizing

α⋅perf^(q,m)−β⋅c^(q,m).\alpha\cdot\widehat{\mathrm{perf}}(q,m)-\beta\cdot\widehat{c}(q,m).

Thus, argmax remains applicable when its scores already incorporate the performance–cost objective. Alternatively, cost-aware routers may employ threshold-based decisions or cascading strategies. For multi-turn routing, the objective accounts for the quality of the final answer and the accumulated cost of all calls, so a locally greedy decision can be suboptimal over the full trajectory.

(ii) Regarding “lightweight designs” in the abstract.

We agree that the original statement was overly broad and did not identify which routing methods benefit from stronger cost constraints. Here, lightweight designs referred to relatively simple routing mechanisms, such as the nearest-neighbor-based kNNRouter, the MLP-based MLPRouter, and the matrix-factorization-based MFRouter.

More specifically, Figure 4 (§5.3) provides three concrete examples of ranking changes across the tested values of the cost weight β\beta:

  • Generic LLM Tasks: kNNRouter improves from 4th at β=0\beta=0 to 1st at β=0.8\beta=0.8.
  • Vision: MLPRouter improves from 10th at β=0\beta=0 to 1st at every tested weight β≥0.4\beta\geq0.4.
  • TimeSeries: MFRouter improves from 5th at β=0\beta=0 to 1st at every tested weight β≥0.2\beta\geq0.2.

These results demonstrate that simpler routing methods can achieve the highest performance–cost reward under tighter cost constraints, with rankings depending on the task and cost weight. To avoid ambiguity, we have replaced “lightweight designs” in the abstract with the following sentence:

“At the highest cost weight tested, kNNRouter ranks first by performance–cost reward on generic LLM tasks, MLPRouter on vision, and MFRouter on time series.”

We have also revised the opening paragraph of §5.3, “Performance–Cost Trade-offs,” to explain the pattern in Figure 4. EloRouter leads both Vision and TimeSeries when only answer quality is scored, but the two tracks favor different routers as the weight on inference cost increases.


Q2: Router-R1 training and evaluation fairness.

Response. We thank the reviewer for this question. To ensure a fair comparison, Router-R1 was retrained on the xRouteBench training split rather than directly evaluated using its publicly released checkpoint. Specifically, we followed its original reinforcement learning framework, using the same benchmark training and evaluation splits as the other routers. Therefore, the reported results reflect training on xRouteBench rather than zero-shot transfer from the original NQ and HotpotQA training distribution.

Furthermore, we retrain Router-R1 using Qwen3-4B as its base model, while keeping the candidate pool, benchmark splits, and evaluation protocol unchanged.

Table 7. Router-R1 performance with different backbones.

BenchmarkQwen2.5-3BQwen3-4B
Generic35.6464.74
LoCoMo24.6028.03
LongMemEval17.2831.20
Geometry3K14.7539.34
MathVista18.0052.00
Video22.2225.93
TimeSeries23.6251.18
Average22.3041.77

With the stronger backbone, Router-R1 achieves a 19.47-point improvement in average performance, including gains of 29.10 points on Generic LLM Tasks, 34.00 points on MathVista, and 27.56 points on TimeSeries.

§5.2 specifies that we retrained Router-R1 on Qwen3-4B under the original training protocol. Table 7 gives the Router-R1 results from manuscript Table 4 for both backbones on the same seven benchmarks.

AuthorsOpenReview ↗

Response 2

W2: The analysis of the main result is weak.

Response. We sincerely thank the reviewer for the insightful comments. We agree that stronger baselines and a clearer interpretation of the reported gains would improve our analysis. Following your suggestion, we additionally evaluate Best-Single-LLM and OracleRouter using the same 18 candidate models and further examine the performance–cost trade-offs.

(i) Clarification of the 17.4% improvement and Largest-LLM baseline.

Largest-LLM uses DeepSeek-V3.1 (671B) for every query. Appendix D.1 (Rule-Based Baselines) now identifies the model explicitly. Appendix E, Table 12 lists both DeepSeek-V3.1 and Cogito at 671B parameters.

The reported 17.4% relative improvement compares GraphRouter (45.46) against Largest-LLM (38.72) under the seven-benchmark macro-average in manuscript Table 3. The 17.4% uses Largest-LLM as the reference. The comparison with the training-selected fixed model appears in Table 1 below (Appendix E, manuscript Table E.1). This motivates the additional comparisons below.

(ii) Additional Best-Single-LLM and OracleRouter baselines.

We introduce two additional baselines using the existing query–model performance records:

  • Best-Single-LLM: Select the best-performing model on each benchmark's training split and use it for all corresponding test queries.
  • OracleRouter: Select the highest-scoring candidate for each test query using its observed outcomes, providing a hindsight performance upper bound rather than a deployable policy.

Table 1. Best-Single-LLM and OracleRouter results (β=0\beta=0). Performance is multiplied by 100; Best-Single cost is in units of 10−510^{-5} USD/query. The final row gives each of the seven benchmarks equal weight for both performance and cost. The full table appears in Appendix E, Table E.1 of the manuscript.

BenchmarkBest-Single-LLM (Selected Model)Best-Single Perf.Best-Single CostOracle Perf.
Generic LLM Taskscogito-v2-1-671b80.5445.7792.93
LoCoModeepseek-v3.126.5928.8643.96
LongMemEvalllama-4-maverick39.3458.3560.01
Geometry3Kllama-4-maverick45.9055.8678.69
MathVistallama-4-maverick50.0040.1365.00
Videoqwen3-coder-next25.9317.7833.33
TimeSeriesllama-3.3-70b-instruct-turbo63.78136.8399.21
Macro Avg.—47.4454.8067.59

The results reveal substantial untapped potential for query-level model selection. OracleRouter improves over Best-Single-LLM from 47.44 to 67.59 (+20.15 points), with improvements across all seven benchmarks, including +35.43 on TimeSeries, +32.79 on Geometry3K, +20.67 on LongMemEval, and +12.39 on Generic LLM Tasks.

The oracle scores higher on all seven benchmarks, even though Best-Single-LLM selects a fixed model separately on each benchmark’s training split.

We acknowledge that Best-Single-LLM (47.44) outperforms the current learned routers on the performance-only macro-average, including GraphRouter (45.46). Thus, the oracle gap represents remaining opportunities for query-dependent routing, rather than gains already achieved by existing methods.

(iii) Query-dependent routing versus EloRouter.

EloRouter achieves a competitive macro-average of 44.68 without conditioning on individual queries, demonstrating the effectiveness of global model ranking. However, its limitations become apparent on heterogeneous workloads.

For example, on Generic LLM Tasks, which covers 13 subtasks spanning mathematics, coding, knowledge, and commonsense reasoning, Table 2 compares performance with the seven-benchmark average.

Table 2. Performance on Generic LLM Tasks and the seven-benchmark average.

MethodGeneric LLM TasksSeven-benchmark Avg.
Largest-LLM70.2938.72
EloRouter64.1544.68
GraphRouter80.5445.46
Best-Single-LLM80.5447.44
OracleRouter92.9367.59

GraphRouter outperforms EloRouter by 16.39 points on Generic LLM Tasks. Although Best-Single-LLM achieves the same score (80.54), OracleRouter reaches 92.93, leaving another 12.39 points of query-level improvement potential beyond the fixed model selected on the Generic training split.

These findings indicate that global model ranking can be effective, but substantial query-level complementarity remains, particularly on heterogeneous workloads. Our claim is not that query-dependent methods always dominate, but that different routing strategies exploit this complementarity to different degrees.

AuthorsOpenReview ↗

Response 3

Continuation of our response to W2 from the previous page:

(iv) Performance–cost trade-offs in practical routing.

Appendix E, Table 13 reports the performance–cost points underlying Figure 5. Largest-LLM has the highest cost among the single-turn methods in that table. Appendix E, Table E.1 reports the additional Best-Single-LLM reference. Table 3 below summarizes representative operating points.

Table 3. Representative performance–cost operating points. Performance is multiplied by 100; cost is in units of 10−510^{-5} USD/query.

Methodβ\betaPerformanceCost
Largest-LLM0.038.7272.06
Best-Single-LLM0.047.4454.80
EloRouter0.044.6852.19
GraphRouter0.045.4650.57
SVMRouter0.045.1032.66
EloRouter0.436.863.74
GraphRouter0.443.0514.22
EloRouter0.836.863.74
GraphRouter0.836.863.74

These results provide three observations:

  • Query-dependent routers can improve both performance and cost relative to EloRouter. At β=0\beta=0, GraphRouter achieves 45.46/50.57 versus EloRouter's 44.68/52.19. SVMRouter achieves 45.10/32.66, providing slightly higher performance with 37.4% lower inference cost.
  • Cost-aware routing enables more economical operating points. At β=0.4\beta=0.4, GraphRouter achieves 43.05/14.22, reducing inference cost by 74% compared with Best-Single-LLM (47.44/54.80), at a 4.39-point performance reduction. Compared with Largest-LLM, it achieves 4.33 points higher performance with 80.3% lower cost.
  • No single method dominates across all budgets. At β=0.4\beta=0.4, GraphRouter achieves higher performance than EloRouter at higher cost, while both reach the same operating point at β=0.8\beta=0.8. Figure 4 further demonstrates substantial ranking changes across cost weights and task families.

Thus, performance-only rankings do not fully characterize routing effectiveness, and query-dependent methods can provide useful quality–cost operating points depending on deployment constraints.

(v) Revised interpretation of the main findings.

Appendix D.1 identifies the Largest-LLM baseline. Appendix E, Tables 13 and E.1 give the evidence for the following conclusions.

  1. Model size alone is not a reliable indicator of performance, as illustrated by Largest-LLM.
  2. Strong fixed-model and query-independent baselines remain competitive, as demonstrated by Best-Single-LLM and EloRouter.
  3. Substantial query-level routing headroom remains, with OracleRouter exceeding Best-Single-LLM by 20.15 points on average.
  4. Performance–cost trade-offs are essential, as different routing strategies provide distinct operating points under varying cost constraints.

Overall, RouterFactory's main contribution is not that one routing algorithm universally outperforms all alternatives, but a unified infrastructure for systematically comparing heterogeneous routing strategies, revealing their strengths, limitations, and remaining opportunities for more effective and cost-efficient query-level selection.

We believe these additional baselines and analyses provide a more rigorous interpretation of our main findings and further strengthen the motivation for unified LLM routing infrastructure.


Q1: LLM-MultiRound scores in Table 2.

Response. Thank you for checking these scores against the test-set sizes. With Qwen2.5-3B-Instruct, LLM-MultiRound answered 31 of 100 questions correctly on MathVista (31.00%), 9 of 61 on Geometry3K (14.75%), and 39 of 127 on TimeSeries (30.71%). The revised manuscript reports these single-run scores in Tables 3 and 4. The mean across all seven benchmarks is 22.49.

Reviewer_SrMsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents RouterFactory, a unified infrastructure for developing, evaluating, and deploying LLM routers. It formalizes routing as a sequential decision process over query, user context, and interaction history, and characterizes any router by five components: context encoder, model encoder, scoring function, decision rule, and learning signal. Single-turn, multi-turn/agentic, and personalized routers become special cases of this formulation. The system automates data construction, training, inference, evaluation, and deployment around a shared query–model matrix, implements 16 routers, and introduces xRouteBench, a five-track benchmark with 4,767 instances covering generic LLM tasks, memory, vision, time-series, and personalization. Experiments report that no single router dominates, learned routing beats always selecting the largest model, multi-turn routing does not consistently outperform single-turn routing, and personalization helps but depends on how user context is modeled.

Critical strengths
  1. Real infrastructure gap addressed. Routers are currently implemented and evaluated in incompatible stacks; RouterFactory shares data construction, training, inference, evaluation, and deployment across routers, so switching routers or candidate pools is largely a configuration change.
  2. Useful organizing abstraction. The five-component sequential-decision formulation is not a new algorithm, but it is a clean and pedagogically valuable way to place single-turn, multi-turn, and personalized routers under one interface.
  3. Benchmark covers under-served scenarios. xRouteBench includes long-context memory, image/video, time-series, and user-conditioned preference data, and jointly tracks quality and token-level cost rather than accuracy alone.
Critical weaknesses
  1. Differentiation from concurrent routing benchmarks is insufficiently sharp. RouterArena, LLMRouterBench, RouterBench, RouterEval, and VL-RouterBench occupy adjacent space. The paper needs a clearer feature-by-feature case for what RouterFactory uniquely enables. 2.Multi-turn negative result is confounded by the base model. All multi-turn routers use Qwen2.5-3B-Instruct for decomposition and aggregation, so the conclusion may reflect base-model capability rather than a property of multi-turn routing.
  2. Multimodal tracks are text-mediated. Vision, video, and time-series inputs are converted to textual descriptions before routing, so the benchmark evaluates routing over unified text representations, not native multimodal routing.
Questions for authors
  1. Multi-turn confounding: All multi-turn routers use Qwen2.5-3B-Instruct as the base model. Can you provide a controlled experiment or analysis with a stronger base model to show whether the negative multi-turn result reflects multi-turn routing itself rather than base-model capability?

  2. Differentiation from concurrent benchmarks: How does RouterFactory differ concretely from RouterArena, LLMRouterBench, RouterBench, RouterEval, and VL-RouterBench? A feature-by-feature comparison of scope, candidate pools, supervision construction, cost modeling, and deployment support would clarify the unique contribution.

  3. Deployment cost model: The paper positions RouterFactory as deployment infrastructure, but cost is measured only in token price. Can you report latency, failure rate, retry behavior, or throughput for at least the Slack and multi-agent experiments?

3 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank you for your positive evaluation of our work. We appreciate your recognition of the Real infrastructure gap addressed with the useful organizing abstraction. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have.

W1: Differentiation from concurrent routing benchmarks is insufficiently sharp.

Response. We thank the reviewer for highlighting these related efforts. RouterFactory differs from existing routing benchmarks primarily in its unified development-to-deployment infrastructure across multiple routing paradigms, rather than benchmark scale alone. Table 1 compares the six frameworks feature by feature.

Table 1. Comparison of RouterFactory with existing routing benchmarks and frameworks.

FrameworkScope and poolSupervision and routersCost modelingInterface
RouterBenchText, 11 LLMs405K+ scored outputs for prediction and cascadesModel prices and cascade costOffline evaluation
RouterEval12 evaluations, 8,500+ LLMs200M+ performance records for model selectionQuality objective onlyOffline evaluation
RouterArena9 domains, 44 categories, router-specific poolsEvaluates submitted routers on graded queriesInference cost and routing latencyPublic leaderboard
LLMRouterBench21 datasets, 33 LLMsConfigurable collector and evaluator, 10 router adaptersPerformance–cost and latency analysesBenchmark API and adapters
VL-RouterBench14 datasets, 17 VLMsAccuracy–cost soft labels for visual queriesToken prices and throughputVLM evaluation toolkit
RouterFactory (Ours)5 tracks, 18 LLMs, configurable poolsTask scores or preference pairs, 16 routers across three familiesToken cost summed over all callsServing API, Slack, multi-agent systems, ComfyUI

The key distinctions are threefold.

(1) Unified support for heterogeneous routing paradigms. Existing benchmarks primarily focus on single-turn model selection. RouterFactory additionally incorporates multi-turn/agentic and personalized routing within a common sequential decision formulation, supporting routers with different interaction histories, learning signals, and decision processes through a shared interface.

(2) Integrated router development and evaluation. RouterFactory goes beyond providing routing data or evaluation adapters by integrating data construction, router implementation, training, inference, and evaluation. Its modular architecture allows new routers to be implemented through a routing method and a corresponding loss function, while reusing the same infrastructure. Although LLMRouterBench also supports configurable data collection and baseline adapters, RouterFactory extends this workflow across all three routing families.

(3) Deployment beyond offline benchmarking. RouterFactory connects the same routing implementations to an OpenAI-compatible serving interface, OpenClaw-based real-user interactions, and multi-agent systems. We further demonstrate these capabilities through human preference collection on Slack and per-agent routing across five multi-agent topologies. The ComfyUI interface also enables visual configuration and prototyping.

RouterFactory is complementary to these benchmarks. RouterEval studies how model selection scales with the candidate pool, LLMRouterBench supports configurable collection and evaluation, RouterArena compares submitted routers on a public leaderboard, and VL-RouterBench evaluates native vision-language routing.

We added the feature-by-feature comparison to §3, “Comparison with routing benchmarks” (manuscript Table 2), and revised Appendix A, “Related Work,” to clarify how RouterFactory builds on prior work. §2.2 explains how we account for every call in a routing trajectory, and §5.4 demonstrates how the same routing interface supports live preference collection on Slack and per-agent model selection across five multi-agent topologies.

AuthorsOpenReview ↗

Response 2

W2: Multi-turn negative result is confounded by the base model.

Response. Thank you for suggesting this control. We replaced Qwen2.5-3B-Instruct with Qwen3-4B for decomposition and aggregation while keeping the candidate pool, test queries, and scoring protocol fixed. We retrained Router-R1 on the new backbone using its original training protocol.

Table 2. Impact of backbone capability on multi-turn routing performance. Scores use the quality-only objective. The average gives each of the seven benchmarks equal weight.

BenchmarkkNN 3BkNN 4BLLM 3BLLM 4BR1 3BR1 4B
Generic13.9974.2812.9871.2135.6464.74
LoCoMo24.7025.8424.6018.7924.6028.03
LongMemEval18.3224.1717.4419.3617.2831.20
Geometry3K16.3922.9514.7524.5914.7539.34
MathVista30.0053.0031.0056.0018.0052.00
Video25.9325.9325.9318.5222.2225.93
TimeSeries33.0744.0930.7140.1623.6251.18
Average23.2038.6122.4935.5222.3041.77

The original multi-turn shortfall depends in part on the backbone. With Qwen3-4B, the seven-benchmark means for kNN-MultiRound, LLM-MultiRound, and Router-R1 increased by 15.41, 13.03, and 19.47 points, respectively (Table 2). All three exceeded Smallest-LLM on Generic LLM Tasks (57.55) and the best single-turn score on MathVista (50.00). The gains varied by task. On Video, kNN-MultiRound stayed at 25.93 and LLM-MultiRound fell from 25.93 to 18.52. Across all seven benchmarks, the strongest multi-turn router, Router-R1, scored 41.77, compared with 45.46 for GraphRouter.

We added this comparison to §5.2, “Main Results” (manuscript Table 4) and revised the corresponding finding in §1, “Introduction.”


W3: Multimodal tracks are text-mediated.

Response. We agree that the main benchmark evaluates routing over text representations. To test whether routers benefit from seeing the visual input directly, we trained MLPRouter, GraphRouter, and RouterDC on Qwen3-VL-Embedding-2B embeddings of the question text paired with the original image (Geometry3K and MathVista), sampled video frames (Video), or a chart plotted from the time series (TimeSeries). RouterDC uses the frozen encoder followed by a trainable linear projection. The candidate models receive the same inputs and are scored as in manuscript Table 3; the change is on the router side.

Table 3. Router scores by query representation. Single-run scores are from manuscript Table 6. Its text rows reproduce manuscript Table 3.

RouterRouter representationGeometry3KMathVistaVideoTimeSeries
MLPRouterText (manuscript Table 3)27.8734.0029.6356.69
MLPRouterQwen3-VL29.5135.0029.6357.48
GraphRouterText (manuscript Table 3)42.6250.0022.2262.99
GraphRouterQwen3-VL27.8728.0022.2263.78
RouterDCText (manuscript Table 3)16.3924.0025.9345.67
RouterDCQwen3-VL26.2335.0025.9359.06

In Table 3, MLPRouter gains on Geometry3K, MathVista, and TimeSeries. RouterDC gains 9.84, 11.00, and 13.39 points on those benchmarks, respectively, while GraphRouter loses 14.75 points on Geometry3K and 22.00 on MathVista. All three routers match their text scores on Video.

We clarify the default text representation in §3, “Design principle,” and report the experiment in §5.2, “Main Results,” manuscript Table 6.

AuthorsOpenReview ↗

Response 3

Q3: Deployment cost model.

Response. Thank you for raising this question. We evaluated eight routing methods in both deployments and report their mean end-to-end latency in Table 4 (Slack) and Table 5 (multi-agent).

Setting. Both deployments used OpenRouter for LLM inference with concurrency set to eight, and we observed zero failures and zero retries across the timed requests. We started the timer when RouterFactory received a query and stopped it when the final answer was ready.

  • Slack. We timed one routed answer per measurement. This includes selecting the model and waiting for its response. The timer stops before the user votes in the preference study.
  • Multi-agent. We timed one complete workflow per query, including routing at each node, all LLM calls, and any retries. When agents ran in parallel, their calls overlapped in elapsed time.

Table 4. Mean end-to-end latency for one routed answer on Slack (seconds).

RouterMean latency (s)
Largest-LLM7.7
kNNRouter4.4
SVMRouter5.4
MLPRouter7.6
MFRouter5.3
EloRouter6.3
GraphRouter3.6
RouterDC5.1

Table 5. Mean end-to-end latency per query for five multi-agent topologies (seconds). Each topology column averages over its queries. The Avg. column gives the five topology means equal weight.

RouterStarTreeGraphChainPlan-Exec-SumAvg.
Largest-LLM23.332.117.313.821.921.7
kNNRouter22.628.521.513.125.222.2
SVMRouter25.627.98.416.622.820.3
MLPRouter23.120.522.921.934.724.6
MFRouter33.232.818.535.829.730.0
EloRouter23.517.015.429.324.922.0
GraphRouter22.331.620.222.535.926.5
RouterDC12.435.814.922.427.422.6

Conclusion. On Slack, GraphRouter returned answers fastest, averaging 3.6 s per query compared with 7.7 s for Largest-LLM. Across the five multi-agent topologies, SVMRouter was the only router with a lower average latency than Largest-LLM (20.3 s versus 21.7 s). The fastest choice depended on the workflow. SVMRouter led on Graph (8.4 s), while RouterDC led on Star (12.4 s).

Appendix J, “Deployment Latency and Reliability,” reports the measurement protocol, Slack latency in manuscript Table 15, and multi-agent latency in manuscript Table 16.

We would like to thank the reviewer once again for their valuable time and thoughtful feedback. We sincerely hope that our responses and revisions have adequately addressed the concerns raised and further improved the clarity and overall quality of the manuscript. We would greatly appreciate the reviewer's further assessment of these improvements and warmly welcome any additional feedback or discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Our Sincere Appreciation to the Area Chair and Reviewers: Summary of Rebuttal Improvements

We sincerely thank the Area Chair and all reviewers for their thoughtful and constructive feedback. During the rebuttal, we substantially strengthened RouterFactory and xRouteBench through additional controlled experiments, stronger baselines, expanded deployment evaluations, and clearer methodological and reproducibility analyses. Below, we summarize the major concerns, our responses, and the key findings.

DimensionKey ConcernsOur Main ActionsKey Findings
Infrastructure Contribution & NoveltyHow does RouterFactory differ from existing routing benchmarks and frameworks?Added feature-by-feature comparisons with related frameworks and reproducible examples demonstrating task switching and router reuse across multi-agent workflows.RouterFactory provides a unified development-to-deployment infrastructure supporting single-turn, multi-turn, and personalized routing through shared interfaces.
Evaluation Fairness & Stronger BaselinesAre the reported routing improvements meaningful compared with strong fixed-model and query-independent baselines?Added Best-Single-LLM and OracleRouter baselines, clarified the Largest-LLM configuration, and expanded performance–cost analyses.Best-Single-LLM achieves 47.44, while OracleRouter reaches 67.59, revealing substantial remaining potential for query-level routing. Cost-aware routers offer different quality–cost trade-offs.
Multi-Turn Routing & Backbone CapabilityAre multi-turn results limited by the weak 3B controller rather than the routing paradigm?Replaced Qwen2.5-3B with Qwen3-4B under controlled settings and retrained Router-R1 using its original training protocol.The three multi-turn routers improve by 13.03–19.47 points on average, demonstrating the importance of backbone capability while revealing task-dependent limitations.
Multimodal & Memory EvaluationDo text-mediated inputs and limited retrieval histories adequately evaluate the intended routing scenarios?Added visual-input routing experiments using Qwen3-VL embeddings and full-history memory evaluations on LoCoMo.Multimodal representations alter router performance and rankings. Full-history access improves memory-routing scores by 7.93–20.22 points, highlighting the importance of input representation and context availability.
Statistical Robustness & ReproducibilityAre rankings robust to benchmark aggregation, cost normalization, and evaluation configurations?Added alternative aggregation schemes, paired uncertainty analyses, numerical performance–cost points, and clarified cost normalization and MMLU-Pro prompting.GraphRouter remains the strongest evaluated router under all three aggregation schemes. Additional analyses clarify where gains are robust and where differences are task-dependent.
Personalization & Real-World DeploymentIs personalization beneficial, and does RouterFactory provide practical deployment evidence beyond offline evaluation?Added same-architecture user-context controls, session-clustered comparisons, and end-to-end latency measurements for Slack and five multi-agent topologies.User-specific conditioning improves PersonalizedRouter from 72.03% to 83.05%. Deployment experiments demonstrate reusable routing policies with measurable latency and reliability.

Overall Assessment After Rebuttal

The additional experiments and clarifications strengthen three central conclusions:

  1. Infrastructure value: RouterFactory provides a reusable and extensible framework that unifies heterogeneous routing paradigms across development, evaluation, and real-world deployment.
  2. Experimental rigor: Stronger baselines, controlled backbone comparisons, multimodal and memory analyses, and statistical robustness checks provide a more comprehensive and transparent assessment of routing performance.
  3. Practical significance: Real-user personalization, multi-agent deployment, and performance–cost evaluations demonstrate the utility of standardized routing infrastructure across diverse application settings.

Importantly, our findings highlight that no single routing strategy consistently dominates across tasks and deployment constraints, reinforcing the need for a unified infrastructure that enables fair, reproducible, and extensible comparisons.

Appreciation

We sincerely appreciate the Area Chair and reviewers for their valuable time, careful assessments, and constructive suggestions. Their feedback has substantially improved the clarity, experimental rigor, and reproducibility of our work. We hope these additional results and clarifications provide a stronger basis for evaluating the contributions and practical significance of RouterFactory and xRouteBench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.