acceptodds
Under review as a conference paper at ICLR 2027

XSR: Rethinking Where Semantic Routing Happens

Abstract

Semantic routing is an essential component of large-scale, multi large language model (LLM) serving, enabling requests to be dynamically directed to the right models based on their content. However, existing semantic routers operate as application-layer intermediaries, requiring requests to incur communication overhead before being forwarded to selected models. This design introduces severe overhead from application-layer processing and communication with external router processors. We present eXpress Semantic Router (XSR), a semantic router architecture that achieves high-throughput and low-latency by running latency-sensitive routing components, such as request processing, signal extraction, and decision policy, in the network layer. Our experiments show that across different routing functions (character (n)-gram, BM25, and distilled intent routing), XSR delivers up to 37.2 higher throughput across the evaluated concurrency ranges while reducing average latency by up to 79.7% compared to a state-of-the-art baselines, such as vLLM Semantic Router.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
semantic routing, network offload, machine learning system, large language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7oK7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Conventional semantic routing systems has the communication and processing overhead. The overhead is introduced by application-layer semantic routers. This paper presents a new semantic routing system called Xpress Semantic Router (XSR). XSR divides the routing process into different components, including request processing, signal extraction, and routing policy execution, and place each component into corresponding layer to reduce this overhead. The authors evaluate several routing mechanisms, including character n-gram, BM25, and distilled intent-based routing, ion XSR. Experimental results show that XSR can achieve up to 37.2× higher throughput across the evaluated concurrency levels and reduce average latency by up to 79.7% compared with a state-of-the-art baseline such as vLLM Semantic Router.

Critical strengths
  1. Figure 2 provides a clear and intuitive comparison between the architectures of XSR and VSR, helping readers understand the key differences between the two designs.
  2. The reported performance improvements are impressive, particularly the substantial throughput gains and latency reductions achieved by XSR.
Critical weaknesses
  1. The methodology could be described in greater detail. For example, the decomposition of the semantic router is currently somewhat high-level, making it difficult to understand how individual routing algorithms are mapped to and executed within XSR. A more detailed walkthrough using a specific routing algorithm as an example would improve the clarity and reproducibility of the proposed design.
  2. The distillation from the mmBERT-32K teacher model to the student model results in a noticeable degradation in routing accuracy. This raises a concern that supporting the routing algorithm within XSR may require a trade-off between system efficiency and routing quality. The paper would benefit from a more thorough discussion and evaluation of this trade-off.
Questions for authors

How do the authors ensure that the proposed four-stage decomposition is general enough to accommodate the different semantic routing algorithms listed in Section 1.1?

3 comments

We thank the reviewer for providing the feedback.

  • Weakness 1: We believe that we did our best to provide a thorough explanation on each step of our methodology, alongside the example routing algorithm: BM25. We assume that our understanding of what “routing algorithm” indicates may be different from yours. We would appreciate it if you could provide more details on your concerns and why BM25 is not sufficient as an example.

  • Weakness 2: We fully agree on your concern. We have discussed the point in the general comment.

  • Reply to the question: We provided the generic four stage decomposition in Section 2: request parsing, signal extraction, policy evaluation, and backend forwarding. And we do believe this is indeed a very general way of structuring semantic routing and generic AI/ML application.

Thank you for your rebuttal. The BM25 example clearly illustrates the meaning of the four stages. However, I remain concerned about the generalizability of this stage decomposition to other algorithms. Moreover, the approach still lacks validation on real-world workloads, where the increased complexity of the routing model may introduce additional system-level challenges. Therefore, I will maintain my original score.

Thank you for your comment.

We would like to point out that our texts mention that our experiments use widely used benchmarks like SPEED-Bench and RouterArena (https://routeworks.github.io/), which is widely considered as a de-facto benchmarks for real-world routing performance. If you can help with elaborating on your concerns on the decomposition with counter-examples, it will help us greatly. Thank you.

Reviewer_6cPoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces XSR, a semantic router that moves supported routing functions from an external userspace router into the Linux socket datapath using eBPF. XSR decomposes routing into request parsing, signal extraction, policy evaluation and backend forwarding. It implements character n-gram matching, BM25 and a distilled linear intent classifier in the datapath. Experiments are conducted with these three routing functions, comparing routing agreement, throughput and latency against vLLM Semantic Router and LLMRouter using fixed-response marker backends.

Critical strengths
  • Framing semantic routing as a function-placement problem is clear and practically motivated, and the paper identifies overhead beyond the routing computation itself.
  • The agreement evaluation and same-student controls are valuable, as they separate implementation consistency from teacher–student approximation. The authors also explicitly reported the classification-quality loss and the unresolved BM25 mismatches.
  • The forwarding controls and latency breakdown in Figure 1 help explain where the improvements come from, and the code is provided.
Critical weaknesses
  • The benefit of kernel placement is not isolated. XSR and VSR also differ in network topology (host veth vs Docker bridge), the Envoy–ExtProc exchange, and BM25 precomputation, which the authors acknowledge inSections 3.3 and 4. A topology-matched, in-process userspace router using the same precomputed BM25 tables and integer student should be compared.
  • The practicality of learned routing is unclear. In Table 1, domain accuracy drops from 62.14% for the teacher to 19.64% for the student, and route accuracy is close to always predicting "others" (85.86% vs 85.71%), although macro-F1 is higher (40.33 vs 30.77). Embedding-based and transformer classifiers are unsupported with no userspace fallback, while most semantic routers rely on them. A hybrid kernel/userspace design that keeps routing quality close to the teacher should be evaluated and quality, fallback rate and cost should be reported together.
  • The deployment-level benefits are not verified. VSR's BM25 routing path adds about 2 ms over Envoy-only forwarding, which is small compared to typical LLM request latency. Marker backends isolate the routing overhead, but give no evidence on TTFT, end-to-end latency or resource savings with real model servers. The 85–97% reduction in routing instances in Appendix A is also only an extrapolation. End-to-end results with real model servers (e.g., vLLM) should be reported.
  • The technical advance over prior offloading work needs to be clearer. Beeline (Brandner et al., 2026) already moves supported L7 policies into an eBPF fast path, with userspace fallback for unsupported ones. XSR applies this to prompt-derived signals, but the authors should clarify what unique novelty XSR has.
  • Missing reference: RouterBench [1] is an early benchmark for evaluating multi-LLM routing and its quality–cost tradeoffs. It should be discussed alongside RouteLLM and RouterArena.

[1] Hu et al. RouterBench: A Benchmark for Multi-LLM Routing System. arXiv 2024.

Questions for authors
  • Which request bodies are used in the throughput and latency runs? Table 5 only describes the correctness workloads, where intent prompts average 59 bytes and RouterArena prompts average 727 bytes. How do throughput and tail latency change with prompt length and policy size?
  • XSR assumes TLS is terminated before the datapath and only supports plaintext HTTP/1.1 with a single string content field. In a real deployment, what terminates TLS, and how would multi-message chat requests or HTTP/2 traffic be handled?
1 comment

We thank the reviewer for recognizing the value of our work that identifies and corrects the overhead beyond the actual routing algorithm.

Please refer to the general comment regarding the concerns of network topology and student model performance.

We also address the reviewer specific questions below.

  • Beeline: is a generic L7 policy and is not optimized for language model routing. XSR’s contribution goes beyond the offload and allows for more sophisticated algorithms like n-gram matching, BM25, and intent classification that Beeline doesn’t provide. Also XSR’s implementation is readily usable with existing inference engines.
  • RouterBench: Thank you for your suggestion, we have addressed this in the revised text.
  • Latency and Prompt Length: While the relationship between prompt length and latency is heavily dependent on system settings, on our setup, we observed that roughly a 10.32× difference in prompt length corresponds to 1.74× increase in latency and respective decrease in throughput.
Reviewer_gCm2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes XSR, which moves the parts of semantic routing that fit eBPF constraints into the Linux kernel socket datapath (SK_SKB stream parsing and SOCKMAP redirection). These parts are character-trigram and BM25 keyword matching, a Boolean policy evaluator, and a distilled 14-class linear intent classifier. This removes the per-request Envoy ExtProc round-trip used by vLLM Semantic Router. The authors frame semantic routing as a function-placement problem and give three datapath criteria: visible inputs, bounded computation and state, and no blocking or userspace-only operations.Correctness is evaluated in two ways. For keyword routing on 9,280 requests, XSR agrees with VSR and LLMRouter on all requests for character n-gram and on 9,276 of 9,280 for BM25. For intent routing on 1,400 labeled requests, the distilled student matches the mmBERT-32K teacher’s domain prediction on 29.8% of requests, and its route-level macro-F1 is 40.3 against 30.8 for a majority-route baseline.Performance is measured with marker backends on a 4-vCPU VM. Relative to VSR, XSR reports 6.76–12.98× throughput for n-gram, 10.60–17.52× for BM25, and 9.51–37.24× for intent, which uses a modified VSR* baseline. At a fixed 100 req/s, mean latency falls by 74.1%, 79.7%, and 73.1% for the three functions. An ablation decomposes the BM25 path into request-path components.

Critical strengths

(a) Careful and checkable datapath reimplementation with strong keyword-routing equivalence. For character n-gram, XSR matches VSR and LLMRouter on all 9,280 requests. Appendix C gives an integer-only threshold transformation that I verified, and it explains how multiplicity-aware overlap is computed in-kernel. The measurement protocol is also transparent: five independent trials, 95% confidence intervals, validity criteria for rejecting runs, and retained raw outputs. This matters because the correctness results are the main evidence that offloading preserves routing behavior, and the paper provides enough detail for an independent check.

(b) A useful design framing with honest scoping. Treating routing as a function-placement problem, with explicit datapath criteria, gives the community a principled way to decide what can be offloaded from a semantic router. The paper also reports limitations that weaken its own headline: the BM25 decomposition states that the gain comes from a restructured request path rather than faster BM25 scoring, and the intent student’s weak classification quality is reported rather than hidden. This makes the reported claims more credible.

Critical weaknesses

(a) The benefit for LLM serving is not established. The measured saving is roughly 1–2.3 ms of mean latency per request. For BM25, the mean falls from 2.75 ms to 0.56 ms (Figure 5), but this is the router path only. The headline 73–80% reductions and throughput multipliers are relative to the router path, not to end-to-end serving. No LLM inference backend is used. Backends return fixed routes, and prompts average about 720 bytes (Table 5). The paper never reports routing overhead as a fraction of time-to-first-token or total generation time, which is the quantity needed to judge whether the problem matters. The gains are also concentrated in the cheapest lexical routing functions. The motivation concerns semantic routing with stronger signals, but Appendix F explicitly excludes embedding-based and transformer-based classifiers. The paper therefore does not characterize the regime in which routing overhead is significant or the regime in which the approach helps.(b) The attribution of gains and the equivalence claims are not isolated. Several confounds weaken the central comparison:• VSR runs in a Docker bridge, while XSR runs on a host veth pair (Appendix E). Topology differences are counted against VSR. No userspace implementation with identical HTTP handling and network path is provided, so the claimed effect of placement is inferred rather than measured.• For intent, the baseline is VSR* (VSR with the student substituted for mmBERT-32K). The default configuration’s latency is never measured, so the abstract’s comparison with “vLLM Semantic Router” does not hold for this function. The 37.2× figure in the abstract comes from this modified baseline at c=8.• Section 2.4 requires identical routes for supported requests under the same configuration. Four BM25 mismatches remain “unresolved” and are not analyzed. The “independent BM25 reference” is not defined.• Boolean composition (AND, NOR, NOT) is claimed as supported in Sections 2.3 and Appendix F. The correctness workload, however, uses single-operator OR rules over four keyword groups, so these claims are untested.

Questions for authors
  1. Please report end-to-end time-to-first-token and total latency with a real LLM inference backend for direct forwarding, VSR, and XSR. Please also report routing overhead as a fraction of each.
  2. Please add a userspace control that implements the same HTTP parsing, signal evaluation, policy, and forwarding over the same network path as XSR, with no Docker bridge. Please also rerun VSR under host networking. How much of the reported gain remains?
  3. Please analyze the four BM25 mismatches. What are the requests, which system is correct under the stated criterion, and what is the root cause (for example, 10⁶ rounding of weights and thresholds, or incomplete stemming surface-form coverage)? If the cause cannot be resolved, the equivalence claim in Section 2.4 must be restricted accordingly.
  4. Please provide load–latency curves, for example at 50% and 80% of each system’s saturation throughput. Please also report per-component CPU utilization at saturation. XSR saturates by c=4 on a shared 4-vCPU host that also runs the load generator and backends, so the bottleneck may not be the router.
  5. What fraction of realistic OpenAI-compatible traffic (multi-turn messages, array-valued content, streaming, chunked encoding) falls within the supported domain? What happens to out-of-domain requests? Is TLS termination included in the measured latency, and if so, where does it run?
  6. Please test AND, OR, NOR, and NOT rules, including mixed signal combinations, against VSR, or remove the corresponding support claims.
1 comment

We thank the reviewer for recognizing the value of our work that identifies our contribution to the community that proposes a principled way to decide what can be offloaded from a semantic router.

Please refer to the general comment above regarding the concerns of network topology and student model performance.

We also address the reviewer specific questions below.

  • BM25 mismatches: The main mismatch arises from having newline characters as BM25 tokenizer strips out whitespaces. This can be resolved by modifying the BM25 algorithm to consider whitespaces as well, but our goal is to analyze the vanilla BM25 algorithm. The paper is updated with the limitations

  • OpenAI-compatible traffic: Given the nature of n-gram and BM25, we do support multi-turn messages and array-valued content. Streaming of the output is done at a response time after the model has been selected, so it is also supported. As for chunked encoding, we have not tested the correctness, but in such scenarios, chunked encoded messages will be routed using system prompts or other headers, which XSR supports.

  • AND, OR, NOR, and NOT rules: We have tested the functionality of these operators and the functionality can be replicated using our code.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment to All Reviewers

We thank the reviewer for recognizing the value of our work. We want to emphasize the works’ contribution to the community that identifies overhead beyond the actual routing algorithm and proposes a principled way to decide what can be offloaded from a semantic router.

  • E2E improvements: Our experiments show that XSR is able to save 2~3ms of inference time. Depending on the model selected, this may be minimal or significant. When comparing this time against a very large frontier model, it is negligible. However, when compared against embedding models or Jev-like decision models, which have a strict latency SLOs of ~20ms, XSR is able to save 10% of total latency, which is substantial.

  • Network topology differences: VSR on Docker Bridge vs XSR on host veth VSR is configured to use Docker bridge for portability, whereas XSR is configured to enable lower-level optimization. With that said, the latency difference between Docker bridge and host veth is only 20~50 microseconds, which is within the realm of noise when compared with overall improvements. Throughput is also not a factor since Docker bridge is capable of supporting 30Gbps, which is plenty for most routers.

  • Student model on Domain Accuracy: First of all, we want to clarify that the student model is not the main contribution of this work. It is a feasibility study to show that XSR can run in environments where kernel memory is limited. With ample kernel memory, this problem does not exist as we can just run the teacher model. With that said, we want this work to motivate future works where an improved student model can be designed and trained to be run inside the kernel for better routing.

  • TLS termination: Similar to VSR, one can definitely add an Envoy proxy or other TLS terminators to XSR. However, in a data center setting, a gateway and/or a load balancer already acts as a TLS terminator. Also, in many Kubernetes environments, e.g., service mesh, encryption and decryption is already taken care of.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.