- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Conventional semantic routing systems has the communication and processing overhead. The overhead is introduced by application-layer semantic routers. This paper presents a new semantic routing system called Xpress Semantic Router (XSR). XSR divides the routing process into different components, including request processing, signal extraction, and routing policy execution, and place each component into corresponding layer to reduce this overhead. The authors evaluate several routing mechanisms, including character n-gram, BM25, and distilled intent-based routing, ion XSR. Experimental results show that XSR can achieve up to 37.2× higher throughput across the evaluated concurrency levels and reduce average latency by up to 79.7% compared with a state-of-the-art baseline such as vLLM Semantic Router.
Critical strengths
- Figure 2 provides a clear and intuitive comparison between the architectures of XSR and VSR, helping readers understand the key differences between the two designs.
- The reported performance improvements are impressive, particularly the substantial throughput gains and latency reductions achieved by XSR.
Critical weaknesses
- The methodology could be described in greater detail. For example, the decomposition of the semantic router is currently somewhat high-level, making it difficult to understand how individual routing algorithms are mapped to and executed within XSR. A more detailed walkthrough using a specific routing algorithm as an example would improve the clarity and reproducibility of the proposed design.
- The distillation from the mmBERT-32K teacher model to the student model results in a noticeable degradation in routing accuracy. This raises a concern that supporting the routing algorithm within XSR may require a trade-off between system efficiency and routing quality. The paper would benefit from a more thorough discussion and evaluation of this trade-off.
Questions for authors
How do the authors ensure that the proposed four-stage decomposition is general enough to accommodate the different semantic routing algorithms listed in Section 1.1?
3 comments
Thank you for your rebuttal. The BM25 example clearly illustrates the meaning of the four stages. However, I remain concerned about the generalizability of this stage decomposition to other algorithms. Moreover, the approach still lacks validation on real-world workloads, where the increased complexity of the routing model may introduce additional system-level challenges. Therefore, I will maintain my original score.
Thank you for your comment.
We would like to point out that our texts mention that our experiments use widely used benchmarks like SPEED-Bench and RouterArena (https://routeworks.github.io/), which is widely considered as a de-facto benchmarks for real-world routing performance. If you can help with elaborating on your concerns on the decomposition with counter-examples, it will help us greatly. Thank you.
We thank the reviewer for providing the feedback.
Weakness 1: We believe that we did our best to provide a thorough explanation on each step of our methodology, alongside the example routing algorithm: BM25. We assume that our understanding of what “routing algorithm” indicates may be different from yours. We would appreciate it if you could provide more details on your concerns and why BM25 is not sufficient as an example.
Weakness 2: We fully agree on your concern. We have discussed the point in the general comment.
Reply to the question: We provided the generic four stage decomposition in Section 2: request parsing, signal extraction, policy evaluation, and backend forwarding. And we do believe this is indeed a very general way of structuring semantic routing and generic AI/ML application.