acceptodds
Under review as a conference paper at ICLR 2027

Routing Without Embeddings: Fast and Interpretable Routing with Regular Expressions

Abstract

Large Language Model (LLM) routers commonly rely on neural query embeddings, with larger encoders expected to better capture query intent and difficulty. Yet scaling Qwen2.5 encoders from 0.5B to 72B parameters brings little improvement in routing accuracy (Figure 1b), suggesting that small encoders may already capture the query properties needed for routing. We therefore investigate which properties matter and whether they can be extracted directly from text without a neural encoder. We introduce RegexRoute, a pipeline that uses sparse autoencoders (SAEs) to discover interpretable regular-expression (regex) features. Using unlabeled text, an LLM turns descriptions of grouped SAE latents into regex extractors and refines them to match latent activation patterns. These extractors supply numerical features to a lightweight routing head, eliminating neural encoding at inference (Figure 1a). Across four benchmarks, a fixed set of 128 features achieves 76.43% average routing accuracy, comparable to the strongest neural text encoder baseline at 76.41%, with much lower latency and greater robustness. These findings establish explicit, interpretable text features as a practical basis for designing and understanding LLM routers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM routing, interpretable feature discovery, regular expressions

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ws55OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether LLM routing can avoid neural query encoding at inference time. RegexRoute groups SAE latents from unlabeled text, uses an LLM to author regex extractors, and refines them against latent-family activation patterns. A supervised Extra-Trees head predicts candidate-model response scores from the frozen features. The same 128-feature library achieves 76.43% average routing score across four benchmark settings, compared with 76.41% for the strongest embedding baseline in Table 1, with substantially lower median routing latency. Ablations examine discovery models, grouping, refinement, feature count, and routing heads; further analyses evaluate perturbation stability and feature recovery from embeddings. Under Equation 1, “routing accuracy” measures the selected model’s response score, rather than agreement with a unique oracle-best model.

Critical strengths
  1. A useful quality–overhead operating point without neural encoding at inference. The explicit 128-feature representation essentially matches the strongest reported embedding baseline on average, while running on CPU with substantially lower median routing latency. The 32-feature variant offers an additional trade-off. This is a consequential practical result: a router need not run a neural encoder merely to retain competitive quality in the evaluated settings. My positive assessment rests on this operating point, rather than the negligible 0.02 percentage-point numerical lead.
  2. A concrete discovery pipeline supported by meaningful controls. Feature discovery uses unlabeled text and the resulting library is reused across routing benchmarks, while only the routing head receives performance labels. The paper removes source-specific wrappers for all representations, includes a substantive word/character TF-IDF baseline, and matches candidate-generation budgets in the discovery ablation. At 128 features, LLM-only, initial guided, and full procedures achieve 75.67%, 76.16%, and 76.43%, respectively. These controls, together with the published extractor list and prompts, make the proposed pipeline informative and assessable, even though the mechanism of its gains needs further isolation.
Critical weaknesses
  1. Limited evidence on queries from new sources. Although the authors remove dataset-specific prompt templates, the remaining text features can still reveal which dataset a query comes from (Figures 5–6 and Appendix E). The router may therefore learn which model usually works best for each familiar source. The paper does not test on source datasets excluded from training, so it is unclear how well this approach would work on unfamiliar queries. CARROT and R2Bench also share queries (Figure 11), making the evaluation less diverse than four separate benchmarks might suggest. This overlap alone does not imply train–test leakage. My main concern is whether the reported performance would hold when the router encounters new types of traffic.
  2. The incremental contribution of SAE guidance over basic surface statistics is incompletely isolated. Appendix D contains 124 count features and four ratio features, with exact duplicates such as f77/f79/f81 and f14/f17/f18/f21/f22. Since both latent-family firing totals and many regex outputs accumulate with text length, some alignment may reflect shared length effects. The TF-IDF and LLM-only controls are useful, but a compact length/punctuation/layout baseline and a length-controlled diagnostic are missing. Appendix C.3 correctly acknowledges that the initial-guided comparison combines family information and alignment-based selection. Without these controls and uncertainty estimates for the small gains, it remains unclear how much new value comes specifically from SAE-derived structure. This limits the methodological contribution, although it does not invalidate the reported deployment operating point.
Questions for authors
  1. Are source datasets shared across training and test splits within each benchmark? Can you provide one limited source-held-out comparison using an existing benchmark with source labels? Clarifying the CARROT/R2Bench overlap and contrasting with a train-derived source-to-model lookup would help distinguish transferable routing from familiar-source recognition. The answer could materially change my assessment of the scope of the contribution.
  2. Under the same head, how does RegexRoute compare with a small fixed set of character/word length, digit, capitalization, punctuation, layout, and code/math-marker statistics? Do the alignment and routing findings persist after controlling for length and removing exact duplicate extractors? A small diagnostic on the existing data would help determine whether the proposed discovery process adds meaningful information beyond simple counts.
  3. Can you report paired uncertainty for the main routing comparisons, explain how Table 1 relates to the five-seed study in Appendix B.4, and clarify whether those repetitions refit the router only or also regenerate the regex library? This would affect how confidently the modest SAE-guidance/refinement gains can be interpreted. I regard 76.43% versus 76.41% as comparable numerical results, not evidence of superiority.
1 comment
AuthorsOpenReview ↗

Remindar to participate in discussion

Thank you again for your review. As the discussion period is about to end, we wonder if our responses have addressed your concerns. We would be glad to provide any further results if any remains :)

Reviewer_TUhKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper replaces the neural query encoder of an LLM router with 128 regex count features. To build them, it groups Gemma Scope SAE latents into families, asks Claude Opus to write a regex for each family from its Neuronpedia descriptions, and refines the regexes to match the family's activation counts. An Extra-Trees regressor on these features then predicts per-model scores. Across four routing benchmarks, the regex router reaches 76.43% mean accuracy, on par with Llama-3.1-8B embeddings (76.41%), at 2.2 ms median latency on CPU.

Critical strengths
  • It is a useful and somewhat sobering finding that cheap surface features match 8B embeddings on these benchmarks.
  • Feature discovery uses no routing labels, and the same library is reused on all four benchmarks. In addition, the full library is listed in the appendix.
  • The ablations are thorough
Critical weaknesses

1. The main differences are within noise. The headline gap is 0.02 pp, and most ablation gaps are a few tenths of a point. However, all results come from a single run of the routing head and a single LLM authoring run. Table 10, where each cell is a separately authored library, moves non-monotonically with N (76.06, 75.98, 76.43, 76.19 for N = 32 to 256), so authoring noise alone seems as large as the claimed effects. I would like to see means and standard deviations over several authoring runs and seeds.

2. The benchmarks have little headroom, and the features mostly detect the source dataset. All representations fall between 74.9% and 76.4%, the best single model gets 73.5%, and a single regex feature already gets 73.48%. Without oracle accuracy, it is hard to tell how much of the achievable gain any router captures. Moreover, Section 7.1 shows that forest neighbors share the same source dataset 68% of the time. This suggests that the router mainly recognizes which benchmark a query came from, so the results say little about whether the features capture query difficulty. Within-source accuracy and held-out-source tests would settle this.

3. The robustness claim does not hold up. The regex router loses more accuracy under perturbation than every sentence-embedding model (0.81 pp vs. 0.20-0.47 pp, Table 5). It has the best perturbed score only because it starts from the best clean score. Furthermore, under paraphrase, the most realistic shift, it falls below Qwen2.5-0.5B (74.92% vs. 75.18%), which is what one would expect from surface features.

4. The efficiency benefit is limited. On a GPU server, regex saves 3-17 ms at the median, but its p99 is worse than every GPU baseline (33.6 ms vs. 7.8 ms for MiniLM-L6 on realistic traffic, Table 8). This saving is small next to LLM time-to-first-token. The real case for the method is CPU-only deployment. However, the paper does not say how the CPU encoders were run. Standard tricks such as ONNX export, int8 quantization, and input truncation would speed them up several-fold. TF-IDF latency is also not reported. A fair comparison should optimize both sides and report end-to-end serving latency.

Questions for authors
  1. What are the oracle accuracy and the within-source accuracy on each benchmark?
  2. What is the variance over several authoring runs and routing-head seeds?
  3. Does the router transfer to held-out source datasets?
  4. How were the CPU encoders run, and how do optimized encoders, static embeddings, and TF-IDF compare in latency?
Reviewer_PDxJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is motivated by an observation that small query encoders seem to suffice for the LLM router. Authors then investigate rule-based regex encoders using SAEs and LLMs. The discovered encoders do not rely on SAEs/LLMs at inference time, thus are very computationally efficient while preserving the routing performance. The proposed method and insights are interesting and useful in LLM routing research.

Critical strengths
  1. The idea is well-motivated and interesting. The results are convincing, and the insights are useful to LLM routing research.
  2. The proposed approach is much simpler and interpretable compared with existing ones. It is a successful use of mechanistic interpretability techniques to guide the design of new methods.
Critical weaknesses
  1. The core step (generation of extractors) is done by an LLM. It is important to inspect this process, such as its stability, patterns, reasons for generating what, etc., or the process itself will become black-box. Detailed analysis on this step is crucial for the community to build on this method.
  2. In Table 1, TF-IDF's performance is close to feature extractors (the latter robustly outperforms for ~0.2-1.5). However, in Figure 1, this method is not shown. Although I understand the main claim in abstract is the potential unnecessity of neural encoders, I am concerned that this will make readers misunderstand that the proposed method wins all existing approaches for that much. Another slight concern is that the main high-level finding of the paper (simple rule-based methods outperforms NN-based methods) is already be explained by comparing TF-IDF to neural encoders.
Questions for authors
  1. In Equation 5, for each document, you calculate two scalars (feature values and reference values) as the execution results of feature extractor and SAE latent family, respectively. However, the two scalars can be the same even if extractor and family's value distributions in the document are very different. Is my understanding correct? If so, does this affect the method's performance?
  2. Given many modern LLM API's indeterminate nature, how robust is the discovered feature extractor set? For example, if doing two identical independent runs, how different will be the two discovered feature extractor sets?
  3. For the LLM used to author extractors, are you using normal LLM API call or agentic harness such as Claude Code? If it is the latter, won't Claude Code do something beyond your specification, such as doing some small experiments alongside when creating the extractors?

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary for AC: responses to reviewers' concerns

Dear AC,

Thank you for reviewing our submission. The current ratings are PDxJ (3), TUhK (2), Ws55 (3). Below we summarize our contribution, what the reviewers recognized, and how we addressed each concern, including three points on which TUhK (2) has not yet replied.


Our contribution

We ask whether an LLM router needs a neural query encoder, and show that it does not.

  1. A finding. 128 regex features route as accurately as the embeddings of 0.5B to 8B decoder LMs on four benchmarks (76.38 ± 0.08% against 76.33–76.39%), with no neural encoder at inference.
  2. A principled way to build the features. SAE latents from unlabeled text are grouped into families, an LLM writes a regex for each family, and the regexes are refined against the family's activations. No routing labels are used, and one library serves all four benchmarks.
  3. Benefits: efficiency and robustness. A routing decision takes 2.2 ms on a CPU and is robust to perturbed queries, so efficiency is not traded for accuracy.
  4. An explanation. Routers gain over the best single model by classifying the content of a query, and surface features are sufficient for this.

Reviewers' recognition

All three reviewers find the paper clear and easy to follow, and they agree on its merits.

The finding is useful: a router doesn't need a neural encoder to perform well.

  • PDxJ (3): "The idea is well-motivated and interesting. The results are convincing, and the insights are useful to LLM routing research."
  • TUhK (2): "It is a useful and somewhat sobering finding that cheap surface features match 8B embeddings on these benchmarks."
  • Ws55 (3): "This is a consequential practical result: a router need not run a neural encoder."

The method is simple, interpretable and free of routing labels.

  • PDxJ (3): "The proposed approach is much simpler and interpretable compared with existing ones. It is a successful use of mechanistic interpretability techniques to guide the design of new methods."
  • TUhK (2): "Feature discovery uses no routing labels, and the same library is reused on all four benchmarks."
  • Ws55 (3): "A concrete discovery pipeline supported by meaningful controls."

The evaluation is thorough and open to inspection.

  • TUhK (2): "The ablations are thorough."
  • Ws55 (3): "These controls, together with the published extractor list and prompts, make the proposed pipeline informative and assessable [...]"

The efficiency gain is practical.

  • Ws55 (3): "A useful quality–overhead operating point without neural encoding at inference."
  • TUhK (2), after the rebuttal: "The CPU latency comparison with optimized encoders is also convincing."

Where the discussion stands

  • PDxJ (3): "The response addressed my concerns. [...] I maintain my score and lean to acceptance."
  • TUhK (2): two of the four weaknesses are confirmed resolved: "The variance analysis resolves my concern about noise: the regex router is on par with the decoder-LM embeddings, and the effect of N is now clear. The CPU latency comparison with optimized encoders is also convincing." The reviewer kept the score on three points. We answered them with the baseline the reviewer asked for and have not heard back (items 7–9 in the next comment).
  • Ws55 (3): we answered all three points with new experiments (items 1, 5 and 6).

[Continued in the next comment]

AuthorsOpenReview ↗

[Cont.] Summary for AC: responses to reviewers' concerns

[Continued from above]

We addressed every weakness and question.

1. Are the differences within noise? (New experiments) We repeated the authoring five times and the routing head with five seeds, at each N. The SD is about 0.1 pp. Ours (76.38 ± 0.08%) is level with the three decoder-LM embeddings and above every other router, and accuracy rises steadily up to N = 128. TUhK (2): W1/Q2 (confirmed resolved), Ws55 (3): Q3, PDxJ (3): W2

2. Is the latency comparison fair? (New experiments) Every encoder now runs in ONNX Runtime with int8 quantization and truncated inputs, and TF-IDF is added. Ours is still the fastest at the median and at p99, with no loss of accuracy. TUhK (2): W4/Q4 (confirmed resolved)

3. Is the LLM authoring step a black box? (New analysis, clarification) Five independent authoring runs differ by at most 0.18 pp and choose the same model for 79% of test queries. Authoring uses plain API calls, and the LLM never sees routing labels. PDxJ (3): W1/Q2, Q1, Q3 (confirmed resolved)

4. Oracle and within-dataset accuracy (New results) Ours improves on the best single model on every benchmark (+2.9 pp on the mean), and the oracle (93.33%) leaves large headroom. Within a dataset, no router is significantly more accurate than the dataset's best model, so the routing gain comes from classifying the content of a query. TUhK (2): Q1, W2

5. Unseen domains (New experiments) With whole domains held out, no router beats the best model of the training domains. Ours is significantly more accurate than 9 of the 11 other routers and level with the other two, and a 5% sample of the new domain lifts it above the best training model. This is a known open problem of LLM routing, and the paper does not aim to solve it. Ws55 (3): W1/Q1, TUhK (2): Q3

6. What does the SAE guidance add over simple statistics? (New experiments) Under the same routing head, primitive count statistics reach 75.81 ± 0.05%, below ours (76.38 ± 0.08%), and every SAE-guided authoring run is more accurate than every unguided one. So the SAE guidance adds information beyond these statistics. Ws55 (3): W2/Q2, Q3


The three points TUhK (2) kept open

We want to be candid: the rating of TUhK (2) is the main obstacle to acceptance. The reviewer confirmed that two of the four weaknesses have been resolved, with three remaining open. We answered them as well but have not heard back, so we kindly ask you to determine whether they have been resolved.

7. "The router reduces to category lookup", so the key baseline is "a simple category classifier, or hand-written surface features [...] followed by a per-category best-model lookup". (New experiments) We ran this baseline. A classifier predicts the dataset for each query and routes the query to that dataset's best model. However, we found:

  • Every lookup is significantly less accurate than our router on the four-benchmark mean (73.60–75.98% against 76.43%). The lookup is limited by coarse category labels and by classifier errors, whereas our router classifies queries by content without supervision from category labels.
  • The hand-written features the reviewer named reach only 73.60% with the lookup and 74.56% with our routing head, far worse than ours, showing the SAE guidance (Sec. 4) from our own pipeline is significant.

So the routing gain is not something a cheap category classifier or these hand-written features provide, and they are not simply category lookup.

8. Is the practical usefulness limited on unseen domains? (Clarification) We explained that unseen domains are an open problem for every router, which the reviewer also acknowledges, and that ours is already the most accurate router there. The practical value of the paper is routing as accurately as decoder-LM embeddings without a neural encoder, at 2.2 ms on a CPU, thereby reducing routing overhead by 97% while maintaining comparable accuracy. And as item 7 shows, a much simpler router does not reach the same level of accuracy. So simple category recognition is not enough, and our method makes a meaningful contribution.

9. Does the router lose more under perturbation than the sentence encoders? (Clarification, claim revised) Our drop is larger (0.81 pp vs. 0.20–0.47 pp), but the drop is the wrong measure of robustness: it favors routers that start weak, and always choosing the best single model yields a drop of 0. In terms of final accuracy on perturbed queries, ours outperforms every sentence encoder. Even if the drop were taken as the measure, our main claim would still stand, because it does not depend on robustness: a router can route as accurately as decoder-LM embeddings without a neural encoder. We report robustness only to show that removing the encoder does not cost accuracy on perturbed queries.


We thank all reviewers for their constructive feedback and believe all concerns have now been addressed. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.