- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLM routing can avoid neural query encoding at inference time. RegexRoute groups SAE latents from unlabeled text, uses an LLM to author regex extractors, and refines them against latent-family activation patterns. A supervised Extra-Trees head predicts candidate-model response scores from the frozen features. The same 128-feature library achieves 76.43% average routing score across four benchmark settings, compared with 76.41% for the strongest embedding baseline in Table 1, with substantially lower median routing latency. Ablations examine discovery models, grouping, refinement, feature count, and routing heads; further analyses evaluate perturbation stability and feature recovery from embeddings. Under Equation 1, “routing accuracy” measures the selected model’s response score, rather than agreement with a unique oracle-best model.
Critical strengths
- A useful quality–overhead operating point without neural encoding at inference. The explicit 128-feature representation essentially matches the strongest reported embedding baseline on average, while running on CPU with substantially lower median routing latency. The 32-feature variant offers an additional trade-off. This is a consequential practical result: a router need not run a neural encoder merely to retain competitive quality in the evaluated settings. My positive assessment rests on this operating point, rather than the negligible 0.02 percentage-point numerical lead.
- A concrete discovery pipeline supported by meaningful controls. Feature discovery uses unlabeled text and the resulting library is reused across routing benchmarks, while only the routing head receives performance labels. The paper removes source-specific wrappers for all representations, includes a substantive word/character TF-IDF baseline, and matches candidate-generation budgets in the discovery ablation. At 128 features, LLM-only, initial guided, and full procedures achieve 75.67%, 76.16%, and 76.43%, respectively. These controls, together with the published extractor list and prompts, make the proposed pipeline informative and assessable, even though the mechanism of its gains needs further isolation.
Critical weaknesses
- Limited evidence on queries from new sources. Although the authors remove dataset-specific prompt templates, the remaining text features can still reveal which dataset a query comes from (Figures 5–6 and Appendix E). The router may therefore learn which model usually works best for each familiar source. The paper does not test on source datasets excluded from training, so it is unclear how well this approach would work on unfamiliar queries. CARROT and R2Bench also share queries (Figure 11), making the evaluation less diverse than four separate benchmarks might suggest. This overlap alone does not imply train–test leakage. My main concern is whether the reported performance would hold when the router encounters new types of traffic.
- The incremental contribution of SAE guidance over basic surface statistics is incompletely isolated. Appendix D contains 124 count features and four ratio features, with exact duplicates such as f77/f79/f81 and f14/f17/f18/f21/f22. Since both latent-family firing totals and many regex outputs accumulate with text length, some alignment may reflect shared length effects. The TF-IDF and LLM-only controls are useful, but a compact length/punctuation/layout baseline and a length-controlled diagnostic are missing. Appendix C.3 correctly acknowledges that the initial-guided comparison combines family information and alignment-based selection. Without these controls and uncertainty estimates for the small gains, it remains unclear how much new value comes specifically from SAE-derived structure. This limits the methodological contribution, although it does not invalidate the reported deployment operating point.
Questions for authors
- Are source datasets shared across training and test splits within each benchmark? Can you provide one limited source-held-out comparison using an existing benchmark with source labels? Clarifying the CARROT/R2Bench overlap and contrasting with a train-derived source-to-model lookup would help distinguish transferable routing from familiar-source recognition. The answer could materially change my assessment of the scope of the contribution.
- Under the same head, how does RegexRoute compare with a small fixed set of character/word length, digit, capitalization, punctuation, layout, and code/math-marker statistics? Do the alignment and routing findings persist after controlling for length and removing exact duplicate extractors? A small diagnostic on the existing data would help determine whether the proposed discovery process adds meaningful information beyond simple counts.
- Can you report paired uncertainty for the main routing comparisons, explain how Table 1 relates to the five-seed study in Appendix B.4, and clarify whether those repetitions refit the router only or also regenerate the regex library? This would affect how confidently the modest SAE-guidance/refinement gains can be interpreted. I regard 76.43% versus 76.41% as comparable numerical results, not evidence of superiority.
Remindar to participate in discussion
Thank you again for your review. As the discussion period is about to end, we wonder if our responses have addressed your concerns. We would be glad to provide any further results if any remains :)