- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper highlights the importance of execution environment information in LLM routing for agentic tasks. The authors introduce EA-ROUTE, a benchmark that evaluates the same queries under different environment configurations, and propose an agent-compiled approach to efficiently extract environment information for routing. Experiments show that incorporating environment information improves model success prediction and leads to better quality–cost trade-offs across different routing methods and environments.
Critical strengths
The paper recognizes and demonstrates the importance of environment information in model routing, which I find quite insightful. The experimental design also reveals several interesting findings, such as the performance differences between different environment representation methods. Overall, I think this is a thought-provoking work.
Critical weaknesses
- The current benchmark constructs environments by starting from a minimal environment and introducing variations through environment knobs. However, the paper does not clearly explain how the additional distractor resources are sourced or constructed. For example, where do the distractor documents come from? Are they semantically relevant to the target query? Are the additional distractor tools actually executable and functional? I am particularly concerned about the tool-use settings, where the authors introduce nearly 1,000 additional tools, which is a very large number. If these constructed environments differ substantially from realistic environments, I think it would be reasonable to question how well the experimental findings generalize. Could the router simply be learning certain patterns introduced by the artificial environment generation process?
- I also feel that the experiments are not sufficiently comprehensive. The current evaluation only considers routing between two models, while practical model routing systems often involve a larger pool of candidate models, potentially including domain-specific expert models. I wonder whether the authors have tested the value of environment information in such multi-model routing settings. After all, routing between two models with relatively clear capability differences seems like a comparatively simple setting, especially considering that the actual improvements in downstream task success rates are relatively modest. Additional experiments would help demonstrate whether environment information remains valuable in more challenging and realistic routing scenarios. I understand that such experiments can be costly, so I consider this a suggestion rather than a strict requirement.
Questions for authors
-
I am not fully convinced that the agent-compiled approach provides substantial additional value. The hand-crafted approach, which uses relatively generic environment information, already captures a large portion of the overall improvement. Could the authors further clarify what makes agent-compiled environment representations particularly useful compared to simpler alternatives?
-
I also have some questions about the minimal environment construction. Could this setting unintentionally make the task easier by revealing information about the solution? For example, retaining only the necessary files may largely remove the difficulty of locating the relevant files in the first place. I understand that the reference answers are not directly exposed to the models, but the environment construction itself might still provide implicit hints. I am curious about the average model performance on minimal environments compared to the original environments. This comparison might help clarify the impact of the construction process.
-
I would also be interested in seeing how the router's model selection behavior changes with environment complexity. For example, as the environment becomes more complex, how does the proportion of queries routed to Opus change? Intuitively, I would expect the router to select Opus more frequently as the environment becomes harder. However, for extremely difficult environments where even Opus is unlikely to succeed, choosing the cheaper Qwen model might actually be more cost-effective. I am curious whether the actual routing results exhibit such a trend, and I think this analysis would provide useful insights into what the router has learned from the environment information.
I would be open to raising my score if the authors can satisfactorily address these concerns and questions.
5 comments
W2 We thank the reviewer for this suggestion and for their understanding of the experimental cost. Our main experiments use two candidate models for two reasons. First, every query–environment pair requires complete executions of the candidate models, and two candidates already require more than 30k executions. Second, a strong–weak pair directly answers whether the stronger model is worth using in the current environment, which is also the standard binary routing setting adopted by prior work such as RouteLLM. At the same time, we fully agree that multi-model routing is a really important setting. Therefore, we added a multi-model experiment on a subset below.
We added two candidate models on BrowseComp-Plus: GPT-6-luna and OpenResearcher-30B-A3B, a smaller model fine-tuned specifically for search tasks, i.e., the kind of domain-specific expert model the reviewer mentions. The four models differ in size and price tier, and no single model dominates across all environments. Our method requires no modification. The router predicts the success probability and cost of each candidate through a shared encoder and a model-identity embedding, so adding new candidates does not change the architecture. Environment descriptions are independent of the candidate pool, so the extractors can be reused without recompilation.
With four candidates, adding environment information substantially improves both prediction and routing:
| Candidates | Router input | Macro-AUC |
|---|---|---|
| 2 | Query only | 55.0 |
| 2 | + Environment | 72.1 |
| 4 | Query only | 56.3 |
| 4 | + Environment | 74.5 |
The Macro-AUC improvement is comparable to the improvement with two candidates on the same dataset. The routing success gain at b=0.5 is +7.4, which is even larger than the average gain in the two-candidate LLMs setting (+6.26). Therefore, in a more challenging setting with a larger candidate pool that includes a domain-specific expert model, environment information remains effective and is even more valuable.
Q1 We thank the reviewer for this question. The value of our method is that the agent can automatically write an extractor that understands the role of each file through its own exploration, without anyone knowing in advance how the environment is organized. In real deployments, different users organize their environments in different ways: some place all indices in a single folder in Markdown format, while others distribute them across multiple directories; some define tools in JSON configuration files, while others declare them directly in code. Automatically understanding the environment is therefore a prerequisite for environment-aware routing to be practical. Hand-crafted does bring improvement, since statistics such as file counts, directory depth, and file sizes already carry some environment information. However, the additional value of Agent-compiled lies in understanding the role of each resource in the environment. Taking the BrowseComp-Plus example in App. D.3, Hand-crafted can only report “15 files, max depth 3, 60% .md,” without distinguishing which files form the corpus and which are indices, or which retrieval method is available. By inspecting file contents and probing interfaces, Agent-compiled identifies that 12 of the files are corpus documents and that the environment supports only grep search. These properties directly determine how much searching a model must do, and they cannot be obtained from generic file statistics.
We would also suggest that Hand-crafted doesn’t already capture most of the overall improvement. Taking Privileged knob metadata (71.87) as an upper bound, the total headroom from Query only (61.31) is 10.56 points. Hand-crafted captures about 41% of it, Agent-compiled about 68%, and Optimized about 84%, reducing the gap to the upper bound from 6.20 to 1.65.
Q2: We thank the reviewer for this question. The reviewer's concern has two parts: first, whether the minimal environment makes the task overly easy by retaining only the necessary resources; and second, whether the environment construction process itself may leave implicit hints. We address each below.
First, the minimal environment is just the starting point of environmental variation. Real user environments range from simple to complex, e.g., some contain only a few relevant documents, while others contain large amounts of irrelevant content. To systematically study the effect of environment complexity, we need a common starting point from which controlled adjustments can be made, and the minimal environment serves as this starting point. On top of it, scale knobs gradually add back the difficulty of locating relevant resources in controlled amounts. All of our experiments take values in the range from the minimal environment to complex environments.
Second, although the minimal environment contains fewer resources, it retains only the evidence needed to support the answer, does not directly reveal the answer, and is not trivial to solve. Macro-averaged across the seven datasets, Qwen3-235B and Opus 4.8 achieve success rates of 66.95% and 94.20%, respectively, in the smallest sampled environments, and 25.06% and 63.52% in the original environments. The minimal environment is easier on average, as intended by design; however, even in the minimal environment, Qwen3-235B still fails on many tasks. We further conducted a case study. For example, in one BrowseComp-Plus case, the minimal environment contains only 6 gold documents (about 48k words), and the question requires combining multiple clues, including voice-acting experience, childhood experience, and band membership, to identify a specific person; Qwen still fails. In SWE-bench Lite, even without any additional files, Qwen fails on a Django task that requires fixing a regular-expression matching bug. These cases show that the minimal environment removes only the difficulty of locating relevant resources, while the difficulty of reasoning itself remains. This is consistent with the view emphasized in our paper: the environment is an important factor in determining task difficulty, but this does not mean that the difficulty of the query itself no longer matters. The two jointly determine task difficulty, which is precisely why we introduce environment information into routing as a complement to the query rather than a replacement for it.
Finally, the environment construction process does not leave exploitable implicit hints. As discussed above, distractor resources follow the same distribution as the gold resources: distractor documents are randomly sampled from the same corpus, and the generated distractor tools are essentially indistinguishable from real tools in both form and semantics. In addition, layout changes reorganize all resources uniformly, so gold resources are not singled out by their position or naming.
We sincerely thank the reviewer for the thorough and constructive review, and for the many insightful questions that have helped us substantially improve the clarity and completeness of our work. We greatly appreciate the reviewer's recognition of the value of our work, and we address each of the reviewer's concerns in detail below.
W1: We thank the reviewer for raising this important question. (1) Distractor documents are randomly sampled from the source dataset’s own corpus. They are real documents, not generated text. They follow the dataset’s real distribution, so distractor documents and gold documents come from the same distribution and share the same style, and the agent cannot dismiss them based on format or writing style. Semantically related documents also appear naturally as the scale increases: at the maximum scale, BrowseComp-Plus uses its full original corpus of about 100K documents. The scale knob therefore moves from the minimal environment back toward the dataset’s original, realistic setting, without introducing the constructor’s own selection preferences. (2) Distractor tools are not randomly generated empty functions. For each dataset, we use Opus-4.8 to generate new tools in the same domain, using the existing tool definitions and data schemas as references. Generation follows the original tools’ naming conventions, parameter formats, and description style to remain consistent with the original distribution. We also implement a full backend for every tool, and all generated tools are verified to be callable and to return results. The added tools also do not collide in name with the original tools.
To further verify whether the generated tools deviate from the real distribution, we conducted a distinguishability experiment. We first sample 5 real tools as in-context references, then sample 30 real tools and 30 generated tools to form 30 pairs for pairwise comparison. For each pair, Claude Opus 5.5 is asked to judge which tool is consistent with the distribution of the reference real tools, and is allowed to answer “uncertain.” In 28 of the 30 judgments, the model answered “uncertain”. That is, even a strong LLM given real tools as references can essentially not distinguish generated tools from real ones. This result shows that the generated tools are built with the same dataset’s real tools and data schemas as references, follow the same naming conventions, parameter formats, and description style, and belong to the same business domain, so they are consistent with real tools in both surface form and semantics. We will report this experiment and examples in the appendix.
The last question is about whether the router merely learns patterns from the environment generation process. We fully agree with the reviewer that this is very important and needs to be verified. Therefore, we conduct the experiment in the native-environment setting in Section 4.6. The router is trained entirely on our constructed environments and then applied directly to the original execution environments provided by each source dataset. Without any adjustment to the router and extractors, the Macro-AUC still improves from 70.27 to 75.62. This demonstrates that the router learns environment properties that genuinely affect model performance and transfer to real-world environments. We will modify this part to make this clear.