acceptodds
Under review as a conference paper at ICLR 2027

Environment-Aware Large Language Model Routing for Agentic Tasks

Abstract

Large language model routing aims to balance task performance and cost by selecting a suitable model for each query. In this work, we find that the preferred model for an agentic task can change with its execution environment, even when the query remains fixed. Changes in available tools, data scale, and execution feedback affect candidate models' performance, revealing an important source of routing information that query-only routers overlook. We introduce EARoute, an environment-aware routing benchmark of 16k query–environment pairs from seven datasets spanning question answering, coding, and tool use. By varying tools, data scale, interface organization, and execution feedback while holding the query, answer, and evaluation fixed, EARoute shows that environmental changes can reverse model preferences for the same task. We propose a method for making routers environment-aware. Its key challenge is that environments are large and heterogeneous, so they cannot be passed to a router directly and must first be condensed into a compact, informative representation. To this end, we introduce an offline environment-understanding agent that explores environments and compiles routing-relevant properties into reusable extractors. These extractors generate compact descriptions without an LLM call at routing time and can be rerun as environment contents change. Optimized with downstream routing feedback, the resulting representation improves macro-averaged model-success prediction AUC by 8.9 points over query-only inputs, with consistent gains across routing paradigms and held-out environments. Our benchmark and agent-compiled representations demonstrate why execution environments matter for model routing and how to incorporate them effectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Model Routing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dtuYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper highlights the importance of execution environment information in LLM routing for agentic tasks. The authors introduce EA-ROUTE, a benchmark that evaluates the same queries under different environment configurations, and propose an agent-compiled approach to efficiently extract environment information for routing. Experiments show that incorporating environment information improves model success prediction and leads to better quality–cost trade-offs across different routing methods and environments.

Critical strengths

The paper recognizes and demonstrates the importance of environment information in model routing, which I find quite insightful. The experimental design also reveals several interesting findings, such as the performance differences between different environment representation methods. Overall, I think this is a thought-provoking work.

Critical weaknesses
  1. The current benchmark constructs environments by starting from a minimal environment and introducing variations through environment knobs. However, the paper does not clearly explain how the additional distractor resources are sourced or constructed. For example, where do the distractor documents come from? Are they semantically relevant to the target query? Are the additional distractor tools actually executable and functional? I am particularly concerned about the tool-use settings, where the authors introduce nearly 1,000 additional tools, which is a very large number. If these constructed environments differ substantially from realistic environments, I think it would be reasonable to question how well the experimental findings generalize. Could the router simply be learning certain patterns introduced by the artificial environment generation process?
  2. I also feel that the experiments are not sufficiently comprehensive. The current evaluation only considers routing between two models, while practical model routing systems often involve a larger pool of candidate models, potentially including domain-specific expert models. I wonder whether the authors have tested the value of environment information in such multi-model routing settings. After all, routing between two models with relatively clear capability differences seems like a comparatively simple setting, especially considering that the actual improvements in downstream task success rates are relatively modest. Additional experiments would help demonstrate whether environment information remains valuable in more challenging and realistic routing scenarios. I understand that such experiments can be costly, so I consider this a suggestion rather than a strict requirement.
Questions for authors
  1. I am not fully convinced that the agent-compiled approach provides substantial additional value. The hand-crafted approach, which uses relatively generic environment information, already captures a large portion of the overall improvement. Could the authors further clarify what makes agent-compiled environment representations particularly useful compared to simpler alternatives?

  2. I also have some questions about the minimal environment construction. Could this setting unintentionally make the task easier by revealing information about the solution? For example, retaining only the necessary files may largely remove the difficulty of locating the relevant files in the first place. I understand that the reference answers are not directly exposed to the models, but the environment construction itself might still provide implicit hints. I am curious about the average model performance on minimal environments compared to the original environments. This comparison might help clarify the impact of the construction process.

  3. I would also be interested in seeing how the router's model selection behavior changes with environment complexity. For example, as the environment becomes more complex, how does the proportion of queries routed to Opus change? Intuitively, I would expect the router to select Opus more frequently as the environment becomes harder. However, for extremely difficult environments where even Opus is unlikely to succeed, choosing the cheaper Qwen model might actually be more cost-effective. I am curious whether the actual routing results exhibit such a trend, and I think this analysis would provide useful insights into what the router has learned from the environment information.

I would be open to raising my score if the authors can satisfactorily address these concerns and questions.

5 comments

We sincerely thank the reviewer for the thorough and constructive review, and for the many insightful questions that have helped us substantially improve the clarity and completeness of our work. We greatly appreciate the reviewer's recognition of the value of our work, and we address each of the reviewer's concerns in detail below.

W1: We thank the reviewer for raising this important question. (1) Distractor documents are randomly sampled from the source dataset’s own corpus. They are real documents, not generated text. They follow the dataset’s real distribution, so distractor documents and gold documents come from the same distribution and share the same style, and the agent cannot dismiss them based on format or writing style. Semantically related documents also appear naturally as the scale increases: at the maximum scale, BrowseComp-Plus uses its full original corpus of about 100K documents. The scale knob therefore moves from the minimal environment back toward the dataset’s original, realistic setting, without introducing the constructor’s own selection preferences. (2) Distractor tools are not randomly generated empty functions. For each dataset, we use Opus-4.8 to generate new tools in the same domain, using the existing tool definitions and data schemas as references. Generation follows the original tools’ naming conventions, parameter formats, and description style to remain consistent with the original distribution. We also implement a full backend for every tool, and all generated tools are verified to be callable and to return results. The added tools also do not collide in name with the original tools.

To further verify whether the generated tools deviate from the real distribution, we conducted a distinguishability experiment. We first sample 5 real tools as in-context references, then sample 30 real tools and 30 generated tools to form 30 pairs for pairwise comparison. For each pair, Claude Opus 5.5 is asked to judge which tool is consistent with the distribution of the reference real tools, and is allowed to answer “uncertain.” In 28 of the 30 judgments, the model answered “uncertain”. That is, even a strong LLM given real tools as references can essentially not distinguish generated tools from real ones. This result shows that the generated tools are built with the same dataset’s real tools and data schemas as references, follow the same naming conventions, parameter formats, and description style, and belong to the same business domain, so they are consistent with real tools in both surface form and semantics. We will report this experiment and examples in the appendix.

The last question is about whether the router merely learns patterns from the environment generation process. We fully agree with the reviewer that this is very important and needs to be verified. Therefore, we conduct the experiment in the native-environment setting in Section 4.6. The router is trained entirely on our constructed environments and then applied directly to the original execution environments provided by each source dataset. Without any adjustment to the router and extractors, the Macro-AUC still improves from 70.27 to 75.62. This demonstrates that the router learns environment properties that genuinely affect model performance and transfer to real-world environments. We will modify this part to make this clear.

W2 We thank the reviewer for this suggestion and for their understanding of the experimental cost. Our main experiments use two candidate models for two reasons. First, every query–environment pair requires complete executions of the candidate models, and two candidates already require more than 30k executions. Second, a strong–weak pair directly answers whether the stronger model is worth using in the current environment, which is also the standard binary routing setting adopted by prior work such as RouteLLM. At the same time, we fully agree that multi-model routing is a really important setting. Therefore, we added a multi-model experiment on a subset below.

We added two candidate models on BrowseComp-Plus: GPT-6-luna and OpenResearcher-30B-A3B, a smaller model fine-tuned specifically for search tasks, i.e., the kind of domain-specific expert model the reviewer mentions. The four models differ in size and price tier, and no single model dominates across all environments. Our method requires no modification. The router predicts the success probability and cost of each candidate through a shared encoder and a model-identity embedding, so adding new candidates does not change the architecture. Environment descriptions are independent of the candidate pool, so the extractors can be reused without recompilation.

With four candidates, adding environment information substantially improves both prediction and routing:

CandidatesRouter inputMacro-AUC
2Query only55.0
2+ Environment72.1
4Query only56.3
4+ Environment74.5

The Macro-AUC improvement is comparable to the improvement with two candidates on the same dataset. The routing success gain at b=0.5 is +7.4, which is even larger than the average gain in the two-candidate LLMs setting (+6.26). Therefore, in a more challenging setting with a larger candidate pool that includes a domain-specific expert model, environment information remains effective and is even more valuable.

Q1 We thank the reviewer for this question. The value of our method is that the agent can automatically write an extractor that understands the role of each file through its own exploration, without anyone knowing in advance how the environment is organized. In real deployments, different users organize their environments in different ways: some place all indices in a single folder in Markdown format, while others distribute them across multiple directories; some define tools in JSON configuration files, while others declare them directly in code. Automatically understanding the environment is therefore a prerequisite for environment-aware routing to be practical. Hand-crafted does bring improvement, since statistics such as file counts, directory depth, and file sizes already carry some environment information. However, the additional value of Agent-compiled lies in understanding the role of each resource in the environment. Taking the BrowseComp-Plus example in App. D.3, Hand-crafted can only report “15 files, max depth 3, 60% .md,” without distinguishing which files form the corpus and which are indices, or which retrieval method is available. By inspecting file contents and probing interfaces, Agent-compiled identifies that 12 of the files are corpus documents and that the environment supports only grep search. These properties directly determine how much searching a model must do, and they cannot be obtained from generic file statistics.

We would also suggest that Hand-crafted doesn’t already capture most of the overall improvement. Taking Privileged knob metadata (71.87) as an upper bound, the total headroom from Query only (61.31) is 10.56 points. Hand-crafted captures about 41% of it, Agent-compiled about 68%, and Optimized about 84%, reducing the gap to the upper bound from 6.20 to 1.65.

Q2: We thank the reviewer for this question. The reviewer's concern has two parts: first, whether the minimal environment makes the task overly easy by retaining only the necessary resources; and second, whether the environment construction process itself may leave implicit hints. We address each below.

First, the minimal environment is just the starting point of environmental variation. Real user environments range from simple to complex, e.g., some contain only a few relevant documents, while others contain large amounts of irrelevant content. To systematically study the effect of environment complexity, we need a common starting point from which controlled adjustments can be made, and the minimal environment serves as this starting point. On top of it, scale knobs gradually add back the difficulty of locating relevant resources in controlled amounts. All of our experiments take values in the range from the minimal environment to complex environments.

Second, although the minimal environment contains fewer resources, it retains only the evidence needed to support the answer, does not directly reveal the answer, and is not trivial to solve. Macro-averaged across the seven datasets, Qwen3-235B and Opus 4.8 achieve success rates of 66.95% and 94.20%, respectively, in the smallest sampled environments, and 25.06% and 63.52% in the original environments. The minimal environment is easier on average, as intended by design; however, even in the minimal environment, Qwen3-235B still fails on many tasks. We further conducted a case study. For example, in one BrowseComp-Plus case, the minimal environment contains only 6 gold documents (about 48k words), and the question requires combining multiple clues, including voice-acting experience, childhood experience, and band membership, to identify a specific person; Qwen still fails. In SWE-bench Lite, even without any additional files, Qwen fails on a Django task that requires fixing a regular-expression matching bug. These cases show that the minimal environment removes only the difficulty of locating relevant resources, while the difficulty of reasoning itself remains. This is consistent with the view emphasized in our paper: the environment is an important factor in determining task difficulty, but this does not mean that the difficulty of the query itself no longer matters. The two jointly determine task difficulty, which is precisely why we introduce environment information into routing as a complement to the query rather than a replacement for it.

Finally, the environment construction process does not leave exploitable implicit hints. As discussed above, distractor resources follow the same distribution as the gold resources: distractor documents are randomly sampled from the same corpus, and the generated distractor tools are essentially indistinguishable from real tools in both form and semantics. In addition, layout changes reorganize all resources uniformly, so gold resources are not singled out by their position or naming.

Reviewer_95t8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies LLM routing for agentic tasks, focusing on how the execution environment can affect model selection even when the user query remains unchanged. The authors construct a benchmark called EA-ROUTE by varying environment properties such as data scale, tools, and retrieval interfaces, while keeping the query, answer, and evaluator fixed.

The benchmark contains 16266 query–environment pairs derived from 2158 original queries across 7 datasets covering QA, coding, data analysis, and tool use. Each query is paired with a reconstructed minimal environment and controlled knob variants. The authors evaluate Qwen3-235B and Claude Opus 4.8 in these environments using the Hermes harness, and show that environmental changes can alter the preferred model in terms of observed performance and cost. The authors then propose an agent-compiled environment representation approach. An offline LLM agent explores environments and generates reusable Python code to produce compact environment descriptions for routing, without requiring additional LLM calls at inference time. Experiments with a query-only router show that adding environment information improves success-prediction Macro-AUC and realized quality–cost trade-offs.

Critical strengths
  1. This paper constructs a well-motivated benchmark that isolates the effect of execution environments on model routing. EA-ROUTE keeps the query and evaluation criteria fixed while varying environmental conditions, as shown in Table 1. This makes it easier to identify information that query-only routers cannot capture.

  2. The authors propose making routers environment-aware by introducing an agent that generates Python extractor code during the offline stage. The generated Python code can convert complex and heterogeneous environment information into compact environment descriptions. During routing inference, routers can directly use these descriptions as part of the input without additional LLM calls, avoiding extra latency and cost.

Critical weaknesses
  1. The benchmark uses predefined environment variations, which may not fully reflect realistic agentic settings.

The varying environment conditions are based on predefined knobs. Many properties in Table 1, such as document count, tool pool size, and retrieval mode, are easy to measure automatically. However, real-world agentic tasks are often affected by other factors, such as hidden states, tool reliability, environment permissions, interface complexity, and data quality.

Moreover, the benchmark mainly evaluates query-only routing with only two candidate models, Qwen3-235B and Opus 4.8, which have large differences in performance and cost. Opus has a significantly higher success rate in most settings. For example, in Table 9, the reported success rates on SWE-bench Lite are 11.76% for Qwen and 66.41% for Opus. Therefore, the router is largely deciding whether it is worth switching to the stronger model.

I understand that constructing these environments is challenging. However, in more realistic settings, routers usually need to choose among multiple candidate models (e.g., the original query-only routing paper IPR [1] considers 11 candidate models). It is unclear whether the proposed method would still work well in such settings.

  1. The experiments do not sufficiently isolate the value of agent-based environment compilation from simpler environment feature extraction.

This paper proposes using an LLM agent to explore environments and compile reusable Python-based extractors. However, in Table 4, the hand-crafted baseline uses a single generic extraction program across all environment types, whereas the proposed approach compiles specialized extractors for 71 dataset–layout families (line 1581) and further optimizes them through 10 rounds of routing feedback. This makes it difficult to determine how much of the improvement comes from agent-based compilation itself, rather than access to layout-specific information and additional optimization.

In addition, the optimized environment descriptions in Appendix D.3 largely consist of straightforward properties, such as document counts, retrieval mode, directory structure, and index size. These properties can be directly measured using static Python extractors. It would be helpful if the authors could include a layout-aware feature extraction baseline.

[1] IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offs

Questions for authors
  1. Why do the authors only consider two candidate models?

  2. How much of the improvement is specifically attributable to agent-based compilation rather than layout-aware feature extraction?

  3. What is the full offline cost of constructing and optimizing environment extractors? Appendix C.6 reports a relatively small API cost for a BrowseComp-Plus extractor. However, the complete approach involves 71 extractors and 10 rounds of harness optimization. How much computation and API expense does the full procedure require, and how frequently would recompilation be necessary when encountering new environment layouts? This would help assess the claimed efficiency advantage in deployments where environment structures change frequently.

4 comments

W1/Q1: We thank the reviewer for this valuable suggestion. We understand that this comment raises two related questions: First, whether predefined environmental changes adequately capture the complexity of real-world scenarios; and second, whether our experimental findings can generalize to multi-model routing settings. We address these two questions separately below.

  1. Realism of Environmental Changes

First, in constructing the benchmark, we have made a lot of effort to bring the environments as close as possible to realistic and diverse agentic use cases. To this end, we built on recently released agent benchmarks and LLMs spanning four common and diverse task types, covering a wide range of environment types, including document corpora, structured data, code repositories, and tool libraries. At the same time, we fully agree with the reviewer that environmental variation in real-world deployments involves more than the attributes captured in our benchmark. Therefore, following the reviewer’s suggestion, we explored a new environmental factor not covered by the original knobs: service reliability. Because each new environment condition requires re-executing both candidate models, we conducted this experiment on BrowseComp-Plus. Each read or search call returns a retryable transient error with probability 20%. In addition, the retrieval results are perturbed: their ranking is shuffled, and irrelevant documents are randomly inserted. For both candidate models, the turns and tokens consumed by retries are counted toward the execution cost. With service failures and retrieval noise, the success rate of Qwen3-235B drops from 61% to 52%, whereas that of Opus 4.8 drops only from 88% to 86%. An unstable service requires more retries and more evidence filtering, and the weaker model is more likely to exhaust its turn budget as a result. This is consistent with the role of factors such as corpus size in our paper: environmental changes affect different models differently and therefore influence model selection.

In this experiment, the environment-understanding agent re-explores the perturbed environment and may also inspect the historical call logs of the retrieval service. These logs come only from the executions of a small number of training tasks. They record the return status of each call, such as whether it failed and how many retries it needed, and contain no information about any test task. The agent identifies the instability of the service and adds corresponding measurements to the extractor, such as the call error rate. We then retrain the router with the new environment descriptions and evaluate it on the test tasks.

Router inputMacro-AUC
Query only58.54
+ Environment78.37 (+19.83)

The environment-aware router again substantially outperforms the query-only router. This shows that a new environmental factor can be incorporated without changing the method.

More broadly, the goal of our paper is not to enumerate every factor that may arise in real deployments, but to show that incorporating environment information into routing is itself important. The stability experiment illustrates that new factors can be incorporated in the same way: they can be added as knobs to the benchmark, and the environment-understanding agent can identify and measure them when they are observable. Other factors mentioned by the reviewer, such as hidden state and permission constraints, can likewise be incorporated within the same framework without changing the method.

  1. More Models For Routing

Our main experiments use two candidate models for two reasons. First, every query–environment pair requires complete executions of the candidate models, and two candidates already require more than 30k executions. Second, a strong–weak pair directly answers the standard binary routing setting adopted by prior work such as RouteLLM.

To examine whether our method remains effective with a larger candidate pool, we added two candidate models: GPT-6-luna and OpenResearcher-30B-A3B, a smaller model fine-tuned for search tasks. Due to time and cost constraints, this experiment was conducted only on BrowseComp-Plus. Our method requires no modification. Environment descriptions are independent of the candidate pool, so the extractors can be reused without recompilation.

With four candidates, environment information continues to bring substantial gains on BrowseComp-Plus:

Router inputMacro-AUC (2 candidates)Macro-AUC (4 candidates)Routing success at b = 0.5 (4 candidates)
Query only55.056.353.3%
+ Environment72.174.560.7%
The AUC gain with four candidates is comparable to that with two candidates on the same dataset. Therefore, environment information continues to substantially improve routing with a larger and more diverse candidate pool.

W2/Q2: We thank the reviewer for the careful observation. We agree that the properties in Appendix D.3, such as document counts, retrieval mode, directory structure, and index size, can all be measured by static Python extractors. This is precisely the design goal of our method: the agent’s final output is itself a static Python extractor, and no LLM is called at runtime (Sec. 3). The question, therefore, is not whether these properties can be measured by static code, but who writes this code and what the writer needs to know in advance.

Consider deploying environment-aware routing in a real product. Different users organize their environments in different ways: some place all indices in a single folder in Markdown format, while others distribute them across multiple directories; some define tools in JSON configuration files, while others declare them directly in code. In real deployments, obtaining the layout specification is therefore itself the problem, and automating this step is a prerequisite for environment-aware routing to be practical. This is the core motivation for introducing an agent to analyze environments. The agent receives no layout labels or specifications; it inspects read-only seed environments and writes the extractors on its own (App. D.1). Layout knowledge is an output of the agent’s exploration, not an input to it.

The reviewer raises two possible confounding factors. First, Agent-compiled writes a separate extractor for each layout family, whereas Hand-crafted uses a single generic program, so the improvement may come from access to layout-specific information. Second, Agent-compiled additionally undergoes 10 rounds of routing-feedback optimization, so the improvement may come from optimization itself. Our existing results partly address both. Regarding the first factor, Hand-crafted is not blind to layout: the directory depth, entries per directory, and file formats it measures already differ across layouts, so its descriptions also vary with layout. The key difference lies in how accurately the representation captures the role of each file. Hand-crafted can only report “15 files, max depth 3, 60% .md,” without determining which files form the corpus and which are indices, whereas the agent-written extractor identifies the 12 corpus documents and recognizes that only grep search is available (App. D.3). Regarding the second factor, Agent-compiled without any optimization (68.53) already outperforms Hand-crafted (65.67) and leads on all seven datasets, so this gain is independent of the 10 optimization rounds.

To address this question more directly, we added a layout-aware extraction baseline, as the reviewer suggested. It uses the same model, compilation prompt, and output requirements as Agent-compiled, and likewise writes one extractor per layout family. The only difference is the source of layout knowledge: instead of exploring the environment, it assumes that the user directly provides the complete layout README.md information, including the directory structure, index locations, and tool declaration formats.

RepresentationGranularitySource of layout informationSuccess AUC
Query only––61.31
Hand-craftedSingle generic programNone65.67
Layout-awareOne per familyProvided by the user68.11
Agent-compiledOne per familyExplored by the agent68.53

The layout-aware baseline reaches 68.11, clearly above Hand-crafted (65.67) and comparable to Agent-compiled (68.53). This indicates that the key factor is layout-specific extraction code that accurately identifies the resources and interfaces of each substrate, regardless of whether the layout knowledge comes from the user or from the agent. The example in Appendix D.3 illustrates the difference. The generic rules can only report "15 files, max depth 3", counting the index and other auxiliary files together with the documents. A layout-specific extractor instead separates the 12 corpus documents and recognizes that the environment offers only literal grep search.

This result also clarifies the practical role of agent compilation. The layout-aware baseline requires the user to supply a complete specification for every layout. The agent needs no layout specification: it reaches comparable performance by exploring read-only seed environments on its own, and it can recompile extractors for new layouts and datasets (Table 5). In real deployments, different users organize their environments differently, and users cannot be expected to write a complete layout specification for each environment. Automating this step therefore makes environment-aware routing more practical and scalable.

Q3: Q3.1: What is the full offline cost, including compiling the 71 extractors and the 10 rounds of optimization?

Each round of routing-guided optimization costs about 21 dollar in API charges, plus one router training run of 1.5–2 GPU hours on an A100 80GB. Over 10 rounds, this totals about 210 dollar in API charges and 15–20 GPU hours. The harness optimization is a one-time cost. And the user will not need to pay for it.

Q3.2: Does this cost recur in deployment?

The 10-round optimization is an offline training expense and is not repeated for each new deployment environment. Once optimized, the compiler prompt is fixed and reused. Deployment requires running the agent to generate an extractor for the new layout, at an average API cost of approximately $0.4 as shown in the Appendix. Subsequent environment descriptions are obtained by executing the generated Python code, without additional LLM calls.

Q3.3: What is the cost in frequently changing environments?

We distinguish two types of environment change. The first, and by far the most common in practice, is a change in content or scale within an existing environment. For example, in an HR question-answering system, the company may add new policy handbooks. In this case, no LLM call is needed to regenerate the code. Because the extractor stores measurement procedures rather than specific values, simply rerunning the existing extractor yields an up-to-date description. This only executes a Python program, takes about 0.056 seconds on average, and incurs no additional cost (App. C.6). The second type is a substantial change in the task type or the environment format. For example, the HR system may move from searching a folder of policy documents to querying a database through tool APIs, or the document collection may be migrated to a different storage structure with a new retrieval index. Only in this case does the agent need to recompile the extractor, at about $0.4 per recompilation. Even if such recompilation were required every 100 queries, the amortized cost would be only about 1.1% of a single Opus execution.

Thank you for the detailed response and additional experiments! The service reliability setting and four-model routing experiments help address my concerns about generalization. The new layout-aware baseline and cost analysis also clarify my other concerns. I appreciate the authors for these additional results and clarifications, and have decided to raise my rating from 2 to 3. I also encourage the authors to include these new experiments and relevant details in the revised paper.

Reviewer_UgziOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how execution environments affect model selection for agentic tasks. EA-ROUTE pairs 2,158 queries with controlled environment variants, yielding 16,266 query-environment pairs across seven datasets. The proposed method uses an offline agent to compile reusable environment extractors and refines the profiling harness through routing feedback. Compact descriptions are then supplied to a router without an online LLM profiling call. Experiments compare success/cost prediction and realized routing performance, include alternative routing paradigms, and examine several distribution shifts and transfer to native environments.

Critical strengths
  1. The controlled benchmark addresses a meaningful limitation of query-only routing. Holding the query, answer, and evaluator fixed while varying resource scale, access interfaces, feedback, and layout makes environmental effects more directly observable. Query-level splits keep all variants of a query together, and the 3,000-pair repeated-execution check helps distinguish environmental preference changes from ordinary execution randomness. The paper also correctly distinguishes preferred-model changes from strict reversals in capability ranking.

  2. The representation pipeline is practical and supported by more than an AUC comparison. The optimized representation improves IPR Macro-AUC from 61.31 to 70.22 and log-cost R2 from 0.029 to 0.410. The fixed-budget analyses in Appendix B.5 show actual macro success gains of roughly 3.2-3.8 percentage points, while native-environment and held-out-layout/dataset evaluations provide evidence beyond the constructed variants. Reusable extractors offer a credible way to avoid repeated LLM exploration when a layout is reused.

Critical weaknesses
  1. The incremental benefit of agent compilation is not fully separated from the specificity and content of the representation baseline. Appendix D.2 defines Hand-crafted as one generic program across heterogeneous substrates, whereas the agent authors 71 dataset-layout extractors. The comparison therefore changes both how the extractor is obtained and how precisely it identifies resources and interfaces. The privileged-knob reference is informative, but does not isolate whether the remaining gain comes from agent-discovered factors, accurate substrate parsing, or packing more measured fields into the input limit. This limits the mechanistic interpretation of the improvement over simpler environment-aware inputs, rather than the main finding that environment information is useful.

  2. The deployment-cost evidence has a narrower scope than the general efficiency discussion. Appendix C.6 measures description latency on three datasets and only records where both profiling methods succeed; its authoring estimate covers a BrowseComp-Plus extractor, rather than the complete ten-round harness optimization and router-fitting process. These are useful, explicitly scoped measurements, but extractor failure coverage, tail latency, and the full amortization of search remain unclear. The realized routing gains are also descriptive point estimates, so uncertainty around the relatively modest success-rate improvements would help assess their practical reliability.

Questions for authors
  1. Could you compare a compact fixed schema of observable fields, such as resource count, retrieval interface, feedback availability, and layout statistics, against the optimized description using the existing measurements and splits? Please explain which gains remain after controlling for available fields and input length. A re-encoding or feature-subset analysis is sufficient; a new manually engineered system is not required.

  2. Could you summarize the offline budget for the ten-round search, including extractor compilation/retries and router fitting, and give a break-even estimate under a few plausible reuse counts? Please also report how many records were excluded from the latency comparison because either profiler failed, with a brief breakdown of failure types and latency by resource scale.

  3. Using the existing predictions and executions, could you provide query-grouped paired uncertainty estimates for the fixed-budget or budget-averaged routing success gains? All environment variants of a query should remain together. This would complement the already reported three split/initialization runs and the execution-randomness study, without repeating the full benchmark.

  4. The experiments use one weak/strong candidate pair. Please state this scope prominently and explain which conclusions concern that pair versus environment-aware routing more generally. It would also help to summarize native-environment routing outcomes, if already available, alongside the native prediction AUC; a larger candidate-model sweep is not necessary for the response.

6 comments

We sincerely thank the reviewer for the careful and constructive review.

W1: We thank the reviewer for the careful analysis. We agree that the comparison between Hand-crafted and Agent-compiled in Table 4 changes two things at once. The first is how the extractor is obtained: a generic Python program generated with human information, versus programs written by an agent after exploring the environment. The second is the granularity of parsing: one program for all environments, versus a separate extractor for each of the 71 dataset–layout families. Therefore, to disentangle these two factors, we added a layout-aware experiment. The results show that the key factor is layout-specific extraction code: it matters little whether the layout knowledge is supplied by the user or discovered by the agent. Agent compilation, however, automates this step and removes the need for users to specify every layout by hand, which makes it more practical in real deployments.

The newly added layout-aware baseline uses the same model, compilation prompt, and output requirements as Agent-compiled, and likewise writes one extractor per layout family. The only difference is that it does not explore the environment using the environment understanding agent. Instead, it assumes the user directly provides the complete layout information, including the directory structure and tool declaration formats.

RepresentationGranularitySource of layout informationSuccess AUC
Query only––61.31
Hand-craftedSingle generic programNone65.67
Layout-awareOne per familyProvided by the user68.11
Agent-compiledOne per familyExplored by the agent68.53

The layout-aware baseline reaches 68.11, comparable to Agent-compiled (68.53) and clearly above Hand-crafted (65.67). We therefore attribute this main gain to accurate parsing of the resources and interfaces in each substrate. The example in Appendix D.3 illustrates the difference. The generic rules can only report "15 files, max depth 3," counting the index and other auxiliary files. A layout-specific extractor instead separates the 12 corpus documents and recognizes that the environment offers only literal grep search.

This result also shows the practical role of agent compilation. The layout-aware baseline requires the user to supply a complete specification for every layout. The agent needs no layout labels or specifications provided by the user: it reaches comparable performance by exploring environments without human labor. In real deployments, different users organize their environments differently. Obtaining the user-written layout specification is itself an obstacle, and users cannot be expected to provide it every time. Automating this step is the main value of agent compilation.

Optimized improves further on Agent-compiled. In both settings, the agent explores the environment and then writes the extractor. What changes are two things: which fields are measured, and how they are formatted. The example in Appendix D.3 shows that routing-guided optimization removes explanatory text and subjective ratings, switches to a compact format, and adds more structural measurements, such as the directory layout, the number of populated shards, document path depth, and index size.

W2/Q2 We thank the reviewer for raising this point. Below, we address extractor failures, tail latency, the amortization of the offline cost, and the uncertainty of the routing gains.

(1) Extractor failures. A compiled extractor is deterministic Python code. During the validation stage of the harness search, every candidate extractor must execute successfully and produce a deterministic description (Appendix D.4). Only extractors that pass validation are used. Across all of our experiments, no compiled extractor failed at runtime. Therefore, the filter in Appendix C.6, which keeps only records where both methods succeed, removes only records where on-demand LLM profiling failed, not records where the extractor failed; these failures arise from the inherent instability of LLM profiling. We will state this explicitly in the revision.

(2) Tail latency. We additionally measured extractor description-generation latency on all seven datasets:

DatasetRecordsp95 (s)p99 (s)
BrowseComp-Plus1,9850.2800.487
CoDA-Bench1,2040.1300.246
CRMArena-Pro1,5990.2440.419
DeepPlanning-Shopping8640.4510.917
EnterpriseBench1,5260.2430.799
OfficeQA-Pro9800.2550.859
SWE-bench Lite1,8800.3680.935

Across all datasets, the p99 latency is below 1 second. Even in the heaviest tail, the latency remains below both the average (11.0 seconds) and the median (10.3 seconds) latency of on-demand LLM profiling. In other words, the slowest 1% of extractor runs are still faster than typical LLM profiling.

(3) Amortization of the offline cost. The 10-round routing-guided harness search is a one-time cost incurred during method development and does not need to be repeated for each deployment. In practice, the user only needs to run the environment-understanding agent once for each new layout to compile its extractor. For example, compiling a BrowseComp-Plus extractor costs about $0.36–$0.63, comparable to a single Opus task execution ($0.35 on average). Once compiled, the extractor is reused for all environments sharing that layout, with no LLM calls at runtime. The deployment cost is therefore determined mainly by this one-time compilation cost, which is amortized as usage grows, and does not depend on the harness optimization process.

(4) Uncertainty of the routing gains. Beyond the point estimates in Table 10, Figure 4 shows the full cost–performance routing curves. The curves show that routing with environment descriptions outperforms query-only routing across most of the budget range, rather than only at a few budget points. We also added query-grouped paired bootstrap confidence intervals in the Q3 section.

Q1: We thank the reviewer for this suggestion. We understand that the reviewer's concern is the contribution of routing-guided harness optimization: the optimized description adopts a more compact format, which fits more measurements within the same input length. After controlling for available fields and input length, how much independent value does the optimization provide? We agree that the format plays a role, which is also what we intended to illustrate in App. D.3: under a fixed input length, a compact format allows more measured properties to reach the router. However, the optimization does more than compress the format. For example, on BrowseComp-Plus, compared with Agent-compiled, the optimized extractor also adds several new measurements, including the directory layout, the number of populated shards, document path depth, and index size. In other words, the optimization improves both what to measure and how to describe it, and both are searched automatically on the development set based on downstream routing performance rather than designed by hand. For a compact fixed schema to achieve the same effect, one would need to know in advance which fields to include for each type of environment, which itself requires a similar search or manual iteration.

Q3: We thank the reviewer for this helpful suggestion. Using the existing predictions and executions, we computed query-grouped paired bootstrap confidence intervals for the routing success gains in Table 10. The table below reports the point estimates with 95% confidence intervals.

Datasetb = 0.25b = 0.50b = 0.75
BrowseComp-Plus+5.54 [2.15, 7.91]+6.26 [2.20, 9.90]+1.85 [−0.28, 4.31]
CoDA-Bench+6.82 [3.09, 11.27]+7.93 [4.27, 12.05]+5.91 [3.01, 8.98]
CRMArena-Pro+2.30 [−0.98, 5.14]−0.15 [−3.54, 3.88]+1.90 [−2.20, 3.78]
DeepPlanning-Shopping+0.14 [−2.33, 3.03]+2.03 [0.01, 4.28]+2.86 [0.24, 4.45]
EnterpriseBench+2.00 [−1.04, 4.61]+0.82 [−2.10, 3.27]−0.04 [−1.89, 1.40]
OfficeQA-Pro+5.15 [−0.46, 10.12]+6.61 [0.43, 11.29]+5.98 [0.34, 10.78]
SWE-bench Lite+4.32 [−4.26, 8.03]+0.92 [−2.68, 4.64]+4.02 [−0.38, 7.92]
Macro-average+3.75 [2.22, 5.28]+3.49 [2.07, 4.91]+3.21 [1.98, 4.45]

Q4 We thank the reviewer for this suggestion. The core goal of our paper is to demonstrate the importance of environment information for routing. To this end, we chose a clear and controlled setting to illustrate this core idea: a strong–weak model pair directly answers whether the stronger model is worth using in the current environment, which separates the role of environment information from the complex interactions among candidate models and more clearly reveals the effect of the environment itself.

Regarding the conclusions, we distinguish two types. The first type concerns this particular model pair and mainly consists of specific numerical results, such as the magnitude of routing success gains, cost savings, and preference flip rates under environmental changes. These values depend on the capability and price differences between the candidate models and would differ for other model combinations. The second type consists of general conclusions about environment-aware routing: the environment changes which model is preferable; incorporating environment information into routing substantially improves the prediction of success and cost; and agent-compiled environment representations are more effective than generic rules. These conclusions do not depend on which two models are used.

To verify this, we extended the candidate pool to four models on BrowseComp-Plus, adding GPT-6-luna and OpenResearcher-30B-A3B, a model fine-tuned specifically for search tasks. Without modifying the method or recompiling the extractors, adding environment information improves Macro-AUC from 56.3 to 74.5, comparable to the improvement in the two-candidate setting, and improves routing success at a normalized budget of 0.5 from 53.3% to 60.7%. This shows that the value of environment information is not limited to a specific pair of candidate models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.