acceptodds
Under review as a conference paper at ICLR 2027

LongContextRouter: Context-Aware Model Routing for Long-Horizon LLM Agents

Abstract

Choosing a model for an LLM agent requires a delicate balance between task success and execution cost. However, standard routing methods are often myopic: they rely predominantly on the initial request, missing critical requirements buried in long-horizon task documents, prior messages, and execution records. We introduce LongContextRouter, a context-aware task-level router designed to capture this extended context through overlapping-window encoding and causal Transformer aggregation. This hierarchical representation interacts with learned candidate-model embeddings to predict task success, trained via pointwise and multi-positive listwise losses. A decoupled cost head allows model selection to dynamically reflect different cost preferences without requiring the success predictor to be retrained. Evaluated across four benchmarks and five candidate models—and leveraging historical execution text during training on SWE-bench Pro—LongContextRouter achieves 76.44% sample-average task success. This outperforms the strongest fixed-model baseline by 4.19 percentage points while simultaneously reducing recorded model-execution costs by 32.0%.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Language Model Routing, LLM Agents, Context-Aware Model Selection, Long-Context Modeling, Multi-Model Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_76FZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LongContextRouter, a task-level router that picks a LLM per agent task by predicting each candidate's chance of success and penalizing expected cost. It encodes the full task input hierarchically. It also uses execution traces from other models as extra training signal. Across four benchmarks it beats the strongest single model in accuracy while cutting cost by about a third.

Critical strengths
  1. the identified scenario, choosing the model that is enough for the task and previous context, is practically useful.

  2. the paper is written clearly and easy to follow.

  3. The evaluation is comprehensive, spanning four diverse benchmarks with per-benchmark accuracy and cost, controlled ablations of each component, and full model-selection distributions.

Critical weaknesses
  1. the central claim of the paper is not tested at inference time. The paper motivates the proposed approach with long-horizon agentic settings, but at test time the router only sees the query. And since a single problem statement usually fits within one 32K window, the hierarchical encoder and the context pathway are largely unused during evaluation.

  2. baselines are limited. The paper does not compare with recent routers that use execution information for agent tasks, such as SWE-Router, which conditions task-level routing on partial trajectories at test time.

Questions for authors

Why does training-only context help a query-only router, and is the gain due to the context itself? Historical traces are included during training but C is empty at test time, yet B outperforms A. What mechanism do the authors believe explains this transfer?

5 comments
AuthorsOpenReview ↗

Question1 & Weakness 1:Why does training-only context help a query-only router, and is the gain due to the context itself? Historical traces are included during training but C is empty at test time, yet B outperforms A. What mechanism do the authors believe explains this transfer?

Thank you for questioning whether the evaluation supports the paper's context-related claims and for identifying the missing execution-informed baseline. Your comments helped us distinguish two settings more clearly: long-context routing at inference on MemoryAgentBench, and history-based training for query-only routing on SWE-bench Pro. Below, we provide the corresponding input protocols, window statistics, and ablations, together with the added SWE-Router comparison. We also clarify which conclusions the experiments support and which aspects of the transfer mechanism remain unisolated. Substantive revisions and additions are highlighted in blue in the revised manuscript.

Question

Why does training-only context help a query-only router, and is the gain due to the context itself? Historical traces are included during training but C is empty at test time, yet B outperforms A. What mechanism do the authors believe explains this transfer?

The central claim of the paper is not tested at inference time. The paper motivates the proposed approach with long-horizon agentic settings, but at test time the router only sees the query. And since a single problem statement usually fits within one 32K window, the hierarchical encoder and the context pathway are largely unused during evaluation.

Baselines are limited. The paper does not compare with recent routers that use execution information for agent tasks, such as SWE-Router, which conditions task-level routing on partial trajectories at test time.

Related W1:

the central claim of the paper is not tested at inference time. The paper motivates the proposed approach with long-horizon agentic settings, but at test time the router only sees the query. And since a single problem statement usually fits within one 32K window, the hierarchical encoder and the context pathway are largely unused during evaluation.

Answer

Long-context aggregation is exercised at inference on MemoryAgentBench; query-only inference applies to SWE-bench Pro. The two benchmarks evaluate different uses of context:

BenchmarkTest-time routing input
BBHQuery
MemoryAgentBenchQuery plus benchmark-provided memory and long text
MMAU (SC)Query
SWE-bench ProProblem statement; execution histories are training-only

All 100 MemoryAgentBench test tasks require multiple windows, accounting for 100/382 test tasks (26.18%), with up to five windows per input. The 32,768-token limit applies to each encoder window, rather than the complete routing input. The ordered window embeddings are combined by the Transformer before candidate-success prediction. Across training tasks, 137/867 (15.80%) require multiple windows, with up to four windows.

The relevant ablations are:

ConfigurationMAB success (%)SWE success (%)Overall success (%)
A-ON: Query / Transformer62.0066.0071.73
D-ON: Query + Context / Mean pooling62.0066.0071.73
B-ON: Query + Context / Transformer67.0078.0076.44

On MemoryAgentBench, B-ON solves five additional tasks relative to either ablation. B versus D retains context while changing aggregation; B versus A compares the context-enabled and query-only configurations. These results support the full configuration on the evaluated long-context tasks. They do not establish that gains increase with input length: all multi-window test tasks belong to MemoryAgentBench, and the configurations also differ in execution cost.

On SWE-bench Pro, the evidence concerns transfer from historical training views to query-only prediction. All configurations route the same 100 test problem statements without execution histories. B-ON achieves 78%, compared with 66% for A-ON, corresponding to 12 additional successful tasks.

History-based training expands 160 SWE tasks into 800 views using five candidate-model-specific execution contexts per task; 40 validation tasks similarly produce 200 views. Query-only training has 160 distinct training inputs because removing the execution contexts leaves the same problem statement across candidates. The configurations share original tasks, task-disjoint splits, the candidate pool, and full-task outcome labels. Views from one task remain in the same partition.

AuthorsOpenReview ↗

Question (continued)&Weakness 2

Weakness 2

baselines are limited. The paper does not compare with recent routers that use execution information for agent tasks, such as SWE-Router, which conditions task-level routing on partial trajectories at test time.

Our interpretation is that historical execution text provides additional evidence about task difficulty and candidate behavior when training the shared aggregator, model embeddings, and success predictor. Those learned parameters subsequently operate on new problem statements. This is a plausible explanation, rather than an experimentally isolated mechanism: the current comparison measures the combined history-based training configuration and does not separately identify the contributions of historical content, view augmentation, and training exposure. The reported comparison uses seed 43 and does not establish robustness across training seeds.

The added SWE-Router comparison addresses the missing execution-informed baseline. Our reproduction uses partial trajectories for selection and is evaluated on the same five-model pool and 382 tasks:

Costs cover selected-model execution only; SWE-Router’s partial-trajectory generation and router encoding, computation, and hosting are excluded. These results therefore do not establish an end-to-end cost advantage.

MethodOverall success (%)Execution cost (USD)SWE success (%)SWE execution cost (USD)
SWE-Router75.3910.8966.008.67
LongContextRouter76.4411.6578.0011.16

LongContextRouter solves 292/382 tasks versus 288/382, an improvement of 1.05 percentage points, at approximately USD 0.76 higher selected-model execution cost. On SWE-bench Pro, the improvement is 12 percentage points. Performance therefore varies across benchmarks; these aggregate results do not imply uniform superiority. For details on reproducing SWE-Router, please refer to our repository: anonymized repository.

Thank you again for prompting this clearer separation of the two evaluation settings. We hope the evidence of multi-window inference, the qualified interpretation of history-based training, and the additional baseline address your main concerns. If these responses resolve them, we would appreciate your reconsideration of the paper and its rating. We welcome further discussion of any remaining concern.

AuthorsOpenReview ↗

Follow-up on Our Responses to Your Additional Questions

Thank you again for your careful review. As the discussion period draws to a close, we wanted to follow up on our responses and the revisions to the manuscript. Do you have any remaining questions or concerns that we could clarify? If our responses and revisions have addressed your main concerns, we would appreciate your reconsideration of the paper and whether an upward revision of your rating would reflect your updated assessment. We would be happy to discuss any points that remain open.

Reviewer_ewEBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LongContextRouter selects a model for each task using predicted success and cost. Frozen embeddings encode overlapping text windows, which a Transformer aggregates to score candidate models. Experiments cover four task families and five models. SWE-bench Pro execution histories are used only during training; test inputs contain the initial request.

Critical strengths
  • Useful gains within a shared model pool. The router succeeds on 292/382 tasks versus 276/382 for the strongest fixed model, with execution costs of \11.65versus11.65 versus \\17.13. These results support practical value, although router overhead is excluded.
  • Clear training/test information boundaries. Data are split by task, and the paper distinguishes training histories from query-only test inputs. Ablations of inputs, aggregation, and cost handling provide a basis for analysis.
Critical weaknesses
  • The long-context contribution is not isolated. A single window holds 32,768 tokens, yet window counts and results by input length are missing. SWE training expands 160 tasks into 640 model-specific views, without clearly matching views, supervision, and updates in the query-only control. Gains could therefore reflect history information or extra training views rather than modeling across windows.
  • The objective and cost comparisons are unclear. Eq.9 specifies BCE plus listwise loss, whereas Table 4 states “BCE + cost ranking” without defining the latter's form or weight. Cost OFF removes cost terms from both training and inference. Some ablations reduce execution cost from \11.65toabout11.65 to about \\3.35, so lower accuracy alone does not establish a worse trade-off.
Questions for authors
  • Implementation: Please specify the full loss and its weights, and explain the implementation and supervision differences between EmbedLLM in Table 1 and the query-only, Cost-OFF configuration in Table 2.
  • Inputs and training: How many training and test inputs exceed one window? Do history and query-only controls match views, labels, and updates? Results grouped by window count would help assess the long-context contribution.
  • Existing-result analysis: Using a fixed checkpoint and existing model outcomes, can you vary the cost preference, select settings on validation data, and compare test performance at equal execution cost?
8 comments
AuthorsOpenReview ↗

Question 1 & Weakness 2:implementation: Please specify the full loss and its weights, and explain the implementation and supervision differences between EmbedLLM in Table 1 and the query-only, Cost-OFF configuration in Table 2.

Thank you for your detailed questions about implementation, training controls, and cost comparisons. They helped us make the experimental protocol more explicit. Below, we specify the training objectives and weights, distinguish EmbedLLM from A-OFF, report input-window statistics and training-view construction, and explain validation-based preference selection. We also state the scope of the fixed-checkpoint comparisons, including the distinction between nearby and exactly matched execution costs. Substantive revisions and additions are highlighted in blue in the revised manuscript.

Question 1

Implementation: Please specify the full loss and its weights, and explain the implementation and supervision differences between EmbedLLM in Table 1 and the query-only, Cost-OFF configuration in Table 2.

Related W2 (objective definition):

The objective and cost comparisons are unclear. Eq.9 specifies BCE plus listwise loss, whereas Table 4 states “BCE + cost ranking” without defining the latter's form or weight.

Answer

Eq. (9) gives the form of the objective; we now specify that the experiments used a listwise weight of 0.3. We also correct the “BCE + cost ranking” label in the submitted manuscript's Table 4: the comparison uses BCE only, and the full model uses

Lsuccess=LBCE+0.3Llist.\mathcal L_{\mathrm{success}}=\mathcal L_{\mathrm{BCE}}+0.3\mathcal L_{\mathrm{list}}.

BCE supervises individual full-task success probabilities. The listwise term assigns probability mass to successful candidates:

Llist=−1∣V+∣∑v∈V+log⁡(∑m∈Mv+Pv(m)).\mathcal L_{\mathrm{list}}=-\frac{1}{|\mathcal V^+|}\sum_{v\in\mathcal V^+}\log\left(\sum_{m\in\mathcal M_v^+}P_v(m)\right).

P_v is the softmax over candidate scores; M_v_plus contains successful candidates, and V_plus contains views with at least one success. After success training, we freeze the router and train only the cost head on fixed query-only features:

Lcost=1NM∑i=1N∑m=1M[ℓi,m−log⁡(1+ci,m)]2.\mathcal L_{\mathrm{cost}}=\frac{1}{NM}\sum_{i=1}^{N}\sum_{m=1}^{M}[\ell_{i,m}-\log(1+c_{i,m})]^2.

Table-1. Supplementary listwise-weight experiments. BCE has coefficient 1. The original 0.3 setting gives the highest accuracy and lowest cost among the five evaluated weights.

Training lossTest accuracy (%)Test cost (USD)
BCE + 0.1 × listwise68.85126.47
BCE + 0.2 × listwise68.5996.69
BCE + 0.3 × listwise76.4411.65
BCE + 0.4 × listwise74.6114.37
BCE + 0.5 × listwise72.7715.58

A-OFF and EmbedLLM differ in representation processing and training objective. A-OFF retains Transformer aggregation and BCE-plus-listwise training, while EmbedLLM uses query projection, model embeddings, and candidate-wise BCE. Both use full-task success labels. The identical original table entries were a copying error.

Table-2. Corrected results. Costs cover selected-model execution.

Method or configurationAccuracy (%)Cost (USD)
A-OFF72.5192.36
EmbedLLM75.1340.08
LongContextRouter76.4411.65

We update the objective description, training-loss table, and baseline comparison. The training weight 0.3 is distinct from the inference-time preference lambda; cost-head fitting remains a separate stage. For detailed training configurations, including the learning rate, optimizer settings, and other hyperparameters, please refer to our repository: anonymized repository.

AuthorsOpenReview ↗

Question 2 & Weakness 1:Inputs and training: How many training and test inputs exceed one window? Do history and query-only controls match views, labels, and updates? Results grouped by window count would help assess the long-context contribution.

Question 2

Inputs and training: How many training and test inputs exceed one window? Do history and query-only controls match views, labels, and updates? Results grouped by window count would help assess the long-context contribution.

Related W1:

The long-context contribution is not isolated. A single window holds 32,768 tokens, yet window counts and results by input length are missing. SWE training expands 160 tasks into 640 model-specific views, without clearly matching views, supervision, and updates in the query-only control. Gains could therefore reflect history information or extra training views rather than modeling across windows.

Answer

100 of 382 test tasks require multiple windows. We count original tasks once, including when they have several training views. A task is multi-window if any full-input view exceeds one 32,768-token window; adjacent windows overlap by 512 tokens.

Table-3. Pooled input-window statistics.

SplitTasksMulti-window tasksProportionMaximum windows
Train86713715.80%4
Test38210026.18%5

All multi-window test tasks are from MemoryAgentBench; the remaining 282 tasks are from the other three benchmarks. Groups follow the full input and remain fixed across ablations. A-ON uses query-only inputs and Transformer aggregation; D-ON uses context and mean pooling; B-ON uses context and Transformer aggregation. All enable cost awareness.

Table-4. Accuracy by input group, from the recorded test outcomes.

Input groupTasksA-ON (%)D-ON (%)B-ON (%)
Single-window28275.1875.1879.79
Multi-window10062.0062.0067.00

B-ON gains 5.00 percentage points on multi-window tasks and 4.61 points on single-window tasks over either ablation. The hierarchical encoder covers the full input across windows, and the context pathway supplies that information to success prediction.

History-based training expands 160 SWE tasks into 800 views using five candidate-model-specific execution contexts per task. Query-only training retains 160 distinct inputs, since removing context leaves the same problem statement across candidates. Both share the original tasks, splits, and outcome labels.

The revised manuscript corrects the SWE training/validation view counts from 640/160 to 800/200 and includes these training controls and grouped window results in the experimental setup and appendix.

AuthorsOpenReview ↗

Question 3 & Weakness 2:Existing-result analysis: Using a fixed checkpoint and existing model outcomes, can you vary the cost preference, select settings on validation data, and compare test performance at equal execution cost?

Question 3

Existing-result analysis: Using a fixed checkpoint and existing model outcomes, can you vary the cost preference, select settings on validation data, and compare test performance at equal execution cost?

Related W2 (cost comparisons):

Cost OFF removes cost terms from both training and inference. Some ablations reduce execution cost from $11.65 to about $3.35, so lower accuracy alone does not establish a worse trade-off.

Answer

The checkpoint and reference λ vector were selected sequentially on the validation set. We first selected the checkpoint with the highest validation task success rate. We then fixed it and searched one λ coordinate at a time over {0, 0.05, 0.1, 0.2, 0.4, 0.8, 1.0, 1.2, 1.4, 1.8, 2.0}, holding the other coefficients and scaling rules fixed. We prioritized validation success rate and used lower total selected-model execution cost to break ties. The checkpoint and selected vector were frozen before test evaluation.

We hold the selected checkpoint, predictions, and task-class posteriors fixed, varying one coordinate of (0,0.2,0.05,0.2) at a time. Each input combines all four cost components, so a coefficient can affect selections across benchmarks. Accuracy and selected-model execution cost cover all 382 test tasks.

The accuracy–cost plot is available at this link.

Table-5. Representative operating points.

Method or configurationAccuracy (%)Cost (USD)
DeepSeek-V4-Flash-073171.733.35
DeepSeek-V4-Pro-081372.2517.13
B-OFF73.8217.78
B-ON, reference vector76.4411.65
B-ON, lambda_MMAU = 0.176.189.20
B-ON, lambda_MMAU = 0.474.083.77
B-ON, lambda_MMAU = 2.072.253.42

The reference B-ON setting improves accuracy over B-OFF by 2.62 percentage points and reduces cost by 34.49%; the 74.08% / USD 3.77 setting also improves both measures. Near DeepSeek-V4-Flash-0731's budget, B-ON solves 276/382 tasks versus 274/382, for USD 0.07 additional cost (2.02%).

The appendix specifies this validation-selection procedure and fixed-checkpoint protocol, together with the measured accuracy and cost differences at nearby budgets.

Thank you again for helping us improve the transparency of the experimental setup. We hope these details and analyses provide a clearer basis for assessing the results, with the remaining limitations stated explicitly. If the responses address your main concerns, we would be grateful if you would reassess the paper and its rating. Please let us know if any aspect still requires clarification.

Reviewer_oH8POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Task-level router for agent workloads: Serialize query + context, chunk into 32k-token overlapping windows, embed each with a frozen Qwen3-Embeddding-9.6B, aggregate with a one layer casual transformer, take the last state, dot it against a learned per-model embedding to get a per model success probability. Trained with BCE plus a multi-positive softmax loss. A separate linear cost head on frozen query-only features predicts per-model cost. At inference, pick argmax of success minus a benchmark specific λ - weighed scaled cost. Evaluated on BBH, MMAU-SC, Memory AgentBench, SWE-bench Pro (382 test tasks), five candidate models. Results: 76.44% sample-average success at $11.65 vs 72.25% at $17.13 for the best fixed model. On SWE-bench Pro, execution trajectories are added to training inputs only while test uses the problem statement alone.

Critical strengths
  1. Good problem, practical design: For agents, most of the cost goes to long multi-step runs, so picking the right model per task matters. The paper trains the router with full execution logs but only needs the problem statement at test time. That’s something one could actually deploy.

  2. The router gets 78% success at dollar 11.16, while the best single model gets 67 percent at dollar 16.77. It does this by sending 42% of tasks to the cheap model and 58% to the expensive one (Fig. 2b). Beating every single model on both accuracy and cost, using only the problem statement, is exactly what a router should do. If this number holds up, the paper has a contribution.

Critical weaknesses
  1. Table 1’s “EmbedLLM” row (98.00/0.11, 63.00/1.68, 59.76/0.07, 67.00/90.50, 71.94, 72.51, $92.36) is identical in every column to Table 2’s A-OFF row — the authors’ own model with context and cost awareness removed. And the other ablations collapse to constant policies: D-ON matches the always-Flash row exactly, A-ON/”w/o listwise”/Table 4 Base are within a task or two of it, C-ON routes 98% of SWE to GLM (3% success), B-OFF routes 100% to Pro-0813. So the “+4.71 from context” and “+4.71 from Transformer aggregation” claims are both “full model vs. always-Flash,” and the paper has no comparison to prior routing work at all.

  2. No seeds, no CIs; the SWE gap is 11 tasks, the overall gap ~16 of 382, and context views on SWE somehow move BBH and MAB numbers, which looks like seed noise. λ_t = 0.0/2.0/0.1/1.0 with two different scaling rules is selected per benchmark by a procedure the paper never states, and benchmark identity is an input at inference that the baselines don’t get. The cost penalty raising accuracy (73.82 → 76.44) says the success head isn’t ranking candidates on its own. And the one thing the decoupled cost head is for — sweeping λ on a fixed checkpoint — is never shown; Fig. 2a’s caption explicitly disclaims it.

Questions for authors

1.Is Table 1’s EmbedLLM row the same run as Table 2’s A-OFF? If not, what accounts for identical values across all 11 columns? If so, what exactly is the EmbedLLM baseline?

  1. How were λ_t and the per-benchmark scaling rules selected?

  2. Provide the accuracy-vs-cost curve for a fixed B-ON checkpoint as λ varies (the sweep Fig. 2a disclaims). Does B-ON Pareto-dominate B-OFF and always-Flash along that curve?

4 comments
AuthorsOpenReview ↗

Question 1&Weakness 1: Is Table 1's EmbedLLM row the same run as Table 2's A-OFF? If not, what accounts for identical values across all 11 columns? If so, what exactly is the EmbedLLM baseline?

Thank you for your careful review and for highlighting the need to distinguish the routing baseline from our ablations and to substantiate the accuracy-cost trade-off. Your questions helped us identify a table-entry error and improve the explanation of cost-preference selection. Below, we correct the EmbedLLM comparison, detail the validation-based selection procedure, and provide fixed-checkpoint sensitivity results that distinguish Pareto improvements over B-OFF from nearby-budget comparisons with always-Flash. Substantive revisions and additions are highlighted in blue in the revised manuscript.

Question 1

Is Table 1's EmbedLLM row the same run as Table 2's A-OFF? If not, what accounts for identical values across all 11 columns? If so, what exactly is the EmbedLLM baseline?

Answer

The identical EmbedLLM and A-OFF entries were a table-entry error: A-OFF results were mistakenly copied into the EmbedLLM row. The corrected results are 75.13% / USD 40.08 for EmbedLLM and 72.51% / USD 92.36 for A-OFF.

A-OFF retains LongContextRouter's Transformer aggregation and BCE-plus-listwise objective while removing context and cost awareness. EmbedLLM uses a query projection, model embeddings, and candidate-wise BCE supervision. Both use full-task success labels.

Table-1. Corrected comparison on the same five-model pool and 382 tasks. Costs cover selected-model execution, excluding SWE-Router's trajectory-generation cost and router overhead.

MethodSuccessful tasksAccuracy (%)Cost (USD)
DeepSeek-V4-Flash-0731274/38271.733.35
DeepSeek-V4-Pro-0813276/38272.2517.13
EmbedLLM287/38275.1340.08
SWE-Router288/38275.3910.89
LongContextRouter292/38276.4411.65

SWE-Router was reproduced following its published method and configuration. LongContextRouter solves four more tasks than SWE-Router for USD 0.76 in additional cost. Against DeepSeek-V4-Pro-0813, it solves 16 more tasks while reducing cost by 32.01%.

A-ON removes context, while D-ON replaces Transformer aggregation with mean pooling. Both obtain 71.73%, so each differs from B-ON's 76.44% by 4.71 percentage points. D-ON selects DeepSeek-V4-Flash-0731 throughout; on SWE, C-ON assigns 98% to GLM and B-OFF assigns 100% to DeepSeek-V4-Pro-0813. B-ON instead combines Flash and Pro-0813 selections and reaches 78.00% success.

With seed 43 fixed, all benchmarks share trainable parameters. Changing SWE training views can therefore produce small changes on BBH and MAB through joint training. We correct the main baseline table and corresponding comparisons; the ablation differences refer to their respective configurations.

AuthorsOpenReview ↗

Question 2& Weakness 2: How were λ_t and the per-benchmark scaling rules selected?

Question 2

How were λ_t and the per-benchmark scaling rules selected?

Related W2 (selection and inference information):

λ_t = 0.0/2.0/0.1/1.0 with two different scaling rules is selected per benchmark by a procedure the paper never states, and benchmark identity is an input at inference that the baselines don’t get

Answer

The checkpoint and cost-preference vector were selected sequentially on the validation set. We first selected the checkpoint with the highest validation task success rate. With that checkpoint fixed, we performed a one-coordinate-at-a-time grid search over the values listed below, holding the other coefficients and scaling rules fixed. Settings were ranked by validation success rate, with lower total selected-model execution cost breaking ties. We then froze the checkpoint and λ vector before test evaluation. The selected reference vector is (0,0.2,0.05,0.2).

Table-2. Cost preferences and scaling rules.

Task classReference λCost scaling
BBH0.00Min–max
MemoryAgentBench0.20Min–max
MMAU (SC)0.05Min–max
SWE-bench Pro0.20Logarithmic, using a training-derived scale

The routing rule combines four task-class components for every input:

Rm(x;λ)=∑t∈Tpt(x)λtϕt(c^t)m.R_m(x;\lambda)=\sum_{t\in\mathcal T}p_t(x)\lambda_t\phi_t(\hat{\mathbf c}_t)_m.

m∗(x)=arg⁡max⁡m∈M[π^m(x)−Rm(x;λ)].m^*(x)=\arg\max_{m\in\mathcal M}[\hat\pi_m(x)-R_m(x;\lambda)].

Here, p_t(x) is the task-class posterior, c_hat_t contains candidate-model cost estimates for class t, and phi_t scales those costs. Thus, changing the MMAU coefficient can also affect SWE inputs with a nonzero MMAU posterior.

The original experiments used the posterior-weighted rule above. We have revised imprecise or ambiguous wording in the original description and added implementation details to the main text, explaining how the four task-class cost components are combined using posterior probabilities. These clarifications and additions leave the routing implementation and original experimental results unchanged.

Min–max scaling puts the corresponding cost vectors on a common range. For SWE, logarithmic scaling accommodates its wider execution-cost range:

ϕSWE(c^SWE)m=log⁡(1+c^SWE,m/aSWE)−log⁡(1+min⁡jc^SWE,j/aSWE).\phi_{\mathrm{SWE}}(\hat{\mathbf c}_{\mathrm{SWE}})_m=\log(1+\hat c_{\mathrm{SWE},m}/a_{\mathrm{SWE}})-\log(1+\min_j\hat c_{\mathrm{SWE},j}/a_{\mathrm{SWE}}).

The scale a_SWE is the median of positive candidate-wise mean training costs. The coordinate-wise search uses

{0, 0.05, 0.1, 0.2, 0.4, 0.8, 1.0, 1.2, 1.4, 1.8, 2.0}.

The accuracy–cost plot is available at this link. The experimental protocol and cost-configuration appendix specify the validation-selection procedure and provide a more detailed explanation of the posterior-weighted inference rule and its implementation.

AuthorsOpenReview ↗

Question 3 & Weakness 2:Provide the accuracy-vs-cost curve for a fixed B-ON checkpoint as λ varies (the sweep Fig. 2a disclaims). Does B-ON Pareto-dominate B-OFF and always-Flash along that curve?

Question 3

Provide the accuracy-vs-cost curve for a fixed B-ON checkpoint as λ varies (the sweep Fig. 2a disclaims). Does B-ON Pareto-dominate B-OFF and always-Flash along that curve?

Related W2 (success ranking and fixed-checkpoint evidence):

The cost penalty raising accuracy (73.82 → 76.44) says the success head isn’t ranking candidates on its own. And the one thing the decoupled cost head is for — sweeping λ on a fixed checkpoint — is never shown; Fig. 2a’s caption explicitly disclaims it.

Answer

Several measured B-ON settings improve both accuracy and cost over B-OFF; the comparison with DeepSeek-V4-Flash-0731 shows higher accuracy at a nearby budget. We fix the B-ON checkpoint and predictions, varying one coordinate of (0,0.2,0.05,0.2) at a time while holding the other three fixed. Accuracy and selected-model execution cost cover all 382 test tasks.

The accuracy–cost plot is available at this link.

Table-3. Representative operating points. B-OFF is a separately trained comparison; the B-ON rows share one checkpoint.

Method or configurationAccuracy (%)Cost (USD)
DeepSeek-V4-Flash-073171.733.35
DeepSeek-V4-Pro-081372.2517.13
B-OFF73.8217.78
B-ON, reference vector76.4411.65
B-ON, lambda_MMAU = 0.474.083.77
B-ON, lambda_MMAU = 2.072.253.42

The reference B-ON setting improves accuracy over B-OFF by 2.62 percentage points and reduces cost by 34.49%. The lambda_MMAU = 0.4 point also improves both measures. At lambda_MMAU = 2.0, B-ON solves two more tasks than DeepSeek-V4-Flash-0731 for USD 0.07 additional cost (2.02%).

The cost penalty can also preserve aggregate success: changing lambda_MAB from 0 to 0.4 retains 289 successes while reducing cost from USD 12.91 to USD 10.19. We report these measured operating points and their budget differences in the fixed-checkpoint appendix; the Flash comparison is at nearby, rather than identical, execution cost.

The reference setting and the 74.08% / USD 3.77 setting Pareto-dominate the reported B-OFF configuration. The nearby-budget Flash comparison improves accuracy at slightly higher cost and therefore does not establish Pareto dominance over always-Flash.

Thank you again for these constructive questions. We hope the corrected comparison and additional analysis address your concerns about baseline identity, preference selection, and the scope of the cost-accuracy claims. If they resolve your concerns, we would appreciate your reconsideration of the paper and its rating. We would also welcome clarification of any remaining issue.

AuthorsOpenReview ↗

Follow-up on Our Responses to Your Additional Questions

Thank you again for your careful review. As the discussion period draws to a close, we wanted to follow up on our responses and the revisions to the manuscript. Do you have any remaining questions or concerns that we could clarify? If our responses and revisions have addressed your main concerns, we would appreciate your reconsideration of the paper and whether an upward revision of your rating would reflect your updated assessment. We would be happy to discuss any points that remain open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.