- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LongContextRouter, a task-level router that picks a LLM per agent task by predicting each candidate's chance of success and penalizing expected cost. It encodes the full task input hierarchically. It also uses execution traces from other models as extra training signal. Across four benchmarks it beats the strongest single model in accuracy while cutting cost by about a third.
Critical strengths
-
the identified scenario, choosing the model that is enough for the task and previous context, is practically useful.
-
the paper is written clearly and easy to follow.
-
The evaluation is comprehensive, spanning four diverse benchmarks with per-benchmark accuracy and cost, controlled ablations of each component, and full model-selection distributions.
Critical weaknesses
-
the central claim of the paper is not tested at inference time. The paper motivates the proposed approach with long-horizon agentic settings, but at test time the router only sees the query. And since a single problem statement usually fits within one 32K window, the hierarchical encoder and the context pathway are largely unused during evaluation.
-
baselines are limited. The paper does not compare with recent routers that use execution information for agent tasks, such as SWE-Router, which conditions task-level routing on partial trajectories at test time.
Questions for authors
Why does training-only context help a query-only router, and is the gain due to the context itself? Historical traces are included during training but C is empty at test time, yet B outperforms A. What mechanism do the authors believe explains this transfer?
5 comments
Question (continued)&Weakness 2
Weakness 2
baselines are limited. The paper does not compare with recent routers that use execution information for agent tasks, such as SWE-Router, which conditions task-level routing on partial trajectories at test time.
Our interpretation is that historical execution text provides additional evidence about task difficulty and candidate behavior when training the shared aggregator, model embeddings, and success predictor. Those learned parameters subsequently operate on new problem statements. This is a plausible explanation, rather than an experimentally isolated mechanism: the current comparison measures the combined history-based training configuration and does not separately identify the contributions of historical content, view augmentation, and training exposure. The reported comparison uses seed 43 and does not establish robustness across training seeds.
The added SWE-Router comparison addresses the missing execution-informed baseline. Our reproduction uses partial trajectories for selection and is evaluated on the same five-model pool and 382 tasks:
Costs cover selected-model execution only; SWE-Router’s partial-trajectory generation and router encoding, computation, and hosting are excluded. These results therefore do not establish an end-to-end cost advantage.
| Method | Overall success (%) | Execution cost (USD) | SWE success (%) | SWE execution cost (USD) |
|---|---|---|---|---|
| SWE-Router | 75.39 | 10.89 | 66.00 | 8.67 |
| LongContextRouter | 76.44 | 11.65 | 78.00 | 11.16 |
LongContextRouter solves 292/382 tasks versus 288/382, an improvement of 1.05 percentage points, at approximately USD 0.76 higher selected-model execution cost. On SWE-bench Pro, the improvement is 12 percentage points. Performance therefore varies across benchmarks; these aggregate results do not imply uniform superiority. For details on reproducing SWE-Router, please refer to our repository: anonymized repository.
Thank you again for prompting this clearer separation of the two evaluation settings. We hope the evidence of multi-window inference, the qualified interpretation of history-based training, and the additional baseline address your main concerns. If these responses resolve them, we would appreciate your reconsideration of the paper and its rating. We welcome further discussion of any remaining concern.
Follow-up on Our Responses to Your Additional Questions
Thank you again for your careful review. As the discussion period draws to a close, we wanted to follow up on our responses and the revisions to the manuscript. Do you have any remaining questions or concerns that we could clarify? If our responses and revisions have addressed your main concerns, we would appreciate your reconsideration of the paper and whether an upward revision of your rating would reflect your updated assessment. We would be happy to discuss any points that remain open.
Question1 & Weakness 1:Why does training-only context help a query-only router, and is the gain due to the context itself? Historical traces are included during training but C is empty at test time, yet B outperforms A. What mechanism do the authors believe explains this transfer?
Thank you for questioning whether the evaluation supports the paper's context-related claims and for identifying the missing execution-informed baseline. Your comments helped us distinguish two settings more clearly: long-context routing at inference on MemoryAgentBench, and history-based training for query-only routing on SWE-bench Pro. Below, we provide the corresponding input protocols, window statistics, and ablations, together with the added SWE-Router comparison. We also clarify which conclusions the experiments support and which aspects of the transfer mechanism remain unisolated. Substantive revisions and additions are highlighted in blue in the revised manuscript.
Question
Related W1:
Answer
Long-context aggregation is exercised at inference on MemoryAgentBench; query-only inference applies to SWE-bench Pro. The two benchmarks evaluate different uses of context:
All 100 MemoryAgentBench test tasks require multiple windows, accounting for 100/382 test tasks (26.18%), with up to five windows per input. The 32,768-token limit applies to each encoder window, rather than the complete routing input. The ordered window embeddings are combined by the Transformer before candidate-success prediction. Across training tasks, 137/867 (15.80%) require multiple windows, with up to four windows.
The relevant ablations are:
On MemoryAgentBench, B-ON solves five additional tasks relative to either ablation. B versus D retains context while changing aggregation; B versus A compares the context-enabled and query-only configurations. These results support the full configuration on the evaluated long-context tasks. They do not establish that gains increase with input length: all multi-window test tasks belong to MemoryAgentBench, and the configurations also differ in execution cost.
On SWE-bench Pro, the evidence concerns transfer from historical training views to query-only prediction. All configurations route the same 100 test problem statements without execution histories. B-ON achieves 78%, compared with 66% for A-ON, corresponding to 12 additional successful tasks.
History-based training expands 160 SWE tasks into 800 views using five candidate-model-specific execution contexts per task; 40 validation tasks similarly produce 200 views. Query-only training has 160 distinct training inputs because removing the execution contexts leaves the same problem statement across candidates. The configurations share original tasks, task-disjoint splits, the candidate pool, and full-task outcome labels. Views from one task remain in the same partition.