acceptodds
Withdrawn from ICLR 2027

Agent-as-a-Router: Closing the Feedback Loop in Model Routing for Coding Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Users of coding agents increasingly have access to several frontier Large Language Models (LLMs) that excel on different tasks, so routing each task to a suitable model matters for both quality and cost. Most prior open-source routers treat this as a one-shot classification problem whose information state is frozen after training. We argue that routing for agents is instead a streaming decision problem with verifiable feedback since gathering information matters for routing. A controlled diagnosis supports this view: supplying per-dimension performance statistics to an LLM router raises its score by 3–9 points over the same statistics with shuffled model labels, whereas scaling the router from 9B to 27B parameters or eliciting longer reasoning does not. We therefore formalize routing as a ContextActionFeedback (C-A-F) loop evaluated by cumulative regret against a per-task oracle, instantiate it as ACRouter (an Orchestrator, a Verifier, and an execution-grounded Memory), and release CodeRouterBench, 10K coding tasks with per-task outcomes for 8 frontier LLMs plus a 176-task out-of-distribution (OOD) agentic-programming split. ACRouter attains the lowest cumulative regret on in-distribution streams, and component ablations show that closing the loop accounts for most of the gain over retrieval from a frozen memory. On OOD tasks, static routers trained offline fall to 24–39% resolve rate, at or below random routing. With an exact verifier, verification-driven escalation resolves 73.3% of OOD tasks, 9.1 points above the strongest single model; with deployable verifiers that do not read the grading tests, the gain falls within noise.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Code Agent, Router, Efficient Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AjZvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Agent-as-a-Router, a framework that uses execution feedback to improve model selection for coding tasks. The authors implement it via ACRouter(an Orchestrator, a Verifier, and an execution-grounded Memory). They also introduce CodeRouterBench, which covers ~ 10K coding tasks and 8 LLMs.

The experiments show that the past execution results help to improve routing on in-distribution tasks. On out-of-distribution (OOD) agentic tasks, the system uses a fixed sequence of models with verification-based escalation. The results show that online feedback can help routing, but the OOD gains depend on verifier accuracy.

Critical strengths
  1. The paper is related to an important problem and also provides a broad evaluation with a bench dataset. The is useful to compare routing quality and cost. The controlled experiment in Table 1 is helpful to understand the problem and motivation. Overall, the paper is easy to follow.

  2. The ID results shows that online feedback helps. In Table 4(b), updating the online-kNN router with feedback improves average performance from 47.32% to 48.74%. This is meaningful and it tests whether learning from previous tasks improves future decisions. The

Critical weaknesses
  1. The OOD results do not show the benefit of the online routing method in a clear way. The ID system uses a trained policy and online memory, but the OOD experiments follow a fixed sequence of models (Algorithm 1). The reported 73.30% resolve rate also relies on an exact verifier that uses benchmark grading results. With practical verifiers, performance drops to 64.77–65.91%, which is not significantly better than GPT-5.4 at 64.20% (Table 5). It difficult to highlight the benefit of online routing compared to simply try multiple models with accurate verification. A comparison with repeated attempts by a single strong model under the same budget is needed.

  2. The OOD evaluation has important data limitations. Appendix E.5 states that 406 of 896 task–model runs on the original 112 SWE-bench tasks never started, but were counted as failures with zero cost. The additional 64 tasks also use different model versions in several columns, and their costs are estimated because token logs were unavailable. I am concerned these could affect both the performance and cost. The paper needs to show that its core conclusions still hold when these issues are addressed.

Questions for authors
  1. Since the OOD experiments use a fixed model sequence, can the authors show whether online memory provides any additional benefit over this fixed cascade? How does the method compare with repeated attempts using a single strong model under the same budget?

  2. Why did 406 task–model runs fail to start? Can the authors provide a complete evaluation with consistent model versions, or show how sensitive the main results are to these missing runs?

  3. How closely do the verifier scores match the actual benchmark scores across different coding tasks? Since the method learns from these scores, understanding their reliability is important.

Reviewer_85pvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reframes LLM routing for coding as a closed Context-Action-Feedback loop evaluated by cumulative regret against a per-task oracle, with reward defined as score minus 0.1 x cost. A diagnostic study (Table 1) argues that routers benefit mainly from accurate per-dimension statistics: correct statistics beat shuffled ones by +3.2/+9.4/+7.7 points at 0.8B/9B/27B, while scaling 9B to 27B or adding longer reasoning gives no consistent gain. The proposed ACRouter combines an Orchestrator (a voting ensemble of a LoRA-tuned Qwen3.5-0.8B, a Memory-kNN and a DimensionBest prior, with 5% exploration), a Verifier (AST checks, sandbox execution, prompt tests) and an online FIFO memory (20K entries, top-10, cosine >= 0.5). The authors also release CodeRouterBench (about 10K tasks over 9 dimensions from 15 sources, 8-model pool) with ID and OOD splits. ACRouter reports 49.98 AvgPerf and cumulative regret 205.5 in-distribution versus 47.50 / 278.1 for DimensionBest, at $13.21 versus $34.02 for Opus-only, and 73.30% on the 176-task agentic OOD split versus 64.20% for the best single model (GPT-5.4).

Critical strengths
  • Useful framing. Treating routing as an online, feedback-driven loop and reporting cumulative regret is a sensible shift from static one-shot routing and matches how routers are deployed.
  • Informative diagnostic. The shuffled-statistics control in Table 1 is a clean way to show what the router actually relies on, and the negative scaling/reasoning results are useful to the community.
  • Benchmark contribution. CodeRouterBench covers 9 coding dimensions with both ID and agentic OOD splits and an 8-model pool; releasing it would be valuable.
  • Cost reporting. Costs are reported alongside quality, and the online-memory ablation (241.2 vs 286.7 regret for online vs frozen kNN) supports the value of feedback.
Critical weaknesses
  • Central concern: deployable OOD advantage is not established. The 73.30% result uses an exact verifier that observes the harness outcome. Deployable verifiers reach 64.77% and 65.91%, versus 64.20% for GPT-5.4, with improvement intervals spanning zero. This is a limitation of the evidence for practical routing gains, not proof of equivalence between methods.
  • ID gains require stronger uncertainty analysis. The reported AvgPerf improvement over DimensionBest is about 2.5 points. The review should not treat that point estimate alone as a reliable improvement; stream-order and seed sensitivity are important to the online-learning claim.
  • Verifier signals on ID are proxies. AST/sandbox/prompt-test checks are weak proxies for correctness on many dimensions; the paper should quantify verifier precision/recall against ground truth.
  • OOD set construction. Of the 176 OOD tasks, 64 added tasks have imputed costs and no patches; results on the original vs added subsets should be separated.
  • Projected, not measured, ablations. Several ablation regret values are projections (Delta-Reg = N|Delta-Perf|/100) rather than measured runs. These should be measured or clearly separated from empirical results.
  • Cost-quality interpretation is narrow. With ID calls costing about one cent and a 0.1 cost weight, the reward is dominated by task score. A cost-weight sensitivity analysis is needed before interpreting the ID result as a substantial quality-cost trade-off.
Questions for authors
  1. Given the reported deployable-verifier results, can you demonstrate a statistically supported quality-cost advantage over the strongest single model after including verifier compute?
  2. Can you separate the OOD results for the 112 original tasks and the 64 added tasks with imputed costs, and quantify sensitivity to that imputation?
  3. Which ablation numbers are measured and which are projected? Can the projected ones be replaced by actual runs?
  4. How sensitive are results to the 0.1 cost weight, the 5% exploration rate and the 0.5 similarity threshold?
Reviewer_P3XBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that coding-task routing shuld be online: the router should keep learning from verified outcomes instead of classifying each task once. A neat diagnostic shows LLM routers improve when given real per-dimension statistics but not when the labels are shuffled, and that a bigger router doesn't help. ACRouter combines a small classifier, a kNN memory of past outcomes and a per-dimension prior, and on agentic tasks it escalates through a model chain until a verifier accepts. The authors also release a roughly 10K-task benchmark with outcomes for eight frontier models. ACRouter gains about 2.5 points in-distribution. On agentic tasks it resolves 73.3% against 64.2% for GPT-5.4, but only when the gradding harness itself serves as the verifier.

Critical strengths

The shuffled-label diagnostic is clean and likely to replicate. The paper is also candid that realistic verifiers erase the agentic gain, which is its most usseful lesson.

Critical weaknesses

The thesis is only shown with feedback a deployed system wouldn't have. On SWE-bench, 406 of 896 runs never started and were scored as failures, mostly for the cheap models that static routers favour. That undermines the claim that static routers fall to random under distribution shift. In-distribution, removing the memory barely hurts (49.65 against 49.98), so the gain mostly comes from a static ensemble.

Questions for authors

Does the in-distribution memory store a test-blind score or the ground truth? What do the agentic results look like restricted to runs that actually started? How was the escalation chain chosen? For future work, deployable verifiers should be tested on all agentic tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.