- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Agent-as-a-Router, a framework that uses execution feedback to improve model selection for coding tasks. The authors implement it via ACRouter(an Orchestrator, a Verifier, and an execution-grounded Memory). They also introduce CodeRouterBench, which covers ~ 10K coding tasks and 8 LLMs.
The experiments show that the past execution results help to improve routing on in-distribution tasks. On out-of-distribution (OOD) agentic tasks, the system uses a fixed sequence of models with verification-based escalation. The results show that online feedback can help routing, but the OOD gains depend on verifier accuracy.
Critical strengths
-
The paper is related to an important problem and also provides a broad evaluation with a bench dataset. The is useful to compare routing quality and cost. The controlled experiment in Table 1 is helpful to understand the problem and motivation. Overall, the paper is easy to follow.
-
The ID results shows that online feedback helps. In Table 4(b), updating the online-kNN router with feedback improves average performance from 47.32% to 48.74%. This is meaningful and it tests whether learning from previous tasks improves future decisions. The
Critical weaknesses
-
The OOD results do not show the benefit of the online routing method in a clear way. The ID system uses a trained policy and online memory, but the OOD experiments follow a fixed sequence of models (Algorithm 1). The reported 73.30% resolve rate also relies on an exact verifier that uses benchmark grading results. With practical verifiers, performance drops to 64.77–65.91%, which is not significantly better than GPT-5.4 at 64.20% (Table 5). It difficult to highlight the benefit of online routing compared to simply try multiple models with accurate verification. A comparison with repeated attempts by a single strong model under the same budget is needed.
-
The OOD evaluation has important data limitations. Appendix E.5 states that 406 of 896 task–model runs on the original 112 SWE-bench tasks never started, but were counted as failures with zero cost. The additional 64 tasks also use different model versions in several columns, and their costs are estimated because token logs were unavailable. I am concerned these could affect both the performance and cost. The paper needs to show that its core conclusions still hold when these issues are addressed.
Questions for authors
-
Since the OOD experiments use a fixed model sequence, can the authors show whether online memory provides any additional benefit over this fixed cascade? How does the method compare with repeated attempts using a single strong model under the same budget?
-
Why did 406 task–model runs fail to start? Can the authors provide a complete evaluation with consistent model versions, or show how sensitive the main results are to these missing runs?
-
How closely do the verifier scores match the actual benchmark scores across different coding tasks? Since the method learns from these scores, understanding their reliability is important.