acceptodds
Under review as a conference paper at ICLR 2027

ORACLE: Agentic AI Orchestrator Routing via Adaptive Verifier Calibration Feedback

Abstract

Modern enterprise agent deployments consist of a heterogeneous pool of large language models (LLMs) having diverse capabilities and cost. Existing model routing strategies optimize the quality-cost trade-off, while providing request-level static decisions. More recent solutions address agentic routing as a task-level selection with a serial verifier based router feedback loop. However, their fixed verifier suitable for homogeneous workloads may not generalize to heterogeneous batches of agentic tasks (example: coding, general conversational). Additionally, due to the verifier placement in the critical path of the loop, serving quality may be affected during multiple concurrent requests routing. To mitigate these issues, we present ORACLE. It is a concurrency-aware online routing mechanism that aligns adaptive routing with adaptive verification for feedback. ORACLE acts as a training-free drop-in 'feedback loop' on top of any model-selection policy to first classify the task type and then dynamically assigns a task-appropriate verifier. Further, we develop a delayed feedback strategy for concurrent requests that largely removes verifier latency from the dispatch critical path. We then present a post-routing dispatch scheduler, namely DISC. DISC reserves each task's peak KV footprint at admission and dispatches to an alternate backend when the reward gain from reduced wait exceeds the reward loss from lower accuracy. Extensive evaluation on SWE-bench, -bench, and Terminal-Bench 2.0 shows that \NAME improves the accuracy–cost frontier by up to 7 percentage points over state-of-the-art routing baselines, while ORACLE with DISC improves program throughput by up to .

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic AI, LLM Routing, LLM Serving, Contextual Bandits

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_35v1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ORACLE, a concurrency-aware framework for routing multi-turn agentic tasks across heterogeneous language models. It addresses three challenges: inappropriate verification feedback across different task types, throughput losses from synchronous verification, and backend load imbalance under concurrent execution. The authors evaluate ORACLE on three benchmarks with Qwen and Mistral model pairs on GH200 and H200 GPUs.

Critical strengths
  1. ORACLE addresses an important systems problem by combining task-appropriate verification, asynchronous shared-policy updates, and program-level resource management within a modular architecture.
  2. The evaluation covers multiple agentic task families, two model pairs, and different GPU configurations. The reported improvements in accuracy–cost trade-offs and program throughput are potentially significant.
Critical weaknesses
  1. The adaptive-verification baseline is artificially weak, limiting the significance of the main accuracy gains. The principal comparison applies a software-patch verifier to all task families, explicitly assigning failure feedback tasks regardless of their actual success (Section 5.1). Consequently, much of the reported improvement follows from replacing systematically incorrect feedback with benchmark-appropriate feedback. Although this is a valid demonstration of verifier mismatch, it does not establish superiority over a straightforward task-aware baseline using benchmark metadata, a rule-based verifier selector, or a stronger learned verifier-selection method.
  2. The evidence for online generalization and robust scheduling performance is limited. The scheduling results are based on small live task sets and few execution seeds, including only one seed for a reported Mistral throughput comparison. Furthermore, the DISC sensitivity analysis shows that increasing throughput can reduce the number of successfully resolved tasks substantially. A more comprehensive assessment of end-to-end task utility, accuracy–throughput trade-offs, and statistical variability is needed.
Questions for authors

Please see weaknesses.

Reviewer_jnMBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ORACLE integrates task-dependent verifier selection, shared asynchronous feedback, and KV-cache reservation-based scheduling through DISC to improve the accuracy–cost trade-off and concurrent execution efficiency of multi-model agent systems. Its main contribution is a systems integration of feedback learning and serving, rather than a new model-selection algorithm.

Critical strengths

The paper addresses an important interaction between model selection and serving constraints. The strongest evidence concerns DISC: admission-control and backend-reassignment ablations, together with resolved-task throughput across different model and hardware configurations, support its practical value under high concurrency and cache contention, beyond improvements in token-generation speed alone.

Critical weaknesses
  1. The verifier comparison does not sufficiently establish the claimed advance. The patch-only baseline in §5.1 assigns failure feedback to every τ²-bench and Terminal-Bench task. The approximately 7-percentage-point improvement therefore primarily demonstrates the benefit of applicable feedback over inapplicable feedback, rather than superiority over an appropriately configured feedback-aware router. Since benchmark/domain metadata already enter the router, and Table 1 reports near-equivalence to fixed per-type verification, a metadata-based verifier mapping should be included as a reference on the main mixed stream. Moreover, §3.3 of the original ACRouter paper already specifies task-dependent verification tools. Please distinguish limitations of the patch-only implementation used here from limitations of the prior framework itself.

  2. Adaptation to repeated tasks is not separated from generalization to unseen tasks. Figure 4 evaluates the final pass after eight passes over the same task set. Appendix E.6 explicitly states that ACRouter retrieves earlier verdicts of identical tasks and attributes its above-single-model performance to this recall. This protocol is legitimate for recurring workloads, but does not establish the accuracy–cost benefit on previously unseen tasks. Please use task-disjoint warm-up and evaluation sets while holding context features fixed. The cold-start experiment in Appendix E.8 also changes the feature configuration, preventing a clean attribution to accumulated feedback alone.

  3. The τ²-bench evaluation confounds agent selection with the interaction environment. The selected model serves as both the agent and the user simulator. Changing the model therefore changes the environment as well as the agent, so success-rate differences cannot be attributed cleanly to agent selection.

Questions for authors

Please check the weakness part.

Reviewer_EFq5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for feedback-aware model routing under concurrent agentic workloads. The framework, ORACLE, consists of three components: 1) adaptive verification, which selects a task-specific verifier using embeding-based classification, rather than applying the same verifier to all tasks; 2) delayed feedback, whichasynchronously updates a shared routing policy without blocking new tasks; and 3) DISC, a scheduler that uses program-level KV-cache reservations and estimated accuracy–wait trade-offs to balance workloads across model backends.

Experiments on SWE-bench Verified, τ²-bench, and Terminal-Bench 2.0 show that adaptive verification improves routing accuracy by up to 7 percentage points at matched cost, while DISC improves program throughput by up to 1.8× over ThunderAgent.

Critical strengths
  1. This paper focuses on the interaction between routing feedback and concurrent agent execution. The proposed method, ORACLE, maintains a shared routing policy that incorporates verification results asynchronously, rather than waiting for entire batches or maintaining independent learners for each concurrent slot. Figure 5 shows that this design achieves 1.7× higher throughput than batch synchronization at 32 concurrent tasks, while resolving 7–13 more tasks than independent learners across the evaluated concurrency levels. This result verifies the effectiveness of the proposed feedback handling mechanism.
Critical weaknesses
  1. The evaluation of adaptive verification relies on a relatively weak baseline. According to Section 5.2, the single verifier baseline uses a patch verifier for all tasks, which always returns failure feedback for τ²-bench and Terminal-Bench tasks. Therefore, the reported accuracy improvement mainly demonstrates the benefit of using appropriate verifiers rather than the effectiveness of the proposed verifier-selection method. It is unclear what advantage it provides over directly assigning verifiers based on task types, especially when the routing experiments already use benchmark and domain metadata.

Besides, the adaptive verification assumes a predefined set of verifiers with labeled exemplars. This assumption would limit its applicability beyond the evaluated benchmarks. How would ORACLE handle tasks that do not clearly belong to any existing verifier class?

  1. The main experiment does not sufficiently validate the claimed effectiveness of online routing under heterogeneous concurrent workloads. In Figure 4, the authors report main accuracy–cost results after 8 passes over the same task stream. Besides, in Appendix E.8, the result of cold-start setting does not show significant improvement over random routing.

Meanwhile, the live concurrency experiments in Figures 5–7 only use SWE-bench tasks. It remains unclear how well ORACLE generalizes to previously unseen tasks in realistic concurrent deployments.

Questions for authors
  1. It is reasonable to adjust the model assignment according to GPU resources. However, the evaluation of the proposed DISC scheduler (including Table 5) mainly considers static deployments of two models on 1-2 GPUs. Real-world serving systems often involve multiple GPU replicas, multiple models, and tensor parallelism. Would DISC achieve similar improvements in these more complex serving settings?

  2. Could the authors compare the proposed adaptive verification against a simple task-type-based verifier assignment baseline? Given the high classification accuracy (see Appendix B.3) and the availability of benchmark information, this comparison would help establish the contribution of the verifier-selection mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.