acceptodds
Under review as a conference paper at ICLR 2027

Ask Your Advisor: Learning When to Consult Stronger Models in Agentic Systems

Abstract

To balance task performance and computational cost, agentic workflows can use a lower-cost model for execution and selectively consult a stronger model. In such executor–advisor systems, the executor decides when to seek advice, making escalation depend on self-assessment, although current language models struggle to detect where their own reasoning fails. The need for stronger reasoning emerges during execution and varies across workflow phases. We introduce ARIA, a meta-controller that learns when to consult by balancing task success against consultation cost. Our approach formulates consultation as a sequential resource-allocation problem. At workflow contact points, ARIA maps execution evidence to a compact state abstraction and selects SILENT or CONSULT, with action values learned by Monte Carlo control from cost-penalized outcomes. Across evaluated tasks of SWE-bench Verified, Terminal-Bench 2.1, and -bench, ARIA recovers 44–81% of the executor-advisor success gap at less than half the advisor model’s cost. ARIA achieves the best observed success-cost trade-off among ten other evaluated conditions, including routing and cascading methods, fixed and prompted consultation, and random placement at matched advice volume. The policies are compact and inspectable, with a single consultation-price parameter governing the success–cost trade-off without retraining either underlying model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic systems, multi-agent orchestration, cost-quality optimization, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SEpoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARIA, a Q-table controller that uses workflow phase, execution evidence, and consultation history to decide when a lower-cost executor should consult a stronger advisor, without fine-tuning either model. Across three agentic benchmarks, ARIA recovers 44%–81% of the executor–advisor success gap at less than half the evaluation cost of advisor-only execution.

Critical strengths
  1. The results provide evidence that consultation timing matters. On SWE-bench, ARIA and Random make a similar number of consultations per episode, yet achieve success rates of 48.0% and 28.0%, respectively.
  2. The method is lightweight and inspectable. It leaves the underlying models unchanged, exposes its decisions through a Q-table, and reports reconciled model costs.
Critical weaknesses
  1. Generalizability is limited. ARIA learns a separate Q-table for each benchmark from only 20–52 distinct development tasks, many of which are revisited; the SWE-bench and Terminal-Bench development sets are also screened based on model outcomes. Evaluation uses held-out tasks from the same benchmark families, but the paper does not show that a learned policy transfers to new task families, model pairs, or workflows. The evidence therefore supports benchmark-specific consultation policies more strongly than a general ability to learn when to consult.
  2. The coarse state representation limits the methodological contribution. ARIA averages Monte Carlo returns for each state–action pair and chooses the action with the higher empirical mean. Different tasks with the same phase, evidence category, and consultation history receive the same decision, regardless of their specific information gaps. Without a comparison to a phase-only policy learned from the same data, it is unclear whether the proposed state representation adds value beyond a simple learned consultation schedule.
Questions for authors
  1. How does ARIA compare with an executor that decides for itself when to consult, under the same contact points, advisor interface, and budget?
  2. Does adding a small amount of task-specific information improve consultation decisions?
  3. Why was the policy learned from so few distinct tasks when larger task pools were available for SWE-bench Verified and τ²-bench Telecom?
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer SEpo

We thank the reviewer for the detailed feedback. We address the concerns about state representation, executor-driven consultation, and generalization, supported by additional evaluations across all benchmarks.

W2: Methodological contribution and state representation

ARIA's contribution is a lightweight framework for learning when to consult stronger black-box models during execution, without modifying either model. Its compact 24-state representation aggregates recurring workflow situations across tasks, allowing experience to be shared when estimating consultation values. This follows established principles of state abstraction in reinforcement learning (Abel et al., 2016), balancing task-specific detail against policy complexity.

To directly address the reviewer's proposed control, we evaluated a learned four-state Phase-only policy fitted from the same training trajectories, alongside a Delivery-only strategy retaining ARIA's review-and-revision mechanism. Both use the same model pair, advisor interface and consultation budget.

BenchmarkPolicySuccessCost/RunER
SWE-benchARIA48.0%$22.090.37
Delivery-only42.7%$22.610.24
Phase-only40.7%$22.840.19
Terminal-BenchARIA34.7%$24.930.15
Delivery-only32.7%$25.780.07
Phase-only29.3%$30.97−0.15
τ²-benchARIA93.3%$7.770.65
Delivery-only82.5%$6.420.31
Phase-only86.7%$9.230.31

ARIA achieves the highest observed success and excess recovery (ER) across all three benchmarks. Neither Phase-only nor Delivery-only reproduces its performance, providing direct empirical support for state-conditioned consultation beyond simpler strategies. We acknowledge that individual state features are not isolated by these comparisons and that resource-terminated SWE-bench evaluations, particularly for Phase-only, limit attribution of the full observed differences solely to the policies.

Q1: Executor-driven consultation

We would like to clarify that this comparison is mostly addressed by our existing LLM-Router baseline (Section 5.1, Appendix B.2). Here, the executor (to be precise the orchestrator agent) independently decides when to consult the advisor, without ARIA's learned policy. On SWE-bench, LLM-Router achieves 29.3% success versus ARIA's 48.0%, while consulting substantially less frequently (0.41 vs. 2.78 consultations per episode).

This provides a relevant comparison with a native executor-driven consultation. However, LLM-Router does not implement ARIA's dedicated pre-delivery review-and-revision mechanism. This distinction is intentional: LLM-Router evaluates the executor's native, self-directed consultation behavior without imposing ARIA's pre-delivery revision mechanism. It therefore addresses the reviewer's request for an executor-driven baseline, while preserving the executor's natural consultation behavior.

Q2: Task-specific information

ARIA deliberately uses observable workflow signals rather than task-specific representations, allowing its compact policy to aggregate experience across heterogeneous tasks. The executor and advisor already receive task context, but this information is not directly encoded in the consultation controller. We agree that incorporating additional task-specific features could further improve consultation decisions and consider this a valuable extension, although its incremental benefit has not yet been isolated experimentally.

W1/Q3: Development tasks and generalization

ARIA learns a compact 24-state Q-table rather than task-specific solutions or model parameters. Repeated task executions follow standard Monte Carlo reinforcement learning practice, where stochastic trajectories provide additional state-action observations for estimating expected returns (Sutton & Barto, 2018). Under a limited training budget, we prioritized outcome-screened development tasks where differences in executor and advisor performance suggested that consultation could meaningfully affect success. This follows the broader motivation of informative task selection in curriculum reinforcement learning (Narvekar et al., 2020), concentrating training on potentially decision-relevant cases rather than simply selecting successful tasks.

We acknowledge that outcome-based screening may introduce selection bias and that repeated trajectories cannot replace broader task diversity. Importantly, training and evaluation tasks remain disjoint, providing evidence of generalization to unseen tasks across three benchmark families.

Reference

Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., & Stone, P. (2020). Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey. Journal of Machine Learning Research, 21(181), 1-50. https://jmlr.org/papers/v21/20-212.html

Reviewer_1zHvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ARIA, an external controller for deciding when a fixed executor model should consult a stronger advisor during an agentic workflow. It encodes workflow history as 24 combinations of temporal position, execution evidence, and prior consultation. The proposed learning objective combines terminal task success with a penalty for consultation, and policies are learned through tabular Monte Carlo control. Separate controllers are evaluated on held-out tasks from SWE-bench Verified, Terminal-Bench 2.1, and τ²-bench Telecom using Magentic-One and a Gemini executor–advisor pair. The experiments compare task success, model cost, consultation placement, and residual failure patterns.

Critical strengths
  1. Determining when additional model capability justifies its cost is an important problem for agentic systems. A small external controller that operates with fixed, black-box models is a practical design direction.

  2. The evaluation spans three different task domains and includes fixed schedules and random consultation alongside autonomous models. These comparisons offer useful evidence that the timing of consultation can affect the success–cost trade-off within the evaluated workflows.

Critical weaknesses
  1. The current experiments do not establish the adequacy of the 24-state abstraction. In Section 3.2 and Appendix A.2, FRAME/EXPLORE/REASON are assigned according to the first/next-two/later contact points rather than measured task progress. PRE_STEP and STALL can map to the same state despite different intervention semantics, while the binary consultation-history feature groups together one and three previous consultations despite their different remaining budgets under B = 4. These choices can pool situations that differ in their recovery needs and their reasons for preserving advisor access for later. Although the paper offers an intuitive rationale for this compression, all experiments use the same mapping. Neither the policy heatmaps nor the within-state analysis in Appendix C.5 establishes whether the information discarded by this mapping changes the preferred action. Important missing controls include a phase-only table, separate ablations of the evidence and history features, and variants that retain contact-point type and remaining budget, all under matched training and consultation budgets. This concern is particularly concrete on SWE-bench: 53.5% of consultations occur at DELIVER (Section 5.3), where advice can allow an additional revision cycle (Appendix A.1). A delivery-only policy with the same revision rules would help determine whether the full state-conditioned controller adds value beyond a simple final-review strategy.

  2. The deployment scope and adaptation requirements remain insufficiently validated. The paper clearly specifies that both language models remain fixed while the external controller is trained; the unresolved question is how reusable the learned controller is. Each benchmark uses its own policy and cost calibration, with 20–52 development tasks revisited over approximately 150–162 episodes, followed by evaluation on 50/50/40 disjoint tasks (Appendices A.3 and B.1). This supports generalization to held-out instances within each benchmark family, but does not test whether a frozen controller transfers across task families, model pairs, or orchestration frameworks. All experiments use one Gemini pair and Magentic-One, and Appendix A.2 explicitly describes the concrete event mapping as framework-specific. Although Section 6 acknowledges that the method is most useful within related workflows, the extent to which a workflow can change before its state mapping or policy requires adaptation remains unmeasured. This matters because the signals themselves change meaning across settings: Section 5.3 notes that τ² errors are often semantic or procedural and may not produce explicit error evidence, while unvisited states default to SILENT. Training also requires terminal success labels, but their availability outside benchmark harnesses is not examined. Finally, the headline costs exclude training and pilot runs (Appendix B.4). Without transfer tests, learning curves for adaptation, and an accounting of the costs of collecting feedback and relearning the controller, the practical benefit for a new application remains uncertain.

Questions for authors
  1. Under matched training and consultation budgets, how does the 24-state policy compare with phase-only control and variants that retain contact-point type and remaining budget?

  2. How sensitive are the policy and its success–cost trade-off to the first/next-two/later stage boundaries and to changes in the frequency of contact points?

  3. What transfers without modification when the task family, model pair, or orchestrator changes, and how does a frozen controller compare with one trained for the new setting?

  4. What success labels and retraining does a new application require, and how many deployment tasks are needed to amortize the costs of data collection and controller training?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer 1zHv

We thank the reviewer for the detailed feedback and concrete suggestions. We conducted additional experiments to address the concerns about state abstraction and clarify ARIA's generalization and deployment requirements.

W1/Q1: State representation and simpler alternatives

We agree that the 24-state abstraction requires comparison against simpler alternatives. We therefore conducted additional online evaluations across all three benchmarks: Delivery-only, retaining ARIA's review-and-revision mechanism, and Phase-only, a four-state policy learned from the same training trajectories. Both retain the same model pair, advisor interface, and consultation budget.

BenchmarkPolicySuccessCost/RunER
SWE-benchARIA48.0%$22.090.37
Delivery-only42.7%$22.610.24
Phase-only40.7%$22.840.19
Terminal-BenchARIA34.7%$24.930.15
Delivery-only32.7%$25.780.07
Phase-only29.3%$30.97−0.15
τ²-benchARIA93.3%$7.770.65
Delivery-only82.5%$6.420.31
Phase-only86.7%$9.230.31

ARIA achieves the highest observed success and excess recovery (ER) across all three benchmarks. Delivery-only does not reproduce its performance despite identical revision rules, while Phase-only achieves a weaker success-cost trade-off. These results provide direct empirical support for the full state-conditioned controller beyond simpler alternatives.

We further examined individual state features, contact-point type, and remaining budget through offline sensitivity analyses. While these checks help characterize the abstraction, they do not isolate the online contribution of each feature. Resource-terminated SWE-bench evaluations, particularly for Phase-only, also limit attribution of the full observed differences only to the policies.

Q2: Stage boundaries and contact-point frequency

We investigated stage-boundary sensitivity through additional offline analyses. Alternative boundaries changed 12-16% of consultation decisions on SWE-bench and 8-10% on Terminal-Bench, indicating that most decisions remained stable. However, these analyses do not establish equivalent task performance, and τ²-bench showed greater sensitivity. We agree that independently evaluating sensitivity to contact-point frequency would be particularly valuable when extending ARIA to other orchestration frameworks, and plan to investigate this in an updated version.

W2/Q3: Generalization and adaptation

ARIA applies the same controller design and learning procedure across three distinct task environments, with separately trained policies evaluated on disjoint, held-out tasks. This supports generalization within benchmark families and demonstrates that the learning framework is applicable across different task settings.

However, framework applicability and transfer of a frozen policy are distinct questions. Changes in task families, model pairs, or orchestration structures may require revised state mappings, cost calibration, or policy retraining. We do not claim that a frozen policy retains its success-cost trade-off across such changes, and agree that controlled transfer and adaptation experiments would provide valuable additional evidence of broader applicability.

W2/Q4: Training requirements and amortization

ARIA learns from terminal success feedback, provided by benchmark validators in our experiments. Each policy is trained on episodes, where repeated stochastic executions provide additional state-action observations for value estimation. Once trained, the controller requires only observable workflow signals, without additional success labels or model updates. Adapting to substantially different applications may require new labeled trajectories and retraining; obtaining reliable outcome feedback outside benchmark harnesses remains an important practical consideration.

We additionally quantify training expenditure: USD 49.77 (SWE-bench), USD 67.42 (Terminal-Bench), and USD 36.26 (τ²-bench) per final training run. Relative to Advisor-only execution, the resulting inference savings amortize these training costs after approximately 94-105 deployment tasks, suggesting promising economic potential for recurring agentic workflows.

Reviewer_KqdvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies when a weaker agent should consult a stronger model during task execution. The authors propose ARIA, a cost-aware meta-controller that makes consultation decisions based on workflow progress, execution feedback, and consultation history. The policy is learned through tabular Monte Carlo control, balancing task success against consultation cost without modifying the underlying models. The authors report improvements in success–cost trade-offs over several baselines. Further analyses examine how consultation timing affects performance and how failures to incorporate advisor guidance limit agent collaboration.

Critical strengths

(1) Cost-Aware Consultation Mechanism. ARIA separates consultation decisions from the executor's self-assessment and formulates them as a cost-aware sequential decision problem. Learning from observable workflow signals and task-level outcomes provides an alternative to confidence-based or heuristic consultation strategies. The controller is also lightweight and interpretable, requiring neither fine-tuning of the underlying models nor access to their internal information. (2) Empirical Analysis of Consultation Timing. The paper compares learned consultation policies with fixed schedules and approximately rate-matched random consultation under a shared advisor intervention mechanism. These comparisons help distinguish the effects of consultation timing from consultation frequency. The additional failure analysis identifies cases where the executor fails to incorporate advisor guidance, providing further observations on the limitations of executor–advisor collaboration.

Critical weaknesses

(1) The methodological novelty of the paper is limited. ARIA mainly applies standard tabular Monte Carlo control to consultation decisions, without introducing substantial advances over existing sequential decision-making and resource allocation methods. While the problem itself is practically relevant, the paper does not clearly establish its methodological contribution beyond prior work on selective agent collaboration. (2) The paper focuses on balancing task success and inference cost, but its objective uses a fixed penalty per consultation, whereas actual model costs vary with consultation timing and execution trajectories. This creates a gap between the training objective and the cost metric used in evaluation, and the paper does not sufficiently examine how this approximation affects the learned policy. (3) Although the paper includes several baselines, many address task-level model routing rather than consultation during execution, making the comparisons less direct. Among the more comparable consultation strategies, ARIA's improvements are inconsistent; for instance, it achieves the same success rate as Heuristic on Terminal-Bench. The lack of comparisons with alternative learning-based consultation policies also makes it difficult to establish the necessity of the proposed approach. (4) The evaluation is restricted to a single executor–advisor model pair and one agentic workflow framework, with separate consultation policies trained for each benchmark. Given that consultation decisions depend on the underlying workflow structure, the current results provide limited evidence of generalization across model combinations or agent architectures, and the transferability of the learned policies remains unclear.

Questions for authors

(1) The paper attributes its performance gains to the learned consultation policy, yet more than half of ARIA's consultations on SWE-bench occur at the final delivery stage. Could simpler strategies, such as consulting only before delivery or allocating consultations based on workflow stages, achieve comparable results? What additional benefit does the sequential decision-making formulation provide over simpler learning approaches, such as contextual bandits? (2) ARIA's improvements over directly comparable consultation baselines are not consistent across benchmarks. For instance, ARIA matches Heuristic in success rate on Terminal-Bench while achieving lower cost, and Random does not strictly match ARIA's consultation volume on τ²-bench. Could the authors provide more tightly controlled comparisons and paired statistical analyses to establish the reliability of the reported improvements? (3) ARIA uses a fixed penalty per consultation during training but evaluates efficiency using actual inference costs. How well does this approximation reflect the cost differences across execution stages, and how might it affect the learned policy? Additionally, given that separate training trajectories must be collected for each domain, how does the controller's training cost affect the overall cost–benefit trade-off? (4) The evaluation uses a single model pair and workflow framework, with separate policies trained for each benchmark. Is there evidence that the proposed approach remains effective with different model combinations or workflow structures? To what extent can the current results support the broader claims about its applicability to agentic systems?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer Kqdv (1/2)

We thank the reviewer for the constructive feedback. We address the concerns through additional experiments and clarify ARIA's methodological contribution, cost modeling, baseline comparisons and generalization scope.

W1/Q1: Contribution and simpler consultation strategies

ARIA addresses a distinct question from task-level model routing: when is consulting a stronger model worthwhile during ongoing execution? We learn an external, cost-aware consultation policy from observable workflow signals and terminal outcomes, without modifying either model. Unlike approaches relying on model-internal signals (STEER; Lee et al., 2026) or executor fine-tuning (SWE-Protégé; Kon et al., 2026), ARIA operates with black-box models. Our contribution is thus the formulation and practical learning of sequential consultation decisions, rather than a new RL algorithm.

To directly test whether simpler strategies are sufficient, we conducted additional online evaluations across all three benchmarks: a Delivery-only strategy retaining ARIA's review-and-revision mechanism and a learned Phase-only policy fitted from the same training trajectories. Both use the same model pair, advisor interface, and consultation budget.

BenchmarkPolicySuccessCost/RunER
SWE-benchARIA48.0%$22.090.37
Delivery-only42.7%$22.610.24
Phase-only40.7%$22.840.19
Terminal-BenchARIA34.7%$24.930.15
Delivery-only32.7%$25.780.07
Phase-only29.3%$30.97−0.15
τ²-benchARIA93.3%$7.770.65
Delivery-only82.5%$6.420.31
Phase-only86.7%$9.230.31

ARIA achieves the highest observed success and excess recovery (ER) across all three benchmarks. Neither Delivery-only, despite identical revision rules, nor learned Phase-only control reproduces its performance. These results strengthen the empirical case for state-conditioned consultation, although resource-terminated SWE-bench evaluations, particularly for Phase-only, limit attribution of the full differences solely to the policies.

We model consultation as a sequential decision problem. A consultation changes how execution continues and reduces the remaining advisor budget. Its value therefore depends on what happens afterwards. Contextual bandits do not capture this, since they assume that actions do not affect future contexts. ARIA instead estimates the expected return of each decision, combining the terminal outcome with later consultation penalties. We use every-visit Monte Carlo for this estimate (Sutton & Barto, 2018). It learns directly from observed returns without bootstrapping, which suits the sparse terminal reward but increases variance. We do not claim superiority over contextual bandits, but consider Monte Carlo a suitable estimator for this sequential resource-allocation problem.

AuthorsOpenReview ↗

Author Response to Reviewer Kqdv (2/2)

W2/Q3: Consultation costs and training overhead

Our fixed consultation penalty follows established resource-allocation approaches, including advice budgets (Torrey & Taylor, 2013), fixed intervention costs (Cortes et al., 2016), and Lagrangian penalties (Altman, 1999). It regulates advisor usage rather than assuming uniform API costs; evaluation reports actual inference costs.

A preliminary offline analysis using measured consultation costs leaves the greedy decisions unchanged, although this does not replace retraining under actual pricing. Our λ-ablation further illustrates the success–cost trade-off: higher penalties reduce consultation frequency and expenditure, but also task success. A final training run costs USD 49.77, USD 67.42, and USD 36.26 on SWE-bench, Terminal-Bench, and τ²-bench. Relative to Advisor-only execution, these training costs are amortized through inference savings after approximately 94-105 deployment tasks, suggesting economic potential for recurring workflows.

W3/Q2: Baseline comparisons and statistical reliability

We consider routing baselines relevant because both routing and consultation aim to improve task success while reducing stronger-model inference costs. While routing provides a system-level comparison, Plan, Heuristic, Always, and Random, alongside our new Delivery-only and Phase-only baselines, share ARIA's intervention mechanism, enabling more direct comparisons of consultation strategies.

We acknowledge that improvements vary across benchmarks. On Terminal-Bench, ARIA matches Heuristic in success (34.7%) while reducing evaluation costs by 9.2%. As discussed in Section 6, greater task heterogeneity may contribute to this smaller advantage.

On τ²-bench, Random was calibrated before evaluation to approximate ARIA's consultation frequency but reached 87.8% of its consultation volume. Matching consultation probabilities does not guarantee equal realized counts, as each policy induces different trajectories and consultation opportunities under the same hard budget. We therefore acknowledge the mismatch and interpret this comparison as descriptive rather than strictly volume-matched.

W4/Q4: Generalization and transfer

Although our experiments use one model pair and orchestrator, ARIA applies the same controller design and learning procedure across three distinct environments. Each policy is evaluated on disjoint, held-out tasks, supporting generalization within benchmark families.

We distinguish this applicability from zero-shot transfer of a frozen policy, as each benchmark uses separately learned policies and cost calibration. Systematic transfer across model pairs and orchestration frameworks remains to be established. We agree that such evaluations would provide valuable additional evidence of ARIA's broader applicability and plan to investigate them in an updated version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.