- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ARIA, a Q-table controller that uses workflow phase, execution evidence, and consultation history to decide when a lower-cost executor should consult a stronger advisor, without fine-tuning either model. Across three agentic benchmarks, ARIA recovers 44%–81% of the executor–advisor success gap at less than half the evaluation cost of advisor-only execution.
Critical strengths
- The results provide evidence that consultation timing matters. On SWE-bench, ARIA and Random make a similar number of consultations per episode, yet achieve success rates of 48.0% and 28.0%, respectively.
- The method is lightweight and inspectable. It leaves the underlying models unchanged, exposes its decisions through a Q-table, and reports reconciled model costs.
Critical weaknesses
- Generalizability is limited. ARIA learns a separate Q-table for each benchmark from only 20–52 distinct development tasks, many of which are revisited; the SWE-bench and Terminal-Bench development sets are also screened based on model outcomes. Evaluation uses held-out tasks from the same benchmark families, but the paper does not show that a learned policy transfers to new task families, model pairs, or workflows. The evidence therefore supports benchmark-specific consultation policies more strongly than a general ability to learn when to consult.
- The coarse state representation limits the methodological contribution. ARIA averages Monte Carlo returns for each state–action pair and chooses the action with the higher empirical mean. Different tasks with the same phase, evidence category, and consultation history receive the same decision, regardless of their specific information gaps. Without a comparison to a phase-only policy learned from the same data, it is unclear whether the proposed state representation adds value beyond a simple learned consultation schedule.
Questions for authors
- How does ARIA compare with an executor that decides for itself when to consult, under the same contact points, advisor interface, and budget?
- Does adding a small amount of task-specific information improve consultation decisions?
- Why was the policy learned from so few distinct tasks when larger task pools were available for SWE-bench Verified and τ²-bench Telecom?
Author Response to Reviewer SEpo
We thank the reviewer for the detailed feedback. We address the concerns about state representation, executor-driven consultation, and generalization, supported by additional evaluations across all benchmarks.
W2: Methodological contribution and state representation
ARIA's contribution is a lightweight framework for learning when to consult stronger black-box models during execution, without modifying either model. Its compact 24-state representation aggregates recurring workflow situations across tasks, allowing experience to be shared when estimating consultation values. This follows established principles of state abstraction in reinforcement learning (Abel et al., 2016), balancing task-specific detail against policy complexity.
To directly address the reviewer's proposed control, we evaluated a learned four-state Phase-only policy fitted from the same training trajectories, alongside a Delivery-only strategy retaining ARIA's review-and-revision mechanism. Both use the same model pair, advisor interface and consultation budget.
ARIA achieves the highest observed success and excess recovery (ER) across all three benchmarks. Neither Phase-only nor Delivery-only reproduces its performance, providing direct empirical support for state-conditioned consultation beyond simpler strategies. We acknowledge that individual state features are not isolated by these comparisons and that resource-terminated SWE-bench evaluations, particularly for Phase-only, limit attribution of the full observed differences solely to the policies.
Q1: Executor-driven consultation
We would like to clarify that this comparison is mostly addressed by our existing LLM-Router baseline (Section 5.1, Appendix B.2). Here, the executor (to be precise the orchestrator agent) independently decides when to consult the advisor, without ARIA's learned policy. On SWE-bench, LLM-Router achieves 29.3% success versus ARIA's 48.0%, while consulting substantially less frequently (0.41 vs. 2.78 consultations per episode).
This provides a relevant comparison with a native executor-driven consultation. However, LLM-Router does not implement ARIA's dedicated pre-delivery review-and-revision mechanism. This distinction is intentional: LLM-Router evaluates the executor's native, self-directed consultation behavior without imposing ARIA's pre-delivery revision mechanism. It therefore addresses the reviewer's request for an executor-driven baseline, while preserving the executor's natural consultation behavior.
Q2: Task-specific information
ARIA deliberately uses observable workflow signals rather than task-specific representations, allowing its compact policy to aggregate experience across heterogeneous tasks. The executor and advisor already receive task context, but this information is not directly encoded in the consultation controller. We agree that incorporating additional task-specific features could further improve consultation decisions and consider this a valuable extension, although its incremental benefit has not yet been isolated experimentally.
W1/Q3: Development tasks and generalization
ARIA learns a compact 24-state Q-table rather than task-specific solutions or model parameters. Repeated task executions follow standard Monte Carlo reinforcement learning practice, where stochastic trajectories provide additional state-action observations for estimating expected returns (Sutton & Barto, 2018). Under a limited training budget, we prioritized outcome-screened development tasks where differences in executor and advisor performance suggested that consultation could meaningfully affect success. This follows the broader motivation of informative task selection in curriculum reinforcement learning (Narvekar et al., 2020), concentrating training on potentially decision-relevant cases rather than simply selecting successful tasks.
We acknowledge that outcome-based screening may introduce selection bias and that repeated trajectories cannot replace broader task diversity. Importantly, training and evaluation tasks remain disjoint, providing evidence of generalization to unseen tasks across three benchmark families.
Reference
Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., & Stone, P. (2020). Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey. Journal of Machine Learning Research, 21(181), 1-50. https://jmlr.org/papers/v21/20-212.html