acceptodds
Under review as a conference paper at ICLR 2027

Learning to Configure Agentic AI Systems

Abstract

Configuring LLM-based agent systems involves choosing workflows, tools, token budgets, and prompts from a large combinatorial design space, and is typically handled today by fixed templates or hand-tuned heuristics that apply the same configuration regardless of query difficulty, leading to brittle behavior and wasted compute. To address this, we formulate agent configuration as a semi-Markov decision process (SMDP) where each configuration acts as a temporally extended option that determines how an agent system processes a query, and introduce ARC (Agentic Resource & Configuration learner), a lightweight hierarchical policy that dynamically selects query-specific agent configurations. Across reasoning, tool-use, and agentic benchmarks, ARC consistently improves over budget-matched tool-augmented LLMs, increasing average reasoning accuracy by 31.3%, tool-use accuracy by 13.95%, and doubling τ-Bench (Airline) Pass^1 success from 9.0% to 18.0%. These results demonstrate that learning per-query agent configurations is a powerful alternative to "one size fits all" designs.

Primary area:
reinforcement learning
Keywords:
Agentic Workflows, Reinforcement Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qrdPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors present ARC, a reinforcement learning-based framework to derive a light-weight agent configuration policy. This policy can dynamically determine the agent configurations, including the structural and prompt designs, based on different input queries. The proposed framework aims to balance the tradeoff between cost and accuracy, minimizing costs on trivial questions while spending more budgets on complicated queries.

The core novelty of this paper lies in the formulating each agent configuration as an option within a semi-Markov Decision Process. To my best knowledge, the adaptation of Hierarchical Reinforcement Learning to agent configuration is also novel, opening new research directions for learning how to configure agent structures and prompts.

Critical strengths

(1) The problem this paper aims to address is very practical and important. Configuring agent systems is challenging due to the large search space. The proposed ARC system learns a policy to configure agents based on queries, and demonstrates improvements over existing one-size-fits-all solutions. The system could be very useful in multiple domains, including physics, medicine and chemistry, where heterogeneous tasks require customized agent workflows, e.g., drug design and disease diagnosis.

(2) The formulation of agent configuration as an option within a semi-Markov Decision Process is novel. This formulation takes the cost of each action into consideration, leading to the capability of balancing the cost-accuracy trade-off for various queries. The authors apply HRL to learn a policy model, the adaptation is sound and proves to be working well in downstream analysis.

(3) I appreciate that the authors did comprehensive analysis in addition to the main comparison table, including adaptive allocation, workflow diversity, policy transfer, error analysis, etc. All these analysis contributes to a deeper understanding of how, why and where the proposed method can work well. The policy transfer experiments are promising, indicating broader generalizability between multiple tasks.

Critical weaknesses

(1) One limitation is that the methods are only tested on Qwen and Gemini models. It would be more intriguing if the authors could test more frontier models because nowadays many domain-specific agent systems are instantiated using top LLMs such as GPT-6 or Opus 5.5.

(2) Current evaluations on GSM8k, HotpotQA, MedQA, DROP and GAIA are most text-only benchmarks. More evaluations on multimodal benchmarks are needed to demonstrate the method’s generalizability beyond text.

(3) The authors claim to address the shortcoming of standard MDP methods. However, in Figure 3, there seems to be a lack of proper comparison with MDP. Adding a straight comparison could be much more helpful to justify the value of the proposed method.

Questions for authors

(1) Does the learned policy ARC generalize across models? For example, can the same policy model be applied to agents instantiated by GPT-6 and Opus 5.5, and still show significant improvements?

(2) How does the performance of training ARC with a stronger LLM compare to training ARC with a cheap model but running inference with a stronger LLM? This question would be critical for practical use, e.g., under the same budget, does the authors recommend investigating more LLM power in the HRL stage or agent inference stage?

(3) How does the proposed method compare to current prevailing RSI methods? Are they complementary to each other? The authors don’t need to run additional methods, more discussions could be much helpful.

3 comments

We thank the reviewer for the positive assessment of our work and for recognizing the practical importance of adaptive agent configuration, our hierarchical formulation, and our empirical analysis across transfer, diversity, and error taxonomy. Below, we address each of your concerns and questions.

W1 & Q1: Generalization to Stronger Models and Cross-Model Transfer

Yes, ARC is a general method that can be applied to any underlying model, including newer frontier models like GPT-6 or Opus 5.5. Our core design decouples the configuration policy from the execution LLM, which allows us to swap the underlying model and apply our algorithm to any backbone without modifying model weights. This is evident from our main experiments with Qwen 2.5 (open-weight) and Gemini 2.5 (closed-source), as well as our cross-scale transfer results in Figure 4 (Section 4.4), where the same policy learned on Qwen 2.5 7B is applied directly to 32B and 72B variants and continues to show consistent improvements across all benchmarks.

To make this statement more concrete, we also applied ARC to a newer and stronger backbone, Qwen 3 30B (A3B Instruct), across multiple tasks and baselines:

MethodGSM8KMedQAGAIAMean
Base Model w/ tools93.2 ± 0.130.5 ± 0.26.7 ± 1.743.5
DSPy94.3 ± 0.183.9 ± 0.21.3 ± 0.559.8
DAAO95.2 ± 0.285.4 ± 0.47.7 ± 2.162.8
AFlow (Sonnet optimizer)95.1 ± 0.283.9 ± 0.39.0 ± 0.062.7
ARC w/o SFT (Ours)95.4 ± 0.182.9 ± 0.18.0 ± 2.562.1
ARC (Ours)95.7 ± 0.384.3 ± 0.29.3 ± 0.563.1

Furthermore, as the underlying model improves, ARC also adapts by learning relevant configurations accordingly. As shown in the table below, as backbone capability improves, ARC shifts from heavy hierarchical coordination (Orchestrator-Workers for GSM8k) to lighter parallel execution (Parallel-Sectioning for GSM8k) and reduces external tool calls by 28%, showing the adaptive nature of ARC.

SpecsQwen 2.5 7B (88.6%)Qwen 3 30B (95.4%)Δ
Dominant workflowOrchestrator-WorkersParallel-SectioningLighter coordination
Avg agents / episode4.013.99~No change
Tool calls / episode5.313.82~28% decrease

Q2: Training ARC with a Stronger LLM vs. Training with a Cheap Model and Running Inference on a Stronger LLM

We thank the reviewer for asking this very relevant and practical question. Under the same budget, we recommend training the ARC controller using a smaller, cheaper model and allocating the budget toward running inference with the stronger LLM. This is mainly because during training, the policy needs to explore a large number of steps (between 4,000 and 12,000 episodes in our experiments, depending on the dataset). With a cheaper model, it is much easier and more economical to run this exploration (costed us under $10 total across five benchmarks with Qwen 2.5 7B), and once the configurations are learned, we can use the same policy across larger model sizes at inference time, as seen in Figure 4 of the paper (7B to 32B and 72B). We do, however, recommend this direct transfer primarily between models from the same model family. As shown in our results in W1 & Q1 above, two different model families or generations (such as Qwen 2.5 and Qwen 3) have different native capabilities and can therefore converge to different relevant configurations. We will include this practical discussion in the final version.

W2: Generalizability to Multimodal Benchmarks

We agree with the reviewer that our current evaluation is concentrated on text-based benchmarks. However, we would like to highlight that ARC naturally extends to multimodal settings. Our query encoder (MetaCLIP-H/14, Appendix D) is natively a multimodal vision-language backbone capable of encoding joint image-text inputs, and the policy network selects over abstract tool and workflow slots. That said, we have not yet evaluated ARC on multimodal benchmarks, and we will make this limitation and future direction explicit in the final version of the paper.

W3 & Q3: Comparison with Standard MDP Methods and Complementarity with RSI Methods

We thank the reviewer for this suggestion in comparison with MDP methods. However, we would like to highlight that we have already covered this comparison in the paper. In Table 1, we included two flat RL baselines that do not use our hierarchical SMDP option decomposition, namely, RL Bandits (a single-step flat contextual bandit) and RL Episodes (a standard flat episode-level MDP policy). Across all benchmarks and both backbones in Table 1, ARC consistently outperforms both baselines. Additionally, we had also included them in Figure 3, where RL Episodes lie on the Pareto frontier at a lower-accuracy as compared to ARC (pareto optima). We will make these points more prominent in Figure 3 and Section 4.3 so the comparison is easier to spot.

Additionally, we thank the reviewer for highlighting Recursive Self-Improvement (RSI) methods. We see RSI methods as highly complementary to ARC. As shown in our earlier experiment moving from Qwen 2.5 7B to Qwen 3 30B (in W1 & Q1), as the underlying model evolves, the previous configuration is no longer the best configuration to follow. In an RSI setting, as the system improves itself, it learns new capabilities while previous bottlenecks or reliance on external tools can disappear, and new tools, workflows, and prompts can be introduced as the system grows. Because ARC is lightweight and fast to retrain, it can evolve alongside an RSI loop to continuously adapt the configuration policy as the agent system's capabilities and action space expand. We thank the reviewer for pointing this out and will include this discussion in the final version.

Conclusion

We hope these additional results and clarifications address all of the review's questions, and we would be grateful if you would consider raising your score.

Thanks for the author's detailed responses and substantial efforts put into improving the manuscript. My concerns have been cleared. I will keep my rating as it is.

Reviewer_mzswOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ARC, a query-conditioned controller for configuring an LLM-based agent system while keeping the backbone LLM frozen. A structure policy selects a workflow, tools, and token budgets, and a conditional prompt policy composes instructions from a fragment library. The controller is trained with masked PPO using correctness and resource costs, followed by supervised refinement on elite trajectories. The paper motivates this design through an SMDP formulation and evaluates it on six benchmarks covering reasoning, tool use, and interactive agents, using Qwen2.5-7B-Instruct and Gemini 2.5 Flash Lite, with additional transfer experiments. Reported results include 88.6% GSM8K accuracy and an increase in airline τ-Bench Pass^1 from 9.0% to 18.0% for Qwen. The main claimed contribution is improved accuracy and resource allocation through per-query configuration rather than a fixed system.

Critical strengths
  1. The configuration problem is useful, and the proposed controller exposes several consequential decisions jointly. Section 3 combines workflow choice, role-specific tools, budgets, and prompt composition without updating the backbone LLM. This is a practical direction for improving existing agent systems, and the explicit workflow library makes the intended deployment model understandable.

  2. The evaluation covers multiple capabilities and includes relevant initial controls. Tables 1–2 include search, prompt/workflow optimization, flat PPO, and a version without SFT across two backbone families. For example, ARC reports 88.6% on GSM8K versus 85.2% for RL Episodes, and 63.9% on DROP versus 54.3%. These results suggest potential beyond the direct baseline, even though the controls need better specification. The paper also acknowledges an important limitation: GEPA substantially outperforms ARC on MedQA.

Critical weaknesses
  1. The contribution beyond prior query-adaptive systems and the theoretical guarantees are insufficiently established. MaAS (Multi-agent Architecture Search via Agentic Supernet, ICML 2025) and MAS-GPT (MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems, ICML 2025) address closely related problems of query-dependent agent-system design, but neither is discussed or evaluated. ARC's joint configuration and hierarchical policy may offer differences, but their added value needs clearer justification. The SMDP rationale is also unclear when selecting one configuration per query. Theorem 3.1 establishes Q-learning convergence rather than convergence of the implemented PPO algorithm, while Theorem 3.2 does not establish a deployment reward floor for unseen queries or fresh stochastic executions.

  2. The accuracy–cost advantage of query-specific configuration is not convincingly isolated. Maximum-budget matching does not separate adaptation from ARC's iterative execution and prompt optimization. Figure 11 appears to show approximately 50% HotpotQA accuracy for fixed Parallel Voting, above ARC's 34.1% in Table 1; whether the evaluation protocols differ needs clarification. Reproducibility is further limited by the missing tool-shaping reward definition, unspecified numerical budget tiers, and absent uncertainty estimates for the main results. Reporting inconsistencies also require correction: Greedy Search is 36.0% in Section 4.5 versus 78.2% in Table 1, and ARC's MedQA result is 58.4% in Section 4.2 versus 64.6% in Table 1.

Questions for authors
  1. What successor state and duration-aware return are used in PPO? How do the theoretical guarantees apply to the implemented PPO and SFT policies, particularly on unseen queries and stochastic executions?

  2. What distinguishes ARC from MaAS and MAS-GPT, and do its accuracy–cost gains hold against comparable adaptive methods and the best fixed configuration?

  3. Please define RtoolR_{\mathrm{tool}} and the numerical budget tiers, reconcile the Greedy Search and MedQA discrepancies, and provide evaluation sample counts and uncertainty across independent training runs.

3 comments

We thank the reviewer for recognizing the practical value of ARC’s ability to jointly select a workflow, tools, token budgets, and prompts for each query while keeping the backbone LLM frozen. We address each concern below, clarifying how ARC differs from prior work, what the comparisons and ablations show, which theoretical claims apply to training, and how to reproduce the results.

Q1 & W1. What return does PPO use, and what do the theoretical results guarantee?

For the single-query benchmarks in Table 1, the structural policy makes one decision per query. The prompt policy then composes the instructions, the backbone executes the workflow, and the episode ends. The successor state is terminal, so its value is zero. The structural PPO return is therefore the observed episode reward R, and the advantage before batch normalization is R-V(s_q). A workflow can take several LLM calls, and its duration affects R through the step and token-budget penalties in Eq. 3. The prompt policy separately discounts the terminal reward across its fragment-selection steps. The reviewer is right that the structural update in this setting can also be described as a contextual bandit with variable-cost actions. Theorem 3.1 establishes convergence for tabular SMDP Q-learning under its stated assumptions. We will revise Section 3.4 to make clear that convergence of the neural PPO implementation remains an empirical question. Theorem 3.2 concerns an idealized exact fit to the empirical distribution of elite configurations. The threshold applies to the reward observed in a retained training episode. Even on the same query, executing that configuration again can yield a different reward. We will remove the claim of a deployment reward floor and report the effect of SFT empirically. For example, SFT improved held-out MedQA accuracy from 58.4% to 64.6% in Table 1. The independent GSM8K runs reported under Q3 show how that effect varies across training runs.

W1, W2 & Q2. How does ARC differ from MaAS and MAS-GPT, and where does adaptive configuration help?

We will add MaAS and MAS-GPT to the related-work discussion. MaAS samples a query-specific system from a learned agentic supernet. MAS-GPT trains a language model to generate executable multi-agent code. ARC learns how to configure a supplied library of nine workflows. Two queries can select the same workflow, yet receive different tools, token budgets, and instructions for each role. A lightweight controller learns these decisions jointly from execution feedback while the backbone stays frozen. We will make this distinction explicit in Section 2.

For the empirical part of the question, we included DAAO because it adapts orchestration to each query, and AFlow because it optimizes a workflow for each benchmark and applies the selected program to its test questions. All methods used Qwen2.5-7B as the execution backbone. Each benchmark entry reports mean accuracy (%) ± standard deviation across three runs. The final column averages the three benchmark means.

MethodDROPHotpotQAGAIAMean
DAAO62.1 ± 0.431.8 ± 0.23.7 ± 1.532.5
AFlow (Sonnet optimizer)61.1 ± 0.226.7 ± 0.14.3 ± 0.630.7
AFlow (7B optimizer)56.2 ± 0.430.3 ± 0.51.0 ± 0.029.2
ARC (Ours)65.9 ± 0.035.7 ± 0.16.0 ± 0.035.9

ARC achieved the highest mean accuracy on all three benchmarks. The clearest gains over DAAO, the strongest baseline on average, were 3.8 percentage points on DROP and 3.9 points on HotpotQA. We agree with the reviewer that a common execution backbone alone does not establish equal inference cost. These results support the accuracy comparison. Whether the gains persist at equal realized cost remains open.

Fixed configurations and Figure 11. In the original HotpotQA evaluation in Table 1, ARC achieved 34.1%, compared with 28.6% for Greedy Search and 27.8% for flat PPO (RL Episodes), which also had access to multi-step workflows. We will describe grid and greedy search as the search procedures we evaluated. An exhaustive optimum over all fixed configurations was not measured. The reviewer correctly reads the Parallel-Voting bar in Figure 11 as approximately 50%. The denominator explains the apparent discrepancy. Figures 10–11 indicate that Parallel Voting answered 1,198 of the 2,371 training episodes routed to it correctly (50.5%). Table 1’s 34.1% is the final policy’s accuracy across held-out HotpotQA questions, where ARC chooses a workflow for each question. To compare directly with the 34.1%, Parallel Voting would have to be applied to that same held-out set. We will correct the caption to identify the training split, routing condition, and number of episodes behind each bar.

W2, Q3. What are the reward, budget tiers, corrected scores, and evaluation uncertainties?

Tool reward and budgets. Let u be the number of tool invocations, m the number of tool assignments summed across roles, and y{0,1} the final-answer correctness indicator. Set r=u/m when m>0, and r=0 otherwise. The trainer uses

\begin{cases} (b+0.2y)u + 0.3r\,\mathbf{1}[r \geq 0.5], & u > 0, \\ -\bigl(0.5+0.3(1-y)\bigr)m, & u = 0. \end{cases}$$ Here b is the per-call bonus, with a default of 0.1. Each invocation earns this bonus and earns an additional 0.2 when the final answer is correct. Equation 3 uses weights of 5 for correctness and 1 for tool shaping, and subtracts step and normalized token penalties with coefficients 0.02 and 0.03. The low, medium, and high output-token tiers are 512 / 1,024 / 2,048 for the reasoner, 256 / 512 / 1,024 for the verifier, and 128 / 256 / 512 for the answerer. Reporting corrections. Table 1 gives the correct Qwen2.5-7B values. Greedy Search achieved 78.2% on GSM8K. We will correct the 36.0% in Section 4.5. Section 4.2 mixed two MedQA rows: 58.4% is ARC without SFT, and 64.6% is full ARC. The 15.5-point gain over the 49.1% base model belongs to full ARC. Evaluation counts and uncertainty. GSM8K used 1,319 test questions and MedQA used 1,273. GAIA used 65 examples for training and 100 for held-out evaluation. The separate full-split baseline evaluations used 9,535 DROP and 7,405 HotpotQA questions. τ-Bench Airline used 20 distinct evaluation tasks with repeated rollouts. We will give the sample count and split alongside each result. The original ARC entries in Table 1 report one training run per configuration. In three additional independent GSM8K retrainings, accuracy before SFT was 87.6%, 89.3%, and 89.1%. After SFT, the corresponding values were 87.3%, 88.3%, and 82.3%. The across-run sample standard deviations were 0.95 and 3.25 percentage points, respectively. Thus, the improvement from SFT in the submitted run varied across retrainings. For baseline evaluations that reused a trained DAAO controller or a selected AFlow workflow, the standard deviation measures variability across executions. We will label the unit of repetition alongside each set of error bars. We hope these clarifications and additional results address all of the reviewer's concerns, and we respectfully ask the reviewer to reconsider their initial evaluation and score in light of this rebuttal.

Thank you for clarifying the HotpotQA results and the scope of the theorems. The new comparisons and ablations show accuracy gains. My main question remains whether ARC can outperform a tuned fixed configuration from the same search space on the same test set at comparable measured inference cost. The new results leave this question open, so my final rating is 2 (weak rejection).

Reviewer_X5mkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper formulates agent configuration as a query-wise decision problem: for each query, a learned controller emits a full configuration c = (ω, t, b, p) — workflow topology ω (one of nine patterns, from single-call Direct up to Evaluator-Optimizer), the set of enabled tools t (calculator, web search, code execution), a token-budget tier b, and a composed prompt p — while the backbone LLM itself is kept frozen. The controller is factored into a structure policy and a prompt policy, trained with PPO under workflow-conditioned action masking, and then refined by a supervised stage that distils "elite" (correct and high-reward) trajectories drawn from the RL buffer. The decision process is presented as a semi-Markov decision process over options so that workflows of different lengths (1–7 LLM calls) can be compared under a single reward that trades correctness against call count and token use. Empirically the authors train one policy per benchmark on six datasets (GSM8k, DROP, MedQA, HotpotQA, GAIA, τ-Bench Airline) with Qwen 2.5 7B and Gemini 2.5 Flash Lite, compare against base models with tools, grid/greedy search, AutoGen, DSPy, GEPA, AFlow, LAP and two flat-RL variants, and add analyses of cross-dataset and cross-scale transfer, workflow diversity, an error taxonomy, and an accuracy–cost Pareto plot.

Critical strengths

S1. A clean, reusable formulation that couples all four configuration axes and keeps the backbone frozen. The paper is the first I have seen in this line to put workflow, tool set, budget tier and prompt composition into a single per-query action, with an explicit masking rule that removes structurally invalid combinations (Direct has one agent, so tool and budget slots for a second agent must be pruned). The nine workflows are specified concretely enough to re-implement (Appendix A gives call counts and per-role tool access for each), the training loop is given as pseudocode (Algorithm 1), and because only the lightweight heads are updated, the recipe is cheap to apply on top of any hosted model. This is consequential because it is the paper's main conceptual contribution and the reason the setting is more general than workflow search or prompt optimisation alone; it is delivered coherently and is independent of whether the empirical claims below hold.

S2. The evaluation is unusually broad in coverage and honest about a large loss. Six benchmarks across three capability axes, two model families (one open-weight, one proprietary), plus transfer across datasets (Table 4) and across Qwen scales (Figure 4), a per-workflow accuracy breakdown (Figure 11) and an accuracy–cost Pareto plot (Figure 3) is materially more than comparable orchestration papers attempt. The paper also does not hide the case where it loses decisively — "On MedQA, GEPA performs best (87.1%), while ARC reaches 58.4%" (Section 4.2) — and offers a specific, checkable diagnosis (GEPA uses a ~1,100-token domain system prompt vs ~50-token general fragments). This is consequential because a paper arguing for architectural adaptation needs exactly this kind of per-task negative result to make its boundary conditions visible, and the diagnosis points at a concrete follow-up (structure search over domain-specific prompts).

Critical weaknesses

W1. The paper's headline claim — higher accuracy and lower cost — is contradicted by its own Figure 3 and by its own reward function. (a) The abstract states the learned policy achieves "up to 25% higher task accuracy while also reducing token and runtime costs", and Section 4.3 states that ARC "occupies Pareto-optimal regions that strictly improve upon prior methods, achieving high accuracy at lower cost". Figure 3 (GSM8K) does not support the cost half: the DSPy and GEPA points sit at roughly 10⁻¹ $/episode while both ARC points sit at roughly 10⁰ $/episode — on the order of 20× more expensive for a few points of accuracy (≈88 vs ≈83). ARC is Pareto-optimal in the sense that nothing is simultaneously cheaper and more accurate, which is a different statement from "lower cost", and the text conflates the two. There is also no runtime table anywhere in the manuscript, so the "runtime" half of the abstract claim has no evidence at all. (b) The reward cannot generate cost sensitivity either. Eq. 3 is R(s,o) = α·I[correct] − βs·ko − βt·ntokens/Tmax + η·Rtool, with "Task success weight: α = 5.0", "Step penalty: βs = 0.02", "Token penalty: βt = 0.03" (Appendix F). Since ntokens/Tmax ∈ [0,1] and ko ≤ 7, the entire efficiency term is bounded by 0.03 + 0.14 = 0.17, i.e. at most ≈3.4% of the 5.0 correctness term — as specified, the reward is effectively accuracy-only. If the real cost pressure comes from the per-option discount γ^ko (0.9⁷ ≈ 0.48, and γ itself is reported as 0.95 in Appendix F), then the mechanism is never identified as the cost weight and is not what the reward-shaping section describes. (c) "up to 25% higher task accuracy" does not match any number reported: the largest average gain in Table 1 is "+31.3" points (Section 4.2) and the largest per-benchmark relative gain is far higher (37.8% → 88.6% on GSM8K), while the gains over the strongest baseline are +5.0 (GSM8K vs GEPA), +4.9 (DROP vs RL Episodes), +2.0 (GAIA vs GEPA), 0.0 (HotpotQA, exactly tied with AutoGen) and −22.5 (MedQA). This is consequential because efficiency is one of the two claims the paper is organised around (RQ2 is devoted to it), and as written a reader who looks at Figure 3 alongside the abstract receives two incompatible messages; the paper is defensible if re-stated as "more expensive but better on the Pareto frontier", but it is not defensible as written.

W2. The reported numbers do not reconcile with each other, and two baselines look broken; the formal apparatus in Section 3 does not describe the algorithm that is trained. (a) Numbers disagree across text, tables, figures and appendix — each of these is a direct contradiction rather than rounding: Section 4.5 says "Greedy search performs even worse (36.0%)" while Table 1 gives 78.2% for Greedy Search (Qwen, GSM8K), and the same sentence benchmarks grid search against "ARC (88.5%)", the Gemini number, not the Qwen 88.6% in that column; Section 4.2 says "ARC reaches 58.4%, still improving over the base model by 15.5 points" although 58.4 − 49.1 = 9.3 and 15.5 is 64.6 − 49.1, i.e. the full-ARC row; the Table 4 caption claims "performance degradation of 6 − 7%" while its own rows are 63.9 → 63.0 (1.4% relative) and 64.7 → 57.0 (11.9% relative); Appendix H reports "GRPO achieved 81.2% … compared to PPO's 85.7%" and "DPO achieved 79.8%" against Table 5's 82.3 / 87.6 / 85.7, with 85.7 appearing under two different labels; Section 3.1 gives "γ = 0.9" while Appendix F gives "γ = 0.95". Separately, Table 3 reports identical Greedy Search statistics (UW = 2, entropy 0.24, Gini 0.46) for all three datasets, Figure 4's 7B points disagree with Table 1's 7B column on three of four benchmarks, and Table 6 reports "0.6853 ± 0.7577" for a metric bounded in [0,1]. None of this is accompanied by a seed count or any variance, so there is no way to tell a typo from a re-run. (b) Baselines look mis-configured, and the protocol is undisclosed. The paper never states whether AutoGen, DSPy, GEPA, AFlow and LAP were re-implemented or quoted, nor with which prompts, budgets or metrics. The base Gemini model scores "10.0%" on MedQA (Table 1), which is below the 25% chance level of a four-option multiple-choice exam — so either answer extraction or budget enforcement is failing, and this row is the denominator of the entire Δ column. The same harness is used for the τ-Bench headline: "ARC doubles base-model Pass^1 performance (9.0%→18.0%)" (Section 4.2) is measured against a baseline that is "restricted to non-iterative tool calls" (Section 4.1) on a benchmark whose entire content is multi-turn tool–user interaction, so doubling it is close to uninformative; DSPy at "3.0%" Pass^1 on the same benchmark (Table 2) looks like a non-functional re-implementation. Finally, "LLM as A Policy (LAP)" is used as a named baseline in Section 4.1 and in Table 1 but has no entry in the reference list. (I searched for a public paper with that name matching the described method and could not identify one; this is my failure to locate it, not evidence that it does not exist — hence a citation request rather than an assertion.) (c) The SMDP/options formalism and both theorems do not apply to what is trained. Section 3.1 states that after masking "all surviving options share Io = S", each episode is one query, and one option is executed before termination — so there is no inter-option state transition and no multi-step credit assignment; structurally this is a contextual bandit with a cost-penalised reward, which the paper itself lists as a baseline ("RL Bandits"). Theorem 3.1 is the textbook SMDP Q-learning result (Sutton et al., 1999) applied to Q-learning, whereas the method trains PPO with function approximation, so its convergence guarantee is about an algorithm that is never run; the inference that Theorem 3.1 means "the elite buffer contains near-optimal configurations" (Section 3.4) does not follow. Theorem 3.2 asserts πSFTarc(c|s) > 0 ⇒ c ∈ Celite(s), which cannot hold for a softmax policy over a finite action set (every action keeps positive probability without hard masking), and Celite(s) is defined only over training-buffer states, so the "reward floor" E[R] ≥ τ says nothing about test queries. This is consequential because a full page of Section 3 plus a section heading rests on this apparatus, and the paper's stated motivation for the SMDP ("Standard MDP formulations fall short here because they assume actions have a fixed, discrete duration", Section 1) is not needed by any component of the method as implemented.

(Deliberately not listed as critical, but relevant to the recommendation: (i) one policy is trained per dataset at "4,000 per dataset minimum" episodes, so the demonstrated adaptation is dataset-level — Figure 10 is described as the policy "sharpen[ing] and different datasets … converg[ing] to distinct dominant workflows" — while within-dataset adaptivity, the actual novelty, is never measured against a per-difficulty-tier fixed configuration or a per-query oracle; (ii) reproducibility gaps: no prompt fragment from P is given although it was generated with a closed model ("generation using GPT-5.2 outperforms other language models"), no policy-network architecture, no held-out split for selecting among PPO/GRPO/DPO and 19 embeddings, and no compute accounting for six policies plus those sweeps, in a paper whose thesis is cost; (iii) positioning: ADAS (Hu et al., 2024), query-adaptive routing (Adaptive-RAG; RouteLLM) and compute-optimal test-time scaling (Snell et al., 2024) are the obvious neighbours and are not cited; (iv) presentation leftovers: nine occurrences of "Agent2 Tools:" in Appendix A, a literal "textbf" in Appendix I, an undefined aρt` in Section 3.2, and a Figure 10 caption naming three panels for a five-panel figure.)

Questions for authors
  1. Cost, restated. Please give the per-episode cost in dollars, tokens and wall-clock for every method and benchmark, and reconcile the abstract's "reducing token and runtime costs" and Section 4.3's "at lower cost" with Figure 3, where ARC sits about 20× to the right of DSPy/GEPA. If the correct statement is "higher cost, better accuracy on the Pareto frontier", please say so explicitly in the abstract and conclusion. Which λ (Eq. 1) and which α, βs, βt, η (Eq. 3) were used for the reported runs, and is the observed cost sensitivity driven by those coefficients or by the γ^ko discount?

  2. Variance and denominators. How many independent training runs underlie Tables 1–3 and Figure 4, and can you report mean ± std? How many evaluation items remain after the GAIA 65/… split ("For GAIA, we partitioned the validation set, using the first 65 samples for training and the rest for evaluation", Section 4.1) and how many tasks in τ-Bench Airline, with raw correct counts rather than percentages? At the reported magnitudes (GAIA 2.0%–6.0%), which of the claimed gains survive a binomial confidence interval?

  3. Number reconciliation. Please reconcile, item by item: greedy search 36.0% (Section 4.5) vs 78.2% (Table 1); "improving over the base model by 15.5 points" at 58.4% (Section 4.2); the "6 − 7%" transfer degradation (Table 4 caption) vs 1.4%/11.9% in the table; Appendix H's 81.2 / 85.7 / 79.8 vs Table 5's 82.3 / 87.6 / 85.7; γ = 0.9 (Section 3.1) vs 0.95 (Appendix F); Figure 4's 7B points vs Table 1's 7B column; the identical Greedy Search statistics in Table 3; and the "0.6853 ± 0.7577" entry in Table 6.

  4. Baselines. Were AutoGen, DSPy, GEPA, AFlow and LAP re-implemented or quoted, and with which prompts, budgets and metrics? Specifically: how does Gemini reach 10.0% on MedQA, below the 25% chance level; how is answer extraction done for the "Base Model w/ Tools" row; what does "restricted to non-iterative tool calls" mean on τ-Bench, where the task requires multi-turn tool use; and why is DSPy's Pass^1 only 3.0%? Please also supply the missing LAP reference.

  5. Formalism. Given Io = S after masking, one option per query, and episode termination after that option, what state transition does the SMDP model that a contextual bandit with a cost-penalised reward does not? How does Theorem 3.1's Q-learning convergence result bear on a PPO-trained softmax policy, and how can Theorem 3.2's support restriction hold when a softmax keeps every action in its support?

  6. Adaptivity at the query level, and reproducibility. Because one policy is trained per dataset, what evidence shows within-dataset adaptation rather than dataset-level specialisation — e.g. the correlation between query difficulty and chosen configuration, a per-difficulty-tier fixed-configuration baseline, or a per-query oracle computed from the existing rollouts? And will the prompt fragment library P, the policy architectures, and the code be released, with a statement of whether any test-set metric was used to select among PPO/GRPO/DPO, the embedding model, and ARC vs ARC w/o SFT?

Criteria under which my score would rise: (i) restating the efficiency claim honestly as "more expensive, Pareto-better" together with a real cost/token/runtime table, or alternatively re-tuning the reward so that the cost term is non-negligible and re-running; (ii) either dropping the SMDP/HRL framing in favour of the bandit-with-cost-penalty formulation the code actually implements, or making the options real (multi-turn state, learned termination) and re-deriving the theorems for PPO; (iii) multi-seed results with confidence intervals and raw counts, plus a full re-derivation of every table from logs to remove the contradictions in W2(a); (iv) baseline re-runs at their own reported configurations plus the missing LAP reference; (v) an oracle/per-difficulty-tier analysis that isolates within-dataset adaptivity. Meeting (i)–(iii) would move me to 3 (weak acceptance); (iv)–(v) would make it a clear accept candidate. As written, the central claim is unsupported by the paper's own figure and the numbers cannot yet be relied upon.

6 comments

We thank the reviewer for recognizing the value of jointly configuring workflows, tools, budgets, and prompts with a frozen backbone, and for appreciating our experiment coverage. We apologize for the typos and phrasing inconsistencies that were unintentionally introduced in our submission. Below, we clarify our intended accuracy-cost framing, provide additional experimental evidence, correct specific reporting inconsistencies, and explain the scope of the theoretical results.

W1 & Q1. Clarification on accuracy-cost trade-off and reward design.

We understand the reviewer’s concern and agree with them that Pareto optimality does not mean lower cost than every baseline, and that our current wording in the abstract and Section 4.3 conflates the two. As correctly identified by the reviewer, our intended claim is that ARC achieves a better accuracy-cost trade-off on the Pareto frontier. As evident in our results shown in Figure 3, ARC reaches higher accuracy than lightweight single-pass methods (DSPy and GEPA) at higher inference cost, while achieving both higher accuracy and substantially lower token and runtime costs relative to heavy multi-agent orchestration (AutoGen). We thank the reviewer for sharing this with us and will update it in the final version to make this trade-off more explicit.

To make the token and runtime comparisons concrete, the table below reports the per-query measurements on GSM8K across all 1,319 evaluation queries corresponding to Figure 3. On GSM8K, ARC achieves 88.6% accuracy compared to AutoGen's 74.8%, while using approximately 60x fewer tokens and 15x less wall-clock runtime. Compared with DSPy and GEPA, ARC achieves +5.5 and +5.0 points higher accuracy at higher inference cost (5.8x and 4.6x more tokens, respectively). These results clearly show ARC lies on the pareto optimal across these baselines.

MethodAccuracy (%)Tokens / querySeconds / query
Base model37.82080.4
DSPy80.44100.8
GEPA83.65201.0
ARC (Ours)88.62,3843.2
AutoGen74.8141,87048.5

Clarification On Reward Design:

Regarding the reward coefficients (Eq. 3), our implementation uses (α,βs,βt,η)=(5.0,0.02,0.03,1.0)(\alpha, \beta_s, \beta_t, \eta) = (5.0, 0.02, 0.03, 1.0) directly, while λCcost\lambda C_{\text{cost}} in Eq. 1 serves only as high-level motivating notation rather than an additional tuned hyperparameter. In our design, task correctness (α=5.0\alpha = 5.0) is intentionally the primary objective, while the step and token penalties (βs,βt\beta_s, \beta_t) act as tie-breakers that favor cheaper configurations whenever multiple workflows yield comparable expected correctness and tool-shaping rewards.

To verify that these penalties meaningfully shape resource usage despite their smaller scale relative to α\alpha, we ran a reward ablation on GSM8K (trained for 2,000 episodes and evaluated on 100 held-out questions) comparing our full reward against βs=βt=0\beta_s = \beta_t = 0. Removing the efficiency penalties increases average token usage from 893 to 1,213 per query (+35.8%), increases average tool calls from 2.00 to 5.04, and slightly reduces accuracy from 88.0% to 85.0% due to unnecessary multi-step over-reasoning. Thus, the explicit penalty terms (βs,βt)(\beta_s, \beta_t) directly curb redundant steps and token consumption without sacrificing task accuracy, and this cost sensitivity comes from the reward terms themselves rather than structural γko\gamma^{k_o} discounting (which we clarify in W2c, Q5).

W2(a) & Q2. Independent runs, evaluation counts, and uncertainty

We apologize for the confusion caused by the number discrepancies in the text. To be completely transparent, the values reported in our main tables (such as Table 1) reflect our final completed evaluation runs and are the authoritative numbers. However, when updating the manuscript from our initial draft, a few numbers in the running text and Appendix were unintentionally carried over from earlier pilot runs (such as the 2,000-episode preliminary runs in Appendix H vs. the 4,000-episode final runs in Table 5, which we reconcile item-by-item in W2(a), Q3 below). None of these text carry-overs were intentional, and we apologize for this oversight and we will fix these typos in the final version.

The results in paper’s Table 1 report a single training run per configuration. Following the reviewer's suggestion to evaluate variance across independent runs, we conducted a 3-seed evaluation with Qwen 2.5 7B as the execution backbone against recent and strong optimization baselines (DAAO and AFlow with both Sonnet and 7B optimizers), shown in the table below.

MethodDROPHotpotQAGAIAMean
DAAO62.1 ± 0.431.8 ± 0.23.7 ± 1.532.5
AFlow (Sonnet optimizer)61.1 ± 0.226.7 ± 0.14.3 ± 0.630.7
AFlow (7B optimizer)56.2 ± 0.430.3 ± 0.51.0 ± 0.029.2
ARC (Ours)65.9 ± 0.035.7 ± 0.16.0 ± 0.035.9

Across all three benchmarks, ARC consistently outperforms the baselines with minimal variance across seeds, achieving the highest overall mean (35.9%). We will update the full table with multi-seed results in the final version.

New results with recent backbone:

Additionally, to verify that these gains hold on newer, more capable models as well, we evaluated ARC across 3 independent seeds using Qwen3-30B-A3B-Instruct as the execution backbone, results in table below. ARC achieves the highest overall average (63.1%) and leads on both GSM8K and GAIA while remaining highly competitive on MedQA, demonstrating that our framework generalizes effectively to recent, stronger backbone models.

MethodGSM8KMedQAGAIAMean
Base Model w/ tools93.2 ± 0.130.5 ± 0.26.7 ± 1.743.5
DSPy94.3 ± 0.183.9 ± 0.21.3 ± 0.559.8
DAAO95.2 ± 0.285.4 ± 0.47.7 ± 2.162.8
AFlow (Sonnet optimizer)95.1 ± 0.283.9 ± 0.39.0 ± 0.062.7
ARC w/o SFT (Ours)95.4 ± 0.182.9 ± 0.18.0 ± 2.562.1
ARC (Ours)95.7 ± 0.384.3 ± 0.29.3 ± 0.563.1

Clarification on evaluation sample count:

For the evaluation sample counts, GAIA uses 65 validation samples for training and the remaining 100 samples for evaluation, while τ\tau-Bench uses 20 test tasks across 5 trials (100 rollouts total, with 9/100 base-model successes and 18/100 ARC successes, consistent with the reported Pass1\text{Pass}^1, Pass2\text{Pass}^2, and Pass3\text{Pass}^3 values in Table 2). We will also state these exact values in the final version.

W2(a) & Q3. Item-by-item reconciliation of reported numbers

As mentioned above, the values in our main tables (Table 1 and Table 5) are our final results. We thank the reviewer for flagging these details so carefully. Some of these differences reflect distinct evaluation settings that we clarify below, while a few others were unintentional carry-overs from an earlier draft that we will fix in the final version. Details as follows:

  • MedQA 58.4% and +15.5 points (Section 4.2), and Greedy Search 36.0% (Section 4.5): In Section 4.2, both numbers are in Table 1 (58.4% is ARC w/o SFT and +15.5 points is full ARC at 64.6% over the 49.1% base model), and the text accidentally mixed the two rows. In Section 4.5, 78.2% in Table 1 is the correct Greedy Search score (36.0% was from an early draft), and 88.5% is the Gemini ARC score right below Qwen's 88.6%. We will fix both sentences to match Table 1.
  • Appendix H (81.2% / 85.7% / 79.8%) vs. Table 5 (82.3% / 87.6% / 85.7%): We want to clarify that both sets of numbers are valid, as they come from different training steps. As stated in Appendix H, those numbers are from a 2,000-episode run, whereas Table 5 reports the full 4,000-episode run. This was unintentional carry-overs from an earlier draft and we will correct this in the final draft.
  • Table 4 caption ("6-7% degradation") and γ=0.9\gamma = 0.9 vs. 0.950.95: In Table 4, "6-7%" was the average of the two relative drops for GSM8K transfer (1.4% on DROP and 11.9% on MedQA, which averages to 6.65%). However, we agree with the reviewer that the current caption can be confusing and we will update it to per-task change in the final version. For the discount factor, γ=0.9\gamma = 0.9 (Section 3.1) is the actual value used, while γ=0.95\gamma = 0.95 in Appendix F was a typo that we will fix.
  • Figure 4, Table 3 (Greedy Search), and Table 6 (0.7577): In Figure 4, we want to assure the reviewer that the 7B points do match the data in Table 1, though we believe the compressed y-axis scale makes the exact values hard to read visually. We will improve the plot readability and add data labels in the revision. In Table 3, although unusual, the Greedy Search values (UW = 2, Entropy = 0.24, Gini = 0.46) are genuine and not a copy-paste error. Because greedy search is deterministic, it consistently collapses onto the same two dominant workflows in the same proportion across all three datasets. Finally, in Table 6 (flava-full projected), ± 0.7577 is a decimal typo for ± 0.007577 (missing the leading zeros from the weighted std across the four metrics: 0.0294, 0.0086, 0.0047, 0.0041), which we will fix it in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.