- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this paper, the authors present ARC, a reinforcement learning-based framework to derive a light-weight agent configuration policy. This policy can dynamically determine the agent configurations, including the structural and prompt designs, based on different input queries. The proposed framework aims to balance the tradeoff between cost and accuracy, minimizing costs on trivial questions while spending more budgets on complicated queries.
The core novelty of this paper lies in the formulating each agent configuration as an option within a semi-Markov Decision Process. To my best knowledge, the adaptation of Hierarchical Reinforcement Learning to agent configuration is also novel, opening new research directions for learning how to configure agent structures and prompts.
Critical strengths
(1) The problem this paper aims to address is very practical and important. Configuring agent systems is challenging due to the large search space. The proposed ARC system learns a policy to configure agents based on queries, and demonstrates improvements over existing one-size-fits-all solutions. The system could be very useful in multiple domains, including physics, medicine and chemistry, where heterogeneous tasks require customized agent workflows, e.g., drug design and disease diagnosis.
(2) The formulation of agent configuration as an option within a semi-Markov Decision Process is novel. This formulation takes the cost of each action into consideration, leading to the capability of balancing the cost-accuracy trade-off for various queries. The authors apply HRL to learn a policy model, the adaptation is sound and proves to be working well in downstream analysis.
(3) I appreciate that the authors did comprehensive analysis in addition to the main comparison table, including adaptive allocation, workflow diversity, policy transfer, error analysis, etc. All these analysis contributes to a deeper understanding of how, why and where the proposed method can work well. The policy transfer experiments are promising, indicating broader generalizability between multiple tasks.
Critical weaknesses
(1) One limitation is that the methods are only tested on Qwen and Gemini models. It would be more intriguing if the authors could test more frontier models because nowadays many domain-specific agent systems are instantiated using top LLMs such as GPT-6 or Opus 5.5.
(2) Current evaluations on GSM8k, HotpotQA, MedQA, DROP and GAIA are most text-only benchmarks. More evaluations on multimodal benchmarks are needed to demonstrate the method’s generalizability beyond text.
(3) The authors claim to address the shortcoming of standard MDP methods. However, in Figure 3, there seems to be a lack of proper comparison with MDP. Adding a straight comparison could be much more helpful to justify the value of the proposed method.
Questions for authors
(1) Does the learned policy ARC generalize across models? For example, can the same policy model be applied to agents instantiated by GPT-6 and Opus 5.5, and still show significant improvements?
(2) How does the performance of training ARC with a stronger LLM compare to training ARC with a cheap model but running inference with a stronger LLM? This question would be critical for practical use, e.g., under the same budget, does the authors recommend investigating more LLM power in the HRL stage or agent inference stage?
(3) How does the proposed method compare to current prevailing RSI methods? Are they complementary to each other? The authors don’t need to run additional methods, more discussions could be much helpful.
3 comments
W3 & Q3: Comparison with Standard MDP Methods and Complementarity with RSI Methods
We thank the reviewer for this suggestion in comparison with MDP methods. However, we would like to highlight that we have already covered this comparison in the paper. In Table 1, we included two flat RL baselines that do not use our hierarchical SMDP option decomposition, namely, RL Bandits (a single-step flat contextual bandit) and RL Episodes (a standard flat episode-level MDP policy). Across all benchmarks and both backbones in Table 1, ARC consistently outperforms both baselines. Additionally, we had also included them in Figure 3, where RL Episodes lie on the Pareto frontier at a lower-accuracy as compared to ARC (pareto optima). We will make these points more prominent in Figure 3 and Section 4.3 so the comparison is easier to spot.
Additionally, we thank the reviewer for highlighting Recursive Self-Improvement (RSI) methods. We see RSI methods as highly complementary to ARC. As shown in our earlier experiment moving from Qwen 2.5 7B to Qwen 3 30B (in W1 & Q1), as the underlying model evolves, the previous configuration is no longer the best configuration to follow. In an RSI setting, as the system improves itself, it learns new capabilities while previous bottlenecks or reliance on external tools can disappear, and new tools, workflows, and prompts can be introduced as the system grows. Because ARC is lightweight and fast to retrain, it can evolve alongside an RSI loop to continuously adapt the configuration policy as the agent system's capabilities and action space expand. We thank the reviewer for pointing this out and will include this discussion in the final version.
Conclusion
We hope these additional results and clarifications address all of the review's questions, and we would be grateful if you would consider raising your score.
Thanks for the author's detailed responses and substantial efforts put into improving the manuscript. My concerns have been cleared. I will keep my rating as it is.
We thank the reviewer for the positive assessment of our work and for recognizing the practical importance of adaptive agent configuration, our hierarchical formulation, and our empirical analysis across transfer, diversity, and error taxonomy. Below, we address each of your concerns and questions.
W1 & Q1: Generalization to Stronger Models and Cross-Model Transfer
Yes, ARC is a general method that can be applied to any underlying model, including newer frontier models like GPT-6 or Opus 5.5. Our core design decouples the configuration policy from the execution LLM, which allows us to swap the underlying model and apply our algorithm to any backbone without modifying model weights. This is evident from our main experiments with Qwen 2.5 (open-weight) and Gemini 2.5 (closed-source), as well as our cross-scale transfer results in Figure 4 (Section 4.4), where the same policy learned on Qwen 2.5 7B is applied directly to 32B and 72B variants and continues to show consistent improvements across all benchmarks.
To make this statement more concrete, we also applied ARC to a newer and stronger backbone, Qwen 3 30B (A3B Instruct), across multiple tasks and baselines:
Furthermore, as the underlying model improves, ARC also adapts by learning relevant configurations accordingly. As shown in the table below, as backbone capability improves, ARC shifts from heavy hierarchical coordination (Orchestrator-Workers for GSM8k) to lighter parallel execution (Parallel-Sectioning for GSM8k) and reduces external tool calls by 28%, showing the adaptive nature of ARC.
Q2: Training ARC with a Stronger LLM vs. Training with a Cheap Model and Running Inference on a Stronger LLM
We thank the reviewer for asking this very relevant and practical question. Under the same budget, we recommend training the ARC controller using a smaller, cheaper model and allocating the budget toward running inference with the stronger LLM. This is mainly because during training, the policy needs to explore a large number of steps (between 4,000 and 12,000 episodes in our experiments, depending on the dataset). With a cheaper model, it is much easier and more economical to run this exploration (costed us under $10 total across five benchmarks with Qwen 2.5 7B), and once the configurations are learned, we can use the same policy across larger model sizes at inference time, as seen in Figure 4 of the paper (7B to 32B and 72B). We do, however, recommend this direct transfer primarily between models from the same model family. As shown in our results in W1 & Q1 above, two different model families or generations (such as Qwen 2.5 and Qwen 3) have different native capabilities and can therefore converge to different relevant configurations. We will include this practical discussion in the final version.
W2: Generalizability to Multimodal Benchmarks
We agree with the reviewer that our current evaluation is concentrated on text-based benchmarks. However, we would like to highlight that ARC naturally extends to multimodal settings. Our query encoder (MetaCLIP-H/14, Appendix D) is natively a multimodal vision-language backbone capable of encoding joint image-text inputs, and the policy network selects over abstract tool and workflow slots. That said, we have not yet evaluated ARC on multimodal benchmarks, and we will make this limitation and future direction explicit in the final version of the paper.