acceptodds
Under review as a conference paper at ICLR 2027

TRAINING RECURSIVE AGENT SWARMS AT DEPTH

Abstract

Agent swarms offer a promising path toward solving complex tasks through coordinated delegation and collective learning. However, most previous studies focus on agent coordination or shared policy optimization. How delegation depth and upstream learning jointly constrain training opportunities for independently updated subagents remains underexplored. In this study, we model this dependence through update coverage. This is the probability that a root rollout group supplies a policy with a usable update. We obtain two findings. Passive sampling costs can grow exponentially with delegation depth. In a finite batch model, individually beneficial upstream updates can jointly reduce downstream coverage and lower two-stage team value even under their best order. We introduce LACA (Learning Aware Controlled Adaptation) to explicitly elicit the training signals needed by nested policies. LACA samples fresh continuations from saved parent states and selects ordered policy subsets using update coverage and estimated two-stage team value. It retains the original task reward and learner objective. We evaluate LACA through paired update programs, controlled entry, and downstream policy rollback. At a common budget of 2,048 charged rollouts, LACA improves task success over GRPO by 8.42% on average across WebWalkerQA, Terminal-Bench 2.1, and BrowseComp-Plus, and its mean success exceeds PPO, DAPO, and Dr.MAS.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic AI, delegation, coverage shift

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xm5qOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the training opportunity problem in recursive agent swarm system. It identifies the problem by introducing the "update coverage" to analyze the sampling probability introduced by the depth. LACA tackles this problem by saving training trajectories to estimate a 2-stage team value to select the subteam and the order of agents to update. LACA is benchmark on 3 benchmarks, showing better performance compared with GRPO and DAPO baselines.

Critical strengths
  • This paper studies swarm training, which is an important problem, where the theoretical and empirical results have large gaps.
  • The experiment is detailed, which is an important evidence to support its theoretical claim.
  • The high level motivation of this work is to increase the training opportunity of subsequent agents, which is reasonable to me.
Critical weaknesses
  • I'm still a bit confused about the motivation of the design. See questions.
Questions for authors
  • What is the motivation of altering subteam and update order?Why would it improve the performance?
  • Is there any theoretical guarantee that the strategy of randomly selecting teams/orders, as described in the paper, can improve performance?
Reviewer_BZdEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the training difficulties of recursive agent swarms, where upstream policy updates can reduce the training opportunities available to downstream agents. The authors introduce update coverage to characterize this dependency and show that individually beneficial updates may become harmful when combined. Building on these findings, they propose LACA, which restores saved parent states to provide downstream training data and selects ordered policy subsets based on estimated two-stage team value. A paired audit determines whether the updated team should be accepted. Experiments on WebWalkerQA, Terminal-Bench 2.1, and BrowseComp-Plus demonstrate improvements over GRPO, DAPO, and Dr.MAS.

Critical strengths
  1. An interesting perspective on recursive agent training. The paper identifies a meaningful problem that is largely overlooked by standard credit assignment methods: upstream learning can deprive downstream agents of usable training data. The update coverage formulation and analysis of harmful update combinations provide a useful theoretical perspective.

  2. Well-motivated framework with extensive analysis. LACA offers a coherent way to combine controlled training access, policy update selection, and team-level auditing. The experiments are encouraging, particularly the controlled coverage comparisons, update-combination studies, and worker rollback analysis.

Critical weaknesses
  1. The two-stage objective is not fully aligned with the main training protocol. Although program selection considers a subsequent downstream learning stage, the main benchmark runs execute only one decision window and do not commit the lookahead continuation. This makes it unclear how directly the estimated two-stage value reflects the performance of the resulting team.

  2. The large contribution of the audit gate is insufficiently explained. In Appendix E.4, two-stage selection without the gate improves average success by only 1.98 points, whereas enabling the gate adds 27.48 points. The factorial results establish an interaction, but the mechanism behind such a large difference remains unclear.

  3. Controlled entry may introduce a mismatch in training states. Restoring saved parent states improves downstream coverage by bypassing ancestor routing. However, these states may become less likely under updated upstream policies, potentially limiting the relevance of the additional worker training to natural root-started execution.

Questions for authors
  1. Since the lookahead continuation is not committed in the main one-window experiments, what makes two-stage team value preferable to the immediate team value for selecting the deployed update program?

  2. Appendix E.4 reports a 27.48-point improvement from adding the audit gate after two-stage selection, despite only three rejected programs in twenty decisions. Could the authors explain why these relatively few rejections have such a large effect on final performance?

  3. How does LACA handle saved parent states that become unlikely to be reached after upstream policies change? Is the relevance of these states to the current root-started policy measured or otherwise accounted for?

Reviewer_4xvcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LACA, which provides training opportunities for deeply nested subagents and selects which agents to update and in what order. It addresses insufficient downstream training caused by upstream learning, improving task performance in recursive multi-agent teams.

Critical strengths
  1. The paper proposes a method for training heterogeneous multi-agent systems, including jointly training multiple distinct LLMs. Although some prior work has explored this setting, it remains relatively uncommon in multi-agent training.
  2. The paper evaluates the method on benchmarks spanning QA, web search, and coding-agent tasks, demonstrating its effectiveness across multiple domains.
Critical weaknesses
  1. The need for deep recursive delegation has not been established. The paper takes this structure as a given without demonstrating that the target tasks require it. Since serial execution introduces additional handoff and context overhead, comparisons with a single agent and single-level delegation under the same resource budget are needed. Performance and actual costs at different depths should also be reported to establish the practical value of addressing training problems in deeply nested teams.
  2. The training method uses crude credit assignment. The main agent and subagents perform independent tasks, so updating every agent along the recursive path using the same group-relative advantage is inappropriate.
  3. Insufficient control of computational budgets prevents a convincing assessment of efficiency gains. The paper constrains the budget using equal rollout counts, but methods may differ in the number of agent calls, recursion depth, and context length per rollout, resulting in different token consumption and computational costs. A common token limit per execution also does not ensure equal total consumption. Without reporting actual token usage or comparing methods under a fixed total token budget, the results only show that LACA performs better with the same number of attempts; they do not rule out gains from additional computation. Consequently, the paper lacks fair comparisons against “training only the main agent” and “jointly training all agents.”
Questions for authors

Overall, I am skeptical of the value of recursive multi-agent delegation at the motivational level, and I have concerns about credit assignment in the RL training procedure at the methodological level. Although the paper is well written, I currently lean toward rejection. However, I welcome a constructive discussion with the authors and will revise my score if they can satisfactorily address these concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

A General Response by the Authors

Dear Area Chairs and Reviewers,

Thank you for handling our submission. To support your evaluation, we briefly summarize below: (1) an overview of our work, (2) the strengths highlighted by the reviewers, and (3) the responses we provided to address their concerns.

(1) Overview of our work. Agent swarms increasingly rely on nested delegation, yet standard credit assignment only reweights team reward across roles and fails to ensure that nested policies receive usable training data. We study how delegation depth and the learning of higher-level agents limit the training opportunities of independently updated subagents, introduce update coverage to make this failure measurable, and propose LACA to address it. Our key contributions are:

  • Update coverage. Define the probability that a group of team rollouts gives an agent a usable training update, and prove that the rollouts needed to train an agent can grow exponentially with its delegation depth.
  • LACA. Train nested agents directly by resuming rollouts from saved parent states, choose which agents to update and in what order by estimated team performance after the next round of learning, and keep an update only if the team improves.
  • Results. Under the same 2,048-rollout cap, improve mean task success on three benchmarks by +6.14 pp over GRPO, +5.57 pp over DAPO, and +4.1 pp over Dr.MAS, and show that co-training nested agents with LACA helps the swarm (+33.96 pp over training the root alone).

(2) Strengths. We are glad that all three reviewers recognized strengths of our work. In particular:

  • Overlooked problem. The paper "identifies a meaningful problem that is largely overlooked by standard credit assignment methods" (BZdE), "an important problem" (xm5q).
  • Theory and framework. The coverage analysis offers "a useful theoretical perspective", and LACA is a "well-motivated framework with extensive analysis" (BZdE).
  • Cross-domain experiments. Detailed experiments support the theory (xm5q, BZdE) across QA, web search, and coding-agent tasks with heterogeneous LLM teams (4xvc).

(3) Responses to address the reviewers' concerns. We provided detailed clarifications and added new experiments along several axes:

  • Value of recursive delegation. Reviewer 4xvc's main reservation is the "value of recursive multi-agent delegation" and "credit assignment in the RL training procedure". Under identical models and inference limits, the LACA-trained depth-3 team solves a multi-stage recovery task with 59.9% fewer tokens than a single agent, while the single-agent and single-level runs both fail. At the same budget, root-only training reaches 24.33% and full LACA 58.29% (4xvc W1).
  • Credit assignment. LACA keeps the same team-return advantage as the matched GRPO baseline, updates one agent at a time with partners fixed, and beats role-rescaled credit (Dr.MAS) by +4.10 to +4.12 pp. With the point above, this resolves both parts of the main reservation (4xvc W2).
  • Computational budget. Since equal rollout counts "do not rule out gains from additional computation", we report measured per-depth inference tokens and training tokens (156M to 158M for LACA vs. 136M to 138M for joint baselines, at most 1,792 executions under the 2,048 cap) (4xvc W3).
  • Two-stage objective. Reviewer BZdE noted that "the two-stage objective is not fully aligned with the main training protocol". At an equal 46M selection-token budget, the lookahead changes which program is committed and adds +2.70 pp in one window (BZdE W1, Q1; 4xvc W3).
  • Audit gate. For "the large contribution of the audit gate is insufficiently explained", the gate effect is an interaction with two-stage selection across four sequential training windows (BZdE W2, Q2).
  • Saved states. For "controlled entry may introduce a mismatch in training states", saved-state entry recovers +10.07 pp on natural root starts. The reviewer replied: "I have a better understanding of the issue now. I'll keep my positive rating." (BZdE W3, Q3).
  • Motivation of the design. Reviewer xm5q was "still a bit confused about the motivation of the design". Upstream updates change routing and shrink downstream training data: updating all three owners loses 8.72 to 9.64 pp against child-only training even in the best order, while LACA's selected program gains +30.32 pp over a fixed order. The reviewer replied: "The response addresses my main concern." (xm5q W1, Q1).

In addition, we have incorporated the above responses in Sections 1, 2, 5, and 6 and Appendices C and E, and highlighted the revisions in blue.

We believe that the detailed point-by-point responses and the updated submission can address the reviewers' concerns. The revised submission presents a clearer motivation for recursive delegation and for LACA, with measured token costs and controlled comparisons across depths and components.

Thank you again for your time and consideration.

Sincerely, Authors of #38820

Dear Reviewers,

The author-reviewer discussion period will end soon. Please review and acknowledge the authors’ responses.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.