acceptodds
Under review as a conference paper at ICLR 2027

SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery

Abstract

Long-running coding agents such as autoresearch can persistently discover optimizations for open-ended problems. However, they tend to converge onto a single high-level approach, then proceed with low-level edits while missing superior approaches to the problem. We hypothesize two harness-level design choices contribute to this behavior: accumulating context in a single long-running agent and only exposing a single program state to edit. We introduce SwarmResearch, a type of orchestrator-subagent harness in which a Shepherd Agent uses global context to steer a population of Search Agents without prescribing exact tasks, while each Search Agent operates with local context in its own git branch. On open-ended optimization tasks, SwarmResearch discovers better or comparable solutions to state-of-the-art LLM-guided evolution and multi-agent techniques on 13/15 tasks, driven by higher-level exploration. Compared with fixed scaling of serial and parallel agents, SwarmResearch's orchestrator-guided scaling discovers better solutions by adapting parallelism at different search depths.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, multi-agent systems, LLM-guided search, open-ended optimization, scientific discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XesnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SwarmResearch, an orchestrator-subagent framework designed to prevent long-running coding agents from converging on suboptimal local solutions. SwarmResearch utilizes a Shepherd Agent to steer Search Agents operating in isolated Git branches with controlled local contexts. The method is evaluated on 15 open-ended optimization tasks, demonstrating competitive performance against existing evolutionary and multi-agent baselines.

Critical strengths

The paper shows context accumulation and single-state editing as core issues in long-running coding agents. The proposed dual-tier context management and Git-branch isolation provide a novel paradigm for multi-agent collaboration.

Critical weaknesses
  1. The evaluation lacks statistical experiments, as all methods are run only once per task.

  2. The complex orchestrator and Git-harness design introduces significant overhead but yields inconsistent benefits. This suggests the architecture is an over-engineered, task-specific trick rather than a universally necessary paradigm for open-ended discovery.

Questions for authors
  1. Given the randomness of LLM sampling, can you provide results from at least three independent runs to prove these small performance gains are statistically significant?

  2. Can you provide an ablation study to show the specific impact of the Git-branch isolation and the Explorer/Optimizer design?

  3. You mentioned your reproduced baselines performed worse than their original papers. How can we ensure a fair comparison if the baselines were not run with their optimal settings or the same model?

  4. Maintaining isolated Git worktrees and an orchestrator adds overhead. Could you provide a simple cost-benefit analysis showing the performance gain per dollar or token spent?

  5. Appendix A shows the Shepherd Agent's steering is highly prompt-dependent. How robust is this design when using less capable models or different prompt strategies?

  6. The method shows clear advantages mainly in heuristic tasks, but small returns in math and systems tasks. What specific problem types make this high-level exploration truly necessary?

2 comments

Thank you for your feedback. Our new Appendix A, “Extended Analysis,” adds ablations, repeated runs, and analyses addressing reviewer concerns (>$1K in experiment credits).

1) Repeated runs and statistical reliability: We ran SwarmResearch and CORAL independently three times each on two tasks. We omit EvoX because it performed substantially worse in our original experiments. SwarmResearch achieves a higher mean and best-of-3 score on both tasks. These runs provide further evidence against gains arising from stochastic variation. We use Claude Sonnet 5.5 to reduce costs and test a different model. Three runs across all tasks with Opus 4.6 would cost over $3,500, beyond our budget. Opus 4.6 High now hits its thinking-token limit about five times as often than the time of experiments, complicating exact replication.

Section 3.2 distinguishes meaningful from small differences through manual analysis. SwarmResearch exceeds CORAL on 10/15 tasks and matches it on two. Differences on Circle Packing, Erdos Min Overlap, and MMD-14-3 are small and could shrink with further polishing at higher budgets; larger differences occur on tasks that reward high-level exploration. Beyond final scores, Appendix A.7 adds search trajectories for each task and method, showing where other methods plateau at worse scores.

Finally, small numerical gains on mathematical optimization problems should not automatically be interpreted as stochastic variation. Remaining gains can be small yet difficult to discover, requiring algorithms that escape basins where previous techniques are stuck. All prior work also reports small numerical gains e.g. AlphaEvolve improved the third-autocorrelation upper bound from 1.45810 to 1.4557.

2) Baseline configurations and comparison fairness: We use CORAL’s public implementation, its original model (Opus 4.6), and its four-agent configuration. For EvoX, we use its public implementation and recommended 100 iterations, but Opus 4.6 rather than GPT-5 due to cost constraints. CORAL’s paper also reproduces EvoX with Opus 4.6 and reports rankings consistent with ours. SwarmResearch and CORAL receive the same $50 budget, at which both show diminishing returns (Appendix A.7).

3) Component ablations: Our primary claim is that the integrated framework improves high-level exploration and matches or outperforms our baselines. Appendix A adds ablations and analysis of the requested components. Since statistically significant ablations are costly, we supplement them with analysis where it provides reasonable evidence.

Git-based branching yields global bests through non-greedy exploration on 5/15 tasks (A.4). Explorers and Optimizers are complementary: from the same parent, Explorers outperform Optimizers in 46% of cases, and Optimizers outperform Explorers in 32% (A.3). Ablations of context resetting between Explorers and history isolation suggest these may have a small effect on final outcomes relative to fresh-context Explorers starting from the baseline and adaptive orchestration (A.2). These ablations clarify where context resetting matters, while existing results support the core design: both CORAL and the fixed-scaling comparison (three independent runs) show that a fixed group of persistent agents performs worse than adaptive orchestration.

4) Orchestration overhead and cost-effectiveness: Search Agents dominate inference cost: orchestration increases total output tokens by 7.7% over Search Agents alone (Section 3.4). Git worktrees don’t inherently add overhead: descendants reuse their parent’s branch. Separate workspaces prevent conflicts in concurrent experiments, making worktrees difficult to ablate. Table 1 shows SwarmResearch’s gains over CORAL under equal budgets, including orchestration overhead.

5) Prompt design and model robustness: Our framework uses only skills for ease of use, so detailed prompts are needed to implement its design. Prompts are general and do not fit to any task, emphasizing simple principles such as avoiding convergence on one approach. This design is simpler than EvoX’s solution-selection heuristics or CORAL’s heartbeat events and shared memory. Small open-source models, such as Minimax-M2.5, can perform poorly as Shepherd Agents in our experiments. The design relies on frontier models’ ability to orchestrate subagents and follow our harness-as-a-skill instructions. Appendix A.6 analyzes behavior with minimal prompting and motivates our prompt design.

6) Tasks that benefit from high-level exploration: On tasks such as Circle Packing, remaining gains may be small and best achieved by tuning hyperparameters or optimizers. Other math tasks like signal processing, systems tasks like transaction scheduling, and heuristics tasks offer more room for algorithmic changes and therefore reward high-level exploration. Also, small numerical gains need not be insignificant; see point 1 (paragraph 2) for details.

I appreciate the authors' efforts in addressing the comments. The issues have been resolved. I will maintain my score.

Reviewer_e8gSOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SwarmResearch, an agent orchestration framework for open-ended program optimization. A Shepherd Agent allocates search resources using global experiment summaries, while Explorer and Optimizer Agents operate on separate Git branches with fresh context or inherited parent conversations, respectively. The framework aims to mitigate search convergence caused by accumulated context and shared program state. It is evaluated on 15 optimization tasks, an adaptive search-scaling experiment, and an approximate speculative decoding case study.

Critical strengths
  1. Preventing long-running coding agents from prematurely committing to one approach is an important problem for autonomous research. The discussion of context anchoring and greedy program updates is clear, and the proposed solution is practically actionable.
  2. The combination of global coordination, local context, and independent branches is relatively simple. The appendix provides concrete skill definitions, making the operational rules substantially more transparent than an architecture-only description.
Critical weaknesses
  1. The paper attributes its gains to context isolation, branch preservation, global orchestration, and the Explorer/Optimizer distinction, but the supplied experiments do not systematically separate these factors. It therefore remains unclear which components are necessary, or whether the gains primarily arise from stronger orchestration models and resource allocation.
  2. Each method is run only once per task in the main benchmark. The authors acknowledge that this does not provide stable performance estimates, but cost constraints do not substitute for uncertainty analysis. On tasks with small score differences, the results cannot reliably distinguish genuine improvements from stochastic variation.
  3. The paper primarily measures high-level exploration through the number of edited lines, reporting larger changes than CORAL and EvoX. However, edit size also depends on program length, refactoring, and implementation style; it does not directly establish exploration of more distinct algorithmic families.
  4. Where does the innovation of this work lie? It seems to mainly consist of some existing engineering techniques. Could the author clarify the innovation of the paper?
Questions for authors

See Critical Weaknesses

2 comments

Thank you for your feedback. Our new Appendix A, “Extended Analysis,” adds ablations, repeated runs, and analyses addressing reviewer concerns (>$1K in experiment credits).

1) Component contributions and ablations: The paper compares SwarmResearch with CORAL and EvoX using the same model, with a matched $50 budget for CORAL and EvoX’s recommended 100 iterations. Appendix A.7 shows diminishing returns at these budgets. Therefore, SwarmResearch’s gains do not arise from stronger orchestration models nor resource allocation.

Our primary claim is that the integrated framework improves high-level exploration and achieves comparable or better performance than our baselines. Appendix A adds ablations and analysis of the requested components. Since statistically significant ablations are costly, we supplement them with analysis where it provides reasonable evidence.

Git-based branching yields global bests through non-greedy exploration on 5/15 tasks (A.4). Explorers and Optimizers are complementary: from the same parent, Explorers outperform Optimizers in 46% of cases, and Optimizers outperform Explorers in 32% (A.3). Ablations of context resetting between Explorers and history isolation suggest these may have a small effect on final outcomes relative to fresh-context Explorers starting from the baseline and adaptive orchestration (A.2). These ablations clarify where context resetting matters, while existing results support the core design: both CORAL and the fixed-scaling comparison (three independent runs) show that a fixed group of persistent agents performs worse than adaptive orchestration.

2) Repeated runs and statistical reliability: We completed three independent runs on two tasks for both SwarmResearch and CORAL. We omit EvoX because it performed substantially worse in our original experiments. SwarmResearch achieves a higher mean and best-of-3 score on both tasks. These runs provide additional evidence against the improvements being due to stochastic variation. We use Claude Sonnet 5.5 to reduce costs and test a different model. Three runs across all tasks with Opus 4.6 would cost over $3,500, beyond our budget. Since the original experiments, Opus 4.6 High has also hit its thinking-token limit approximately five times as often, making exact replication challenging.

Section 3.2 distinguishes meaningful improvements from small differences through manual analysis of solutions. SwarmResearch exceeds CORAL on 10/15 tasks and matches it on two. Differences on Circle Packing, Erdos Min Overlap, and MMD-14-3 are small and could shrink with further polishing at higher budgets; larger differences occur on tasks that reward high-level exploration. Beyond final scores, Appendix A.7 adds search trajectories for each task and method, showing where other methods plateau at worse scores.

Finally, small numerical gains on mathematical optimization problems should not automatically be interpreted as stochastic variation. Remaining gains can be small yet difficult to discover, requiring algorithms that escape basins where previous techniques are stuck. All prior work also reports small numerical gains e.g. AlphaEvolve improved the third-autocorrelation upper bound from 1.45810 to 1.4557.

3) Measuring high-level exploration: See Appendix A.5 for details. We perform LLM-based classification of changes for each method as high-level or not. High-level changes are 68.6% of SwarmResearch’s sampled edits, compared with 33.9% for CORAL and 45.6% for EvoX. High-level changes edit a median of 228 lines, versus 60 for low-level changes. Using patch size alone to classify changes as high-level achieves an AUC of 0.79, which increases to 0.85 when patch size is normalized by the lines of code in the change's parent and resulting solutions. Patch size is thus an imperfect but useful, deterministic, free, and well-defined proxy for high-level changes. Authors agree with the LLM 92% of the time on a subset of 25 changes and 3-way LLM agreement is 97%.

4) Novelty/innovation: Although the architecture is implemented through simple means (skills, coding agent sessions, and Git), the architecture is substantially different from existing approaches in open-ended discovery. Our paper demonstrates its improved performance and increased high-level exploration. Existing agent-based methods like CORAL use a fixed number of self-organized agents, each in long-running experimental loops with one editable program state. SwarmResearch does not run a fixed number of agents; instead, an orchestrator adaptively spawns any number of Search Agents to diversify and concentrate on areas of the problem’s search space (Section 3.4 shows the benefit of orchestrator-guided scaling). Additionally, unlike existing agents keeping one program state per agent, branch-based versioning preserves competing approaches to avoid greedy local search (see Appendix A.4 for evidence).

Reviewer_e8gSOpenReview ↗

Response

Thank you for the author's reply. I will improve my score, but I still have concerns about the novelty of this work.

Reviewer_i7akOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SwarmResearch, a multi-agent framework designed to overcome the limited exploration of long-running coding agents. It uses a Shepherd Agent to coordinate independent Search Agents, each operating with local context in its own Git branch to explore diverse solutions. By leveraging Git-based isolation and adaptive parallel exploration, SwarmResearch achieves more effective optimization than existing LLM-guided and multi-agent approaches.

Critical strengths
  1. Clear Motivation: The paper addresses a meaningful limitation of existing long-running coding agents. The motivation is well articulated, and the proposed direction is relevant to improving open-ended problem solving.

  2. Intuitive Framework Design: The proposed framework is conceptually clear and well structured. Its design provides a reasonable approach to balancing exploration and exploitation in agent-based research.

  3. Comprehensive Empirical Evaluation: The paper evaluates the proposed approach across diverse optimization tasks and compares it with relevant baselines. The experimental results provide useful evidence supporting the effectiveness of the framework. I like the budget-controlled comparison; it clearly shows SwarmResearch is on the Pareto frontier.

Critical weaknesses
  1. Some high-level ideas overlap with one of the baseline COARL (https://arxiv.org/pdf/2604.01658), e.g., multi-agent system orchestration for open-ended problems, Git-based context management.

  2. The poor performance of EvoX may be attributable to the limited computational budget. In the original EvoX paper, the method achieved a score of approximately 2.636 on the same circle packing problem, whereas it only achieves around 2.1 in this paper. This discrepancy raises concerns about whether the comparison is conducted under sufficiently competitive settings. I would like to see experiments with larger computational budgets for both SwarmResearch and EvoX (and ideally other baselines as well) to better assess their relative performance.

Questions for authors

See my weakness

  • What's the conceptual difference between Swarmresearch's Git branch context management and the similar context management idea in COARL (https://arxiv.org/pdf/2604.01658)?
1 comment

Thank you for your feedback. Our new Appendix A, “Extended Analysis,” adds ablations, repeated runs, and analyses addressing reviewer concerns (>$1K in experiment credits).

1) SwarmResearch vs. CORAL: Although CORAL and SwarmResearch are both multi-agent systems for open-ended tasks, their architectures are very different. CORAL uses a fixed number of self-organized agents with shared memory, each in long-running experimental loops with one editable program state, but SwarmResearch does not run a fixed number of agents. Instead, an orchestrator adaptively spawns any number of Search Agents on top of any parent solution to diversify and concentrate on areas of the problem’s search space (Section 3.4 shows the advantage of orchestrator-guided agent scaling).

Although CORAL uses Git branches and worktrees to isolate persistent agents, each agent iterates on its own workspace containing its most recent solution. In contrast, SwarmResearch treats branching and parent selection as explicit primitives for the orchestrator scheduling new Search Agents. Each Search Agent operates on a separate Git branch, allowing the Shepherd Agent to launch any number of Search Agents from any completed solution, including non-best solutions. Appendix A.4 provides evidence of this non-greedy search behavior: 5/15 tasks have global bests that arise from non-best-parent lineages. Finally, CORAL involves a more complex harness involving heartbeat events and requires manually configuring the number of agents. SwarmResearch is fully implemented as skills and the orchestrator adapts the number of Search Agents.

2) EvoX reproduction and budget fairness: We configured EvoX with the same computational budget as in their paper (100 iterations) and use their provided open-source repository. We note the discrepancy between our reproduction and their paper’s results in “3.1 Experimental Setup, Baselines and configurations”. We attribute this to the difference in models. EvoX’s paper uses GPT-5 while we use Opus 4.6. We only report Opus 4.6 due to cost limitations. CORAL’s paper also reproduces EvoX with Opus 4.6 and reports rankings consistent with ours. SwarmResearch and CORAL receive the same $50 budget, which we selected since both methods show diminishing returns by this point (Appendix A.7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.