- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces SwarmResearch, an orchestrator-subagent framework designed to prevent long-running coding agents from converging on suboptimal local solutions. SwarmResearch utilizes a Shepherd Agent to steer Search Agents operating in isolated Git branches with controlled local contexts. The method is evaluated on 15 open-ended optimization tasks, demonstrating competitive performance against existing evolutionary and multi-agent baselines.
Critical strengths
The paper shows context accumulation and single-state editing as core issues in long-running coding agents. The proposed dual-tier context management and Git-branch isolation provide a novel paradigm for multi-agent collaboration.
Critical weaknesses
-
The evaluation lacks statistical experiments, as all methods are run only once per task.
-
The complex orchestrator and Git-harness design introduces significant overhead but yields inconsistent benefits. This suggests the architecture is an over-engineered, task-specific trick rather than a universally necessary paradigm for open-ended discovery.
Questions for authors
-
Given the randomness of LLM sampling, can you provide results from at least three independent runs to prove these small performance gains are statistically significant?
-
Can you provide an ablation study to show the specific impact of the Git-branch isolation and the Explorer/Optimizer design?
-
You mentioned your reproduced baselines performed worse than their original papers. How can we ensure a fair comparison if the baselines were not run with their optimal settings or the same model?
-
Maintaining isolated Git worktrees and an orchestrator adds overhead. Could you provide a simple cost-benefit analysis showing the performance gain per dollar or token spent?
-
Appendix A shows the Shepherd Agent's steering is highly prompt-dependent. How robust is this design when using less capable models or different prompt strategies?
-
The method shows clear advantages mainly in heuristic tasks, but small returns in math and systems tasks. What specific problem types make this high-level exploration truly necessary?
2 comments
I appreciate the authors' efforts in addressing the comments. The issues have been resolved. I will maintain my score.
Thank you for your feedback. Our new Appendix A, “Extended Analysis,” adds ablations, repeated runs, and analyses addressing reviewer concerns (>$1K in experiment credits).
1) Repeated runs and statistical reliability: We ran SwarmResearch and CORAL independently three times each on two tasks. We omit EvoX because it performed substantially worse in our original experiments. SwarmResearch achieves a higher mean and best-of-3 score on both tasks. These runs provide further evidence against gains arising from stochastic variation. We use Claude Sonnet 5.5 to reduce costs and test a different model. Three runs across all tasks with Opus 4.6 would cost over $3,500, beyond our budget. Opus 4.6 High now hits its thinking-token limit about five times as often than the time of experiments, complicating exact replication.
Section 3.2 distinguishes meaningful from small differences through manual analysis. SwarmResearch exceeds CORAL on 10/15 tasks and matches it on two. Differences on Circle Packing, Erdos Min Overlap, and MMD-14-3 are small and could shrink with further polishing at higher budgets; larger differences occur on tasks that reward high-level exploration. Beyond final scores, Appendix A.7 adds search trajectories for each task and method, showing where other methods plateau at worse scores.
Finally, small numerical gains on mathematical optimization problems should not automatically be interpreted as stochastic variation. Remaining gains can be small yet difficult to discover, requiring algorithms that escape basins where previous techniques are stuck. All prior work also reports small numerical gains e.g. AlphaEvolve improved the third-autocorrelation upper bound from 1.45810 to 1.4557.
2) Baseline configurations and comparison fairness: We use CORAL’s public implementation, its original model (Opus 4.6), and its four-agent configuration. For EvoX, we use its public implementation and recommended 100 iterations, but Opus 4.6 rather than GPT-5 due to cost constraints. CORAL’s paper also reproduces EvoX with Opus 4.6 and reports rankings consistent with ours. SwarmResearch and CORAL receive the same $50 budget, at which both show diminishing returns (Appendix A.7).
3) Component ablations: Our primary claim is that the integrated framework improves high-level exploration and matches or outperforms our baselines. Appendix A adds ablations and analysis of the requested components. Since statistically significant ablations are costly, we supplement them with analysis where it provides reasonable evidence.
Git-based branching yields global bests through non-greedy exploration on 5/15 tasks (A.4). Explorers and Optimizers are complementary: from the same parent, Explorers outperform Optimizers in 46% of cases, and Optimizers outperform Explorers in 32% (A.3). Ablations of context resetting between Explorers and history isolation suggest these may have a small effect on final outcomes relative to fresh-context Explorers starting from the baseline and adaptive orchestration (A.2). These ablations clarify where context resetting matters, while existing results support the core design: both CORAL and the fixed-scaling comparison (three independent runs) show that a fixed group of persistent agents performs worse than adaptive orchestration.
4) Orchestration overhead and cost-effectiveness: Search Agents dominate inference cost: orchestration increases total output tokens by 7.7% over Search Agents alone (Section 3.4). Git worktrees don’t inherently add overhead: descendants reuse their parent’s branch. Separate workspaces prevent conflicts in concurrent experiments, making worktrees difficult to ablate. Table 1 shows SwarmResearch’s gains over CORAL under equal budgets, including orchestration overhead.
5) Prompt design and model robustness: Our framework uses only skills for ease of use, so detailed prompts are needed to implement its design. Prompts are general and do not fit to any task, emphasizing simple principles such as avoiding convergence on one approach. This design is simpler than EvoX’s solution-selection heuristics or CORAL’s heartbeat events and shared memory. Small open-source models, such as Minimax-M2.5, can perform poorly as Shepherd Agents in our experiments. The design relies on frontier models’ ability to orchestrate subagents and follow our harness-as-a-skill instructions. Appendix A.6 analyzes behavior with minimal prompting and motivates our prompt design.
6) Tasks that benefit from high-level exploration: On tasks such as Circle Packing, remaining gains may be small and best achieved by tuning hyperparameters or optimizers. Other math tasks like signal processing, systems tasks like transaction scheduling, and heuristics tasks offer more room for algorithmic changes and therefore reward high-level exploration. Also, small numerical gains need not be insignificant; see point 1 (paragraph 2) for details.