- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Mixture of Self-Improving Branches to address limited search diversity in automated agent harness optimization. The method dynamically adjusts branch-specific development subsets based on comparative frontier coverage and updates proposal guidance using each branch's search history. A router then selects between the final branch heads for new tasks.
Experiments on mathematical reasoning, Terminal-Bench 2.0, and SWE-bench Lite show improvements over Meta-Harness, with the largest gain of 16 percentage points on Math–Gemini.
Critical strengths
Meaningful problem and concrete mechanism. The paper addresses the limitations of optimizing harnesses against a fixed development set. Dynamic task assignment provides a potentially useful mechanism for encouraging complementary search trajectories.
Evidence of branch specialization. Figure 2 and Appendix C illustrate distinct harness mechanisms and evolving proposal guidance across branches. Table 3(b) provides encouraging evidence that combining development-set updates with guidance adaptation improves performance.
Held-out evaluation and complementarity analysis. The paper evaluates development-selected harnesses on held-out tasks and demonstrates complementary capabilities between the resulting branch heads.
Critical weaknesses
1.Insufficient explanation of how dynamic task assignment induces specialization.
Dynamic development-set assignment is the paper's main methodological contribution. While the trajectory and guidance analyses demonstrate that branches evolve differently, the causal role of task assignment remains unclear.
In particular, the final development subsets overlap substantially in some settings. For Math–Sonnet, the two branches retain 92 and 103 tasks, sharing 81; for SWE-bench Lite, they retain 27 and 28 tasks, sharing 24. The paper does not analyze whether the branch-specific tasks differ meaningfully in category, difficulty, or failure modes.
Moreover, independent searches can also discover different harness mechanisms. Although Table 3(b) provides a useful fixed-development baseline, it does not compare the resulting branch complementarity. Stronger evidence is needed to establish the specific contribution of dynamic task assignment.
- Large performance differences across action models remain unexplained.
The improvement over Meta-Harness is substantial on Math–Gemini (+16.0 points) but much smaller on Math–Sonnet (+1.5 points), despite using the same mathematical reasoning setting.
It is unclear whether the large Gemini gain reflects the general effectiveness of the proposed search mechanism or model-specific characteristics, such as baseline capability and sensitivity to harness design.
Both action models are also relatively old compared with models available before submission. Evaluation on a newer model, together with an analysis of the different search outcomes, would strengthen the generalizability claim.
- Baseline search configuration and computational fairness need clarification.
The proposed method uses two branches with 20 iterations each, but the Meta-Harness baseline's search iterations and candidate budget are not sufficiently specified.
Table 3(b) includes a useful comparison against two independent fixed-development searches, but only for Math–Gemini. Appendix A also reports substantially higher total token consumption for the proposed method in several settings.
This makes it difficult to separate the benefits of dynamic specialization from those of additional search resources. Matched-budget comparisons and repeated full search runs would strengthen the conclusions.
Questions for authors
- Dynamic development subsets and specialization.
Could the authors analyze how the final branch-specific development subsets differ in task categories, difficulty, or failure modes? Given their substantial overlap in some settings, what evidence connects dynamic task assignment to the distinct harness mechanisms and complementary test performance? How does this complementarity compare with two independent fixed-development searches?
- Differences across action models.
What explains the large discrepancy between Math–Gemini (+16.0 points) and Math–Sonnet (+1.5 points)? Have the authors examined whether these differences arise from model capabilities, search trajectories, or harness-specific effects? Does the method retain its benefits with newer action models?
- Search budget and baseline configuration.
Could the authors clarify the Meta-Harness baseline's search iterations, number of generated candidates, and total compute budget? How does the proposed method compare with a longer single Meta-Harness search or two independent searches under matched candidate or token budgets?
- Robustness and implementation.
How are missing coverage evaluations and historical candidate rankings handled after development cases are removed from a branch?
2 comments
Thanks for clearing that up and running the extra tests. That adressed my major concerns. I'll increase my score up accordingly. Thanks for the effort!
We thank Reviewer ZQyF for the detailed questions about specialization, variation across action models, baseline comparisons, and implementation. We address each concern below.
Question 1: Development-subset differences and complementary mechanisms
Our analysis characterizes branch strengths by the mechanisms that solve different cases. Section 4.3 identifies concrete answer checks and proof-obligation routing in Branch 1, and free-form derivation, two-sided closure, and agreement challenges in Branch 2. The polynomial example shows how Branch 1’s counterexample check recovers a valid family on a case missed by Branch 2.
Development-subset overlap does not imply identical search objectives: branch-exclusive cases can change candidate rankings, selected heads, and subsequent proposals. Appendices B.1–B.2 document this process. Shared SWE-bench Lite cases decrease from 30 to 24 as exclusive cases increase from 1 to 7. At Math–Gemini iteration 13, Branch 2’s subset shrinks from 44 to 40 cases and a free-form derivation harness becomes the head, which later proposals extend through progressive refinement and two-sided closure. With proposal adaptation enabled, adding development pruning increases Math–Gemini accuracy from 50.0% to 62.0%.
Revision: Section 4.3 now explicitly connects the branch-exclusive cases to differing search objectives and points to the recorded examples in Appendices B.1–B.2.
Question 2: Model-dependent gains and newer-model transfer
We interpret the difference between the Gemini and Sonnet gains as reflecting the action models’ underlying capabilities: a harness can improve how a model approaches a task, while its benefit can remain constrained by the model’s task-solving capability.
We also evaluate transfer to Gemini 3.8 Flash using the best single harness generated with Gemini 3 Flash as the action model. The transferred harness achieves 75.0% accuracy, compared with 68.0% for BM25 retrieval, a gain of 7.0 percentage points. The transfer uses the existing evolved harness rather than rerunning evolution with the newer model.
Table 5: Test accuracy (%) across action models
Question 3: Search budget and repeated baseline configuration
Section 4.5 includes two independent Meta-Harness runs on Math–Gemini, achieving 46.0% and 51.0% test accuracy. We report the better individual test result, 51.0%, compared with 62.0% for our method. This is an 11.0-percentage-point improvement even with favorable test-based selection for Meta-Harness.
Revision: Section 4.2 reports the development/test scores of both runs. Section 4.5 explicitly distinguishes the better individual baseline result from routed deployment. Appendix A reports full-run token usage.
Question 4: Missing coverage evaluations and historical rankings after pruning
All candidates are scored on the current development set using their stored per-case evaluations. This is feasible because we only remove cases: after pruning, we recompute each candidate’s mean reward on the retained cases and update the rankings.
Revision: Appendix B describes the rescoring procedure.