acceptodds
Under review as a conference paper at ICLR 2027

Recursive Self-Improvement through Multi-Agent Self-Supervision

Abstract

Recursive self-improvement (RSI) of a model on non-verifiable tasks, such as open-ended research, faces a supervision bottleneck when its outputs exceed what even human experts can reliably assess, leaving the model itself (optimizee) as the best available optimizer and evaluator. However, a single model instance often struggles to reliably critique and improve its own complex reasoning under this homogeneous loop. To address this, we propose Multi-Agent Self-Supervision (MASS), an RSI method that alternates between evolutionary workflow optimization and supervised fine-tuning on self-generated trajectories. Guided by early findings that multi-agent topologies excel at complex reasoning, MASS prompts a single base model to iteratively propose, execute, and self-evaluate multi-agent workflows. Through an evolutionary search constrained by structural guardrails, the model optimizes these computational-graph-like orchestrations, discovering the most effective distinct roles and information routing for a given task. Across multiple open-ended research tasks, the same model on these optimized multi-agent execution traces yields higher performance per training token than training on single-agent trajectories. Because the improved model subsequently acts as a better optimizer and evaluator, this alternating framework enables a continuous, recursive bootstrapping of the model’s capabilities. These findings suggest that jointly learning orchestration and bounded subagent execution from multi-agent trajectories can provide an effective signal for RSI.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recursive self-improvement, Multi-agent systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ukxVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MASS, which alternates between optimizing multi-agent workflows and performing SFT on self-generated trajectories. The updated shared model continues to serve all three roles of execution, optimization, and evaluation. Over two rounds of updates, the authors report improvements on private scientific research tasks and two major public benchmarks, and analyze coordination behavior and training token efficiency.

Critical strengths

The paper combines multi-agent workflow search with joint SFT on controller and subagent trajectories, incorporating collaborative experience into shared parameters for use in the next round of search. This design offers a valuable methodological idea for constructing self-supervision signals in the absence of a stronger supervisor.

Critical weaknesses
  1. The experimental coverage is limited. §4.1 reports eight tasks for post-training and three private test tasks. Although evaluations on public benchmarks are also provided, this task scale and two rounds of updates remain insufficient to fully support the conclusion of sustained recursive improvement.
  2. The reliability of the self-supervision targets is questionable. Problem solving and evaluation use models from the same source, so filtering may amplify shared biases; the agreement with external LLMs measured in Figure 2 cannot, on its own, demonstrate that the scientific research results are more correct.
Questions for authors
  1. Please provide the actual number of steps, and the training and validation loss curves for both rounds, to check whether training is stabilizing, whether there are signs of overfitting, and the basis for selecting the checkpoints.
  2. When the same model is sampled multiple times on the same tasks, how much do the trajectories and windows differ? Please provide statistics on diversity or duplication rates to explain the effective supervision provided by these data.
  3. Please report the actual training efficiency and examine whether repeated SFT on a small number of tasks leads to overfitting or forgetting of general capabilities.
  4. How do different checkpoints perform on the tasks during training? Does the model selected based on validation loss also outperform the final checkpoint?
  5. How does MASS compare with existing related methods on ScienceAgentBench and MLR-Bench?
12 comments

Thanks for the comments. Please check our responses.

W1. Evalution

We have run two more cycles (L(3),L(4)\mathcal{L}^{(3)},\mathcal{L}^{(4)}) on four more benchmarks (total six), and we also report perofmrnace per outputToken. Running one cycle (workflow optimization + post-training) on qwen-3.6-27B takes about 5 days with 8 H100s.

For fairness, we keep each cycle of post-training as follows:

  • the same number of supervised tokens (17~19M);
  • the same hyperparameter config and hardware setup (8 H100s).

Here are performance results:

BenchmarkL(0)\mathcal{L}^{(0)}L(1)\mathcal{L}^{(1)}L(2)\mathcal{L}^{(2)}L(3)\mathcal{L}^{(3)}L(4)\mathcal{L}^{(4)}
(1) ScienceAgentBench28.8±1.528.8 {\scriptstyle\pm 1.5}30.4±2.030.4 {\scriptstyle\pm 2.0}31.7±2.031.7 {\scriptstyle\pm 2.0}33.3±1.833.3 {\scriptstyle\pm 1.8}35.8±2.035.8 {\scriptstyle\pm 2.0}
(2) MLR-Bench1.58±0.381.58 {\scriptstyle\pm 0.38}1.80±0.301.80 {\scriptstyle\pm 0.30}2.40±0.282.40 {\scriptstyle\pm 0.28}3.10±0.323.10 {\scriptstyle\pm 0.32}3.95±0.253.95 {\scriptstyle\pm 0.25}
(3) AstaBench0.053±0.0020.053 {\scriptstyle\pm 0.002}0.055±0.0040.055 {\scriptstyle\pm 0.004}0.067±0.0170.067 {\scriptstyle\pm 0.017}0.082±0.0120.082 {\scriptstyle\pm 0.012}0.099±0.0090.099 {\scriptstyle\pm 0.009}
(4) DSbench0.473±0.0290.473 {\scriptstyle\pm 0.029}0.465±0.0110.465 {\scriptstyle\pm 0.011}0.482±0.0250.482 {\scriptstyle\pm 0.025}0.504±0.0180.504 {\scriptstyle\pm 0.018}0.530±0.0220.530 {\scriptstyle\pm 0.022}
(5) Terminal-Bench 2.037.5±2.237.5 {\scriptstyle\pm 2.2}35.3±2.135.3 {\scriptstyle\pm 2.1}35.2±1.735.2 {\scriptstyle\pm 1.7}34.8±1.934.8 {\scriptstyle\pm 1.9}33.2±1.533.2 {\scriptstyle\pm 1.5}
(6) SWE-bench Verified62.6±1.462.6 {\scriptstyle\pm 1.4}61.8±1.961.8 {\scriptstyle\pm 1.9}61.1±0.661.1 {\scriptstyle\pm 0.6}60.2±1.160.2 {\scriptstyle\pm 1.1}58.9±0.958.9 {\scriptstyle\pm 0.9}

We also report an performance-per-outputToken comparison relative to L(0)\mathcal{L}^{(0)}.

BenchmarkL(2)/L(0)\mathcal{L}^{(2)} / \mathcal{L}^{(0)}L(3)/L(0)\mathcal{L}^{(3)} / \mathcal{L}^{(0)}L(4)/L(0)\mathcal{L}^{(4)} / \mathcal{L}^{(0)}
(1) ScienceAgentBench1.20x1.32x1.68x
(2) MLR-Bench1.60x1.85x2.10x
(3) AstaBench1.28x1.42x1.89x
(4) DSbench1.35x1.50x1.61x
(5) Terminal-Bench 2.00.96x0.95x0.91x
(6) SWE-bench Verified0.94x0.93x0.93x

Discussion. Self-improvement continues through L(3)\mathcal{L}^{(3)} and L(4)\mathcal{L}^{(4)}, supporting that RSI with MASS remains sustainable. On benchmarks (1)–(4), the gains from L(2)→L(3)\mathcal{L}^{(2)} \to \mathcal{L}^{(3)} and L(3)→L(4)\mathcal{L}^{(3)} \to \mathcal{L}^{(4)} are slightly larger than those from L(0)→L(1)\mathcal{L}^{(0)} \to \mathcal{L}^{(1)} and L(1)→L(2)\mathcal{L}^{(1)} \to \mathcal{L}^{(2)}. This is consistent with Figure~2(a),(c), where the optimizer and evaluator improve even when training uses only executor-generated data. Also, further experiments (additional cycles) further suggest that these improved roles can accelerate performance improvement via continuous bootstrapping from the improved executor.

For a fairer comparison, we also report performance per output token, which controls for the total number of output tokens used by the multi-agent system. This matches our observation that our multi-agent workflow evolution improves performance mainly by strengthening exploitation (e.g., removing failed attempts and redundant exploration from trajectories).

Finally, software benchmarks (5)–(6) show less improvement across all four cases relative to L(1)\mathcal{L}^{(1)}. One plausible reason is that our training data consists of synthetic, non-verifiable research tasks spanning finance, robotics, and pharmacy, which appear to transfer more readily to in-domain public research benchmarks. We are optimistic that benchmarks (5) and (6) would also improve if we include relevant software-engineering synthetic tasks in the training data.

W2. Reliablity of optimizer, evaluator improvement

Since our training tasks are non-verifiable and the evaluation rubrics are LLM-generated, it is difficult to rigorously test the reliability of the self-supervision training signal. Figure 2(c) uses a strong LLM to measure how similar the judgments of the post-trained Qwen3.6-27B evaluator are to those of the strong LLM.

But to more rigorously verify whether the improved optimizer/evaluator is helpful for model training, we conduct a following ablation that tests on verifiable public benchmarks.

In the draft, the update from L(1)\mathcal{L}^{(1)} to L(2)\mathcal{L}^{(2)} uses L(1)\mathcal{L}^{(1)} as the executor, optimizer, and evaluator. Here, we instead lower the optimizer and/or evaluator to L(0)\mathcal{L}^{(0)} and repeat the experiment. The hypothesis is that using L(0)\mathcal{L}^{(0)} instead of L(1)\mathcal{L}^{(1)} for either the optimizer or the evaluator will degrade the performance of the post-trained model. If so, this supports that using the post-trained model as the optimizer and evaluator can reliably boost performance in the next cycle.

Therefore, we conduct the following ablation.

Let the first-cycle model be L(1)\mathcal{L}^{(1)} and the second-cycle model be L(2)\mathcal{L}^{(2)} (the reported model in the submitted draft).

  • Case 1: Exec =L(1)→= \mathcal{L}^{(1)} \to (Opt =L(1)= \mathcal{L}^{(1)}, Eval =L(1)= \mathcal{L}^{(1)}) →Lpaper(2)\to \mathcal{L}^{(2)}_{\text{paper}} : reported
  • Case 2: Exec =L(1)→= \mathcal{L}^{(1)} \to (Opt =L(0)= \mathcal{L}^{(0)}, Eval =L(0)= \mathcal{L}^{(0)}) →Lweak opt,eval(2)\to \mathcal{L}^{(2)}_{\text{weak opt,eval}} : new experiment
  • Case 3: Exec =L(1)→= \mathcal{L}^{(1)} \to (Opt =L(1)= \mathcal{L}^{(1)}, Eval =L(0)= \mathcal{L}^{(0)}) →Lweak eval(2)\to \mathcal{L}^{(2)}_{\text{weak eval}} : new experiment
  • Case 4: Exec =L(1)→= \mathcal{L}^{(1)} \to (Opt =L(0)= \mathcal{L}^{(0)}, Eval =L(1)= \mathcal{L}^{(1)}) →Lweak opt(2)\to \mathcal{L}^{(2)}_{\text{weak opt}} : new experiment
BenchmarkL(1)\mathcal{L}^{(1)}Lpaper(2)\mathcal{L}^{(2)}_{\text{paper}}Lweak opt,eval(2)\mathcal{L}^{(2)}_{\text{weak opt,eval}}Lweak eval(2)\mathcal{L}^{(2)}_{\text{weak eval}}Lweak opt(2)\mathcal{L}^{(2)}_{\text{weak opt}}
(1) ScienceAgentBench30.4±2.030.4 {\scriptstyle\pm 2.0}31.7±2.0‾\underline{31.7 {\scriptstyle\pm 2.0}}30.6±1.830.6 {\scriptstyle\pm 1.8}30.9±2.130.9 {\scriptstyle\pm 2.1}31.5±1.9‾\underline{31.5 {\scriptstyle\pm 1.9}}
(2) MLR-Bench1.80±0.301.80 {\scriptstyle\pm 0.30}2.40±0.28‾\underline{2.40 {\scriptstyle\pm 0.28}}1.86±0.311.86 {\scriptstyle\pm 0.31}1.98±0.331.98 {\scriptstyle\pm 0.33}2.36±0.29‾\underline{2.36 {\scriptstyle\pm 0.29}}
(3) AstaBench0.055±0.0040.055 {\scriptstyle\pm 0.004}0.067±0.017‾\underline{0.067 {\scriptstyle\pm 0.017}}0.056±0.0050.056 {\scriptstyle\pm 0.005}0.058±0.0060.058 {\scriptstyle\pm 0.006}0.065±0.012‾\underline{0.065 {\scriptstyle\pm 0.012}}
(4) DSbench0.465±0.0110.465 {\scriptstyle\pm 0.011}0.482±0.025‾\underline{0.482 {\scriptstyle\pm 0.025}}0.468±0.0160.468 {\scriptstyle\pm 0.016}0.471±0.0180.471 {\scriptstyle\pm 0.018}0.479±0.021‾\underline{0.479 {\scriptstyle\pm 0.021}}
(5) Terminal-Bench 2.035.3±2.135.3 {\scriptstyle\pm 2.1}35.2±1.7‾\underline{35.2 {\scriptstyle\pm 1.7}}35.0±2.035.0 {\scriptstyle\pm 2.0}35.1±1.935.1 {\scriptstyle\pm 1.9}35.3±1.8‾\underline{35.3 {\scriptstyle\pm 1.8}}
(6) SWE-bench Verified61.8±1.961.8 {\scriptstyle\pm 1.9}61.1±0.6‾\underline{61.1 {\scriptstyle\pm 0.6}}61.4±1.561.4 {\scriptstyle\pm 1.5}61.3±1.261.3 {\scriptstyle\pm 1.2}61.2±1.0‾\underline{61.2 {\scriptstyle\pm 1.0}}

Discussion. On research benchmarks (1)–(4), strong evaluation (Lpaper(2)L_{paper}^{(2)}) outperforms weak evaluation (Lweakeval(2)L_{weak eval}^{(2)}, Lweakopt,eval(2)L_{weak opt,eval}^{(2)}). Notably, Lweakopt(2)L_{weak opt}^{(2)} performs similarly to Lpaper(2)L_{paper}^{(2)}. Evaluator quality is more critical because the evaluator is more heavily involved in post-training (e.g., self-ranking and checkpoint selection), whereas the optimizer mainly affects workflow optimization.

The finding that optimizer strength is less critical is also consistent with our observation in Figure 7(a): a weak optimizer run for many iterations can attain performance similar to a strong optimizer run for fewer iterations.

Software benchmarks (5)–(6) show less improvement across all four cases relative to L(1)\mathcal{L}^{(1)}. One plausible reason is that our training data consists of synthetic, non-verifiable research tasks spanning finance, robotics, and pharmacy, which appear to transfer more readily to in-domain public research benchmarks. We are optimistic that benchmarks (5) and (6) would also improve if we include relevant software-engineering synthetic tasks in the training data.

Q1,4. Training details 1

Steps. Cycle 1 (L(0)→L(1)\mathcal{L}^{(0)}\to\mathcal{L}^{(1)}): 1,624; Cycle 2 (L(1)→L(2)\mathcal{L}^{(1)}\to\mathcal{L}^{(2)}): 1,983; Cycle 3 (L(2)→L(3)\mathcal{L}^{(2)}\to\mathcal{L}^{(3)}): 1,758; Cycle 4 (L(3)→L(4)\mathcal{L}^{(3)}\to\mathcal{L}^{(4)}): 1,841. All use 3 windows per step.

Loss curves. Train = mean over the preceding 150 steps; Val = full validation set, measured every 150 steps.

StepC1 trainC1 valC2 trainC2 valC3 trainC3 valC4 trainC4 val
0–0.1737–0.1228–0.1048–0.0912
1500.17790.15400.14980.12160.11240.10360.09850.0904
3000.16420.15210.14070.12070.10810.10240.09420.0896
4500.14600.14970.13650.12010.10370.10150.09080.0890
6000.14850.14970.12630.11980.10020.10070.08760.0884
7500.14320.14890.12620.11730.09680.10000.08490.0879
9000.13230.14800.11850.11650.09310.09930.08210.0874
1,0500.12850.14820.10950.11550.08970.09880.07940.0870
1,2000.11870.14780.10290.11490.08640.09830.07680.0866
1,3500.12130.14800.09510.11450.08320.09840.07450.0862
1,5000.11710.14840.09240.11450.08050.09850.07230.0863
1,624 (C1 end) / 1,6500.10430.14850.09180.11470.07810.09860.07040.0864
1,758 (C3 end) / 1,8000.08800.11490.07640.09860.06870.0865
1,841 (C4 end) / 1,9500.09040.11470.06750.0865
1,983 (C2 end)0.08820.1145

Stabilization. Validation loss converges to a plateau in all four cycles; later cycles start lower and improve by smaller absolute amounts, under a matched token/step budget.

Cycle 1Cycle 2Cycle 3Cycle 4
Val loss: start →\to minimum (step) →\to end0.1737 →\to 0.1478 (1,200) →\to 0.14850.1228 →\to 0.1145 (1,350) →\to 0.11450.1048 →\to 0.0983 (1,200) →\to 0.09860.0912 →\to 0.0862 (1,350) →\to 0.0865
End vs. minimum+0.5%+0.03%+0.3%+0.3%
Validation trajectories that improved23 / 2323 / 2323 / 2322 / 23

Overfitting. Training loss keeps decreasing after validation plateaus, so a train–validation gap opens. It comes from windows that are drawn again (sampling is with replacement), not from worse generalization:

Orchestrator windowsTrain, first drawTrain, re-drawnVal
Cycle 1, steps 601–9000.1950.1500.193
Cycle 2, steps 901–1,2000.1690.1180.164
Cycle 3, steps 601–9000.1510.1020.148
Cycle 4, steps 901–1,2000.1380.0910.136

Loss on windows seen for the first time stays at the validation level; only re-drawn windows are memorized. Validation loss rises by at most 0.5% after its minimum.

Checkpoint basis. Cycle 1: step 1,200, the validation-loss minimum. self-evalutor used step 1200. Cycle 2: validation loss is flat from step 1,350 to the end (within 0.35%) and does not separate these checkpoints; Self-evalutor use step 1,600, the best plateau checkpoint on training-task rollouts. The validation minimum is step 1,350. Cycle 3: step 1,350 (near the validation minimum at 1,200; flat within 0.3%),self-evalutor used step 1350 . Cycle 4: step 1,500 (plateau after the validation minimum at 1,350; flat within 0.4%), self-evalutor used step 1500.

Reviewer_166NOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes Multi-Agent Self-Supervision (MASS), a recursive self-improvement framework that extends RSI from single-agent systems to multi-agent systems.

Critical strengths
  1. The paper presents an interesting insight: as modern agentic systems increasingly adopt multi-agent architectures, RSI should also be investigated in multi-agent settings.
Critical weaknesses
  1. Limited methodological novelty. This work largely extends RHI with multi-agent training, while the fundamental workflow-optimization methodology remains largely unchanged. Closely related work, MASS [1], is also missing from the references.

  2. Preliminary experimental evaluation. The multi-agent and single-agent training configurations differ. For example, the main multi-agent student is trained on 119 trajectories collected from eight tasks, whereas the single-agent controls use different task coverage. Furthermore, on ScienceAgentBench, the score increases only from 28.8% to 31.7%. The paper also evaluates only two cycles of evolution, which provides limited evidence of sustained recursive self-improvement.

[1] Zhou, Han, Xingchen Wan, Ruoxi Sun, Hamid Palangi, Shariq Iqbal, Ivan Vulić, Anna Korhonen, and Sercan Arik. "Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies." International Conference on Learning Representations (ICLR), 2026.

Questions for authors

No particular questions.

5 comments

W2. Evalution.

W2.1 config inconsistency.

The single-agent and multi-agent training configurations are trained on the same task coverage of eight tasks. If the paper states that these configurations differ, we admit that this is a typo, and we thank the reviewer for raising the issue.

W2.2. Weak evaluation.

Thanks for raising this issue. We have run two more cycles (L(3),L(4)\mathcal{L}^{(3)},\mathcal{L}^{(4)}) on four more benchmarks (total six), and we also report perofmrnace per outputToken. Running one cycle (workflow optimization + post-training) takes about 5 days with 8 H100s.

For fairness, we keep each cycle of post-training as follows:

  • the same number of supervised tokens (17~19M);
  • the same hyperparameter config and hardware setup (8 H100s).

Here are performance results:

BenchmarkL(0)\mathcal{L}^{(0)}L(1)\mathcal{L}^{(1)}L(2)\mathcal{L}^{(2)}L(3)\mathcal{L}^{(3)}L(4)\mathcal{L}^{(4)}
(1) ScienceAgentBench28.8±1.528.8 {\scriptstyle\pm 1.5}30.4±2.030.4 {\scriptstyle\pm 2.0}31.7±2.031.7 {\scriptstyle\pm 2.0}33.3±1.833.3 {\scriptstyle\pm 1.8}35.8±2.035.8 {\scriptstyle\pm 2.0}
(2) MLR-Bench1.58±0.381.58 {\scriptstyle\pm 0.38}1.80±0.301.80 {\scriptstyle\pm 0.30}2.40±0.282.40 {\scriptstyle\pm 0.28}3.10±0.323.10 {\scriptstyle\pm 0.32}3.95±0.253.95 {\scriptstyle\pm 0.25}
(3) AstaBench0.053±0.0020.053 {\scriptstyle\pm 0.002}0.055±0.0040.055 {\scriptstyle\pm 0.004}0.067±0.0170.067 {\scriptstyle\pm 0.017}0.082±0.0120.082 {\scriptstyle\pm 0.012}0.099±0.0090.099 {\scriptstyle\pm 0.009}
(4) DSbench0.473±0.0290.473 {\scriptstyle\pm 0.029}0.465±0.0110.465 {\scriptstyle\pm 0.011}0.482±0.0250.482 {\scriptstyle\pm 0.025}0.504±0.0180.504 {\scriptstyle\pm 0.018}0.530±0.0220.530 {\scriptstyle\pm 0.022}
(5) Terminal-Bench 2.037.5±2.237.5 {\scriptstyle\pm 2.2}35.3±2.135.3 {\scriptstyle\pm 2.1}35.2±1.735.2 {\scriptstyle\pm 1.7}34.8±1.934.8 {\scriptstyle\pm 1.9}33.2±1.533.2 {\scriptstyle\pm 1.5}
(6) SWE-bench Verified62.6±1.462.6 {\scriptstyle\pm 1.4}61.8±1.961.8 {\scriptstyle\pm 1.9}61.1±0.661.1 {\scriptstyle\pm 0.6}60.2±1.160.2 {\scriptstyle\pm 1.1}58.9±0.958.9 {\scriptstyle\pm 0.9}

We also report an performance-per-outputToken comparison relative to L(0)\mathcal{L}^{(0)}.

BenchmarkL(2)/L(0)\mathcal{L}^{(2)} / \mathcal{L}^{(0)}L(3)/L(0)\mathcal{L}^{(3)} / \mathcal{L}^{(0)}L(4)/L(0)\mathcal{L}^{(4)} / \mathcal{L}^{(0)}
(1) ScienceAgentBench1.20x1.32x1.68x
(2) MLR-Bench1.60x1.85x2.10x
(3) AstaBench1.28x1.42x1.89x
(4) DSbench1.35x1.50x1.61x
(5) Terminal-Bench 2.00.96x0.95x0.91x
(6) SWE-bench Verified0.94x0.93x0.93x

Discussion. Self-improvement continues through L(3)\mathcal{L}^{(3)} and L(4)\mathcal{L}^{(4)}, supporting that RSI with MASS remains sustainable. On benchmarks (1)–(4), the gains from L(2)→L(3)\mathcal{L}^{(2)} \to \mathcal{L}^{(3)} and L(3)→L(4)\mathcal{L}^{(3)} \to \mathcal{L}^{(4)} are slightly larger than those from L(0)→L(1)\mathcal{L}^{(0)} \to \mathcal{L}^{(1)} and L(1)→L(2)\mathcal{L}^{(1)} \to \mathcal{L}^{(2)}. This is consistent with Figure~2(a),(c), where the optimizer and evaluator improve even when training uses only executor-generated data. Also, further experiments (additional cycles) further suggest that these improved roles can accelerate performance improvement via continuous bootstrapping from the improved executor.

For a fairer comparison, we also report performance per output token, which controls for the total number of output tokens used by the multi-agent system. This matches our observation that our multi-agent workflow evolution improves performance mainly by strengthening exploitation (e.g., removing failed attempts and redundant exploration from trajectories).

Finally, software benchmarks (5)–(6) show less improvement across all four cases relative to L(1)\mathcal{L}^{(1)}. One plausible reason is that our training data consists of synthetic, non-verifiable research tasks spanning finance, robotics, and pharmacy, which appear to transfer more readily to in-domain public research benchmarks. We are optimistic that benchmarks (5) and (6) would also improve if we include relevant software-engineering synthetic tasks in the training data.

Thank the authors for the response.

I agree with the authors insights on novelty clarification, especially considering current RSI community is crowded. Regarding the experiment, thank you for the results, which takes large computation.

I have one follow-up question. I understand this may sound like an LLM-generated review point but it isn't. I am also actively working on related research and have observed that experimental results can vary considerably across different random seeds. I wonder whether the authors have investigated this or have any results to report across runs with different seeds.

One minor point to correct: "when a self-improvement loop lacks external feedback for post-training, a reliable feedback signal can be obtained by distilling the output of copeis of itself". Authors may want to check following work:

[1] Srivastava, Gaurav, Zhenyu Bi, Meng Lu, and Xuan Wang. "DEBATE, TRAIN, EVOLVE: Self‐Evolution of Language Model Reasoning." In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 32752-32798. 2025. [2] Subramaniam, Vighnesh, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. "Multiagent finetuning: Self improvement with diverse reasoning chains." In International Conference on Learning Representations, vol. 2025, pp. 10840-10862. 2025. [3] Luo, Yinyi, Yiqiao Jin, Weichen Yu, Mengqi Zhang, Srijan Kumar, Xiaoxiao Li, Weijie Xu, Xin Chen, and Jindong Wang. "Agentark: Distilling multi-agent intelligence into a single llm agent." NeurIPS 2026. [4] Yao, Dixi, Tahseen Rabbani, Manzil Zaheer, and Tian Li. "Federation over Text: Insight Sharing for Multi-Agent Reasoning." arXiv preprint arXiv:2604.16778 (2026). [5] Nie, Jun, Yonggang Zhang, Qianshu Cai, Yiu-ming Cheung, Xinmei Tian, and Bo Han. "EvolveNet: Collaborative harness evolution for agent self-improvement." arXiv preprint arXiv:2608.04968 (2026).

Thanks for the follow-up!

Rebuttal point1. Different random seeds -> Different test results. Why?

Thanks for the question; indeed, this is a very good point. In fact, we observed the same phenomenon during our experiments, and addressing it was one of the major engineering challenges that required substantial effort to make the MASS cycle work effectively.

Here is what we think and concluded: We think that the "long-horizon" nature of agentic tasks inevitably introduces large variance in the generated outputs, which can lead to large variations in test results, regardless of whether the mean performance is high or low. This effect can be particularly pronounced when evaluation relies on verifiable (or binary, such as correct/incorrect) outcome rewards rather than more fine-grained outcome rewards or process-level evaluation metrics.

Let me provide a simple example. Suppose that the initial multi-agent workflow w0w_0 exhibits low average performance and high variance across its outputs. After iteratively optimizing w0w_0, we obtain an improved workflow w⋆w^\star. Although w⋆w^\star achieves substantially higher mean performance, we may still observe high variance across its generated outputs.

We believe this observation has three implications.

  • First, high output variance may be somewhat inevitable when the horizon length of a coding agent exceeds a certain critical threshold. For reference, our multi-agent system using Qwen 3.6 27B generated approximately 150K–200K tokens per task on average. We speculate that this critical horizon length may increase as model capabilities improve, although this hypothesis requires further investigation.

  • Second, this observation may suggest that the community needs additional harness-level mechanisms or algorithms specifically designed to reduce variance in long-horizon coding agent tasks for accurate evaluation. It seems that we are on the same page that accurate evaluation is one of the most important aspects of self-improvement.

  • Third, this observation highlights the need for further research on developing fine-grained, process-level evaluation metrics to mitigate the high variance associated with binary verifiable rewards.

So, how did we address this challenge in MASS?

Our practical solution was relatively simple. We observed that even the optimized workflow w⋆w^\star exhibited substantial variance in its generated trajectories. Therefore, for each training task, we sampled 12 trajectories using w⋆w^\star, selected the top-kk trajectories through self-evaluation, and used only these selected high-quality trajectories for supervised fine-tuning (SFT).

Although this approach is conceptually straightforward, developing an effective trajectory-selection strategy was one of the major engineering bottlenecks we encountered. It required substantial trial and error to make the MASS self-improvement cycle work reliably.

Therefore, here is our suggestion for the reviewer: Suppose a benchmark contains 100 tasks, each evaluated with 10 different random seeds. I would first sort the 100 tasks by the coding agent's output token length and examine whether there is a positive correlation between output token length and task-level variance. If so, we may need to devise an external algorithm to reduce output variance.

Rebuttal point2. Related Work

Thank you for pointing out these works! We will add them to the related work section in the revised version.

Reviewer_GHFXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Multi-Agent Self-Supervision, which alternates between optimizing multi-agent workflows and finetuning a shared model on collected execution trajectories. The updated model then resumes the roles of executor, workflow optimizer, and evaluator. The intended contribution is to obtain useful supervision through coordination when the model being improved is also its own supervisor.

The experiments use Qwen3.6-27B, with eight synthetic research tasks contributing training data. Across two cycles, the authors report improvements on three held-out synthetic tasks, ScienceAgentBench, and MLR-Bench. Additional experiments examine transfer to workflow discovery and evaluation, training-token efficiency, and the internalization of coordination behaviors.

Critical strengths

Clarity and Depth I really appreciate the clarity and depth of thought with which the authors have conducted experiments in this paper. Many of my concerns were addressed in the discussions section.

Critical weaknesses

My main concern is with the intent of various experiments. Right now, the experiments do not sufficiently distinguish the contribution of the proposed recursive algorithm from prompting useful behavior, repeated sampling, and additional supervised training.

Evidence for Recursive Self-improvement. The workflow-transfer result is relevant, and Figure 2 also reports increased evaluator agreement and improved second-generation task performance. However, showing that the model becomes a better executor, workflow discoverer, or evaluator does not establish that feeding those improvements back into the pipeline produces a better next-generation student. During workflow discovery, all three roles change together. The role ablations in Figure 6 hold model weights fixed and do not carry the alternatives through another training cycle.

One potential experiment to resolve this: start both second-cycle conditions from the same first-cycle student. In one condition, use the updated model for all three roles. In the other, retain the original optimizer and evaluator, including for training-trajectory ranking, while using the same first-cycle student as executor. Both conditions should generate fresh data and continue training under matched task and computational budgets. Comparing the resulting students would test whether updating the improvement process contributes beyond another round of data generation and fine-tuning.

Contribution of Evolutionary Workflow Search. Right now, the benefit of training on multi-agent executions isn't being distinguished from the benefit of evolving the workflows that generate those executions. Instead of using an evolutionary workflow search, what do we expect to see with a simpler baseline search algorithm such as a agent-driven workflow search or a simple branch and bound search? I think Appendix C.4 has a baseline experiment like this where the base model is prompted to organize its own team, and it seems to yield acceptable performance.

I think this is an important baseline because the existing control compares the base and MASS-trained models when both receive the directive at evaluation time. It does not compare students trained on executions generated by the two approaches. Hence, the control establishes that coordination can be elicited without search, but leaves unresolved whether search produces better supervision for subsequent learning.

Here's a simple experiment for this: initialize two compute-matched students from the same base model where one is trained with the evolutionary workflow's executions and the other using the prompt based workflow's executions. My assumption is that the evolutionary workflow's student would have an advantage over the prompt-based workflow; and it will show that optimizing the workflows yields measurable gains over simply prompting useful coordination behaviors and training on the resulting executions.

Questions for authors

Two questions that are separate from those raised in the weaknesses section:

  1. How much does the framework rely on the initial model’s capabilities? Does the homogeneous loop still generate useful supervision with a substantially smaller, roughly 9B-class backbone? Conversely, with a more capable backbone, do the authors expect the evolutionary workflow search to still be better than simply asking that model to organize and execute the task?

  2. How sensitive is the framework to the quality of self-generated feedback?. How much evaluation capability is required for this feedback to remain useful? For example, what happens when the in-loop evaluator receives a smaller reasoning budget or is replaced with a less capable model, while the executor and optimizer remain unchanged? Does degraded feedback merely slow workflow discovery, or does it produce poorer training data that undermines subsequent learning?

Overall, I agree that coordinating copies of a model to generate its own supervision is an interesting problem to study. However, I am not yet convinced that the evaluation establishes the benefit of the proposed recursive search-and-training algorithm over simpler alternatives.

3 comments

Q1. Small Model Experiment

Experimenting with a smaller model requires an additional algorithm on top of MASS for our framework to work. We initially started this project by testing MASS with Qwen 3.5 9B (with a 131K context window and a 20K output limit). However, the main challenge was that workspace evaluation often became infeasible due to the Qwen coding agent's built-in safeguards, particularly its detection of repeated or malformed tool calls, of the base language model's output.

Specifically, during workflow evolution in MASS, the executor solves tasks using the current workflow wiw_i and generates outputs (e.g., .py, .md, and .json files) in a workspace. The evaluator (LLM-as-a-judge) then compares the workspace outputs generated by wiw_i against those produced by the best-so-far workflow wjw_j, where j<ij < i.

We initially experimented with the full set of 30 candidate training tasks (from which we selected 12 tasks for our paper). We observed that the initial task completion rates for workflows w0w_0, w1w_1, and w2w_2 were 14/30, 15/30, and 15/30, respectively. The two main reasons for failure were (1) repeated identical tool-call loops and (2) reaching the maximum number of tool calls allowed by the Qwen coding harness (three), which serves as a safety safeguard.

To address this issue, we first attempted repeated sampling until all 30 tasks were successfully completed. For example, given w0w_0, if the initial completion rate was 14/30, we retried the remaining 16 failed tasks. If 10 additional tasks succeeded, we then retried the remaining 6 tasks. We continued this process within our compute budget until all 30 tasks were completed.

After each failed attempt, we also incorporated the failure cases into the workflow optimizer. This allowed the optimizer to consider not only successfully completed tasks but also failure cases, with the goal of improving the workflow's ability to complete previously unsuccessful tasks.

However, this approach did not lead to consistent performance improvements when using Qwen 3.5 9B evaluated on hidden tests using a stronger LLM. Specifically, although we observed an initial improvement from w0w_0 to w1w_1, subsequent workflows (w2w_2, w3w_3, w4w_4, and so on) achieved nearly identical performance to w1w_1. Interestingly, the total number of retries required for workflows w0w_0, w1w_1, w2w_2, and w3w_3 was 61, 76, 75, and 43, respectively. This suggests that the workflow optimizer may have gradually improved its ability to handle execution failures. Nevertheless, the reduction in retries did not translate into a corresponding improvement in task performance.

Our hypothesis is that repeatedly sampling the Qwen coding agent's executions until all tasks are completed introduces a form of survivorship bias. Specifically, we selectively retain executions that satisfy the task-completion criteria, even though their actual quality, as measured by the evaluation metrics, may remain low. Overall, this suggests that the MASS cycle may require an additional outer-loop mechanism to handle execution failures more effectively when using smaller models. Alternatively, it may indicate that the MASS framework requires a certain minimum level of model capability (or agentic harness capability) to function effectively. At least, our experiments show that MASS works with Qwen 3.6 27B but does not work effectively with Qwen 3.5 9B under the same Qwen coding harness.

However, the primary objective of our work is to investigate whether optimizing multi-agent workflows and subsequently performing SFT on their high-quality execution trajectories can provide an effective learning signal for self-improvement. Since this is our central research hypothesis, we decided to use a more capable base model to isolate and study this mechanism, rather than introducing additional algorithmic components specifically designed to address the execution failures of smaller models. Such modifications could go beyond the scope of our current work and potentially constitute a separate research direction.

We believe that one straightforward way to make MASS work with a smaller model (Qwen 3.5 9B) would be to use simpler, shorter-horizon training tasks than those currently included in our training set. This could reduce execution failures and allow us to investigate whether the MASS self-improvement mechanism remains effective at smaller model scales.

We are currently conducting additional experiments with Qwen 3.8 27B, although completing four full MASS cycles is expected to take approximately four weeks due to the computational cost. While these results may not be available within the rebuttal period, we believe our responses to W1, W2, and Q2 provide relevant empirical evidence and discussion that partially address the reviewer's concerns regarding the sensitivity of MASS to base-model capability!

Thank you for the response and the detailed experiments!

W1. Additional component level ablations

Thank you for the additional ablations. A couple of additional observations:

  • Thank you for the case breakdown presented here -- these are all reasonable and competitive baselines.
  • I agree that improved evaluator capability seems to provides the maximal benefit. This motivates a clear direction of improvement for the community.
  • On all benchmarks, the marginal improvement for the suggested method is somewhat limited. On TB2 and SWE-Bench Verified, L^(2)'s improvement is largely in reducing the variance around L^(1). In my experience, this is usually an indication of distribution sharpening rather than of overall capability improvement.
    • Since the delta improvement is small, qualitative analysis of the successful cases could help resolve what behavior is being learned between the cycles.

W2. evolutionary / prompt based. RSI w/ Sub-optimal workflows

Sorry about the confusion and I agree, the provided ablation hits at my main underlying concern. It also makes sense that amortized reasoning helps provide small improvements in each cycle.

Q2. Results with different evaluators

Thank you for the experiments. I generally agree that blindly increasing capability may not yield scalable benefits from verification. However, when more capable models are paired with good tooling (e.g. to lazily monitor large quantities of signals), they tend to outperform cheaper models. However, the results and analysis presented here make sense to me.

Q1. Small models lack basic tool calling capabilities; out of domain.

This is understandable. Thanks for the explanation.


The presented ablations, in my reading, seem to corroborate with the presented results as well and many of my initial concerns have now been addressed.

MASS presents an in-depth investigation into whether a simple technique -- collecting multi-agent workflow trajectories and fine tuning on them -- provides an effective signal for self-improvement.

Pros:

  • In depth analysis of a problem of interest to the ICLR community.
  • Useful testbed for self-improving post-training. Assuming the code and artifacts are responsibly released and the results are reproducible, MASS could be a useful and simple to understand baseline for future work in this area.

Cons:

  • Results are somewhat lackluster -- but this is generally a consequence of having datasets and metrics that weren't designed for RSI research.
  • Lacking methodological novelty -- as reported by Reviewer 166N. I agree with this assessment but a re-analysis of such a simple methodology would be useful for the RSI community.

For me, the pros now outweigh the cons and I'm happy to increase the score and champion this paper for acceptance.

AuthorsOpenReview ↗

Thanks for the rebuttal

As we approach the end of the rebuttal period, we would like to smoothly wrap up this discussion thread.

First and foremost, we sincerely thank the reviewer for taking the time to carefully read our paper, provide thoughtful comments, and engage in further discussions throughout the rebuttal period. We recognize that this process requires substantial time and effort, and we genuinely appreciate your engagement.

We particularly appreciate the reviewer for suggesting two ablation studies in the Weaknesses section. Those suggestions have been very helpful in allowing us to more clearly articulate where the performance gains actually come. Overall, we believe that these additional experiments have helped strengthen the positioning and main messages of our paper. We will include the results of both suggested ablation studies in the main experimental section of the revised manuscript.

On a related note, we are also continuing our experiments with the Qwen 3.8 27B model. We have found that these experiments take substantially longer than those with Qwen 3.6 27B (even with more computational resources than we had secured during the initial manuscript submission). This is because Qwen 3.8 27B tends to generate more tokens and significantly longer trajectories, making the workflow search process considerably more time-consuming. Nevertheless, we appreciate the reviewer's question regarding how our method performs across models with different capabilities. We believe that addressing this question will further strengthen our work. More broadly, we believe this is an important question for the research community: to what extent can self-improvement be sustained, and when might it converge, as the capabilities of underlying LLMs continue to advance rapidly?. We therefore intend to complete these experiments as much as possible and include the results in the revised version.

Overall, we are glad that the reviewer's initial concerns have been resolved through our discussions.

Once again, we sincerely thank the reviewer for the thoughtful comments and continued engagement throughout the rebuttal process. If you have any further suggestions regarding our research direction, potential follow-up investigations, or questions that you believe would be particularly valuable for the community to explore, we would be very grateful to hear them. Your feedback would be highly valuable as we continue developing this research project. Please feel free to drop any such thoughts.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.