- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MASS, which alternates between optimizing multi-agent workflows and performing SFT on self-generated trajectories. The updated shared model continues to serve all three roles of execution, optimization, and evaluation. Over two rounds of updates, the authors report improvements on private scientific research tasks and two major public benchmarks, and analyze coordination behavior and training token efficiency.
Critical strengths
The paper combines multi-agent workflow search with joint SFT on controller and subagent trajectories, incorporating collaborative experience into shared parameters for use in the next round of search. This design offers a valuable methodological idea for constructing self-supervision signals in the absence of a stronger supervisor.
Critical weaknesses
- The experimental coverage is limited. §4.1 reports eight tasks for post-training and three private test tasks. Although evaluations on public benchmarks are also provided, this task scale and two rounds of updates remain insufficient to fully support the conclusion of sustained recursive improvement.
- The reliability of the self-supervision targets is questionable. Problem solving and evaluation use models from the same source, so filtering may amplify shared biases; the agreement with external LLMs measured in Figure 2 cannot, on its own, demonstrate that the scientific research results are more correct.
Questions for authors
- Please provide the actual number of steps, and the training and validation loss curves for both rounds, to check whether training is stabilizing, whether there are signs of overfitting, and the basis for selecting the checkpoints.
- When the same model is sampled multiple times on the same tasks, how much do the trajectories and windows differ? Please provide statistics on diversity or duplication rates to explain the effective supervision provided by these data.
- Please report the actual training efficiency and examine whether repeated SFT on a small number of tasks leads to overfitting or forgetting of general capabilities.
- How do different checkpoints perform on the tasks during training? Does the model selected based on validation loss also outperform the final checkpoint?
- How does MASS compare with existing related methods on ScienceAgentBench and MLR-Bench?
12 comments
W2. Reliablity of optimizer, evaluator improvement
Since our training tasks are non-verifiable and the evaluation rubrics are LLM-generated, it is difficult to rigorously test the reliability of the self-supervision training signal. Figure 2(c) uses a strong LLM to measure how similar the judgments of the post-trained Qwen3.6-27B evaluator are to those of the strong LLM.
But to more rigorously verify whether the improved optimizer/evaluator is helpful for model training, we conduct a following ablation that tests on verifiable public benchmarks.
In the draft, the update from to uses as the executor, optimizer, and evaluator. Here, we instead lower the optimizer and/or evaluator to and repeat the experiment. The hypothesis is that using instead of for either the optimizer or the evaluator will degrade the performance of the post-trained model. If so, this supports that using the post-trained model as the optimizer and evaluator can reliably boost performance in the next cycle.
Therefore, we conduct the following ablation.
Let the first-cycle model be and the second-cycle model be (the reported model in the submitted draft).
- Case 1: Exec (Opt , Eval ) : reported
- Case 2: Exec (Opt , Eval ) : new experiment
- Case 3: Exec (Opt , Eval ) : new experiment
- Case 4: Exec (Opt , Eval ) : new experiment
| Benchmark | |||||
|---|---|---|---|---|---|
| (1) ScienceAgentBench | |||||
| (2) MLR-Bench | |||||
| (3) AstaBench | |||||
| (4) DSbench | |||||
| (5) Terminal-Bench 2.0 | |||||
| (6) SWE-bench Verified |
Discussion. On research benchmarks (1)–(4), strong evaluation () outperforms weak evaluation (, ). Notably, performs similarly to . Evaluator quality is more critical because the evaluator is more heavily involved in post-training (e.g., self-ranking and checkpoint selection), whereas the optimizer mainly affects workflow optimization.
The finding that optimizer strength is less critical is also consistent with our observation in Figure 7(a): a weak optimizer run for many iterations can attain performance similar to a strong optimizer run for fewer iterations.
Software benchmarks (5)–(6) show less improvement across all four cases relative to . One plausible reason is that our training data consists of synthetic, non-verifiable research tasks spanning finance, robotics, and pharmacy, which appear to transfer more readily to in-domain public research benchmarks. We are optimistic that benchmarks (5) and (6) would also improve if we include relevant software-engineering synthetic tasks in the training data.
Q1,4. Training details 1
Steps. Cycle 1 (): 1,624; Cycle 2 (): 1,983; Cycle 3 (): 1,758; Cycle 4 (): 1,841. All use 3 windows per step.
Loss curves. Train = mean over the preceding 150 steps; Val = full validation set, measured every 150 steps.
| Step | C1 train | C1 val | C2 train | C2 val | C3 train | C3 val | C4 train | C4 val |
|---|---|---|---|---|---|---|---|---|
| 0 | – | 0.1737 | – | 0.1228 | – | 0.1048 | – | 0.0912 |
| 150 | 0.1779 | 0.1540 | 0.1498 | 0.1216 | 0.1124 | 0.1036 | 0.0985 | 0.0904 |
| 300 | 0.1642 | 0.1521 | 0.1407 | 0.1207 | 0.1081 | 0.1024 | 0.0942 | 0.0896 |
| 450 | 0.1460 | 0.1497 | 0.1365 | 0.1201 | 0.1037 | 0.1015 | 0.0908 | 0.0890 |
| 600 | 0.1485 | 0.1497 | 0.1263 | 0.1198 | 0.1002 | 0.1007 | 0.0876 | 0.0884 |
| 750 | 0.1432 | 0.1489 | 0.1262 | 0.1173 | 0.0968 | 0.1000 | 0.0849 | 0.0879 |
| 900 | 0.1323 | 0.1480 | 0.1185 | 0.1165 | 0.0931 | 0.0993 | 0.0821 | 0.0874 |
| 1,050 | 0.1285 | 0.1482 | 0.1095 | 0.1155 | 0.0897 | 0.0988 | 0.0794 | 0.0870 |
| 1,200 | 0.1187 | 0.1478 | 0.1029 | 0.1149 | 0.0864 | 0.0983 | 0.0768 | 0.0866 |
| 1,350 | 0.1213 | 0.1480 | 0.0951 | 0.1145 | 0.0832 | 0.0984 | 0.0745 | 0.0862 |
| 1,500 | 0.1171 | 0.1484 | 0.0924 | 0.1145 | 0.0805 | 0.0985 | 0.0723 | 0.0863 |
| 1,624 (C1 end) / 1,650 | 0.1043 | 0.1485 | 0.0918 | 0.1147 | 0.0781 | 0.0986 | 0.0704 | 0.0864 |
| 1,758 (C3 end) / 1,800 | 0.0880 | 0.1149 | 0.0764 | 0.0986 | 0.0687 | 0.0865 | ||
| 1,841 (C4 end) / 1,950 | 0.0904 | 0.1147 | 0.0675 | 0.0865 | ||||
| 1,983 (C2 end) | 0.0882 | 0.1145 |
Stabilization. Validation loss converges to a plateau in all four cycles; later cycles start lower and improve by smaller absolute amounts, under a matched token/step budget.
| Cycle 1 | Cycle 2 | Cycle 3 | Cycle 4 | |
|---|---|---|---|---|
| Val loss: start minimum (step) end | 0.1737 0.1478 (1,200) 0.1485 | 0.1228 0.1145 (1,350) 0.1145 | 0.1048 0.0983 (1,200) 0.0986 | 0.0912 0.0862 (1,350) 0.0865 |
| End vs. minimum | +0.5% | +0.03% | +0.3% | +0.3% |
| Validation trajectories that improved | 23 / 23 | 23 / 23 | 23 / 23 | 22 / 23 |
Overfitting. Training loss keeps decreasing after validation plateaus, so a train–validation gap opens. It comes from windows that are drawn again (sampling is with replacement), not from worse generalization:
| Orchestrator windows | Train, first draw | Train, re-drawn | Val |
|---|---|---|---|
| Cycle 1, steps 601–900 | 0.195 | 0.150 | 0.193 |
| Cycle 2, steps 901–1,200 | 0.169 | 0.118 | 0.164 |
| Cycle 3, steps 601–900 | 0.151 | 0.102 | 0.148 |
| Cycle 4, steps 901–1,200 | 0.138 | 0.091 | 0.136 |
Loss on windows seen for the first time stays at the validation level; only re-drawn windows are memorized. Validation loss rises by at most 0.5% after its minimum.
Checkpoint basis. Cycle 1: step 1,200, the validation-loss minimum. self-evalutor used step 1200. Cycle 2: validation loss is flat from step 1,350 to the end (within 0.35%) and does not separate these checkpoints; Self-evalutor use step 1,600, the best plateau checkpoint on training-task rollouts. The validation minimum is step 1,350. Cycle 3: step 1,350 (near the validation minimum at 1,200; flat within 0.3%),self-evalutor used step 1350 . Cycle 4: step 1,500 (plateau after the validation minimum at 1,350; flat within 0.4%), self-evalutor used step 1500.
Thanks for the comments. Please check our responses.
W1. Evalution
We have run two more cycles (L(3),L(4)) on four more benchmarks (total six), and we also report perofmrnace per outputToken. Running one cycle (workflow optimization + post-training) on qwen-3.6-27B takes about 5 days with 8 H100s.
For fairness, we keep each cycle of post-training as follows:
Here are performance results:
We also report an performance-per-outputToken comparison relative to L(0).
Discussion. Self-improvement continues through L(3) and L(4), supporting that RSI with MASS remains sustainable. On benchmarks (1)–(4), the gains from L(2)→L(3) and L(3)→L(4) are slightly larger than those from L(0)→L(1) and L(1)→L(2). This is consistent with Figure~2(a),(c), where the optimizer and evaluator improve even when training uses only executor-generated data. Also, further experiments (additional cycles) further suggest that these improved roles can accelerate performance improvement via continuous bootstrapping from the improved executor.
For a fairer comparison, we also report performance per output token, which controls for the total number of output tokens used by the multi-agent system. This matches our observation that our multi-agent workflow evolution improves performance mainly by strengthening exploitation (e.g., removing failed attempts and redundant exploration from trajectories).
Finally, software benchmarks (5)–(6) show less improvement across all four cases relative to L(1). One plausible reason is that our training data consists of synthetic, non-verifiable research tasks spanning finance, robotics, and pharmacy, which appear to transfer more readily to in-domain public research benchmarks. We are optimistic that benchmarks (5) and (6) would also improve if we include relevant software-engineering synthetic tasks in the training data.