- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MAS-Loop jointly trains an Orchestrator that constructs a multi-agent system and a Refiner that diagnoses its execution and proposes a bounded graph or configuration edit. The revised system is executed again, and source/updated outcomes supply separate rewards for the two policies. The Refiner receives execution evidence but no gold answer. Experiments cover interactive tasks, multi-hop QA, and reasoning, including transfer to held-out task families and other MAS generators. The paired rollouts also form MAS-JudgeCorpus for training a system-selection reward model.
Critical strengths
-
The paper connects execution diagnosis to an explicit, executable edit interface and learns both system construction and refinement. The results are useful beyond running two policies together: the standalone Orchestrator is strong, and the Refiner improves systems from several other generators. This makes the contribution more convincing than an improvement tied to one bespoke workflow.
-
The evaluation includes several relevant controls. Appendix D doubles MAS-Orchestra’s rollout group to match the number of system executions and includes a prompted reflection pass with the same gold-free diagnostic input and bounded edit interface. Both remain below MAS-Loop on HotpotQA and BrowseComp-Plus. Together with the reward-coefficient ablations and held-out transfer results, these comparisons provide meaningful support for learning execution-grounded refinement. I lean toward acceptance on this evidence.
Critical weaknesses
-
The reward signal does not separate a patch effect from execution randomness. Each source graph has one source execution and one independent updated execution; a training group samples different systems rather than repeating the same intervention. Consequently, max(Δs,0) in Eq. (11) can penalize apparent recoverability caused by noise, even when a valid edit has no effect on expected performance. The unchanged-graph branch avoids this penalty, but does not establish how much of the valid-patch signal comes from actual repairs. The paper correctly calls Δs a realized score change; its interpretation as evidence of a recoverable architectural weakness still needs support.
-
Several incremental gains from adding the Refiner are small relative to the evaluation sets: AIME has 57 questions and GPQA has 198, and the overall average gain over the standalone Orchestrator is 1.96 points. Repeated training/evaluation results and complete inference costs are not reported. These limitations weaken the strength of the co-evolution and deployment claims, although the larger standalone-policy gains and matched-execution controls keep my recommendation positive.
Questions for authors
-
For fixed source graphs, how does the distribution of improvements after a valid patch compare with independent unchanged-graph reruns? A repeated-continuation control would clarify how much the rewards measure actual repair rather than execution noise.
-
How stable are the standalone and Refiner gains across independent training/evaluation seeds? Please report Accept, invalid-edit, improvement, and regression rates alongside the principal task scores.
-
What is the total inference token/latency cost of source execution, diagnosis, and re-execution relative to the strongest baselines? For the best-of-16 selection experiment, please distinguish verification-only tokens from candidate generation and execution costs.
3 comments
Dear Reviewer DdLi,
We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. Your insights have been instrumental in refining our work.
To address your concerns, we have provided additional clarifications and experiments to enhance the clarity of our contributions. We hope these responses effectively resolve any ambiguities, provide a more comprehensive understanding of our work, and encourage you to reconsider your rating.
If you have any further questions or require additional clarification, we would be happy to discuss them. We truly appreciate your time and consideration and look forward to your feedback.
Best regards,
The Authors
Dear Reviewer DdLi,
Thank you again for your time and constructive feedback.
With about one day remaining in the author–reviewer discussion period, we would greatly appreciate it if you could take a moment to review our rebuttal and revised manuscript. We have addressed your concerns through additional analyses, experiments, and clarifications.
We would be happy to clarify any remaining questions and welcome your further feedback.
Thank you for your time and consideration.
Best regards,
The Authors
Response to Reviewer DdLi
We thank the reviewer for recognizing policy transfer and the existing execution-matched controls.
[W1 + Q1] Fixed-patch effects versus unchanged-graph reruns
We agree that single-sample feedback can be affected by execution noise. To test whether initial gains persist, we randomly select 200 initially improving pairs from HotpotQA training log. Both graphs and the original patch remain fixed, and each source MAS and updated MAS are independently executed eight times. Comparing each pair's eight-execution means, 90.50% retain the initial positive direction. These results provide empirical support for using a single Refiner sample per source MAS in our RL training. We have added the sampling trade-off to Section 3.3 and the experiment to Appendix D.3.1 in the revised paper.
[W2: stability + Q2] Repeated evaluation and Refiner decisions
Following your suggestion, we report mean ± sample standard deviation over three evaluations. MAS-Loop achieves the highest overall average under both O-only and O+R deployment. Refinement further improves MAS-Loop and all three external orchestrators. These results align with our original findings. We have incorporated the complete repeated-evaluation results into Section 4 in the revised paper.
We also report the requested HotpotQA behavior rates. All rates use the complete HotpotQA test set, and Edit denotes valid non-Accept edits. Larger improvement proportions (Imporve - Regress) on external orchestrators agree with their larger average refinement gains. This supports R's independent utility and is consistent with a stronger co-trained O leaving less room for refinement. We have incorporated the repeated evaluations into Section 4 and the uncertainty and behavior analyses into Appendices D.3.2–D.3.4 in the revised paper.
[W2: costs + Q3] Deployment costs and verification-only accounting
Methods differ in their upfront optimization, from validation-set workflow search to GPU-based policy training. In practical deployment, these upfront costs can be amortized over repeated deployments, whereas inference costs recur with each use. We therefore focus on inference as a common basis for comparing deployment costs. Following your suggestion, we add HotpotQA inference costs analysis for the strongest overall baseline AOrchestra and each MAS-Loop stage. Source execution includes O construction and execution, diagnosis includes R diagnosis/editing, and re-execution runs the updated system. We have added this breakdown to Appendix D.4.1 in the revised paper.
We also compare against AOrchestra under comparable inference token budgets, using three-sample majority voting to increase its token budget. MAS-Loop exceeds the ensemble by 8.00 percentage points while using fewer tokens, indicating that its advantage beyond increased token budget. We have added this result to Appendix D.4.2 in the revised paper.
The best-of-16 experiment already reports verification-only tokens. All judges use the same 16 MAS candidates, so shared generation and execution costs are excluded and the Token (M) in Table 2 is actually the verification-only tokens, as specified in Section 5.2.