- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces two new mechanisms to improve scientific discovery of LLM: Simple Loop and LoopRL. Simple loop builds a collaborative working environment for multiple agents with the same policy. These agents share the same memory, but can propose different improvements. Different agents proceed independently. Approaches like reflection and knowledge consolidation are removed to grant agents more freedom for decision. LoopRL is a reinforcement learning algorithm with a carefully designed reward function. This reward function encourages the improvement relative to the current best, and also gives credit to the experience which insights the improvement. The paper claims that these two mechanisms can improve the diversity of exploration, and thus make the quality of discoveries higher.
Critical strengths
- Details. This paper provides a lot of details of how the proposed approaches work on different problems, and the observations on how the improvements are generated through the collaboration between multiple agents.
- Width of benchmarks. This work tests the performance of the proposed approaches on a wide range of test cases.
Critical weaknesses
- Clarity. The reviewer finds the paper really hard to follow and understand. For example, in section 4.2, the authors analyze how the improvements are generated by the collaboration between agents in a very lengthy manner. However, this section is named results, and is supposed to show some quantitative results.
- Lack of comparisons with other approaches. Despite a lengthy discussion of the improvements introduced by multiple agents, this work has no mention of what other approaches can achieve. Although in lines 251-252 the authors promise to compare with CORAL, later this comparison is missing. Without a fair benchmark with other approaches, one cannot assess the performance of the proposed methods.
- Contribution of reinforcement learning is not discussed. There are two approaches introduced: Simple Loop and LoopRL. Simple Loop introduces a collaborative environment, and seems to work without reinforcement learning. In the "results" section, it seems the contribution from multiple agents is sound, but RL is not essential. An ablation study with just Simple Loop and without LoopRL would help figure out how important the RL is in the proposed combination of methods.
Questions for authors
See critical weaknesses section. Overall, this work is not clearly presented, and not well-organized. The contributions cannot be assessed based on its current format.
2 comments
Response to Reviewer ZRRL [2/n]
Weakness 3: Contribution of reinforcement learning
Thank you for your question. Regarding the impact of reinforcement learning on scientific discovery, Section 4.3 discusses the effectiveness of training, showing that LoopRL improves communication across branches, cooperation strategies, and search patterns.
In Section 4.4 and Figure 5, we analyze how different credit assignment strategies affect RL training. We find that LoopRL’s credit assignment strategy outperforms that used in conventional multi-turn RL, achieving faster search, measured by the number of iterations, and higher final performance under the same experimental settings.
Additionally, our responses to Weaknesses 1 and 2 provide detailed comparisons between SimpleLoop and LoopRL on ten common discovery tasks and two challenging stress-test tasks. LoopRL consistently outperforms SimpleLoop, quantitatively demonstrating the effectiveness of RL training in enabling the model to autonomously learn search strategies for scientific discovery.
We sincerely appreciate your time in reviewing our paper and your valuable suggestions for improving it. We hope that our detailed responses address your concerns. If you have any remaining concerns, we will promptly provide further details. Thank you.
Response to Reviewer ZRRL [1/n]
We sincerely thank you for your careful and thorough review. We appreciate your recognition of our work, particularly the detailed explanations and the breadth of our benchmark evaluation. Below, we address your concerns point by point.
Weakness 1: Clarity and quantitative results
Thank you for pointing this out. Following the presentation style of AlphaEvolve [1], ShinkaEvolve [2], and CORAL [3], we provided detailed accounts of the search process and its progression. Section 4.2 was intended to illustrate how communication among agents can substantially improve discovery and to analyze how specific communication patterns affect discovery outcomes.
To complement this analysis with quantitative evidence, we provide results across all ten benchmarks below. Under the same resource budget, SimpleLoop outperforms CORAL on every task, while LoopRL further improves performance by enabling the model to learn search strategies specific to each problem.
[1] AlphaEvolve: A coding agent for scientific and algorithmic discovery
[2] ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution
[3] CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discovery
Weakness 2: Comparison with CORAL
In addition to the comparisons with CORAL presented in our response to Weakness 1, we provide further results on the two challenging benchmarks used for stress testing in CORAL: Anthropic Kernel Engineering and Frontier-CS Polyominoes Packing. All methods are evaluated under the same benchmark settings, computational resources, and wall-clock time limits.
With the same underlying model, SimpleLoop achieves better final performance and faster search than CORAL. With the smaller open-source model, Qwen 3.8 27B, SimpleLoop also clearly outperforms CORAL, while LoopRL further improves performance.