- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper presents RSI-Master, a system for autonomous model improvement through iterative training experiments. It uses ExpOS to manage experiments and enforce experimental rules, and uses separate agents for planning, execution, and review. The system is evaluated on PostTrainBench and additional tasks across two model sizes, with component ablations and case studies.
Critical strengths
-
The paper studies an important problem: how agents can run reliable experiments and improve models autonomously. It considers both experimental integrity and the exploration of different research directions. These are practical concerns for improving autonomous models.
-
The evaluation covers multiple tasks and two model sizes. The proposed system achieves a higher average score than the evaluated baselines on PostTrainBench. The paper also includes component ablations and case studies to help explain its behavior.
Critical weaknesses
- Concern about the claimed contributions
The contribution of ExpOS is unclear. Preventing evaluation tampering, test-data leakage, and unauthorized model substitution is a basic requirement for credible experiments. If an agent can manipulate the evaluation itself, the reported results cannot be trusted. The paper does not explain what makes these safeguards a research contribution beyond meeting this basic requirement.
The benefit of a separate Reviewer is not established. DataMaster focuses on data optimization, while the more closely related TREX is not compared. TREX uses the same agent to propose experiments and analyze results; this paper separates these roles. It remains unclear why this separation is better.
- Concern about the missing implementation details and unclear presentation
Key details are missing. When the Reviewer is removed, does another agent take over its responsibilities and access to evidence? The paper does not explain this or clearly describe the Kimi Agent Swarm implementation. Without code or sufficient detail, these experiments are hard to assess and reproduce. Figures 1, 3, and 4 also have text that is too small to read comfortably.
Questions for authors
- Does a separate Reviewer outperform having the same agent propose experiments and analyze results, as in TREX? Could you compare these designs under the same settings?
- In the Reviewer ablation, who takes over its responsibilities and evidence access? Please also clarify how the Kimi Agent Swarm baseline was implemented.
5 comments
Response to the Authors
I thank the authors for the response. I have two follow-up questions.
-
Since the Main Agent does not inherit the Reviewer’s permissions, the w/o Reviewer ablation changes both the role structure and the access to evidence, so it cannot isolate the benefit of separating the roles. So my main concern remains. TREX uses two roles: a Researcher that plans experiments and analyzes their results, and an Executor that runs them. RSI-Master splits this into three roles: a Main Agent, Workers, and a separate Reviewer. Why is the three-role design better, and what evidence shows it?
-
I agree that hacking is a real problem. However, a credible autonomous research system must prevent it in the first place; otherwise, its results would not be meaningful. In this sense, ExpOS addresses a necessary requirement rather than a research question, and its design largely builds on established engineering practices such as sandboxing and permission control. I therefore still view it as a solid engineering component rather than a core contribution of the paper.
Follow up response
Q1: Why Is the Three-Role Design Better, and What Evidence Supports It?
We thank the reviewer for the follow-up questions. You raise an important point, which we also encountered during system development. In our early experiments with a two-role architecture, we observed that having the Main Agent both plan research and analyze experimental outcomes could lead to insufficient evidence verification and premature commitment to existing research directions. This motivated us to introduce a dedicated Reviewer. We provide further clarification and evidence below.
-
Development observations. We initially explored a two-role architecture in which the Main Agent handled both research planning and result analysis, similar to TREX. During development, we observed potential issues with insufficient evidence inspection and premature commitment to existing research directions, motivating the separation of evidence verification from research planning.
-
Behavioral analysis. Beyond the end-to-end ablation, we examine two aspects of research behavior.
(a) Evidence Inspection Frequency. We compare how frequently the Main Agent and Reviewer inspect different types of experimental evidence.
| Inspection Operation | w/o Reviewer: Main | RSI-Master: Main | RSI-Master: Reviewer |
|---|---|---|---|
| Evaluation results | 5.8 | 2.8 | 6.4 |
| Training configurations | 1.6 | 0.9 | 4.8 |
| Data & checkpoint provenance | 0.7 | 0.5 | 3.9 |
| Sample-level diagnostics | N/A | N/A | 7.1 |
| Cross-experiment comparisons | 1.2 | 0.9 | 4.0 |
| Total | 9.3 | 5.1 | 26.2 |
| Avg. per accessible category | 2.3 | 1.3 | 5.2 |
(b) Major-Breakthrough Worker Rate. We divide research progress into early, middle, and late stages. A Worker counts as a major breakthrough if its best checkpoint improves upon the best-so-far score at the time of its creation by at least 10% (relative).
| Stage | w/o Reviewer: Workers | Breakthroughs | Rate ↑ | RSI-Master: Workers | Breakthroughs | Rate ↑ |
|---|---|---|---|---|---|---|
| Early (0–33%) | 22 | 6 | 27.3% | 23 | 7 | 30.4% |
| Middle (33–66%) | 21 | 2 | 9.5% | 24 | 4 | 16.7% |
| Late (66–100%) | 17 | 0 | 0.0% | 20 | 2 | 10.0% |
| Overall | 60 | 8 | 13.3% | 67 | 13 | 19.4% |
- End-to-end performance. Removing the Reviewer reduces the PostTrainBench average from 54.49 to 44.67 (−9.82 points) (Table 3). The behavioral analyses above are designed to investigate whether dedicated evidence verification leads to more thorough inspections and more productive research directions, particularly in later stages.
The existing ablation supports the overall Reviewer-guided design. Isolating the causal effect of role separation itself requires controlling evidence access.
Q2: Is ExpOS Primarily an Engineering Component?
We thank the reviewer for the follow-up question. We agree that sandboxing and permission control are important foundations for credible autonomous research. However, in long-horizon adaptive optimization, how to constrain experimental actions while preserving the agent's freedom to explore research directions remains an important challenge.
-
Connection to Reward Hacking. This problem is closely related to reward hacking and reward tampering in reinforcement learning. Prior work has formally characterized how optimizing imperfect proxy rewards can lead to worse performance on the true objective [1], and demonstrated the risks of reward overoptimization [2]. In autonomous post-training, benchmark evaluators similarly serve as proxies for model capabilities. Meanwhile, agents control the entire development pipeline, including data, training, checkpoints, inference, and evaluation, creating additional opportunities to manipulate the experimental process rather than improve the intended capabilities [3].
-
Evidence from Frontier Labs. This is not merely a hypothetical concern. Anthropic reported rolling back three days of Mythos Preview RL training after detecting reward hacking, and subsequently flagged over 10% of its production RL environments for issues including reward hacking, broken tasks, and misconfiguration [4]. OpenAI also documented an incident in which an agent exploited a vulnerability in a testing interface to access an original software implementation and submit it for a higher reward [5]. These cases demonstrate that preventing reward hacking remains an ongoing challenge even in frontier model development.
-
ExpOS as More than Hacking Prevention. ExpOS regularizes agent behavior while also structuring the workspace and context through which agents organize, share, and retrieve experimental information. This enables consistent experiment execution and context management throughout long-horizon autonomous research.
Thus, ExpOS goes beyond conventional access control by providing experimental action-space regularization and persistent evidence tracking for long-horizon autonomous model development.
Response to Reviewer AwQe
We sincerely appreciate your time and thoughtful comments. Below, we respond to each point in detail.
W1 & Q1: The Contribution of ExpOS Beyond Basic Integrity Safeguards
We agree that experimental integrity is a prerequisite; our contribution is not the requirement itself, but how experiment execution and evidence are structured for autonomous research.
These results support the practical contribution of ExpOS beyond integrity enforcement.
For evaluation credibility, experiments identified as violating the protocol are excluded from accepted results and final checkpoint selection (Appendix E.1).
Q2 : The Benefit of a Separate Reviewer Compared with TREX
This demonstrates the practical benefit of Reviewer-guided orchestration within RSI-Master.
Q3: Missing Implementation Details and Reproducibility (Weakness 2 & Question 2)
We clarify the Reviewer ablation protocol, Kimi Agent Swarm configuration, and figure readability.
Reviewer ablation: In the w/o Reviewer ablation, no agent replaces the Reviewer. Workers continue conducting experiments and receiving aggregate feedback, while the Main Agent retains responsibility for research planning and orchestration. Independent evidence verification, cross-Worker review, and structured Reviewer reports are removed. The Main Agent retains its original ExpOS access without inheriting Reviewer-specific permissions.
Kimi Agent Swarm implementation: We have expanded Appendix A.2 to specify the agent configuration, parallel task allocation, coordination and context-sharing mechanisms, backbone version, tool access, and resource budget, making the baseline implementation more reproducible.
Figure readability: We have enlarged the fonts and labels in Figures 1, 3, and 4, and revised their layouts for better readability.
Finally: We really appreciate your insightful suggestion. We hope the above clarifications resolve the concerns. If the response is helpful, we would be grateful if you could consider adjusting your score accordingly.