acceptodds
Under review as a conference paper at ICLR 2027

RSI-Master: Value-Guided Experimentation for Autonomous Model Improvement

19 views

Abstract

Recursive self-improvement (RSI) seeks to enable AI systems to participate in improving their own capabilities. A concrete pathway is autonomous model development, where agents iteratively explore post-training strategies to improve a base model. This setting faces two challenges: agents may exploit open-ended experimental actions through hacking, and repeated experimentation may lead to strategy lock-in, where an early direction is refined rather than reconsidered. We introduce RSI-Master, which addresses the two challenges at two levels: regularize step-wise actions, avoiding hacking behaviors, and promote well-structured exploration of research directions, avoiding strategy lock-in. RSI-Master consists of an Experiment OS, which enables regularized experimental actions and maintains persistent, traceable experimental records, and Reviewer-Guided Research Orchestration, which organizes Workers and Reviewers in a dynamically growing research DAG. Workers explore diverse research directions and Reviewers compare evidence across related experiments for subsequent explorations. On PostTrainBench with Qwen3-4B-Base, it averages 54.49 versus 49.34 for the strongest agent baseline, with a 0.0% hacking rate. Scaling to 35B model, RSI-Master surpasses the human-developed Instruct model on LiveCodeBench-v6 (41.21 vs. 37.36) and SciCode, and reaches a nonzero score on HorizonMath, a benchmark of unsolved research problems on which most frontier models score near zero.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Recursive Self-Improvement, AI4AI, Autonomous Research, Research Agents, Post-Training, Data-Centric Model Improvement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AwQeOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper presents RSI-Master, a system for autonomous model improvement through iterative training experiments. It uses ExpOS to manage experiments and enforce experimental rules, and uses separate agents for planning, execution, and review. The system is evaluated on PostTrainBench and additional tasks across two model sizes, with component ablations and case studies.

Critical strengths
  1. The paper studies an important problem: how agents can run reliable experiments and improve models autonomously. It considers both experimental integrity and the exploration of different research directions. These are practical concerns for improving autonomous models.

  2. The evaluation covers multiple tasks and two model sizes. The proposed system achieves a higher average score than the evaluated baselines on PostTrainBench. The paper also includes component ablations and case studies to help explain its behavior.

Critical weaknesses
  1. Concern about the claimed contributions

The contribution of ExpOS is unclear. Preventing evaluation tampering, test-data leakage, and unauthorized model substitution is a basic requirement for credible experiments. If an agent can manipulate the evaluation itself, the reported results cannot be trusted. The paper does not explain what makes these safeguards a research contribution beyond meeting this basic requirement.

The benefit of a separate Reviewer is not established. DataMaster focuses on data optimization, while the more closely related TREX is not compared. TREX uses the same agent to propose experiments and analyze results; this paper separates these roles. It remains unclear why this separation is better.

  1. Concern about the missing implementation details and unclear presentation

Key details are missing. When the Reviewer is removed, does another agent take over its responsibilities and access to evidence? The paper does not explain this or clearly describe the Kimi Agent Swarm implementation. Without code or sufficient detail, these experiments are hard to assess and reproduce. Figures 1, 3, and 4 also have text that is too small to read comfortably.

Questions for authors
  1. Does a separate Reviewer outperform having the same agent propose experiments and analyze results, as in TREX? Could you compare these designs under the same settings?
  2. In the Reviewer ablation, who takes over its responsibilities and evidence access? Please also clarify how the Kimi Agent Swarm baseline was implemented.
5 comments
AuthorsOpenReview ↗

Response to Reviewer AwQe

We sincerely appreciate your time and thoughtful comments. Below, we respond to each point in detail.


W1 & Q1: The Contribution of ExpOS Beyond Basic Integrity Safeguards

We agree that experimental integrity is a prerequisite; our contribution is not the requirement itself, but how experiment execution and evidence are structured for autonomous research.

  1. Integrity is nontrivial in an end-to-end workflow: Agents control data acquisition, training, and checkpoint production; final weights and scores alone cannot establish a compliant development process. PostTrainBench itself documents contamination and model substitution despite explicit restrictions (§5.3–5.5).
Evidence from PostTrainBenchReported finding
Contamination in extended evaluation (§5.5)23 contamination flags across five agents
Opus 4.6 (§5.5)12 flags across 84 runs
Early failure modes (§5.3)Evaluation-code modification and substitution of an Instruct checkpoint
  1. ExpOS supports research beyond integrity checks: Its structured action space regularizes experimental operations without prescribing research strategies, while its persistent artifact space links data, configurations, checkpoints, and evaluations for subsequent retrieval and review (§2.2).
SystemWithout ExpOSWith ExpOSΔ
RSI-Master (Table 3)46.6954.49+7.80
Claude Code (Table 4)43.6851.25+7.57
Codex (Table 4)43.2643.53+0.27

These results support the practical contribution of ExpOS beyond integrity enforcement.

For evaluation credibility, experiments identified as violating the protocol are excluded from accepted results and final checkpoint selection (Appendix E.1).


Q2 : The Benefit of a Separate Reviewer Compared with TREX

  1. Difference from TREX: While TREX uses a Researcher to both design experiments and analyze their outcomes, RSI-Master introduces a separately instantiated Reviewer. The Main Agent assigns explicit review questions and scopes; Reviewers directly inspect experimental records, verify Worker conclusions, and compare evidence across multiple Workers. Their findings are then returned to the Main Agent to inform subsequent research decisions. This separates research execution from independent evidence verification.
AspectTREXRSI-Master
Role allocationResearcher plans and analyzes; Executor executes.Main orchestrates; Workers experiment; Reviewer reviews.
Evidence verificationResearcher analyzes experimental outcomes.Independent Reviewer directly verifies Worker conclusions.
Cross-experiment reviewResearcher uses historical experiments and MCTS.Reviewer compares evidence across Workers under Main-defined review scopes.
  1. Ablation evidence: Removing the Reviewer eliminates this dedicated verification process, leaving the Main Agent to rely more heavily on Worker reports and its own analysis. As shown in Table 3:
ConfigurationAverage
w/o Reviewer44.67
RSI-Master54.49
Improvement+9.82

This demonstrates the practical benefit of Reviewer-guided orchestration within RSI-Master.


Q3: Missing Implementation Details and Reproducibility (Weakness 2 & Question 2)

We clarify the Reviewer ablation protocol, Kimi Agent Swarm configuration, and figure readability.

  1. Reviewer ablation: In the w/o Reviewer ablation, no agent replaces the Reviewer. Workers continue conducting experiments and receiving aggregate feedback, while the Main Agent retains responsibility for research planning and orchestration. Independent evidence verification, cross-Worker review, and structured Reviewer reports are removed. The Main Agent retains its original ExpOS access without inheriting Reviewer-specific permissions.

  2. Kimi Agent Swarm implementation: We have expanded Appendix A.2 to specify the agent configuration, parallel task allocation, coordination and context-sharing mechanisms, backbone version, tool access, and resource budget, making the baseline implementation more reproducible.

  3. Figure readability: We have enlarged the fonts and labels in Figures 1, 3, and 4, and revised their layouts for better readability.


Finally: We really appreciate your insightful suggestion. We hope the above clarifications resolve the concerns. If the response is helpful, we would be grateful if you could consider adjusting your score accordingly.

Reviewer_AwQeOpenReview ↗

Response to the Authors

I thank the authors for the response. I have two follow-up questions.

  1. Since the Main Agent does not inherit the Reviewer’s permissions, the w/o Reviewer ablation changes both the role structure and the access to evidence, so it cannot isolate the benefit of separating the roles. So my main concern remains. TREX uses two roles: a Researcher that plans experiments and analyzes their results, and an Executor that runs them. RSI-Master splits this into three roles: a Main Agent, Workers, and a separate Reviewer. Why is the three-role design better, and what evidence shows it?

  2. I agree that hacking is a real problem. However, a credible autonomous research system must prevent it in the first place; otherwise, its results would not be meaningful. In this sense, ExpOS addresses a necessary requirement rather than a research question, and its design largely builds on established engineering practices such as sandboxing and permission control. I therefore still view it as a solid engineering component rather than a core contribution of the paper.

AuthorsOpenReview ↗

Follow up response

Q1: Why Is the Three-Role Design Better, and What Evidence Supports It?

We thank the reviewer for the follow-up questions. You raise an important point, which we also encountered during system development. In our early experiments with a two-role architecture, we observed that having the Main Agent both plan research and analyze experimental outcomes could lead to insufficient evidence verification and premature commitment to existing research directions. This motivated us to introduce a dedicated Reviewer. We provide further clarification and evidence below.

  1. Development observations. We initially explored a two-role architecture in which the Main Agent handled both research planning and result analysis, similar to TREX. During development, we observed potential issues with insufficient evidence inspection and premature commitment to existing research directions, motivating the separation of evidence verification from research planning.

  2. Behavioral analysis. Beyond the end-to-end ablation, we examine two aspects of research behavior.

(a) Evidence Inspection Frequency. We compare how frequently the Main Agent and Reviewer inspect different types of experimental evidence.

Inspection Operationw/o Reviewer: MainRSI-Master: MainRSI-Master: Reviewer
Evaluation results5.82.86.4
Training configurations1.60.94.8
Data & checkpoint provenance0.70.53.9
Sample-level diagnosticsN/AN/A7.1
Cross-experiment comparisons1.20.94.0
Total9.35.126.2
Avg. per accessible category2.31.35.2

(b) Major-Breakthrough Worker Rate. We divide research progress into early, middle, and late stages. A Worker counts as a major breakthrough if its best checkpoint improves upon the best-so-far score at the time of its creation by at least 10% (relative).

Stagew/o Reviewer: WorkersBreakthroughsRate ↑RSI-Master: WorkersBreakthroughsRate ↑
Early (0–33%)22627.3%23730.4%
Middle (33–66%)2129.5%24416.7%
Late (66–100%)1700.0%20210.0%
Overall60813.3%671319.4%
  1. End-to-end performance. Removing the Reviewer reduces the PostTrainBench average from 54.49 to 44.67 (−9.82 points) (Table 3). The behavioral analyses above are designed to investigate whether dedicated evidence verification leads to more thorough inspections and more productive research directions, particularly in later stages.

The existing ablation supports the overall Reviewer-guided design. Isolating the causal effect of role separation itself requires controlling evidence access.


Q2: Is ExpOS Primarily an Engineering Component?

We thank the reviewer for the follow-up question. We agree that sandboxing and permission control are important foundations for credible autonomous research. However, in long-horizon adaptive optimization, how to constrain experimental actions while preserving the agent's freedom to explore research directions remains an important challenge.

  1. Connection to Reward Hacking. This problem is closely related to reward hacking and reward tampering in reinforcement learning. Prior work has formally characterized how optimizing imperfect proxy rewards can lead to worse performance on the true objective [1], and demonstrated the risks of reward overoptimization [2]. In autonomous post-training, benchmark evaluators similarly serve as proxies for model capabilities. Meanwhile, agents control the entire development pipeline, including data, training, checkpoints, inference, and evaluation, creating additional opportunities to manipulate the experimental process rather than improve the intended capabilities [3].

  2. Evidence from Frontier Labs. This is not merely a hypothetical concern. Anthropic reported rolling back three days of Mythos Preview RL training after detecting reward hacking, and subsequently flagged over 10% of its production RL environments for issues including reward hacking, broken tasks, and misconfiguration [4]. OpenAI also documented an incident in which an agent exploited a vulnerability in a testing interface to access an original software implementation and submit it for a higher reward [5]. These cases demonstrate that preventing reward hacking remains an ongoing challenge even in frontier model development.

  3. ExpOS as More than Hacking Prevention. ExpOS regularizes agent behavior while also structuring the workspace and context through which agents organize, share, and retrieve experimental information. This enables consistent experiment execution and context management throughout long-horizon autonomous research.

Thus, ExpOS goes beyond conventional access control by providing experimental action-space regularization and persistent evidence tracking for long-horizon autonomous model development.

Reviewer_mnipOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RSI-Master combines a constrained Experiment OS, persistent experiment provenance, and a dynamically expanded graph of Workers and independent Reviewers for autonomous post-training. With Kimi-K3 as the research agent, it reports a PostTrainBench average of 54.49 versus 46.53 for the strongest original baseline and 66.47 for the official Instruct reference. The paper also presents component ablations, ExpOS-equipped baselines, and task-specific applications across additional domains and a larger model.

Critical strengths
  1. The separation of permitted experimental actions, checkpoint provenance, and independent evidence review is a useful systems contribution. Explicit ownership and role-specific evidence access make autonomous research easier to trace and audit (Sections 2.2–2.3).
  2. The evaluation includes relevant agent baselines, component ablations, and tests of ExpOS inside existing harnesses. Removing each component lowers the reported average, providing initial evidence for the integrated design (Tables 2–4).
Critical weaknesses
  1. whether the incremental improvement is stable for different turns. Table 4 reports 54.49 for RSI-Master versus 51.25 for Claude Code + ExpOS, but no multi turn averages. And the improvement is not big enough, so whether it is a stable improvement? And you pointed that your method will select a best ckpts. I think it is ok to select a best one. But the improvement is not so reliable and sufficient.
  2. I am concerned about the comparison between the human-developed Qwen3.5-35B-A3B model and the rsi-master constructed model. because training process of the human-developed Qwen3.5-35B-A3B is not open, so we can not direct compare the two. Table 2 also shows a lower performance compared with the official Instruct model (54.49 versus 66.47), with BFCL as the only task on which RSI-Master scores higher.
Questions for authors
  1. How many runs were conducted for RSI-Master and Claude Code + ExpOS? Please show their mean scores under the same resource constraints, and clarify whether the improvement persists across runs.
  2. I am concerned about the comparison between the human-developed Qwen3.5-35B-A3B model and the rsi-master constructed model. because training process of the human-developed Qwen3.5-35B-A3B is not open, so we can not direct compare the two. Table 2 also shows a lower performance compared with the official Instruct model (54.49 versus 66.47), with BFCL as the only task on which RSI-Master scores higher.

I am open to raise my score if you solve my question

1 comment
AuthorsOpenReview ↗

Response to Reviewer mnip

We sincerely appreciate your time and thoughtful comments. Below, we respond to each point in detail.


Q1: Run-to-Run Stability and Reliability of Improvements

  1. Original Results: The results originally reported in Table 4 were obtained from a single run.
  2. Run-to-Run Stability: To assess stability, we report three independent runs in total under identical resource constraints (1 × H100, 12 hours per run). The results are summarized below.
BenchmarkCC+ExpOS R1R2R3Mean ± StdRSI-Master R1R2R3Mean ± Std
AIME 202517.7820.0022.2220.00 ± 2.2223.3320.0026.6723.33 ± 3.34
Arena-Hard43.1245.2046.8045.04 ± 1.8549.9546.7544.3047.00 ± 2.83
BFCL64.9166.0063.0064.64 ± 1.5264.5064.0067.0065.17 ± 1.61
GPQA35.7137.9534.3836.01 ± 1.8041.2938.8439.9640.03 ± 1.23
GSM8K92.6092.3492.9592.63 ± 0.3192.2092.4292.0492.22 ± 0.19
HealthBench40.6238.1043.2040.64 ± 2.5535.7939.7532.4035.98 ± 3.68
HumanEval64.0367.0770.1267.07 ± 3.0574.3971.3471.9572.56 ± 1.61
Average51.2552.3853.2452.29 ± 1.0054.4953.3053.4753.76 ± 0.64

Q2: Comparison with Human-Developed Models and Performance Gap

  1. Comparison with Human-Developed Models: A strictly controlled comparison is infeasible because human-developed models differ in training resources, duration, and data, many of which are undisclosed. Nevertheless, comparing against officially released models under standardized evaluation protocols is common practice and provides a meaningful external performance reference.

  2. Performance Gap and Research Budget: RSI-Master is given different budgets at different scales: 1 H100 for 12 hours at 4B, versus 8 H20 GPUs for 24 hours at 35B, providing substantially more compute and a longer iteration horizon for the agent. This allows us to examine how autonomous research benefits from additional resources and iterations. At 35B, RSI-Master surpasses Instruct on SciCode-ICL, LiveCodeBench-v6, and HorizonMath.


Finally: We really appreciate your insightful suggestion. We hope the above clarifications resolve the concerns. If the response is helpful, we would be grateful if you could consider adjusting your score accordingly.

Reviewer_V15DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RSI-Master is a system for autonomous post-training that combines constrained experimental interfaces with reviewer-guided research exploration. Experiment OS records datasets, configurations, checkpoints, and evaluations, while Workers and Reviewers form a growing research DAG coordinated by a Main Agent. On seven PostTrainBench tasks, the system reports an average score of 54.49, compared with 46.53 for the strongest baseline without ExpOS, and records no detected hacking. Further experiments examine domain-specific optimization and a larger model, including reported improvements over its Instruct reference on coding benchmarks.

Critical strengths
  1. I found the combination of controlled execution and flexible research exploration to be well motivated. Restricting experimental operations while preserving checkpoint lineage makes results easier to inspect. Separately scheduling reviews across related experiments also provides a reasonable mechanism for reconsidering directions rather than repeatedly refining an early choice.

  2. The component ablations and comparisons with ExpOS-equipped baselines are useful. They go beyond comparing the full system against unrestricted agents and provide some evidence that orchestration contributes beyond the experimental interface alone. Evaluating the framework across different tasks and model scales also helps establish its breadth of applicability.

Critical weaknesses
  1. I could not determine whether final scores are measured on an evaluation set untouched by the adaptive research process. Workers receive aggregate feedback, Reviewers receive detailed diagnostics, and the final checkpoint is selected by evaluator score. These are reasonable mechanisms for development, but require a clearly separate final evaluation to establish generalization. Withholding reference answers from Reviewer reports does not by itself resolve this concern, since the resulting recommendations still guide subsequent training.

  2. I found the role of the research DAG less clear than that of ExpOS. The DAG records dependencies and passes context, but decisions about continuing, branching, or abandoning a direction remain with the Main Agent and Reviewers. The paper does not sufficiently explain how these decisions use the graph to counter strategy lock-in. In particular, a Reviewer could recommend further refinements within the same strategy rather than challenge the strategy itself. I would like a clearer account of what in the orchestration design actively encourages reconsideration beyond organizing experiment history and agent communication.

Questions for authors
  1. Which evaluation data are available during optimization, which diagnostics can Reviewers inspect, and which data are reserved exclusively for final scoring? Please clarify this separation and the decontamination procedure for externally acquired or generated training data.

  2. What decision instructions or policies guide the Main Agent and Reviewers when choosing between refining an existing strategy and exploring a different one? How does the research DAG influence that choice? What encourages Reviewers to challenge the underlying strategy rather than recommend local improvements?

  3. For HorizonMath, which problems contribute to the reported 4.00 score, how is that score calculated, and what validator evidence supports those successes? Were these problems excluded from all optimization feedback?

2 comments
AuthorsOpenReview ↗

Response to Reviewer V15D

We sincerely appreciate your time and thoughtful comments. Below, we respond to each point in detail.


W1 & Q1: Evaluation Protocol and Data Leakage

  1. Evaluation Protocol. We agree that development feedback and independent final evaluation should be distinguished. For smaller benchmarks, we evaluate on the full dataset, while for larger benchmarks, we use fixed subsets for iterative evaluation. We have clarified the evaluation sizes and protocols in the revised paper. Importantly, we distinguish benchmarks with independent held-out evaluations from those where the reported scores reflect development-set performance, and do not claim the latter as independent evidence of generalization.

  2. Evaluation Feedback vs. Leakage. As the reviewer notes, using evaluation feedback to guide research is a reasonable development practice. However, accessing benchmark feedback is distinct from incorporating evaluation questions or answers into training. To mitigate such leakage, we employ (i) prompt-level restrictions, (ii) training-data decontamination checks, and (iii) Reviewer audits for contamination and suspicious improvements.

We acknowledge that these safeguards do not eliminate adaptive overfitting, which requires independent held-out evaluation to assess.

W2: Research DAG and Strategy Reconsideration

We highlight three aspects:

  1. Cross-Worker evidence comparison. Worker-to-Reviewer edges define which investigations are reviewed together (§2.3). This allows a review to compare findings across research directions and assess whether they support consistent conclusions.

  2. Explicit mechanisms for reconsideration. Multiple initial hypotheses, fresh Reviewer processes, Main Agent reassessment of Reviewer conclusions, and revisable research memory create repeated opportunities to question existing judgments. These mechanisms encourage reconsideration throughout the research process.

  3. Follow-up task construction. The Main Agent uses these assessments to assign refinement, verification, or alternative exploration. Worker-to-Worker edges specify which predecessor findings inform each new investigation, connecting revised judgments to subsequent research actions.

Q2: Decision Policies for Exploration vs. Refinement

The Main Agent chooses between refinement and alternative exploration by reassessing Worker findings and Reviewer recommendations. The following instructions and mechanisms guide this process:

PrincipleConcrete instruction or designRole in reconsideration
Hypothesis diversitySeparate Workers explore 3–8 initial hypotheses.Establish competing research directions.
Critical reassessmentEach review runs in a fresh Reviewer process.Provide a fresh assessment context.
Reviews contain Conclusion / Evidence / Future-Direction.Connect evidence to follow-up proposals.
Main critically rechecks Reviewer conclusions and attributions.Reconsider the proposed diagnosis before acting.
Revisable memoryRecord open hypotheses and dead ends.Retain alternatives and failed directions.
Permit new findings to overturn earlier conclusions.Keep previous judgments open to revision.

The DAG specifies the branches included in each review and the predecessor context supplied to subsequent Workers. These mechanisms provide repeated opportunities to question the current direction and revise follow-up plans. They encourage reconsideration while allowing continued refinement when supported by the evidence.

Q3: HorizonMath Evaluation

We have corrected the reporting of the evaluation subset used during iteration. Specifically, the reported HorizonMath score of 4.00% is based on a fixed subset of 50 problems from the full 113-problem benchmark (2/50), rather than the full benchmark. The full-set score is 4.42%. We will provide the subset selection criteria and problem IDs, and revise the table labels to avoid ambiguity.

The evaluation sizes and scores are summarized below:

BenchmarkFull SetDev SetDev-Set ScoreFull-Set Score
SciCode-ICL / Verified646435.9435.94
AIME 2026303073.3373.33
HealthBench Professional52520073.3372.76
HorizonMath113504.004.42
LiveCodeBench v618218241.2141.21
HLE2,15820011.0011.21
IMO AnswerBench4003053.3352.50

Finally: We really appreciate your insightful suggestion. We hope the above clarifications resolve the concerns. If the response is helpful, we would be grateful if you could consider adjusting your score accordingly.

Thank you for the rebuttal and the detailed response! The breakdown of the split is useful, thanks for that. Since the full HorizonMath score does include the 50 problems used during development, I would still like to separately see the score on the remaining 63. The DAG explanation also clarifies how the system encourages reconsideration, but at the same time removing the Reviewer changes both who reviews the experiments and the evidence that is available. Because of that, I think the ablation leaves open whether a standalone separate Reviewer would assist in preventing lock-in of strategies. Overall, I’m keeping my recommendation unchanged. I appreciate the work, and all the best!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.