- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DataOS, a four-layer framework that organizes experimental artifacts and execution histories to guide revisions to training data, target models, and agent harnesses. It reports improvements over native coding agents on PostTrainBench and Curation-Bench, with ablations and transfer experiments intended to support the value of structured history and harness evolution.
Critical strengths
The evaluation covers both open-ended post-training and fixed-recipe data curation. The component ablations also provide some evidence that structured history and harness evolution contribute beyond the underlying coding agents.
Critical weaknesses
- The RSI claim is stronger than the demonstrated feedback loop. GPT-5.6 Sol or Claude Opus 5 serves as the researcher while smaller, separate target models are trained. The paper demonstrates an agent improving target models and parts of its harness, but does not show whether an improved target model can itself become the researcher for subsequent improvement. This distinction is central to interpreting the “self-improvement” claim.
- The evolution mechanism and its dynamics are insufficiently evidenced. The appendix defines the three Plan types and provides one joint-optimization trajectory, but does not clearly show representative filled-in plans, actual prompt/operator/pipeline revisions, or comparable round-by-round performance trajectories on both benchmarks. Consequently, it is difficult to assess what evolved in practice and when the reported gains arose.
Questions for authors
- Could you provide a concrete executed example of each Plan type, including its specifications, version changes, and resulting actions? For harness evolution, which of the four predefined role prompts, operators, or pipelines were actually changed, and why was this initial role decomposition chosen?
- What training interventions were permitted and actually attempted on PostTrainBench? Please distinguish these from Curation-Bench, where the training recipes are fixed and only data subsets are selected.
- Can an improved target model replace the frontier researcher in a subsequent iteration, and does this yield further improvement? If not, how do the authors delimit the paper’s RSI claim?
- Could the authors report per-iteration candidate and best-so-far scores, alongside harness-revision events, for both benchmarks? The existing joint-optimization table does not provide a time-aligned comparison, and Curation-Bench lacks an analogous trajectory.
3 comments
Official Response to Reviewer ogCo (2)
Q4. Per-iteration performance trajectories
We have added Figures 2 and 3 in Appendix D.6, presenting PTB trajectories aligned by elapsed time and per-iteration Curation-Bench trajectories, both with candidate and best-so-far scores. These trajectories illustrate our view of improvement: sustained exploration guided by experimental feedback, where candidate performance may fluctuate or regress while better solutions are discovered and retained. In Curation-Bench, the model is reset and the training configuration fixed at each iteration, yet later iterations still identify better data subsets. This illustrates improvement in the research process through feedback, rather than continuous accumulation of model-weight updates.
I appreciate the authors’ response and their careful answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.
Official Response to Reviewer ogCo (1)
We are grateful to the reviewer for the thorough review and helpful suggestions for improving the manuscript.
Q1. RSI scope and researcher replacement
We agree that target-to-researcher replacement is a key question for RSI. This is particularly challenging because frontier researcher models benefit from large-scale training and partly undisclosed data, while comparable compute and data resources are difficult to obtain for target training. RSIBench-Data [1] reports a related difficulty: its exploratory same-family researcher–target experiment failed to improve upon the unadapted target.
Our scope is environment-data-driven, system-level RSI with a fixed researcher backbone. DataOS uses accumulated experimental evidence to revise data, target models, and the research harness. Adopted prompt, operator, and pipeline revisions are reused in later iterations, changing the procedures that produce further improvements. Our PTB and Curation-Bench targets are trained and evaluated on downstream tasks; we have not tested researcher replacement. Reviewer kCXr also recognizes persistent environment-data management as a unifying perspective and the paper’s most important contribution.
These mechanisms provide a foundation for future researcher replacement. A successor researcher still needs to understand prior experiments, reuse effective artifacts and procedures, and propose and validate further changes. DataOS preserves this experience across iterations and system versions by maintaining evidence, provenance, and dependencies. We will explore turning accumulated research trajectories, experimental feedback, and reusable strategies into resources for training and evaluating researcher capabilities, then test whether improved models can take over the researcher role and continue producing improvements on bounded research tasks with controlled budgets.
References
[1] Fanqing Meng, Lingxiao Du, Qiguang Chen, Ziqi Zhao, Haocheng Lu, Mengkang Hu, and Michael Qizhe Shieh. RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement. arXiv:2607.25886, 2026.
Q2. Executed Plan instances and concrete component revisions
We decompose the system by decision objective, evidence scope, and action space—connected through versioned Plans and shared environment data—to prevent low-level execution and debugging traces from polluting high-level research decisions and to preserve cross-iteration state. The Leader plans experiments under task objectives and budgets (Task Plan); when cross-iteration evidence reveals procedural bottlenecks, the Evolver diagnoses limitations and specifies system revisions and validation criteria (Evolution Plan); the Worker handles implementation and execution; and the Maintainer commits artifact provenance and component versions (Maintenance Plan) for downstream reuse.
In the documented example, the Leader provides experimental feedback and improvement objectives; the Evolver analyzes linked records and specifies revisions and validation; the Worker adds and registers ref_loss_margin_selector and integrates it into the Task Plan's data.recipe, enabling checkpoint-conditioned preference-data selection; and the Maintainer preserves component versions and dependencies for subsequent reuse.
We have clarified the rationale for this initial role decomposition in Sections 3.1–3.5 of the revised manuscript, and added executed instances of the Task, Evolution, and Maintenance Plans in Appendix D.1–D.3 (including specifications, version relationships, executed actions, and outcomes). Appendix D.4 provides role handoffs and an index of supporting records, while Appendix D.5 details the operator and pipeline changes and their subsequent reuse.
Q3. Training interventions on PTB
On PTB, DataOS supports full-parameter SFT, LoRA SFT, full-parameter DPO, and GRPO, together with hyperparameter tuning and checkpoint reuse. It autonomously selects and executes interventions based on task requirements and experimental feedback, within fixed data, compute, and evaluation constraints.
We analyzed its actual training-method choices and obtained the following distribution:
DataOS predominantly selects SFT in standard runs (95%), while DPO accounts for 50% of its choices in scaling.
We document the supported interventions, observed choices, and distinction from Curation-Bench’s fixed training recipes in Appendix B.1.4 and Table 6 of the revised manuscript.