- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark for agents doing data work. The agent is handed a pool of agent trajectories of mixed quality, some of them deliberately broken, and has to decide what to keep, fix, or throw away. Everything downstream is frozen, i.e., same fine-tuning recipe, same evaluation, so the dataset the agent submits is the only variable, and it's judged by how well the resulting model does on tasks it was never shown during the research loop.
Critical strengths
- The error construction is the best part of the paper. Replacing one step and letting the original model re-roll in the original container, so the downstream observations are real consequences rather than a model’s impression of failure, is the right call and the authors explain clearly why the obvious alternative is worse.
- Freezing the recipe and isolating public feedback from the hidden score is clean. I believe the headline number is measuring the dataset and not something else.
Critical weaknesses
-
The RSI framing doesn't hold up. The researcher is GPT-5.6 or Opus or GLM. The model being trained is Qwen3-8B. Different labs, different scales, no connection between them. A big model preparing supervision for a small one is data curation, and in places it's closer to distillation. There's no self anywhere in the loop. To be about self-improvement, the checkpoint from round t would have to come back as the researcher in round t+1, and the paper neither does that nor explains why not. The title and the whole introduction promise something the experiment doesn't deliver.
-
And even ignoring that, nothing can compound. Recipe fixed, hyperparameters fixed, no RL, no access to training code, one base model, one student size, one frozen corpus of verified rollouts to draw from. Everything that would let the next iteration be better in kind rather than degree has been held constant deliberately. What's left is one round of data cleaning.
-
The task has a low ceiling and the paper doesn't confront it. When you train on the clean twin of the same 1,500 instances, you get 25.0. The best agent gets 21.7. So the whole exercise tops out at undoing damage the authors introduced, and the result is that agents undo most of it.
-
I don’t believe the ranking in Figure 5. A hundred hidden instances puts the binomial standard error near 4 points at a 20% resolve rate, before you add run-to-run noise, and the reported standard deviations run from 1 to 5.5 over three runs. GLM at 21.7 and sol (High) at 16.0 are not reliably apart, and eleven configurations are ranked in between them. Give confidence intervals and say which pairs actually separate.
Questions for authors
Why isn't the researcher the same model as the student, and what happens if you close the loop so the trained checkpoint curates the next round? Without that, what makes any of this recursive?
7 comments
Third, in our paper, we explicitly defines RSI motivation in terms of AI helping improve subsequent generations of models, without requiring the researcher and target to be the same model. The abstract states:
Recursive self-improvement (RSI) envisions AI systems that can help improve future generations of models.
The introduction likewise describes models taking over the research loop of designing better training methods, running experiments, evaluating outcomes, and iterating toward the next generation of models. This is the system-level meaning used in our paper: AI takes on research responsibilities that contribute to developing successor models. Under this definition, the successor may have a different architecture, parameter scale, or model family; it need not be the researcher's own checkpoint.
Accordingly, RSI-Wild evaluates the data-research component of this process: whether an agent can diagnose training-data problems, develop and execute data strategies, and use training and evaluation feedback to improve a subsequent model. The main experiments keep the researcher's weights fixed while it iterates on data strategies.
2. (W2, W3) Do the fixed training recipe and initial trajectory pool reduce the task to a single round of data cleaning, with performance capped at the 25.0% achieved by Train on Successful 1500?
Our agents’ data tasks include data synthesis: they can construct new tasks and training trajectories, expanding the scale and coverage of the training data. The 25.0% score for Train on Successful 1500 is therefore a reference obtained by training on correct trajectories for the original 1,500 instances, not a performance ceiling for our task.
SWE-Lego provides a direct example of expanding training supervision through synthetic data. Its Table 3 reports that 9,100 of the 14,110 valid trajectories that fully resolve their tasks originate from synthetic tasks, approximately 64.5%. Using a training set combining real and synthetic data, SWE-Lego achieves 42.2% Pass@1 with Qwen3-8B, matching our model scale, and 52.6% Pass@1 with Qwen3-32B, through SFT alone and before verifier-based parallel test-time scaling. This illustrates how synthesis can expand supervision beyond an initial trajectory pool: using SFT does not restrict performance to the level obtained by restoring existing correct trajectories. SWE-Lego, Table 3 and training-strategy ablations.
| Reference setting | Model size | Data and method | SWE-bench Verified score | What the reference establishes |
|---|---|---|---|---|
| Train on Successful 1500 | 8B | Correct trajectories for the original 1,500 instances; fixed SFT service | 25.0% | How much value can be recovered by replacing failed trajectories in the initial pool with correct versions |
| Our human-researcher reference | 8B | Data research within the same scaffold; 2,500 trajectories in the final training set | 29.0% | Data research within our framework can exceed the Train on Successful 1500 reference |
| SWE-Lego-Qwen3-8B | 8B | Approximately 18k validated trajectories combining real and synthetic data; SFT only | 42.2% Pass@1 | SFT performance at the same initial model scale is not capped at 25.0% |
Our results in this table use the fixed 100-task hidden set. SWE-Lego results are the original paper's SWE-bench Verified evaluations, with different training-data scales, budgets, training procedures, and evaluation conditions. We use them as external examples of headroom for SFT and data design, rather than treating 42.2% or 52.6% as an established attainable limit within our 25-hour budget.
Fixing the recipe and hyperparameters isolates the effects of data-research decisions. Researchers can still use model feedback to identify coverage gaps, synthesize new tasks and trajectories, adjust data mixtures, and repair erroneous supervision, thereby testing data hypotheses that differ from the previous round. A fixed optimizer does not require an unchanged training set or repetition of the same cleaning strategy. Our evaluation asks whether agents can produce better target models through these data decisions under a common training service.
Accordingly, we interpret the results as follows: the strongest current agent has an observed mean of 21.7%, below the 25.0% of Train on Successful 1500 and the human researcher's 29.0%. This reflects the agents' incomplete use of the available data-research space; these three observed values do not establish a theoretical ceiling for the task. We will clarify the role of Train on Successful 1500 as a reference, emphasize the availability of task synthesis and new-trajectory construction, and distinguish feedback-driven data-research iterations from compounding growth through researcher-model upgrades across generations.
3. (W4) With only 100 hidden tasks and three runs per configuration, how reliable is the ranking in Figure 5, and which differences are statistically supported?
Thank you for pointing this out. Figure 5 presents configurations in order of their observed mean scores to support analysis of shortcomings in current agents’ data-research capabilities alongside their research traces. Its purpose is not to establish a definitive model leaderboard. We agree that the current results do not establish a complete, statistically separated ranking of the 13 configurations. To make the uncertainty explicit, we retrieved the final per-task hidden-set reports for all 39 independent research runs, paired outcomes on the same 100 tasks, and accounted for variability across runs. We now provide ordinary 95% confidence intervals for each configuration and all 78 pairwise differences, together with unadjusted approximate two-sided tests. Under the crossed task/run bootstrap, every pairwise difference interval includes zero. We will display means and confidence intervals in the figure and report pairwise differences, aligning the presentation with this analytical purpose.
For the specific comparison raised by the reviewer, GLM-5.3 (Max) scores 23%, 19%, and 23% across its three hidden-set runs, while GPT-5.6-sol (High) scores 15%, 16%, and 17%. Their observed means are 21.67% and 16.00%, a difference of 5.67 percentage points. Resampling complete research runs and shared tasks gives an approximate 95% confidence interval of [0.00, 11.33] percentage points for this difference, with an unadjusted approximate two-sided bootstrap . Thus, GLM has a higher observed mean, while this comparison does not reach the 0.05 significance threshold under this analysis.
Scores and confidence intervals for all configurations
All values below are percentages. Means and standard deviations match the main paper. Confidence intervals use 200,000 crossed task/run bootstrap resamples. Each resample uses the same sampled tasks across configurations and independently samples complete research runs within each configuration, preserving task pairing and dependence within a research run. Run indices are not artificially paired across configurations.
| Configuration | Three hidden-set scores | Mean ± SD | Approximate 95% CI |
|---|---|---|---|
| GLM-5.3 (Max) | 23, 19, 23 | 21.7 ± 2.3 | [14.3, 29.7] |
| Claude Sonnet 5 (High) | 20, 19, 21 | 20.0 ± 1.0 | [13.0, 27.3] |
| GPT-6-astra (High) | 16, 21, 21 | 19.3 ± 2.9 | [12.3, 27.0] |
| GPT-5.6-sol (Medium) | 16, 23, 18 | 19.0 ± 3.6 | [12.0, 27.0] |
| GPT-5.6-luna (High) | 19, 17, 20 | 18.7 ± 1.5 | [12.0, 26.0] |
| Kimi K3 (Max) | 17, 19, 18 | 18.0 ± 1.0 | [11.3, 25.0] |
| Claude Opus 4.8 (High) | 21, 15, 17 | 17.7 ± 3.1 | [11.0, 25.0] |
| Claude Opus 5 (High) | 17, 20, 15 | 17.3 ± 2.5 | [10.7, 24.7] |
| GPT-5.6-terra (High) | 19, 18, 14 | 17.0 ± 2.6 | [10.7, 24.3] |
| DeepSeek V4 Pro (High) | 17, 18, 15 | 16.7 ± 1.5 | [10.3, 23.7] |
| Gemini 3.1 Pro Preview (High) | 18, 18, 13 | 16.3 ± 2.9 | [10.0, 23.3] |
| GPT-5.6-sol (High) | 15, 16, 17 | 16.0 ± 1.0 | [10.0, 22.7] |
| GPT-5.6-sol (Low) | 8, 19, 14 | 13.7 ± 5.5 | [6.7, 22.0] |
Pairwise differences between GLM and the other configurations
Differences below are GLM minus the corresponding configuration, in percentage points. Each interval is an approximate 95% confidence interval for that individual difference, and each value comes from an unadjusted approximate two-sided bootstrap test. Overlap between individual configurations' intervals is not used as a substitute for testing their difference.
| Comparison configuration | Mean difference | Ordinary 95% difference CI | Unadjusted p |
|---|---|---|---|
| Claude Sonnet 5 (High) | +1.67 | [-5.33, +8.67] | 0.6393 |
| GPT-6-astra (High) | +2.33 | [-4.33, +9.33] | 0.4851 |
| GPT-5.6-sol (Medium) | +2.67 | [-4.33, +10.00] | 0.4589 |
| GPT-5.6-luna (High) | +3.00 | [-3.67, +9.67] | 0.3911 |
| Kimi K3 (Max) | +3.67 | [-2.00, +10.00] | 0.2228 |
| Claude Opus 4.8 (High) | +4.00 | [-3.67, +11.33] | 0.2989 |
| Claude Opus 5 (High) | +4.33 | [-2.33, +11.33] | 0.2288 |
| GPT-5.6-terra (High) | +4.67 | [-2.00, +11.00] | 0.1632 |
| DeepSeek V4 Pro (High) | +5.00 | [-0.67, +11.33] | 0.0984 |
| Gemini 3.1 Pro Preview (High) | +5.33 | [-1.67, +12.33] | 0.1248 |
| GPT-5.6-sol (High) | +5.67 | [+0.00, +11.33] | 0.0515 |
| GPT-5.6-sol (Low) | +8.00 | [+0.00, +16.00] | 0.0529 |
Thank you for your careful comments on the paper’s research framing, experimental design, and the reliability of the reported results. Drawing on supplementary experiments, related literature, and per-task statistical analyses, we address three issues below: the definition of RSI and the division of roles between researcher and target models, the scope for improvement through data synthesis and statistical uncertainty in the reported ranking. We also added them to the paper
1. (W1, Q1) Why is this relevant to recursive self-improvement when the researcher and the trained model differ? What happens when the trained checkpoint becomes the next researcher?
First, our choice of separate researcher and target models is a practical compromise between research capability and training resources. We directly tested the proposed alternative: a locally deployed Qwen3-8B acting as both researcher and student. Both roles started from the same checkpoint. The intended loop was that the current model Wt would curate data Dt, undergo full-parameter SFT initialized from Wt, and produce Wt+1, which would then become both the next researcher and the student initializer. The experiment used the original 1,500 SWE trajectories, four A800 GPUs, and a shared 25-hour budget, with no external researcher model.
The locally deployed 8B model did not bootstrap the self-training loop. It failed to start even the first training experiment or submit a model within the budget. Its saved trace repeatedly returned to basic file inspection and empty-output initialization. Thus, in this task and configuration, the model that we could afford to train locally could not carry out the research workflow needed to initiate its own evolution. At the other end, training the larger frontier models used as researchers is beyond the compute resources available to this project. We therefore adopt a practical compromise: use a capable frontier model for research decisions and an open, locally trainable Qwen3-8B model to test those decisions through actual training and evaluation. This allows us to study the research process within an executable and reproducible training budget.
Second, separating the researcher from the trained model is an established design in closely related work. Autonomous model-development studies repeatedly use a capable research model to improve a distinct target model, including work explicitly motivated by RSI. The following examples closely match this division of roles:
These precedents support evaluating AI's ability to conduct research that improves another model. Our choice of a separate student follows this experimental practice and makes data-research decisions measurable through the resulting model's performance.