acceptodds
Under review as a conference paper at ICLR 2027

RSI-Wild: Benchmarking AI Data Researchers with In-the-Wild Agentic Trajectories for Recursive Self-Improvement

Abstract

Recursive self-improvement (RSI) envisions AI systems that can help improve future generations of models. Recent benchmarks evaluate research agents on data-centric post-training tasks, but rarely expose them to the mixed-quality trajectory pools encountered in real-world data research. We introduce RSI-Wild, a benchmark for evaluating data research agents in this more realistic setting. RSI-Wild defines 23 error types across six categories and constructs execution-grounded failed trajectories, which are mixed with verified successes to form controlled mixed-quality pools. Our experiments reveal a utility–reliability gap. Frontier agents recover substantial value from noisy data, but the strongest reaches only 21.7% on the software engineering task, compared with 29% for human researchers. They fail to consistently produce better models across iterations, use research time inefficiently, and have difficulty translating external resources into concrete experiments. Error-level analysis further shows that agents more readily address errors with explicit contradictory evidence, while often overlooking locally coherent failures that require reasoning about missing actions or verification. Together, these findings suggest that current agents can make useful data improvements, but still struggle to turn feedback and external knowledge into sustained performance gains.

Primary area:
datasets and benchmarks
Keywords:
Recursive Self-Improvement, Auto Research

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hJEDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for agents doing data work. The agent is handed a pool of agent trajectories of mixed quality, some of them deliberately broken, and has to decide what to keep, fix, or throw away. Everything downstream is frozen, i.e., same fine-tuning recipe, same evaluation, so the dataset the agent submits is the only variable, and it's judged by how well the resulting model does on tasks it was never shown during the research loop.

Critical strengths
  • The error construction is the best part of the paper. Replacing one step and letting the original model re-roll in the original container, so the downstream observations are real consequences rather than a model’s impression of failure, is the right call and the authors explain clearly why the obvious alternative is worse.
  • Freezing the recipe and isolating public feedback from the hidden score is clean. I believe the headline number is measuring the dataset and not something else.
Critical weaknesses
  • The RSI framing doesn't hold up. The researcher is GPT-5.6 or Opus or GLM. The model being trained is Qwen3-8B. Different labs, different scales, no connection between them. A big model preparing supervision for a small one is data curation, and in places it's closer to distillation. There's no self anywhere in the loop. To be about self-improvement, the checkpoint from round t would have to come back as the researcher in round t+1, and the paper neither does that nor explains why not. The title and the whole introduction promise something the experiment doesn't deliver.

  • And even ignoring that, nothing can compound. Recipe fixed, hyperparameters fixed, no RL, no access to training code, one base model, one student size, one frozen corpus of verified rollouts to draw from. Everything that would let the next iteration be better in kind rather than degree has been held constant deliberately. What's left is one round of data cleaning.

  • The task has a low ceiling and the paper doesn't confront it. When you train on the clean twin of the same 1,500 instances, you get 25.0. The best agent gets 21.7. So the whole exercise tops out at undoing damage the authors introduced, and the result is that agents undo most of it.

  • I don’t believe the ranking in Figure 5. A hundred hidden instances puts the binomial standard error near 4 points at a 20% resolve rate, before you add run-to-run noise, and the reported standard deviations run from 1 to 5.5 over three runs. GLM at 21.7 and sol (High) at 16.0 are not reliably apart, and eleven configurations are ranked in between them. Give confidence intervals and say which pairs actually separate.

Questions for authors

Why isn't the researcher the same model as the student, and what happens if you close the loop so the trained checkpoint curates the next round? Without that, what makes any of this recursive?

7 comments

Thank you for your careful comments on the paper’s research framing, experimental design, and the reliability of the reported results. Drawing on supplementary experiments, related literature, and per-task statistical analyses, we address three issues below: the definition of RSI and the division of roles between researcher and target models, the scope for improvement through data synthesis and statistical uncertainty in the reported ranking. We also added them to the paper

1. (W1, Q1) Why is this relevant to recursive self-improvement when the researcher and the trained model differ? What happens when the trained checkpoint becomes the next researcher?

First, our choice of separate researcher and target models is a practical compromise between research capability and training resources. We directly tested the proposed alternative: a locally deployed Qwen3-8B acting as both researcher and student. Both roles started from the same checkpoint. The intended loop was that the current model WtW_t would curate data DtD_t, undergo full-parameter SFT initialized from WtW_t, and produce Wt+1W_{t+1}, which would then become both the next researcher and the student initializer. The experiment used the original 1,500 SWE trajectories, four A800 GPUs, and a shared 25-hour budget, with no external researcher model.

Supplementary same-model experimentActual setting or outcome
Initial researcher and studentThe same Qwen3-8B checkpoint
Planned inheritance across roundsThe updated checkpoint becomes both the next researcher and the student initializer
Research budget and hardware25 hours; 4 × A800
Researcher inferenceNon-thinking mode; 32,768-token context; up to 8,192 new tokens
Tool calls in the saved trace17, all shell operations
Repeated behavior observedRepeatedly inspected input-file line counts, file beginnings/endings, and the first example; initialized the output training file as an empty array three times
Training experiments started0
Checkpoint updates completed0
Final model submissionNo submission

The locally deployed 8B model did not bootstrap the self-training loop. It failed to start even the first training experiment or submit a model within the budget. Its saved trace repeatedly returned to basic file inspection and empty-output initialization. Thus, in this task and configuration, the model that we could afford to train locally could not carry out the research workflow needed to initiate its own evolution. At the other end, training the larger frontier models used as researchers is beyond the compute resources available to this project. We therefore adopt a practical compromise: use a capable frontier model for research decisions and an open, locally trainable Qwen3-8B model to test those decisions through actual training and evaluation. This allows us to study the research process within an executable and reproducible training budget.

Second, separating the researcher from the trained model is an established design in closely related work. Autonomous model-development studies repeatedly use a capable research model to improve a distinct target model, including work explicitly motivated by RSI. The following examples closely match this division of roles:

StudyResearcherTrained targetRelevant design
PostTrainBenchFrontier agents, e.g., Claude Code with Opus 4.6Qwen3-4B, Gemma-3-4BThe researcher autonomously acquires data and conducts post-training experiments on a separate target model
RSIBench-DataOpus-4.8, Sonnet-5, and GPT-5.6 researchersQwen3.5-35B-A3B-BaseResearchers iteratively revise data strategies through shared training and evaluation services
RSI-MasterKimi-K3 drives the research systemQwen3-4B-Base and Qwen3.5-35B-A3B-BaseAutonomous experimentation improves independent target models under an RSI research framing
RSI ControlFrontier researchers including Claude, GPT, and GLMQwen3.5-4B, with 2B used for some tasksResearchers execute stages of a successor model's training pipeline; researcher and target need not share weights or model families

These precedents support evaluating AI's ability to conduct research that improves another model. Our choice of a separate student follows this experimental practice and makes data-research decisions measurable through the resulting model's performance.

Third, in our paper, we explicitly defines RSI motivation in terms of AI helping improve subsequent generations of models, without requiring the researcher and target to be the same model. The abstract states:

Recursive self-improvement (RSI) envisions AI systems that can help improve future generations of models.

The introduction likewise describes models taking over the research loop of designing better training methods, running experiments, evaluating outcomes, and iterating toward the next generation of models. This is the system-level meaning used in our paper: AI takes on research responsibilities that contribute to developing successor models. Under this definition, the successor may have a different architecture, parameter scale, or model family; it need not be the researcher's own checkpoint.

Accordingly, RSI-Wild evaluates the data-research component of this process: whether an agent can diagnose training-data problems, develop and execute data strategies, and use training and evaluation feedback to improve a subsequent model. The main experiments keep the researcher's weights fixed while it iterates on data strategies.

2. (W2, W3) Do the fixed training recipe and initial trajectory pool reduce the task to a single round of data cleaning, with performance capped at the 25.0% achieved by Train on Successful 1500?

Our agents’ data tasks include data synthesis: they can construct new tasks and training trajectories, expanding the scale and coverage of the training data. The 25.0% score for Train on Successful 1500 is therefore a reference obtained by training on correct trajectories for the original 1,500 instances, not a performance ceiling for our task.

SWE-Lego provides a direct example of expanding training supervision through synthetic data. Its Table 3 reports that 9,100 of the 14,110 valid trajectories that fully resolve their tasks originate from synthetic tasks, approximately 64.5%. Using a training set combining real and synthetic data, SWE-Lego achieves 42.2% Pass@1 with Qwen3-8B, matching our model scale, and 52.6% Pass@1 with Qwen3-32B, through SFT alone and before verifier-based parallel test-time scaling. This illustrates how synthesis can expand supervision beyond an initial trajectory pool: using SFT does not restrict performance to the level obtained by restoring existing correct trajectories. SWE-Lego, Table 3 and training-strategy ablations.

Reference settingModel sizeData and methodSWE-bench Verified scoreWhat the reference establishes
Train on Successful 15008BCorrect trajectories for the original 1,500 instances; fixed SFT service25.0%How much value can be recovered by replacing failed trajectories in the initial pool with correct versions
Our human-researcher reference8BData research within the same scaffold; 2,500 trajectories in the final training set29.0%Data research within our framework can exceed the Train on Successful 1500 reference
SWE-Lego-Qwen3-8B8BApproximately 18k validated trajectories combining real and synthetic data; SFT only42.2% Pass@1SFT performance at the same initial model scale is not capped at 25.0%

Our results in this table use the fixed 100-task hidden set. SWE-Lego results are the original paper's SWE-bench Verified evaluations, with different training-data scales, budgets, training procedures, and evaluation conditions. We use them as external examples of headroom for SFT and data design, rather than treating 42.2% or 52.6% as an established attainable limit within our 25-hour budget.

Fixing the recipe and hyperparameters isolates the effects of data-research decisions. Researchers can still use model feedback to identify coverage gaps, synthesize new tasks and trajectories, adjust data mixtures, and repair erroneous supervision, thereby testing data hypotheses that differ from the previous round. A fixed optimizer does not require an unchanged training set or repetition of the same cleaning strategy. Our evaluation asks whether agents can produce better target models through these data decisions under a common training service.

Accordingly, we interpret the results as follows: the strongest current agent has an observed mean of 21.7%, below the 25.0% of Train on Successful 1500 and the human researcher's 29.0%. This reflects the agents' incomplete use of the available data-research space; these three observed values do not establish a theoretical ceiling for the task. We will clarify the role of Train on Successful 1500 as a reference, emphasize the availability of task synthesis and new-trajectory construction, and distinguish feedback-driven data-research iterations from compounding growth through researcher-model upgrades across generations.

3. (W4) With only 100 hidden tasks and three runs per configuration, how reliable is the ranking in Figure 5, and which differences are statistically supported?

Thank you for pointing this out. Figure 5 presents configurations in order of their observed mean scores to support analysis of shortcomings in current agents’ data-research capabilities alongside their research traces. Its purpose is not to establish a definitive model leaderboard. We agree that the current results do not establish a complete, statistically separated ranking of the 13 configurations. To make the uncertainty explicit, we retrieved the final per-task hidden-set reports for all 39 independent research runs, paired outcomes on the same 100 tasks, and accounted for variability across runs. We now provide ordinary 95% confidence intervals for each configuration and all 78 pairwise differences, together with unadjusted approximate two-sided tests. Under the crossed task/run bootstrap, every pairwise difference interval includes zero. We will display means and confidence intervals in the figure and report pairwise differences, aligning the presentation with this analytical purpose.

For the specific comparison raised by the reviewer, GLM-5.3 (Max) scores 23%, 19%, and 23% across its three hidden-set runs, while GPT-5.6-sol (High) scores 15%, 16%, and 17%. Their observed means are 21.67% and 16.00%, a difference of 5.67 percentage points. Resampling complete research runs and shared tasks gives an approximate 95% confidence interval of [0.00, 11.33] percentage points for this difference, with an unadjusted approximate two-sided bootstrap p=0.0515p=0.0515. Thus, GLM has a higher observed mean, while this comparison does not reach the 0.05 significance threshold under this analysis.

Scores and confidence intervals for all configurations

All values below are percentages. Means and standard deviations match the main paper. Confidence intervals use 200,000 crossed task/run bootstrap resamples. Each resample uses the same sampled tasks across configurations and independently samples complete research runs within each configuration, preserving task pairing and dependence within a research run. Run indices are not artificially paired across configurations.

ConfigurationThree hidden-set scoresMean ± SDApproximate 95% CI
GLM-5.3 (Max)23, 19, 2321.7 ± 2.3[14.3, 29.7]
Claude Sonnet 5 (High)20, 19, 2120.0 ± 1.0[13.0, 27.3]
GPT-6-astra (High)16, 21, 2119.3 ± 2.9[12.3, 27.0]
GPT-5.6-sol (Medium)16, 23, 1819.0 ± 3.6[12.0, 27.0]
GPT-5.6-luna (High)19, 17, 2018.7 ± 1.5[12.0, 26.0]
Kimi K3 (Max)17, 19, 1818.0 ± 1.0[11.3, 25.0]
Claude Opus 4.8 (High)21, 15, 1717.7 ± 3.1[11.0, 25.0]
Claude Opus 5 (High)17, 20, 1517.3 ± 2.5[10.7, 24.7]
GPT-5.6-terra (High)19, 18, 1417.0 ± 2.6[10.7, 24.3]
DeepSeek V4 Pro (High)17, 18, 1516.7 ± 1.5[10.3, 23.7]
Gemini 3.1 Pro Preview (High)18, 18, 1316.3 ± 2.9[10.0, 23.3]
GPT-5.6-sol (High)15, 16, 1716.0 ± 1.0[10.0, 22.7]
GPT-5.6-sol (Low)8, 19, 1413.7 ± 5.5[6.7, 22.0]

Pairwise differences between GLM and the other configurations

Differences below are GLM minus the corresponding configuration, in percentage points. Each interval is an approximate 95% confidence interval for that individual difference, and each pp value comes from an unadjusted approximate two-sided bootstrap test. Overlap between individual configurations' intervals is not used as a substitute for testing their difference.

Comparison configurationMean differenceOrdinary 95% difference CIUnadjusted p
Claude Sonnet 5 (High)+1.67[-5.33, +8.67]0.6393
GPT-6-astra (High)+2.33[-4.33, +9.33]0.4851
GPT-5.6-sol (Medium)+2.67[-4.33, +10.00]0.4589
GPT-5.6-luna (High)+3.00[-3.67, +9.67]0.3911
Kimi K3 (Max)+3.67[-2.00, +10.00]0.2228
Claude Opus 4.8 (High)+4.00[-3.67, +11.33]0.2989
Claude Opus 5 (High)+4.33[-2.33, +11.33]0.2288
GPT-5.6-terra (High)+4.67[-2.00, +11.00]0.1632
DeepSeek V4 Pro (High)+5.00[-0.67, +11.33]0.0984
Gemini 3.1 Pro Preview (High)+5.33[-1.67, +12.33]0.1248
GPT-5.6-sol (High)+5.67[+0.00, +11.33]0.0515
GPT-5.6-sol (Low)+8.00[+0.00, +16.00]0.0529
Reviewer_bSUNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RSI-Wild evaluates agents that curate, repair, or synthesize training trajectories under a fixed fine-tuning recipe and a shared research/training/evaluation time budget. The benchmark creates controlled failures by injecting one of 23 error types into successful trajectories and continuing execution in the original environment. Their datasets train Qwen3-8B, with public feedback and disjoint hidden evaluation. The paper evaluates 13 SWE researcher configurations with three runs each, adds a smaller terminal track, and studies data operations, external resources, and error-specific interventions.

Critical strengths
  1. The benchmark isolates an important research capability. Fixing the model, recipe, and evaluation services while varying data decisions makes this a useful test of automated data research. A mixed-quality trajectory pool offers a complementary challenge to clean-data curation or synthesis from scratch.

  2. The construction is more grounded than unconstrained text corruption. Continuing execution from a modified step ensures that subsequent observations arise from the environment. Retaining hidden type labels enables analyses that aggregate model scores alone cannot provide.

Critical weaknesses
  1. Intervention is not equivalent to diagnosis or successful repair. Section 5.3 counts dropping, rewriting, or semantic masking as intervention. An agent that drops all trajectories can intervene on every labeled error without recognizing any type; GLM's terminal intervention rate of 100% with an E–C gap of zero illustrates the distinction (Table 3). The E–C and within-run normalizations are useful controls, but do not validate correction quality. Claims that agents “detect” or “address” particular errors should be phrased as intervention patterns unless supported by a repair-validity audit. With only 20 examples per type, uncertainty and source-task clustering also matter.

  2. Section 4.4 interprets approximately 3% active research time as poor budget allocation, but a fixed expensive full-parameter training service can dominate wall-clock time even for a sensible strategy. A scheduled or simple scripted policy would help establish avoidable inefficiency. Similarly, 22/34 second attempts regressing on public scores does not isolate an intrinsic inability to use feedback from evaluation noise or weak checkpoint selection. The single half-pool long-horizon run is suggestive, but changes the data regime and does not establish a general iterative trend.

Questions for authors
  1. Can the existing runs separate training latency, evaluation latency, active analysis, and idle time, and compare the agent's allocation to a simple fixed schedule?

  2. For a sample of rewritten or masked errors, how often is the defect actually corrected or appropriately excluded? Can intervention results distinguish broad pruning from type-specific repair?

6 comments

Thank you for distinguishing trajectory intervention from diagnosis and successful repair, and for highlighting the effects of service cost and evaluation variability on our interpretation of research outcomes. In response, we have revised how we quantify and report error repair, added a repair-validity assessment, decomposed service time, and examined fixed scheduling and continued research. We report these results below and have added them to the paper.

1. (W1, Q2) How often do rewrites or mask changes correct or appropriately exclude a defect, and how can broad pruning be distinguished from specific repair?

We recognize the limitation of the original error-intervention metric: it records whether a trajectory is dropped, rewritten, or semantically masked, whereas repair validity requires direct examination of the rewritten content. We therefore added an error-repair validity assessment, checking whether the established defect was removed in each case and reporting successful and failed repairs across all 23 error types.

Assessment method. The analysis covers the 13 configurations and 39 runs in the main SWE results, using the finally selected iteration of each run. The assessment has three steps:

  1. Identify rewrite candidates. We align the trajectories actually read by each run with its final output, distinguishing deletion, mask-only changes, unchanged content, and content rewrites. Content rewrites enter the repair assessment; deletions and mask changes remain separately recorded in the intervention statistics.
  2. Establish necessary repair criteria. We compare the actual input with a correct trajectory for the same task, check the defect location and necessary actions, and formulate criteria for the corresponding task and input version. The criteria accept semantically equivalent repairs. Additional exploration and nonessential tests in the reference are not imposed as requirements.
  3. Determine the repair outcome. Using the criteria and relevant original text from the input and correct reference, the evaluator model examines each candidate rewrite and provides message citations. A repair succeeds when the necessary conditions are met, the defect is removed, and no new related problem is observed. It fails when the defect remains or a necessary condition is unmet. Missing actions are assessed from the candidate's actual action records, with completion claims checked separately from action execution.

Overall results. Among 4,989 evaluable rewrites, 814 repairs succeeded and 4,175 failed, giving a success rate of 16.3%.

Repair outcomeCountPercentage
Successful repair81416.3%
Failed repair4,17583.7%
Total4,989100%

Results for all 23 types. The following table groups cases by their original injection label. Each success rate uses the evaluable rewrites of that type as its denominator, rather than all interventions.

Original label and error typeEvaluable rewritesSuccessesFailuresSuccess rate
PL.1 Ineffective Planning2202519511.4%
PL.2 Goal Drift2286216627.2%
PL.3 Constraint Ignorance2292920012.7%
PL.4 Impossible Action1962217411.2%
R.1 Hallucination200141867.0%
R.2 Reasoning Error2182619211.9%
R.3 Calculation Error2062218410.7%
R.4 Task Misunderstanding2405918124.6%
M.1 Retrieval Failure2114916223.2%
M.2 Hallucination (False Memory)2375218521.9%
M.3 Over-simplification / Incomplete Summary2274018717.6%
A.1 Tool / Parameter Error2192819112.8%
A.2 Hallucinated Tool or Action2263818816.8%
A.3 Output Format Error2236016326.9%
A.4 Planning-Action Disconnect220172037.7%
A.5 Premature Termination172131597.6%
A.6 Looping Behavior2425718523.6%
V.1 Context Loss195181779.2%
V.2 Inadequate Verification190141767.4%
RF.1 Progress Misassessment1801816210.0%
RF.2 Outcome Misinterpretation2336117226.2%
RF.3 Causal Misattribution2295617324.5%
RF.4 Hallucination2483421413.7%
Total4,9898144,17516.3%

The repair success rate is 23.6% for Looping Behavior, 7.6% for Premature Termination, and 7.4% for Inadequate Verification. The rates for Goal Drift, Output Format Error, and Outcome Misinterpretation are 27.2%, 26.9%, and 26.2%, respectively. These results describe repair performance among rewritten cases and are interpreted alongside the rewrite coverage of each type.

2. (W1) How should 20 source tasks per type and repeated observations across runs affect uncertainty?

Within the input pool, each of the 460 labeled-error trajectories corresponds to a distinct source task, with 20 distinct tasks per error type. The pool therefore does not inflate its task count by including multiple labeled-error trajectories generated from the same source task.

The same tasks are processed across 39 researcher runs, so a task may contribute more than one intervention outcome. Each type still has 20 distinct tasks, not 780 independent tasks. When estimating uncertainty, we treat multiple outcomes for the same task as repeated observations rather than independent samples.

3. (W2, Q1) Can training, evaluation, active analysis, and idle time be separated and compared with a human-prescribed fixed schedule?

To examine whether the model's autonomous time scheduling leaves room for improvement, we added a human-prescribed fixed-schedule control. With the same initial data, training/evaluation recipe, and 25-hour cap, humans specified the research phases and minimum time allocations before the run. Our discussion of time scheduling therefore draws on a comparison between human-prescribed and autonomous allocation, rather than the approximately 3%/97% time split alone.

Time decomposition. Service records identify the start and end of training and public evaluation separately, while the scheduler records analysis/curation and synthesis phases. For the GPT-5.6-terra High fixed-schedule run, we clipped these intervals to the period between budget start and final submission, checked that they did not overlap, and reported the remaining time separately.

Time componentHoursShare of elapsed time before submission
Training service16.4968.7%
Public evaluation service4.6419.3%
Analysis/curation phase0.793.3%
Synthesis phase1.958.1%
Other non-service time0.150.6%
Total before submission24.02100%

Training and public evaluation together took 21.13 hours, or 88.0% of elapsed time. Analysis/curation and synthesis together took 2.74 hours, or 11.4%. The remaining 0.98 hours of the 25-hour budget are recorded separately as unused budget.

Training and evaluation times are measured as service-call wall-clock durations, including startup and execution overhead. Analysis and synthesis are scheduler-defined wall-clock phases that include tool execution and API waiting. The remaining 0.15 hours cover phase transitions, control operations, final selection, and other time not separately classified. Thus, the logs distinguish service latency, research phases, and other time; active analysis and waiting within research phases are not individually timed, and the residual is not treated as pure idle time. Section 4.4 will specify this measurement basis for the original 3%/97% statistic.

Human-prescribed fixed-schedule policy. The comparison uses the same initial trajectory pool, four-GPU training/evaluation recipe, and 25-hour budget cap. Before the run, humans required at least 20 minutes of analysis/curation and 40 minutes of synthesis per round, at least one accepted new trajectory, and at most two training attempts. The script enforces these requirements; the minimum phase allocations are not chosen autonomously by the model. Within this fixed schedule, the agent processes the data, decides whether to continue to another round, and selects the final checkpoint. The completed run is summarized below.

MetricRound 1Round 2
Analysis/curation phase23.25 min24.17 min
Synthesis phase69.94 min47.04 min
New synthetic trajectories accepted / proposed14/159/9
Training service8.23 h8.27 h
Public evaluation service2.24 h2.40 h
Complete training/evaluation cycle10.47 h10.66 h
Public score15/10017/100
Final selection—Second round, hidden score 19/100
GPT-5.6-terra High settingComplete research runsPublic scoreHidden score
Original autonomous scheduling314.7±4.5%17.0±2.6%
Additional human-prescribed fixed schedule117/10019/100

The human-prescribed schedule achieved 17/100 public and 19/100 hidden, compared with historical autonomous-scheduling means of 14.7% and 17.0%. This provides a preliminary indication of limitations in the model's autonomous time scheduling and suggests that externally prescribing research phases and allocations is a direction for improvement.

Reviewer_Yb47OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RSI-Wild evaluates AI data researchers on mixed-quality agent trajectories under a fixed training and evaluation pipeline. The authors define 23 error types across six categories, inject errors into verified successful trajectories, and resume execution in the original environments to obtain failed trajectories with known injection labels. Researchers can filter, repair, mask, or synthesize training data, train Qwen3-8B, and use public evaluation feedback within a 25-hour budget; submitted checkpoints are scored on a disjoint hidden split. Experiments on software-engineering and terminal tasks show that agents can improve over direct training on the mixed pool, while remaining below the successful-data and human references. Further analyses examine intervention patterns, iterative behavior, budget use, and access to external research resources.

Critical strengths
  1. Controlled errors with execution-grounded consequences. Resuming trajectories in their original environments is a useful improvement over purely textual fabrication of failures. The known injection labels enable a more detailed analysis of data-processing behavior than aggregate downstream scores alone.

  2. A substantive closed-loop evaluation. The benchmark combines actual training feedback, disjoint final scoring, and a human reference operating under the same stated time and tool constraints. Repeated SWE runs, operation ablations, and general-capability evaluations provide useful evidence about both gains and trade-offs.

Critical weaknesses
  1. The intervention metric does not directly establish error recognition or successful repair. Section 5.3 counts a trajectory as intervened on when it is dropped, rewritten, or semantically loss-masked. A generic rewrite or length-based filter can satisfy this definition without locating or correcting the injected error. Consequently, the error-type preferences support conclusions about which trajectories are modified, but not necessarily which errors agents understand. The interpretation concerning explicit contradictions versus missing actions would be stronger with error-localization and repair-validity checks, controlling for trajectory length and repetition. This matters because error-level diagnosis is a central contribution.

  2. Several explanations of research bottlenecks are stronger than the supporting comparisons. In Section 4.4, spending 97% of runtime on training/evaluation could reflect the fixed service costs, and early stopping may be reasonable if another complete trial cannot fit. Similarly, a lower second-attempt score (Section 4.3) does not by itself show ineffective feedback use: useful exploration can regress before checkpoint selection. Alternative allocation policies, no-feedback search, and repeated-training controls are needed to distinguish these explanations. The synthetic, failure-conditioned pool and single-run terminal results further limit how broadly these behavioral conclusions can be generalized.

Questions for authors
  1. How often does a counted intervention actually target the injected step and remove the specified error? Do you have a manual audit or execution-based validation, and do the error-type preferences remain after controlling for length and repetition?

  2. At early termination, was enough time available for another complete training/evaluation round? Is there evidence that additional analysis or continued experimentation would improve the submitted result?

  3. Can you compare feedback-driven research against an equal-budget no-feedback or fixed-strategy baseline, and quantify training variability? Even a focused comparison would help substantiate the claimed bottlenecks.

  4. What validation establishes the realism and label fidelity of the constructed failures? Have the main error-level findings been checked on naturally occurring failures?

6 comments

We thank the reviewer for the specific suggestions on the intervention metric, interpretations of the research process, and data validity. We have supplemented the analysis with a repair validity assessment, controls for trajectory length and repetition, and experiments on continued research, a prescribed schedule, research without evaluation feedback, retraining on fixed data, and naturally occurring failures. We report these results below and have added them to the paper.

1. (W1, Q1) How often does a counted intervention target and remove the injected error? Do error-type preferences remain after controlling for length and repetition?

We recognize the limitation of the original intervention metric: it records whether a trajectory is dropped, rewritten, or semantically loss-masked, while repair validity requires direct examination of the rewritten content. We therefore added a repair validity assessment that checks whether each established defect has been removed and reports successes and failures across all 23 error types.

Assessment procedure. The analysis covers the final selected iteration of each of the 39 runs across the 13 configurations in the main SWE results. It has three steps:

  1. Identify rewritten trajectories. We align the actual input of each run with its final output and distinguish deletion, mask-only changes, unchanged content, and content rewriting. Rewritten trajectories enter the repair assessment; deletions and mask changes remain separate categories in the intervention statistics.
  2. Establish necessary repair criteria. We compare the actual input with the correct reference trajectory for the same task, identify the defect location and necessary actions, and construct criteria for each task and input version. The criteria accept semantically equivalent repairs; additional exploration and optional tests in the reference are not required.
  3. Assess the repair. The evaluator model checks each candidate rewrite against these criteria and the relevant input and reference text, providing message citations. A case is successful when all necessary conditions are met, the defect is removed, and no new related problem is observed. It fails when the defect remains or a necessary condition is unmet. For missing actions, we examine action records in the candidate and assess completion claims separately from the actions themselves.

Overall results. Among 4,989 evaluable rewrites, 814 repairs succeeded and 4,175 failed, for a success rate of 16.3%.

Repair validity assessmentCountPercentage
Successful repair81416.3%
Failed repair4,17583.7%
Total4,989100%

Results for all 23 error types. Rows are grouped by the original injection label. The success rate is the number of successes divided by the number of evaluable rewrites. This assessment concerns textual repair of defects established in the actual input; consistency between the assessment criteria and the original labels is reported in Q4.

Original label and error typeEvaluable rewritesSuccessesFailuresSuccess rate
PL.1 Ineffective Planning2202519511.4%
PL.2 Goal Drift2286216627.2%
PL.3 Constraint Ignorance2292920012.7%
PL.4 Impossible Action1962217411.2%
R.1 Hallucination200141867.0%
R.2 Reasoning Error2182619211.9%
R.3 Calculation Error2062218410.7%
R.4 Task Misunderstanding2405918124.6%
M.1 Retrieval Failure2114916223.2%
M.2 Hallucination (False Memory)2375218521.9%
M.3 Over-simplification / Incomplete Summary2274018717.6%
A.1 Tool / Parameter Error2192819112.8%
A.2 Hallucinated Tool or Action2263818816.8%
A.3 Output Format Error2236016326.9%
A.4 Planning-Action Disconnect220172037.7%
A.5 Premature Termination172131597.6%
A.6 Looping Behavior2425718523.6%
V.1 Context Loss195181779.2%
V.2 Inadequate Verification190141767.4%
RF.1 Progress Misassessment1801816210.0%
RF.2 Outcome Misinterpretation2336117226.2%
RF.3 Causal Misattribution2295617324.5%
RF.4 Hallucination2483421413.7%
Total4,9898144,17516.3%
Repair success rates are 27.2% for Goal Drift, 26.9% for Output Format Error, and 26.2% for Outcome Misinterpretation. For the types emphasized in the original discussion, the rates are 23.6% for Looping Behavior, 7.6% for Premature Termination, and 7.4% for Inadequate Verification.

Controls for length and repetition. We use all 58,500 input observations across the 39 runs to fit a linear probability model with error type, run fixed effects, nonlinear terms for input length, and literal repetition features. A sensitivity analysis additionally includes assistant-text length and message count. The table reports differences in intervention probability in percentage points, with approximate 95% intervals using two-way clustering by task and run.

Type comparisonUnadjusted difference (controlling for run)Add input length and repetitionAlso add assistant length and message count
Looping Behavior − Premature Termination+15.90 [+5.14, +26.66]+7.66 [+1.03, +14.29]+4.22 [-1.52, +9.96]
Looping Behavior − Inadequate Verification+11.54 [+2.43, +20.65]+6.55 [+0.80, +12.31]+2.44 [-2.00, +6.89]
Premature Termination − Correct trajectories-6.42 [-11.91, -0.93]+1.22 [-3.60, +6.05]+1.92 [-3.32, +7.16]
Inadequate Verification − Correct trajectories-2.06 [-8.54, +4.42]+2.33 [-3.11, +7.77]+3.69 [-2.20, +9.58]

After adjustment for input features, the intervention gap between Looping Behavior and Premature Termination decreases from 15.90 to 7.66 percentage points, and to 4.22 after further adjustment for assistant length and message count. The gap between Looping Behavior and Inadequate Verification decreases from 11.54 to 6.55 and then to 2.44 percentage points. Both intervals include zero under the final specification. The initially negative difference between Premature Termination and correct trajectories also becomes positive after adjustment.

Among the 4,989 repair assessment records, controlling for input length, repetition, and run yields a repair-success probability difference of +5.72 percentage points between Looping Behavior and Premature Termination, with a 95% interval of [-12.25, +23.69]. The corresponding difference between Looping Behavior and Inadequate Verification is +10.97 percentage points, with an interval of [-4.11, +26.05].

These results show that the observed differences across types are associated with input length, repetition, and task composition. In Section 5.3, we will report intervention rates and repair success rates separately and interpret type differences alongside the adjusted estimates.

2. (W2, Q2) Was enough time available for another complete training/evaluation round at early termination? Can continued experimentation or additional analysis improve the final result?

We added continuation and prescribed-schedule experiments, assessing early termination using three quantities: the remaining budget at stopping, the duration of a complete experimental cycle, and the final selection after continuation.

ExperimentBudget conditionWork completedOutcome
SWE, GPT-5.6-terra High continuation15.64 h1 additional round; training and public evaluation took 10.43 hPublic score 15/100 → 10/100; original candidate retained
Terminal, GPT-5.6-sol High continuation22.18 h7 additional candidates, 8 in totalOriginal candidate retained
SWE, GPT-5.6-terra High prescribed schedule25 h2 rounds, with cycles of 10.47 and 10.66 hPublic score 15/100 → 17/100; second round selected, hidden score 19/100

In the SWE continuation experiment, the remaining 15.64 h accommodated an additional round that actually took 10.43 h. The new candidate scored 10/100 on the public set, and the researcher retained the original candidate, whose existing hidden score was 18/100. The Terminal continuation experiment added seven candidates and also retained the original candidate. In both cases, continuation produced additional candidates while leaving the final submitted model unchanged. The SWE continuation restored the workspace and research history. The Terminal continuation also involved service repairs, and its outcome is reported in terms of whether the final model selection changed.

The prescribed-schedule experiment required at least 20 minutes of analysis/curation and 40 minutes of synthesis per round, allowed at most two training attempts, and required acceptance of new synthetic trajectories. Its execution was as follows.

MetricRound 1Round 2
Analysis/curation phase23.25 min24.17 min
Synthesis phase69.94 min47.04 min
New trajectories accepted / proposed14/159/9
Training service time8.23 h8.27 h
Public evaluation service time2.24 h2.40 h
Complete training/evaluation cycle10.47 h10.66 h
Public score15/10017/100
Final selection—Second round; hidden score 19/100

At submission, this run had used 24.02 h and had 0.98 h remaining, less than either observed complete cycle. Stopping was therefore consistent with the remaining budget. This result comes from one complete run and reflects the combined scheduling strategy of prescribed phase durations, additional synthesis, and two training rounds. Phase durations are measured as wall-clock time and include waiting and API latency.

These experiments distinguish two situations: some stopping points leave enough time to continue, but the additional candidates do not change the final selection; the prescribed-schedule run ends with too little time for another complete cycle. In Section 4.4, we will report the 97% training/evaluation share as a breakdown of service costs and assess budget allocation by whether the remaining time can accommodate a complete cycle.

3. (W2, Q3) Under the same budget, how does feedback-driven research compare with no-feedback or fixed-strategy research? Can training variability explain candidate-score regressions?

We added two SWE research runs without evaluation feedback. Both used GPT-5.6-sol High, the same initial 1,500 trajectories, the same four-GPU training/evaluation recipe, and a 25 h budget cap. Training and public evaluation proceeded normally and counted against the budget, while public scores, per-task results, and rollouts were hidden from the researcher.

We report them alongside the original feedback setting below.

ConditionComplete research runsPublic scoreHidden score
Original feedback setting, GPT-5.6-sol High319.7±2.5%16.0±1.0%
Additional no-feedback run 1111/1009/100
Additional no-feedback run 2118/10019/100
Additional no-feedback runs, aggregate214.5±4.9%14.0±7.1%

This comparison shows that researchers without evaluation feedback can also produce trainable data strategies, suggesting that agents are currently not utilizing feedback effectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.