acceptodds
Under review as a conference paper at ICLR 2027

RSIE: Recursive Self-Improvement through Interaction with Expert Environments

Abstract

Recent work on recursive self-improvement (RSI) has demonstrated that agents can improve through repeated interaction with their tasks and environments. Yet it remains unclear whether such improvement can persist in expert environments, where executions are long-horizon and costly, and feedback is sparse, delayed, and domain-specific. Research and professional tasks naturally exhibit these characteristics, making them a demanding testbed for recursive self-improvement. We introduce RSIE, a system for recursive self-improvement through interaction with expert environments. RSIE repeatedly executes expert tasks, extracts reusable experience from completed trajectories and their evaluation feedback, and retains candidate experience only when its reuse leads to measurable improvement. Confirmed experience is accumulated through two complementary paths: externalized guidance for subsequent executions and parameter internalization via on-policy self-distillation. As the agent improves, saturated tasks are replaced by new ones, continually exposing the system to its evolving capability frontier. We evaluate RSIE with Qwen3.6-35B and Qwen3.5-9B on ResearchClawBench, SGIBench, and GDPVal, spanning scientific research and professional work. RSIE consistently improves over the corresponding base agents, showing that recursive self-improvement can extend to expert environments with sparse task-level feedback.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Self-evolving agents, LLM agents, on-policy self-distillation, Real Expert Research

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8jNPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce RSIE, a closed-loop RSI system over "expert environments."

FrontierWorld constructs executable task packages from already-published research and professional deliverables (released studies with data/code, clinical notes, financial models, engineering reports). Each package consists of an instruction, an assembled workspace, and a task-specific reference rubric; reference material is hidden from the agent, and a static validation certifies package quality without executing a solver. A task state machine (reserve / active / mastered / revisit) refreshes the active frontier when tasks saturate.

Experience admission. A GPT-5.4 diagnoser writes candidate experience entries in a fixed schema (trigger / procedure / validation / repair; four functional views). Eq. 3 admits an entry only when its reuse raises the same task's valid score beyond the previous best under the unchanged evaluator; invalid runs (timeouts, worker failures, contract failures) are excluded.

Internalization. Verified experience is used through (i) externalized retrieval — conservative, transferable domain-skill and general-workflow views only, at most one entry retrieved, never across the task boundary — and (ii) parametric internalization via on-policy self-distillation (LoRA, generalized JSD, λ = 1, ZeRO-3, no distilled answer).

Evaluation. Ten cycles from Qwen3.6-35B and Qwen3.5-9B on ResearchClawBench (600 active tasks), SGIBench (5,000 generated tasks; 318 DeepResearch + 68 wet experiments), and GDPVal (1,000 active tasks; 220 scored), all executed through the common ResearchHarness (4 CPU cores / 8 GB; 2–4 h per-task budgets).

Headline results. 35B: RCB 12.1 → 20.1, Wet 23.37% → 56.90%, DeepResearch 48.43% → 59.50%, GDPVal 838 → 1339 Elo. 9B: 8.83 → 14.9, 17.80% → 48.80%, 44.97% → 48.56%, 608 → 825 Elo.

Critical strengths
  1. Table 1 organizes representative systems by improvement mechanism (task/world, model update, experience reuse, verified retention) and feedback regime (judge, cost per run), cleanly positioning the paper at the "hours per run, external rubric" corner that existing RSI work does not cover.
  2. Admission requires strict same-task improvement over the agent's own prior best under the unchanged evaluator; infrastructure failures are dropped rather than promoted; the static validation cannot roll out a solver and hence cannot certify solvability; the public/hidden boundary ensures the agent never reads the reference reports. These are the correct instincts for a setting with sparse positive feedback.
  3. Experience organization (four functional categories, transferable vs. task-bound, with the 56.7%/87.0% skill-overlap diagnostic), the checklist variant (18.7), progressive exposure vs. one-shot injection (19.2 vs. 16.9), fixed vs. refreshed frontier (best 19.2; cycle-10 18.7), data scaling (N = 64…600), and an offline SFT control accompanied by a credible trajectory audit (1,000 → 348 retained examples; 87.3% repeated-skill overlap), plus a DPO control with a frozen base agent and disjoint training pools.
  4. Benchmarks are never averaged; native evaluators and formats are retained; the wet subset is kept separate; the authors decline to fit a scaling law to four measured points; the lack of an SFT validation split and the shared diagnoser/judge model are disclosed rather than hidden.
Critical weaknesses
  1. GPT-5.4 is simultaneously the experience extractor and the judge for three of four metrics. I am therefore not able to attribute the reported gains to improved expert-task competence rather than to improved alignment with one model family's rubric.
  2. No RSI method is included as an experimental baseline.
  3. Pretraining contamination of the published-deliverable source is not examined. FrontierWorld is built from real, already-released documents (released studies, clinical notes, financial models, reports). The public/hidden boundary protects the reference during a run, but the Qwen base models may already have encountered the source deliverables during pretraining. There is no source-to-pretraining overlap analysis (titles, URLs, n-grams, citations) and no memorization control, and no analysis of whether high scores reflect prior knowledge of the source text rather than executed competence. Held-out-by-task-identifier (600 / 5,000 / 1,000 active pools versus 40 / 386 / 220 scored items) is not held-out-by-task-family.
  4. Cross-task-family transfer rests only on aggregate benchmark totals. The conclusion that improvement "transfers across task types" is supported by benchmark totals, not by a transfer test. Retrieval is deliberately conservative (one entry, transferable views only), but no experiment evaluates on a task family entirely unseen during experience accumulation; no endpoint retrieval-hit-rate or counterfactual (retrieval disabled) breakdown is given; and the 56.7% skill-overlap number is a text-statistic proxy for transfer, not behavioral evidence of transfer.
  5. No total compute/cost accounting and no uncertainty reporting. Ten cycles over 600 / 5,000 / 1,000 active tasks, with 2–4 h per-task budgets and pervasive GPT-5.4 judging, warrant a total compute ledger (CPU/GPU hours, GPT-5.4 calls and tokens, estimated cost, Elo pairwise-comparison counts); none is reported. Results appear to be single runs: no seeds, no standard deviations, no confidence intervals, and no CIs or bootstrap intervals for the Elo estimates, despite post-hoc "best checkpoint" selection (e.g., 19.2 at cycle 5).
Questions for authors
  1. Please re-score all three benchmarks using judges from a different model family (ideally two; e.g., Claude, Gemini, or DeepSeek), blind to the training condition, and report (i) inter-judge agreement (Cohen's κ or pairwise agreement) against GPT-5.4, (ii) the complete score tables under the alternative judges, and (iii) whether the gains and rank orderings survive. A human-rated subset as an anchor would further strengthen the claim. Does the wet-experiment subset remain the only independent signal, and do gains hold there with more tasks?
  2. Please (i) define the exact configuration behind Table 3's "RSIE" (19.2) and Table 4's N = 600 (19.20) — model size, retrieval on/off, frontier refresh, checkpoint selection — and state how they relate to the complete 20.1 RSIE in Table 2; (ii) reprint Tables 3 and 4 with model size and retrieval flags on every row and explicitly discuss the 19.20 (35B, w/o retrieval) vs. 14.9 (9B, complete) comparison; (iii) restore the wet values in §4.2 or state explicitly that the column was omitted; (iv) correct the Table 9 header and confirm the row-to-model mapping; (v) state architecture, total/activated parameters, and checkpoint IDs for both bases, including whether "35B" is a MoE active-parameter count.
  3. What is the admission gate's measured gain on task families never seen during experience accumulation (e.g., extract experience on two domains/benchmarks and evaluate on a third, with no extraction from it)? Please report retrieval hit rates at the endpoints, a per-task counterfactual with retrieval disabled, and transfer broken down by task family rather than only by benchmark totals. 4 Please adapt at least one RSI method (R-Zero, ReasoningBank, or KOPE) into ResearchHarness under matched task budgets and the same native evaluators, on at least one benchmark. It need not win; the point is to make the claimed "expensive setting" contrast experimentally testable.
  4. Please provide (i) a contamination audit of the source deliverables against pretraining (title/URL/n-gram overlap and memorization probes) and (ii) a full compute/cost ledger (CPU/GPU hours, GPT-5.4 calls/tokens, Elo pair counts); repeat the headline runs with multiple seeds and report standard deviations or confidence intervals for the Elo estimates.
6 comments
AuthorsOpenReview ↗

Answer of Q1 To Reviewer 8jNP

We thank the reviewer for this valuable suggestion.

Q1. Reliability of evaluation across LLM judges, human experts, and deterministic scoring

RSIE consistently outperforms the base model and SFT across all three benchmarks when evaluated by alternative judges, with the method ranking unchanged. Independent evaluations by two human experts also confirm the improvement. In addition, after expanding the wet-experiment evaluation to 200 tasks, RSIE retains a 35.62 percentage-point gain over the base model under deterministic scoring. Below, we provide evidence from these three perspectives.

1. Consistent improvements across judge families.

We rescored the same final artifacts using DeepSeek-V4.1-Flash and Gemini-3.8-Flash, keeping the tasks, original evaluation criteria, and score aggregation procedures unchanged. Only the judge model was replaced. Method names and training conditions were hidden, and the presentation order of paired artifacts was randomized. All results below use Qwen3.6-35B:

JudgeMethodRCB ↑SGI-DR ↑GDPVal Elo ↑
GPT-5.4Base model12.148.43838
GPT-5.4SFT13.450.63866
GPT-5.4RSIE20.159.501339
DeepSeek-V4.1-FlashBase model12.750.01855
DeepSeek-V4.1-FlashSFT14.552.77901
DeepSeek-V4.1-FlashRSIE19.760.331353
Gemini-3.8-FlashBase model11.9347.62804
Gemini-3.8-FlashSFT12.748.90849
Gemini-3.8-FlashRSIE19.858.921288

All three judges preserve the ranking RSIE > SFT > base model on all three benchmarks. Relative to the base model under each judge, RSIE improves RCB scores by 7.0–8.0 points, SGI-DR scores by 10.32–11.30 percentage points, and GDPVal ratings by 484–501 Elo.

We further measure pairwise agreement by comparing each alternative judge’s win/loss/tie decisions with those of GPT-5.4 on base-model and RSIE artifacts from the same tasks. Agreement is the proportion of decisions that match exactly.

Pairwise agreement with GPT-5.4RCBSGI-DRGDPVal
DeepSeek-V4.1-Flash80.0%88.7%85.0%
Gemini-3.8-Flash85.0%90.6%89.1%

These judge families produce similar scores, preserve the same method ranking, and agree on relative output quality for most tasks. Together, these results support the improvement in expert task performance beyond the scoring preferences of GPT-5.4.

2. Independent human evaluation confirms improved artifact quality.

We included all RCB test tasks in a human evaluation conducted by two experts with relevant research backgrounds. Each expert independently evaluated the code, analysis results, and reports produced by the base model and RSIE, following the original checklist and its weights. Task inclusion did not depend on model scores. Method identities were hidden, artifact order was randomized, and the experts did not discuss their assessments before submitting scores.

MethodExpert 1: RCB score ↑Expert 2: RCB score ↑Mean ↑
Base model101512.5
RSIE152520.0
Improvement+5+10+7.5

Although the experts differ in their absolute scores, both rate RSIE above the base model, with a mean improvement of 7.5 points. This provides independent human evidence of improved artifact quality, complementing the wet-experiment evaluation as evidence that does not rely on an LLM judge.

3. Gains persist under deterministic scoring on an expanded task set.

The SGIBench wet-experiment evaluation uses deterministic action-scoring rules and does not rely on an LLM judge. We expanded the evaluation from 68 to 200 tasks while retaining the same scoring procedure:

Wet-experiment settingNumber of tasksBase model ↑RSIE ↑Gain (percentage points)
Original evaluation6823.3756.90+33.53
Expanded evaluation20023.3758.99+35.62

On the expanded set of 200 tasks, RSIE improves over the base model by 35.62 percentage points, confirming that the gains persist across a larger set of wet-experiment tasks. Taken together, evaluation across judge families, independent expert assessment, and deterministic scoring support the interpretation that the observed gains reflect improved expert task performance beyond adaptation to a single judge’s preferences.

AuthorsOpenReview ↗

Answer of Q2 To Reviewer 8jNP

Q2. Clarification of experimental settings, table labels, and model details

The scores of 19.2 in Table 3 and 19.20 in Table 4 both correspond to Qwen3.6-35B-A3B without external experience at inference time, matching the “w/o experience” configuration in Table 2. The score of 20.1 corresponds to the full RSIE configuration using the same model with external experience enabled. We have standardized model names and configuration labels across these tables and added per-round wet-experiment results and model details.

1. The scores of 19.2 and 20.1 reflect different inference configurations of the same model.

The RSIE reference row in Table 3 and the Qwen3.6-35B-A3B result at N=600N=600 in Table 4 both follow the RSIE training pipeline, including FrontierWorld task refreshing, and use checkpoint-2000, obtained after 2,000 post-training steps. External experience is disabled at inference time in both cases, yielding the same score of 19.2 reported in Table 2.

Here, “experience” refers to external guidance loaded from the experience bank at inference time. The “w/o experience” setting disables only this inference-time guidance; experience extraction, validation, and OPSD updates remain part of training. Enabling external experience for the same model increases the RCB score from 19.2 to 20.1, indicating that the experience bank provides additional benefits beyond the capabilities acquired through post-training.

2. Tables 3–4 now include complete model names and an Experience label for each row.

All rows in Table 3 use Qwen3.6-35B-A3B. Table 4 reports results for Qwen3.6-35B-A3B and Qwen3.5-9B. The Experience column is marked “w/o” throughout both tables.

Qwen3.6-35B-A3B achieves 19.20 without external experience, exceeding the 14.9 achieved by Qwen3.5-9B under the full configuration. However, this comparison varies both the model and the Experience setting, so it cannot isolate the contribution of external experience. The corresponding within-model comparisons are 19.2 → 20.1 (+0.9) for Qwen3.6-35B-A3B and 13.1 → 14.9 (+1.8) for Qwen3.5-9B. Both comparisons support the additional benefit of external experience.

3. Appendix A.2 now includes complete per-round results, including deterministic wet-experiment scores.

Table 6 in Appendix A.2 of the revised manuscript reports ten rounds of evaluation for Qwen3.6-35B-A3B on RCB, GDPVal, SGI-DR, and SGI-Wet. The wet-experiment evaluation uses the “w/o experience” configuration and follows the benchmark’s deterministic scoring rules.

The scores across the ten rounds are 23.37, 25.66, 25.91, 27.02, 30.72, 35.77, 41.62, 48.83, 52.21, and 50.01. Performance peaks at 52.21 in round 9, matching the result for this configuration in Table 2. The round-10 score is 50.01, still 26.64 points above the first round. These results show that improvements also hold under deterministic evaluation without an LLM judge, while revealing performance fluctuations in later rounds. The full RSIE score of 56.90 in Table 2 uses external experience and therefore corresponds to a different inference configuration.

4. Table 9 in the original manuscript, now Table 12, includes complete model names and explicit score labels.

The revised table separately reports the base-model score, the score after SFT, and the corresponding change. The mapping between models and scores remains unchanged.

5. We have added complete model names, parameter counts, and evaluation checkpoint details.

Qwen3.6-35B-A3B uses an MoE architecture with 35B total parameters and 3B active parameters per token. Qwen3.5-9B is a dense model with 9B parameters. Their base checkpoints come from the official repositories Qwen/Qwen3.6-35B-A3B and Qwen/Qwen3.5-9B, respectively. RSIE evaluations in this paper use checkpoint-2000, obtained after 2,000 post-training steps.

AuthorsOpenReview ↗

Answer of Q3 To Reviewer 8jNP (Part 1/2)

Q3. Cross-domain transfer of validated skills, retrieval coverage, and comparison with RSI methods

We thank the reviewer for this valuable suggestion. We provide additional evidence through two routes: parameter internalization and external experience reuse. After experience accumulation and OPSD training exclusively on Math tasks, the model improves by 5.9 and 7.0 points on the fully held-out Life and Earth domains, respectively, without external experience at test time. In the standard RCB evaluation, external experience is loaded for every task, and enabling it further improves the average score in all ten domains. RSIE also achieves the highest score in our additional comparison with representative RSI methods on expert research tasks, outperforming the strongest baseline by 5.0 points.

1. Validated experience supports capability improvement in domains excluded from learning.

Following the reviewer’s suggested setting of extracting experience from some domains and evaluating on others, we started from the same Qwen3.6-35B-A3B base model and selected 98 Math-related tasks within the RSIE pipeline. Only execution feedback from these tasks was used to extract and validate experience, and only experience passing the admission criterion was used for OPSD parameter updates. Life and Earth seeds were excluded from training task retrieval, and all tasks derived from those seeds were excluded from experience extraction, selection, and parameter updates.

Evaluation used the original official RCB test tasks and evaluation protocol, with all test tasks kept separate from training tasks. External experience was disabled at test time to directly assess cross-domain capabilities retained in the model parameters after OPSD training.

Model / Training settingMath ↑Life ↑Earth ↑
Qwen3.6-35B-A3B base model10.08.515.0
RSIE trained only on selected Math tasks22.514.422.0
Improvement+12.5+5.9+7.0

Although expert research tasks involve different domain knowledge, they share requirements such as planning analytical steps, checking assumptions, validating results, and diagnosing and correcting errors based on feedback. These processes provide learning content that can be reused across domains. Life and Earth both improve despite being excluded from experience accumulation and receiving no external experience at test time. The setup and results of this held-out domain experiment have been added to Appendix A.5.3, Table 11 of the revised manuscript.

2. External experience is retrieved and loaded for 100% of tasks in the final RCB evaluation.

When experience is enabled, each test task retrieves and loads the highest-similarity eligible entry from the experience bank. Thus, defining the loading rate as the proportion of test tasks for which experience is actually loaded, coverage is 100% both overall and within each of the ten RCB domains.

This rate describes the coverage of external experience. To assess its actual contribution, we additionally compare the same model with and without external experience. The retrieval settings have been clarified in Appendix A.5.2.

3. Comparisons on the same tasks show additional benefits from external experience across all ten RCB domains.

As clarified in our response to Q2, 19.2 and 20.1 correspond to the same trained model with external experience disabled and enabled, respectively. We fixed the model checkpoint and evaluation settings, executed both configurations on the same test tasks, and aggregated results over the ten original RCB domains, each containing four tasks.

RCB domainBase modelRSIE w/o experienceRSIE w/ experienceGain from external experience
Astronomy23.732.934.7+1.8
Chemistry3.09.610.3+0.7
Earth15.029.030.6+1.6
Energy18.424.425.5+1.1
Information5.512.913.3+0.4
Life8.59.711.1+1.4
Material12.318.919.2+0.3
Math10.022.022.3+0.3
Neuroscience1.16.87.2+0.4
Physics23.525.826.8+1.0
Mean12.119.220.1+0.9

This table reports the standard RCB evaluation after full RSIE training, separately from the Math-only held-out domain experiment in point 1. Without external experience, RSIE already outperforms the base model in all ten domains. Enabling experience further improves domain-level average scores by 0.3–1.8 points. These results break down the overall gains by domain and support the complementary roles of parameter learning and external experience. The complete comparison has been added to Appendix A.5.2.

Reviewer_2CwCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper's method RSIE focuses on recursive self-improvement through interaction with expert environments. The RSIE method repeatedly executes expert tasks, extracts from trajectories and their corresponding eval feedback reusable experience, and only retains such experience when reusing it leads to improvement. The paper demonstrates improvements with this method on scientific and professional benchmarks.

Critical strengths

The paper demonstrates results and improvements on multiple recent models and benchmarks as well as provides hyperparameter details and different ablations.

Critical weaknesses

The proposed framework lacks significant novelty. It largely combines existing techniques that have already been employed in prior self-improvement for LLM papers. The paper moreover lacks comparison against relevant RSI-style methods like ReasoningBank (https://arxiv.org/abs/2509.25140) for instance.

Questions for authors
  1. Can the authors please clarify the methodological novelty of their approach with respect to prior work in LLM self-improvement?
  2. Could the authors compare their approach against relevant RSI-style methods like ReasoningBank (https://arxiv.org/abs/2509.25140) for instance under equal wall clock and compute budgets?
  3. Could the authors perform more detailed stratification of results across components of the method to better attribute gains to the proposed method?
4 comments
AuthorsOpenReview ↗

Answer of Q1 To Reviewer 2CwC

Thank you for the valuable time and feedback you have invested in this review. We sincerely hope our further clarification could address your concerns:

Q1. Methodological novelty of RSIE for recursive Self-Improvement on expert tasks

Our work studies recursive self-improvement for expert tasks, focusing on a central challenge: when execution is expensive, feedback centers on final deliverables, and learning on familiar tasks gradually saturates, how can we continue to obtain useful learning signals and translate them into transferable model capabilities? Our approach constructs and refreshes expert task environments, selects and organizes experience from execution feedback, and combines internal training for capability development with external indexing for knowledge access. Below, we position our work relative to prior studies along three dimensions.

1. Research scope: task-driven model self-improvement.

R-Zero introduces a Challenger–Solver co-evolution framework that autonomously generates learning tasks and updates the model. Agent0 further incorporates tool use, allowing curriculum generation and execution capabilities to improve together. These studies establish a loop between autonomous task generation and model improvement. However, this loop alone does not ensure that generated tasks capture the professional requirements of real expert work. Expert tasks often depend on specific data, tools, and workspaces, and their quality must be assessed through the resulting deliverables. Extending autonomous learning to expert domains therefore requires additional attention to task authenticity, material completeness, and outcome evaluability.

To address these requirements, FrontierWorld constructs environments from completed scientific studies and professional deliverables, grounding learning tasks in problems that experts have actually solved and the artifacts they produced. It checks whether each task can be instantiated, whether the required materials are complete, whether the outcomes are verifiable, and whether the boundary between public and hidden materials is properly defined. This provides grounded learning tasks for post-training expert capabilities.

2. Core challenge: sustained self-improvement under costly interaction.

Dream-RSI uses historical search trees as replay environments to improve exploration policies at lower cost. DGM modifies the agent’s own code and combines benchmark evaluation with an archive of candidate agents to explore better implementations. These approaches advance the reuse of existing experience and the improvement of execution mechanisms. A further question is how to sustain fresh learning signals as the available tasks become saturated.

In model post-training for expert tasks, repeatedly executing familiar tasks can produce highly overlapping experience, so additional interaction costs may yield diminishing learning value. RSIE therefore uses execution history to refresh active tasks when they reach full scores or exhibit persistent stagnation, allowing a limited interaction budget to cover new professional requirements. As shown in Figure 5, task refreshing outperforms the fixed-task setting on held-out evaluations in later stages, supporting the value of continually updating tasks in this setting.

3. Proposed approach: experience organization and selective internalization.

ReasoningBank distills and retrieves experience from successes and failures at test time, allowing subsequent requests to reuse learned strategies. KOPE organizes decisions, correctness feedback, and performance feedback from hardware optimization into an experience graph that guides subsequent optimization. Both approaches improve the execution of fixed models through external experience. We further study how such experience can be translated into model capabilities.

Our analysis shows that more execution does not necessarily produce more useful learning content. An audit of trajectories from repeated execution reveals a skill overlap of 87.3% (Appendix A.6.1). Directly replaying these trajectories can overemphasize repetitive, task-specific behaviors while providing limited new learning signals. RSIE therefore organizes experience into four categories according to its scope of applicability. Only domain skills and general workflows are used for OPSD; task-specific facts and conditional error-prevention guidance remain externally accessible for use when needed. This keeps parameter updates focused on transferable methods and workflows. As shown in Table 3, introducing this categorization reduces skill overlap from 87.0% to 56.7% and improves the RCB score from 18.4 to 19.2.

AuthorsOpenReview ↗

Answer of Q2 To Reviewer 2CwC

Q2. Comparison with representative RSI methods

We have added comparisons with three representative methods: RSIGym (Data Track) [1], which improves models through autonomous training data construction; Dream-RSI [2], which improves task execution by recursively optimizing exploration policies; and ReasoningBank [3], which supports self-improvement through experience accumulation and retrieval. Together, these methods cover three complementary aspects: data, exploration policies, and external memory.

All methods start from the same Qwen3.6-35B-A3B base model and are evaluated on an independent ResearchClawBench (RCB) test set under a shared 100-hour runtime limit, covering data generation, decision-making, training, and evaluation. We use GPT-5.4 consistently for all components requiring an LLM judge. The specific configurations are as follows:

  • RSIGym (Data Track): The research agent autonomously generates the same number of training samples as our method, and the target model is updated using a fixed LoRA SFT recipe. The research agent and execution framework remain fixed.
  • Dream-RSI: Using our training tasks and their environments, this method explores candidate solutions and replays historical exploration trees to optimize decisions about branch selection, further refinement, and stopping. Model weights remain fixed, and the exploration policy is frozen before testing.
  • ReasoningBank: Successful and unsuccessful experiences are extracted from execution trajectories on our training tasks to construct a memory bank, without updating model parameters. At test time, the method only retrieves from this bank; test trajectories are not written back into memory.
  • RSIE: Following our proposed pipeline, RSIE refreshes tasks, selects and categorizes experience, updates the model through OPSD, and uses external experience retrieval when solving test tasks.

To preserve test-set independence, Dream-RSI and ReasoningBank use the transfer adaptations described above. RCB test tasks and their feedback are excluded from data construction, model selection, policy updates, and experience extraction.

MethodRCB score ↑
Qwen3.6-35B-A3B12.1
RSIGym-style Data14.6
Dream-RSI (transfer adaptation)13.7
ReasoningBank (frozen memory bank)15.1
RSIE (ours)20.1

RSIE achieves the highest score in this comparison, improving over the initial model by 8.0 points and over the strongest baseline, ReasoningBank, by 5.0 points. These results support the effectiveness of combining task refreshing, experience organization, and model post-training for expert tasks under the evaluated setting.

Note: RSIGym includes Data, Harness, and Joint tracks. We use the Data Track because its focus on updating models through training data construction is most closely aligned with our model post-training setting.

References

[1] Meng et al. RSIGym: A Flexible Environment for Recursive Self-Improvement.

[2] Zheng et al. Dream-RSI: Recursive Self-Improvement through Evolving Worlds.

[3] Ouyang et al. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory.

AuthorsOpenReview ↗

Answer of Q3 To Reviewer 2CwC

Q3. Contributions of task data, experience organization, parameter learning, and test-time retrieval

To clarify how each component contributes to capability improvement, we have reorganized the existing results along four dimensions: task data, experience organization and introduction, parameter learning, and test-time retrieval. All scores below are reported on ResearchClawBench (RCB).

1. Task data: expanding task coverage and refreshing saturated tasks.

Table 4 shows that both models improve as the amount of expert task data increases:

Model64 examples140 examples400 examples600 examples
Qwen3.5-9B8.9710.5612.7113.10
Qwen3.6-35B-A3B14.2016.1017.8019.20

Increasing the data size from 400 to 600 examples yields further gains of 0.39 and 1.40 points for the 9B and 35B models, respectively. Figure 5 further compares fixed and refreshed task pools, both maintaining 600 active tasks. The fixed pool reaches a plateau earlier, whereas refreshing the pool sustains higher performance in later rounds. Specifically, a task is replaced by one from the reserve pool when it reaches the maximum score or fails to exceed its historical best score over three consecutive valid executions.

These results support our task supply design: broader expert task coverage improves model capabilities, while continued task refreshing supports further improvement beyond repeatedly optimizing a fixed set of tasks.

2. Experience organization and introduction: determining what to learn and when to learn it.

Table 3 provides the following comparisons on Qwen3.6-35B-A3B:

ComponentControlProposed settingGain
Four-category experience classificationWithout classification: 18.4With classification: 19.2+0.8
Experience representationChecklist variant: 18.7Full execution guidance: 19.2+0.5
Timing of experience introductionAll at once: 16.9Progressive: 19.2+2.3

Experience classification separates experience into task-specific knowledge, domain skills, general workflows, and failure avoidance. Only domain skills and general workflows are used for parameter updates. Introducing this classification reduces skill overlap from 87.0% to 56.7%, alongside improved performance.

Full execution guidance specifies operational steps, deliverable requirements, verification procedures, and recovery strategies following failures, making the experience directly actionable in subsequent execution. This representation improves performance by 0.5 points over the checklist variant.

Progressive introduction distributes the same set of verified experiences across multiple improvement rounds. The model applies an initial subset, undergoes evaluation and updating, and then receives the next subset. The all-at-once control introduces the entire set together. With the experience set and total data volume held constant, progressive introduction improves performance by 2.3 points.

Together, these comparisons indicate that organizing both the content and timing of experience enables more effective use of existing interactions and improves transfer performance.

3. Parameter learning: comparing trajectory imitation, preference learning, and experience-conditioned self-distillation.

Appendices A.2 and A.5–A.6 provide the following results:

ModelInitial modelSFTDPORSIE (OPSD, retrieval disabled)
Qwen3.5-9B8.836.029.2213.1
Qwen3.6-35B-A3B12.113.4—19.2

SFT imitates historical execution trajectories that have passed quality checks and deduplication. DPO uses preference pairs constructed from scores assigned to candidate trajectories on training tasks. In RSIE’s OPSD, the student generates responses under the standard task context, while a teacher additionally conditioned on domain skills and general workflows provides the distillation signal. This allows the model to retain the corresponding capabilities without explicit experience prompts. DPO was evaluated only on the 9B model. Under the evaluated configurations, the RSIE training pipeline with OPSD achieves the best results, outperforming DPO by 3.88 points on the 9B model and SFT by 5.8 points on the 35B model. These findings support the effectiveness of using transferable experience for model post-training.

4. Test-time retrieval: providing additional gains beyond parameter learning.

ModelRSIE without retrievalRSIE with retrievalGain
Qwen3.5-9B13.114.9+1.8
Qwen3.6-35B-A3B19.220.1+0.9

With retrieval disabled, both models already substantially outperform their initial versions, indicating that capability gains are retained in the model parameters. Adding retrieval yields further improvements of 1.8 and 0.9 points, respectively. These results support the complementary roles of parameter updates and external experience retrieval, with the full RSIE framework achieving better performance at both model scales.

Dear Reviewer,

Thank you again for your thoughtful feedback. We hope our response above has helped clarify the points you raised. We would be grateful if you could give us the opportunity to address any further questions or concerns you may have. If any concerns remain, we would be happy to provide additional clarification.

Thank you for your time and consideration.

Reviewer_7GuoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether an agent can continually improve from a small number of expert-task interactions when each execution is costly and feedback arrives only once, at completion. RSIE extracts reusable experience from completed executions, screens it through subsequent reuse, and incorporates it into external guidance and model parameters through on-policy self-distillation. FrontierWorld replaces saturated tasks to focus subsequent interactions on the agent's evolving capability frontier.

Critical strengths
  • The paper extends the experimental scope of RSI to costly expert workflows, including hour-long tasks assessed through external rubrics. This makes learning from sparse terminal feedback a concrete research setting for RSI in scientific and professional work.
  • Connecting experience memory with parameter learning offers a useful way to study how episodic experience becomes reusable capability. Evaluation without retrieval makes this question empirically testable by examining what the model retains after explicit guidance is removed.
Critical weaknesses
  • To decide whether an individual candidate experience is worth retaining, the admission rule uses the task's historical best across cycles. It does not directly compare the current model's performance on the same task with and without that candidate.
  • Figure 4 is labeled as cumulative best scores, yet the SGIBench curve decreases. The paper does not provide the underlying per-cycle scores, so the stability of the reported gains is difficult to assess.
Questions for authors

How does FrontierWorld expand skill coverage beyond the seed-task distribution as its task pool grows? Do the gains persist when evaluation holds out entire seed-task families, including their derived tasks?

6 comments
AuthorsOpenReview ↗

Answer of Q1 To Reviewer 7Guo

Thank you for your thoughtful questions about our comparisons and uncertainty estimates.

Q1. Comparison of skill selection criteria: best-so-far scores and paired evaluation with and without skills

We thank the reviewer for this valuable suggestion. We use the historical best score as the experience admission threshold to connect newly introduced learning content with measurable progress on expert tasks. Together with task refreshing in FrontierWorld, this design aims to make more effective use of costly interactions. Below, we explain its motivation and effects through an iterative analysis of a single task and an additional comparison against paired evaluation with and without candidate experience.

1. Selecting experience based on continued task progress.

Expert tasks involve complex analysis and execution, and experience distilled from initial feedback often requires refinement through subsequent practice. Taking Math_003 from RCB40 as an example, we conducted eight rounds of self-improvement training. In each round, we refined the guidance using existing experience and new execution results, obtaining the following task scores:

Round12345678
Score14.19.912.415.517.216.817.017.1

Early execution shows substantial fluctuations, suggesting that the initial feedback and distilled guidance were insufficient to address the task’s difficulties. With continued refinement based on subsequent execution results, the score reaches 17.2 in round 5, improving over the first round by 3.1 points, and remains within 16.8–17.1 over the next three rounds. This case illustrates how useful experience for expert tasks can develop through repeated execution; an initial distillation does not necessarily provide sufficiently effective guidance.

We therefore admit candidate experience into subsequent learning only when its reuse produces a score that strictly exceeds the historical best on the same task under unchanged evaluation criteria. This threshold ties experience admission to new task progress and avoids treating recovery from an earlier decline as a new performance breakthrough.

2. Continually expanding learning coverage through FrontierWorld.

RSIE combines experience refinement within tasks with expanded learning coverage across tasks. While a task continues to improve, execution feedback is used to refine the experience. Once the task reaches the maximum score or fails to exceed its historical best over three consecutive valid executions, FrontierWorld replaces it with a new expert task.

The example above also illustrates this process: after reaching a new best in round 5, the following three rounds yield no further improvement, meeting the stagnation criterion for task refreshing. Experience selection retains learning content associated with progress, while task refreshing introduces new professional requirements, allowing a limited interaction budget to support continued capability improvement.

3. Balancing validation effectiveness and execution cost.

Comparing the current model with and without candidate experience directly measures the experience’s benefit to current execution. This requires execution results under both conditions. When no reusable result from the current model without experience is available, an additional control execution is needed. For expert tasks where a single execution can take hours, this adds substantial validation cost. Our rule instead reuses historical scores as the reference, allowing subsequent execution with experience to serve both task improvement and experience validation.

To further examine this choice, we added a control experiment using the same amount of data as the original setting. For each candidate experience, we performed one paired evaluation with and without it, holding the current model, task environment, and evaluation criteria fixed. Only candidates yielding a positive score improvement were retained, and the rest of the pipeline remained unchanged.

Experience admission ruleRCB score ↑
One paired evaluation of the current model with and without candidate experience17.4
Historical best-score threshold (ours)20.1

Our rule achieves an RCB score of 20.1, outperforming the single paired-evaluation criterion by 2.7 points. These results suggest that selecting experience against historical task progress provides more effective learning content in this setting and leads to higher final performance within the full RSIE pipeline.

AuthorsOpenReview ↗

Answer of Q2 To Reviewer 7Guo

Q2. Per-cycle performance, experience saturation, and task refresh during recursive improvement

We thank the reviewer for pointing out the inconsistency between the curves in Figure 4 and the definition of cumulative best performance. Figure 4 is intended to show the best score achieved up to each round. This definition remains unchanged, and we will check and correct the corresponding plots. To distinguish cumulative progress from fluctuations in actual execution, we have added the raw per-round scores in Appendix A.2, Table 6 of the revised manuscript. On SGIBench, most gains occur in the early rounds. Later raw scores fluctuate at a relatively high level, while the cumulative best remains at the highest score already achieved.

1. Detailed per-round evaluation results.

Appendix A.2, Table 6 now provides the complete raw scores for all four evaluations across rounds. Below, we report the three evaluations corresponding to Figure 4. GDPVal is reported using continuous judge scores.

RoundRCB ↑GDPVal ↑SGI-DR ↑
112.116.948.43
213.020.050.6
312.723.853.9
415.623.559.5
515.223.259.1
619.228.658.2
718.935.558.6
818.735.158.5
919.035.058.0
1019.135.358.6

The main gains on SGI-DR occur within the first four rounds, with performance increasing from 48.43 to 59.5. Over the subsequent six rounds, scores fluctuate between 58.0 and 59.1 without exceeding this peak, indicating saturation in later rounds.

2. Increasingly repetitive experience provides fewer new learning signals.

To better understand this finding, we examined how execution feedback and candidate experience evolved. Early experience provides concrete remedies for distinct execution problems. As common issues are resolved, subsequent experience increasingly repeats existing checking procedures and offers limited new guidance for the remaining domain-specific judgments. The following case illustrates this pattern:

In SGI-DR training task A, “Comparing material conductivity across studies,” the agent initially overlooked measurement conditions reported in supplementary materials and directly compared values expressed in different units. The distilled experience instructed the agent to recover missing information, standardize units, and record the material state, measurement conditions, and source for each value. Reusing this experience improved the task score to 54.

The remaining difficulty was then to determine whether results obtained under different material phases, preparation procedures, and measurement conditions were comparable. However, subsequent candidate experience continued to emphasize “cross-checking sources” and “verifying experimental conditions,” substantially overlapping with earlier guidance. These instructions called for further checks but did not explain which differences would invalidate a comparison or how the conclusions should be adjusted accordingly. The revisions therefore failed to provide a new approach to the remaining difficulty. The next three valid executions scored 48, 52, and 51, with no improvement on the relevant evaluation criteria or over the historical best.

This case illustrates that continued experience generation does not necessarily provide new useful learning content. When subsequent guidance becomes repetitive and fails to address additional capability gaps, the benefits of experience refinement can saturate, and further execution is not guaranteed to exceed the previous best.

3. Refreshing tasks based on progress introduces new capability requirements.

FrontierWorld replaces a task with a preregistered new task when it reaches the maximum score or fails to exceed its historical best over three consecutive valid executions. New tasks allow the agent to reuse acquired experience while exposing different capability gaps, enabling subsequent interactions to produce new learning content.

Experience selection and task refreshing thus connect experience refinement within tasks with expanded learning across tasks. Validated experience is retained, while new professional requirements are introduced once progress on the current task saturates, providing fresh learning signals for subsequent capability improvement.

AuthorsOpenReview ↗

Answer of Q3 To Reviewer 7Guo

Q3. Skill coverage expansion in FrontierWorld and generalization to held-out task families

FrontierWorld retrieves new real-world scientific literature to expand task and skill coverage beyond the initial seed tasks. In an additional experiment, training RSIE exclusively on Math tasks improves performance on the fully held-out Life and Earth task families by 5.9 and 7.0 points, respectively. These results support the transferability of the acquired capabilities to task families excluded from learning. Below, we explain the task expansion mechanism and the cross-domain transfer results.

1. Seed-guided retrieval expands task and skill coverage through new source material.

FrontierWorld uses the domains and semantic content of seed tasks to generate retrieval queries at different levels of granularity, identifying relevant scientific literature and professional deliverables. Task construction is then grounded in the newly retrieved and screened sources: task instructions are derived from their actual research questions, workspaces are assembled using public data, code, and supporting materials, and hidden evaluation criteria are constructed from the reported findings.

The specific requirements of each new task therefore come from the expert work documented in its source. Topical similarity does not imply identical skill requirements: differences in research methods, tools, and validation procedures expose the model to professional requirements beyond those covered by the initial seeds.

For example, studies on “material conductivity” involve organizing experimental data, fitting physical models, and running numerical simulations. Constructing tasks from these studies introduces different skills: data cleaning and condition checking, parameter estimation and fit diagnostics, or simulation configuration and result validation. The seed provides a shared retrieval direction, while the new sources determine the specific work the model must perform.

This construction process turns the diversity of real expert work into learning content, allowing task pool expansion to introduce new skill requirements beyond simply increasing the number of similar tasks.

2. Held-out domain evaluation demonstrates transfer to task families excluded from learning.

Starting from the same Qwen3.6-35B-A3B base model, we selected Math-related tasks within the RSIE pipeline, obtaining 98 tasks. Only execution feedback from these tasks was used for experience extraction, validation, and parameter updates. Life and Earth seeds were excluded from training task retrieval, and all tasks derived from those seeds were excluded from experience extraction, selection, and parameter updates.

We evaluated on the official benchmark’s original test tasks, reporting results separately for Math, Life, and Earth under the official evaluation protocol. All test tasks were kept separate from training tasks. Experience retrieval was disabled at test time to directly assess capabilities retained in the model parameters.

Model / Training settingMath ↑Life ↑Earth ↑
Qwen3.6-35B-A3B base model10.08.515.0
RSIE trained only on selected Math tasks22.514.422.0
Improvement+12.5+5.9+7.0

Training exclusively on Math tasks improves Math performance by 12.5 points and also yields gains of 5.9 and 7.0 points on the fully held-out Life and Earth families. Although expert research tasks require different domain knowledge, they commonly involve planning experiments or analytical steps, checking assumptions, validating results, and diagnosing and correcting errors based on feedback. The experimental setup and results have been added to Appendix A.5.3.

For example, a workflow developed through Math tasks—identifying variables and assumptions, planning solution steps, checking boundary conditions, validating results, and locating errors—can provide reusable guidance. In Life tasks, such a workflow can help organize experimental analysis, check statistical assumptions, and verify conclusions. In Earth tasks, it can support checks of variable units, model applicability, and computational results. The domain-specific objects change, while the methods for planning, analysis, and error correction remain useful.

These results indicate that RSIE’s benefits extend beyond the task families used for training. Experience accumulated from expert tasks can be translated into transferable model capabilities, improving performance in held-out domains even without test-time experience retrieval.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal summary (Part 2/2)

4. Evaluation reliability and result stability. [7Guo; 8jNP]

Reviewers raised concerns about dependence on a single LLM judge and the stability of the improvements.

We rescored the same artifacts using judges from the DeepSeek and Gemini model families. All three benchmarks preserve the ranking RSIE > SFT > base model. Independent assessments of RCB artifacts by two human experts also confirm the improvement, with a mean gain of 7.5 points. Furthermore, expanding the deterministic wet-experiment evaluation from 68 to 200 tasks yields a 35.62 percentage-point improvement over the base model. Together, evaluation across judge families, human expert assessment, and deterministic scoring support improved task performance.

To assess stability, we added complete raw per-round scores for all four evaluations, distinguishing cumulative best performance from execution fluctuations. Case studies further examine increasingly repetitive experience and diminishing gains in later rounds. Five runs on RCB preserve the ranking observed in the main experiments. Full RSIE achieves a GDPVal Elo of 1339, with a task-level bootstrap 90% confidence interval of [1300, 1380].

5. Clarifications to experimental settings and presentation. [8jNP]

We standardized model names and Experience labels across tables, clarified that 19.2 and 20.1 correspond to the same trained model with external experience disabled and enabled, and added model sizes, checkpoint details, and the retrieval protocol. Table 6 in the revised manuscript provides complete per-round results for all four evaluations, while Table 12 clarifies the correspondence between models and SFT scores.

We also added computational resource and model-call statistics and acknowledged the potential risk of pretraining contamination from public source materials. We clarified that our study examines subsequent capability improvement from a shared pretrained starting point.

Thank you again for your time and constructive suggestions. Your feedback has helped us present the methodological contributions more clearly and further examine the effectiveness, transferability, and evaluation reliability of experience learning.

Sincerely,
The Authors

AuthorsOpenReview ↗

Rebuttal summary (Part 1/2)

Rebuttal summary

Dear Area Chair and Reviewers,

Thank you for your careful evaluation of our work. To facilitate your assessment of this rebuttal, we summarize the paper’s contributions, our responses to the main concerns, and the key additional results below.

Our paper introduces RSIE, a recursive model post-training framework for expert tasks. These tasks are expensive to execute, feedback centers on final deliverables, and repeated execution of familiar tasks produces increasingly redundant learning content. RSIE addresses these challenges by constructing and refreshing grounded expert tasks through FrontierWorld, selecting and organizing experience from execution feedback, and combining internal training for capability development with external indexing for knowledge access.

Our responses provide additional evidence on method comparisons, experience selection, cross-domain transfer, and evaluation reliability.

1. Methodological contributions and comparisons with related work. [2CwC; 8jNP]

Reviewers asked us to clarify RSIE’s methodological contributions relative to existing self-improvement approaches and provide comparisons with representative methods.

We clarified three contributions: constructing executable and evaluable learning tasks from real expert work; refreshing learning content when tasks saturate; and selecting experience that supports new progress while selectively translating it into model capabilities.

We also added comparisons with RSIGym (Data Track), Dream-RSI, and ReasoningBank, covering autonomous data construction, exploration policy optimization, and external memory accumulation, respectively. Under the same base model, an independent RCB test set, and a shared 100-hour runtime limit, RSIE achieves 20.1 points, improving over the base model by 8.0 points and the strongest baseline by 5.0 points.

2. Component contributions and the rationale for experience selection. [2CwC; 7Guo]

Reviewers sought a clearer account of which components drive the gains and why experience admission uses the historical best score as its threshold.

We reorganized the existing results around task data, experience organization, parameter learning, and test-time retrieval. Data scaling and fixed-versus-refreshed task pool comparisons support the value of expanding coverage of professional requirements. Experience classification reduces skill overlap from 87.0% to 56.7% and improves RCB performance from 18.4 to 19.2. With the experience set and total data volume held constant, progressive introduction outperforms all-at-once introduction by 2.3 points. The RSIE training pipeline with OPSD also outperforms the evaluated SFT and DPO configurations.

For the admission rule, we added a direct comparison against a single paired evaluation of the current model with and without candidate experience. The historical best-score threshold achieves 20.1, compared with 17.4 for the paired-evaluation criterion. This result supports tying experience admission to new task progress. Execution case studies further illustrate how initial guidance requires refinement through practice, and how task refreshing introduces new learning requirements once experience becomes repetitive.

3. Cross-domain transfer and the contribution of external experience. [7Guo; 8jNP]

Reviewers asked whether task expansion introduces new skill requirements and whether experience learning improves performance on task families excluded from training.

We added a strict held-out domain experiment using only 98 Math-related tasks for experience accumulation and parameter updates. Life and Earth seeds and all tasks derived from them were excluded from learning. With external experience disabled at test time, performance still improves by 5.9 points on Life and 7.0 points on Earth, supporting transfer to domains excluded from learning.

We also added retrieval comparisons across all ten domains in the standard RCB evaluation. The same trained model scores 19.2 without external experience and 20.1 with it, with additional gains in every domain. These experiments separately demonstrate capabilities retained through parameter learning and the complementary value of external experience during execution. The held-out domain results and ten-domain comparisons appear in Tables 11 and 9 of the revised manuscript, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.