acceptodds
Under review as a conference paper at ICLR 2027

Dream-RSI: Recursive Self-Improvement through Evolving Worlds

21 views

Abstract

Recursive self-improvement is becoming essential for autonomous AI agents, whose progress depends on discovering high-value solutions across complex domains. Effective exploration drives this process, yet managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization must navigate vast meta-search spaces under delayed, expensive feedback from long-horizon rollouts. We introduce Dream-RSI, a framework for scalable, recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying base agent unchanged. Our key insight is that accumulated discovery history can act as a replay simulator over the realized search space. By dreaming within this simulator built from historical discovery trees, Dream-RSI obtains immediate, low-cost off-policy feedback to evaluate and refine exploration policies without repeated, expensive online evaluation. The improved policy is then redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across 9 tasks in 4 domains, Dream-RSI achieves competitive quality and improves discovery efficiency in several settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recursive Self-Improvement, Scientific Discovery, Exploration Strategy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kLG2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to improve the exploration policy of LLM discovery agents, i.e., the rules that decide which attempts to refine, how many to run in parallel, and when to stop. The authors propose Dream-RSI, which writes this policy as code and improves it between rounds of discovery. The tree of attempts recorded in each round is reused as a “replay simulator.” A candidate policy is scored by replaying it over the recorded trees, which reveals only already stored outcomes, so no new agent calls are needed. An LLM writes three new versions of the policy each round. The best version by replay score is used in the next round, and the score rewards reaching high results with few calls and in parallel. Across nine tasks in four domains Dream-RSI matches or beats the same agent with a fixed policy while using fewer agent calls.

Critical strengths
  • Reusing completed discovery runs to evaluate new exploration policies without new agent calls is a simple and practical idea. It addresses the slow, expensive feedback that makes improving exploration policies hard.
  • The main baseline, Recursive Fixed Exploration, is well controlled. It uses the same agent, evaluator, starting policy, and per-round budget, and differs only in never updating the policy.
  • The evaluation covers diverse domains, and the paper includes full prompts and the discovered programs.
Critical weaknesses

Main concerns are:

  • The replay simulator is biased and never validated: Replay only reveals outcomes already stored in the recorded trees. It can never reward exploring beyond what was recorded, only reaching recorded results with fewer calls, which favors pruning and stopping early. Recorded outcomes also depend on the original order of exploration, since the agent reads all earlier and sibling attempts before proposing (Listing 1). The paper never checks whether replay scores predict online results.
  • Experiments show small or mixed gains, all from single runs, and no task reports repeated discovery runs or variance.
  • Call savings may come from simple early stopping: The replay score directly rewards fewer calls. On VGG16 and LayerNorm, the final scores are about the same. Fixed exploration improves only about 1–3% after roughly 450 calls, so a simple stopping rule might give similar savings. The paper compares with neither such a rule nor rewritten policies deployed without replay selection. It also does not count the calls of the LLM that rewrites the policy.
Questions for authors
  • Do replay scores predict online results? For example, could the authors run several candidate policies online and report how well their replay scores match their online results?
  • How does Dream-RSI compare with fixed exploration plus a simple stopping rule (e.g., stop after a round without improvement), and with rewritten policies deployed without replay selection?
  • Could the authors report independent discovery runs per task, with variance?
  • How many calls and tokens does the LLM that rewrites the policy use? How do total cost and wall-clock time compare with fixed exploration?
4 comments
AuthorsOpenReview ↗

Author Response (1/n)

We thank the reviewer for recognizing the practicality of replay-based policy evaluation, the controlled baseline, and the breadth of our evaluation. We address your concerns below regarding replay validity, early stopping and replay-based selection, statistical robustness, and total computational overhead.

Regarding replay validity and limitations (Weakness 1; Question 1)

(a) Do replay scores predict online performance?

We have added an online validation experiment to examine whether replay-score rankings agree with online performance.

Setting. Using the first-round history from Lasso-Flash, we generate three candidate policies through dreaming and evaluate each independently for one online round. The online final score is the highest score achieved during the round, computed as the inverse geometric mean of solver runtimes over the 17 synthetic evaluation instances used during discovery; solvers that fail any correctness check receive a score of zero.

Results. The replay-score ranking largely agrees with the online final-score ranking across these three candidates. The selected policy achieves the highest online final score with an intermediate number of calls: Policy 1 uses fewer calls but achieves lower quality, while Policy 2 uses more calls than the selected policy yet achieves lower quality.

TaskCandidate PolicyReplay Score ↑Online Final Score ↑Online Calls
Lasso-FlashPolicy 10.880.122171
Lasso-FlashPolicy 20.900.125278
Lasso-FlashPolicy 3 (selected)0.920.130255

(b) Recorded coverage and dependence on exploration history

We acknowledge that replay evaluates policies only on recorded nodes and outcomes; it cannot evaluate unrecorded branches or reward improvements beyond the recorded outcomes. We have added this in the limitation section of our current version of paper.

However, we clarify that the solution-generation agent does not observe attempts from other branches within the current round. Its context consists of the node’s ancestor chain and any previous-round history, which is fixed for that recorded round. Replay strictly preserves ancestor dependencies: a child is revealed only after its parent has been reached. Thus, changing the order of exploration across branches does not change the generation context associated with a recorded node. The orchestration policy observes results as they are revealed, but these observations do not become additional cross-branch context for the solution-generation agent.

2. Do the gains come from early stopping or iterative LLM rewriting alone? (Weakness 3; Question 2)

(a) How does Dream-RSI compare with fixed exploration plus a simple stopping rule?

Following your suggestion, we apply a simple stopping rule to the recorded fixed-exploration runs: stop after one completed round without improvement in the best-so-far score. We show the results below. We can see this approach achieves comparable results on LayerNorm but terminates prematurely on ConvDiv, missing later improvements. Dream-RSI achieves a higher score with fewer calls on ConvMax and a similar score with substantially fewer calls on VGG16. These results show that this simple stopping rule can be effective on some tasks but does not consistently reproduce Dream-RSI’s quality–cost trade-off.

TaskMethodFinal Score ↑Budget (Calls)
VGG16Fixed + Early Stopping (Patience = 1)0.534759358880
VGG16Dream-RSI0.531914894408
LayerNormFixed + Early Stopping (Patience = 1)1.117318436550
LayerNormDream-RSI1.122334456553
ConvDivFixed + Early Stopping (Patience = 1)0.531914894440
ConvDivDream-RSI1.897533207786
ConvMaxFixed + Early Stopping (Patience = 1)0.299401198770
ConvMaxDream-RSI0.431034483669

(b) Does replay-based selection improve over iterative LLM rewriting alone?

We conducted an additional experiment to isolate the effect of replay-based selection. We keep the seed policy, agents, prompt, replay feedback for rewriting, three rewrites per round, number of rounds, and call budget unchanged, but select the next policy uniformly at random among the four candidates.

We can see that random selection can also improve performance, which may be attributed to LLM-based rewriting guided by replay feedback. Using replay scores to select policies provides further gains, as randomly selected policies may have suboptimal efficiency or performance.

TaskSelection MethodCallsFinal Score ↑
ConvDivRandom Selection8871.55
ConvDivDream-RSI7861.90
AuthorsOpenReview ↗

Author Response (2/n)

3. Independent runs and variability (Weakness 2; Question 3)

The original results were based on one discovery run per setting. Following your suggestion, we added two independent runs for both Recursive Fixed Exploration and Dream-RSI on ConvDiv, ConvMax, and Lasso-Flash, giving three runs per method on each of these tasks. Across these three tasks, Dream-RSI achieves better mean final scores while using fewer discovery calls.

TaskMethodAvg. CallsFinal Score (Mean ± STD)
ConvDiv ↑Recursive Fixed Exploration990.001.3987 ± 0.1515
ConvDiv ↑Dream-RSI570.671.6848 ± 0.2547
ConvMax ↑Recursive Fixed Exploration990.000.5374 ± 0.2669
ConvMax ↑Dream-RSI689.330.6112 ± 0.2240
Lasso-Flash ↓Recursive Fixed Exploration3200.002866.8 ± 320.6
Lasso-Flash ↓Dream-RSI1790.672619.8 ± 323.4

4. Policy-development overhead, total cost, and wall-clock time (Weakness 3; Question 4)

We conduct a cost analysis of the Lasso-Flash experiments. Throughout the discovery process, policy development occurs once between each pair of consecutive rounds, with three LLM calls per update, totaling 12 calls, 66.6M total tokens, and 0.36M output tokens. Including policy development, Dream-RSI uses 40.9% fewer LLM calls and 47.4% fewer total tokens than fixed exploration. Policy development accounts for approximately 0.6% of its total calls and 2.0% of its total tokens.Dream-RSI takes approximately 1.41× as long in active elapsed time (34.31 h versus 24.35 h).

MetricFixed ExplorationDream-RSI: DiscoveryDream-RSI: Policy DevelopmentDream-RSI: Total
LLM calls32001879121891
Total tokens6343M3271M66.6M3337.6M
Output tokens53.0M27.6M0.36M27.96M
Active elapsed time24.35 h30.72 h3.59 h34.31 h
AuthorsOpenReview ↗

Regarding Concerns

Thank you again for your time and constructive feedback. As the discussion deadline approaches, we would appreciate any further feedback or remaining concerns and would be happy to provide additional clarification.

Thank you to the authors for the additional experiments and responses. The online validation, comparison with fixed stopping rule and repeated run, are all helpful results. Through these the authors have largely addressed my concerns so I have raised my score to 3.

Reviewer_bFmxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose an LLM -agent system in which a coding agent explores a tree of programs with the aim to improve the exploration of the policy adjusting such params as which node to expand, the number of parallel jobs and so on. In online phase the recorded tree is added to a pool of trees. In offline phase the LLM modifies the code of policy and the best candidate is evaluated by replaying it over the pool of trees. After the best candidate is selected it is redeployed.

Main contribution points:

  1. Stored search trees are reused as a simulator: a candidate exploration policy is run over them, sees only recorded children, and is scored by result quality per LLM call across several budget settings. This is policy evaluation on logged data, though the paper does not frame it that way or state when it is reliable.
  2. Each round the LLM writes three new versions of the policy, all four are scored by replay, the best is redeployed, and the tree pool grows by one (up to ten rounds). Compared with earlier LLM-evolved search strategies, the novelty is offline scoring and accumulating history. The improver itself never changes and nothing is trained.
  3. Single-run results on nine tasks in four domains. The only controlled baseline is the same system with the policy frozen while comparisons to outside systems are not controlled. There are two ablations and two discovered programs that readers can inspect.
Critical strengths
  1. A simple, cheap idea that others can reuse. If the authors can show that replay scores predict real online gains, this is the kind of idea practitioners would adopt quickly.
  2. Two ablations provide useful insights: a) scoring policies by replay works better than summarizing the history into the prompt and b) the learned policy first cuts its budget and then expands it again.
Critical weaknesses
  1. The results are promising but the authors do not provide standard deviations or confidence intervals. Without them it is difficult to estimate the statistical significance of the provided results. With several runs instead of one, these would make the paper convincing.
  2. No experiment separates the effect of replay scoring from the effect of simply having an LLM rewrite the policy each round from a carefully hand-written prompt. The one ablation on history gives the two arms very different compute, and at equal compute the result goes the other way. The paper never shows whether a high replay score actually leads to a better online run. It also ignores the known problems with scoring policies on logged data: a candidate can only pick nodes that were already recorded.
Questions for authors
  1. How many independent runs underlie each curve/table entry? Please provide 3 seeds for both methods on at least ConvDiv, ConvMax, and Lasso-Flash, with CIs or STDs on the efficiency ratios and final-vs-final numbers alongside matched-compute ones.
  2. Table 1: are external rows best-of-run or means? Which autocorrelation inequality does each row refer to, and which n for Circle Packing? Please acknowledge that the fixed baseline wins on Autocorrelation and that Circle Packing is saturated across systems.
  3. Please report speedup of kernels over the PyTorch reference and a call-matched (not wall-clock-capped) PaceEvolve run, with the LLM used.
  4. What are λ\lambda, K1, K2, and the decoding temperature? Which model was the policy-development agent in 4.1-4.3 paragraphs. How are lower-is-better scores mapped into a=clip(s/Si)a = clip(s/S_i)?
  5. The only controlled baseline (Recursive Fixed Exploration) freezes the policy entirely, so it removes both the LLM rewrite step and the replay-based selection step. The reported gains could come from the rewrite step alone, since the policy-development prompt already encodes strong exploration heuristics and sees the previous round’s live results. Please run the following control: keep everything unchanged (seed policy, agents, prompt, three rewrites per round, rounds, call budget), but select the next policy uniformly at random among the four candidates instead of by replay score, and report the final score and the calls-to-reach-score curve. This isolates the contribution of the replay simulator from iterative LLM rewriting.
5 comments
AuthorsOpenReview ↗

Author Response (1/n)

We thank the reviewer for recognizing our approach’s simplicity, low cost, and useful ablations. We address your concerns below regarding replay validity, the contribution of replay-based selection, statistical robustness, and experimental fairness and reproducibility.

Regarding replay validity and limitations (Weakness 2)

(a) Do replay scores predict online performance? (Weakness 2)

We have added an additional experiment to show replay scores can indeed predict online performance. The results are shown below.

Setting: Using the first-round history from Lasso-Flash, we generate three candidate policies through dreaming and evaluate each independently for one online round. Online scores are measured on the discovery set using the same evaluator as during search.

Results: The policy with the highest replay score achieves the best online final score with an intermediate call budget. Policy 1 uses fewer calls but achieves lower quality, while Policy 2 uses more calls than the selected policy yet achieves a lower score. In this experiment, replay-score rankings agree with online final-score rankings, providing initial evidence of replay–online consistency.

TaskCandidate PolicyReplay Score ↑Online Final Score ↑Online Calls
Lasso-FlashPolicy 10.880.122171
Lasso-FlashPolicy 20.900.125278
Lasso-FlashPolicy 3 (selected)0.920.130255

(b) What can recorded-tree replay evaluate? (Weakness 2)

We acknowledge that replay evaluates candidate exploration policies only on recorded nodes and outcomes. Our core idea is to reuse limited search history as a low-cost simulator for evaluating and improving exploration policies before online deployment. Although replay is only a surrogate for policy selection, our added online validation provides initial evidence that its rankings can transfer to live search. Future work could explore learned world models to extend evaluation beyond recorded trees. We have added a Limitations and Future Work section to discuss these limitations and potential extensions.

Regarding the contribution of replay-based selection

(a) Isolating replay-based selection from iterative LLM rewriting. (Weakness 2; Question 5)

Following your suggestion, we conducted an additional experiment to isolate the contribution of replay-based selection. We keep all other settings unchanged, including replay feedback for iterative rewriting, but select the next policy uniformly at random among the current policy and three rewritten candidates. We can see Dream-RSI achieves a higher final score with fewer discovery calls than random selection. This result supports the additional benefit of replay-based selection beyond replay-guided iterative LLM rewriting

TaskSelection MethodCallsFinal Score ↑
ConvDivRandom Selection8871.55
ConvDivDream-RSI7861.90

(b) Compute fairness of the history ablation. (Weakness 2)

Our intended point is that optimizing against a single replay world may overfit the policy to that recorded trajectory, leading to overly aggressive reductions in exploration computation. The all-history variant uses more calls and achieves higher final quality, which is consistent with mitigating this behavior. This ablation was not intended to establish an advantage at matched computation.

Regarding statistical robustness: Independent runs and variability. (Weakness 1; Question 1)

In our current paper, we run each experiment once and report the results. To address your concern on statistical robustness, we further conduct two additional runs for both methods on ConvDiv, ConvMax, and Lasso-Flash and the results are presented below.

TaskMethodAvg. CallsFinal ScoreScore at Matched Compute
ConvDiv ↑Recursive Fixed Exploration990.001.3987 ± 0.15151.3267 ± 0.1818
ConvDiv ↑Dream-RSI570.671.6848 ± 0.2547—
ConvMax ↑Recursive Fixed Exploration990.000.5374 ± 0.26690.5208 ± 0.2734
ConvMax ↑Dream-RSI689.330.6112 ± 0.2240—
Lasso-Flash ↓Recursive Fixed Exploration3200.002866.8 ± 320.63118.4 ± 135.3
Lasso-Flash ↓Dream-RSI1790.672619.8 ± 323.4—
TaskCall Ratio (Fixed / Dream-RSI, Mean ± STD)
ConvDiv1.8420 ± 0.5233
ConvMax1.4368 ± 0.0374
Lasso-Flash1.7915 ± 0.1111
AuthorsOpenReview ↗

Author Response (2/n)

Regarding experimental fairness and reproducibility

(a) Table 1 reporting and task configurations. (Question 2)

We source the external scores from prior work, following the reporting settings of each source. When both mean and best results are provided, as in EvoX and its evaluated baselines, we report the best results. We found that we mistakenly reported EvoX’s mean autocorrelation score (1.4589) and have corrected it to its best score (1.4558). We use the third autocorrelation inequality and circle packing with (n=26) in a unit square. We acknowledge the fixed baseline’s win on Autocorrelation and the ties on Circle Packing at the reported precision.

(b) Kernel speedups and call-matched PaceEvolve comparison. (Question 3)

We compare Dream-RSI and PaceEvolve using Gemini-3.1-Pro under the same discovery-agent call budget for each task: 408 / 553 / 786 / 669. The table reports the best valid kernel found within each budget, with speedup computed as PyTorch runtime divided by kernel runtime under the same evaluation protocol and the test environment.

MethodLLMVGG16LayerNormConvDivConvMax
PyTorch—1.00×1.00×1.00×1.00×
PaceEvolveGemini-3.1-Pro3.28×5.98×7.05×2.16×
Dream-RSIGemini-3.1-Pro3.58×6.91×13.51×2.81×

(c) Hyperparameters, models, and score definitions. (Question 4)

Thank you for pointing out these missing details. We have clarified the following in the revised manuscript. Hyperparameters: We use (\lambda=0.05). We set (K_1) to the maximum non-root node budget of an online tree and (K_2) to the number of non-root nodes in the recorded replay tree; the policy may stop earlier. Each decision round may involve multiple calls in parallel. The decoding temperature is 1.0. The policy-development agent uses the same model as the discovery agent in each experiment. We transform lower-is-better objectives into higher-is-better scores before computing attainment: inverse runtime for KernelBench, inverse geometric-mean runtime over the 17 discovery instances for Lasso, and s=1.4556427953745406/C₃ for Autocorrelation. Invalid candidates receive zero score. We then compute a=clip(s/Sᵢ,0,1), where s is the best transformed score attained during replay and Sᵢ is the best transformed score in the same recorded trace.

Thank you for the detailed response and the new experiments. I still have three concerns.

  1. As I understand it, the random-selection control is compared with the original Dream-RSI run (1.90). Your new multi-seed table shows 1.6848 ± 0.2547 for Dream-RSI on ConvDiv, so the random-selection result (1.55) is within the run-to-run spread. It is also above the Recursive Fixed Exploration baseline’s result (1.3987 ± 0.1515), which suggests that rewriting alone may explain much of the gain. Could you run random selection two more times and compare the averages, both for the final score and for the score at matched calls?
  2. In the multi-seed results, the differences between the two methods are within one standard deviation on ConvDiv and ConvMax. The original single run also appears to be the strongest of the three on ConvDiv (1.90 vs. an average of 1.68), so the current headline numbers in the paper overstate the typical result and should be updated accordingly.
  3. The replay-versus-online check covers three policies with one online run each, and the online scores differ by only 2.5–6.5%. The run-to-run variation of this score is not reported, while for the final runtime on the same task your multi-seed table shows about 11–12%. Could you repeat each policy’s online round two or three times, so that we can see whether the ranking is stable?

I will raise my score if, with three runs each, Dream-RSI’s average final score on ConvDiv is still clearly above that of random selection, both at the end of the search and at the same number of calls (the table “Isolating replay-based selection from iterative LLM rewriting”).

Reviewer_pZLeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

DREAM-RSI presents a framework for searching a solution space using AI agents.

There are many components to the framework: Trees of solutions. Notably, I believe these are starlike trees (no non-root branching), but the paper does not make this appropriately clear. An exploration orchestration policy that for a given round of tree growth determines a batch of nodes for extending the current tree: adding new children to the root and or adding descendants to the branches. A solution proposing policy (the base agent) that takes in a history of all previous tree solution searches, and proposes a solution based on the current node in the current tree. This base agent is what the exploration orchestration policy orchestrates.

Core to the framework is the idea of online orchestration from the orchestration policy, and an offline phase where the orchestration policy is improved.

By recording past tree history (and restricting the trees to be starlike), the authors elegantly efficiently simulate the problem of orchestration (without the compute required for the online continuations) and the idea appears to me clever, elegant, and genuinely novel. The authors explain that this past record acts as a simulator for exploration (analogous to previous RL work, such as Dreamer).

The authors present their framework as solving a critical problem in orchestrating exploration in complex search spaces, and tie this, for motivation, to Recursive self-improvement in particular. They cite Darwin Godel Machine (DGM) paper (Zhang et al, ICLR, 2026) in support of a claim that orchestration is critical. While there is plausible value in orchestration, I do not recall DGM making any such claim that a lack of orchestration is a critical limitation, and when I checked I could not find such a claim either. Further, DMG uses general trees (not just starlike trees) and has each node a coding agent that proposes improvements to itself.

The authors seem to see the core contribution of DREAM-RSI is ‘closing a meta-layer RSI loop.’ However, as noted above, generally RSI methods do not operate on starlike trees, and indeed, operating only on starlike trees is potentially quite limiting. Based on my understanding of the field, I would also not characterize their method, as presented standalone, as necessarily being RSI, given that the orchestration policy is essentially just being updated/improved on the complex task of optimizing orchestration. It is in no way improving itself. While the field of RSI is a good motivation (orchestration being an important component of many RSI systems), it is worth being clear about what exactly DREAM-RSI is as currently presented. DREAM-RSI could potentially be a part of an RSI framework, or be developed into RSI, etc… but I believe it is not in itself a system that improves itself.

They also do not in my opinion make sufficiently clear that their method does not learn exploration (i.e., how to best generate a solution that explores solution space) but instead a way of efficiently orchestrating exploration.

Critical strengths

The underlying idea of the paper appears to me clever, elegant, and genuinely novel: using a record of orchestration to efficiently train an orchestration policy without requiring the base policy to act. Much of the paper is well written, and it seems to me that the authors have put substantial effort into communicating their idea.

For example, the authors' mathematical articulation of their framework seems correct, precise, and they provide appropriate natural language descriptions of their algebraic terms. This kind of writing care matters and is appreciated.

Critical weaknesses

Unfortunately, the claims and motivation of the paper appear overstated, and the empirical evidence seems lacking.

Their motivation is to address a fundamental issue with RSI frameworks, but the frameworks they mention in general do not operate on starlike trees, and much of the difficulty of orchestration is orchestrating across complex trees that branch recursively at multiple levels. Paying careful attention to the mathematical description of their method, one can identify this limitation, but it is a substantive limitation that should be clearly communicated to the reader. It also strongly undermines their motivation of improving RSI orchestration (e.g., methods such as the Darwin Godel Machine) which employ complex non-starlike trees (and orchestrating recursively branching trees introduces important challenges that their framework does not address and renders their stated framework—in its current form—inapplicable to how these RSI methods actually exist and are used in practice).

The authors do not make clear that this is a limitation of their method, and indeed the authors provide no limitations section.

Their empirical comparisons appear inadequate: when comparing to other preexisting methods they do not control for the foundation model used, and generally appear to use a stronger base model than the methods they compare to. Finally, their own control (Recursive Fixed Exploration) is an ablation of their own method and feels highly artificial. As such, I believe they only have experimental support for the fact their method works to some degree, and that their framework benefits from updating the orchestration policy. They do not have sufficient support for their claim that they are solving a critical problem, or that their framework can outperform other methods (or indeed has any value compared to any other method).

Questions for authors

Does your method operate on non-starlike trees (trees that do not branch only at the root)? If the method only operates on starlike trees, why did you not make this appropriately clear?

Why is there no limitations section? Your paper has multiple limitations (e.g., limited empirical comparisons, only operating on starlike trees, etc...) and it would seem to be good scientific communication to clearly flag these to a reader (so they can understand the genuinely good contribution you make, and not be confused into thinking it is more than it is).

Could you elaborate on how you believe solving the problem of orchestrating exploration is the same as solving the problem of exploration?

Following the last question, could you position your work relative to work that aims to learn an exploration policy? For example, in the non-agentic meta-RL setting, First-Explore (Norman and Clune, NeurIPS, 2024) explicitly learns an exploration-mode, and one could apply the same technique to agentic AI/harness optimization. How do you position your orchestration relative to that?

6 comments
AuthorsOpenReview ↗

Author Response (1/n)

We thank the reviewer for recognizing the novelty of our approach and the clarity of our formulation, and for providing valuable feedback. We address the concerns below.

Regarding the tree structure and its implications for our motivation and impractical in RSI system

We first clarify that DGM is cited as an example of agent self-improvement (Introduction, lines 35–36), rather than as evidence that orchestration is a critical bottleneck.

We acknowledge our current implementation uses a starlike orchestration structure and will clarify this explicitly. However, this does not invalidate our motivation of improving RSI orchestration: an explicitly maintained, recursively branching tree is one design choice, rather than a prerequisite. In our implementation, each branch maintains a workspace containing its previous attempts, and the coding agent can decide which earlier solutions to revisit, refine, or combine. For example, given attempts A, B, and C, a new attempt D may build on any of them or combine elements from multiple attempts. Although attempts are organized as a chain, their underlying relationships can form a more complex structure through revisiting and combining earlier solutions.

Recent work also supports the viability of this design: Meta-Harness allows the proposer to inspect all prior candidates’ code, scores, and execution traces without imposing a parent-selection rule, and reports stronger results than OpenEvolve and a PUCT-based TTT-Discover search variant in its harness-optimization experiments (Lee et al., 2026). Building on this design, DREAM-RSI formulates the upper-level orchestration problem as deciding when to initiate new branches, which existing branches to continue, and how much computation to allocate to them. Our contribution is to improve these decisions by converting recorded discovery trees into replay simulators, enabling the agent to “dream” within them for inexpensive offline policy evaluation and improvement.

Regarding fair comparisons under the same foundation model and the role of our control baseline

Thank you for raising this concern. Our original submission already included external baselines evaluated using the same foundation models, and we have further expanded these comparisons during the rebuttal period.

Specifically, we compare against PaceEvolve (2026) using Gemini-3.1-Pro on kernel engineering tasks (Figure 3) and Meta-Harness (2026) using the same discovery agent, Opus-4.8 (Figure 5). We additionally compare against SimpleTES on algorithm engineering tasks using Gemini-3.7-Flash, with SimpleTES receiving a substantially larger sampling budget. The results are presented below. We can see that using the same foundation model, DREAM-RSI achieves the lowest average runtime among the three methods. Moreover, we include Recursive Fixed Exploration as a strong internal baseline that isolates the benefit of updating the orchestration policy relative to keeping it fixed within our framework. Therefore we believe our empirical evaluation is already fair and sufficient.

MethodModelComputeGisetteRCV1DNALeukemiaColonDuke BreastAvg. ↓
SimpleTES †Gemini-3.7-Flash51,2001746.122889.255.516.910.918.24122.8
Recursive Fixed ExplorationGemini-3.7-Flash3,2001133.113873.029.824.115.724.42516.7
Dream-RSIGemini-3.7-Flash1,8791091.912923.431.421.012.223.62350.6
AuthorsOpenReview ↗

Author Response (2/n)

Regarding exploration orchestration and its relation to First-Explore

We do indeed improve exploration by optimizing exploration orchestration. Section 2 of our paper explicitly defines the responsibilities of the exploration policy: determining which directions to explore, which candidate solutions to further refine, which branches to pursue in parallel, and when to terminate.

Intuitively, exploration is akin to searching for treasure in a vast mountainous region with numerous paths and unknown treasure locations. An explorer must not only move along these paths but also—based on findings made thus far—decide whether to continue on the current path, pivot in a different direction, or explore multiple paths. Optimizing these decisions inherently improves the exploration process. In our system, the orchestration policy uses feedback from observations to terminate branches deemed to have limited potential, reallocates computational resources toward more promising directions, and dynamically adjusts the degree of parallelism. Thus, our claim of "improving exploration" specifically refers to the optimization of these exploration-control decisions. We further clarify this meaning and the scope of our research in the revised manuscript.

Relation to First-Explore: First-Explore is an elegant and well-known exploration paper. Specifically, it addresses how to explore for better subsequent exploitation by training an exploration policy using downstream exploitation returns. Different from it, DREAM-RSI addresses delayed and expensive feedback in orchestration-policy optimization by converting recorded discovery trees into replay simulators, enabling the agent to “dream” within them for inexpensive offline policy evaluation and improvement. We will add it to our related work section.

Regarding lack of limitations section

Thank you for highlighting this omission. We agree that explicitly discussing limitations is important for accurately communicating the scope of our contribution.We explicitly acknowledge the structural limitation discussed above:

  • Tree structure. Our current implementation uses a starlike orchestration structure, with each branch maintaining a workspace of previous attempts. Although the proposer can revisit and combine ancestral solutions, the orchestrator does not explicitly schedule multiple children of a non-root node. Future work could extend the framework to general branching structures, representing each node as a workspace state and allowing multiple independently scheduled continuations. We have added this discussion to a dedicated Limitations section in the revised paper.

Thank you again for helping us improve the clarity and completeness of our paper.

I appreciate the many changes and additions, and I believe they much improve the paper. As I said, I believe the idea is elegant and I like the work. I shall read through the revised paper, and reflect on it further and I am definitely open (after further reflection) to potentially raising my score.

A few quick notes: Looking at the current draft, I do not find the "Orchestration structure" limitation section particularly clear, but I very much appreciate the inclusion. I would value if if you could refine it a bit further. The content is there, but the expression of said content could be far clearer. At the moment, I imagine that the section is only clear to those who already know the issue. In particular, a simple statement alone the lines of, "DREAM-RSI as currently articulated is not directly applicable to most RSI frameworks due to this limitation, however [future work, etc...]." might help. You may disagree, and believe it applicable, and I open to you arguing that, but regardless you should state clearly what the implication of "[not] cover]ing the full range of scheduling decisions" is.

I appreciate the comparison to First-Explore. I think it is helpful to situate your work (and its focus on orchestrating exploration) against other ways of framing exploration. I note that I could find no current comparison in the draft though.

I have been thinking on the paper some more, and one other general piece of feedback is that you almost exclusively cite work published this year. Only citing very recent work is usual, and risks failing to properly contextualize your work relative to the field. In particular, the use of world models (and trajectories being proxies for such) goes back the 1990s, and there seem to be many works that might be relevant.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

Thank you for your constructive feedback and for engaging with us throughout the rebuttal. We appreciate that the reviewers highlighted several strengths of Dream-RSI:

  • A novel and practical approach. Reviewers highlighted the novelty of replaying recorded discovery trees (pZLe) and the practical value of evaluating orchestration policies without additional discovery-agent calls (bFmx, kLG2).
  • A precise and clearly explained mathematical formulation of the framework (pZLe).
  • Insightful ablations on replay and budget allocation (bFmx), together with a well-controlled fixed-policy baseline and evaluation across diverse domains (kLG2).

During the rebuttal period, we provided detailed clarifications and conducted substantial additional analyses to address raised concerns. Our updates fall into two major categories:

1. New experiments and analyses

  • Online validation of replay rankings on Lasso-Flash (three candidate policies, three online runs each).
  • Three independent discovery runs per method for DREAM-RSI and Recursive Fixed Exploration on ConvDiv, ConvMax, and Lasso-Flash.
  • Random-selection ablation on ConvDiv with the same replay-guided rewriting procedure.
  • Early-stopping baseline for Recursive Fixed Exploration on four kernel tasks.
  • External baseline comparisons with PaceEvolve under matched models and call budgets, and with SimpleTES using Gemini-3.7-Flash on Lasso.
  • End-to-end efficiency analysis on Lasso covering LLM calls, token usage, and elapsed time, including policy development.

2. Manuscript improvements

  • Figure 3 now identifies each curve as a single run. Appendix C.5, Table 6 reports three-run means and standard deviations for ConvDiv, ConvMax, and Lasso.
  • Section 6 (Related Work) now discusses First-Explore and broader related work about world model, experience reuse and planning .
  • Add Section 8 (Limitations) to clearly discuss the limitation of our paper
  • We clarified more details. We also modified some statement to make it more precise.

Following these revisions, reviewers pZLe and kLG2 both raised their scores to 3. Reviewer kLG2 specifically cited the online validation, early-stopping comparison, and repeated runs as largely addressing their concerns.

Thank you again for your time and for helping us strengthen the evaluation and clarify the scope of Dream-RSI.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.