- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve the exploration policy of LLM discovery agents, i.e., the rules that decide which attempts to refine, how many to run in parallel, and when to stop. The authors propose Dream-RSI, which writes this policy as code and improves it between rounds of discovery. The tree of attempts recorded in each round is reused as a “replay simulator.” A candidate policy is scored by replaying it over the recorded trees, which reveals only already stored outcomes, so no new agent calls are needed. An LLM writes three new versions of the policy each round. The best version by replay score is used in the next round, and the score rewards reaching high results with few calls and in parallel. Across nine tasks in four domains Dream-RSI matches or beats the same agent with a fixed policy while using fewer agent calls.
Critical strengths
- Reusing completed discovery runs to evaluate new exploration policies without new agent calls is a simple and practical idea. It addresses the slow, expensive feedback that makes improving exploration policies hard.
- The main baseline, Recursive Fixed Exploration, is well controlled. It uses the same agent, evaluator, starting policy, and per-round budget, and differs only in never updating the policy.
- The evaluation covers diverse domains, and the paper includes full prompts and the discovered programs.
Critical weaknesses
Main concerns are:
- The replay simulator is biased and never validated: Replay only reveals outcomes already stored in the recorded trees. It can never reward exploring beyond what was recorded, only reaching recorded results with fewer calls, which favors pruning and stopping early. Recorded outcomes also depend on the original order of exploration, since the agent reads all earlier and sibling attempts before proposing (Listing 1). The paper never checks whether replay scores predict online results.
- Experiments show small or mixed gains, all from single runs, and no task reports repeated discovery runs or variance.
- Call savings may come from simple early stopping: The replay score directly rewards fewer calls. On VGG16 and LayerNorm, the final scores are about the same. Fixed exploration improves only about 1–3% after roughly 450 calls, so a simple stopping rule might give similar savings. The paper compares with neither such a rule nor rewritten policies deployed without replay selection. It also does not count the calls of the LLM that rewrites the policy.
Questions for authors
- Do replay scores predict online results? For example, could the authors run several candidate policies online and report how well their replay scores match their online results?
- How does Dream-RSI compare with fixed exploration plus a simple stopping rule (e.g., stop after a round without improvement), and with rewritten policies deployed without replay selection?
- Could the authors report independent discovery runs per task, with variance?
- How many calls and tokens does the LLM that rewrites the policy use? How do total cost and wall-clock time compare with fixed exploration?
4 comments
Author Response (2/n)
3. Independent runs and variability (Weakness 2; Question 3)
The original results were based on one discovery run per setting. Following your suggestion, we added two independent runs for both Recursive Fixed Exploration and Dream-RSI on ConvDiv, ConvMax, and Lasso-Flash, giving three runs per method on each of these tasks. Across these three tasks, Dream-RSI achieves better mean final scores while using fewer discovery calls.
| Task | Method | Avg. Calls | Final Score (Mean ± STD) |
|---|---|---|---|
| ConvDiv ↑ | Recursive Fixed Exploration | 990.00 | 1.3987 ± 0.1515 |
| ConvDiv ↑ | Dream-RSI | 570.67 | 1.6848 ± 0.2547 |
| ConvMax ↑ | Recursive Fixed Exploration | 990.00 | 0.5374 ± 0.2669 |
| ConvMax ↑ | Dream-RSI | 689.33 | 0.6112 ± 0.2240 |
| Lasso-Flash ↓ | Recursive Fixed Exploration | 3200.00 | 2866.8 ± 320.6 |
| Lasso-Flash ↓ | Dream-RSI | 1790.67 | 2619.8 ± 323.4 |
4. Policy-development overhead, total cost, and wall-clock time (Weakness 3; Question 4)
We conduct a cost analysis of the Lasso-Flash experiments. Throughout the discovery process, policy development occurs once between each pair of consecutive rounds, with three LLM calls per update, totaling 12 calls, 66.6M total tokens, and 0.36M output tokens. Including policy development, Dream-RSI uses 40.9% fewer LLM calls and 47.4% fewer total tokens than fixed exploration. Policy development accounts for approximately 0.6% of its total calls and 2.0% of its total tokens.Dream-RSI takes approximately 1.41× as long in active elapsed time (34.31 h versus 24.35 h).
| Metric | Fixed Exploration | Dream-RSI: Discovery | Dream-RSI: Policy Development | Dream-RSI: Total |
|---|---|---|---|---|
| LLM calls | 3200 | 1879 | 12 | 1891 |
| Total tokens | 6343M | 3271M | 66.6M | 3337.6M |
| Output tokens | 53.0M | 27.6M | 0.36M | 27.96M |
| Active elapsed time | 24.35 h | 30.72 h | 3.59 h | 34.31 h |
Regarding Concerns
Thank you again for your time and constructive feedback. As the discussion deadline approaches, we would appreciate any further feedback or remaining concerns and would be happy to provide additional clarification.
Thank you to the authors for the additional experiments and responses. The online validation, comparison with fixed stopping rule and repeated run, are all helpful results. Through these the authors have largely addressed my concerns so I have raised my score to 3.
Author Response (1/n)
We thank the reviewer for recognizing the practicality of replay-based policy evaluation, the controlled baseline, and the breadth of our evaluation. We address your concerns below regarding replay validity, early stopping and replay-based selection, statistical robustness, and total computational overhead.
Regarding replay validity and limitations (Weakness 1; Question 1)
(a) Do replay scores predict online performance?
We have added an online validation experiment to examine whether replay-score rankings agree with online performance.
Setting. Using the first-round history from Lasso-Flash, we generate three candidate policies through dreaming and evaluate each independently for one online round. The online final score is the highest score achieved during the round, computed as the inverse geometric mean of solver runtimes over the 17 synthetic evaluation instances used during discovery; solvers that fail any correctness check receive a score of zero.
Results. The replay-score ranking largely agrees with the online final-score ranking across these three candidates. The selected policy achieves the highest online final score with an intermediate number of calls: Policy 1 uses fewer calls but achieves lower quality, while Policy 2 uses more calls than the selected policy yet achieves lower quality.
(b) Recorded coverage and dependence on exploration history
We acknowledge that replay evaluates policies only on recorded nodes and outcomes; it cannot evaluate unrecorded branches or reward improvements beyond the recorded outcomes. We have added this in the limitation section of our current version of paper.
However, we clarify that the solution-generation agent does not observe attempts from other branches within the current round. Its context consists of the node’s ancestor chain and any previous-round history, which is fixed for that recorded round. Replay strictly preserves ancestor dependencies: a child is revealed only after its parent has been reached. Thus, changing the order of exploration across branches does not change the generation context associated with a recorded node. The orchestration policy observes results as they are revealed, but these observations do not become additional cross-branch context for the solution-generation agent.
2. Do the gains come from early stopping or iterative LLM rewriting alone? (Weakness 3; Question 2)
(a) How does Dream-RSI compare with fixed exploration plus a simple stopping rule?
Following your suggestion, we apply a simple stopping rule to the recorded fixed-exploration runs: stop after one completed round without improvement in the best-so-far score. We show the results below. We can see this approach achieves comparable results on LayerNorm but terminates prematurely on ConvDiv, missing later improvements. Dream-RSI achieves a higher score with fewer calls on ConvMax and a similar score with substantially fewer calls on VGG16. These results show that this simple stopping rule can be effective on some tasks but does not consistently reproduce Dream-RSI’s quality–cost trade-off.
(b) Does replay-based selection improve over iterative LLM rewriting alone?
We conducted an additional experiment to isolate the effect of replay-based selection. We keep the seed policy, agents, prompt, replay feedback for rewriting, three rewrites per round, number of rounds, and call budget unchanged, but select the next policy uniformly at random among the four candidates.
We can see that random selection can also improve performance, which may be attributed to LLM-based rewriting guided by replay feedback. Using replay scores to select policies provides further gains, as randomly selected policies may have suboptimal efficiency or performance.