- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Gödel Forest, a multi-agent framework for data-centric recursive self-improvement that balances search depth and breadth. Each agent maintains a persistent search tree to iteratively refine data generation and training strategies, while multiple agents explore different directions in parallel. An asynchronous shared memory stores distilled lessons from successful and failed experiments, along with a global leaderboard, to guide future exploration. The method is evaluated on six RSIBench-Data benchmarks, achieving an average improvement of 10.70% over a single-agent baseline. Ablation studies further show a +7.00 gain over independent parallel search, supporting the benefit of shared memory and cross-agent coordination.
Critical strengths
- The paper is well written and accompanied by nicely made figures. It was easy to follow the paper.
- The method shows consistent improvements across six benchmarks. The ablation study comparing against independent parallel workers further helps isolate the benefit of shared memory and cross-agent coordination.
Critical weaknesses
-
My main concern is the reliability of shared lessons and the potential accumulation of errors over time. If an agent draws an incorrect conclusion from an experiment and shares it with other workers, the error may propagate through subsequent evolution. Have the authors encountered such cases, and is there a mechanism to detect, correct, or remove misleading lessons?
-
Generalization to unseen tasks is not sufficiently evaluated. Checkpoint selection and official evaluation use the same task instances, raising concerns about overfitting to the benchmarks used during search. The authors should evaluate the final models on separate held-out instances that were never used during data strategy search to assess their generalization.
-
The contributions of different components are not fully isolated. The current ablation removes both shared lessons and the global leaderboard, making it unclear which component drives the improvement. The authors should ablate these two components separately under matched compute budgets to clarify their individual contributions.
-
Generalization across target models is not evaluated. All experiments use Qwen3.5-35B-A3B-Base as the target model. The authors should include experiments with other target models to verify whether the proposed framework generalizes beyond this specific model.
Questions for authors
Please see the weaknesses section above
3 comments
Response to Reviewer R1oM (2/2)
3. Contributions of the leaderboard and shared lessons
The existing ablation directly tests whether coordination adds value beyond running more workers. Both conditions use three workers and the same resource ceilings. Removing the leaderboard and shared lessons gives 47% on SWE-bench Verified, while the full forest reaches 54%. The full forest also spends less on Tinker (239.67 versus 421.63), so its gain cannot be explained by higher training-service expenditure. This establishes the benefit of the combined communication mechanism under the tested protocol.
The channels have complementary roles. The leaderboard provides a common reference grounded in full evaluations; procedural lessons communicate interventions, observed effects, and operational rules that other workers can test. This includes information from failed experiments that would not appear as the best-performing strategy. The termination-and-verification case in §4.5 illustrates how such information helps construct a new candidate, beyond identifying an existing high-scoring one.
The new leaderboard-only control, now included in Figure 3 and §4.4, provides the requested finer attribution. Keeping K = 3, the researcher, target model, local search protocol, evaluation interfaces, and global budget ceilings fixed, leaderboard-only sharing reaches 49%, close to the 47% of three independent workers. Elapsed time is nearly identical (13.50 versus 13.38 hours), and expenditure falls from 421.63 to 378.00. Rankings alone therefore yield a modest 2-percentage-point gain. With procedural lessons added, the full forest reaches 54% in 14.06 hours at 239.67. Compared with leaderboard-only sharing, this is a further 5-percentage-point gain and 36.6% lower Tinker expenditure, with 0.56 additional hours. The similarity between independent workers and leaderboard-only sharing, followed by the larger improvement from adding lessons, supports the incremental value of procedural knowledge when a shared leaderboard is already available.
4. Generality across researchers and target models
Following the RSIBench-Data [1] setting, all experiments use Qwen3.5-35B-A3B-Base as the target model. Our paper asks whether a researcher can generate suitable data for a given task and thereby train a stronger model. Gödel Forest changes how the researcher generates, evaluates, and shares data strategies; the target model is only the object being trained, and the method does not depend on its architecture or internals. With the target model fixed, the score difference between vanilla and Gödel Forest comes only from the difference in the research procedure.
For our method, the key variable is the researcher that makes the decisions. In §4.3 we replace Claude Code with Codex as the researcher, and Gödel Forest still improves over the vanilla baseline: from 33% to 44% on SWE-bench Verified and from 20.22% to 23.60% on Terminal-Bench 2.0.
References
[1] RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement. arXiv:2607.25886v1, 2026.
Thank you for the detailed response and the new experimental results. The new leaderboard-only control helps clarify some of my concerns.
However, I remain concerned about the generalization of the learned models to unseen instances. My concern is not about cross-task transfer to unrelated domains (though showing that would be even better), but rather that using the same benchmark instances for both iterative search feedback and final evaluation risks overfitting to those specific test instances. Without validation on a held-out split of unseen instances, the true generalizability of the evolved checkpoints is not sufficiently established.
I will keep my current rating.
Response to Reviewer R1oM (1/2)
We thank the reviewer for recognizing the consistent gains and the evidence for cross-worker coordination. We address the four concerns below, focusing on the mechanisms and controlled comparisons that support our contribution.
1. Reliability and correction of shared lessons
Shared lessons guide new hypotheses; measured candidate performance determines selection. A lesson does not directly replace a checkpoint or certify an improvement. The receiving worker must implement the intervention, train the target model, and obtain experimental feedback. Only fully evaluated candidates enter the shared leaderboard used for final selection. This places an empirical check between a shared interpretation and an accepted model improvement.
Persistent private trees provide a complementary protection: earlier strategies and evaluated candidates remain available when a new intervention fails. Workers retain separate research contexts and can revisit productive branches, so one misleading proposal does not overwrite the search history or eliminate previously successful alternatives. The architecture preserves both evidence and options for subsequent decisions.
Section 4.5 documents how an initially useful but incomplete lesson is refined. Final-turn loss masking addresses unstable termination, while later feedback reveals that intermediate verification actions also require supervision. Prefix-truncated trajectories combine the two observations by making verification actions supervised terminal targets. Correction therefore occurs through subsequent experiments and revised interventions. The demonstrated mechanism is evidence-driven revision of how a lesson is applied; the paper does not posit an automatic rule for identifying and deleting every incorrect lesson.
The system-level result is also positive: the communicating forest reaches 54%, compared with 47% for independent workers, with lower Tinker expenditure (239.67 versus 421.63). Together, the evaluation gate, preserved alternatives, and observed refinement explain how procedural sharing remains useful even when individual interpretations require revision.
2. Task-conditioned research and unseen-instance generalization
Our paper studies data-centric RSI: given a target task, can the researcher generate suitable training data for that task and use it to evolve a stronger model? Data generation is inherently task-conditioned: the researcher must know the target task to diagnose where the model falls short on it and to synthesize data that addresses those gaps. Our evaluation therefore measures how effectively the research procedure improves the model on the given task, rather than how well the resulting model transfers to other tasks.
The RSIBench-Data [1] protocol is designed for exactly this question, and we follow it without modification: the researcher develops data and training strategies for a given benchmark, receives only evaluator feedback during search without access to gold solutions, and the frozen selected checkpoint is then scored in an isolated official run on the same tasks. Under this protocol, vanilla and Gödel Forest face identical tasks, feedback, and budgets, so the score difference directly reflects the difference in the research procedure.