acceptodds
Under review as a conference paper at ICLR 2027

RSI-Forge: From Research Papers to Environments for Recursive Self-Improvement

12 views

Abstract

Environments are the foundation of recursive self-improvement: they provide the problems agents work on and the feedback used to evaluate progress. Yet constructing challenging research environments with reliable evaluation still depends on domain experts, limiting their scale and disciplinary coverage. We introduce RSI-Forge, a multi-agent pipeline that turns published papers into executable environments for self-improvement. Three agents coordinate construction, reproduction, and review to produce tasks with automated evaluators; each paper’s method is independently reimplemented to establish a baseline score. We present 210 environments across 18 fields, including 90 reviewed by independent human domain experts. Both experts and agent judges give high ratings to the potential for improving the provided starting solutions and the evaluators’ ability to distinguish solution quality, whereas experts are more critical of shortcut resistance, faithfulness to the source paper, and whether a single idea can exhaust a task. To validate their use for repeated improvement, we evaluate four models over 3 successive attempts on 120 environments, with each attempt inheriting prior code and notes while model weights remain fixed. At least one model improves after the first attempt in 84% of environments. Models also outperform the reproduced paper methods in 68 of the 120 environments, demonstrating room for gains beyond these baselines. Transcript analysis identifies work beyond parameter tuning in 95% of these successful attempts. Analysis of the resulting trajectories shows that models scoring lower on these tasks explore less, more often accept gains smaller than the reported standard error, and rely more heavily on tuning to the development set. RSI-Forge provides a scalable approach to constructing research environments for training and evaluating self-improving agents.

Primary area:
datasets and benchmarks
Keywords:
recursive self-improvement, LLM agents, agentic coding, environment construction, research code, self-improvement trajectories, evaluation, synthetic data, multi-agent system

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uP8vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the lack of environments and tasks for improving RSI agents, and proposes a scalable pipeline for generating them. Specifically, it uses published papers as sources, then runs a multi-agent workflow to generate task environments and corresponding baselines, and uses LM judges to score them. The authors used this approach to build numerous tasks and environments, running experiments with different models across 120 of them to demonstrate the pipeline's effectiveness.

Critical strengths
  1. the problem this paper tries to solve is crucial and very practical. If our AI agents need to continuously improve their research capabilities, we genuinely need a mature pipeline to automatically generate a large volume of high-quality research tasks with automated evaluation.

  2. the scale and coverage of the tasks built in the paper are impressive, featuring 210 environments, 18 research domains, and multiple models. Furthermore, all the generated tasks have been evaluated by human experts.

Critical weaknesses
  1. regarding the value of the tasks generated by this pipeline: I'm skeptical about whether they're actually useful for an agent to learn new capabilities, and that's my main concern. For instance, the RSI experiment in the paper only consists of three sessions, with the first two lasting just one hour, and most of the gains come from the very first session. This suggests the tasks might not be particularly difficult, meaning they may not really help agents learn new skills. Furthermore, since these environments are derived from papers, we don't know whether the generated tasks genuinely capture the research problems explored in the original papers or if they're just oversimplified versions. If they are oversimplified, they might be too basic to support a true RSI experiment.
  2. the pipeline uses a multi-agent workflow where an author agent generates the content, a reviewer reviews it, an independent reproducer reproduces the baseline, and then another reviewer reviews it again. However, the paper provides no ablation studies to justify whether all these design choices are actually necessary. For example, I could potentially have a single, powerful agent generate the tasks, evaluation, and baseline algorithms all at once based on the paper. While that might consume more upfront compute cost, if the multi-step design doesn't bring a significant improvement in task quality, wouldn't a one-shot generation approach be better?Also, the authors claim the pipeline is scalable. When it comes to scalability, there are two aspects: whether it can automatically generate environments in bulk, and what it takes to produce a high-quality environment. Regarding the former, it remains unclear how much capital, compute, and time are required to generate a single high-quality environment, and we don't yet know whether this approach scales effectively or not.
Questions for authors
  1. are these generated environments sufficient to support long-term self-improvement? Specifically, if we run real RSI on them, won't agent performance plateau after the first couple of sessions?

  2. regarding the multi-agent pipeline built in this paper for generating task environments: how important is the independent reproducer agent within it? I think that adding more agents increases the pipeline overhead, so I'd like to know what the advantages are of using this multi-agent pipeline compared to just using a single, capable agent to generate all the environments and tasks directly.

6 comments

Thank you for the thoughtful comments. We address W1/Q1 together and W2/Q2 together below.

W1/Q1: Value of the generated tasks.

We'd like to highlight a few findings that showcase the value of the generated tasks, focusing on difficulty, fidelity, and room for continued improvement.

To assess difficulty beyond the starting solution, we compare agents against an independently reproduced paper method under the same task interface and execution budget. Models surpassed that reference in 67/120 environments after excluding all three sessions of model-environment chains flagged for access to restricted materials. Our transcript coding identifies work beyond parameter tuning in 94% of the attempts that surpassed the reference. These comparisons assess progress against a reproduced research method within the adapted task (Sections 5.2-5.3).

On fidelity, domain experts reviewed 90 of the 180 main-run environments with access to the source papers, giving mean ratings of 4.39/5 for research content and 3.72/5 for faithfulness. Their assessments support the presence of research content while identifying losses in fidelity from adapting problems to bounded compute (Section 4.4). The improvement result also holds in the higher-fidelity subset: 24/34 environments rated 4-5 show later gains (Appendix J.2).

Although most gains occur early, improvement continues beyond the first session. After the same whole-chain exclusion, at least one model improves its best graded solution from session 1 to session 3 by more than 0.01 in normalized held-out score in 100/120 environments (83%). This includes 15/20 environments rated 4-5 for resistance to saturation. Some experts saw solver summaries, so the subgroup comparisons are descriptive. All three sessions have the same 60-minute cap, including the third, with fixed model weights. The evidence supports the tasks' use for iterative solution improvement; three sessions cannot establish long-term capability gains or rule out later plateaus, and do not isolate inherited records from additional attempts. We have updated the results in Sections 5.2-5.3, added the expert-rated subset analysis in Appendix J.2, and clarified the scope in Section 6.

W2/Q2: Value and cost of separate reproduction and review.

To test whether the separate agents are worth the overhead, we took 15 papers for which our pipeline had completed a build and asked one agent to build, reproduce, and review each environment in a single session. We kept the model, prompts, and checks the same. With separate agents, the reproducer works from the paper and solver materials without seeing the author's implementation.

The single-agent version completed 11 of the 15 builds. In the other four, it stopped because it could not independently review work it had created itself. Even among the 11 completed builds, nine were marked as passing while the review report said an independent assessment was not possible. None of the 15 builds with separate agents had this contradiction (Appendix G.2).

Taking one of the tasks as an example, the solver was supposed to use observations as they arrived, but the environment built by a single agent also exposed future observations. When we changed only those future observations, the solver's first output changed in all 12 tested runs. The environment built with separate agents supplied observations one step at a time, and the same test changed the first output in none of the 12 runs (Appendix G.3). The broader robustness tests were tied at 9 of 11 tasks.

For these 15 builds, recorded model costs were 6,083 (USD) with separate agents and 5,811 with one agent, a 4.7% difference. The single-agent cost includes the four incomplete builds. In the main production run, we built 180 environments over 14 days. The recorded median cost was 377 per environment across the 163 builds with usage logs. Appendix G.1 gives the cost accounting and logging coverage.

We hope these clarifications and additional results address your concerns. Please let us know if there is anything else we can clarify.

Dear Reviewer uP8v,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Thanks for the authors' response, but my core confusion remains unresolved. First, regarding the ratings by domain experts, the paper shows that on the resistance-to-saturation dimension, the human domain experts scored it very low. This demonstrates that they consider these tasks prone to saturation. Second, regarding the authors' mention in the rebuttal that it improves its best-graded solution by more than 0.01, I find this 0.01 magnitude hard to interpret. To me, it simply means that this scale is already saturated. Additionally, the authors noted that these sessions fail to establish long-term capability gains. Therefore, while the paper describes a scalable task generation pipeline, I don't believe the current results are sufficient to show that this is a meaningful approach for recursive self-improvement tasks. Correct me if you think my interpretation is wrong.

Reviewer_8SLVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RSI-Forge addresses the lack of scalable, reliable environments for evaluating recursive self-improvement (RSI) in AI agents. It introduces a multi-agent pipeline that converts published research papers into executable optimization tasks with automated evaluators and independently reproduced baselines. An author agent constructs each environment, a reproducer agent independently implements the original paper’s method to establish a baseline, and a reviewer agent checks task clarity and evaluator validity. The authors build 210 environments across 18 research fields, with 90 independently reviewed by human experts.

To validate these environments, the authors evaluate four models over three successive attempts on 120 environments, allowing each attempt to inherit previous code and notes while keeping model weights fixed. At least one model improves after the first attempt in 84% of environments, and the reproduced baseline is surpassed in 68 of 120 environments. Trajectory analysis also reveals differences in exploration, experimental judgment, and solution selection across models. The key contribution is a scalable framework for constructing and validating research environments that enable measurable iterative improvement, rather than a method that directly improves model capabilities. The authors note that improvements on compute-constrained tasks do not necessarily translate into advances on the original research problems.

Critical strengths

1.RSI-Forge decomposes environment construction into distinct authoring, reproduction, and review stages, with independent reproduction establishing a baseline and a separate reviewer checking task clarity and evaluator validity. This modular design helps reduce reliance on the original environment generator's own judgments and makes the resulting tasks more reproducible and verifiable. By converting published research into executable tasks with explicit objectives and automated evaluators, the framework also provides a practical way to scale environment construction across 18 research fields.

2.Rigorous validation is particularly important for RSI benchmarks, since apparent improvements can be misleading if evaluators are unreliable, baselines are weak, or tasks can be easily saturated or exploited. RSI-Forge addresses these concerns through independent baseline reproduction, automated quality checks, human expert reviews of 90 environments, and iterative solver evaluations across 120 environments. The experiments show measurable improvements over successive attempts and, in many cases, beyond reproduced baselines. This multi-faceted evaluation provides stronger evidence for the usefulness of the benchmark than solver performance alone, while also revealing limitations in task saturation and fidelity to the source papers.

Critical weaknesses
  1. Limited mitigation of environment quality issues. Although RSI-Forge incorporates automated checks and human expert reviews, these evaluations reveal limitations in shortcut resistance, fidelity to the source papers, and the availability of continued improvement opportunities. However, the paper does not sufficiently address how these issues could be mitigated or demonstrate that the proposed pipeline can systematically resolve them. This leaves uncertainty about the reliability of the constructed environments and whether improvements measured on them consistently reflect meaningful progress on the original research problems.

  2. Limited evidence for recursive self-improvement. The solver experiments demonstrate iterative performance improvements, but models retain the same weights across attempts and inherit previous code and notes. Thus, the results primarily establish the feasibility of iterative optimization within fixed models, rather than demonstrating recursive improvement of the models' own capabilities. Further controlled experiments would help distinguish the benefits of accumulated context from those of additional attempts and establish how well the benchmark measures RSI itself.

Questions for authors
  1. Impact of environment quality on the main results The human expert evaluations identify limitations in shortcut resistance, faithfulness to the source papers, and task saturation. Could the authors clarify whether these issues affect the 120 environments used in the solver experiments, and whether the reported improvement rates remain consistent when considering only environments with stronger expert assessments? This would help establish how robust the main findings are to potential environment quality issues.
  2. Reliability of the measured improvements The paper reports that at least one model improves after the first attempt in 84% of environments, while the trajectory analysis also finds that agents sometimes accept gains smaller than the reported standard error and rely more heavily on development-set tuning. Could the authors clarify how they determine whether individual improvements are statistically meaningful and quantify how development-set tuning may affect the reported gains?
  3. Fairness and strength of reproduced baselines The paper describes how the reproducer agent independently implements the original paper’s method to establish a baseline. The paper further reports that solver models surpass these reproduced baselines in 68 of 120 environments. How do the authors verify that the reproduced baselines are sufficiently faithful to and competitive with the original methods? Could the authors provide evidence that the reported improvements are robust to potential variations in baseline reproduction quality?
4 comments

Thank you for the careful review and suggestions for checking environment quality.

W1/Q1: Environment quality and its effect on the results. We'd like to take this opportunity to highlight that the performance improvements across attempts persist when we evaluate only tasks rated highly by human experts. Of the 120 tasks evaluated with four models over three successive attempts, 62 had expert ratings. On these 62 tasks, models improved beyond their first attempt on 49 (79%) and surpassed the reproduced baseline on 35 (56%). For tasks rated 4-5/5 on the listed criterion:

  • Resistance to saturation: 15/20 (75%) show later improvement; 13/20 (65%) surpass the reproduced baseline.
  • Shortcut resistance: 39/46 (85%) show later improvement; 25/46 (54%) surpass the reproduced baseline.
  • Faithfulness to the paper: 24/34 (71%) show later improvement; 18/34 (53%) surpass the reproduced baseline.

Review requests also led to edits in files used by the solver or evaluator in 163/262 logged repair cases. An independent LLM judge assessed 60 reviewer-author exchanges; after weighting by construction stage, an estimated 63% of replies were judged to address the reported issue. In 31/33 cases where the reviewer quoted text needing a change, that text was absent from the final files.

We added the expert-rated results in Appendix J.2, the repair checks in Appendix F, and the example of preventing access to future observations in Appendix G.3.

W2: Evidence for recursive self-improvement. The experiments demonstrate repeated improvement of task solutions with fixed model weights. Across three attempts that combine extra solving time with inherited code and notes, at least one model improves beyond its first attempt on 100/120 tasks (Sections 5.1-5.2). We clarified the scope of these experiments in Section 6.

Q2: Reliability of gains and development-set tuning. We'd like to clarify that the reported improvements are measured on held-out data, while agents use development scores to choose changes. At least one model's best saved solution improves from the first to the third attempt on 100/120 tasks (83%). The task counts throughout this reply exclude all three attempts by a model on a task if its logs show access to restricted evaluation materials.

The improvement criterion measures score gains, not statistical significance. The existing small-gain analysis (Appendix D.4.5) describes agents' choices during development: in 139 sessions with usable scores and standard errors, 73% of Haiku's accepted gains and 44% of Opus's were smaller than the selected solution's reported standard error.

Among attempts that surpassed the reproduced baseline after these exclusions, transcript coding found that 94% included work beyond parameter tuning, such as modifying the method, repairing code, or testing alternatives; 6% involved tuning alone (Section 5.3). We clarified the score criterion in Section 5.2 and expanded the exclusion analysis in Appendix D.4.6.

Q3: Fairness and strength of the reproduced baselines. To validate the strength of our baselines, we compared two implementations of the paper's method: one from the author agent and one from the independent reproducer. We tested seven tasks from the 120-task panel, selected based on available code and evaluation data. The reproducer's implementation was stronger on four tasks when both were scored using each task's held-out data and evaluator. Solvers surpassed the reported baseline on all seven tasks and the stronger of the two implementations on six. These results showcase the robustness of the solver gains on these tasks to the choice of baseline implementation: most wins remain even when we use the stronger baseline.

We hope these clarifications and additional results address your concerns. Please let us know if there is anything else we can clarify.

Dear Reviewer 8SLV,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Thank you for the clarifications and additional analyses. The results on expert-rated task subsets substantively address my concern in Q1. The alternative-baseline comparison also provides initial support for the robustness of the reported gains, partially addressing Q3.

Dear Reviewer 8SLV,

Thank you for taking the time to read our response and for confirming that the expert-rated subset analysis addresses Q1 and that the alternative-baseline comparison partially addresses Q3.

We are glad these additions were helpful, and we will retain them in the final version (Appendices J.2, F, and G.3). If any aspect of Q3 or our other responses remains unclear, we would be happy to provide further analysis during the remaining discussion period and future.

Best regards, The Authors

Reviewer_XbjgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a multi-agent pipeline (Author, Reproducer, Reviewer) that converts published research papers into executable environments for recursive self-improvement (RSI). Each environment pairs a starting solution, a reproduced baseline, and a normalization target, and the authors build 210 environments spanning 18 domains. They evaluate several models (Opus 5, GPT-5.6-sol, DeepSeek, Haiku 4.5) as solvers and report improvement over sessions, plus an expert study on a subset of environments.

Critical strengths
  1. Automatically sourcing RSI environments from the literature addresses a genuine bottleneck, and 210 environments across 18 domains is a meaningful scale for this line of work.
  2. The end-to-end construction has sensible validation stages: requiring a reproduction to beat the starting solution before an environment passes is a reasonable gate against trivial or broken environments.
Critical weaknesses
  1. The most serious issue is that held-out data leakage is retained in the main results. The paper reports that 22 of 480 chains (4.6%) accessed held-out material or another run's graded record, yet keeps these chains in the headline numbers and relegates the exclusion analysis to an appendix ("change at most 0.006"). Leakage in the main results undermines the validity of the reported improvements.
  2. The baseline definition is circular. A reproduced baseline must outperform the starting solution to pass acceptance, which turns the starting solution into a tunable knob for controlling difficulty, since the authors can weaken it to guarantee passage. Table 12's starting solution ("raw sample covariance, without localization or inflation") appears deliberately weak.
  3. The normalization target is uncalibrated. The paper does not report the target source per environment, nor analyze attainability. Figure 6 shows the reproduced baseline's median normalized score is 0.55, but without target calibration these normalized scores are hard to interpret.
  4. There is no ablation of the construction roles. The three agents share one model (Opus 5), so their independent contributions are untested. 46% of environments trigger Reviewer repair requests, but no evidence shows these repairs improve environment quality.
  5. The expert-versus-agent comparison suffers from information asymmetry. Expert packets include solver-result summaries that are withheld from the agent judges, yet the paper directly compares the two rating streams (Figure 2).
  6. Selection bias is not addressed: only 213 of 2,117 eligible papers entered the run (p.5, l.536–537), and the sampling criterion is not stated, so the 85% construction success rate may be optimistic.
  7. The novelty is incremental, because the core idea of automatically building executable environments from published papers is largely anticipated by work the paper itself cites: EnvScaler (Song et al., 2026), EnvFactory (Xu et al., 2026), and SPADE (Liu et al., 2026) already synthesize executable, verifiable environments programmatically, TermiGen (Zhu et al., 2026) already uses iterative multi-agent refinement, and Theiler et al. (2026) already translate published methods into a shared benchmark framework, yet the paper distinguishes itself from Theiler in a single sentence ("across fields / with independent reproduction") with no quantitative or mechanistic comparison.
Questions for authors

See Weakness.

9 comments

Thank you for your detailed review, we address W1-W3 here and W4-W7 in the second reply.

W1: Leakage and the main results. Thank you for pointing this out. We have updated the results reported in the abstract, introduction, and Section 5.2 to exclude runs that accessed held-out data or other restricted evaluation materials. For each affected model and task, we excluded all three attempts. After this exclusion, at least one model improves beyond its first attempt on 100/120 tasks (83%), and solvers surpass the reported baseline on 67/120.

We identified 26 affected runs across 24 tasks; each run consists of one model's three attempts on a task. The earlier audit reported 22. All 120 tasks remain in the denominator. Table 4 excludes affected attempts individually; the stricter exclusion described above keeps the model ordering unchanged and changes any final-score median by at most 0.003 (updated Appendix D.4.6).

W2: Starting-solution difficulty and baseline wins. We'd like to clarify that we compare solvers with each task's baseline implementation, rather than its starting solution. Making the starting solution weaker would not change whether a solver beats that fixed baseline.

In the weather example you cite, the starting solution is a basic ensemble Kalman method using raw sample covariance. Its held-out error is 1.786, compared with 1.390 for the reproduced paper method. A solver must beat the latter to count as surpassing the baseline (Appendix C.3).

We also examined whether repairs weakened starting solutions to pass the reproduction check. Of 34 tasks requiring repairs at that stage, 27 had readable repair records; none showed a lowered starting solution. Only one repair edited starting-solution code, changing comments or docstrings. We added this audit to Appendix J.3.

W3: Target sources and interpretation. We use a mathematical bound or an idealized solution to set each task's normalization target. For example, one change-detection task uses a detector supplied with the true change point and distribution parameters, which the solver must infer. The supplement includes such target descriptions in the per-task tests/anchors.json files (Appendix D.2).

The scale maps the starting score to 0 and the target to 1. A normalized score of 0.55 therefore means 55% of that interval; it does not mean 55% of the best performance achievable by a solver. Appendix J.3 identifies the 84-reference subset used for the median you cite.

We added a comparison of each task's best final solution with its normalization target to Appendix I. Using the same exclusions as Table 4, we found that the target was exceeded on 11/120 tasks.

W4: Value of separate roles and evidence from repairs. Following your suggestion, we have added the workflow comparison in Appendix G and the repair checks in Appendix F. We compared the three-agent workflow with one agent performing construction, reproduction, and review in a single session. We used the same model, prompts, and checks on 15 papers for which the three-agent workflow had completed a build. The single-agent version completed 11 builds; four stopped because the agent could not independently review its own work.

Among the 11 completed single-agent builds, nine were marked as passing despite a review report saying that independent assessment was not possible. None of the 15 builds with separate agents had this contradiction (Appendix G.2).

The difference also appeared in one task where observations should arrive one step at a time. The single-agent environment exposed future observations: changing only those observations changed the solver's first output in all 12 tested runs. The environment built with separate agents supplied observations one step at a time, and the same test changed none of the 12 outputs. The broader robustness tests were tied at 9/11 tasks (Appendix G.3).

Review requests led to edits in files used by solvers or evaluators in 163/262 logged repair cases. An independent LLM judge assessed 60 reviewer-author exchanges; after weighting by construction stage, an estimated 63% of replies were judged to address the reported issue. In 31/33 searchable cases, the text flagged for change was absent from the final files.

W5: Information available to experts and agent judges. To check how the different review materials affected the ratings, we repeated the agent ratings on the 46 tasks whose expert packets included solver scores. We used one judge and the same rubric, comparing the original prompt with a version that added the build records and solver scores given to experts.

With these materials added, the judge's average rating matches the experts' average, compared with a 0.13-point higher average when rerunning the original prompt. For resistance to saturation, the average difference falls from 0.57 to 0.13 points. The average size of disagreements on individual ratings is unchanged. We have added this comparison to Appendix J.1 and updated Figure 2's caption to explain the original difference in review materials.

W6: Paper selection and the 85% completion rate. We selected 213 papers by hand from the 2,117 eligible papers, without a formal priority score or ranking rule. We stopped construction at 180 deliveries, giving an 85% completion rate over those 213 attempts.

Eligibility was assessed separately by compute budget, using the research objective, measured baseline, executable metric, computational requirements, room for improvement, and redistribution licence. The existing screening criteria and construction funnel are in Sections 4.1-4.2 and Appendix B.1.

W7: Contribution relative to related systems. Thank you for suggesting a more detailed comparison. We have expanded Section 2 and added Appendix H to explain how RSI-Forge differs from these systems. RSI-Forge builds a task and evaluator for each source problem, includes a separate reproduction stage, and measures further solution improvements against the task's reported baseline.

EnvScaler and EnvFactory synthesize environments for tool-use training; SPADE adapts executable environments through self-play; TermiGen constructs terminal environments and training trajectories.

Theiler et al. is the closest comparison: it translates methods from 16 prognostics and health-management papers into a shared benchmark framework. In RSI-Forge, the reproducer works without access to the task author's implementation; its method is then graded under the same interface and execution budget as subsequent solver solutions. The resulting corpus contains 210 environments across 18 fields, with expert assessments on 90 and repeated solver evaluation on 120 (Section 2; Appendix H).

We hope these clarifications and additional results address your concerns. Please let us know if there is anything else we can clarify.

Dear Reviewer Xbjg,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Comments on the paper

1 comment

Thank you for the careful reviews and constructive suggestions. We appreciate the recognition of the importance of automatically generating research tasks (uP8v, Xbjg), the scale and breadth of our 210 environments across 18 fields (uP8v, Xbjg), and the value of combining expert review with iterative solver evaluation (8SLV).

Following your suggestions, we have made three main updates:

  • Task value and robustness. Added checks on expert-rated tasks and alternative baseline implementations: improvement persists on highly rated tasks, and six of seven tested wins remain against the stronger baseline.
  • Construction design. Added a comparison with single-agent construction, repair checks, and cost analysis. Separate sessions preserved independent review with similar total recorded cost in this comparison.
  • Reporting and related work. Updated the results after excluding affected runs, repeated agent ratings with the additional materials given to experts, and expanded the related-work comparison.

Rebuttal-related changes in the paper are highlighted in purple. Our individual replies provide the detailed evidence and address each reviewer's points.

We hope these clarifications and additional results address your concerns. Please let us know if there is anything else we can clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.