acceptodds
Under review as a conference paper at ICLR 2027

ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents

Abstract

ScienceBuddy is an interactive scientific research workspace that brings evidence access, computational analysis, and researcher interaction into a single conversational workflow. At its core is recursive-in-recursive self-improvement, which couples harness evolution with model reinforcement learning: the inner recursion improves the harness with the model fixed, while the outer recursion trains the model under the improved harness. Harness evolution shapes training experience, and model learning creates new opportunities for harness adaptation. Given executable scientific tasks and independent verifiers, interaction feedback guides full-program revisions, while fresh verifier-scored rollouts train the model. We examine the resulting execution effects, training effects, and dependence on update order. Across seven scientific task categories, the reported two-cycle system reaches 70.4% Test accuracy from a 43.2% initial baseline. Crossed training and evaluation harnesses distinguish model learning from test-time program replacement; a separate control favors alternating updates over one-shot sequencing at equal proposal and optimizer-update counts. Reply-availability and editing-scope controls further characterize procedural adaptation. These findings connect the framework to observable improvements in scientific task execution, with task and verifier construction outside the evaluated method.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
recursive self-improvement, scientific agents, harness optimization, agent evolution, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_52kQOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces ScienceBuddy, an interactive scientific workspace with recursive-in-recursive self-improvement (R²SI). The inner loop fixes the task model and uses a separate auxiliary model to revise the execution harness from trajectories and feedback; the outer loop fixes the selected harness and updates the task model through GRPO using fresh, independently verified rollouts. Evaluated on 548 training tasks and 70 test tasks across seven scientific categories, the two-cycle system improves accuracy from 43.3% to 71.0%. Crossed training/evaluation harnesses and update-order controls investigate whether the gains arise from execution changes, model learning, or their scheduling.

Critical strengths

+) The crossed training/evaluation design is useful. Comparing learned models under a common evaluation harness helps distinguish improvements in model learning from gains due to test-time program replacement.

+) The paper integrates scientific tools, executable analysis, and inspectable trajectories into a practical workspace. The substantial reported improvement provides encouraging evidence for combining harness adaptation with model training.

Critical weaknesses

-) The methodological novelty over existing harness–model co-optimisation approaches is insufficiently established, and some claimed contributions extend beyond the evaluated pipeline.

-) The main findings rely on single search/training trajectories, while several controls introduce sampling or compute differences. Consequently, the robustness and attribution of the reported gains remain uncertain.

Questions for authors
  1. The central methodological idea is to alternate between harness adaptation and model updates. However, closely related approaches, including Co-Harness, SIA, and MetaClaw, already combine harness or skill adaptation with weight updates. Table 14 distinguishes this work through full-program revisions, fresh-rollout GRPO, and execution/training/scheduling analyses. These are potentially useful differences, but the experiments primarily compare against component-only baselines and internal variants. Such comparisons establish the benefit of combining components within ScienceBuddy, but do not establish the added value over existing joint optimisation methods. What is the essential methodological distinction, and what evidence isolates its benefit?

  2. The strongest empirical claims concern improved learning under an adapted harness and the advantage of alternating updates. However, Sections 3.4–3.6 and Appendix D clarify that these comparisons generally involve one search/training path per condition, followed by repeated evaluations of fixed checkpoints. Evaluation repeats cannot quantify variability in program search or RL training. Moreover, matching proposal and optimiser-update counts does not match sampled trajectories or tokens; the complete trace-only control changes the refill cap; and Table 13 reports substantially different auxiliary-model usage for full-program and prompt/skill editing. The authors appropriately acknowledge these limitations, but they leave uncertainty about which gains are reproducible and attributable to the intended intervention. Are independent learning runs available, and what are the actual sampling and auxiliary-compute costs for the central comparisons?

  3. The contribution statement includes deriving executable tasks and evaluation rubrics from scientific collaboration. Yet Sections 2.2–2.4 and Appendix C specify that the experiments use supplied tasks and fixed verifiers, without environment augmentation, while Appendix H explicitly separates real researcher interactions from the evaluated learning process. The quantitative experiments instead use simulated user replies. This supports a narrower claim about alternating optimisation on predefined scientific tasks, but does not validate the full interaction-to-learning pipeline. Which parts of this claimed contribution have been implemented and evaluated, and should the contribution statement be narrowed accordingly?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 52kQ (Part 1/2)

We sincerely thank the reviewer for the careful reading and for recognizing the value of our crossed training/evaluation design. We address the three questions point by point below; all changes in the revised manuscript are highlighted in blue. Below, H0/M0 are the initial harness/model, Hk/Mk those after cycle k, and all numbers are accuracy (%) on the 70 held-out Test tasks.

Q1: Co-Harness, SIA, and MetaClaw already combine harness or skill adaptation with weight updates, while the experiments compare only with component-only baselines and internal variants. What is the essential methodological distinction, and what evidence isolates its benefit?

A1: The key methodological difference is that R²SI updates the harness and the model with different learning signals: the harness is revised only from interaction feedback (execution traces and user replies), without access to verifier outcomes or reference answers, while the model is trained with RL only on verifier rewards. In addition, R²SI measures how the harness used during training changes what the model learns, separately from how the harness helps at test time. Co-Harness, SIA, and MetaClaw show that combining harness adaptation with weight updates is effective, and R²SI builds on this line of work.

1. How R²SI differs from these methods.

Co-HarnessSIAMetaClawR²SI
What the harness update changesLocal diffs to prompts, tools, skills, middlewareScaffold rewrites (tools, prompts, retries, search)New skills from failuresThe whole harness program, incl. harness-run input checks
Signal for the harness updateFailed trajectories and validation rolloutsTask verifierFailed trajectoriesExecution traces and user replies only
Model updateSFT on verifier-filtered trajectoriesTest-time RL with LoRARL with a process reward model, LoRAOn-policy GRPO under the current harness
  • The harness never sees verifier outcomes or reference answers. They are hidden from the simulated user, the feedback interpreter, and the harness improver (Appendix B.1); the verifier serves only as the RL reward. This mirrors real use: researchers give feedback, not reference answers.
  • A harness revision can rewrite the whole program and is accepted by automatic checks. A revision may change control flow or add input checks, and it is accepted once it passes interface and execution checks, so no labeled validation set is needed.
  • The model is trained with on-policy GRPO and evaluated on held-out tasks. Training uses 548 Train tasks, and all results are on 70 disjoint Test tasks.
  • We measure the effect of the training harness on the model. The crossed design evaluates models trained under different harnesses under the same harness, so differences come from training, not from the test-time harness.

2. Evidence for each difference.

  • Interaction feedback improves harness revision. With the model fixed, revising the harness from user replies reaches 47.62 vs. 44.29 from traces alone (Table 7); over two full cycles, 71.0 vs. 62.86 (Table 8), although the trace-only run sampled more permissively (see A2).
  • Editing the whole program outperforms editing only prompts and skills. With the model fixed: 47.62 vs. 44.29 (Table 7). With the final model, the evolved harness reaches 71.0 vs. 46.67 for a hand-designed fix of H0 (Table 6). This agrees with Meta-Harness (Lee et al., 2026), where end-to-end optimization of harness code outperforms hand-designed harnesses.
  • The training harness changes what the model learns. Starting from the same M1 with the same 15 RL updates, training under the adapted harness H2 instead of the previous harness H1 improves accuracy by +6.19 when both models are evaluated under H1, and by +9.52 under H2 (Table 11).
  • Alternating the two updates outperforms running them once in sequence. With equal proposals and RL updates, alternation reaches 71.0 vs. 54.29 for all proposals followed by all RL updates (Table 12).
  • On-policy GRPO is well supported. In Shao et al. (2024), GRPO outperforms offline and online rejection-sampling fine-tuning because it also learns from incorrect responses.

3. Why we compare design choices within one setting. Tasks, backbones, and budgets differ across these papers, so their numbers are not directly comparable (Appendix G). We therefore compare each design choice with its alternative in one setting, as these works do for their own designs: Co-Harness compares its joint rounds with a hand-designed harness and harness-only evolution, SIA compares harness-only with harness-plus-weight updates, and MetaClaw compares skill-only adaptation with its full system. We follow the same protocol and add an RL-only baseline, the crossed comparison, and the schedule comparison.

Following your suggestion, we have revised Table 14 and Appendix A to state these differences and their evidence.

AuthorsOpenReview ↗

Response to Reviewer 52kQ (Part 2/2)

Q2: The main findings rely on one search/training path per condition, and some controls differ in sampling or auxiliary compute. Are independent learning runs available, and what are the actual sampling and auxiliary-compute costs for the central comparisons?

A2: The paper reports one learning run per condition, which is common for full self-improvement runs (e.g., the main SWE-bench result of the Darwin Gödel Machine comes from a single run), and it contains two complete, independent alternating runs whose results agree. The central comparisons are matched in the number of attempts used for training, and the remaining differences in sampling and auxiliary compute either produce no gradient or favor the condition that R²SI is compared against.

1. Independent learning runs.

  • The paper contains two complete alternating runs. Besides the main run, the trace-only run (Appendix D.2) performs its own harness search and RL starting from M0. The two end at 71.0 and 62.86, both far above the single-component baselines (47.62) and the one-shot schedule (54.29).
  • Both runs show the same pattern. Every RL phase improves accuracy (main: 38.57→60.95 and 64.29→71.0; trace-only: 49.52→51.43 and 61.43→62.86), and so does returning to harness adaptation after the first RL phase (60.95→64.29; 51.43→61.43).
  • The effect of the training harness holds under both evaluation harnesses (+6.19 and +9.52; Table 11).

2. Sampling cost. Each harness proposal is evaluated on 16 Train interactions, and each RL update trains on 8 task groups × 8 attempts (Table 3).

ConditionHarness proposalsTrain interactions for harness searchRL updatesRL attempts used in updates
Harness-only3048000
RL-only00402,560
R²SI (alternating)14224301,920
One-shot (all proposals, then all RL updates)14224301,920
  • Alternating and one-shot have the same training signal. Both use the same proposals, interactions, updates, and attempts used in updates. The number of sampled trajectories can differ, because each update keeps sampling batches until it has 8 groups with mixed rewards. The extra groups have identical rewards within the group, so their GRPO advantage is zero (Eq. 4) and they contribute no gradient.
  • Each single-component baseline receives more budget for its own component than R²SI.
  • The trace-only comparison favors the trace-only run. Its refill cap was removed, so it always collects the full 8 mixed-reward groups per update, whereas the main run uses fewer groups when 4 batches do not suffice. Normal feedback still reaches higher accuracy (71.0 vs. 62.86).

3. Auxiliary compute. The alternating and one-shot schedules use the same auxiliary model and the same 14 proposals. Table 13 compares two harness-adaptation conditions with the model fixed. They have identical budgets (7 proposals, 112 Train interactions, and the same per-candidate limits of 3 evidence reads, 2 repair attempts, and 6 auxiliary calls; Appendix B.1) and similar numbers of auxiliary requests (201 vs. 182). This comparison does not involve the joint optimization.

We have added the budget table and these clarifications to Appendix B.

Q3: The contribution statement includes deriving tasks and rubrics from collaboration, but the experiments use supplied tasks, fixed verifiers, and simulated replies. Which parts are implemented and evaluated, and should the statement be narrowed?

A3: Yes, we have narrowed the contribution statement to what is implemented and evaluated: the ScienceBuddy workspace, R²SI on given scientific tasks with fixed verifiers and simulated user replies, and the controlled comparisons. Constructing tasks and rubrics from researcher collaboration and augmenting the environment belong to the general framework, which we now describe separately in Appendix I. The revised contributions are:

  1. ScienceBuddy, a scientific research workspace with a pluggable harness interface, on which all experiments run.
  2. R²SI, which alternates whole-program harness revision guided by interaction feedback with GRPO training on verifier rewards.
  3. Controlled comparisons that separate three effects: the harness at test time, the training harness on the model, and the order of the updates.

We now present the researcher cases in Appendix H as illustrations of the workspace, and we have removed "returning the updated system to researchers" from the claims. We have also restructured Sections 3-4 so that the main text describes exactly the configuration that was run, which removes most of the qualifiers that made the paper hard to follow.

We thank the reviewer again for the detailed feedback. We hope these clarifications and revisions address your concerns, and we would kindly ask you to reconsider your score. Please feel free to let us know if you have any further questions.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 52kQ,

We sincerely appreciate the time and effort you dedicated to reviewing our paper. Your thoughtful questions and insightful feedback have been extremely beneficial. In response to your queries and concerns, we have prepared detailed answers .

Should there be any further points that require clarification or improvement, please know that we are fully committed to addressing them promptly.

Thank you once again for your invaluable contribution to our research.

Warm Regards, The Authors

Reviewer_xoyfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ScienceBuddy, an interactive scientific research agent framework that combines harness evolution with model reinforcement learning through a Recursive-in-Recursive Self-Improvement (R2SI) paradigm. Specifically, the inner loop iteratively improves the agent harness based on execution trajectories and user feedback while keeping the model fixed, whereas the outer loop updates the model parameters through GRPO using verifier-scored trajectories collected under the evolved harness. The two processes alternate, allowing harness improvements to influence model training and updated models to provide new opportunities for harness adaptation. The authors evaluate ScienceBuddy on 548 training tasks and 70 test tasks across seven scientific task categories. The proposed approach improves overall test accuracy from 43.3% to 71.0%, outperforming harness-only and RL-only baselines. Additional experiments investigate the effects of training with different harnesses, alternating versus sequential optimization, and the roles of user feedback and editable harness components. The results provide preliminary evidence that harness adaptation and model training can complement each other in scientific agent development.

Critical strengths
  1. I appreciate the authors’ effort in developing ScienceBuddy and exploring the co-evolution of agent harnesses and model parameters. The proposed framework addresses an important direction in agent self-improvement: harness modifications can improve not only immediate task execution but also the training experiences from which the underlying model learns. Integrating these two optimization processes into a unified scientific agent framework is a meaningful contribution.

  2. Beyond the overall performance improvements, I appreciate the additional experiments that attempt to disentangle the effects of harness adaptation and model training. In particular, the crossed training/evaluation harness experiments and the comparison between alternating and sequential optimization provide useful insights into how harness changes may influence subsequent model learning. These analyses strengthen the empirical contribution of the paper.

Critical weaknesses

Overall, I feel this is good work, and the proposed system is useful and inspiring. The following points are more reservations and open questions than fundamental weaknesses.

  1. The joint optimization of harnesses and model parameters has already been explored in related works such as MetaClaw, SIA, and Co-Harness. This paper provides useful additional analyses and explores somewhat different settings, such as scientific task execution versus more general agent environments, and verifier-based GRPO training versus SFT-based approaches. While the core methodology may not be particularly novel, I personally do not consider this a major concern. Given the increasing convergence of research directions in this area, I value solid empirical evidence and careful analysis alongside methodological novelty, and I think this paper makes a meaningful contribution in that regard.

  2. This could also be an open discussion rather than a true weakness. I observe that the demonstrated improvements mainly involve general file-handling and data-parsing procedures, rather than more sophisticated scientific reasoning or planning strategies. Moreover, most performance gains are concentrated in Database QA and Genetics QA, suggesting that the improvements may largely stem from dataset-specific tool-use heuristics. I would be more excited to see evidence of harness evolution that develops more generalizable scientific reasoning and problem-solving skills beyond these relatively narrow procedural improvements.

Questions for authors
  1. The case studies mainly demonstrate improvements in file parsing and data structure inspection. Could the authors provide a more systematic characterization of the evolved harness modifications? For example, how many modifications involve general file-handling procedures versus scientific reasoning, planning, or tool-use strategies? Are there examples of more sophisticated harness improvements that transfer across different scientific task categories?

  2. In particular, could the authors further clarify the key methodological differences from Co-Harness and MetaClaw? It would be helpful to understand whether the observed improvements are primarily attributable to full-program editing, GRPO-based model training, or the alternating optimization schedule, and whether these choices offer advantages over existing approaches.

  3. The crossed-harness experiments suggest that harness adaptation can improve subsequent model training. However, it remains unclear what new capabilities or behaviors the model actually acquires. Could the authors provide representative before/after RL trajectories under a fixed harness, illustrating whether the improvements arise from better tool use, reasoning, error recovery, or more reliable task completion?

3 comments
AuthorsOpenReview ↗

Response to Reviewer xoyf (Part 1/2)

We sincerely thank the reviewer for the thoughtful assessment, for valuing careful empirical analysis, and for finding the crossed training/evaluation design and the schedule comparison useful. We address each point below; all changes in the revised manuscript are highlighted in blue. Below, H0/M0 are the initial harness/model, Hk/Mk those after cycle k, and all numbers are accuracy (%) on the 70 held-out Test tasks.

W2 & Q1: The demonstrated improvements mainly involve file handling and data parsing, and the gains concentrate in Database QA and Genetics QA, which suggests dataset-specific tool-use heuristics. Could the authors characterize the evolved harness modifications more systematically, and are there more sophisticated improvements that transfer across categories?

A1: The evolved edits are general procedures rather than dataset-specific shortcuts, and they transfer to unseen Test questions. At the same time, we agree with the reviewer that harness evolution alone mainly improves the retrieval-heavy categories, while the reasoning-heavy categories need RL on the model. We discuss both points below.

1. The accepted edits are data-inspection procedures, and they account for the gains from harness evolution.

  • A proposal may revise instructions, skills, context construction, input inspection, and execution control (Section 3.3).
  • The accepted edits documented in the paper add input checks that the harness runs before the model's first turn, exposing the actual structure of the data: the field layout of a GMT gene-set file and the nested hierarchy of ClinVar records (Appendix F).
  • The ablations show that these procedures, rather than prompt wording or answer formatting, account for the gains: restricting edits to prompts and skills lowers accuracy from 47.62 to 44.29 (Table 7), and a hand-designed fix of H0's answer and error handling reaches only 46.67, compared with 71.0 for the evolved harness with the same model (Table 6).

2. The edits are general rules that improve unseen Test questions. They are learned from Train interactions and help on Test without any further feedback.

  • The GMT rule (the set name is the transcription factor and the members are its targets) applies to any GTRD gene set. Learned from a Train question about ZFP91, it turns a different Test question about ZNF436 from consistently incorrect to consistently correct.
  • The nested-structure check applies to any nested record. Learned from a path error in one ClinVar comparison, it lets the model solve a different variant-classification question.
  • By design, task-specific answers and facts stay local to their task (Section 3.3), so an accepted edit cannot carry answers across tasks.
  • At the category level, edits learned on Train improve the held-out Test tasks in Database QA (+15.6) and Genetics QA (+28.3).

3. Harness evolution helps retrieval; reasoning-heavy categories need RL on the model. We agree with the reviewer on this point (Table 1):

  • Harness-only (harness search with the model fixed) improves Database QA (+15.6) and Genetics QA (+28.3), but lowers the self-contained categories, e.g., Step Ordering (−26.7), Error Detection (−26.7), and Physical/Quantum Chemistry (−14.8).
  • RL-only (GRPO with the harness fixed at H0) improves some of these reasoning categories, e.g., Step Ordering (+6.7) and Other Chemistry (+5.6).
  • R²SI improves them further: Step Ordering +20.0, Error Detection +6.7, and Other Chemistry +5.6.

We interpret this as a division of labor: harness procedures make evidence reliably accessible, while reasoning ability is built in the weights. This agrees with SIA (Hebbar et al., 2026): "harness updates make the model agentic, shaping how it searches and acts, while weight updates build the domain intuition that no prompt or scaffold can instil." Improvements that transfer across categories therefore appear when both are combined. Evolving harnesses that encode more general reasoning strategies is a promising direction suggested by these results. We have added this discussion to Sections 4.2–4.3.

AuthorsOpenReview ↗

Response to Reviewer xoyf (Part 2/2)

Q2: What are the key methodological differences from Co-Harness and MetaClaw? Are the improvements primarily attributable to full-program editing, GRPO-based training, or the alternating schedule, and do these choices offer advantages over existing approaches?

A2: R²SI differs from Co-Harness and MetaClaw mainly in the learning signals: the harness is revised from interaction feedback without verifier outcomes, and the model is trained with on-policy GRPO. Among the design choices, the alternating schedule contributes most, followed by training under the adapted harness and then editing the whole harness program.

1. How R²SI differs.

Co-HarnessMetaClawR²SI
What the harness update changesLocal diffs to prompts, tools, skills, and middlewareNew skills synthesized from failuresThe whole harness program, including input checks run by the harness
Signal for the harness updateFailed trajectories and validation rolloutsFailed trajectoriesExecution traces and user replies only
Model updateSFT on verifier-filtered trajectoriesRL with a process reward model, LoRAOn-policy GRPO under the current harness

2. Each ablation changes one factor.

  • Alternating schedule: 71.0, compared with 54.29 for all harness proposals followed by all RL updates, at equal proposal and update counts (+16.67; Table 12).
  • Training under the adapted harness: starting from the same M1, training under H2 instead of H1 gives +6.19 when evaluated under H1 and +9.52 under H2 (Table 11).
  • Editing the whole harness program: +3.33 over editing only prompts and skills, with the model fixed (Table 7).

3. Each choice outperforms the corresponding alternative in prior work. The ablations above compare each choice with its alternative in our setting. For GRPO versus SFT on filtered trajectories, as in Co-Harness, we follow Shao et al. (2024), where GRPO outperforms offline and online rejection-sampling fine-tuning because it also learns from incorrect responses. Numbers across papers are not directly comparable because tasks, backbones, and budgets differ (Appendix G). We have revised Table 14 accordingly.

Q3: What new capabilities or behaviors does the model acquire under a fixed harness? Do the improvements arise from better tool use, reasoning, error recovery, or more reliable task completion? Could the authors provide representative before/after RL trajectories?

A3: Under a fixed harness, RL mainly makes the model answer correctly more often at an unchanged submission rate, and the learned behavior transfers best to the harnesses used in training; at the system level, executions also become more reliable and shorter.

1. RL improves answer correctness, not just the submission rate.

  • Under the fixed hand-designed fix of H0, the final model reaches 46.67 vs. 40.00 for the initial model, at nearly the same submission rate (90.95% vs. 91.43%; Table 6). The gain thus comes from answering correctly, not from completing more attempts.
  • The gains depend on the harness used in training. Relative to the initial model, the final model gains +28.10 under H1 and +22.43 under H2, the two harnesses it was trained with, but only +6.67 under the fix of H0, which it never saw (Tables 6 and 11). The model therefore learns to use the procedures of its training harness, which is the effect of the training harness measured in Table 11.

2. The gain is mainly in correctness, with more reliable and efficient execution. From the initial to the final system, unsubmitted attempts fall from 21.9% to 3.8%, truncated runs from 6.7% to 2.9%, model calls per task from 3.23 to 2.69, and tool calls per task from 2.48 to 1.91, while correct answers per 1,000 output tokens double from 0.187 to 0.376 (Figures 9 and 10).

3. On representative trajectories. We report fixed-harness comparisons aggregated over all 70 Test tasks rather than selected examples, which characterizes the behavior change across all categories; trajectory-level examples of the harness edits are given in Appendix F. We have added these fixed-harness comparisons to Appendix D.5.

Ethics flag.

We assume the flag concerns the researcher interactions in Appendix H. Both researchers agreed to the recording of these sessions and to their presentation in the paper. The displayed interfaces are anonymized English reconstructions with account and model identifiers masked, and these sessions were not used for training or quantitative evaluation; all learning results use benchmark tasks and simulated user replies. We have added this statement to Appendix H, and we are happy to provide further details if the reviewer had a different concern in mind.

We thank the reviewer again for the constructive feedback. We hope these responses address your concerns, and please feel free to let us know if you have any further questions.

Reviewer_xoyfOpenReview ↗

Response to Authors' Rebuttal

Thank you for the comprehensive responses and additional analyses. The clarifications and experimental results have addressed my main questions. I maintain my original rating.

Reviewer_6NBbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ScienceBuddy, a workspace combining a research environment with researcher interactions, which acts as a platform to evaluate scientific agents. The paper proposes a two-layer recursive framework for self-improving scientific research agents. The inner layer iterates on improving the agentic harness while the outer layer uses reinforcement learning to train the backbone LLM on trajectories from the inner loop. The framework improves performance of the agent across multiple domains compared to the initial harness H0 and also against iterating on just the harness or the model only.

Critical strengths

The paper proposes a novel two-layer self-improvement framework for agents. Although other meta-frameworks exist that iterates on the agentic harness itself, the paper is unique in that it interleaves RL training (gradient updates) with agentic harness improvements (text-based updates).

The findings and ablations are interesting and insightful. In particular, I found the finding interesting that the evolved harness does more than just program repair since submission rate does not predict performance. The paper also had nice ablations for changing how you alternate between the inner and outer loops and ablating the type of feedback (trace only vs normal.

Critical weaknesses

I don't think it is too surprising that the recursive self-improvement improves performance over the initial harness H0. I would like to see how this method compares to other harnesses and methods from previous work. The paper could benefit a lot from having these additional baselines. Similarly, the comparison to Harness-only and RL-only settings seem unclear whether it is fair (30 harness updates vs 40 RL updates vs 14 harness + 30 RL updates). Why were these chosen and what makes these reasonable numbers for comparison?

Does not explain why some types of tasks have the initial H0 have best (tied) performance. It seems odd that recursive self-improvement worsens performance (harness-only and RL-only) for Troubleshooting and Physical/Quantum chem topics. The results of the paper would be a lot stronger if these are explained.

Questions for authors

The ablation results claim that alternating is better than one-shot. Did you try alternating even more? e.g. H3 - RL5 - H3 - RL5 - ...

Is this method sensitive at all to the starting harness H0? Depending on what this seed harness is, would we expect different results?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6NBb (Part 1/3)

We sincerely thank the reviewer for the thoughtful review and for recognizing the novelty of interleaving RL training with harness improvement, the finding that the evolved harness does more than program repair, and the schedule and feedback ablations. We address each point below; all changes in the revised manuscript are highlighted in blue. Below, H0/M0 are the initial harness/model, Hk/Mk those after cycle k, and all numbers are accuracy (%) on the 70 held-out Test tasks.

W1: Improving over H0 is not surprising. How does the method compare with other harnesses and methods from previous work? Are the Harness-only and RL-only budgets (30 harness updates and 40 RL updates vs. 14 + 30) fair, and why were these numbers chosen?

A1: Beyond H0, the paper compares R²SI with a hand-engineered harness, automated harness optimization, and agentic RL, and R²SI outperforms all three under budgets that favor the baselines. We agree that improving over H0 alone would not be sufficient, and we address the three parts of the question below.

1. R²SI outperforms existing harness designs and training methods. R²SI reaches 71.0, compared with 46.67 for a hand-engineered harness, 47.62 for automated harness optimization, and 47.62 for agentic RL.

  • H0 is a standard harness, not a deliberately weak one. It is a ReAct-style loop with the scientific tool catalog and execution utilities of Biomni (Huang et al., 2025).
  • Hand-engineered harness: 46.67. A generic fix of H0 improves how it handles answers and errors (Table 6). With the same final model, it reaches 46.67 vs. 71.0 for the evolved harness, at a similar submission rate (90.95% vs. 96.2%).
  • Automated harness optimization: 47.62. Harness-only (harness search with the model fixed) optimizes the whole harness program with 30 proposals, in the spirit of Meta-Harness (Lee et al., 2026).
  • Agentic RL: 47.62. RL-only (GRPO with the harness fixed at H0) trains the model for 40 updates.

Tasks, backbones, and budgets differ across papers, so their numbers are not directly comparable (Appendix G). We therefore follow prior joint methods and compare within one setting: Co-Harness compares with a hand-designed harness and harness-only evolution, SIA with its initial scaffold and harness-only updates, and MetaClaw with a baseline and skill-only adaptation.

2. The budgets favor the baselines. Each baseline receives more of its own component than R²SI: more harness proposals for Harness-only (30 vs. 14), and more RL updates (40 vs. 30) and more Train attempts used for learning (2,560 vs. 2,144) for RL-only, where R²SI's 2,144 = 224 interactions in harness phases + 1,920 attempts in RL updates. We gave the baselines longer runs because a single checkpoint can be noisy, and a longer run shows the trend. Figure 4 evaluates Harness-only every three proposals: even its best Test checkpoint (57.14), a choice that favors the baseline, remains well below R²SI (71.0).

3. The numbers were fixed in advance by two rules. They were not tuned on Test. First, each baseline's budget exceeds R²SI's total for the same component (30 > 14 proposals; 40 > 30 updates), so no baseline is limited by a smaller budget. Second, within R²SI, a harness phase and an RL phase use similar amounts of Train data (7 × 16 = 112 interactions vs. 15 × 8 = 120 task groups), so neither component dominates a cycle.

We have added this discussion and the budget comparison to Section 4.1.

AuthorsOpenReview ↗

Response to Reviewer 6NBb (Part 2/3)

W2: Why does H0 have the best (tied) performance on some categories, and why do Harness-only and RL-only lower performance on Troubleshooting and Physical/Quantum Chemistry?

A2: The pattern follows the type of task: these two categories are self-contained reasoning questions, where single-component updates, learned mostly from retrieval failures, bring little benefit or even interfere, while R²SI returns to the H0 level. We find this an important observation and give our analysis below. Grouping the categories by whether they require data retrieval makes the pattern clear (change from H0 in percentage points; Table 1):

TypeCategoryHarness-onlyRL-onlyR²SI
RetrievalDatabase QA+15.6+11.1+44.4
Genetics QA+28.3+13.3+55.0
Self-containedTroubleshooting−10.0−10.00.0
Error Detection−26.7−6.7+6.7
Step Ordering−26.7+6.7+20.0
Physical/Quantum Chem.−14.8−7.40.0
Other Chemistry0.0+5.6+5.6

1. H0 ties for best because these categories mainly test the model's own knowledge. Troubleshooting questions (LAB-Bench ProtocolQA) give the protocol in the prompt, and Physical/Quantum Chemistry questions (QCBench) are numerical calculations; neither needs a database lookup. Troubleshooting is H0's best category (76.7%), which leaves little headroom.

2. Harness-only drops because one shared harness learns mostly from retrieval failures. All categories share one harness, and its edits are learned from Train failures, of which retrieval tasks are the largest share (47%, 260/548). The accepted edits therefore add data-inspection steps (Appendix F). These help retrieval (+15.6 and +28.3) but add unnecessary steps to self-contained questions, and Harness-only is flat or lower on all five self-contained categories. We see this as negative transfer across task types within a shared harness.

3. RL-only drops slightly because these categories give little GRPO signal. This is our hypothesis. GRPO learns only from task groups with mixed outcomes. Questions that the 4B model mostly answers correctly (Troubleshooting) or that require knowledge it lacks (Physical/Quantum Chemistry) give mostly uniform outcomes, so the updates are driven by other categories. This agrees with Yue et al. (2025), who find that the reasoning abilities of RLVR-trained models "originate from and are bounded by the base model." The drops are small: 3 of 30 and 2 of 27 attempts, about one task each.

4. Joint training recovers these categories and improves others. In R²SI, the harness runs its data-inspection steps before the model's first turn, and RL then adapts the model to this harness. The final system uses fewer model calls (3.23 → 2.69) and tool calls (2.48 → 1.91) per task (Figure 10), and it exceeds both single components on Step Ordering (+20.0) and Error Detection (+6.7). Troubleshooting and Physical/Quantum Chemistry only return to the H0 level, consistent with these categories being limited by domain knowledge that neither procedural edits nor RL on about 80 Train tasks per category can add.

We have added this per-category analysis to Section 4.2.

AuthorsOpenReview ↗

Response to Reviewer 6NBb (Part 3/3)

Q1: Did you try alternating even more often (e.g., H3 → RL5 → H3 → RL5 → …)?

A3: We compare alternation with no alternation at an equal budget, because the goal of this paper is to establish the framework of jointly evolving the harness and the model, and to show that the two depend on each other during learning and must be adapted together; how often to alternate is a hyperparameter that we leave for a dedicated study.

  • Alternating helps at the same budget. With 14 proposals and 30 updates in both schedules, H7 → RL15 → H7 → RL15 reaches 71.0, compared with 54.29 for H14 → RL30 (Table 12).
  • The model and the harness depend on each other during learning. What the model learns depends on the harness it is trained under (+6.19 and +9.52 under a common evaluation harness; Table 11), and the learned behavior transfers best to the harnesses seen in training: relative to M0, the final model gains +28.10 under H1 and +22.43 under H2, but only +6.67 under a generic fix it never saw (Tables 6 and 11). A harness designed for M0 and then frozen cannot follow these changes, whereas re-adapting it after RL helps in both complete learning runs (60.95 → 64.29 with normal feedback; 51.43 → 61.43 with trace-only feedback).
  • Two cycles already show steady gains, as in prior work. In the current schedule, each harness phase uses 112 Train interactions and each RL phase 15 updates. In Figure 6, the first RL phase gains most in its last five updates (to 61.0), and the second rises steadily (64.3 → 68.1 → 69.5 → 71.0). Prior work likewise uses few rounds: two in Co-Harness, and two rounds of iterative RL in Shao et al. (2024).

We have added this discussion to Section 4.5.

Q2: Is the method sensitive to the starting harness H0? Would a different seed harness give different results?

A4: The existing results indicate that R²SI does not depend on a particular starting point: it improves from every starting configuration in the paper.

  • It improves when starting from a different model and harness. The second cycle starts from (M1, H1), and both of its phases improve the system: harness adaptation raises M1 from 60.95 to 64.29 (Table 10), and RL then reaches 71.0.
  • It improves from a different first-cycle harness. The trace-only run (Appendix D.2) performs its own harness search and obtains a different first-cycle harness (49.52 vs. 38.57 under M0). It still improves through both cycles to 62.86, well above both single-component baselines (47.62).
  • The adapted harness also helps the initial model. H2 raises M0 from 43.33 under H0 to 48.57 (Table 10).

For the choice of the seed, we follow common practice in self-improving agent work and start from a single simple harness, so that the gains reflect learning rather than manual harness engineering. The Darwin Gödel Machine starts from one lightweight coding agent "to provide a simple starting point from which self-improvement can be clearly demonstrated"; SIA starts from one initial scaffold (a system prompt, a single-tool dispatch loop, and an output parser); and GEPA starts from the base system's seed prompts. Since R²SI edits the whole harness program, it can start from any harness. We have added this discussion to Appendix D.3.

We thank the reviewer again for the constructive feedback. We hope these clarifications address your concerns, and we would kindly ask you to reconsider your score. Please feel free to let us know if you have any further questions.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 6NBb,

We sincerely appreciate the time and effort you dedicated to reviewing our paper. Your thoughtful questions and insightful feedback have been extremely beneficial. In response to your queries and concerns, we have prepared detailed answers .

Should there be any further points that require clarification or improvement, please know that we are fully committed to addressing them promptly.

Thank you once again for your invaluable contribution to our research.

Warm Regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for their detailed and constructive reviews. We are glad that the reviewers found the crossed training/evaluation design useful (Reviewers 52kQ, xoyf), the interleaving of RL training with harness improvement distinctive (Reviewer 6NBb), and the schedule and feedback ablations informative (Reviewers 6NBb, xoyf).

We have revised the manuscript according to the reviewers' suggestions, with all changes highlighted in blue. We summarize our responses to the shared questions below.

1. Difference from existing joint harness–weight methods (all reviewers; Section 2, Table 14, Appendix A). R²SI updates the harness and the model with different learning signals: the harness is revised only from interaction feedback, without access to verifier outcomes or reference answers, while the model is trained with GRPO on verifier rewards. R²SI also measures how the training harness changes what the model learns, separately from the harness's effect at test time. Co-Harness, SIA, and MetaClaw show that combining harness and weight updates is effective, and R²SI builds on this line of work. Following the evaluation practice of these works, we compare each design choice with its alternative in one setting, and each choice performs better: alternating vs. one-shot updates (71.0 vs. 54.29), training under the adapted vs. the previous harness (+6.19 and +9.52 under a common evaluation harness), and whole-program vs. prompt/skill-only editing (+3.33 with the model fixed). The choice of GRPO follows Shao et al. (2024).

2. Baselines, budgets, and robustness (Reviewers 6NBb, 52kQ; Section 4.1, Appendix B). R²SI outperforms a hand-engineered harness (71.0 vs. 46.67 with the same model), automated harness optimization, and agentic RL (both 47.62), although each baseline receives more of its own component than R²SI (30 vs. 14 harness proposals; 40 vs. 30 RL updates). The paper contains two complete alternating runs, which both end well above all baselines and show the same pattern in every cycle. The central comparisons use the same number of training attempts, and the remaining sampling differences either produce no GRPO gradient or favor the condition that R²SI is compared against.

3. Why the gains differ across task types (Reviewers 6NBb, xoyf; Sections 4.2–4.3). We agree that this is an important question and have added a per-category analysis. Harness evolution alone mainly improves retrieval-heavy categories, while reasoning-heavy categories need RL on the model; combining the two also improves categories where neither alone helps, such as Error Detection (+6.7, vs. −26.7 and −6.7 for the single components). We interpret this as a division of labor, consistent with findings in SIA: harness procedures make evidence reliably accessible, while reasoning ability is built in the weights. The evolved edits are general procedures that transfer to unseen Test questions, and under a fixed harness, RL raises accuracy at an unchanged submission rate. This also shows that the model and the harness depend on each other during learning and should be adapted together.

4. Scope of the contribution statement (Reviewer 52kQ; Sections 1–4). We agree, and we have narrowed the claims to what is evaluated: alternating optimization on given scientific tasks with fixed verifiers and simulated user replies. We have also restructured Sections 3–4 so that the main text describes exactly the configuration that was run.

Ethics (Reviewer xoyf; Appendix H). The researcher sessions in Appendix H are presented as translated, abridged summaries and anonymized interface reconstructions with account and model identifiers masked; they were not used for training or quantitative evaluation. Both researchers agreed to the recording of these sessions and to their presentation in the paper.

We reply to each reviewer's concerns in detail below their individual reviews. Thank you again, and please feel free to reach out if you need any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.