acceptodds
Under review as a conference paper at ICLR 2027

CoHARL: Co-Evolving Agent Policies and Executable Harnesses with Reinforcement Learning

Abstract

Large language model (LLM) agents interact with their environments through executable harnesses that govern tool use and feedback. Optimizing a harness for a fixed policy overlooks how its behavior changes during learning, while training a policy under a fixed harness leaves the interface that generates its training experience unchanged. This motivates adapting the harness throughout policy training. We introduce CoHARL, a population-based reinforcement learning framework that co-evolves a shared policy and a population of executable harnesses. The policy learns from rollouts across the population, and a proposer uses its execution traces to revise harness code during training. CoHARL achieves a 27.6% mean resolved rate on SWE-rebench, improving over harness-only search by 6.4 percentage points. Experiments on SWE-bench Verified and Terminal-Bench 2.0 show gains in software engineering and terminal-based tasks. Under matched training budgets, CoHARL also outperforms searching for a harness once before RL and alternating policy updates with revisions to a single active harness.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM Agents, Agent Harness, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_abMhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoHARL which brings policy update and harness evolution into one integrated RL. The proposer LLM mutates multiple pieces of harness using the parent source, current policy trajectories information, and historical accuracy-token-cost for harness candidate proposal. The accepted candidates enter the training pool with a scheduler for rollout assignment in the next round. Then, a regular rollout & policy update (GRPO-like) is performed. Experiments on SWE-rebench, SWE-verified, terminal-bench show that the gains are observed over baselines that only target policy or harness improvement.

Critical strengths
  • The problem itself is clear and fascinating. The integration of the two variables (policy, harness) is important.
  • The mutation/modification proposal makes sense.
  • The illustrations and algorithm is clear and easy to follow.
Critical weaknesses
  • Some detailed information are missing. For example, the staging relies on domain-specific candidate checks (e.g., contract to pool trials for SWE), but the reviewer did not know whether the execution success satisfies or it must have a threshold (pass/fail resolve rate). Are weak harness variations considered in the pool for staging?
  • The contribution of each aspect of the harness improvement is not clearly quantified or discussed deeply. For example, the evolved version might change test score/process reward and then GRPO optimizes policy under such a moving interface. The performance gains might come from both interface bug fixes and reward shaping?
  • There might exist an off-policy nature for the proposal where the history trajectory information rollouted with old/outdated policy weight (correct me if I misunderstand). If so, the off-policy problem (stale archive) should be considered and discussed.
Questions for authors
  • Please detail the conditions for each task (e.g., SWE/terminal) on whether a candidate harness can pass pool trials. In addition, please give the computation budget for the pool trials as well.
  • Can the authors use simple but clear explanations to clarify whether the evovled harness might affect reward std filtering/resolve rate if the process reward is involved into such proposal?
  • One concern about the training and evaluation setting is the benchmark used in experiments. The training/validation is split by date (before or after 2026), but can the authors claim there exist no overlapping between the tasks in training and validation? Especially for the SWE-rebench and SWE-verified datasets, the potential leakage by sharing git repos across tasks should be handled. In addition, the paper does not clearly indicate the dataset used to train terminal agent for validation on TB2.
  • In Sec.4.1, the computational complexity is only briefly mentioned in appendix C.1. Can the authors provide more detailed analysis on the complexity of each part during training, including the comparison with baselines (e.g., coHARL/Search-then-RL/Single-harness alternative/fixed).
5 comments

Thank you for the detailed review. We answer each point below, including the protocol details the submission left under-specified, and would welcome any further questions.

Q1: Candidate admission: is there a resolve-rate threshold, are weak harnesses admitted, what are the SWE/TB-2 conditions, and what is the pool-trial budget?

A1: There is no accuracy threshold at admission: candidates are judged by their training rollouts, so weak harnesses are admitted and then evicted by the scheduler.

  • SWE admission: a structural contract check (loads, exposes the episode interface and a valid manifest, compatible with the training loop) and a diversity check that rejects scaffolds duplicating an existing member.
  • TB-2 admission: additionally a one-task smoke test and recorded validation scores on 20 tasks, still without a threshold.
  • Trials and eviction: each newcomer gets 6 forced trials (one task group of 4 rollouts in each of its next 6 steps) and cannot be evicted before they end; at capacity (8 members), the non-seed member with the lowest Eq. 5 score is evicted.
  • Budget: forced trials come out of the regular per-step budget (12 groups × 4 rollouts), not on top of it, and are about 38% of training episodes in the 30-step full run. They enter the policy update like any other rollout, and because advantages are centered within each task–harness group (Eq. 3), a weak candidate lowers only its own group mean rather than pushing the policy in a harmful direction.

These details will be added to Appendix C.1.

Q2: Harness edits may change test score / process reward, so GRPO optimizes under a moving interface; are gains from interface fixes or reward shaping, and do evolved harnesses affect reward-std filtering?

A2: Harness evolution did not change the training reward. The reward formula (Eq. 7) and the length penalty (Eq. 8) live in the training framework, which the proposer cannot edit, so they are identical across all harness versions. The only reward-related field a harness can touch is test_score, used solely for zero-variance group filtering: the seed stores the test pass fraction and some evolved versions store the harness reward (Appendix C.1), which changes which groups are retained but not the advantages within them. Evaluation uses an external scorer that reads only the final patch and runs the official tests, so no harness edit can affect a reported resolved rate. We audited all 6 revisions accepted in the 30-step full run: the reward code is byte-identical to the seed; edits are confined to prompting, action parsing, and a pre-submission diff check.

To test this empirically we re-ran the 30-step protocol with the reward and test_score code under the same edit ban as the training framework (Table R3). With the training signal held fixed, CoHARL retains 3.9 of the 5.5 points it gains over fixed-harness RL.

Table R3: Reward-locked CoHARL (30 steps, 110 tasks, avg@3, strict resolution).

ConfigurationResolved
CoHARL (full, Table 4)25.8
CoHARL, reward-locked24.2
A1: Fixed-harness RL (Table 4)20.3

A2 (continued): which edits matter. We grouped every edit between the seed and the final harness into four categories, reverted each separately, and evaluated the same step-30 policy (Table R4). Most of the gain comes from the verification gates; the other categories contribute 0.6–1.6 points each, and effects overlap because the prompt text describes the gates.

Table R4: Leave-one-category-out evaluation of harness edits (110 tasks, avg@3, strict resolution, step-30 policy).

Harness variantResolvedΔ vs. final
Final evolved harness25.8–
− verification gates22.4−3.4
− submission control24.5−1.3
− prompt text24.2−1.6
− tool-call normalization25.2−0.6
Seed harness (all reverted)21.5−4.3

The gates run no tests: one checks the working-tree diff and blocks empty patches or edits to test files; the other runs code examples extracted from the issue text on the original and patched code. The agent may run the repository's existing tests on its own, but the hidden FAIL_TO_PASS tests are never visible to it; they are applied only at grading time in a fresh container (Appendix B). The dominant edit is therefore a control-flow constraint in the episode loop, not a change to what is rewarded or to what the agent can observe.

Q3: The proposer's history trajectories come from outdated policy weights; the off-policy / stale-archive problem should be discussed.

A3: Correct for one of the two proposer inputs. Traces are recent rollouts of the parent harness in the current run (the latest rollout per task), so they mostly reflect the current policy, though traces from a parent's first steps in the pool can lag. Archive scores are historical by design. The archive is context for the proposer only; it never enters the advantage, the scheduler, admission, or eviction, so staleness cannot bias the policy gradient. Empirically, removing it (A2) costs 6.4 points under the three-attempt protocol (Table R1), so stale context is still more useful than none.

Q4: Can the authors rule out train/eval overlap, including shared git repos with SWE-rebench and SWE-bench Verified? What data trains the TB-2 agent?

A4: The SWE-rebench split is both time-disjoint and repository-disjoint. The 400 training tasks were created between April 2016 and April 2025; the 110 evaluation tasks between March and May 2026. They share no repository (303 training vs. 79 evaluation repositories), so no evaluation task or repository is seen in training. The gap is eleven months rather than the three implied by the submission's "before 2026 / after March 2026" wording, which we will correct in Section 4.1. Against SWE-bench Verified, exactly one repository overlaps (sympy/sympy, 2 training tasks, neither a Verified instance); re-scoring Table 2 without the sympy instances leaves the ordering of all four methods unchanged. For TB-2, the agent is trained on 843 tasks from the synthetic Endless Terminals set (2,492 tasks), filtered following Shepherd's protocol by dropping tasks the base model solves in all 8 attempts; none of the 89 TB-2 evaluation instructions shares an 8-word span with any training prompt. The data source and this overlap check will be stated in Appendix C.1.

Q5: Provide a detailed per-component cost analysis, compared with Search-then-RL, single-harness alternation, and fixed harness.

A5: Table R5 gives per-run accounting for the four 30-step runs in Table 4 (8 GPUs each, same per-step rollout budget; A4's 1.9 h search phase runs before RL and is included).

Table R5: Training cost of the 30-step runs in Table 4.

RunTrain wall-clock (h)GPU·hProposer calls / acceptedProposer wall-clock (min)
CoHARL (full)47.23788 / 657
A1: Fixed-harness RL23.819000
A4: Search-then-RL22.11778 / 116
A5: Single-harness alternating RL25.32027 / 321

Three points. (1) Table 4 matches training steps and the per-step rollout budget that feeds policy updates; total compute is not matched. (2) CoHARL's roughly two-fold wall-clock is the cost of maintaining a population, not of the proposer: episodes per step are the same, but several harness versions stay active at once, each collecting trial and scheduled rollouts under its own scaffolding. Proposer inference is minor (57 min, about 2%) and independent of pool size. (3) Compute alone does not explain the gain: Plain GRPO under the fixed seed harness for 150 steps, about five times A1's budget, reaches 20.3 (Table 1), still below CoHARL's 25.8 after 30 steps.

We hope these additions resolve your concerns and would be glad to clarify anything further.

Dear Reviewer abMh,

We sincerely appreciate your careful review and valuable comments. With the discussion period ending soon, we wanted to follow up and ask whether our rebuttal has resolved the issues you raised. Please let us know if there is anything else we can clarify. Thank you for your time and consideration.

Reviewer_uC3pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies co-evolution of LLM agent policies and executable harnesses. Existing methods either optimize harness for a fixed policy or train policy under a static harness, failing to adapt the harness while policy learns. The authors introduce CoHARL, a population-based reinforcement learning framework that jointly evolves a single shared policy alongside a population of harnesses. The policy is trained on rollouts sampled across this harness population, while a proposer revises harness code during training using execution traces. Empirically, CoHARL achieves a mean resolved rate of 27.6% on SWE-rebench, outperforming harness-only search by 6.4 percentage points. Gains are also demonstrated on SWE-bench Verified and Terminal-Bench 2.0 for both software engineering and terminal control tasks. Ablations under matched training budgets confirm that both the Pareto archive and runtime execution traces are critical context for the proposer; removing either component substantially degrades final performance. Alternate training paradigms such as search-then-RL and single-harness alternating RL also underperform the full co-evolution setup.

Critical strengths

The paper proposes a promising evolutionary framework that jointly optimizes model parameters and a population of executable harnesses. The dedicated harness proposal and validation mechanisms enable iterative improvements to the execution environment throughout reinforcement learning.

Critical weaknesses

Limited experimental validation. Although the reported results are promising, the experimental evidence remains insufficient to fully establish the generalizability and robustness of the proposed approach. Evaluations across additional models, training configurations, and datasets would strengthen the paper's conclusions.

Questions for authors
  1. Effectiveness with stronger harnesses. it seems that most of the benefits come from the improvement of harness. Would the proposed approach still provide substantial improvements when initialized with stronger, more mature agent harnesses, such as those used by Codex?

  2. Model diversity and LoRA-based training. The experiments use only one model with LoRA-based reinforcement learning. Since LoRA may behave differently from full-parameter training, it remains unclear whether the observed improvements generalize across model architectures and training configurations. Could the authors evaluate additional models with different sizes and capabilities, or compare LoRA with full-parameter RL?

  3. Evaluation diversity and ablation reliability. Although the paper includes cross-dataset evaluation on SWE-bench Verified, additional coding benchmarks and out-of-distribution settings would provide stronger evidence of generalization. Furthermore, the ablation experiments involve only 110 tasks with a single attempt per task, making some of the reported improvements less convincing. Could the authors provide larger-scale evaluations to better support these conclusions?

2 comments

Thank you for the thorough summary and the concrete questions. We address each one below and would appreciate hearing whether our response resolves your concerns.

Q1: Most of the benefit seems to come from the harness; would gains persist from a stronger, mature harness such as Codex's?

A1: The benefit is not predominantly a harness effect. Table R2 evaluates each policy under each harness on the same 110 tasks: the harness alone gives +4.5 under the initial policy, the trained weights alone give +4.2 under the seed harness, and the two are approximately additive. The trained policy accounts for about half of the gain under either interface.

Table R2: Policy × harness decomposition (110 tasks, avg@3, strict resolution; step-30 policy and final-pool harness from the Table 4 protocol).

Policy \ HarnessSeed (SWE-mini)CoHARL harness
Initial Qwen3.5-35B-A3B17.321.8
CoHARL-trained policy21.525.8

On stronger seeds, the Terminal-Bench 2.0 experiment already starts from one: Terminus-2, the reference harness shipped with Harbor, is a production-quality agent, and from that seed CoHARL still gains 10.2 points over the initial agent and 3.1 over harness-only search (Table 3). The gain comes from adapting the harness to the policy as it changes during RL, not from a weak seed: a mature harness is tuned for a frozen frontier model, not for a 3B-active policy midway through training. We expect the absolute gain to shrink as the seed approaches the optimum for a given policy.

Q2: Only one model, LoRA only; evaluate other models or compare LoRA with full-parameter RL.

A2: All compared methods, including Plain GRPO and every Table 4 control, use the same policy, the same rank-32 LoRA, and the same GRPO configuration, so the reported gaps measure the harness population and proposer rather than the adaptation method. Nothing in the method is LoRA-specific: the proposer, scheduler, and archive act on rollouts and harness code, and the policy update is standard GRPO, so the procedure applies unchanged to full-parameter training. We were not able to run a second model in the discussion window (47 h per 30-step run on 8 GPUs, plus the fixed-harness and harness-only controls for a matched comparison); it is now the first item of future work.

Q3: Ablations use 110 tasks with one attempt; more benchmarks and OOD settings would help.

A3: We re-evaluated all six Table 4 checkpoints with three attempts per task and report 95% bootstrap intervals (Table R1). The ordering is unchanged and every control stays below the full method.

Table R1: Table 4 re-evaluated with three attempts per task (avg@3, temperature 0.7, strict resolution; 95% bootstrap CI over 110 tasks, 10,000 resamples).

ConfigurationTable 4 (1 attempt)avg@395% CI
CoHARL (full)25.525.8[17.9, 33.9]
A1: Fixed-harness RL20.020.3[14.2, 27.0]
A2: No archive context20.019.4[13.6, 25.8]
A3: No execution traces20.920.6[14.5, 27.3]
A4: Search-then-RL22.723.3[16.4, 30.6]
A5: Single-harness alternating RL19.120.0[13.9, 26.4]

On benchmarks, the paper already covers three across two domains: SWE-rebench, SWE-bench Verified as cross-dataset transfer without further adaptation, and Terminal-Bench 2.0. Within the discussion period we prioritized the decomposition (Table R2) and the re-evaluation (Table R1) over adding a fourth benchmark; Table R1 will replace the single-attempt Table 4 in the revision.

We would appreciate it if you could let us know whether your concerns are addressed by our response.

Dear Reviewer uC3p,

Thank you again for your thoughtful review and constructive feedback. As the discussion period is drawing to a close, we would like to kindly check whether our responses and additional experiments have addressed your concerns. If any questions remain, we would be happy to provide further clarification. We greatly appreciate your time and effort.

Reviewer_x6PrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LLM agents have two parts: the policy (model weights) and the harness (the code around it). The paper proposes a method that trains these parts together. In particular, one policy learns from rollouts generated by using a population of harnesses, with a separate proposer LLM rewriting the harnesses based on the policy's historical traces. The claimed contributions are a formulation of the co-evolution of a policy and a population of harnesses, a search-and-training procedure that proposes new harnesses, and evaluations on three benchmarks.

Critical strengths

The paper shows that CoHARL beats the harness-only and policy-only baselines on all three benchmarks, which supports the paper's claim that adapting policy and harness together beats adapting either alone. The authors show that the method works on both swebench and TB-2, which means that the method is general and not limited to a particular domain. The evaluations are carefully designed with matched-budget controls and temporal data splits.

Critical weaknesses

Since the unique contribution is not joint adaptation, then the paper's novelty should be focused on the idea that using a population of harnesses is helpful. However, the only direct test of that is Table 4 (a single run with no confidence intervals). Since RL training is known to have high variance, a 7-task gap may not be significant. The paper could be strengthened by making the novelty a bit more clear or adding more evaluations comparing to the necessary baselines (e.g., A5).

Another concern is that the paper does not directly compare to strong baselines / the closest competing methods that also do co-evolution (e.g., WHALE, HarnessForge).

Questions for authors

Which harness is evaluated for each method in Table 4?

3 comments

Thank you for your valuable feedback to help us improve our paper. We detail our response below and please kindly let us know if our response addresses your concerns.

Q1: Which harness is evaluated for each method in Table 4?

A1: Every row evaluates that run's own step-30 policy checkpoint; what differs is the harness it is paired with. We apologize that Appendix C.1 stated this only for A4 and A5; the mapping below is now a note under Table 4.

RowHarness at evaluation
CoHARL (full), A2, A3highest-ranked member of that run's final pool (Eq. 5 score)
A1: Fixed-harness RLSWE-mini (seed), unchanged
A4: Search-then-RLthe harness selected before RL on 32 validation tasks, then frozen
A5: Single-harness alternating RLits only pool member at step 30 (A5 keeps one active harness)

Q2: Novelty should center on the population of harnesses, but its only direct test is Table 4, a single run with no confidence intervals; a 7-task gap may be within RL variance.

A2: We agree that the population design is the core novelty and state it as such in the revision. We also re-evaluated all six Table 4 checkpoints with three attempts per task and report 95% bootstrap intervals (Table R1). The ordering is unchanged and the gap to A5 is 5.8 points.

Table R1: Table 4 re-evaluated with three attempts per task (avg@3, temperature 0.7, strict resolution; 95% bootstrap CI over 110 tasks, 10,000 resamples).

ConfigurationTable 4 (1 attempt)avg@395% CI
CoHARL (full)25.525.8[17.9, 33.9]
A1: Fixed-harness RL20.020.3[14.2, 27.0]
A2: No archive context20.019.4[13.6, 25.8]
A3: No execution traces20.920.6[14.5, 27.3]
A4: Search-then-RL22.723.3[16.4, 30.6]
A5: Single-harness alternating RL19.120.0[13.9, 26.4]

Q3: No direct comparison to WHALE or HarnessForge.

A3: Both have public code, but neither fits our setting: WHALE (verl; search QA, math, chess) and HarnessForge (LLaMA-Factory; tool-use QA) both train 2B–8B models by SFT/RSFT, so running them on SWE-rebench with a 35B MoE policy and GRPO would require porting their training and proposer pipelines, plus one to three days per 30-step run, which does not fit the discussion period. Our controls isolate the axis on which each differs from us: A4 is search-then-train; A5 is WHALE's alternation of harness search and weight updates under one selected harness, with GRPO in place of RSFT so only the single-versus-population axis varies. HarnessForge's per-harness adapters have no matched control and are listed as future work. Appendix F states these correspondences and that they are local controls, not reproductions.

We would appreciate it if you could let us know whether your concerns are addressed by our response.

Thank you for your response. My concerns have been addressed and I have raised my score.

Dear Reviewer x6Pr,

Thank you for taking the time to review our responses and for raising your rating. We are glad that our clarifications and the additional re-evaluation have addressed your concerns. Thank you again for your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.