- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CoHARL which brings policy update and harness evolution into one integrated RL. The proposer LLM mutates multiple pieces of harness using the parent source, current policy trajectories information, and historical accuracy-token-cost for harness candidate proposal. The accepted candidates enter the training pool with a scheduler for rollout assignment in the next round. Then, a regular rollout & policy update (GRPO-like) is performed. Experiments on SWE-rebench, SWE-verified, terminal-bench show that the gains are observed over baselines that only target policy or harness improvement.
Critical strengths
- The problem itself is clear and fascinating. The integration of the two variables (policy, harness) is important.
- The mutation/modification proposal makes sense.
- The illustrations and algorithm is clear and easy to follow.
Critical weaknesses
- Some detailed information are missing. For example, the staging relies on domain-specific candidate checks (e.g., contract to pool trials for SWE), but the reviewer did not know whether the execution success satisfies or it must have a threshold (pass/fail resolve rate). Are weak harness variations considered in the pool for staging?
- The contribution of each aspect of the harness improvement is not clearly quantified or discussed deeply. For example, the evolved version might change test score/process reward and then GRPO optimizes policy under such a moving interface. The performance gains might come from both interface bug fixes and reward shaping?
- There might exist an off-policy nature for the proposal where the history trajectory information rollouted with old/outdated policy weight (correct me if I misunderstand). If so, the off-policy problem (stale archive) should be considered and discussed.
Questions for authors
- Please detail the conditions for each task (e.g., SWE/terminal) on whether a candidate harness can pass pool trials. In addition, please give the computation budget for the pool trials as well.
- Can the authors use simple but clear explanations to clarify whether the evovled harness might affect reward std filtering/resolve rate if the process reward is involved into such proposal?
- One concern about the training and evaluation setting is the benchmark used in experiments. The training/validation is split by date (before or after 2026), but can the authors claim there exist no overlapping between the tasks in training and validation? Especially for the SWE-rebench and SWE-verified datasets, the potential leakage by sharing git repos across tasks should be handled. In addition, the paper does not clearly indicate the dataset used to train terminal agent for validation on TB2.
- In Sec.4.1, the computational complexity is only briefly mentioned in appendix C.1. Can the authors provide more detailed analysis on the complexity of each part during training, including the comparison with baselines (e.g., coHARL/Search-then-RL/Single-harness alternative/fixed).
5 comments
A2 (continued): which edits matter. We grouped every edit between the seed and the final harness into four categories, reverted each separately, and evaluated the same step-30 policy (Table R4). Most of the gain comes from the verification gates; the other categories contribute 0.6–1.6 points each, and effects overlap because the prompt text describes the gates.
Table R4: Leave-one-category-out evaluation of harness edits (110 tasks, avg@3, strict resolution, step-30 policy).
| Harness variant | Resolved | Δ vs. final |
|---|---|---|
| Final evolved harness | 25.8 | – |
| − verification gates | 22.4 | −3.4 |
| − submission control | 24.5 | −1.3 |
| − prompt text | 24.2 | −1.6 |
| − tool-call normalization | 25.2 | −0.6 |
| Seed harness (all reverted) | 21.5 | −4.3 |
The gates run no tests: one checks the working-tree diff and blocks empty patches or edits to test files; the other runs code examples extracted from the issue text on the original and patched code. The agent may run the repository's existing tests on its own, but the hidden FAIL_TO_PASS tests are never visible to it; they are applied only at grading time in a fresh container (Appendix B). The dominant edit is therefore a control-flow constraint in the episode loop, not a change to what is rewarded or to what the agent can observe.
Q3: The proposer's history trajectories come from outdated policy weights; the off-policy / stale-archive problem should be discussed.
A3: Correct for one of the two proposer inputs. Traces are recent rollouts of the parent harness in the current run (the latest rollout per task), so they mostly reflect the current policy, though traces from a parent's first steps in the pool can lag. Archive scores are historical by design. The archive is context for the proposer only; it never enters the advantage, the scheduler, admission, or eviction, so staleness cannot bias the policy gradient. Empirically, removing it (A2) costs 6.4 points under the three-attempt protocol (Table R1), so stale context is still more useful than none.
Q4: Can the authors rule out train/eval overlap, including shared git repos with SWE-rebench and SWE-bench Verified? What data trains the TB-2 agent?
A4: The SWE-rebench split is both time-disjoint and repository-disjoint. The 400 training tasks were created between April 2016 and April 2025; the 110 evaluation tasks between March and May 2026. They share no repository (303 training vs. 79 evaluation repositories), so no evaluation task or repository is seen in training. The gap is eleven months rather than the three implied by the submission's "before 2026 / after March 2026" wording, which we will correct in Section 4.1. Against SWE-bench Verified, exactly one repository overlaps (sympy/sympy, 2 training tasks, neither a Verified instance); re-scoring Table 2 without the sympy instances leaves the ordering of all four methods unchanged. For TB-2, the agent is trained on 843 tasks from the synthetic Endless Terminals set (2,492 tasks), filtered following Shepherd's protocol by dropping tasks the base model solves in all 8 attempts; none of the 89 TB-2 evaluation instructions shares an 8-word span with any training prompt. The data source and this overlap check will be stated in Appendix C.1.
Q5: Provide a detailed per-component cost analysis, compared with Search-then-RL, single-harness alternation, and fixed harness.
A5: Table R5 gives per-run accounting for the four 30-step runs in Table 4 (8 GPUs each, same per-step rollout budget; A4's 1.9 h search phase runs before RL and is included).
Table R5: Training cost of the 30-step runs in Table 4.
| Run | Train wall-clock (h) | GPU·h | Proposer calls / accepted | Proposer wall-clock (min) |
|---|---|---|---|---|
| CoHARL (full) | 47.2 | 378 | 8 / 6 | 57 |
| A1: Fixed-harness RL | 23.8 | 190 | 0 | 0 |
| A4: Search-then-RL | 22.1 | 177 | 8 / 1 | 16 |
| A5: Single-harness alternating RL | 25.3 | 202 | 7 / 3 | 21 |
Three points. (1) Table 4 matches training steps and the per-step rollout budget that feeds policy updates; total compute is not matched. (2) CoHARL's roughly two-fold wall-clock is the cost of maintaining a population, not of the proposer: episodes per step are the same, but several harness versions stay active at once, each collecting trial and scheduled rollouts under its own scaffolding. Proposer inference is minor (57 min, about 2%) and independent of pool size. (3) Compute alone does not explain the gain: Plain GRPO under the fixed seed harness for 150 steps, about five times A1's budget, reaches 20.3 (Table 1), still below CoHARL's 25.8 after 30 steps.
We hope these additions resolve your concerns and would be glad to clarify anything further.
Dear Reviewer abMh,
We sincerely appreciate your careful review and valuable comments. With the discussion period ending soon, we wanted to follow up and ask whether our rebuttal has resolved the issues you raised. Please let us know if there is anything else we can clarify. Thank you for your time and consideration.
Thank you for the detailed review. We answer each point below, including the protocol details the submission left under-specified, and would welcome any further questions.
Q1: Candidate admission: is there a resolve-rate threshold, are weak harnesses admitted, what are the SWE/TB-2 conditions, and what is the pool-trial budget?
A1: There is no accuracy threshold at admission: candidates are judged by their training rollouts, so weak harnesses are admitted and then evicted by the scheduler.
These details will be added to Appendix C.1.
Q2: Harness edits may change test score / process reward, so GRPO optimizes under a moving interface; are gains from interface fixes or reward shaping, and do evolved harnesses affect reward-std filtering?
A2: Harness evolution did not change the training reward. The reward formula (Eq. 7) and the length penalty (Eq. 8) live in the training framework, which the proposer cannot edit, so they are identical across all harness versions. The only reward-related field a harness can touch is
test_score, used solely for zero-variance group filtering: the seed stores the test pass fraction and some evolved versions store the harness reward (Appendix C.1), which changes which groups are retained but not the advantages within them. Evaluation uses an external scorer that reads only the final patch and runs the official tests, so no harness edit can affect a reported resolved rate. We audited all 6 revisions accepted in the 30-step full run: the reward code is byte-identical to the seed; edits are confined to prompting, action parsing, and a pre-submission diff check.To test this empirically we re-ran the 30-step protocol with the reward and
test_scorecode under the same edit ban as the training framework (Table R3). With the training signal held fixed, CoHARL retains 3.9 of the 5.5 points it gains over fixed-harness RL.Table R3: Reward-locked CoHARL (30 steps, 110 tasks, avg@3, strict resolution).