- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Self-HPO, a framework that alternates harness optimization and policy optimization for a coding agent. In each H-step, the current policy analyzes its failed trajectories on a search set and proposes six harness candidates. The harness with the highest resolve rate on the search set is retained. In each P-step, the policy is trained with GRPO and Balanced Adaptive Rollout (BAR) for 50 updates under the selected harness. Tasks resolved at most once across their rollouts in that P-step form the next search set. Starting from Qwen3.5-35B-A3B and the mini-SWE-agent seed harness, two rounds improve SWE-Bench Verified from 61.0% to 69.5%, SWE-Bench Multilingual from 54.2% to 59.7%, and SWE-Bench Pro from 58.3% to 65.2%, exceeding policy-only RLVR and harness-only Self-Harness. Using Opus 4.8 as the harness proposer raises SWE-Bench Verified to 71.6%. Further analyses cross-evaluate harnesses and policies, report overall pass@k and pass@1 by task difficulty, and track zero-success GRPO groups during training.
Critical strengths
-
Consistent gains, including out of domain. Self-HPO and HPO outperform their respective policy-only and harness-only baselines on all three benchmarks. On SWE-Bench Multilingual and SWE-Bench Pro, which are not used for training, Self-HPO improves over RLVR by 3.4 percentage points on each. RLVR uses the same RL configuration, while Self-Harness and Meta-Harness receive the same harness search budget.
-
Both the harness and policy retain improvements. In Table 3, H2 improves the base policy P0 from 61.0% to 64.5%, while P2 reaches 64.4% under the seed harness H0. In the SWE-rebench training analysis, zero-success GRPO groups fall to 6.5% over the final ten updates of Self-HPO's second P-step, compared with mostly 11–17% over the last 50 RLVR updates. The paper also provides the optimized HPO harness code, and its Self-HPO case study shows the harness recovering a submission containing prose instead of a patch.
Critical weaknesses
-
Alternation is not compared with sequential optimization. The baselines optimize either the harness or the policy. No baseline first trains the policy and then optimizes the harness, or first optimizes the harness and then trains the policy. Such a comparison could use the same 100 RL updates and 12 harness candidates as Self-HPO. Table 3 combines harnesses and policies produced during the same Self-HPO run, so it does not answer this question. The RLVR policy is also not tested with an optimized harness. It therefore remains unclear whether alternating updates improve on optimizing both components in sequence.
-
Total optimization cost and variance across training seeds remain unclear. Self-HPO combines RL training with harness search. RLVR receives no harness search, and Self-Harness receives no RL training. The paper does not report total rollout counts or GPU-hours for each method, leaving unclear whether the baselines would improve with the same additional compute. The standard deviations in Table 1 come from three evaluation runs, and Table 2 reports no uncertainty. Repeated training runs are needed to assess how consistently Self-HPO improves over the baselines.
Questions for authors
-
I may have missed this, but have you compared Self-HPO with training the policy first and then optimizing the harness, or optimizing the harness first and then training the policy, using 100 RL updates and 12 harness candidates in each case?
-
Do the Self-Harness and Meta-Harness proposers receive the same reference fixes for search-set tasks as the Self-HPO proposer?
3 comments
Response to Reviewer veFA [Part 2/2]
[Q2]: Total optimization cost and variance across training seeds remain unclear. Self-HPO combines RL training with harness search. RLVR receives no harness search, and Self-Harness receives no RL training. The paper does not report total rollout counts or GPU-hours for each method, leaving unclear whether the baselines would improve with the same additional compute. Repeated training runs are needed to assess how consistently Self-HPO improves over the baselines.
[A2]: We thank the reviewer for raising this question. Below, we report the total optimization budget of each method in terms of generated rollouts in both RL training and harness search.
| Method | Optimization | RL rollouts | Harness search rollouts | Total | Avg. Performance |
|---|---|---|---|---|---|
| Base | N/A | 0 | 0 | 0 | 57.8 |
| Self-Harness | harness-only | 0 | 4k | 4k | 60.1 |
| RLVR | policy-only | 48k | 0 | 48k | 61.9 |
| Self-HPO | joint | 48k | 4k | 52k | 64.8 |
To compare these methods under the same compute, we further scale both Self-Harness and RLVR to a total budget of 52k rollouts, in line with Self-HPO.
| Method | Budget | SWE-Verified | SWE-Multi | SWE-Pro | Avg. |
|---|---|---|---|---|---|
| Base | 0 | 61.0 | 54.2 | 58.3 | 57.8 |
| Self-Harness | 4k | 65.2 | 54.3 | 60.7 | 60.1 |
| Self-Harness | 52k | 62.8 | 52.7 | 60.3 | 58.6 |
| RLVR (100 steps) | 48k | 67.5 | 56.3 | 61.8 | 61.9 |
| RLVR (125 steps) | 52k | 64.7 | 51.4 | 57.3 | 57.8 |
| Self-HPO | 52k | 69.5 | 59.7 | 65.2 | 64.8 |
Under the same budget, both baselines still underperform Self-HPO for different reasons.
- Scaling up harness search alone does not help Self-Harness. With the policy fixed, increasing the search budget from 4k to 52k rollouts does not yield further gains on the evaluation benchmarks. Our analysis shows that Self-Harness with more compute can overfit the fixed search set: it evolves the harness toward more targeted mechanisms for the search tasks, which however fail to generalize to the evaluation tasks and lead to degradation compared to the 4k counterpart.
- Spending more compute on RL updates also degrades RLVR. Increasing RL training from 100 to 125 steps (48k to 52k rollouts) even lowers the average performance of RLVR. This is consistent with the known instability of RL training for agentic tasks [1, 2, 3], where training longer does not always keep improving the policy. Instead, as RLVR training continues, the policy entropy can rise sharply and the model can collapse. We have added this analysis in Appendix G (Figure 13) of the updated PDF.
While we fully acknowledge the importance of reporting the variance across training runs, we would like to clarify that we choose to run the training once for two practical reasons:
- Agentic RL training for long-horizon tasks such as coding is particularly expensive. Each training rollout involves up to hundreds of interactions with the environment, which require heavy GPU generation, let alone the time for policy updates (forward and backward passes) and command execution in the sandbox. For example, in our setup, a full Self-HPO run takes about 1,350 GPU-hours on 8×B200 GPUs.
- Running training once is common practice in this field. Due to this cost, relevant works in agentic RL typically report a single training run per method [4, 5, 6, 7], with multiple evaluation runs to reduce variance. We follow this practice and report the mean and standard deviation over three evaluation runs.
[Q3]: Do the Self-Harness and Meta-Harness proposers receive the same reference fixes for search-set tasks as the Self-HPO proposer?
[A3]: Yes, both Self-Harness and Meta-Harness share the same harness-search pipeline as Self-HPO, where the proposer reads the trajectories on the search set together with their reference fixes, and a new harness is kept only if it improves the search-set score. The only differences are:
- (1) Self-Harness searches on a fixed search set, while Self-HPO uses an evolving search set built from the preceding P-step. The initial search set of Self-HPO is the same as that of Self-Harness.
- (2) Meta-Harness further uses a stronger external proposer (Claude Opus 4.8) to perform harness optimization on the fixed search set.
We have updated the implementation details in Appendix A of the updated PDF to make this clearer.
Please let us know if you have any further questions, and we are happy to incorporate additional suggestions you might have. Thanks again for your time and efforts!
References
References
[1]. Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. COLM 2025
[2]. Xue et al. SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning. ICLR 2026
[3]. Wang et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073
[4]. Wei et al. SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution. NeurIPS 2025
[5]. Yang et al. Kimi-Dev: Agentless Training as Skill Prior for SWE-Agents. ICLR 2026
[6]. Luo et al. DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL. Blog, 2025
[7]. Ivison et al. TMax: A Simple Recipe for Terminal Agents. arXiv:2606.23321
Response to Reviewer veFA [Part 1/2]
We sincerely thank the reviewer for recognizing our work, as well as for the constructive comments. To incorporate the reviewer's suggestions, we have uploaded an updated PDF, where all revisions are highlighted in blue. Below, we address each of the reviewer's questions point by point.
[Q1]: Alternation is not compared with sequential optimization. It therefore remains unclear whether alternating updates improve on optimizing both components in sequence.
[A1]: As suggested, we compare Self-HPO with two sequential baselines under the same budget of 100 RL updates and 12 rounds of harness search:
It is clear that neither sequential baseline matches Self-HPO. In a sequential schedule, once one component is optimized, it stays fixed while the other is optimized, which makes the sequential optimization prone to getting stuck in a local optimum. In contrast, Self-HPO alternates between the two, so that each harness search adapts to the evolving failures of the current policy and each policy update adapts to the current harness. This allows both components to keep evolving and makes it easier to escape such local optima. We have added this comparison as an ablation study in Section 3.3 (Table 3) of the updated PDF.