acceptodds
Under review as a conference paper at ICLR 2027

Self-HPO: Self-Evolving Coding Agents via Harness-Policy Optimization

Abstract

While large language model (LLM) agents have shown promising potential in software engineering, their performance is jointly determined by the model weights and the harness that mediates their interaction with the environment. The two are mutually reinforcing: a better harness exposes and compensates for the weaknesses of the current weights, while stronger weights allow the harness to elicit more capable behavior. Yet existing methods optimize one component with the other fixed, so a harness tuned for one checkpoint may no longer suit an updated policy, while policy training remains limited to the trajectories that its fixed harness elicits. In this work, we propose Self-HPO, a recursively self-improving framework in which the harness and the policy co-evolve through alternating optimization. Specifically, in each round, the current policy first analyzes its own failures and proposes harness revisions (H-step); reinforcement learning (RL) then updates the policy under the selected harness (P-step), and hard tasks encountered during training guide the next H-step, where the updated policy serves as the proposer. Experiments show that Self-HPO consistently outperforms harness-only and policy-only optimization, improving Qwen3.5-35B-A3B from 61.0% to 69.5% on SWE-Bench Verified, with gains extending to out-of-domain benchmarks (e.g., 54.2% → 59.7% on SWE-Bench Multilingual). Extensive analysis indicates that the gains are retained in both components and are complementary, and that fewer training prompts fail on every rollout during co-evolution than under RL with a fixed harness. Finally, we show that with a stronger external model as the harness proposer, the same framework yields a further performance boost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Agents, Harness, Self-improvement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_veFAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Self-HPO, a framework that alternates harness optimization and policy optimization for a coding agent. In each H-step, the current policy analyzes its failed trajectories on a search set and proposes six harness candidates. The harness with the highest resolve rate on the search set is retained. In each P-step, the policy is trained with GRPO and Balanced Adaptive Rollout (BAR) for 50 updates under the selected harness. Tasks resolved at most once across their rollouts in that P-step form the next search set. Starting from Qwen3.5-35B-A3B and the mini-SWE-agent seed harness, two rounds improve SWE-Bench Verified from 61.0% to 69.5%, SWE-Bench Multilingual from 54.2% to 59.7%, and SWE-Bench Pro from 58.3% to 65.2%, exceeding policy-only RLVR and harness-only Self-Harness. Using Opus 4.8 as the harness proposer raises SWE-Bench Verified to 71.6%. Further analyses cross-evaluate harnesses and policies, report overall pass@k and pass@1 by task difficulty, and track zero-success GRPO groups during training.

Critical strengths
  1. Consistent gains, including out of domain. Self-HPO and HPO outperform their respective policy-only and harness-only baselines on all three benchmarks. On SWE-Bench Multilingual and SWE-Bench Pro, which are not used for training, Self-HPO improves over RLVR by 3.4 percentage points on each. RLVR uses the same RL configuration, while Self-Harness and Meta-Harness receive the same harness search budget.

  2. Both the harness and policy retain improvements. In Table 3, H2 improves the base policy P0 from 61.0% to 64.5%, while P2 reaches 64.4% under the seed harness H0. In the SWE-rebench training analysis, zero-success GRPO groups fall to 6.5% over the final ten updates of Self-HPO's second P-step, compared with mostly 11–17% over the last 50 RLVR updates. The paper also provides the optimized HPO harness code, and its Self-HPO case study shows the harness recovering a submission containing prose instead of a patch.

Critical weaknesses
  1. Alternation is not compared with sequential optimization. The baselines optimize either the harness or the policy. No baseline first trains the policy and then optimizes the harness, or first optimizes the harness and then trains the policy. Such a comparison could use the same 100 RL updates and 12 harness candidates as Self-HPO. Table 3 combines harnesses and policies produced during the same Self-HPO run, so it does not answer this question. The RLVR policy is also not tested with an optimized harness. It therefore remains unclear whether alternating updates improve on optimizing both components in sequence.

  2. Total optimization cost and variance across training seeds remain unclear. Self-HPO combines RL training with harness search. RLVR receives no harness search, and Self-Harness receives no RL training. The paper does not report total rollout counts or GPU-hours for each method, leaving unclear whether the baselines would improve with the same additional compute. The standard deviations in Table 1 come from three evaluation runs, and Table 2 reports no uncertainty. Repeated training runs are needed to assess how consistently Self-HPO improves over the baselines.

Questions for authors
  1. I may have missed this, but have you compared Self-HPO with training the policy first and then optimizing the harness, or optimizing the harness first and then training the policy, using 100 RL updates and 12 harness candidates in each case?

  2. Do the Self-Harness and Meta-Harness proposers receive the same reference fixes for search-set tasks as the Self-HPO proposer?

3 comments
AuthorsOpenReview ↗

Response to Reviewer veFA [Part 1/2]

We sincerely thank the reviewer for recognizing our work, as well as for the constructive comments. To incorporate the reviewer's suggestions, we have uploaded an updated PDF, where all revisions are highlighted in blue. Below, we address each of the reviewer's questions point by point.

[Q1]: Alternation is not compared with sequential optimization. It therefore remains unclear whether alternating updates improve on optimizing both components in sequence.

[A1]: As suggested, we compare Self-HPO with two sequential baselines under the same budget of 100 RL updates and 12 rounds of harness search:

  • (1) Harness-then-Policy, which first runs 12 rounds of harness search with the base policy as the proposer and then trains the policy for 100 RL steps under the fixed optimized harness.
  • (2) Policy-then-Harness, which first trains the policy for 100 RL steps under the seed harness and then runs 12 rounds of harness search with the trained policy as the proposer. This baseline also evaluates the RLVR policy with a harness optimized for it.
MethodOptimizationSearch budgetRL updatesSWE-VerifiedSWE-MultiSWE-ProAvg.
BaseN/A0061.054.258.357.8
Self-Harnessharness-only12 rounds065.254.360.760.1
RLVRpolicy-only0100 steps67.556.361.861.9
Policy-then-Harnesssequential12 rounds100 steps66.858.761.162.2
Harness-then-Policysequential12 rounds100 steps65.958.564.362.9
Self-HPOalternating12 rounds100 steps69.559.765.264.8

It is clear that neither sequential baseline matches Self-HPO. In a sequential schedule, once one component is optimized, it stays fixed while the other is optimized, which makes the sequential optimization prone to getting stuck in a local optimum. In contrast, Self-HPO alternates between the two, so that each harness search adapts to the evolving failures of the current policy and each policy update adapts to the current harness. This allows both components to keep evolving and makes it easier to escape such local optima. We have added this comparison as an ablation study in Section 3.3 (Table 3) of the updated PDF.

AuthorsOpenReview ↗

Response to Reviewer veFA [Part 2/2]

[Q2]: Total optimization cost and variance across training seeds remain unclear. Self-HPO combines RL training with harness search. RLVR receives no harness search, and Self-Harness receives no RL training. The paper does not report total rollout counts or GPU-hours for each method, leaving unclear whether the baselines would improve with the same additional compute. Repeated training runs are needed to assess how consistently Self-HPO improves over the baselines.

[A2]: We thank the reviewer for raising this question. Below, we report the total optimization budget of each method in terms of generated rollouts in both RL training and harness search.

MethodOptimizationRL rolloutsHarness search rolloutsTotalAvg. Performance
BaseN/A00057.8
Self-Harnessharness-only04k4k60.1
RLVRpolicy-only48k048k61.9
Self-HPOjoint48k4k52k64.8

To compare these methods under the same compute, we further scale both Self-Harness and RLVR to a total budget of 52k rollouts, in line with Self-HPO.

MethodBudgetSWE-VerifiedSWE-MultiSWE-ProAvg.
Base061.054.258.357.8
Self-Harness4k65.254.360.760.1
Self-Harness52k62.852.760.358.6
RLVR (100 steps)48k67.556.361.861.9
RLVR (125 steps)52k64.751.457.357.8
Self-HPO52k69.559.765.264.8

Under the same budget, both baselines still underperform Self-HPO for different reasons.

  • Scaling up harness search alone does not help Self-Harness. With the policy fixed, increasing the search budget from 4k to 52k rollouts does not yield further gains on the evaluation benchmarks. Our analysis shows that Self-Harness with more compute can overfit the fixed search set: it evolves the harness toward more targeted mechanisms for the search tasks, which however fail to generalize to the evaluation tasks and lead to degradation compared to the 4k counterpart.
  • Spending more compute on RL updates also degrades RLVR. Increasing RL training from 100 to 125 steps (48k to 52k rollouts) even lowers the average performance of RLVR. This is consistent with the known instability of RL training for agentic tasks [1, 2, 3], where training longer does not always keep improving the policy. Instead, as RLVR training continues, the policy entropy can rise sharply and the model can collapse. We have added this analysis in Appendix G (Figure 13) of the updated PDF.

While we fully acknowledge the importance of reporting the variance across training runs, we would like to clarify that we choose to run the training once for two practical reasons:

  • Agentic RL training for long-horizon tasks such as coding is particularly expensive. Each training rollout involves up to hundreds of interactions with the environment, which require heavy GPU generation, let alone the time for policy updates (forward and backward passes) and command execution in the sandbox. For example, in our setup, a full Self-HPO run takes about 1,350 GPU-hours on 8×B200 GPUs.
  • Running training once is common practice in this field. Due to this cost, relevant works in agentic RL typically report a single training run per method [4, 5, 6, 7], with multiple evaluation runs to reduce variance. We follow this practice and report the mean and standard deviation over three evaluation runs.

[Q3]: Do the Self-Harness and Meta-Harness proposers receive the same reference fixes for search-set tasks as the Self-HPO proposer?

[A3]: Yes, both Self-Harness and Meta-Harness share the same harness-search pipeline as Self-HPO, where the proposer reads the trajectories on the search set together with their reference fixes, and a new harness is kept only if it improves the search-set score. The only differences are:

  • (1) Self-Harness searches on a fixed search set, while Self-HPO uses an evolving search set built from the preceding P-step. The initial search set of Self-HPO is the same as that of Self-Harness.
  • (2) Meta-Harness further uses a stronger external proposer (Claude Opus 4.8) to perform harness optimization on the fixed search set.

We have updated the implementation details in Appendix A of the updated PDF to make this clearer.

Please let us know if you have any further questions, and we are happy to incorporate additional suggestions you might have. Thanks again for your time and efforts!

AuthorsOpenReview ↗

References

References

[1]. Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. COLM 2025

[2]. Xue et al. SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning. ICLR 2026

[3]. Wang et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073

[4]. Wei et al. SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution. NeurIPS 2025

[5]. Yang et al. Kimi-Dev: Agentless Training as Skill Prior for SWE-Agents. ICLR 2026

[6]. Luo et al. DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL. Blog, 2025

[7]. Ivison et al. TMax: A Simple Recipe for Terminal Agents. arXiv:2606.23321

Reviewer_XMPYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-HPO, a framework for improving coding agents by jointly optimizing the policy and the harness that governs how the agent interacts with its environment. The agent analyzes its failures to propose harness revisions, then updates its policy through reinforcement learning under the selected harness. The updated policy and difficult tasks encountered during training guide the next round of harness search. The central idea is to let the policy and harness adapt to each other through a repeated cycle of self-improvement.

Critical strengths
  1. The paper addresses a practical problem that an agent’s performance depends on both its policy and the harness that controls its interaction with the environment.
  2. Self-HPO improves over both policy-only and harness-only baselines on three software-engineering benchmarks, including two out-of-domain evaluations.
  3. The experimental study is substantial. The ablations, component cross-evaluations, multi-attempt results, and concrete harness examples provide useful evidence beyond the main performance table.
Critical weaknesses
  1. The experiments establish the value of combining harness search with policy training, but do not isolate the benefit of repeated co-adaptation. It does not show whether alternating harness search and policy training is more effective than optimizing the harness first and then training the policy. The authors should include this two-stage baseline, keeping the optimized harness fixed during policy training and matching the search budget and number of RL updates.
  2. Self-HPO uses additional harness-search rollouts beyond the RL budget used by RLVR. The paper acknowledges this in Section 5 but does not report the total optimization cost. The performance gains are therefore difficult to assess relative to the extra computation. The authors should report the search overhead and compare performance under a comparable total budget.
  3. The decrease in zero-success groups does not establish that more groups provide a GRPO learning signal. All-success groups also have zero reward variance and are included in the denominator. The authors should report the proportions of mixed-reward and all-success groups to support the proposed explanation.
Questions for authors

See critical weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer XMPY [Part 1/2]

We sincerely thank the reviewer for their insightful comments. To incorporate these suggestions, we have uploaded an updated PDF, where all revisions are highlighted in blue. Below, we address each of the reviewer's questions point by point.

[Q1]: The experiments does not show whether alternating harness search and policy training is more effective than optimizing the harness first and then training the policy. The authors should include this two-stage baseline, keeping the optimized harness fixed during policy training and matching the search budget and number of RL updates.

[A1]: As suggested, we compare Self-HPO with two sequential baselines under the same budget:

  • (1) Harness-then-Policy, which first runs 12 rounds of harness search with the base policy as the proposer and then trains the policy for 100 RL steps under the fixed optimized harness
  • (2) Policy-then-Harness, which first trains the policy for 100 RL steps under the seed harness and then runs 12 rounds of harness search with the trained policy as the proposer
MethodOptimizationSearch budgetRL updatesSWE-VerifiedSWE-MultiSWE-ProAvg.
BaseN/A0061.054.258.357.8
Self-Harnessharness-only12 rounds065.254.360.760.1
RLVRpolicy-only0100 steps67.556.361.861.9
Policy-then-Harnesssequential12 rounds100 steps66.858.761.162.2
Harness-then-Policysequential12 rounds100 steps65.958.564.362.9
Self-HPOalternating12 rounds100 steps69.559.765.264.8

It is clear that neither sequential baseline matches Self-HPO. In a sequential schedule, once one component is optimized, it stays fixed while the other is optimized, which makes the sequential optimization prone to getting stuck in a local optimum. In contrast, Self-HPO alternates between the two, so that each harness search adapts to the evolving failures of the current policy and each policy update adapts to the current harness. This allows both components to keep evolving and makes it easier to escape such local optima. We have added this comparison as an ablation study in Section 3.3 (Table 3) of the updated PDF.

[Q2]: Self-HPO uses additional harness-search rollouts beyond the RL budget used by RLVR. The authors should report the search overhead and compare performance under a comparable total budget.

[A2]: We thank the reviewer for this insightful suggestion. Below, we report the total budget of Self-HPO and RLVR in terms of generated rollouts in both RL training and harness search.

MethodOptimizationRL rolloutsHarness search rolloutsTotalAvg. Performance
BaseN/A00057.8
RLVRpolicy-only48k048k61.9
Self-HPOjoint48k4k52k64.8

Compared to RLVR, Self-HPO introduces only a small additional overhead from harness search, about 8% of its total budget, while achieving 2.9 points higher average performance, demonstrating the advantage of our method.

As suggested, to compare performance under a comparable budget, we further train RLVR to 125 steps, which matches the total budget of Self-HPO.

MethodBudgetSWE-VerifiedSWE-MultiSWE-ProAvg.
Base061.054.258.357.8
RLVR (100 steps)48k67.556.361.861.9
RLVR (125 steps)52k64.751.457.357.8
Self-HPO52k69.559.765.264.8

Under the same budget, RLVR still underperforms Self-HPO, and spending more compute on RL updates does not improve and even degrades its performance. This aligns with the finding in the community that training multi-turn agentic RL longer does not always keep improving it, as training can become unstable and eventually collapse [1, 2, 3]. To discuss this, we have added a new Appendix G and Figure 13 in the updated PDF, which show that the policy entropy of RLVR rises sharply and performance drops as training continues after a certain point.

This suggests that allocating part of the compute budget to harness search could be more effective than spending it all on policy updates for agentic optimization. However, how to best allocate compute between the two remains an open question, which we believe is an interesting direction for future work.

AuthorsOpenReview ↗

Response to Reviewer XMPY [Part 2/2]

[Q3]: The decrease in zero-success groups does not establish that more groups provide a GRPO learning signal. The authors should report the proportions of mixed-reward and all-success groups to support the proposed explanation.

[A3]: Per the reviewer's suggestion, we report the proportions of mixed-reward, all-success, and zero-success groups during RL, and have added a new Figure 7 in Appendix B of the updated PDF, extending Figure 4c in Section 3.4.

  • Mixed-reward groups, which receive a non-zero GRPO advantage: Figure 7(a) shows that Self-HPO keeps a larger ratio than RLVR (73.5% vs. 67.4% on average), as the ratio of Self-HPO increases and then plateaus while that of RLVR declines. Since only these groups produce a policy gradient, Self-HPO obtains more learning signals than RLVR.
  • All-success groups, which correspond to tasks that the policy already solves and provide no learning signal: Figure 7(b) shows that the two methods have similar ratios (16.8% vs. 18.6% on average), suggesting that the additional mixed-reward groups of Self-HPO do not come from easier tasks.
  • Zero-success groups, which correspond to tasks beyond the current capability of the policy: Figure 7(c) shows that the ratio of Self-HPO keeps decreasing, indicating that Self-HPO keeps expanding the range of tasks the policy can solve.

Together, these results confirm that Self-HPO enables more effective RL training than RLVR, mainly due to the denser training signals from the increased mixed-reward groups that come from fewer zero-success groups, as alternating harness optimization and policy optimization keeps expanding the range of tasks the policy can solve.

Please let us know if you have any further questions, and we are happy to incorporate additional suggestions you might have! If you find our response satisfactory, we would be grateful if you could consider raising your score. Thanks again for your time and efforts!

References

[1]. Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. COLM 2025

[2]. Xue et al. SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning. ICLR 2026

[3]. Wang et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073

AuthorsOpenReview ↗

A Gentle Reminder to Reviewer XMPY

Dear Reviewer XMPY,

We sincerely appreciate your detailed review and valuable feedback! We would like to gently remind you that the discussion period will end on November 18 (AoE). In our rebuttal, we have provided additional experimental results and clarifications in response to your comments, and we hope our rebuttal satisfactorily addressed your concerns.

Please let us know if you have any further questions, and we are happy to incorporate additional suggestions you might have. If our response resolves your concerns, we would greatly appreciate it if you could consider updating your score.

Thank you again for your time and efforts in reviewing our paper!

Best regards,

Authors of Submission 35303

Reviewer_vR3TOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Self-HPO, a framework that jointly optimizes the harness and policy to enable coding agents to continuously improve themselves. The main idea is to let the agent learn from its own failures to improve the harness, and then use RL to train the policy under the updated harness. The hard tasks identified during RL training are further used to guide the next round of harness optimization, forming a self-improvement loop. The experimental results are quite promising. Self-HPO improves the resolve rate on SWE-Bench Verified from 61.0% to 69.5%, with consistent gains on two other benchmarks. It also outperforms methods that optimize only the harness or the policy. Further analysis shows that improvements in the harness and policy are complementary, and using a stronger external model as the harness proposer can bring additional gains. Overall, I think this is a solid piece of work with a clear motivation, a reasonable approach, and convincing experimental results.

Critical strengths
  • The paper presents a well-motivated framework for jointly optimizing the harness and policy. Instead of optimizing them separately, Self-HPO allows the two components to improve each other through alternating optimization. I think this is an interesting and promising direction for building self-improving agents.

  • The experimental results are strong and the analysis is fairly comprehensive. Self-HPO consistently outperforms harness-only and policy-only baselines across three benchmarks. The authors also provide useful ablations and cross-evaluation results, showing that the improvements in the harness and policy are complementary. These results provide good evidence for the effectiveness of the proposed framework.

Critical weaknesses
  • The baseline comparisons could be more comprehensive. While the paper includes comparisons with harness-only and policy-only methods, it would be helpful to include more direct comparisons with recent approaches that jointly optimize the harness and policy. Such comparisons would help better demonstrate the advantages of Self-HPO over existing methods.

  • The experimental evaluation is somewhat limited, as it mainly focuses on software engineering tasks. It would be interesting to explore whether the proposed framework can generalize to other agentic domains, such as search and tool use. In particular, search tasks could provide an interesting setting to investigate whether the framework can automatically discover effective context management strategies through harness optimization.

[1] https://arxiv.org/abs/2609.00196 WHALE: A Simple Recipe for Joint Harness-Weight Optimization

[2] https://arxiv.org/pdf/2607.03935 Harness-Aware Self-Evolving: Co-Evolving Model Weights, Harness, and Task Solutions

[3] https://arxiv.org/abs/2607.22688 Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents

Questions for authors
  • Could the authors provide more analysis of how the model's harness modification behavior evolves during joint optimization? For example, does the model learn to propose more targeted or sophisticated harness modifications as training progresses? It would be interesting to see some concrete examples of how the proposed modifications change across different rounds.

  • Could the authors provide more details on the computational cost of Self-HPO, especially the additional overhead introduced by harness search? It would be helpful to understand the trade-off between the extra optimization cost and the performance gains, and how Self-HPO compares with policy-only training under a similar compute budget.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vR3T [Part 1/2]

We sincerely thank the reviewer for recognizing our work and for the thoughtful comments, which help us further strengthen the paper. To incorporate the reviewer's suggestions, we have uploaded an updated PDF, where all revisions are highlighted in blue. Below, we address each of the reviewer's questions point by point.

[Q1]: The baseline comparisons could be more comprehensive. It would be helpful to include more direct comparisons with recent approaches that jointly optimize the harness and policy.

[A1]: In our submission, we have cited and discussed the mentioned papers[1,2,3], and provided a conceptual comparison with relevant concurrent works in Table 6. Notably, none of them is evaluated on coding tasks since they focus on settings such as math reasoning and search QA, where an episode typically takes only a few interaction turns, whereas Self-HPO targets long-horizon coding agents that often take over a hundred turns per task.

Per the reviewer's suggestion, we further add a direct empirical comparison with WHALE, which is the only one that has released its code. For fair comparison, we apply its recipe in our setting with the same base model (Qwen3.5-35B-A3B), training tasks, and evaluation protocol.

MethodOptimizationSWE-VerifiedSWE-MultiSWE-ProAvg.
BaseN/A61.054.258.357.8
Self-Harnessharness-only65.254.360.760.1
RLVRpolicy-only67.556.361.861.9
WHALEjoint68.655.659.861.3
Self-HPOjoint69.559.765.264.8

The above results show that Self-HPO consistently outperforms WHALE across three coding benchmarks, even though WHALE relies on a stronger external proposer (Opus 4.8). We attribute the gap to two design choices: (1) WHALE updates the policy with rejection-sampling SFT and (2) searches harnesses on a fixed search set (as shown in Table 6), whereas Self-HPO updates the policy with RL and searches on the tasks that the current policy still fails, so that each H-step targets the updated policy's own failure modes.

[Q2]: The experimental evaluation is somewhat limited. It would be interesting to explore whether the proposed framework can generalize to other agentic domains, such as search and tool use.

[A2]: Following previous works [1,4], we apply our method Self-HPO to agentic search tasks, where the agent uses search tools to retrieve external information and answer open-domain and multi-hop questions. Specifically, we use Qwen3.5-35B-A3B as both the proposer and the policy, with the same seed harness and evaluation setup (seven QA benchmarks) from previous work [1].

Method2WikiBamboogleHotpotMuSiQueNQPopQATriviaAvg.
Base32.061.059.032.051.051.081.052.4
Self-HPO (H-step 1)49.069.061.034.055.048.082.056.9
Self-HPO (P-step 1)66.072.068.035.068.059.085.064.7
Self-HPO (H-step 2)61.068.065.037.068.057.085.063.0
Self-HPO (P-step 2)60.071.068.044.070.060.084.065.3

We find that Self-HPO already improves the average accuracy from 52.4 to 64.7 after the first round: H-step 1 alone adds 4.5 points, and the subsequent P-step 1 adds another 7.8 points. Interestingly, in H-step 2, the proposer introduces a harness that keeps only the most question-relevant sentences of earlier search results to trim the context. Based on our analysis (Appendix F), this simple context management reduces the context seen by the model at each step by 14% on average, while only slightly dropping the accuracy (64.7 to 63.0). After P-step 2, the accuracy recovers to 65.3, exceeding P-step 1 while using less context.

We fully agree that it is interesting to study how the proposer introduces such context-management mechanisms for search agents, while we conjecture that eliciting more advanced strategies may require a more carefully designed proposer instruction, which we leave for future work. We have added a new section Appendix F​ in the updated PDF to present these results and the harness analysis.

AuthorsOpenReview ↗

Response to Reviewer vR3T [Part 2/2]

[Q3]: Could the authors provide more analysis of how the model's harness modification behavior evolves during joint optimization? It would be interesting to see some concrete examples of how the proposed modifications change across different rounds.

[A3]: We thank the reviewer for this insightful question. In the updated PDF, we extended Appendix D and added a new Table 7 to present how the proposed harness modifications evolve across 12 rounds of Self-HPO, where each H-step runs 6 rounds of proposals by the current policy. We summarize the main findings below:

  • Progressive evolution on top of prior harnesses. Each proposal is built on top of the current best harness, so the harness can accumulate mechanisms across rounds. For instance, the final H-step 2 harness keeps the H-step 1 coverage check and combines it with two mechanisms proposed in different rounds of H-step 2.

  • From where to edit (H-step 1) to what the edit does (H-step 2). In H-step 1, the proposals mainly check whether the agent edits the right places. For example, in round 2 the harness asks the agent to also check related documentation, configuration, and modules before submitting, and in round 4 (the selected harness) it compares the files the agent explored with the files in its patch. In H-step 2, the proposals instead check whether the edit itself is correct and addresses the issue. For example, in round 1 the harness reminds the agent to import newly used decorators, in round 5 it checks the signatures of newly added functions, and in round 6 it warns when the key terms of the issue do not appear in the patch.

Overall, as Self-HPO progresses, the proposed harnesses become more sophisticated as they accumulate mechanisms across rounds, while the focus of proposed modifications shifts from where to edit (H-step 1) to what the edit does (H-step 2).

[Q4]: Could the authors provide more details on the computational cost of Self-HPO, especially the additional overhead introduced by harness search? It would be helpful to understand the trade-off between the extra optimization cost and the performance gains, and how Self-HPO compares with policy-only training under a similar compute budget.

[A4]: We thank the reviewer for this suggestion. The computational cost of RL training and harness search depends on many factors that are hard to track exactly, including rollout generation, policy updates (forward and backward passes), sandbox initialization, and command execution. For this analysis, we measure it by the number of generated rollouts, which dominate the cost of both RL training and harness search.

MethodOptimizationRL rolloutsHarness search rolloutsTotalAvg. Performance
BaseN/A00057.8
RLVRpolicy-only48k048k61.9
Self-HPOjoint48k4k52k64.8

Compared to RLVR, Self-HPO introduces only a small additional overhead from harness search, about 8% of its total budget, while achieving 2.9 points higher average performance, demonstrating the advantage of our method.

As suggested, to compare with policy-only training under a similar budget, we further train RLVR to 125 steps, which matches the total budget of Self-HPO.

MethodBudgetSWE-VerifiedSWE-MultiSWE-ProAvg.
Base061.054.258.357.8
RLVR (100 steps)48k67.556.361.861.9
RLVR (125 steps)52k64.751.457.357.8
Self-HPO52k69.559.765.264.8

Under the same budget, RLVR still underperforms Self-HPO, and spending more compute on RL updates does not improve and even degrades its performance. This aligns with the finding in the community that training multi-turn agentic RL longer does not always keep improving it, as training can become unstable and eventually collapse [4, 5, 6]. We have updated the PDF and added a new Figure 13 in Appendix G to make this clear: the policy entropy of RLVR rises sharply and performance drops as training continues after a certain point.

This suggests that allocating part of the compute budget to harness search could be more effective than spending it all on policy updates for agentic optimization. However, how to best allocate compute between the two remains an open question, which we believe is an interesting direction for future work.

Please let us know if you have any further questions, and we are happy to incorporate additional suggestions you might have. Thanks again for your time and efforts!

AuthorsOpenReview ↗

References

References

[1]. Kim et al. WHALE: A Simple Recipe for Joint Harness-Weight Optimization. arXiv:2609.00196

[2]. Luo et al. Harness-Aware Self-Evolving: Co-Evolving Model Weights, Harness, and Task Solutions. arXiv:2607.03935

[3]. Chen et al. Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents. arXiv:2607.22688

[4]. Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. COLM 2025

[5]. Xue et al. SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning. ICLR 2026

[6]. Wang et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.