acceptodds
Under review as a conference paper at ICLR 2027

DuetHarness: Bidirectional Policy–Harness Co-Optimization for LLM Agents

Abstract

The capability of a large language model (LLM) agent depends jointly on its policy and its harness, the executable runtime that mediates interaction with the environment. Existing work either optimizes one side or updates both, but learning still remains essentially one-sided. Co-optimization is difficult because the policy and harness control separate strategies, yet each update changes the problem faced by the other. In this paper, we cast their interaction as a common-payoff game in which each side is improved against the other's current configuration. The idealized fixed point of this coupling is a mutual best response, a Nash equilibrium of this game. We therefore introduce DuetHarness, which implements this coupling by alternating learned updates analogous to approximate best-response steps. With the policy frozen, the editor learns to revise the harness from rollout feedback; with the harness fixed, the policy adapts to it by training against both the current and historical harnesses. Across three agent benchmarks with Qwen3.5-9B, DuetHarness consistently outperforms the corresponding vanilla agent and strong baselines. Ablations show that these gains rely on the alternating co-optimization rather than simply optimizing both sides. Our code is available at https://anonymous.4open.science/r/DuetHarness_ano-F0CE/.

Primary area:
reinforcement learning
Keywords:
LLM agents, agent harness, reinforcement learning, policy–harness co-optimization, alternating optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qWLaOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors present DuetHarness, which jointly trains the policy and the harness during RL. They perform alternating training phases, finding a Nash equilibrium of a common-payoff game with their method. The method is evaluated on ALFWorld, WebShop, and DBBench. The method achieves the best (or tied for the best) results in nearly every category.

Critical strengths
  • The paper is well-written and clearly explains the idea with good logical flow.
  • The diagrams are very clear, especially Fig 1 and Fig 2.
  • The results are good. The method outperforms baselines on a variety of datasets.
  • The paper focuses on an important problem - improving harnesses and RL optimization through a joint optimization.
  • The method is simple enough to be easily understood while representing a novel contribution.
  • The paper includes a lot of detail, as well as code for reproducibility.
Critical weaknesses

None

Questions for authors

Can you add a couple of concrete examples of what the harness editor learned/changed to help build intuition of what it did to improve?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qWLa: Case studies of learned harness edits

We thank the reviewer for the positive assessment and for this helpful suggestion. Below, we give three concrete examples, one per benchmark, from the harnesses retained for testing.

Q1. Concrete examples of what the harness editor learned.

Example 1 (WebShop). Fixing clicks that do not match any button. In WebShop, a click only works if its text exactly matches a button on the page. Otherwise it is ignored but still costs a step. Under h0h_0, the agent often clicks text that is not a button, such as click[size] or a full product title, and many episodes run out of steps before buying. The editor learned a hook that checks each click before it is sent. If the click is close to a listed button, the hook changes it to that button. If not, the hook blocks it and tells the agent which buttons are available. For example, in one test task the agent needs a scented candle in the "voyager" option. The agent clicks the long product title click[Mer Sea Luxury Scented Candle in Tin - Voyager, 25 Hour Burn], and the hook changes it to click[voyager], so the correct option is selected and the purchase succeeds. Together with the policy trained to use this hook, episodes that end without a purchase drop from 128 to 2 out of 500 (Table 11 (now Table 13)).

Example 2 (ALFWorld). Going to the right place before acting. In ALFWorld, an action such as putting an object on a countertop only works when the agent is standing at that countertop. Under h0h_0, the agent often tries to act from the wrong place, so the action fails and a step is wasted. For example, in the task "clean some knife and put it in countertop", the agent under h0h_0 keeps issuing move pan 1 to countertop 1 from elsewhere and the episode ends after repeated failed actions. The editor learned a hook that catches such actions and turns them into the missing step. When the agent issues move knife 1 to countertop 1 away from the countertop, the hook changes it to go to countertop 1, and the agent then completes the task. Together with the policy trained to use this hook, these failed move actions drop from 285 to 37, and success on the unseen split rises from 41.8% to 94.8% (Table 1).

Example 3 (DBBench). Checking a change once before submitting. In DBBench, a task that modifies the database (INSERT, UPDATE, or DELETE) is judged by the final content of the table. Under h0h_0, the agent often loses track of whether its change has already been applied. For example, in a task that asks to insert one record into the table Athlete Performances, the agent under h0h_0 issues the same INSERT four times, which leaves duplicate rows, and the task fails. The editor learned two hooks for such tasks. After a modification, the first hook asks the agent to check the change with a SELECT before submitting. Once this SELECT returns the new row, the second hook tells the agent that the change has been applied and that it should submit without repeating it. With these hooks, the agent inserts the record once, checks it with one SELECT, and submits. Together with the policy trained to use them, these hooks are involved in 25 of the 34 test tasks that DuetHarness solves but the vanilla agent does not.

These hooks were not written in one step. On WebShop, for example, the editor first added guards that blocked clicks on option names, but these guards also blocked some valid "Buy Now" clicks and hurt performance. In a later round, the editor deleted them and wrote the hook in Example 1 instead. This shows that the editor learns from the results of its own edits, and can remove a harmful hook rather than only adding new ones.

We have added these examples to Appendix K.

Thank you for the information and addition to the appendix. I will maintain my score.

AuthorsOpenReview ↗

Thank you for your positive assessment

Dear Reviewer qWLa,

Thank you for the positive assessment and thoughtful feedback. We are glad that the additional examples helped clarify the learned harness edits and their impact. We appreciate your time and consideration throughout the review process.

Best regards,

The Authors

Reviewer_UXTwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DuetHarness, an alternating framework for jointly improving an agent policy and its executable harness. It models the policy–harness pair as a common-payoff game with mutual best response as an idealized target. During Phase-E, a trained editor proposes harness patches using rollout summaries, with rewards measured on both evidence tasks and a task-disjoint transfer batch. During Phase-P, the policy is trained under the updated harness and historical harnesses. Experiments use Qwen3.5-9B on WebShop, ALFWorld, and DBBench. The paper reports improved success rates, WebShop ablations on alternation and editor rewards, and analyses of interaction cost and harness evolution.

Critical strengths
  1. The paper directly studies the interaction between policy learning and harness adaptation. Training the editor to respond to the changing policy is a meaningful extension beyond repeatedly asking a fixed proposer for edits.

  2. The WebShop ablations include policy-only, editor-only, sequential, and frozen-editor settings. Appendix I specifies matched rollout budgets, which makes these comparisons more informative about the contribution of alternation.

  3. The method includes explicit information boundaries, patch validation, and task-disjoint evidence and transfer batches. The detailed hook descriptions and learning traces help understand better what the harness actually changes.

  4. The evaluation reports both task performance and interaction costs. The reductions in environment steps and prompt tokens on WebShop and ALFWorld are useful practical results. The paper also reports the small cost increases on DBBench rather than presenting efficiency as uniform across benchmarks.

Critical weaknesses
  1. Eq. (7) leaves a potential issue in the editor’s learning signal. Invalid or incomplete candidates receive zero reward, whereas valid candidates can receive negative reward. Under group normalization, an invalid patch can therefore receive positive advantage if the other candidates are harmful. The paper does not specify a validity mask or a separate treatment that resolves this incentive. This needs clarification before the reward design can be fully assessed.

  2. The comparison with the closest co-evolution approaches is incomplete. Co-Harness, HarnessForge, and HarnessX are discussed as related methods but are not directly evaluated in Table 1. The frozen-editor ablation is useful, although it uses the cold-start proposer and accepts the first legal, smoke-tested patch. A stronger frozen proposer with reward-based patch selection would provide a more demanding comparison, using same rollout budget.

  3. The equilibrium interpretation has limited empirical support. The paper already qualifies the residual probe as operator-relative, but continuing the same optimizer for three rounds does not bound improvements obtainable through stronger unilateral searches. In particular, deteriorating harness edits can indicate limitations of the response operator. Testing different search budgets and retaining the best candidate response would make this interpretation more clear.

  4. The robustness of the reported margins is difficult to assess. Table 2 averages three repeated runs but does not show their dispersion, while Table 1 reports point estimates. This matters especially for DBBench, where the margin over Harness-R1 is only one percentage point. The authors acknowledge limited multi-seed coverage, and additional independent training runs would help establish the stability of the comparisons.

Questions for authors
  1. Are invalid or incomplete patch candidates included in GRPO normalization and gradient updates? If so, how do you prevent their zero rewards from becoming preferable to negative rewards from valid patches? Reporting invalid-patch rates during training would also be useful.

  2. Could you compare against a stronger frozen proposer with reward-based patch selection, or one of the cited policy–harness co-evolution methods under a common runtime? This would help isolate the benefit of learning the editor beyond the current frozen-editor ablation.

  3. Could you repeat the unilateral residual probe with larger search budgets and best-candidate retention? Please also clarify the +0.013 residual in Appendix L versus +0.012 in Table 14.

  4. Could you provide per-run results or standard deviations for Table 2, and independent training-seed results for the main comparisons? This would give more confidence on the smaller margins, particularly on DBBench.

9 comments
AuthorsOpenReview ↗

Response to Reviewer UXTw (1/4): Invalid patches in the GRPO signal(W1 & Q1)

We thank the reviewer for the careful reading and the constructive questions. We address each weakness together with the corresponding question below.

W1 & Q1. Invalid patches in the editor's GRPO signal.

Thank you for this insightful and careful observation! In our design, invalid or incomplete candidates are included in the group normalization with reward 0. Our original intent was to rank invalid patches below valid patches that improve the paired reward, so that the editor is pushed toward producing patches that pass the validator. However, we did not adequately account for the side effect the reviewer points out. In a group where every valid patch is harmful, an invalid patch ranks above them and receives a positive advantage. Following this comment, we analyzed how much this affects training, using the per-candidate rewards and advantages recorded for all Phase-E groups of the reported runs.

WebShopALFWorldDBBench
Invalid or incomplete candidates21.4%19.1%25.7%
Groups where an invalid candidate receives a positive advantage46.2%44.4%46.4%
Share of positive advantage mass on valid candidates80.2%80.6%74.2%
Mean advantage of invalid candidates+0.080+0.155+0.209
Legal-patch rate across rounds64.1–92.2%68.8–90.6%56.2–87.5%

The above table demonstrates that, in nearly half of the groups, an invalid candidate receives a positive advantage, and the mean advantage of invalid candidates is slightly positive, so the gradient does mildly favor invalid outputs. However, this is not the dominant signal. About three quarters or more of the positive advantage mass goes to valid patches that actually improve the paired reward, and the legal-patch rate fluctuates within a stable range rather than collapsing over training. The results in Tables 1 and 2 are all obtained under this reward, so this issue makes the editor's learning signal less efficient but does not affect the reported conclusions. A natural refinement, which we leave to future work, is to normalize only over valid candidates and assign invalid ones a fixed penalty, or a reward below the worst valid candidate in the group, so that invalid patches can never be preferred. Thank you once again for raising this careful observation.

AuthorsOpenReview ↗

Response to Reviewer UXTw (2/4): Concurrent co-evolution methods and a stronger frozen proposer(W2 & Q2)

W2 & Q2. Comparison with co-evolution methods and a stronger frozen proposer.

We thank the reviewer for pointing out these concurrent works. Co-Harness, HarnessForge, and HarnessX each define the harness in their own way. HarnessForge decomposes it into planning, action, and memory modules, HarnessX composes typed primitives through a substitution algebra, and Co-Harness edits prompts, tools, skills, middleware, and memory, whereas DuetHarness edits a hook-based runtime through validated patches. These methods are also evaluated on different benchmarks and protocols (e.g., pass@2 in HarnessX and tool-integrated math reasoning on AIME and HMMT in Co-Harness), and not all of them have released code. A direct comparison in Table 1 would therefore mix differences in harness design with differences in the optimization scheme, and would be hard to make fair to either side.

Nevertheless, these methods share the paradigm in Figure 1(c), where both sides (policy-harness) are updated but harness edits come from a fixed proposer, and our frozen-editor ablation (Table 2 and Appendix I) instantiates this paradigm within our own harness space. It keeps the alternation, the harness space, and the policy training unchanged, and only stops the editor from learning. It reaches 18.6% success on WebShop, compared with 46.0% for DuetHarness, which isolates the benefit of an editor that keeps learning as the policy moves.

We also agree that this ablation, which keeps the cold-start proposer θ0\theta_0 and installs the first legal, smoke-tested patch, is not the strongest frozen alternative. To address this, we replace θ0\theta_0 with Qwen3.8-Max as a stronger frozen proposer, and keep everything else in the alternating loop unchanged. In each round, Qwen3.8-Max samples K=8K=8 candidate patches from the same evidence packets, and we install the candidate with the highest paired reward.

Variant (WebShop)ScoreSucc.
Frozen editor (θ0\theta_0, first legal patch)56.818.6
Frozen Qwen3.8-Max proposer, best-of-KK by paired reward66.841.2
DuetHarness74.346.0

The stronger proposer raises success from 18.6% to 41.2%, but still falls 4.8% short of DuetHarness. Our failure analysis shows that its main failure is producing invalid patches, since a frozen proposer never learns the patch interface or the validator from feedback. DuetHarness reaches higher success with a 9B editor and no re-ranking, which indicates that the gain comes mainly from the editor continuing to learn rather than from a stronger proposer.

Additionally, we clarify that DuetHarness installs the first patch that passes the smoke test by design. Phase-E has just trained the editor with the paired reward, so patch quality should come from the editor itself. Re-ranking would hand this role back to external search, increase the rollout cost, and risk overfitting to a small batch, while Phase-P adapts the policy to a patch that does not help immediately. We have added this experiment and discussion to the ablation studies in Section 5.3.

AuthorsOpenReview ↗

Response to Reviewer UXTw (3/4): Strength of the equilibrium interpretation(W3 & Q3)

W3 & Q3. Strength of the equilibrium interpretation.

Thank you for this comment. We agree that the probe only measures residual gain under our own operators and does not bound what a stronger unilateral search could find, which is why the paper regards the pair only as an empirical Nash under this operator (§3, §5.5, Limitations).

Following the suggestion, we apply the unilateral residual probe to the pair retained by a DuetHarness variant that installs the best of K=8K=8 candidate patches by paired reward in each round, following the original protocol of three frozen-side rounds on the 200-task validation split. The only change is on the harness side, where each round installs the best of K=8K=8 sampled patches by paired reward.

PairProbe (frozen side)ScoreResidual
DuetHarness (original)Policy (harness frozen)0.727 → 0.739+0.012
Harness (policy frozen), first passing patch0.727 → 0.6150
DuetHarness (best-of-8)Policy (harness frozen)0.728 → 0.730+0.002
Harness (policy frozen), best-of-80.728 → 0.6620

With best-of-8 selection, the harness score falls less than with the first passing patch (0.662 vs. 0.615), but the stronger harness operator still finds no improvement over the retained pair, and the policy residual is only +0.002. Both residuals remain negligible, which supports regarding the retained pair as an empirical Nash under this stronger operator as well. We have revised the paper accordingly, moving the probe into a new subsection, "Equilibrium Analysis" (Section 5.5), and adding these results in a new Table 4.

We also thank the reviewer for catching the inconsistency between Appendix L and Table 14 (now Table 4). In our experiment records, the starting and final validation scores of the policy probe are 0.7266 and 0.7394, so the exact residual is +0.0128. Appendix L rounds this value to +0.013, whereas Table 14 (now Table 4) subtracts the rounded scores (0.739 and 0.727) and reports +0.012. We have corrected this in the paper, and the residual is now reported consistently as +0.012 in Table 4 (Section 5.5) and Appendix L.

Reviewer_SsQrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DuetHarness models an LLM agent as the combination of a policy and an executable harness, jointly optimizing both components. The harness consists of programmable hooks that can modify context, reject actions, or adjust execution budgets. A dedicated editor, initialized through supervised patch demonstrations and subsequently trained with SFT and GRPO, proposes harness modifications while the agent policy remains frozen.

Critical strengths

1.The split is clear: the policy chooses actions, and the harness controls observation, execution, and termination. The editor is trained, rather than left as a frozen proposer. 2.On WebShop, the ablation isolates alternation. Policy-only reaches 38.6% success, editor-only 27.4%, sequential editor-then-policy 40.0%, and the full loop 45.8%. A frozen editor falls to 18.6%.

Critical weaknesses

1.The alternation gap is shown only on WebShop, and it is 7.2 points over policy-only and 5.8 over sequential training. ALFWorld’s jump of about 50 points has no policy-only or sequential control. DBBench ends at 66.0%, against 65.0% for Harness-R1. The long-horizon ALFWorld gains come from hooks that rewrite navigation before the environment steps, such as walking to the right room before heat, cool, or clean. That moves part of the decision rule into code. The Nash claim is stronger than the probe. With the policy frozen, three further harness updates drop the WebShop validation score from 0.727 to 0.615. The paper sets the harness residual to zero because the score fell. The deployed patch is the first candidate that passes a smoke test, not the highest-reward patch. 2.Table 1 has no variance. The limitations still list multi-seed runs as future work, while Table 2 reports three-run averages on WebShop only. The starting harness h0 is weaker than the baseline harness on WebShop (22.2% versus 32.0%), so the +23.6 point within-stack gain includes recovery from that start. Editor demonstrations are filtered from a stronger teacher, qwen3.8-max. Evaluation uses one 9B backbone and three established environments. 3.Limited benchmark diversity and model generalizability. The evaluation is restricted to three established environments and a single model family and scale (Qwen3.5-9B).

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer SsQr (1/4): Ablations on all benchmarks and the source of ALFWorld gains(W1.1–W1.2)

We sincerely thank the reviewer for the careful and detailed reading of our paper. Below, we respond to each point, and we will revise the paper accordingly.


W1. Ablations on other benchmarks, source of the ALFWorld gains, the Nash claim, and experimental details

We split this comment into four sub-points.

W1.1 Ablations on other benchmarks.

Thanks for this comment. We agree this is the most important gap. We have run all ablations of Table 2 on ALFWorld and DBBench. The table below extends Table 2 to all three benchmarks. All entries are means and standard deviations over three runs. For DuetHarness, we report the three-run mean from W2.1, so that it is directly comparable with the ablation variants.

VariantWebShop ScoreWebShop Succ.ALFWorld unseenALFWorld seenDBBench
Qwen3.5-9B + h0h_047.0 ± 1.322.8 ± 1.041.8 ± 2.047.9 ± 3.059.0 ± 0.5
Which sides learn, and whether they alternate
Policy-only (h≡h0h\equiv h_0)70.1 ± 1.538.6 ± 1.376.1 ± 2.576.4 ± 2.361.9 ± 0.8
Editor-only (π≡πϕ0\pi\equiv\pi_{\phi_0})55.8 ± 2.427.4 ± 3.274.5 ± 0.874.5 ± 1.259.5 ± 1.0
Sequential (E→P)70.5 ± 0.640.0 ± 0.884.1 ± 1.480.0 ± 1.264.2 ± 0.6
Frozen editor (θ0\theta_0)56.8 ± 3.918.6 ± 4.343.3 ± 1.650.0 ± 3.856.3 ± 0.3
Mechanisms for two-sided optimization
w/o transfer batch (α=1\alpha=1)71.9 ± 1.243.6 ± 0.985.6 ± 0.883.3 ± 0.864.5 ± 0.2
w/o evidence batch (α=0\alpha=0)69.7 ± 1.040.2 ± 0.781.5 ± 1.080.4 ± 1.260.5 ± 0.3
DuetHarness74.3 ± 0.846.0 ± 0.693.6 ± 3.588.8 ± 2.266.0 ± 0.7

As shown in the above table, the results on ALFWorld and DBBench are consistent with the conclusions on WebShop. The full method is best on all three benchmarks. Training only one side (policy-only, editor-only) or training both sides sequentially (E→P) falls short. For example, on ALFWorld policy-only reaches 76.1 / 76.4 and sequential reaches 84.1 / 80.0 (unseen / seen), versus 93.6 / 88.8 for DuetHarness. Freezing the editor leaves the agent near or below its h0h_0 starting point (43.3 / 50.0 on ALFWorld, 56.3 on DBBench). Removing either term of the paired reward also reduces success on both benchmarks. We have updated Table 2 in the paper accordingly, which now reports the ablations on all three benchmarks with standard deviations.

W1.2 The ALFWorld gains come from hooks that rewrite navigation, moving parts of the decision rule into code.

Thank you for this observation. This is exactly the behavior DuetHarness is designed to produce. The navigation logic (go to the right place before heat/cool/clean) was not hand-written by us. The editor discovered it from rollout evidence and wrote it into the harness, and Phase-P then trained the policy to work with it. Learning which parts of the decision rule are better handled in executable code is the point of making the harness a learned, editable object.

AuthorsOpenReview ↗

Response to Reviewer SsQr (2/4): The equilibrium claim and patch selection(W1.3–W1.4)

W1.3 The Nash claim is stronger than the probe. The harness residual is set to 0 because the score fell.

Thank you for this careful reading. The probe is meant to check whether the retained policy-harness pair is an approximate Nash equilibrium, that is, with one side frozen, does continuing the other side's update still yield a meaningful gain? On the policy side, three further Phase-P rounds change the score by only +0.012. On the harness side, three further Phase-E rounds find no improvement, i.e., the residual is 0. We note that the paper already frames the result as approximate rather than exact. §3 states that both responses are approximate due to finite rollouts, restricted patch spaces, and finite budgets. §5.4 (now §5.5) states that the alternating updates "need not reach a Nash equilibrium" and regards the pair as an empirical Nash under this operator. The Limitations describe the updates as "approximate best responses rather than a Nash solver". To further check this with a stronger harness operator, we apply the same probe to the pair retained by the best-of-8 variant described in W1.4, where each harness round installs the best of K=8K=8 sampled patches by paired reward instead of the first one that passes the smoke test.

PairProbe (frozen side)ScoreResidual
DuetHarness (original)Policy (harness frozen)0.727 → 0.739+0.012
Harness (policy frozen), first passing patch0.727 → 0.6150
DuetHarness (best-of-8)Policy (harness frozen)0.728 → 0.730+0.002
Harness (policy frozen), best-of-80.728 → 0.6620

With best-of-8 selection, the harness score falls less than with the first passing patch (0.662 vs. 0.615), but the stronger harness operator still finds no improvement over the retained pair, and the policy residual is only +0.002. Both residuals remain negligible, which supports regarding the retained pair as an empirical Nash under this stronger operator as well. We have revised the paper accordingly, moving the probe into a new subsection, "Equilibrium Analysis" (Section 5.5), and adding these results in a new Table 4.

W1.4 The deployed patch is the first candidate to pass a smoke test, not the highest-reward patch.

Thank you for pointing this out. This is a deliberate design choice. In DuetHarness, patch quality is meant to come from the learned editor itself. Phase-E has just trained the editor with the paired reward to produce patches that help on both the evidence and transfer batches, so the persistent patch is already a sample from an editor that has learned what helps. Re-ranking candidates by reward at this step would shift part of this role back to an external search procedure rather than the learned editor. It would also increase the rollout cost of every round, and selecting on a small batch risks overfitting to it. The smoke test therefore guards only against crashes, since the installed harness is inherited by all later rounds. A patch that does not immediately raise the score is acceptable, because the following Phase-P adapts the policy to it.

To verify this choice, we run a DuetHarness variant that, in each round, samples K=8K=8 candidate patches from the updated editor and installs the one with the highest paired reward, instead of the first one that passes the smoke test. All other settings are unchanged.

Variant (WebShop)ScoreSucc.Training cost
DuetHarness (first passing patch)74.045.81.00×
DuetHarness (best-of-8 by paired reward)72.844.61.09×

Best-of-8 selection raises the training cost by about 9%, which comes entirely from evaluating the 8 extra candidates on the paired evidence and transfer batches in every round. It does not improve performance, with success 1.2% lower than the original DuetHarness, possibly because selecting on small batches overfits to them. This supports installing the first passing patch and leaving patch quality to the trained editor.

AuthorsOpenReview ↗

Response to Reviewer SsQr (3/4): Variance, starting harness, and model choices(W2)

W2. Missing variance, gains from a low starting point, and the choice of teacher and evaluation models

W2.1 Variance is not reported in Table 1, and Table 2 reports WebShop only.

Thank you for raising this comment. For Table 1, we ran each method once so that DuetHarness is compared with the baselines under the same protocol as the reported baseline results. To address the reviewer's concern, we have re-run DuetHarness and the strongest baseline, Harness-R1, three times each on all three benchmarks and report the mean and standard deviation below.

WebShop ScoreWebShop Succ.ALFWorld unseenALFWorld seenDBBench
Harness-R1 (Table 1, single run)69.041.052.245.765.0
Harness-R1 (3 runs, mean ± std)68.7 ± 0.841.3 ± 1.454.2 ± 2.346.0 ± 3.263.9 ± 3.5
DuetHarness (Table 1, single run)74.045.894.890.766.0
DuetHarness (3 runs, mean ± std)74.3 ± 0.846.0 ± 0.693.6 ± 3.588.8 ± 2.266.0 ± 0.7

For both methods, the single-run numbers in Table 1 lie within the three-run spread, and DuetHarness's mean remains above Harness-R1's on every benchmark. DuetHarness also shows lower variance in most settings, most notably on DBBench (0.7 vs. 3.5).

Regarding Table 2, which reports results of WebShop only, we have extended all ablations to ALFWorld and DBBench with standard deviations, as detailed in our response to W1.1.

W2.2 The gain on WebShop builds on a weak starting harness (22.2 vs. 32.0).

Thank you for this comment. We rewrote the default harness of Harness-R1 as h0h_0, a minimal set of guidance-only hooks in our runtime (Appendix A). In Harness-R1, the default harness is part of the fixed runtime, and its editor can only add hooks on top of it. In our runtime, a harness consists entirely of hooks and parameters, so restating the default harness in this form allows our editor to replace or delete any part of the starting harness rather than only add to it. On WebShop, h0h_0 scores lower mainly because its default budgets are tighter (15 steps instead of 20, and an episode ends after three consecutive ignored actions), so more episodes end without a purchase (128 of 500, Table 11 (now Table 13)).

We agree that part of the large within-stack gain on WebShop reflects this lower starting point. Nevertheless, DuetHarness still achieves the best final performance. It reaches 45.8% success on WebShop, compared with 41.0% for Harness-R1, which starts from the stronger default harness (32.0%). The other benchmarks lead to the same conclusion. On DBBench, h0h_0 also starts below the default harness (59.0 vs. 61.7), yet DuetHarness still ends above Harness-R1 (66.0 vs. 65.0). On ALFWorld, h0h_0 starts higher (41.8 / 47.9 vs. 29.9 / 41.4), but the final gap over Harness-R1 (94.8 / 90.7 vs. 52.2 / 45.7) is far larger than this initial difference. The final performance of DuetHarness is therefore not determined by the strength of its starting harness, but reflects the benefit of policy–harness co-optimization. Moreover, all ablation variants in Table 2 start from the same h0h_0, so those comparisons are unaffected by the choice of starting point.

W2.3 Choice of teacher and evaluation models.

Thank you for this comment. We use a stronger teacher only to build the small cold-start SFT set (306 / 169 / 124 demonstrations), whose purpose is to teach the editor the patch format before RL. Without it, most patches sampled by the editor would fail validation and receive the same zero reward, leaving GRPO with no learning signal. Actually, even prompted frontier models often emit illegal patches (Section 5.2), so the teacher provides a usable starting point. The frozen-editor ablation in Section 5.3, which keeps the teacher-distilled θ0\theta_0 throughout training, reaches only 18.6% on WebShop, so the gains come from the editor continuing to learn in the alternating RL loop rather than from distillation.

For the evaluation model, we follow the setting of Harness-R1, which uses Qwen3.5-9B on the same three benchmarks, so that our results are directly comparable with this closest baseline. A 9B model is also small enough to train both the policy and the editor with RL within our compute budget. To assess other model scales, we additionally evaluate DuetHarness on Qwen3.5-4B, as detailed in our response to W3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chairs, Senior Area Chairs, and Program Chairs,

Thank you sincerely for the time and care you have devoted to our submission, DuetHarness (#16868). We are also grateful to the three reviewers for their thoughtful and constructive feedback, which has helped us improve the paper considerably. We have responded point by point to every weakness and question raised in the reviews, and we have completed additional experiments to address the reviewers' core concerns. All revisions made to the paper during the rebuttal are highlighted in red in the updated version. A brief summary is provided below for your convenience.

Reviewer SsQr

  • Ablations only on WebShop: Following the reviewer's suggestion, we extended all ablations to ALFWorld and DBBench. The full method remains the best on every benchmark, which is consistent with our original findings on WebShop. (Updated in Table 2.)
  • Variance and robustness: We agree that variance should be reported. We therefore re-ran DuetHarness and the strongest baseline, Harness-R1, three times each on all benchmarks. DuetHarness stays ahead on every benchmark and also shows lower variance.
  • Equilibrium claim and patch selection: We clarified that the paper frames the equilibrium as approximate and relative to our update operators. We also repeated the equilibrium probe with a stronger best-of-8 harness operator, and it leads to the same conclusion. In addition, we found that selecting the best of 8 patches instead of the first valid one increases training cost without improving results. (The equilibrium analysis is added as Section 5.5 and Table 4.)
  • Weaker starting harness and stronger teacher: We explained that the final gains do not depend on the starting harness, and that they come from continued RL training of the editor rather than from distillation of the teacher.
  • Benchmark and backbone diversity: To examine generalization across model scales, we added results on Qwen3.5-4B, where DuetHarness again achieves the best results on all test sets.
  • Follow-up discussion: The reviewer confirmed that the additional ablations and multi-seed results address the concerns on these points. For the two remaining points, we clarified that the equilibrium probe shows that neither side finds a better point by updating alone under our operators, while acknowledging that a stronger search might find one, and we agreed to discuss broader environments and model families as future work.

Reviewer UXTw

  • Invalid patches in GRPO: We quantified the effect using the training logs and found that most of the positive advantage still goes to valid patches, so the conclusions of the paper continue to hold. In the follow-up discussion, we further implemented a validity-aware advantage that removes all such noise groups and re-trained DuetHarness on all three benchmarks. It raises the legal-patch rate and speeds up early training but does not further improve final performance, which confirms that the original reward already provides an effective learning signal. (To be discussed in the revised version.)
  • Stronger baselines: As suggested, we added a stronger frozen proposer, Qwen3.8-Max with best-of-8 patch selection. It still falls short of DuetHarness, which supports the benefit of an editor that keeps learning. We also explained why a direct comparison with the concurrent co-evolution methods would be difficult to make fair. (Added to Section 5.3.)
  • Equilibrium interpretation: We agree that the probe is relative to our operators. Following the reviewer's suggestion, we repeated it with a stronger operator, and the residuals remain negligible. We have also corrected a rounding inconsistency in the reported residual. (Added to Section 5.5 and Table 4; corrected in Appendix L.)
  • Dispersion and the DBBench margin: We re-ran the main comparison (DuetHarness and Harness-R1) with three independent seeds. The DBBench margin holds, and DuetHarness is more stable across runs. (Standard deviations for the ablations are added to Table 2.)

Reviewer qWLa

  • Concrete examples: Following the reviewer's suggestion, we added one example per benchmark of a hook that the editor learned, as well as a case where the editor removed a harmful hook it had written earlier. (Added to Appendix K.)

We hope that these additions address the central concerns of all three reviews, and we believe they have substantially strengthened the paper. During the discussion, Reviewer UXTw raised the score, and Reviewer SsQr confirmed that the concerns on ablations and variance have been addressed.

Thank you again for your time and consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.