- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how agent harnesses influence learned policies and their transferability across interfaces. It formulates harness-induced decision processes using KL-regularized control and introduces a ceiling–mismatch decomposition to distinguish harness potential from policy adaptation errors. Through controlled MiniWoB++ experiments and coding-agent evaluations, the authors demonstrate that harness changes can significantly degrade frozen-policy performance, while further adaptation can recover and improve performance. The paper also explores mismatch-based estimation for predicting harness performance.
Critical strengths
-
The ceiling–mismatch decomposition provides an intuitive and principled perspective for understanding performance degradation and recovery under harness changes.
-
The use of exactly solvable MiniWoB++ environments enables precise analysis of optimal policies, readout drift, and adaptation effects.
-
The coding-agent experiments extend the discussion beyond controlled settings and suggest potential applications to harness selection and adaptation.
Critical weaknesses
-
The ceiling depends on the reference policy rather than solely on harness capabilities. For example, H5 achieves a higher ceiling than H0 despite identical execution dynamics, raising concerns about its interpretation as intrinsic harness potential.
-
The coding-agent experiments replace the optimal policy with a teacher and compress actions into 13 coarse categories. These approximations lack theoretical guarantees and may not preserve the true performance gap.
-
The controlled experiments use a single training seed, while coding-agent estimates rely on only three probe tasks. The reported perfect ranking correlation may be unstable. Computational costs also differ across harnesses.
Questions for authors
Do the main conclusions hold under a shared reference policy or without KL regularization?
4 comments
The coding-agent experiments replace the optimal policy with a teacher and compress actions into 13 coarse categories. These approximations lack theoretical guarantees and may not preserve the true performance gap.
Response: Thank you for your important point about the teacher approximation and the action categories. Both approximations now carry explicit theoretical bounds and preserve true harness rankings across teachers, category granularities, and benchmarks.
First, for the teacher approximation, we prove two guarantees. Normal task success under any teacher is a valid lower bound on the plain ceiling at with no optimality assumption or action categories, and the proxy mismatch bias is bounded linearly by the teacher's action-score error relative to the optimum, so harness rankings stay unchanged whenever the true return gap exceeds the difference in proxy bias. Across all 39 Terminal-Bench Lite tasks with 3 runs per task, our GPT-5.6-luna teacher outperforms frozen Qwen3.5-9B on every harness.
| Harness | Target (%) | Teacher (%) | Teacher − target |
|---|---|---|---|
| Terminus-2 | 67.52 | 77.78 | +10.26 |
| mini-SWE-agent | 60.68 | 69.23 | +8.55 |
| DeepSeek Harness | 54.70 | 66.67 | +11.97 |
| Pi coding-agent | 52.14 | 77.78 | +25.64 |
| DeepAgents | 47.86 | 71.79 | +23.93 |
| Five-harness mean | 56.58 | 72.65 | +16.07 |
Because teacher success lower-bounds the plain ceiling , the gap between teacher and target success gives a category-free lower bound on plain mismatch . Across the five harnesses, this lower bound averages 16.07 percentage points and explains why Pi coding-agent and DeepAgents rank lower when frozen. The teacher reaches 77.78% and 71.79% on them, while frozen Qwen3.5-9B suffers at least 25.64 and 23.93 percentage points of mismatch. Replacing GPT-5.6-luna with Claude Haiku 5.5 on the 3-task probe reproduces the exact five-harness accuracy order at Spearman at high reasoning effort and at medium effort.
Second, for the 13 action categories, the KL chain rule guarantees that category-level KL is a nonnegative lower bound on full output KL at every decision step. We also parse every shell command with a Bash syntax tree parser to route syntax errors and malformed tool calls into dedicated invalid and opaque buckets, while closed-loop container execution accumulates fine-grained execution errors across retry steps. On Terminal-Bench Lite, ranks all five harnesses at with 13 categories and even when merged into 7 coarser categories.
The controlled experiments use a single training seed, while coding-agent estimates rely on only three probe tasks. The reported perfect ranking correlation may be unstable. Computational costs also differ across harnesses.
Response: Thank you for your detailed feedback on training seeds, probe tasks, and compute cost. All three concerns are addressed by new multi-seed runs, an expanded 6-task probe, and trajectory-level token accounting.
First, we repeated the controlled MiniWoB++ experiments across three training seeds on all 576 held-out tasks.
| Comparison on 576 held-out tasks | Metric | Seed 42 | Seed 43 | Seed 44 | Mean ± SD |
|---|---|---|---|---|---|
| Initial H0 to frozen batched clicks H1 | Percentage points | −12.15 | −15.53 | −15.05 | −14.24 ± 1.83 |
| Original to reversed fields under H0 | Percentage points | −65.88 | −59.44 | −64.87 | −63.40 ± 3.46 |
| Initial H0 to adapted batched clicks H1 | Percentage points | +3.12 | +3.42 | +4.69 | +3.74 ± 0.83 |
| H0 continuation to adapted batched clicks H1 | Percentage points | +4.80 | +3.75 | +5.00 | +4.52 ± 0.67 |
| H0 continuation to adapted batched clicks H1 | Plain return change | +0.0530 | +0.0362 | +0.0490 | +0.0461 ± 0.0088 |
| H0 continuation to adapted H5, evaluated under H0 | Percentage points | +10.45 | +10.88 | +11.03 | +10.79 ± 0.30 |
Every finding holds with the same sign and clear magnitude on every training seed, including the frozen drop on H1, the field-reversal drop, the adaptation gain on batched clicks over both initial H0 and continued H0 training, and the 10.79 percentage point gain retained in the weights after H5 adaptation.
Second, for the coding agents, we doubled the Terminal-Bench Lite probe set from 3 tasks to 6 tasks by taking the next three tasks in the recorded manifest across three rollout seeds per harness under the same state-sampling protocol.
| Harness | Accuracy (%) | |||
|---|---|---|---|---|
| Terminus-2 | 0.592 | 0.179 | 0.413 | 64.1 ± 5.1 |
| mini-SWE-agent | 0.778 | 0.689 | 0.089 | 59.8 ± 7.8 |
| DeepSeek Harness | 0.641 | 0.636 | 0.006 | 54.7 ± 3.9 |
| Pi coding-agent | 0.619 | 0.691 | −0.072 | 51.3 ± 2.6 |
| DeepAgents | 0.356 | 1.046 | −0.689 | 47.9 ± 3.9 |
| Spearman | 0.40 | −0.90 | 1.00 |
Just as on 3 probes, either alone at or alone at misorders the five harnesses, whereas their difference recovers the exact accuracy ranking at across all 39 tasks and across the 33 unseen tasks. Doubling from 3 to 6 probe tasks also widens the margin between first-place Terminus-2 and second-place mini-SWE-agent from 0.297 versus 0.296 to 0.413 versus 0.089, ranking Terminus-2 first in 99.98% of within-task trajectory bootstrap resamples.
Third, higher-scoring harnesses also use less or equal inference compute. Across all 117 trajectories per harness on Terminal-Bench Lite, top-ranked Terminus-2 achieves 67.52% accuracy while averaging 117k input tokens, 7.5k output tokens, and 16.8 model calls per trajectory, the lowest token footprint among all five harnesses, whereas the other four harnesses average from 173k to 194k input tokens per trajectory. In MiniWoB++, adapting to batched clicks H1 lowers decisions per task from 4.11 to 3.87 with KL training and from 3.44 to 2.57 without KL training, and evaluating the H5-adapted model back under H0 beats H0 continuation by 10.79 percentage points across three seeds at the exact same H0 inference cost.
We will incorporate key parts of the findings and discussions above into the revised manuscript. We hope our responses address your concerns, and please feel free to let us know if you have any further questions.
Dear reviewer DReX, thank you again for your positive and constructive feedback. Since the discussion period is coming to a close and we have not yet heard back from you, we wanted to kindly follow up. We have carefully revised our manuscript and made several substantial updates in direct response to your comments.
- We verified that our main conclusions hold under a shared reference policy and under zero-KL, including equalized H0 and H5 ceilings.
- We tried to add theoretical bounds for the teacher proxy and action categories, and confirmed consistent rankings across Claude Haiku 5.5, and 7 coarser categories.
- We repeated our controlled experiments across three training seeds, doubled the coding probe to 6 tasks, and added full token and step-count comparisons.
We hope these updates address your concerns, and we look forward to hearing your thoughts. If you still have any questions, we are very happy to discuss them further.
Thank you for your thoughtful review and for recognizing the ceiling-mismatch decomposition as a principled perspective, the value of exactly solvable environments for analyzing optimal policies and readout drift, and the extension to coding agents. Below we answer each point with new experiments.
Response: Thank you for your thoughtful question and your careful H5 example. All of our main conclusions on harness adaptation, frozen interface switches, and observation-field reversal hold both under a shared reference policy and when KL regularization is removed from training and evaluation.
First, at τ>0, the ceiling C reflects both the action space of the harness and the reference probabilities induced by its instructions. Across all 576 held-out tasks with Qwen3.5-2B, replacing H5's reference policy with the shared H0 reference policy equalizes the ceilings of H0 and H5 because they share identical actions, transitions, and rewards, whereas batched clicks H1 retain a 0.0458 ceiling advantage at τ=0 by saving about one decision per task.
Under the shared H0 reference or at τ=0, H5 adds zero ceiling over H0, confirming that extra instructions alone leave the highest achievable task return unchanged. At the same time, evaluating both the H0-continued model and the H5-adapted model under the exact same H0 interface and H0 reference shows a 10.45 percentage point success gain, demonstrating that H5's instructions guide exploration during training and leave a lasting gain in the weights.
Second, we retrained Qwen3.5-2B from scratch and during adaptation with the KL weight set to zero, and evaluated all plain task returns at τ=0 on the same 576 held-out tasks.
With zero KL regularization in both training and evaluation, frozen switching to H1 and reversing observation fields still cause drops of 4.89 and 76.97 percentage points in success. Because initial H0 already solves 98.79% of tasks near the 100% success limit, adapting to batched clicks H1 improves plain discounted task return by 0.0395 over the initial model and 0.0437 over continued H0 training by cutting decisions per task from 3.44 to 2.57, closely tracking H1's 0.0458 ceiling gain at τ=0, while H5 adaptation still leaves a 0.77 percentage point gain in the weights when tested back under H0 near the 100% ceiling.