acceptodds
Under review as a conference paper at ICLR 2027

Your Policy Is Secretly a Harness: The Decision Geometry of Agent Interfaces

11 views

Abstract

Language model agents are trained through a harness of prompts, tools, and control logic. Under KL-regularized control, an optimal policy's log odds between executed actions recover the relative action scores that its harness induces, which determine every optimal decision. In web tasks whose control problems we solve exactly, post-training moves a model's action scores toward its training harness's optimum and makes them specific to that harness. Even reversing the order of the observation fields, which keeps every optimal decision, moves these scores and lowers a trained model's success from 85.0% to 19.2%. Action scores fix decisions but leave out the best return a harness offers, which we call its ceiling. An agent's return equals the ceiling minus its mismatch, the return lost to suboptimal decisions along its own trajectories. Training changes only the mismatch. Adding batched clicks raises the ceiling but lowers a trained model's success, and after equal further training under each harness, the batched-click harness gives the best agent. Among swaps between several harnesses, a frozen comparison rejects seven that raise the ceiling, and after adaptation all seven exceed the original agent's return. Ceiling and mismatch measured before adaptation predict the adaptation gains far better than the frozen return change. With the optimum known, mismatch estimated from sampled actions predicts how harness changes and adaptation move return. For coding agents, teacher-based estimates of ceiling minus mismatch order several harnesses by accuracy. A frozen comparison of harnesses therefore mixes what each harness offers with how well the checkpoint uses it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Harnesses, Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_DReXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how agent harnesses influence learned policies and their transferability across interfaces. It formulates harness-induced decision processes using KL-regularized control and introduces a ceiling–mismatch decomposition to distinguish harness potential from policy adaptation errors. Through controlled MiniWoB++ experiments and coding-agent evaluations, the authors demonstrate that harness changes can significantly degrade frozen-policy performance, while further adaptation can recover and improve performance. The paper also explores mismatch-based estimation for predicting harness performance.

Critical strengths
  1. The ceiling–mismatch decomposition provides an intuitive and principled perspective for understanding performance degradation and recovery under harness changes.

  2. The use of exactly solvable MiniWoB++ environments enables precise analysis of optimal policies, readout drift, and adaptation effects.

  3. The coding-agent experiments extend the discussion beyond controlled settings and suggest potential applications to harness selection and adaptation.

Critical weaknesses
  1. The ceiling depends on the reference policy rather than solely on harness capabilities. For example, H5 achieves a higher ceiling than H0 despite identical execution dynamics, raising concerns about its interpretation as intrinsic harness potential.

  2. The coding-agent experiments replace the optimal policy with a teacher and compress actions into 13 coarse categories. These approximations lack theoretical guarantees and may not preserve the true performance gap.

  3. The controlled experiments use a single training seed, while coding-agent estimates rely on only three probe tasks. The reported perfect ranking correlation may be unstable. Computational costs also differ across harnesses.

Questions for authors

Do the main conclusions hold under a shared reference policy or without KL regularization?

4 comments

Thank you for your thoughtful review and for recognizing the ceiling-mismatch decomposition as a principled perspective, the value of exactly solvable environments for analyzing optimal policies and readout drift, and the extension to coding agents. Below we answer each point with new experiments.

The ceiling depends on the reference policy rather than solely on harness capabilities. For example, H5 achieves a higher ceiling than H0 despite identical execution dynamics, raising concerns about its interpretation as intrinsic harness potential.

Do the main conclusions hold under a shared reference policy or without KL regularization?

Response: Thank you for your thoughtful question and your careful H5 example. All of our main conclusions on harness adaptation, frozen interface switches, and observation-field reversal hold both under a shared reference policy and when KL regularization is removed from training and evaluation.

First, at τ>0\tau > 0, the ceiling CC reflects both the action space of the harness and the reference probabilities induced by its instructions. Across all 576 held-out tasks with Qwen3.5-2B, replacing H5's reference policy with the shared H0 reference policy equalizes the ceilings of H0 and H5 because they share identical actions, transitions, and rewards, whereas batched clicks H1 retain a 0.0458 ceiling advantage at τ=0\tau = 0 by saving about one decision per task.

Comparison on 576 held-out tasksMetricBaselineResultChange
H0 to H5 under each harness's own reference at τ=0.1\tau = 0.1Ceiling CC0.43790.5183+0.0804
H0 to H5 under the shared H0 reference at τ=0.1\tau = 0.1Ceiling CC0.43790.43790.0000
H0 to H5 with no KL penalty at τ=0\tau = 0Ceiling CC0.88460.88460.0000
H0 to batched clicks H1 with no KL penalty at τ=0\tau = 0Ceiling CC0.88460.9304+0.0458
Initial H0 to adapted H5Success (%)85.0793.80+8.73
H0 continuation to adapted H5, both evaluated under H0Success (%)83.3693.81+10.45

Under the shared H0 reference or at τ=0\tau = 0, H5 adds zero ceiling over H0, confirming that extra instructions alone leave the highest achievable task return unchanged. At the same time, evaluating both the H0-continued model and the H5-adapted model under the exact same H0 interface and H0 reference shows a 10.45 percentage point success gain, demonstrating that H5's instructions guide exploration during training and leave a lasting gain in the weights.

Second, we retrained Qwen3.5-2B from scratch and during adaptation with the KL weight set to zero, and evaluated all plain task returns at τ=0\tau = 0 on the same 576 held-out tasks.

Comparison on 576 held-out tasksMetricNo-KL baselineNo-KL resultNo-KL changeKL-trained change
Initial H0 to frozen batched clicks H1Success (%)98.7993.90−4.89−12.15
Original to reversed fields under H0Success (%)98.7921.82−76.97−65.88
Initial H0 to adapted batched clicks H1Plain task return0.87360.9131+0.0395+0.0367
H0 continuation to adapted batched clicks H1Plain task return0.86940.9131+0.0437+0.0530
H0 continuation to adapted H5, both evaluated under H0Success (%)98.2499.01+0.77+10.45

With zero KL regularization in both training and evaluation, frozen switching to H1 and reversing observation fields still cause drops of 4.89 and 76.97 percentage points in success. Because initial H0 already solves 98.79% of tasks near the 100% success limit, adapting to batched clicks H1 improves plain discounted task return by 0.0395 over the initial model and 0.0437 over continued H0 training by cutting decisions per task from 3.44 to 2.57, closely tracking H1's 0.0458 ceiling gain at τ=0\tau = 0, while H5 adaptation still leaves a 0.77 percentage point gain in the weights when tested back under H0 near the 100% ceiling.

The coding-agent experiments replace the optimal policy with a teacher and compress actions into 13 coarse categories. These approximations lack theoretical guarantees and may not preserve the true performance gap.

Response: Thank you for your important point about the teacher approximation and the action categories. Both approximations now carry explicit theoretical bounds and preserve true harness rankings across teachers, category granularities, and benchmarks.

First, for the teacher approximation, we prove two guarantees. Normal task success under any teacher is a valid lower bound on the plain ceiling CC at τ=0\tau = 0 with no optimality assumption or action categories, and the proxy mismatch bias is bounded linearly by the teacher's action-score error relative to the optimum, so harness rankings stay unchanged whenever the true return gap exceeds the difference in proxy bias. Across all 39 Terminal-Bench Lite tasks with 3 runs per task, our GPT-5.6-luna teacher outperforms frozen Qwen3.5-9B on every harness.

HarnessTarget (%)Teacher (%)Teacher − target
Terminus-267.5277.78+10.26
mini-SWE-agent60.6869.23+8.55
DeepSeek Harness54.7066.67+11.97
Pi coding-agent52.1477.78+25.64
DeepAgents47.8671.79+23.93
Five-harness mean56.5872.65+16.07

Because teacher success lower-bounds the plain ceiling CC, the gap between teacher and target success gives a category-free lower bound on plain mismatch M=C−JM = C - J. Across the five harnesses, this lower bound averages 16.07 percentage points and explains why Pi coding-agent and DeepAgents rank lower when frozen. The teacher reaches 77.78% and 71.79% on them, while frozen Qwen3.5-9B suffers at least 25.64 and 23.93 percentage points of mismatch. Replacing GPT-5.6-luna with Claude Haiku 5.5 on the 3-task probe reproduces the exact five-harness accuracy order at Spearman ρ=1.00\rho = 1.00 at high reasoning effort and ρ=0.90\rho = 0.90 at medium effort.

Second, for the 13 action categories, the KL chain rule guarantees that category-level KL is a nonnegative lower bound on full output KL at every decision step. We also parse every shell command with a Bash syntax tree parser to route syntax errors and malformed tool calls into dedicated invalid and opaque buckets, while closed-loop container execution accumulates fine-grained execution errors across retry steps. On Terminal-Bench Lite, C~−M~\widetilde C - \widetilde M ranks all five harnesses at ρ=1.00\rho = 1.00 with 13 categories and ρ=0.90\rho = 0.90 even when merged into 7 coarser categories.

The controlled experiments use a single training seed, while coding-agent estimates rely on only three probe tasks. The reported perfect ranking correlation may be unstable. Computational costs also differ across harnesses.

Response: Thank you for your detailed feedback on training seeds, probe tasks, and compute cost. All three concerns are addressed by new multi-seed runs, an expanded 6-task probe, and trajectory-level token accounting.

First, we repeated the controlled MiniWoB++ experiments across three training seeds on all 576 held-out tasks.

Comparison on 576 held-out tasksMetricSeed 42Seed 43Seed 44Mean ± SD
Initial H0 to frozen batched clicks H1Percentage points−12.15−15.53−15.05−14.24 ± 1.83
Original to reversed fields under H0Percentage points−65.88−59.44−64.87−63.40 ± 3.46
Initial H0 to adapted batched clicks H1Percentage points+3.12+3.42+4.69+3.74 ± 0.83
H0 continuation to adapted batched clicks H1Percentage points+4.80+3.75+5.00+4.52 ± 0.67
H0 continuation to adapted batched clicks H1Plain return change+0.0530+0.0362+0.0490+0.0461 ± 0.0088
H0 continuation to adapted H5, evaluated under H0Percentage points+10.45+10.88+11.03+10.79 ± 0.30

Every finding holds with the same sign and clear magnitude on every training seed, including the frozen drop on H1, the field-reversal drop, the adaptation gain on batched clicks over both initial H0 and continued H0 training, and the 10.79 percentage point gain retained in the weights after H5 adaptation.

Second, for the coding agents, we doubled the Terminal-Bench Lite probe set from 3 tasks to 6 tasks by taking the next three tasks in the recorded manifest across three rollout seeds per harness under the same state-sampling protocol.

HarnessC~\widetilde CM~\widetilde MC~−M~\widetilde C - \widetilde MAccuracy (%)
Terminus-20.5920.1790.41364.1 ± 5.1
mini-SWE-agent0.7780.6890.08959.8 ± 7.8
DeepSeek Harness0.6410.6360.00654.7 ± 3.9
Pi coding-agent0.6190.691−0.07251.3 ± 2.6
DeepAgents0.3561.046−0.68947.9 ± 3.9
Spearman ρ\rho0.40−0.901.00

Just as on 3 probes, either C~\widetilde C alone at ρ=0.40\rho = 0.40 or M~\widetilde M alone at ρ=−0.90\rho = -0.90 misorders the five harnesses, whereas their difference C~−M~\widetilde C - \widetilde M recovers the exact accuracy ranking at ρ=1.00\rho = 1.00 across all 39 tasks and across the 33 unseen tasks. Doubling from 3 to 6 probe tasks also widens the margin between first-place Terminus-2 and second-place mini-SWE-agent from 0.297 versus 0.296 to 0.413 versus 0.089, ranking Terminus-2 first in 99.98% of within-task trajectory bootstrap resamples.

Third, higher-scoring harnesses also use less or equal inference compute. Across all 117 trajectories per harness on Terminal-Bench Lite, top-ranked Terminus-2 achieves 67.52% accuracy while averaging 117k input tokens, 7.5k output tokens, and 16.8 model calls per trajectory, the lowest token footprint among all five harnesses, whereas the other four harnesses average from 173k to 194k input tokens per trajectory. In MiniWoB++, adapting to batched clicks H1 lowers decisions per task from 4.11 to 3.87 with KL training and from 3.44 to 2.57 without KL training, and evaluating the H5-adapted model back under H0 beats H0 continuation by 10.79 percentage points across three seeds at the exact same H0 inference cost.

We will incorporate key parts of the findings and discussions above into the revised manuscript. We hope our responses address your concerns, and please feel free to let us know if you have any further questions.

Dear reviewer DReX, thank you again for your positive and constructive feedback. Since the discussion period is coming to a close and we have not yet heard back from you, we wanted to kindly follow up. We have carefully revised our manuscript and made several substantial updates in direct response to your comments.

  1. We verified that our main conclusions hold under a shared reference policy and under zero-KL, including equalized H0 and H5 ceilings.
  2. We tried to add theoretical bounds for the teacher proxy and action categories, and confirmed consistent rankings across Claude Haiku 5.5, and 7 coarser categories.
  3. We repeated our controlled experiments across three training seeds, doubled the coding probe to 6 tasks, and added full token and step-count comparisons.

We hope these updates address your concerns, and we look forward to hearing your thoughts. If you still have any questions, we are very happy to discuss them further.

Reviewer_cqDROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This work investigates how agent policies interact with their deployment harnesses. Under KL-regularized control, the authors prove that an optimal policy fundamentally learns the relative action scores induced by its specific harness, causing post-training to embed harness action preferences into LLM weights. Consequently, transferring a post-trained policy to a new harness often triggers severe performance drops due to target shift, training residual, and readout drift. Even when simple observation field reordering leaves every optimal decision unchanged. To address this, this work decomposes deployed regularized return into the harness ceiling (maximum achievable return) minus policy mismatch (trajectory-accumulated KL loss). It demonstrates that frozen evaluations misleadingly reject candidate harnesses with higher ceilings due to temporary mismatch, whereas subsequent adaptation eliminates mismatch and allows the new harness to surpass the original agent.

Critical strengths
  1. Establishes a formal mathematical framework connecting regularized MDP theory, soft Bellman optimality, and Bregman divergences to quantify decision errors across agent interfaces.
  2. The J=C−MJ = C - M formulation neatly decouples what a harness offers (ceiling CC) from how well a checkpoint utilizes it (mismatch MM), exposing why frozen comparisons create misleading evaluation traps.
  3. Combines exact dynamic programming on MiniWoB++ graphs (Qwen3.5-2B) with practical coding benchmarks (Terminal-Bench Lite with Qwen3.5-9B and GPT-5.6-luna as a teacher).
Critical weaknesses
  1. The theoretical identities rely strictly on KL-regularized MDP formulations, which may not fully hold for post-training algorithms that use hard clipping or alternative objectives.
  2. Accurate mismatch estimation hatMhat{M} without exact task graphs requires tens of thousands of model queries, limiting its practicality for rapid harness search.
  3. In complex, open-ended domains like software engineering, CC and MM rely on an external teacher (GPT-5.6-luna), inheriting teacher proxy biases if the teacher makes suboptimal choices.
  4. Categorizing free-form command outputs into 13 macro categories on Terminal-Bench Lite risks smoothing over fine-grained execution bugs or syntax-level misalignments.
  5. Exact DP proofs are evaluated on small-scale MiniWoB++ web tasks with short decision horizons and lightweight models (Qwen3.5-2B), leaving open whether frontier models show equal sensitivity to readout drift.
Questions for authors
  1. How well does the J=C−MJ = C - M framework generalize to post-training schemes without explicit KL penalty terms (e.g., GRPO or unregularized policy gradient)?
  2. Can readout drift η12\eta_{12} be effectively mitigated at test time via inference-time prompt re-alignment without full parameter adaptation?
  3. How sensitive are the coding-agent harness rankings (C~−M~\tilde{C} - \tilde{M}) to the choice of the teacher model?
  4. How can the sample complexity of mismatch estimation be reduced to make real-time harness optimization computationally feasible?
5 comments

Thank you for your thoughtful review and for recognizing the formal framework, the way J=C−MJ = C - M separates what a harness offers from how well a checkpoint uses it, and the combination of exact dynamic programming with production coding benchmarks. Below we answer each point with new experiments.

The theoretical identities rely strictly on KL-regularized MDP formulations, which may not fully hold for post-training algorithms that use hard clipping or alternative objectives.

How well does the J=C−MJ = C - M framework generalize to post-training schemes without explicit KL penalty terms (e.g., GRPO or unregularized policy gradient)?

Response: Thank you for your thoughtful question. The decomposition J=C−MJ = C - M is an evaluation identity that holds for any deployed policy and any τ≥0\tau \ge 0, so it applies directly to GRPO, PPO with hard clipping, and unregularized policy gradient, and reduces at τ=0\tau = 0 to the standard performance difference lemma on plain task return.

Only exact score identification assumes a KL optimum, where action log-probabilities shift relative to the reference model in proportion to Q∗Q^*, and this linear fit still explains 62% to 68% of variance in our PPO-Clip runs with hard clipping. We also retrained Qwen3.5-2B with unregularized policy gradient at zero KL weight and evaluated on all 576 held-out tasks at τ=0\tau = 0, where batched clicks H1 raise plain return by cutting decisions per task from 3.44 to 2.57.

Comparison on 576 held-out tasksMetricNo-KL baselineNo-KL resultNo-KL changeKL-trained change
Initial H0 to frozen batched clicks H1Success (%)98.7993.90−4.89−12.15
Original to reversed fields under H0Success (%)98.7921.82−76.97−65.88
Initial H0 to adapted batched clicks H1Plain task return0.87360.9131+0.0395+0.0367
H0 continuation to adapted batched clicks H1Plain task return0.86940.9131+0.0437+0.0530

Even with no KL penalty in training, switching frozen to H1 or reversing field order causes clear drops in success, while adapting to H1 still outperforms both the initial H0 model and continued H0 training in plain task return.

Accurate mismatch estimation M^\hat{M} without exact task graphs requires tens of thousands of model queries, limiting its practicality for rapid harness search.

How can the sample complexity of mismatch estimation be reduced to make real-time harness optimization computationally feasible?

Response: Thank you for your practical question. Dense state-level sampling M^\widehat M is an offline diagnostic for pinpointing which saved states lose return, avoiding repeated 25 to 35 minute live coding runs. For rapid harness search without exact task graphs, four lightweight methods cut query cost sharply.

  1. Step-level probes on a few trajectories rank full benchmarks. Probing just 3 Terminal-Bench Lite tasks ranks all five coding harnesses across the full 39-task benchmark at ρ=1.00\rho = 1.00 with zero target runs on the other 36 tasks.
  2. Normal teacher rollouts bound the ceiling at low cost. Running a teacher for 3 runs per task needs no action categories or state resampling and gives a guaranteed lower bound on CC, exceeding frozen Qwen3.5-9B on every coding harness by 16.1 percentage points on average.
  3. The pre-training threshold TT screens candidate harnesses with zero RL training. Equal to the new ceiling minus the initial return, TT orders post-training return gains across all nine harness swaps rejected by frozen evaluation at ρ=0.90\rho = 0.90 while the frozen return change reaches only ρ=0.27\rho = 0.27, and ceiling gain ΔC=C2−C1\Delta C = C_2 - C_1 gets the sign of all nine gains right.
  4. Full episode rollouts measure overall mismatch without state resampling. When the ceiling CC is known, estimating return JJ from normal rollouts and computing M=C−JM = C - J tells whether a low frozen score comes from a low ceiling or from large mismatch that adaptation can remove. To keep return-difference RMSE within 0.0025 when comparing two setups, this cuts model calls per task from about 85,000 for state sampling down to 16 when action probabilities are available, or 256 when only final task outcomes are available.

These four methods already make screening candidate harnesses practical at low cost, and we agree that cutting queries further by estimating mismatch from a single trajectory pass is a promising next step for real-time optimization.

In complex, open-ended domains like software engineering, CC and MM rely on an external teacher (GPT-5.6-luna), inheriting teacher proxy biases if the teacher makes suboptimal choices.

How sensitive are the coding-agent harness rankings (C~−M~\tilde{C} - \tilde{M}) to the choice of the teacher model?

Response: Thank you for your thoughtful question about teacher bias and teacher choice. First, our framework provides two safeguards when a teacher is suboptimal. Normal task success under any teacher is a guaranteed lower bound on the plain ceiling CC with no optimality assumption or action categories, and across all 39 Terminal-Bench Lite tasks the teacher outperforms frozen Qwen3.5-9B on every harness by 16.1 percentage points on average. In addition, the proxy mismatch bias is bounded by the teacher's action-score error relative to the optimum, so harness rankings remain unchanged whenever the true gap exceeds the difference in proxy bias.

Second, we repeated the 3-task probe comparison with Claude Haiku 5.5 at high and medium reasoning effort. Below we compare the paper's GPT-5.6-luna scores with Haiku 5.5 under the same probe protocol, where Haiku's M~\widetilde M combines the action log-odds penalty with the excess rate of hitting the context limit, alongside 3-run accuracy across all 39 Terminal-Bench Lite tasks.

HarnessGPT-5.6-lunaHaiku 5.5 highHaiku 5.5 mediumAccuracy (%)
Terminus-20.2970.8380.84564.1 ± 5.1
mini-SWE-agent0.2960.6210.64259.8 ± 7.8
DeepSeek Harness0.2080.3710.33154.7 ± 3.9
Pi coding-agent0.1770.3580.39851.3 ± 2.6
DeepAgents−0.1680.2760.19447.9 ± 3.9
Spearman ρ\rho1.001.000.90

Across two different model families, all three teacher settings rank Terminus-2 first, mini-SWE-agent second, and DeepAgents last. Haiku 5.5 high reproduces the exact five-harness order of GPT-5.6-luna and full-benchmark accuracy at ρ=1.00\rho = 1.00, and Haiku 5.5 medium agrees on nine of ten pairs at ρ=0.90\rho = 0.90, with only DeepSeek Harness and Pi exchanging places within one standard deviation.

Categorizing free-form command outputs into 13 macro categories on Terminal-Bench Lite risks smoothing over fine-grained execution bugs or syntax-level misalignments.

Response: Thank you for your precise point about syntax and fine-grained execution errors. First, we parse every shell command with a Bash syntax tree parser and route parse failures, malformed tool calls, and missing commands into dedicated invalid and opaque buckets, so syntax misalignments are penalized directly. For valid actions, the 13 semantic categories align different native interfaces such as file-edit tools and raw shell commands across harnesses, and by the KL chain rule they give a nonnegative lower bound on full output KL at each step. Moreover, in live multi-step containers, a fine-grained execution bug inside a valid action changes subsequent container output and triggers extra debugging and retry steps, which M~\widetilde M captures by summing mismatch along the full trajectory, ranking all five coding harnesses at ρ=1.00\rho = 1.00 with 13 categories and ρ=0.90\rho = 0.90 even when merged into 7 coarser categories.

Second, when harnesses share an executable action space, our framework operates directly on exact commands with no macro grouping. On two ALFWorld household benchmarks, 271 validation tasks and 180 sampled tasks, we evaluated five harnesses with frozen Qwen3.5-9B by projecting onto exact executable commands with full object and receptacle arguments, where M~\widetilde M combines exact-command KL with the excess rate of hitting the step limit.

HarnessExact-command C~−M~\widetilde C - \widetilde MVal 271 (%)Sampled 180 (%)
Pi0.305574.271.6
smolagents0.304473.267.9
DeepAgents0.303669.966.6
DeepSeek Harness0.264272.366.0
Codex0.223659.857.4
Spearman ρ\rho0.901.00

Here C~\widetilde C and M~\widetilde M are estimated solely from state-level action probes on the 271 validation trajectories, with zero probes on the 180 sampled tasks and no target task outcomes. Exact-command C~−M~\widetilde C - \widetilde M predicts full-benchmark accuracy rankings at ρ=0.90\rho = 0.90 on the 271 validation tasks and ρ=1.00\rho = 1.00 on the unprobed 180 sampled tasks, confirming that the decomposition works cleanly at exact command granularity as well.

Exact DP proofs are evaluated on small-scale MiniWoB++ web tasks with short decision horizons and lightweight models (Qwen3.5-2B), leaving open whether frontier models show equal sensitivity to readout drift.

Response: Thank you for raising this question about readout sensitivity in larger and frontier models. Exact dynamic programming indeed requires finite task graphs such as MiniWoB++, so to test whether larger and frontier models show similar sensitivity, we added controlled scaling experiments on MiniWoB++ alongside frontier evaluations on long-horizon coding tasks.

First, extending Table 1 across all 576 held-out MiniWoB++ tasks under H0, we evaluated 2B, 9B, and 27B models while holding every optimal action, transition, and reward fixed.

Model on 576 held-out tasksOriginal (%)Reversed (%)Original − reversed
Qwen3.5-2B, frozen11.997.794.20
Qwen3.5-2B, post-trained on H085.0519.1765.88
Qwen3.5-9B, frozen43.4326.2317.20
Qwen3.5-9B, post-trained on H076.6241.3635.26
Qwen3.8-27B, frozen58.2326.9731.26

Before task post-training, reversing JSON field order already lowers frozen Qwen3.5-9B by 17.20 percentage points and frozen Qwen3.8-27B by 31.26 percentage points. Post-training on H0 widens this gap at both scales, from 4.20 to 65.88 percentage points on Qwen3.5-2B and from 17.20 to 35.26 percentage points on Qwen3.5-9B. Scaling from 2B to 9B moderates the post-trained drop from 65.88 to 35.26 percentage points, while still leaving a 35.26 percentage-point loss from field order alone.

Second, on all 39 long-horizon Terminal-Bench Lite coding tasks, frontier GPT-5.6-luna also varies substantially across five coding harnesses.

Coding harnessGPT-5.6-luna (%)
Terminus-277.78
Pi coding-agent77.78
DeepAgents71.79
mini-SWE-agent69.23
DeepSeek Harness66.67

With the frontier model and task set fixed, changing the harness alone shifts accuracy by 11.11 percentage points, consistent with recent findings that frontier models remain sensitive to input order and harness choices [1, 2, 3].

[1] Guan et al., The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs, KDD 2025.

[2] Kapoor et al., Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation, ICLR 2026.

[3] Vats and Golev, The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation, arXiv 2026.

Can readout drift η12\eta_{12} be effectively mitigated at test time via inference-time prompt re-alignment without full parameter adaptation?

Response: Thank you for your valuable question about test-time mitigation. We tested two simple inference-time prompt adjustments, adding a format note or two worked examples, and found that these direct prompt edits do not recover readout drift and lower task success further, because post-training binds the checkpoint closely to its training prompt and format. Without full parameter retraining, remapping the interface back to the original training format restores full success at zero weight cost, and a few dozen lightweight updates recover most of the drop. Across all 576 held-out tasks in Table 1, we evaluated the H0-trained checkpoint with zero weight updates.

Test-time settingReversed fields (%)Shifted codes (%)
Original training format85.085.0
New format with unchanged prompt19.257.1
Prompt re-alignment with format note15.343.5
Prompt re-alignment with two examples13.440.6
Remap interface back to original format85.085.0

Both simple prompt edits lower success further from 19.2% to 15.3% and 13.4% on reversed fields and from 57.1% to 43.5% and 40.6% on shifted codes, while interface remapping restores 85.0% with no weight update. When interface remapping is unavailable, lightweight adaptation on reversed fields recovers 53% of the drop to 54.4% in 32 updates and 67% to 63.4% in 128 updates. We believe that guiding format changes without disturbing post-trained weights requires more carefully designed prompt methods, and we view this as valuable future work.

We will incorporate key parts of the findings and discussions above into the revised manuscript. We hope our responses address your concerns, and please feel free to let us know if you have any further questions.

Reviewer_JrU7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper models agent harnesses as KL-regularized control problems and decomposes deployed return into the harness’s optimal return minus the policy’s KL-based mismatch. On compiled MiniWoB++ tasks with Qwen3.5-2B, it shows harness-specific preferences after training, large drops from optimum-preserving presentation changes, and gains after adapting to harnesses with higher ceilings, plus teacher-based estimates ranking five coding-agent harnesses.

Critical strengths

None

Critical weaknesses
  • The theory is standard KL-regularized RL so there is nothing inherently new.
  • The studies problem is basically distribution shift across environments, which is close to cross-embodied deployment of policies which paper does not discuss.
  • Harness vs Environment. The inferface abstraction mixes presentation with execution. One could treat the harnesses as seperate envs, and main findings are expected results of RL training on an env and the evaluating on another one.
  • Readout study is probably overfitting or capability bottleneck, any capable model will be able to change the order of json.
  • KL is the main source of results, but many modern post-training recipes set KL regularization to 0.
  • Adaptation result is again expected.
  • Lastly, but most importantly paper is written very poorly. Abstract reads like an summary of key results which is not good for a reader who is reading it for the first time. Introduction is also like an extended abstract with no real motivation. There are many fancy words for unnecessary reasons. It needs significant revision for human understanding.
Questions for authors

None

4 comments

Thank you for your comments. Below we address each point with targeted experiments, including cross-training on reversed observation fields and scaling up to 27B to test model capacity, retraining from scratch with the KL weight set to zero, and predicting adaptation gains before any training on the target harness. We also rewrote the abstract, restructured the introduction around concrete practical motivation, and simplified phrasing for clarity.

The theory is standard KL-regularized RL so there is nothing inherently new.

Response: Thank you for raising this question. Within a single fixed Markov decision process, our soft Bellman and performance-gap identities build directly on established regularized reinforcement learning, and our contribution lies in establishing a theoretical foundation for agent harnesses. To the best of our knowledge, prior work on agent harnesses has been largely empirical, leaving open a formal theory of how software interfaces shape post-trained policies and harness evaluations. By bringing regularized control onto shared semantic actions across context encoding cc, response decoding dd, and execution ee, our framework derives and experimentally verifies three cross-harness insights that empirical trials alone do not explain.

  1. Post-training writes harness preferences into weights. Optimal action log odds identify only the training harness's relative action scores, independent of transitions or absolute value, explaining why adapting under richer instructions leaves a 10.45 percentage point gain in the weights even after reverting to the baseline H0 interface.
  2. Cross-harness transfer fails through three distinct error terms. On aligned actions, deployed score error splits into target shift, training residual, and readout drift, isolating pure format drift when optimal decisions stay fixed and explaining why one interface edit can help a general model while hurting a post-trained model.
  3. Adaptation overturns frozen comparisons when the ceiling rises. In J=C−MJ = C - M, weight updates reduce only mismatch MM while harness revisions move both CC and MM. This yields pre-training threshold TT and ceiling gain ΔC=T−M1\Delta C = T - M_1, ordering adaptation gains across nine rejected swaps at ρ=0.90\rho = 0.90 with ΔC\Delta C getting all nine signs right, and teacher proxy C~−M~\widetilde C - \widetilde M, ranking five coding harnesses on Terminal-Bench Lite at ρ=1.00\rho = 1.00 without retraining.

The studies problem is basically distribution shift across environments, which is close to cross-embodied deployment of policies which paper does not discuss.

Response: Thank you for pointing to distribution shift and cross-embodiment transfer. We added a Related Work discussion covering cross-embodied policy transfer [1, 2], observational overfitting [3], and MDP homomorphisms [4]. Beyond standard distribution shift across fixed embodiments, software harnesses introduce two structural properties.

  1. Harnesses share one underlying environment with exact action alignment. They wrap the same task environment and map token outputs onto shared operations, letting us hold transitions, rewards, and all optimal decisions fixed to isolate post-training readout drift and compare ceilings CC on one reward scale.
  2. Harnesses are editable interfaces selected before training. Because candidate harnesses are mutable software routinely ranked with a frozen checkpoint, distribution shift explains why unfamiliar interfaces drop when frozen, whereas separating CC from MM distinguishes worse revisions from higher-ceiling revisions before any target-harness training.

[1] Gupta et al., Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning, ICLR 2017.

[2] Zakka et al., XIRL: Cross-embodiment Inverse Reinforcement Learning, CoRL 2021.

[3] Song et al., Observational Overfitting in Reinforcement Learning, ICLR 2020.

[4] van der Pol et al., MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning, NeurIPS 2020.

Harness vs Environment. The inferface abstraction mixes presentation with execution. One could treat the harnesses as seperate envs, and main findings are expected results of RL training on an env and the evaluating on another one.

Response: Thank you for raising this point. Separate RL environments are fixed external problems with distinct tasks and unaligned reward scales, whereas harnesses are editable layers of prompts, parsers, tools, and control flow wrapping one shared task and success criterion, following standard practice in harness evolution [1, 2] and co-evolution [3, 4]. Within this shared task, our interface definition separates presentation through context encoding cc and response decoding dd, isolated in our readout study, from execution ee, which governs transitions and raises ceiling CC in batched clicks H1. Treating harnesses as unrelated environments only predicts that frozen transfer drops, whereas sharing one task scale predicts before training which rejected harness will surpass the original setup after adaptation, holding on seven swaps that raise CC and failing on two that lower CC.

[1] Hu et al., Automated Design of Agentic Systems, ICLR 2025.

[2] Zhang et al., Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents, ICLR 2026.

[3] Chen et al., Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents, arXiv 2026.

[4] Kim et al., WHALE: A Simple Recipe for Joint Harness-Weight Optimization, arXiv 2026.

Readout study is probably overfitting or capability bottleneck, any capable model will be able to change the order of json.

Response: Thank you for raising this question about the readout study. Recent studies show that even strong frontier models change performance when input order or agent harnesses change [1, 2, 3], and that the same interface change can help a general model while hurting a fine-tuned model [4]. Our framework explains why and tests it cleanly. Post-training writes the training harness's action preferences into the model weights, and the model expresses them through the prompt format and output parser. Holding every optimal action, transition, and reward fixed isolates prompt formatting alone. In Table 1, reversing JSON field order lowers post-trained Qwen3.5-2B from 85.0% to 19.2%, whereas rendering the menu as a list reaches 86.0%.

First, across all 576 held-out tasks in Table 1, test-time format notes or worked examples leave reversed-field success at 15.26% and 13.37%. Training Qwen3.5-2B from scratch on reversed fields and on both orders tests 2B capacity directly on the same 576 tasks.

2B training formatOriginal (%)Reversed (%)
Original fields85.0519.17
Reversed fields26.6984.07
Both field orders81.4782.19

Reversed-field training reaches 84.07% while original fields drop to 26.69%, and training on both orders together reaches 81.47% and 82.19%, showing that 2B has full capacity to handle both JSON orders.

Second, larger models on the same 576 tasks show the same field-order sensitivity, and H0 post-training widens the gap further.

Larger model on 576 held-out tasksOriginal (%)Reversed (%)Original − reversed
Qwen3.5-9B, frozen43.4326.2317.20
Qwen3.5-9B, post-trained on H076.6241.3635.26
Qwen3.8-27B, frozen58.2326.9731.26

Before task post-training, reversing JSON fields already lowers frozen Qwen3.5-9B by 17.20 percentage points and frozen Qwen3.8-27B by 31.26 percentage points, and post-training Qwen3.5-9B on H0 widens this gap to 35.26 percentage points. Across all 39 Terminal-Bench Lite tasks, frontier GPT-5.6-luna also varies across five coding harnesses.

Coding harnessGPT-5.6-luna (%)
Terminus-277.78
Pi coding-agent77.78
DeepAgents71.79
mini-SWE-agent69.23
DeepSeek Harness66.67

With the frontier model and task set fixed, changing the harness alone shifts accuracy by 11.11 percentage points.

[1] Guan et al., The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs, KDD 2025.

[2] Kapoor et al., Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation, ICLR 2026.

[3] Vats and Golev, The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation, arXiv 2026.

[4] Mu et al., GUI-360: A Comprehensive Dataset and Benchmark for Computer-Using Agents, arXiv 2025.

KL is the main source of results, but many modern post-training recipes set KL regularization to 0.

Response: Thank you for raising this question about zero-KL post-training. The decomposition J=C−MJ = C - M and the adaptation threshold hold for any deployed policy and any KL weight τ≥0\tau \ge 0, regardless of how the model was trained. Setting τ=0\tau = 0 removes the KL term completely, so JJ is plain expected task return, CC is the highest return the harness allows, MM is the return lost to suboptimal actions, and J=C−MJ = C - M reduces to the standard performance difference lemma. Only recovering relative action scores from action log-probabilities uses a positive KL weight. To test zero-KL training directly, we retrained Qwen3.5-2B from scratch and during adaptation with the KL weight set to zero, and evaluated on all 576 held-out tasks with no KL term.

Comparison on 576 held-out tasksMetricNo-KL baselineNo-KL resultNo-KL changeKL-trained change
Initial H0 to frozen batched clicks H1Success (%)98.7993.90−4.89−12.15
Original to reversed fields under H0Success (%)98.7921.82−76.97−65.88
Initial H0 to adapted batched clicks H1Plain task return0.87360.9131+0.0395+0.0367
H0 continuation to adapted batched clicks H1Plain task return0.86940.9131+0.0437+0.0530

With zero KL penalty in training and evaluation, the model still specializes strongly to H0. Switching frozen to H1 drops task success by 4.89 percentage points, and reversing JSON field order drops success by 76.97 percentage points, an even steeper drop than under KL training. Because initial H0 already solves 98.79% of tasks without KL, adapting to batched clicks H1 improves plain task return by 0.0395 over initial H0 and 0.0437 over continued H0 training by cutting decisions per task from 3.44 to 2.57, matching the KL-trained results.

Adaptation result is again expected.

Response: Thank you for raising this point. That fine-tuning improves over zero-shot transfer in single-environment RL is well known and unrelated to our contribution. Because harness benchmarks and search methods discard candidates from frozen scores alone, our contribution is predicting before any target training which rejected harnesses will surpass the original trained system after adaptation and which will still fail. Across 22 swaps rejected when frozen, our threshold T=C2−J1T = C_2 - J_1 rules out 13 swaps with T≤0T \le 0 where no policy can beat J1J_1. Across the remaining nine swaps with T>0T > 0, adapting beats the original setup on the seven that raise CC, still loses return on the two that lower CC, and beats equal-budget H0 continuation on H1 and H5.

Pre-training predictor across 9 rejected swapsSpearman ρ\rhoCorrect signs
Frozen return change0.272 of 9
Threshold TT0.907 of 9
Residual-adjusted threshold T−M1=ΔCT - M_1 = \Delta C0.909 of 9

Subtracting the initial checkpoint's mismatch M1M_1 from TT gives T−M1=ΔCT - M_1 = \Delta C, which needs zero candidate training, gets all nine gain signs right, and orders final adaptation gains at ρ=0.90\rho = 0.90 versus 0.270.27 for frozen return change.

Lastly, but most importantly paper is written very poorly. Abstract reads like an summary of key results which is not good for a reader who is reading it for the first time. Introduction is also like an extended abstract with no real motivation. There are many fancy words for unnecessary reasons. It needs significant revision for human understanding.

Response: Thank you for your feedback on the writing. We revised the abstract, introduction, and phrasing for clarity so the paper is easy to follow on a first read.

  1. We rewrote the abstract so a first-time reader first sees what an agent harness is and why scoring candidate harnesses with an already trained model can be misleading, followed by our central question, the return decomposition, and the main results.
  2. We restructured the introduction to open with concrete practical motivation from automated harness design, walk through two plain-language examples of field-order reversal and batched clicks, and only then introduce the formal setup.

Please see the abstract and Section 1 in the revised manuscript for the full text, with the modified parts highlighted in blue.

We will also incorporate selected key findings and discussions from above into the updated manuscript. We hope our responses and revisions address your concerns, and please feel free to let us know if you have any further questions.

Dear Reviewer JrU7, thank you again for your time and feedback. Since the discussion period is coming to a close and we have not yet heard back from you, we wanted to kindly follow up. We have carefully revised our manuscript and made several substantial updates in direct response to your comments.

  1. We explained why harnesses (prompts, parsers, and tools around one shared task) differ from separate RL environments, and added related work on cross-embodied policy transfer.
  2. Consistent with recent studies, we showed with models up to Qwen3.8-27B and GPT-5.6-luna that strong models still drop when JSON order or the harness changes, and trained the 2B model on both orders to show it can handle both.
  3. We retrained with KL regularization set to zero to confirm that our findings still hold, and explained how our method predicts before fine-tuning which new harnesses will beat the original trained setup.
  4. We rewrote the abstract and restructured the introduction in plain language so the motivation is clear on a first read.

We hope these updates address your concerns, and we look forward to hearing your thoughts. If you still have any questions, we are very happy to discuss them further.

Reviewer_NwYAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper offers a compelling conceptual contribution by challenging the conventional practice of comparing agent harnesses using frozen checkpoints. Its ceiling–mismatch decomposition provides a useful framework for separating harness capability from model–harness compatibility. The findings have potentially significant implications for agent benchmarking and system design. Further validation across broader settings and more practical estimation methods would strengthen its generality and applicability.

Critical strengths

The paper offers a compelling conceptual contribution by challenging the conventional practice of comparing agent harnesses using frozen checkpoints. Its ceiling–mismatch decomposition provides a useful framework for separating harness capability from model–harness compatibility.

Critical weaknesses

Further validation across broader settings and more practical estimation methods would strengthen its generality and applicability.

Questions for authors

The paper offers a compelling conceptual contribution by challenging the conventional practice of comparing agent harnesses using frozen checkpoints. a few questions

  1. Extending the analysis to more diverse agent architectures, domains, and realistic tasks beyond web interaction and coding will be more helpful.
  2. Clarify how well the KL-regularized control formulation captures practical agent training and inference, especially under stochastic environments and approximate optimization.
3 comments

Thank you for your thoughtful review and for recognizing that the paper challenges the common practice of comparing harnesses with frozen checkpoints, and that the ceiling-mismatch decomposition separates harness capability from model-harness compatibility. Below we answer each point and report the new experiments.

Further validation across broader settings and more practical estimation methods would strengthen its generality and applicability.

Extending the analysis to more diverse agent architectures, domains, and realistic tasks beyond web interaction and coding will be more helpful.

Response: Thank you for your valuable suggestions. On two ALFWorld household benchmarks across five agent harnesses, C~−M~\widetilde C - \widetilde M predicts accuracy rankings at ρ=0.90\rho = 0.90 and ρ=1.00\rho = 1.00, and lightweight estimators make each diagnostic step practical at low budget.

First, we evaluated five harnesses on two ALFWorld benchmarks, consisting of 271 validation tasks and 180 tasks randomly sampled from the training split at 30 per task type. Together with Terminus-2 and mini-SWE-agent from our coding study, our experiments cover seven harnesses across three domains. With frozen Qwen3.5-9B and the teacher, where M~\widetilde M combines action KL with the excess rate of hitting the step limit, we obtain the results below.

HarnessC~\widetilde CM~\widetilde MC~−M~\widetilde C - \widetilde MVal 271 (%)Sampled 180 (%)
Pi0.81780.51230.305574.271.6
smolagents0.82790.52350.304473.267.9
DeepAgents0.86180.55820.303669.966.6
DeepSeek Harness0.86680.60250.264272.366.0
Codex0.87920.65570.223659.857.4
Spearman ρ\rho0.9/1.0

Here C~\widetilde C and M~\widetilde M are estimated solely from state-level action probes on the 271 validation trajectories, with zero probes on the 180 sampled tasks and no target task outcomes. These scores match the accuracy ranking at ρ=0.90\rho = 0.90 on the 271 validation tasks and predict the unprobed 180 sampled tasks at ρ=1.00\rho = 1.00. C~\widetilde C alone ranks Codex first and Pi last, and subtracting M~\widetilde M restores the order.

Second, our framework explains what post-training learns from a harness and why frozen evaluations misrank harnesses. Exact dynamic programming and dense state sampling M^\widehat M isolate each error source and pinpoint which saved states lose return through offline forward passes with no live environment execution. Compared with running full live benchmarks or RL training on every candidate harness, a small inference budget on a few trajectories saves far slower container runs and wasted training, making each diagnostic step practical on heavy tasks.

  1. Step-level probes on a few trajectories predict full-benchmark rankings. On Terminal-Bench Lite, where a single live coding task takes 25 to 35 minutes, probing just 3 tasks ranks all five coding harnesses across the full 39-task benchmark at ρ=1.00\rho = 1.00 with no target runs on the other tasks.
  2. Normal teacher rollouts check whether a low frozen score hides a high ceiling. Running a teacher model for 3 runs per task needs no action categories or state resampling and gives a guaranteed lower bound on the ceiling. Across all five coding harnesses and all five ALFWorld harnesses, teacher accuracy exceeds frozen Qwen3.5-9B on every harness, by 16.1 and 25.9 percentage points on average.
  3. The pre-training threshold TT predicts adaptation gains before any RL training. Equal to the new ceiling minus the initial return, TT needs zero training on the candidate harness. Across all nine harness swaps rejected by frozen evaluation, TT orders post-training return gains at ρ=0.90\rho = 0.90 while the frozen return change reaches only ρ=0.27\rho = 0.27, and ceiling gain ΔC=C2−C1\Delta C = C_2 - C_1 gets the sign of all nine gains right.
  4. Full episode rollouts measure overall mismatch at low query cost. When the ceiling CC is known, estimating return JJ from normal rollouts and computing M=C−JM = C - J tells whether a low frozen score comes from a low ceiling or from large mismatch that adaptation can remove. To keep return-difference RMSE within 0.0025 when comparing two setups, this cuts model calls per task from about 85,000 for state sampling down to 16 when action probabilities are available, or 256 when only final task outcomes are available.

When a harness acts as a black box without closed-form action probabilities, estimating action distributions or mismatch from discrete outputs naturally requires multiple samples, as seen between 16 calls with probabilities and 256 from outcomes alone. Faster single-pass estimators with no resampling remain a valuable direction for future work.

Clarify how well the KL-regularized control formulation captures practical agent training and inference, especially under stochastic environments and approximate optimization.

Response: Thank you for your thoughtful question. The decomposition J=C−MJ = C - M is an evaluation identity that holds for any deployed policy and any KL weight τ≥0\tau \ge 0, regardless of how the model was trained. For practical inference, setting τ=0\tau = 0 removes the KL penalty completely, so JJ is plain expected task return, CC is the highest return the harness allows, MM is the return lost to suboptimal actions, and J=C−MJ = C - M reduces to the standard performance difference lemma.

  1. For stochastic environments, the identity J=C−MJ = C - M holds directly under general stochastic state transitions. To test stochasticity empirically, we made MiniWoB++ clicks fail randomly by varying click reliability from 50% to 100% alongside action cost and layout across 72 held-out deployment comparisons. Under these stochastic transitions, ΔC−ΔM^\Delta C - \Delta \widehat M predicts the direction of 71 out of 72 return changes and cuts return-change RMSE from 0.2347 with ΔC\Delta C alone down to 0.0023.
  2. Under approximate RL optimization, only exact score identification assumes a KL-regularized optimum, where a policy shifts its action log-probabilities relative to the reference model in proportion to optimal action values Q∗Q^*. To test approximate RL, we examined both our finite-step policy-gradient checkpoints and additional PPO-Clip runs with hard clipping. Fitting each trained policy's log-probability shift linearly to Q∗Q^* explains 62% to 94% of variance across all trained policies, including 62% to 68% under PPO-Clip, showing that the action-score geometry holds well under approximate optimization.
  3. To test training and evaluation with no KL penalty, we also retrained Qwen3.5-2B from scratch with the KL weight set to zero and evaluated on all 576 held-out tasks with no KL term. Under the baseline harness H0, unregularized training already reaches 98.79% task success, so adapting to the batched-click harness H1 improves plain discounted task return by cutting decisions per task from 3.44 to 2.57. Every core finding from the paper holds.
Comparison on 576 held-out tasksMetricNo-KL baselineNo-KL resultNo-KL changeKL-trained change
Initial H0 to frozen batched clicks H1Success (%)98.7993.90−4.89−12.15
Original to reversed fields under H0Success (%)98.7921.82−76.97−65.88
Initial H0 to adapted batched clicks H1Plain task return0.87360.9131+0.0395+0.0367
H0 continuation to adapted batched clicks H1Plain task return0.86940.9131+0.0437+0.0530

The first two rows show that even with no KL penalty in training, the model still specializes strongly to H0. Switching frozen to H1 drops task success by 4.89 percentage points, and reversing field order alone drops success by 76.97 percentage points. The last two rows show that despite this frozen drop, adapting to H1 still outperforms both the initial H0 model and continued H0 training in plain task return, matching the KL-trained results.

We will incorporate key parts of the findings and discussions above into the revised manuscript. We hope our responses address your concerns, and please feel free to let us know if you have any further questions.

Dear reviewer NwYA, thank you again for your valuable and constructive feedback. Since the discussion period is coming to a close and we have not yet heard back from you, we wanted to kindly follow up. We have carefully revised our manuscript and made several substantial updates in direct response to your comments.

  1. We extended our experiments to ALFWorld across five harnesses, confirming that our method remains effective.
  2. We provided practical low-cost estimation methods using a few probe tasks and normal rollouts.
  3. We tested stochastic environments and approximate optimization, and confirmed that our findings hold without KL regularization.

We hope these updates address your concerns, and we look forward to hearing your thoughts. If you still have any questions, we are very happy to discuss them further.

Comments on the paper

2 comments

The author-reviewer discussion period will end soon. Please review and acknowledge the authors’ responses. If your concerns have been addressed, please reconsider your score.

AuthorsOpenReview ↗

Summary of Rebuttal Discussion

Dear Area Chairs and Reviewers,

Thank you very much for your time and valuable feedback on our work. We sincerely appreciate the reviewers' constructive suggestions and positive comments, recognizing our compelling conceptual contribution and principled perspective on frozen-checkpoint comparisons, our useful framework for decoupling harness capability from model compatibility (Reviewers NwYA, cqDR, and DReX), and our combination of precise analysis on solvable MiniWoB++ environments with practical coding-agent evaluations (Reviewers cqDR and DReX). In response to the reviewers' valuable comments, we have conducted substantial new experiments and analyses to further strengthen the paper and improve clarity. Below is a brief summary of our main updates in response to the reviews.

  • We extended our evaluation beyond web interaction and coding to ALFWorld across five harnesses, and added practical low-cost estimation methods to strengthen generality and applicability.
  • We verified that our main conclusions hold without KL regularization, under a shared reference policy, and under stochastic transitions and approximate optimization, addressing the shared concern of all four reviewers.
  • We verified stability across multiple training seeds, an additional teacher model, coarser action categories, and 6 probe tasks, alongside theoretical bounds and computational cost comparisons.
  • We clarified how agent harnesses differ from separate RL environments, ruled out capability bottlenecks, and showed alongside recent studies that larger and frontier models remain sensitive to readout drift and harness choice.
  • We rewrote the abstract and restructured the introduction in plain language with clear motivation so the paper is easy to follow for first-time readers.

During the discussion period, Reviewer cqDR replied that "maintain my positive scores" with no further concerns raised. While the other reviewers have not yet joined the discussion, we believe our responses and new experiments directly address each of their questions and concerns. To help the ACs quickly review our responses during the rebuttal, we have also provided brief summaries in our follow-up comments under each review discussion.

Finally, we sincerely thank the ACs and Reviewers for all the effort dedicated to our paper. These new experiments and analyses have further strengthened both our contributions and our presentation, and we have incorporated key updates into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.