acceptodds
Under review as a conference paper at ICLR 2027

One Policy, Many Harnesses: Reinforcement Learning for Transferable Harness Adaptation

Abstract

Reinforcement learning has improved the performance of coding agents on repository-level software engineering tasks. However, it remains unclear whether the improvement after RL reflects success on more distinct tasks or on the same tasks under more harnesses. We conduct eight training experiments using PPO, alongside one GRPO comparison. Together with the base model, these yield ten Qwen3.5-27B checkpoints evaluated in 40 checkpoint–harness combinations on 731 SWE-Bench Pro tasks, combination produce 87,720 evaluation sessions for Pass@3 evaluation. The key findings are: ① Task-level analysis shows that harness-specific gains primarily transfer from tasks the base model had already solved through another harness. The main gains increase success shared across harnesses without a comparable expansion in total observed task coverage. ② Gains can reach harnesses excluded from RL training, but transfer is direction-dependent: improvements in one harness can coexist with regressions in others. ③ Execution analysis links improved task success to more frequent delivery of nonempty patches, highlighting the importance of carrying edits through to an executable artifact. Together, these findings identify harness adaptation as a substantial component of the observed RL gains. They motivate training and evaluating coding agents for reliable execution across deployment harnesses, rather than treating a higher score in one harness as evidence of broader task-solving competence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Harness, Reinforement Learning for LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5pjbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether reinforcement learning for coding agents improves general task-solving ability or mainly adapts a policy to particular execution harnesses. Starting from Qwen3.5-27B, the authors train nine PPO/GRPO checkpoints using single- and multi-harness configurations and evaluate them with four harnesses on SWE-Bench Pro. The results suggest that RL primarily increases the consistency with which previously solvable tasks are completed across harnesses, rather than substantially expanding overall task coverage.

Critical strengths
  1. The paper identifies an important distinction between acquiring new task-solving capabilities and adapting existing capabilities to different execution harnesses.
  2. The evaluation covers several training and deployment harnesses and includes task-level overlap, patch-delivery behavior, and paired statistical analyses rather than reporting only aggregate benchmark scores.
Critical weaknesses
  1. The experiments use only one base model. All checkpoints are initialized from Qwen3.5-27B, so it is unclear whether the observed harness adaptation is specific to this model's prior capabilities and tool-use behavior. Evaluations across multiple checkpoints derived from the same initialization do not establish generality across model families or scales. At least one additional base model is needed to support the broader methodological claims.
  2. The task scope is too narrow. Both training and primary evaluation are restricted to repository-level software engineering tasks from SWE-rebench and SWE-Bench. Consequently, the paper does not show whether the proposed harness-conditioned learning framework transfers to other agentic domains, such as web navigation, computer use, information retrieval, or data analysis. Even within SWE-Bench Pro, all models achieve zero success on the Go and TypeScript tasks, so the reported gains are effectively limited to Python and JavaScript. This substantially weakens the claim of transferable harness adaptation.
  3. The necessity of reinforcement learning is not demonstrated. The paper contains no compute- or data-matched SFT, or trajectory-distillation baseline. Many reported gains, especially the increased tendency to apply edits and submit nonempty patches, may be learnable directly from successful trajectories without online RL. Without such comparisons, the experiments show that PPO can induce harness adaptation, but not that RL is required or preferable.
  4. The training evidence has limited robustness. Each training recipe is represented by only one run, while some comparisons simultaneously change the harness mixture, training-task pool, update budget, reconstruction procedure, and optimization algorithm. The supplementary SWE-Bench Verified evaluation is also weak: the best trained model only matches the base model's four-harness mean Pass@3, while most trained checkpoints perform worse.
Questions for authors
  1. Do the cross-harness adaptation patterns persist when the same training framework is applied to another base-model family or to models at different scales?
  2. Can the authors evaluate the method outside repository-level software engineering and explain why the current models obtain zero success on all Go and TypeScript tasks?
  3. How does PPO compare with compute- and data-matched SFT or behavior cloning on successful multi-harness trajectories, particularly for patch delivery and held-out-harness performance?
  4. Can the authors repeat the main training recipes with multiple random seeds and isolate the effects of harness diversity, training budget, reconstruction, and optimization method?
Reviewer_fU1xOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the success rate improvements in code-generation agents following reinforcement learning (RL) stem from the agents’ ability to solve a broader range of distinct tasks, or from their increased capacity to complete tasks – previously solvable within only one specific tool-execution framework – across a wider variety of such frameworks. The authors employ a “harness” to formalize the decision-making process, accounting for variations in tool interpretation, context management, and termination criteria. They then construct different training recipes using techniques such as grouped sampling, shared critics, and trajectory-normalized PPO.

The experiments utilize a Qwen3.5-27B base model and nine training checkpoints, with cross-evaluation performed across four frameworks – Codex, OpenCode, Pi, and Claude Code – on 731 SWE-Bench Pro tasks. Each task was attempted three times per framework, resulting in a total of 87,720 sessions. For the Codex-trained checkpoint, the Codex Pass@3 metric increased from 14.23% to 29.55%. The number of tasks successfully completed across all four frameworks rose from 82 to 159, while the union of tasks successfully completed in at least one framework shifted from 266 to 262. Notably, among the 113 newly successful tasks for the Codex-trained agent, 113 had already been solved by the base model using one of the other frameworks. The study also observes policy transfer – specifically from Pi to the Codex-trained agent (which was not trained on Pi data) – and notes that improvements in success rates are accompanied by an increase in the submission of non-empty patches.

Critical strengths
  1. Task identity analysis offers insights that go beyond simple average scores across individual frameworks. For Codex training checkpoints, the intersection of successful tasks increased from 82 to 159, while the union shifted from 266 to 262; notably, 113 of the 118 newly successful tasks for Codex had already been solved by the Base model in other frameworks. This result clearly distinguishes between an increase in shared successes and an expansion of task coverage under a fixed budget.
  2. The experimental design is well-aligned with the empirical research questions. The study utilizes a comprehensive matrix – covering ten checkpoints across four frameworks – that accounts for failures and timeouts. It employs task-paired bootstrapping, McNemar’s test, and corrections for multiple comparisons, while also distinguishing between task completion, patch submission, and post-submission success. Furthermore, the authors explicitly constrain the scope to fixed checkpoints and a limited budget of attempts, which strengthens the credibility of the empirical findings.
Critical weaknesses

The observed adaptation gains are concentrated within a specific pairing of a weaker base model and a particular framework, with a narrow scope for extrapolation. All seven statistically significant positive shifts occurred exclusively with the Codex framework; under this framework, the rate of non-empty patch delivery following a “normal” termination of the base model was only 39.55%, whereas it exceeded 94% for all other frameworks. Consequently, some of these gains may simply reflect the correction of mismatches regarding specific execution contracts; it remains unclear to what extent standard inference-time interfaces or completion-validation checks might recover such performance. Furthermore, given that the experiments involved only a single model and a single training run per configuration – with zero successes across all Go and TypeScript tasks – and that the inclusion of a “Verified” dataset showed no overall improvement in the mean performance across the four frameworks, these findings do not negate the observed phenomena, but they do limit the strength of “transferable adaptation” as a generalized training conclusion.

Questions for authors
  1. Using existing experimental records, recalculate the union, intersection, and incremental successful overlaps for each framework (k=1, 2, 3), and provide the task-level pairing intervals. Does the observed pattern – where the union remains relatively stable while the set of successful outcomes grows – still hold?
  2. Disaggregate the sources of “empty patches” (cases where no code is generated) from Codex: Was it due to no modifications being performed, actions not being accepted, or failures in the export/extraction process? Conduct a controlled comparison on representative tasks, evaluating the “Base/Post-training” vs. “Native/Unified” patch delivery pathways (noting that the unified pathway can only export modifications that have already been made). Otherwise, the underlying mechanism should be interpreted as a correlation rather than a direct causal link.
Reviewer_eu5hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how RL changes coding-agent performance across execution harnesses. Using Qwen3.5-27B and nine trained checkpoints, it evaluates task-level outcomes through four harnesses on SWE-Bench Pro. Many newly successful tasks under Codex were already solved by Base through other harnesses, while the observed all-harness intersection grows substantially. The paper also reports directional transfer to unseen harnesses and increased nonempty-patch delivery.

Critical strengths

Tracking task identities across harnesses provides useful information beyond aggregate success rates. The distinction between observed task coverage and success shared across harnesses is practically relevant: a deployment improvement can matter even when it does not expand the observed set of solvable tasks.

The full evaluation matrix exposes regressions alongside gains. Paired testing and multiple-comparison correction strengthen the descriptive results. The execution-stage analysis is also useful, particularly the distinction between completing a session and actually delivering a repository modification.

Critical weaknesses

The strongest results are concentrated in a particularly weak Base–Codex pairing. All seven Holm-significant gains occur under Codex, where Base delivers nonempty patches in only 39.55% of completed sessions, compared with over 94% under the other harnesses. This does not invalidate the adaptation finding, but leaves its scope uncertain. A controlled intervention on patch application or termination would help establish whether the main gains persist after addressing this execution bottleneck.

The fixed-budget interpretation of the task statistics is appropriate, and Section 6 explicitly acknowledges the limits of causal interpretation. Still, the evidence does not clearly distinguish execution adaptation from improved problem-solving within an overlapping task set. A task already solved by Base elsewhere can benefit from either. The patch-success analysis also compares checkpoint-specific subsets, so changes in conditional success cannot isolate changes in patch quality. The proposed behavioral explanation is plausible, but remains incompletely tested.

Robustness is limited by one backbone and one training run per recipe. The task-level intervals are informative for the evaluated checkpoints but do not quantify training-run variability. Some recipe comparisons also change multiple factors, including the 500-task PPO pool versus the 190-task GRPO pool. The 45-task Verified extension shows a similar directional-transfer pattern in point estimates, but offers limited additional evidence about the main task-coverage finding. Its lack of an overall improvement is not itself a weakness; the concern is how broadly the central observations generalize.

Questions for authors
  1. What causes Base’s low patch-delivery rate under Codex? Does a controlled intervention on patch application or termination, within the same execution budget, substantially reduce the gap to the trained models?

  2. Do the union, intersection, and task-overlap findings persist with more than three attempts per harness? This would help assess their sensitivity to sampling budget.

  3. Can the key comparisons be repeated across training seeds with matched data and rollout budgets? Separately, what explains zero successes on all 300 Go and TypeScript tasks, and what checks confirm that their execution and grading pipelines are functioning correctly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.