- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether reinforcement learning for coding agents improves general task-solving ability or mainly adapts a policy to particular execution harnesses. Starting from Qwen3.5-27B, the authors train nine PPO/GRPO checkpoints using single- and multi-harness configurations and evaluate them with four harnesses on SWE-Bench Pro. The results suggest that RL primarily increases the consistency with which previously solvable tasks are completed across harnesses, rather than substantially expanding overall task coverage.
Critical strengths
- The paper identifies an important distinction between acquiring new task-solving capabilities and adapting existing capabilities to different execution harnesses.
- The evaluation covers several training and deployment harnesses and includes task-level overlap, patch-delivery behavior, and paired statistical analyses rather than reporting only aggregate benchmark scores.
Critical weaknesses
- The experiments use only one base model. All checkpoints are initialized from Qwen3.5-27B, so it is unclear whether the observed harness adaptation is specific to this model's prior capabilities and tool-use behavior. Evaluations across multiple checkpoints derived from the same initialization do not establish generality across model families or scales. At least one additional base model is needed to support the broader methodological claims.
- The task scope is too narrow. Both training and primary evaluation are restricted to repository-level software engineering tasks from SWE-rebench and SWE-Bench. Consequently, the paper does not show whether the proposed harness-conditioned learning framework transfers to other agentic domains, such as web navigation, computer use, information retrieval, or data analysis. Even within SWE-Bench Pro, all models achieve zero success on the Go and TypeScript tasks, so the reported gains are effectively limited to Python and JavaScript. This substantially weakens the claim of transferable harness adaptation.
- The necessity of reinforcement learning is not demonstrated. The paper contains no compute- or data-matched SFT, or trajectory-distillation baseline. Many reported gains, especially the increased tendency to apply edits and submit nonempty patches, may be learnable directly from successful trajectories without online RL. Without such comparisons, the experiments show that PPO can induce harness adaptation, but not that RL is required or preferable.
- The training evidence has limited robustness. Each training recipe is represented by only one run, while some comparisons simultaneously change the harness mixture, training-task pool, update budget, reconstruction procedure, and optimization algorithm. The supplementary SWE-Bench Verified evaluation is also weak: the best trained model only matches the base model's four-harness mean Pass@3, while most trained checkpoints perform worse.
Questions for authors
- Do the cross-harness adaptation patterns persist when the same training framework is applied to another base-model family or to models at different scales?
- Can the authors evaluate the method outside repository-level software engineering and explain why the current models obtain zero success on all Go and TypeScript tasks?
- How does PPO compare with compute- and data-matched SFT or behavior cloning on successful multi-harness trajectories, particularly for patch delivery and held-out-harness performance?
- Can the authors repeat the main training recipes with multiple random seeds and isolate the effects of harness diversity, training budget, reconstruction, and optimization method?