- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Co-Harness alternates between optimizing the agent harness and the model weights. In each round, an LLM-based HarnessCritic diagnoses failed trajectories and proposes local harness edits, which are accepted only after validation. Quality-filtered trajectories collected under the new harness are then used for SFT, and the updated model seeds the next harness search. The method is evaluated on SWE-bench Verified/Pro and on tool-integrated mathematical reasoning. The headline result is that Qwen3-32B improves from 12.9% to 61.8% on SWE-bench Verified .
Critical strengths
- A natural and timely idea, with a transparent optimization log. Treating the harness and the weights as jointly optimized components is a sensible systems view of agent training. The main text shows the full harness evolution including degraded candidates and rollbacks.
Critical weaknesses
- Harness selection and evaluation do not appear to be separated. The harness is searched directly on SWE-bench Verified and AIME24, and the selected configuration is then used to report results on the same benchmarks. The paper says final performance is measured on held-out tasks, but it never specifies which tasks are used for search, SFT data collection, validation, and final evaluation. Without this, part of the reported gain may come from selection on the test set rather than from the method itself.
- The SFT data is not described. It is unclear which tasks or repositories the trajectories come from, how many trajectories were collected, and how many remain after filtering. SWE-bench covers only a small number of repositories, so overlap between training and evaluation repositories is a real risk. Given that two rounds of SFT raise performance from 12.9% to 61.8%, this information is essential.
- Key ablations are missing. There is no "fixed best harness + SFT" baseline, so it is impossible to tell how much co-evolution adds over plain SFT. The critic model is also not specified, so part of the improvement may come from distillation from a stronger model.
- It is unclear whether the evolved harness contains benchmark-specific shortcuts. The paper does not analyze what the discovered harness changes actually do. Some of them may exploit properties of the benchmark (e.g., test-file patterns, answer formats, or evaluation-script behavior) rather than improve general agent capability. Could the authors list the concrete harness modifications and discuss whether any of them amount to benchmark hacking?
Questions for authors
- What are the training, selection, and evaluation splits for each benchmark? On which split were the 28 Verified configurations in §5.3 selected, and how is non-overlap with the data behind Table 2 guaranteed?
- For each SFT round, which tasks and repositories were used, and how many trajectories? Were the evaluation repositories excluded?
- Which model is used as the HarnessCritic, and what is its computational cost?
- Re-select the harness on an independent validation split, and add compute-matched ablations: harness-only, SFT-only (fixed best harness + SFT), and joint optimization.
- Describe the initial code harness, and quantify how much of the TIR gain comes from backend/cleanup fixes versus the proposed loop itself
6 comments
Response to Reviewer SZRF Part (2/2)
Q5. What changes did the harness search retain, and could they exploit benchmark shortcuts?
A5. Thank you for asking for concrete examples. In coding, edit reflection reaches 15.4% from 12.9%; combining stall handling and test-error analysis reaches 16.4%; the selected bundle adds submission gating, scope checks, and repetitive-probe detection, reaching 19.2%. Probe detection uses sequence similarity at least 0.85 and issues a soft hint after three similar probes. These are bundled configuration outcomes, not isolated effect sizes for each edit. Stacking the initial interventions instead scores 12.1%, showing that plausible changes need validation.
For TIR, the reported changes include inference-backend repair, replacing ThreadPool with ProcessPool and process cleanup, batching, restoring reasoning on follow-up turns, adjusting turn limits, and a separately reported voting ensemble. These mechanisms have ordinary runtime interpretations. The method explicitly places benchmark definitions and scoring logic outside the edit surface.
The editable surface excludes benchmark definitions and scoring logic. The retained coding changes address repeated probes, patch scope, and premature submission.
Q6. What does the initial coding harness do, and what did the TIR repairs change?
A6. Thank you for raising both points. The initial coding harness lets the agent navigate a repository, edit files, and execute shell commands. An executable test suite verifies the resulting patch.
For the extended TIR case, the initial 32-second crash has no accuracy score; the later failed run’s 22.5% is incomplete and cannot be used as a comparable baseline. The first completed configuration reaches 59.6% in 3.78 hours, and a later configuration reaches 61.5% in 1.11 hours: a 1.9-point difference and approximately 3.4× runtime ratio. That later configuration combines batching, follow-up reasoning, and changed turn limits, so neither difference isolates cleanup or batching alone. The six-trajectory voting ensemble reaches 63.3% in 2.29 hours under a different aggregation rule from main-table pass@1.
We sincerely hope that our clarifications can address your concerns. We appreciate your recognition of the value of the configuration history and your suggestions for strengthening the evaluation. The reviewer's suggestions can be easily and effectively addressed, and we genuinely hope that the reviewer can consider increasing the score. Thank you very much for your time and thoughtful feedback!
Summary of Response to Reviewer SZRF
Dear Reviewer SZRF, thank you for recognizing the value of the harness revision history and for asking us to clarify how we selected and evaluated the configurations. We clarified that the 28 coding configurations were selected on SWE-smith training tasks and scored separately on held-out SWE-bench Verified. We also added two comparisons on Qwen3-32B Verified: Co-Harness reached 61.8%, compared with 22.8% for two rounds of SFT with the R0 harness fixed and 56.4% for the harness-first schedule.
Your feedback helped us explain the evaluation protocol more clearly and show how Co-Harness compares with both training schedules. We sincerely hope these clarifications and additional results address your concerns, and we would be grateful if you would consider raising your score in light of this response. Thank you very much for your time and thoughtful feedback!
Response to Reviewer SZRF Part (1/2)
Dear Reviewer SZRF,
Thank you for recognizing the value of the configuration history and for asking how we selected and evaluated the harnesses. We appreciate the opportunity to clarify the data protocol and report the additional comparisons.
Q1. Which split was used to select the 28 Verified configurations, and how was evaluation kept separate?
A1. Thank you for raising this concern. We apologize that the figure did not clearly distinguish selection from reporting. Our coding training tasks come from both SWE-smith and R2E-Gym. Our TIR training problems come from the Nemotron-Math collection. Harness iteration and SFT use training tasks only, and final evaluation uses held-out tasks. Failure diagnosis uses failed training rollouts, candidate selection and regression checks use training-side outcomes, and SFT uses verified or quality-filtered training trajectories.
We selected among the 28 configurations using SWE-smith training tasks while keeping Qwen3-32B weights fixed. The displayed Verified scores were measured separately after selection and did not determine which configuration we retained. The selected R0 configuration scored 19.2% on Verified. Table 2 reports held-out evaluation, and its task outcomes were not used for selection. This is an instance-level protocol.
Q2. Where do the SFT trajectories come from and how are they filtered?
A2. Thank you for asking us to clarify the training data. Each SFT round uses trajectories generated by the current task model on the training sources described above. For coding, we verify patches using executable tests. For TIR, we verify the final answer.
Q3. How does Co-Harness compare with fixed-harness SFT and harness-first training?
A3. Thank you for suggesting these comparisons. We ran the requested fixed-R0 comparison on Qwen3-32B Verified. The control uses the R0 harness for both rounds and collects fresh trajectories for round two using its first-round SFT model.
Both arms used the same training task list and scored 19.2% on held-out Verified before the first SFT stage. The fixed-harness arm used the R0 harness for trajectory collection in both rounds. The gap was 20.8 percentage points after the first SFT stage and 39.0 points after the second. The second round therefore widened an existing gap by 18.2 points across the two schedules.
Starting from the same 19.2% R0 score, the harness-first schedule reaches 42.2% with initial weights and the final selected harness, then 56.4% after one SFT stage. Co-Harness finishes 5.4 points higher. Human to R0 measures the initial harness-only search (12.9% to 19.2%). The 28-configuration search already used training-side selection, as described in S1.
Q4. Which model serves as HarnessCritic, and how does it contribute?
A4. Thank you for asking us to clarify the critic’s role. We use GLM-5.3 as HarnessCritic and Qwen3-8B or Qwen3-32B as the task model. The critic reads failed traces and the harness configuration, then proposes diagnoses and local edits. The current Qwen3 model generates the SFT trajectories under the accepted harness.
The critic contributes through the harness edits that shape subsequent task-model rollouts. The 200+ hours reported for TIR cover the entire search case.