- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that a single globally optimized harness is not necessarily suitable for every test instance. The authors summarize the logs left during Meta-Harness search into a playbook, and then train an editor using GRPO: for each test instance, it reads the playbook and patches the global harness. Experiments cover 7 benchmarks, showing an improvement on SWE-smith-MR and a 5-point improvement over Meta-Harness.
Critical strengths
- Good motivation: Recycling the by-products of harness search is a clean and intuitive idea.
- Well-designed ablation: Neither using only the playbook nor only doing RL yields improvements; combining both is what makes it effective.
- Rigorous reporting: Standard errors over 3 to 5 runs are reported for coding tasks.
Critical weaknesses
-
Limited novelty: Customizing prompts or harnesses per instance already exists; the playbook directly adopts ACE; training uses standard GRPO. The patches shown in Figure 1 are essentially workflow descriptions and command examples: closer to "instance-customized prompts" rather than structural modifications to harness code. The paper lacks analysis of what the patches actually modify and what proportion consists of code-level changes.
-
Unfair compute comparison. Turbo performs additional RL training on top of Meta-Harness, requiring heavy environment interactions, whereas Meta-Harness is not allocated this extra budget. Two key baselines are missing: continuing to run Meta-Harness search using the same budget, and a simple router that selects a harness from existing ones in the search logs for each instance. Without these baselines, it is impossible to tell whether the gains come from "instance adaptation" or simply from more compute.
-
Potential data leakage/memorization at the repository level. Both coding benchmarks split their training and test sets within the same set of repositories. The editor likely learns "how to handle repository X" rather than adapting to individual instances. The fact that Gemini uses only 8.7 steps on SWE-smith-MR with costs dropping to a seventh also needs explanation. Furthermore, SWE-smith-MR is a custom subset built by the authors with a test set of only 50 problems; Terminal bench 2.1 has only 44 problems; and the 40-step limit further amplifies the impact of the harness.
Questions for authors
-
Under equivalent additional compute, how do results compare if we continue running Meta-Harness or use a router to pick a harness from search logs for each instance?
-
If test repositories are completely excluded from the training set, how much of the performance improvement remains? What is the proportion of code-logic modifications within the patches?
3 comments
Thank you again for your feedback. As the discussion period is coming to a close, we wanted to follow up on our rebuttal and would be happy to clarify any points if helpful.
Reply to authors
Thanks for your clarification! I'll keep my positive score.
Thank you for your thoughtful review! We appreciate your recognition of the
good motivationbehind our work and your positive assessment of ourwell-designed ablationstudy. We are also pleased that you highlighted therigorous reportingof our experimental results.W1. Limited novelty.
We contextualize our contributions with regard to existing prompt/harness optimization methods.
Search space. Prompt optimization methods (e.g., GEPA, ACE) optimize model instructions, whereas harness optimization (in our setting) considers a broader search space that includes workflows, tool interfaces, and verification mechanisms. This broader search space makes finding a strong harness much more challenging than finding an effective prompt, but it also offers more space for performance improvements.
Code modifications. Our additional analysis confirms that Turbo Harness modifies executable scaffold logic:
Empirical comparison.
Table 2shows that Turbo Harness outperforms commonly used prompt-optimization baselines (e.g., GEPA and ACE) on SWE-bench Verified, achieving 59.3% versus 41.3% for GEPA, and 34.7% for ACE on Claude Haiku 4.5. Additionally, compared with SOTA harness optimization methods (e.g., Meta harness), Turbo harness consistently achieves a more favorable performance across all benchmarks we have tested.We will incorporate the analysis and clarify these distinctions in the revision.
W2 & Q1. Unfair compute comparison.
Great point! First, to clarify, Turbo Harness's RL training involves relatively lightweight patch generation and application. In contrast, Meta-Harness search relies on Claude Code to analyze execution trajectories and propose new harnesses, making each search iteration more computationally expensive.
Following your suggestion, we conducted two additional experiments on ALFWorld under the same setting as
Table 1: (1) extending Meta-Harness search with a comparable additional compute budget, and (2) training a router to select instance-specific harnesses from the existing search archive.Specifically, we extended Meta-Harness from 10 to 40 iterations but found that none of the additional proposed harnesses surpassed the existing training-set frontier, leaving the test performance unchanged. This observation further motivates Turbo Harness: rather than continuing text-based harness search, we introduce weight-based optimization through RL to learn an instance-adaptive editing policy, potentially overcoming the stagnation encountered in global harness optimization.
These results demonstrate that Turbo Harness outperforms both extended global search and the harness router.
W3 & Q2. Generalization and evaluation settings.
Our coding experiments use held-out issues from shared repositories to evaluate within the same distribution. To further assess generalization, we also evaluate Turbo Harness on Terminal-Bench 2.1 (see
Table 3), which spans diverse terminal tasks with substantially different execution requirements. Turbo Harness outperforms Meta-Harness and achieves the highest pass rate among the evaluated harnesses, providing complementary evidence of its generalization across heterogeneous tasks. Since Terminal-Bench 2.1 contains only 89 tasks, we reserve 44 for held-out testing to ensure that evaluation tasks are not used during training.Regarding the 40-step budget, we additionally evaluated both methods under a larger 250-step limit on SWE-bench Verified (Haiku 4.5):
Turbo Harness maintains its advantage under the higher execution budget.
Regarding code-logic modifications, their frequency depends on the benchmark and the initial harness structure. As detailed in our response to
W1, our additional analysis confirms substantial executable scaffold modifications on both SWE-bench Verified and DBBench.