acceptodds
Under review as a conference paper at ICLR 2027

Turbo Harness: Instance-Adaptive Harness Optimization

Abstract

Designing effective harnesses with minimal human intervention is an important challenge for building modern agents. Existing automated harness optimization methods typically produce a single global harness that is applied uniformly across future problem instances. However, a harness that works well on average may not be optimal for every instance. We introduce turbo harness, a framework that can adapt a globally optimized harness to each individual instance by reusing information generated during the original optimization process. Specifically, turbo harness recycles artifacts, including execution trajectories and reflections, produced during a completed global harness optimization run. We train a harness editor, an RL-fine-tuned open-source model, to leverage these artifacts to generate instance-specific patches to the globally optimized harness. At inference time, the proposer uses the instance and prior optimization experience to construct a tailored harness for the execution model. We validate our method through comprehensive numerical experiments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Harness Optimization, LLM-Agent, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XhsnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that a single globally optimized harness is not necessarily suitable for every test instance. The authors summarize the logs left during Meta-Harness search into a playbook, and then train an editor using GRPO: for each test instance, it reads the playbook and patches the global harness. Experiments cover 7 benchmarks, showing an improvement on SWE-smith-MR and a 5-point improvement over Meta-Harness.

Critical strengths
  1. Good motivation: Recycling the by-products of harness search is a clean and intuitive idea.
  2. Well-designed ablation: Neither using only the playbook nor only doing RL yields improvements; combining both is what makes it effective.
  3. Rigorous reporting: Standard errors over 3 to 5 runs are reported for coding tasks.
Critical weaknesses
  1. Limited novelty: Customizing prompts or harnesses per instance already exists; the playbook directly adopts ACE; training uses standard GRPO. The patches shown in Figure 1 are essentially workflow descriptions and command examples: closer to "instance-customized prompts" rather than structural modifications to harness code. The paper lacks analysis of what the patches actually modify and what proportion consists of code-level changes.

  2. Unfair compute comparison. Turbo performs additional RL training on top of Meta-Harness, requiring heavy environment interactions, whereas Meta-Harness is not allocated this extra budget. Two key baselines are missing: continuing to run Meta-Harness search using the same budget, and a simple router that selects a harness from existing ones in the search logs for each instance. Without these baselines, it is impossible to tell whether the gains come from "instance adaptation" or simply from more compute.

  3. Potential data leakage/memorization at the repository level. Both coding benchmarks split their training and test sets within the same set of repositories. The editor likely learns "how to handle repository X" rather than adapting to individual instances. The fact that Gemini uses only 8.7 steps on SWE-smith-MR with costs dropping to a seventh also needs explanation. Furthermore, SWE-smith-MR is a custom subset built by the authors with a test set of only 50 problems; Terminal bench 2.1 has only 44 problems; and the 40-step limit further amplifies the impact of the harness.

Questions for authors
  1. Under equivalent additional compute, how do results compare if we continue running Meta-Harness or use a router to pick a harness from search logs for each instance?

  2. If test repositories are completely excluded from the training set, how much of the performance improvement remains? What is the proportion of code-logic modifications within the patches?

3 comments

Thank you for your thoughtful review! We appreciate your recognition of the good motivation behind our work and your positive assessment of our well-designed ablation study. We are also pleased that you highlighted the rigorous reporting of our experimental results.

W1. Limited novelty.

We contextualize our contributions with regard to existing prompt/harness optimization methods.

  1. Search space. Prompt optimization methods (e.g., GEPA, ACE) optimize model instructions, whereas harness optimization (in our setting) considers a broader search space that includes workflows, tool interfaces, and verification mechanisms. This broader search space makes finding a strong harness much more challenging than finding an effective prompt, but it also offers more space for performance improvements.

  2. Code modifications. Our additional analysis confirms that Turbo Harness modifies executable scaffold logic:

    • SWE-bench Verified (Gemini): 148/150 patches modify the agent loop's exploration, with instance-dependent thresholds, etc.
    • DBBench: 48/60 patches modify scaffold code.
  3. Empirical comparison. Table 2 shows that Turbo Harness outperforms commonly used prompt-optimization baselines (e.g., GEPA and ACE) on SWE-bench Verified, achieving 59.3% versus 41.3% for GEPA, and 34.7% for ACE on Claude Haiku 4.5. Additionally, compared with SOTA harness optimization methods (e.g., Meta harness), Turbo harness consistently achieves a more favorable performance across all benchmarks we have tested.

We will incorporate the analysis and clarify these distinctions in the revision.

W2 & Q1. Unfair compute comparison.

Great point! First, to clarify, Turbo Harness's RL training involves relatively lightweight patch generation and application. In contrast, Meta-Harness search relies on Claude Code to analyze execution trajectories and propose new harnesses, making each search iteration more computationally expensive.

Following your suggestion, we conducted two additional experiments on ALFWorld under the same setting as Table 1: (1) extending Meta-Harness search with a comparable additional compute budget, and (2) training a router to select instance-specific harnesses from the existing search archive.

Specifically, we extended Meta-Harness from 10 to 40 iterations but found that none of the additional proposed harnesses surpassed the existing training-set frontier, leaving the test performance unchanged. This observation further motivates Turbo Harness: rather than continuing text-based harness search, we introduce weight-based optimization through RL to learn an instance-adaptive editing policy, potentially overcoming the stagnation encountered in global harness optimization.

MethodALFWorld (%)
Meta-Harness (10 iterations)60.7
Meta-Harness (40 iterations)60.7
Meta-Harness Router59.3
Turbo Harness70.7

These results demonstrate that Turbo Harness outperforms both extended global search and the harness router.

W3 & Q2. Generalization and evaluation settings.

Our coding experiments use held-out issues from shared repositories to evaluate within the same distribution. To further assess generalization, we also evaluate Turbo Harness on Terminal-Bench 2.1 (see Table 3), which spans diverse terminal tasks with substantially different execution requirements. Turbo Harness outperforms Meta-Harness and achieves the highest pass rate among the evaluated harnesses, providing complementary evidence of its generalization across heterogeneous tasks. Since Terminal-Bench 2.1 contains only 89 tasks, we reserve 44 for held-out testing to ensure that evaluation tasks are not used during training.

Regarding the 40-step budget, we additionally evaluated both methods under a larger 250-step limit on SWE-bench Verified (Haiku 4.5):

Method40 steps250 steps
Meta-Harness56.759.3
Turbo Harness59.361.3

Turbo Harness maintains its advantage under the higher execution budget.

Regarding code-logic modifications, their frequency depends on the benchmark and the initial harness structure. As detailed in our response to W1, our additional analysis confirms substantial executable scaffold modifications on both SWE-bench Verified and DBBench.

Thank you again for your feedback. As the discussion period is coming to a close, we wanted to follow up on our rebuttal and would be happy to clarify any points if helpful.

Reviewer_XhsnOpenReview ↗

Reply to authors

Thanks for your clarification! I'll keep my positive score.

Reviewer_kvSQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. Turbo Harness studies whether a globally optimized agent harness can be improved by adapting it to each task instance. The execution model stays frozen; the object being changed is the surrounding harness, including its workflow and executable control logic.
  2. The method summarizes successful and failed edits from an earlier harness search into a playbook. A Qwen3.5-9B editor is then trained with GRPO to use the task, global harness, and playbook to propose a patch. Deployment requires one editor call before task execution.
  3. Experiments cover seven benchmarks spanning interactive environments, coding, and terminal tasks. The paper reports task performance, executor cost, and ablations of the playbook and editor training. The largest improvements appear in the coding experiments.
Critical strengths
  1. Reusing the search archive is a useful contribution. Failed candidates can contain strategies that are poor globally but helpful for particular instances. The method gives those intermediate artifacts a practical role without requiring a new multi-round search at deployment.
  2. The playbook/RL ablation supports the proposed combination. On SWE-smith-MR with Haiku, the untrained editor with a playbook reaches 50.0%, and the RL editor without it reaches 49.3%, versus 50.7% for Meta-Harness. Combining both reaches 64.0%. This is stronger evidence than showing a gain from an additional trained model alone.
  3. The evaluation covers several kinds of agent behavior and multiple executors. For example, Gemini improves from 38.4% to 54.4% on the held-out SWE-bench Verified subset. Terminal-Bench also improves from 50.5% to 55.5% over Meta-Harness. These results suggest that adaptation is useful beyond one benchmark-specific scaffold.
Critical weaknesses
  1. The efficiency claim needs complete cost accounting. Table 2 excludes editor inference and offline optimization. On SWE-smith-MR with Haiku, the executor saving is only $0.015 per issue, so the editor could materially change the conclusion. A single call to a 9B model is convenient, but its latency and cost still need to be measured.
  2. The error bars capture a limited source of variation. Appendix B.5 states that coding runs reuse the same generated per-instance harnesses and do not vary editor-training seeds. Four interactive benchmarks have only one evaluation run. The reported means therefore provide less evidence about reproducibility of the full training-and-adaptation pipeline than the repeated-run counts might suggest.
  3. Generalization is mainly within the environments used to train the editor. Both coding splits share repositories between training and testing, and a separate editor/playbook is built for each benchmark and executor. The authors acknowledge transfer as future work, but this limits how broadly the results support reusable harness adaptation. The 40-step coding budget also leaves open whether the gains persist when executors have more room to recover from poor guidance.
Questions for authors
  1. Including editor inference, what are the per-task cost and latency? How many deployment tasks are needed to amortize editor training?
  2. How much do pass rates vary when both editor training and per-instance patch generation are independently repeated, rather than reusing the same adapted harnesses?
  3. Do the coding gains persist on held-out repositories or under a larger execution budget? Either comparison would help establish how much the benefit depends on the current training distribution and tight interaction limit.
3 comments

Thank you for your thoughtful review! We appreciate your recognition of our approach as a useful contribution, your positive assessment of our ablations, and your observation that instance-adaptive harness optimization is useful beyond one benchmark-specific scaffold.

W1 & Q1. Complete cost accounting and deployment efficiency.

We provide detailed cost measurements below.

  1. Offline optimization cost. Turbo Harness reuses artifacts from the global harness search. Its additional costs are playbook construction and RL training. For SWE-bench Verified with Gemini 3.7 Flash:
StageAPI CallsGPU hoursWall Clock
Meta-harness (outer-loop search)109k06.9h
Building the playbook (Ours)0.25k00.31h
Training editor (Ours)80k50.16.3h

Turbo Harness introduces comparable or fewer LLM API calls than Meta-Harness global search, plus 50.1 H100 GPU-hours incurred once per editor configuration.

  1. Editor inference latency. Deployment requires only one inference call to a lightweight Qwen3.5-9B editor. We measured latency on a single H100:
Per-task latencyTB2.1 (Claude Sonnet 4.5)SWE-bench Verified (Gemini 3.7 Flash)
Default harness902 s (Terminus-2)224 s
Turbo Harness (Ours)887 s186 s
- Editor inference8.8 s (0.99%)1.32 s (0.71%)
- Executor878 s (99.01%)185 s (99.29%)

The editor accounts for less than 1% of per-task latency, while Turbo Harness also reduces overall execution time relative to default harnesses.

  1. Cost amortization. It is a bit hard to determine precisely how many deployment tasks are needed to amortize editor training, as the cost savings depend on the specific workload, execution model, and API pricing. Moreover, Turbo Harness improves performance over baselines (e.g., Meta-Harness and Harness-R1), which compute costs alone cannot capture. In accuracy-critical settings such as cybersecurity or production systems, weaker harnesses may require retries, verification, or manual inspection, incurring additional costs.

We will include these measurements and discussion in our revision.


W2 & Q2. Reproducibility across training and inference.

Great question! We conducted additional experiments to assess robustness across different sources of randomness.

  1. Multi-seed training. We trained three independent editors on ScienceWorld with different seeds, following Table 1. Results show consistent improvements with limited variance.
BenchmarkSeed 1Seed 2Seed 3MeanStd
ScienceWorld42.438.441.840.92.2
  1. Multi-seed evaluation. We repeated evaluations on ScienceWorld and ALFWorld with three seeds. For SWE-smith, SWE-bench Verified, and Terminal-Bench 2.1, we also report multi-seed means and standard deviations in Tables 2 and Table 3.
BenchmarkDefaultMeta-HarnessTurbo (Ours)
ScienceWorld23.7 ± 0.531.3 ± 2.943.1 ± 1.4
ALFWorld39.8 ± 1.061.3 ± 2.967.6 ± 2.22
  1. Independent patch generation. We independently generated and evaluated three sets of instance-specific patches using the trained editor on SWE-bench Verified (Gemini 3.7 Flash).
BenchmarkRun 1Run 2Run 3MeanStd
SWE-bench Verified51.348.047.348.92.1

These experiments confirm consistent gains across training seeds, evaluation runs, and independent patch generations. We will add these results to our revision.


W3. Generalization is mainly within the environments used to train the editor.

We evaluate two types of coding benchmarks: SWE tasks for in-domain performance and Terminal-Bench for out-of-distribution robustness. For SWE-smith-MR, we intentionally hold out issues from the same repositories to test generalization to unseen issues in familiar environments. In contrast, Terminal-Bench contains diverse long-horizon tasks, with training and test instances drawn from entirely different tasks, providing a broader test of instance-specific adaptation.


Q3. Do the coding gains persist on held-out repositories or under a larger execution budget?

Yes! We evaluated Turbo Harness on SWE-bench Verified with Haiku 4.5 under a larger 250-step budget, using the editor trained with 40 steps without retraining.

Method40 steps250 steps
Meta Harness56.759.3
Turbo Harness (Ours)59.361.3

Turbo Harness maintains its advantage over Meta-Harness at 250 steps, demonstrating generalization beyond the training-time execution budget.

Thank you again for your feedback. As the discussion period is coming to a close, we wanted to follow up on our rebuttal and would be happy to clarify any points if helpful.

I thank authors for their response and additional experiment results. I will retain my positive score.

Reviewer_B3ynOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Turbo Harness, an instance-adaptive harness optimization framework that extends prior global harness optimization approaches by adapting the harness to each individual task instance. The method reuses artifacts collected during a completed outer-loop harness search and summarizes them into a structured playbook containing successful and unsuccessful editing strategies. A lightweight RL-trained harness editor then conditions on the task instance, the playbook, and the globally optimized harness to generate instance-specific patches at inference time. Experiments on seven benchmarks, including SWE-bench Verified, SWE-smith-MR, Terminal-Bench-2.1, and several agentic benchmarks, show consistent improvements over Meta-Harness and prompt-based baselines in both task performance and, in several cases, execution efficiency.

Critical strengths
  1. The paper studies an important and timely problem in harness optimization. The motivation for instance-specific adaptation is intuitive and well justified.

  2. The written is clear and easy to follow.

  3. The paper includes useful ablations demonstrating that both RL training and playbook conditioning contribute to performance improvements.

Critical weaknesses
  1. The efficiency analysis remains incomplete. Although the paper reports reductions in execution steps and execution-model cost, these metrics exclude the cost of editor training, playbook construction, and outer-loop optimization. The full computational overhead of the approach is therefore unclear.

  2. Some baseline comparisons could be stronger. While the paper discusses related adaptive harness optimization approaches, comparisons against closely related methods such as [1] are missing from the main experimental section.

  3. The dependence on a high-quality outer-loop search archive may limit applicability in practice. The method assumes access to strong globally optimized harnesses and sufficiently rich search artifacts, which may require substantial compute and engineering effort.

[1] Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

Questions for authors

Please refer to weaknesses

2 comments

Thank you for your thoughtful review! We appreciate your recognition that our paper addresses an important and timely problem with an intuitive and well-justified motivation. We are also glad that you find our writing clear and easy to follow and our ablation studies useful. We address your concerns below.

W1. The efficiency analysis remains incomplete.

Below, we provide a breakdown of the computational cost of each component of our method. Specifically, we report the number of LLM API calls, GPU hours, and wall-clock time for the Gemini 3.7 Flash experiment on SWE-bench (similar observations hold across other experiments). As shown, the cost of Turbo Harness is comparable to Meta-Harness, while Turbo Harness achieves a higher pass rate, as detailed in Table 2 of our paper.

StageAPI CallsGPU hoursWall Clock Time
Meta-harness (outer-loop search)109k06.9h
Building the playbook (Ours)0.25k00.31h
Training editor (Ours)80k50.16.3h

W2. Some baseline comparisons could be stronger.

Agreed! Thank you for pointing out this new baseline. Below, we compare our method with AHE and Meta Harness. As a reminder, both AHE and Meta Harness produce a global harness through outer-loop search, whereas our Turbo Harness goes a step further by adapting the global harness to each individual instance. Accordingly, in the table below, we evaluate the performance of Turbo Harness when applied on top of the global harnesses generated by AHE and Meta Harness.

MethodScience World (Qwen 3.5 9B)SWE-bench Verified (Gemini Flash 3.7)
Default25.724.4
Meta Harness35.138.4
Meta Harness + Turbo Harness (Ours)42.454.4
AHE47.962.7
AHE + Turbo Harness (Ours)59.171.3

As shown, AHE achieves a higher score than Meta-Harness. Importantly, Turbo Harness further improves AHE from 47.9 to 59.1 on Science World, from 62.7 to 71.3 on SWE-bench Verified, outperforming both AHE alone and Meta-Harness + Turbo Harness reported in our Table 1. These results demonstrate that Turbo Harness is not tied to a specific global harness optimizer and can effectively leverage search artifacts from different optimization methods. We will incorporate these new results into the revised paper


W3. The dependence on a high-quality outer-loop search archive may limit applicability in practice.

You are right: outer-loop harness search does require substantial effort, as finding a strong harness is an extremely hard optimization problem. Currently, most approaches either rely on developer heuristics to manually design harnesses or use search-based methods which are effective but costly.

In fact, this work was motivated when we were considering a closely related question: Does outer-loop harness optimization fully leverage the substantial compute spent in the search process? We show that many artifacts generated during this process remain underutilized and can be reused to further improve the resulting harness. Importantly, our method is flexible and complementary to most global harness optimizations (e.g., in our response to your W2, we showed it can be combined with either AHE or Meta Harness to further improve their performance).

Thank you again for your feedback. As the discussion period is coming to a close, we wanted to follow up on our rebuttal and would be happy to clarify any points if helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.