acceptodds
Under review as a conference paper at ICLR 2027

Composing Task-specific Agent Harnesses at Test Time with Reusable Primitives

Abstract

Agent harnesses govern how large language models (LLMs) gather context, invoke tools, verify results, preserve state, and terminate, largely affecting agent performance. However, the value of each harness mechanism can differ across heterogeneous tasks: a mechanism that improves one task may impose overhead or context distraction on another, leading to the suboptimality of a global harness. We characterize this suboptimality as a mismatch induced by fixed mechanism choices, motivating task-specific harness construction. Nonetheless, generating harness code for each task introduces generation and debugging costs, with execution risks that can compound as more mechanisms are generated. To address those challenges, we introduce **Harness Primitives**, reusable harness mechanisms with clear application scope and composition contract mined from failed task trajectories. Based on Harness Primitives, we propose **STITCH**, a framework that **S**elects suitable primitives given **T**ask **I**nformation and compiles them into **T**ask-spe**C**ific **H**arnesses at test time. This separation enables task-specific harnesses without generating or repairing mechanism code at test time. Extensive experiments demonstrate that STITCH not only improves harness adaptability and robustness, but also scales with the primitive library size, boosting task success rates by up to 12 points over fixed harness baselines, surpassing human-designed harnesses like Codex CLI while maintaining a minimal test-time harness composition overhead of only 2.7%, 638 times more efficient than generating task-specific harnesses from scratch. Ultimately, our work demonstrates that building task-adaptive harnesses can be beneficial for completing diverse tasks and that building reusable primitives can be a promising path towards this goal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Harness, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jor1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes STITCH, a framework for constructing task-specific agent harnesses through reusable Harness Primitives. The primitives are developed offline from recurring failures in development trajectories, and each primitive is described by an application scope and a composition contract. At test time, an LLM-based composer selects a subset of primitives for a given task, while a deterministic compiler handles dependency resolution, graph construction, and validity checks. The resulting harness is then used by the task-solving actor.

The central idea is to obtain task-specific adaptation without generating or debugging an entire harness from scratch at test time. Experiments on SWE-bench Verified and Terminal-Bench 2 show consistent improvements over the seed harness and several fixed-harness baselines, with relatively small online composition overhead.

Critical strengths

The paper addresses a relevant and increasingly important problem. The observation that a fixed harness can be suboptimal across heterogeneous tasks is intuitive, but the paper gives it a useful mechanism-level formulation and connects it to concrete execution failures.

The separation between primitive selection and primitive implementation is a sensible design choice. Having the language model produce a composition intent while delegating dependency handling and graph wiring to a deterministic compiler is more robust than asking the model to generate an entire executable harness. The scope and contract abstraction also provides a reasonably clear interface for reusable harness mechanisms.

The experimental results are promising. STITCH improves Pass@1 from 73.0% to 80.5% on SWE-bench and from 60.0% to 72.2% on Terminal-Bench, and the cross-model and cross-domain experiments provide some evidence that the approach is not entirely tied to one actor model. The library-size analysis and the sparse primitive-selection pattern are also useful for supporting the proposed task-adaptive formulation.

Critical weaknesses

The main limitation is the experimental positioning. The paper discusses test-time harness evolution methods such as TTHE and compositional systems such as HarnessX, but does not provide a direct comparison with either of them. These are substantially closer baselines than fixed harnesses or fixed individual primitives. The current experiments therefore show that STITCH is effective relative to the selected baselines, but do not establish whether frozen primitive composition is preferable to trace-driven harness evolution in terms of success rate, adaptability, or total cost.

It is also difficult to isolate where the reported gains come from. Some improvement may be due to the individual mechanisms themselves rather than the task-conditioned composer. The random-composer and fixed-primitive baselines are useful, but an oracle selector and ablations of the composer, composition contracts, and deterministic compiler would make the contribution of task-specific selection much clearer. The reported 638× cost reduction is also somewhat optimistic, since it relies on an idealized code-generation lower bound and excludes the offline cost of developing and maintaining the primitive library.

Questions for authors

Can the authors provide a direct comparison with TTHE and HarnessX under matched task sets, actor models, execution budgets, and total token/API costs? In particular, it would be useful to separate the benefit of selecting frozen primitives from the benefit of allowing the harness to evolve from execution traces.

The proposed Composer–Compiler design seems potentially well suited to a plugin-oriented runtime such as DeepSeek Harness. Would implementing the primitives and compiler on such a runtime change the reliability, composition overhead, or ability to recompose the harness during execution? A comparison could clarify whether the contribution is tied to Mini-SWE-agent or reflects a more general runtime abstraction.

How much of the final performance gain comes from the primitive library itself, and how much comes from the task-conditioned selector? An oracle selector, a learned or heuristic selector, and ablations removing contract-guided compilation would help answer this.

How were the application scopes and composition contracts created? Please clarify the amount of human involvement, the number of candidate primitives considered and rejected, and the total offline development cost. This is important for assessing whether the proposed library can scale beyond the small collections evaluated in the paper.

6 comments
AuthorsOpenReview ↗

Response to Reviewer jor1 (Part 1/4)

We thank Reviewer jor1 for the thoughtful review and positive assessment. We appreciate your recognition that the paper addresses a relevant and increasingly important problem with a useful mechanism-level formulation. We address each weakness and question below:

Weakness 1 & Question 1: Comparison with TTHE and HarnessX

Weakness 1: The main limitation is the experimental positioning. The paper discusses test-time harness evolution methods such as TTHE and compositional systems such as HarnessX, but does not provide a direct comparison with either of them. These are substantially closer baselines than fixed harnesses or fixed individual primitives. The current experiments therefore show that STITCH is effective relative to the selected baselines, but do not establish whether frozen primitive composition is preferable to trace-driven harness evolution in terms of success rate, adaptability, or total cost.

Question 1: Can the authors provide a direct comparison with TTHE and HarnessX under matched task sets, actor models, execution budgets, and total token/API costs? In particular, it would be useful to separate the benefit of selecting frozen primitives from the benefit of allowing the harness to evolve from execution traces.

Following the reviewer's advice, we additionally add TTHE as a baseline. As shown in the following table,

MethodOverall: Pass@1Pass@2Pass²Django: Pass@1Pass@2Pass²SymPy: Pass@1Pass@2Pass²Other repos: Pass@1Pass@2Pass²
Mini-SWE-agent (Seed Harness)73.078.068.075.078.871.275.078.671.469.176.561.8
SWE-agent47.563.032.053.871.236.542.950.035.739.755.923.5
Codex CLI79.083.075.078.880.876.964.371.457.185.391.279.4
MemoHarness70.577.064.070.276.963.567.971.464.372.179.464.7
MetaHarness73.579.068.074.078.869.264.371.457.176.582.470.6
TTHE72.077.067.070.275.065.467.978.657.176.579.473.5
Our developed primitives (fixed across tasks)
Change-surface Tracer73.577.070.076.980.873.160.764.357.173.576.570.6
Compatibility-envelope Gate72.077.067.076.980.873.157.164.350.070.676.564.7
Contract-case Explorer72.578.067.069.276.961.571.478.664.377.979.476.5
Discriminating-oracle Runner71.576.067.075.078.871.257.157.157.172.179.464.7
Domain-trace Template73.077.069.076.080.871.260.764.357.173.576.570.6
Environment-capability Runner76.079.073.077.980.875.067.971.464.376.579.473.5
Uniform random composer73.378.068.675.479.671.264.369.658.973.979.068.8
STITCH (Ours)80.585.076.079.884.675.075.078.671.483.888.279.4

As we can see from the table above, the proposed method outperforms the trace-driven harness evolution baseline TTHE by +8.5 points. MemoHarness is also a trace-driven harness evolution baseline, which STITCH outperforms by +10 points.

We thank the reviewer for the feedback and have added the experiment above in Appendix G.2 of the revised paper, marked in blue for your review.

AuthorsOpenReview ↗

Response to Reviewer jor1 (Part 2/4)

Weakness 2 & Question 3: Ablations on Composer and Compiler

Weakness 2: It is also difficult to isolate where the reported gains come from. Some improvement may be due to the individual mechanisms themselves rather than the task-conditioned composer. The random-composer and fixed-primitive baselines are useful, but an oracle selector and ablations of the composer, composition contracts, and deterministic compiler would make the contribution of task-specific selection much clearer.

Question 3: How much of the final performance gain comes from the primitive library itself, and how much comes from the task-conditioned selector? An oracle selector, a learned or heuristic selector, and ablations removing contract-guided compilation would help answer this.

As suggested, we first added two baselines mentioned by the reviewer:

  • Oracle Selector: select the best-performing single-primitive harness for each task, serving as an upper bound baseline.
  • Heuristic Selector: select the best-performing single primitive for each type of task (Categorized by repository name for SWE Bench and difficulty level for Terminal Bench). This is also a baseline with oracle information
MethodSWE: Pass@1Pass@3Pass³Terminal: Pass@1Pass@3Pass³
Seed harness72.2 ± 3.194.048.849.5 ± 2.384.818.2
Codex CLI77.9 ± 2.295.059.047.5 ± 2.378.821.2
Best fixed harness72.8 ± 3.493.849.855.1 ± 3.286.425.8
All on67.3 ± 0.888.244.252.0 ± 1.783.322.7
Heuristic Selector75.4 ± 4.494.253.857.6 ± 2.686.428.8
Oracle Selector85.0 ± 2.697.869.868.7 ± 5.392.445.5
STITCH79.3 ± 2.396.059.862.6 ± 4.489.436.4

As shown in the table above, Oracle Selector shows a +21.2 points advantage over Codex CLI, demonstrating the potential of task-specific harnesses.

Heuristic Selector, on the other hand, is 2.6 points over the best single-primitive baseline despite using oracle information. Our application-scope-based composer design is +5 points higher compared to heuristic alternatives, showing the performance gain mainly comes from composing task-specific harnesses rather than individual mechanisms.

As requested by the reviewer, we conduct ablation studies on the contract-guided compilation by comparing it with an alternative agent-based integrator implementation: given the raw code of the primitives, the agent is asked to integrate the mechanism into the base harness.

BenchmarkAgent integration: first passAgent integration: within 50 callsInitial integration (k tokens)Feedback debugging (k tokens)Total integration (k tokens)
SWE-bench2/65/6162.9253.2416.1
Terminal-Bench0/55/5189.1441.9631.1

As shown in the table above, the agent-based integrator integrated the mechanisms into the base harness with a 10/11 success rate, despite its 631.1k token cost, while the proposed deterministic compiler design integrates the mechanisms with negligible cost and 0 failures in all of our experiments.

As suggested, these baselines and the compiler ablation are now included in Appendices G.1 and H of the revised paper, highlighted in blue for your review.

AuthorsOpenReview ↗

Response to Reviewer jor1 (Part 3/4)

Weakness 2: Cost Reduction and Offline Library Cost

Weakness 2: The reported 638× cost reduction is also somewhat optimistic, since it relies on an idealized code-generation lower bound and excludes the offline cost of developing and maintaining the primitive library.

We provide the offline cost of developing the primitives as follows, which is about $400 in total API cost.

Pipeline stageEstimated inference tokens
Trajectory collection≈110M
Primitive proposal≈120M
Primitive implementation≈400M
Primitive validation and revision≈570M
Total≈1.2B

To make the test-time cost comparison more practical, we compare the cost of STITCH and a test-time code-generation baseline that adds mechanisms to the base harness instead of generating from scratch. We provide the specifications of the primitives and ask the agent to add the corresponding mechanism to the base harness.

Benchmark libraryInitial generation (k tokens)debugging (k tokens)Generation total (k tokens)STITCH (k tokens/task)Token ratio
SWE-bench127.6247.9375.52.044183.7×
Terminal-Bench96.8403.3500.12.042244.9×

As shown above, compared to a test-time code-generation baseline that only incrementally implements task-specific logic, STITCH is 244.9 times more cost-efficient due to the high debugging cost (4 times that of initial code generation) of code-generation baselines.

We appreciate this comment; the cost analyses above have been added to Appendices H and I of the revised paper, marked in blue for your reference.

Question 2: Implementation on a Plugin-Oriented Runtime

Question 2: The proposed Composer–Compiler design seems potentially well suited to a plugin-oriented runtime such as DeepSeek Harness. Would implementing the primitives and compiler on such a runtime change the reliability, composition overhead, or ability to recompose the harness during execution? A comparison could clarify whether the contribution is tied to Mini-SWE-agent or reflects a more general runtime abstraction.

We agree that a plugin-oriented runtime such as DeepSeek Harness can be well-suited for the Composer–Compiler design, as the plugin marketplace can be viewed as a natural source of the primitive library.

Without empirical evidence available, our analysis of implementing the Composer–Compiler design on a plugin-oriented runtime is as follows:

  • Composition overhead. The Composer design carries over directly. However, a production plugin marketplace may contain thousands of plugins, far more than the small libraries evaluated in our paper, so their application scopes can no longer all fit into a single Composer prompt. An additional retrieval stage that first shortlists candidate plugins by their application scopes would therefore be needed. Given a reliable retriever, the Composer only reads a fixed-size shortlist, so the cost of its LLM call should stay roughly constant as the library grows.
  • Compilation reliability and mid-execution recomposition. On a plugin-oriented runtime, compiling a composition reduces to installing the selected plugins through the runtime's existing plugin interface, which is typically well-supported and well-tested. We therefore expect compilation to be at least as reliable as in our Mini-SWE-agent implementation, and recomposing the harness during execution becomes a matter of loading or unloading plugins. Composition contracts would remain useful for checking that co-selected plugins do not conflict with each other.

So, for whether the contribution is tied to Mini-SWE-agent: The proposed method is not tied to specific harnesses and uses nothing specific to Mini-SWE-agent. As formally analyzed in Proposition 1, the contribution is a runtime abstraction that resolves the task-mechanism mismatch problem with selective mechanism activation. What would make a difference, however, is whether the base harness is cleanly implemented, which we believe is the case for DeepSeek Harness, and would make extending the harness with new mechanisms easier. Nonetheless, this is a separate issue of code extensibility.

Reviewer_dc5KOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that a fixed agent harness may not work equally well across tasks, as some harness mechanisms can help certain tasks but hurt others. To address this, it proposes STITCH, which builds task-specific harnesses by selecting and combining pre-built, validated primitives rather than generating harness code at test time. These primitives are developed offline using GPT-5.6-Sol, which analyzes failures of a seed harness (Mini-SWE-agent), implements potential improvements, and retains those that improve performance in matched runs. Each primitive is assigned an application scope and a composition contract. At test time, a composer selects suitable primitives for each task, while a deterministic compiler integrates them into the seed harness, with a fallback if validation fails. The paper also provides theoretical results showing when task-specific composition can outperform fixed harnesses and one-shot code generation (Sec. 3).

Experiments cover 100 SWE-bench Verified tasks and 45 Terminal-Bench 2 tasks, using GPT-5.6-Luna as the actor with two attempts per task. STITCH achieves 80.5% Pass@1 on SWE-bench, compared with 73.0% for the seed harness and 79.0% for Codex CLI. On Terminal-Bench, it reaches 72.2%, compared with 60.0% and 56.7%, respectively. It also outperforms individual fixed primitives and a random composer (Tabs. 1–2). Additional experiments examine transfer across actor models (Tab. 3), transfer of primitives between benchmarks (Fig. 3), the effect of library size (Fig. 2), and composition overhead, reported as 4.3% and 2.7% of one actor attempt (Fig. 4).

Overall, the paper presents an interesting and practical direction for making agent harnesses more adaptive and reusable. Its main contribution is shifting harness optimization from a fixed, one-size-fits-all design toward task-specific composition of validated mechanisms. The empirical results are promising, although the evaluation remains relatively small, and broader testing would help establish how well the approach generalizes.

Critical strengths

Two main strengths of the paper I can identify are as follows:

1. A practical design that addresses a real problem. The idea of developing and validating harness mechanisms offline, while limiting the online stage to selecting and composing them, is sensible. It avoids the cost and potential instability of generating harness code for every task. The application scopes and composition contracts also provide useful guidance for selection. The random-composer baseline is particularly valuable: it performs close to the average fixed primitive (73.3% on SWE-bench and 63.5% on Terminal-Bench), while STITCH achieves 80.5% and 72.2%, respectively. This suggests that task-specific selection, rather than simply having more mechanisms available, contributes to the gains.

2. Promising empirical results across multiple settings. STITCH outperforms the seed harness across all four actor models and every individual fixed primitive on both benchmarks. The primitives also show some transferability across benchmarks (Fig. 3), suggesting that these mechanisms may be reusable beyond their original development setting. These results are encouraging and provide initial evidence for the value of task-specific harness selection, although their statistical reliability and generalizability still require further validation.

Critical weaknesses

Two main weaknesses of the paper I can identify are as follows:

1. The evaluation is too limited to support strong conclusions. The experiments cover only 100 SWE-bench tasks and 45 Terminal-Bench tasks, with two attempts per task and no confidence intervals. Several reported improvements are quite small in absolute terms. For example, STITCH achieves 161 successful attempts out of 200 on SWE-bench, compared with 158 for Codex CLI. The results are also not fully consistent across settings, as Codex CLI outperforms STITCH when GPT-5.6-Terra is used as the actor (Tab. 3). Given the small sample sizes and limited uncertainty analysis, it is difficult to determine whether STITCH consistently outperforms Codex CLI or whether increasing the primitive library size genuinely improves performance.

2. The actual use of composition is limited, and important baselines are missing. On roughly half of the tasks, the composer selects no primitive, while only 10 SWE-bench tasks and no Terminal-Bench tasks use two or more primitives (App. F). This suggests that STITCH mainly functions as a task-specific router rather than a mechanism composition framework. While this is still a reasonable approach, the paper lacks two important comparisons: a harness with all primitives enabled and a simpler task-specific routing baseline. The reported 638× efficiency improvement is also somewhat misleading, as it compares against generating an entire harness from scratch, which is not representative of the cited methods, and excludes the offline cost of developing the primitive library.

Overall, these concerns mainly relate to the strength of the empirical evidence and whether the proposed composition framework offers meaningful advantages over simpler alternatives. I would consider raising my score if the authors could demonstrate statistically reliable improvements on a larger evaluation set, and show that STITCH consistently outperforms both an all-primitives harness and a simple task-specific router.

Questions for authors
  1. Statistical reliability: Can the authors report paired confidence intervals for the results in Tabs. 1–3? It would also be useful to examine the tasks where the composer selects no primitive (51 on SWE-bench and 23 on Terminal-Bench). Since STITCH uses the same harness as the seed baseline on these tasks, any performance differences should largely reflect run-to-run variability.

  2. Missing baselines: How does STITCH compare with a harness that enables all primitives, and with a simpler per-task router or an oracle that selects among the single-primitive harnesses? These comparisons would help clarify how much of the improvement comes from composition rather than task-specific selection alone.

  3. Total cost and efficiency: What is the overall cost of STITCH when offline primitive development and validation are included? How does this compare with an actual test-time code-generation approach, such as JIT-Agent, evaluated on the same tasks? This would help establish whether the reported efficiency gains hold under a more realistic comparison.

11 comments
AuthorsOpenReview ↗

Response to Reviewer dc5K (Part 1/4)

We sincerely thank Reviewer dc5K for the detailed and constructive review. We appreciate the reviewer's recognition of STITCH as a practical design that addresses a real problem. We are also glad that you found the results across actor models and the cross-benchmark transferability encouraging. We address each weakness and question below:

Weakness 1 & Question 1: Statistical Reliability of the Results

Weakness 1: The evaluation is too limited to support strong conclusions. The experiments cover only 100 SWE-bench tasks and 45 Terminal-Bench tasks, with two attempts per task and no confidence intervals. Several reported improvements are quite small in absolute terms. For example, STITCH achieves 161 successful attempts out of 200 on SWE-bench, compared with 158 for Codex CLI. The results are also not fully consistent across settings, as Codex CLI outperforms STITCH when GPT-5.6-Terra is used as the actor (Tab. 3). Given the small sample sizes and limited uncertainty analysis, it is difficult to determine whether STITCH consistently outperforms Codex CLI or whether increasing the primitive library size genuinely improves performance.

Question 1: Statistical reliability: Can the authors report paired confidence intervals for the results in Tabs. 1–3? It would also be useful to examine the tasks where the composer selects no primitive (51 on SWE-bench and 23 on Terminal-Bench). Since STITCH uses the same harness as the seed baseline on these tasks, any performance differences should largely reflect run-to-run variability.

Following the reviewer's advice, we expanded our experiments. Specifically, we expanded the SWE-Bench Verified evaluation to a full set of 400 tasks (100 tasks for development), with 3 trials per task, summing up to 1200 trials in total. The performance on the full SWE-Bench Verified is as follows, along with its standard deviation.

MethodOverall: Pass@1Pass@3Pass³Django: Pass@1Pass@3Pass³SymPy: Pass@1Pass@3Pass³Other repos: Pass@1Pass@3Pass³
Mini-SWE-agent (Seed Harness)72.2 ± 3.194.048.875.1 ± 1.094.853.265.6 ± 9.288.345.071.7 ± 4.295.245.5
SWE-agent52.2 ± 1.780.824.054.5 ± 1.282.127.749.4 ± 6.973.323.350.7 ± 1.982.020.4
Codex CLI77.9 ± 2.295.059.079.6 ± 1.895.461.370.0 ± 6.090.051.779.0 ± 3.296.459.3
MemoHarness68.4 ± 2.791.842.571.9 ± 0.994.245.161.7 ± 12.683.338.367.3 ± 4.492.241.3
MetaHarness71.6 ± 1.993.048.573.6 ± 1.294.250.363.9 ± 8.285.043.372.3 ± 2.894.648.5
Our fixed harnesses
Change-surface Tracer70.8 ± 2.493.045.073.0 ± 1.294.246.862.8 ± 7.985.040.071.3 ± 2.294.644.9
Compatibility-envelope Gate70.8 ± 3.291.047.875.0 ± 1.593.154.362.8 ± 11.181.740.069.5 ± 2.692.243.7
Contract-case Explorer72.3 ± 2.893.847.875.5 ± 2.795.450.968.3 ± 7.688.348.370.5 ± 1.994.044.3
Discriminating-oracle Runner71.2 ± 4.091.547.874.4 ± 2.693.650.363.9 ± 10.883.343.370.5 ± 3.892.246.7
Domain-trace Template71.4 ± 2.993.546.575.5 ± 1.996.550.965.0 ± 10.983.341.769.5 ± 2.194.043.7
Environment-capability Runner72.5 ± 3.193.248.878.0 ± 3.596.056.163.3 ± 11.786.738.370.1 ± 1.892.844.9
All on67.3 ± 0.888.244.272.8 ± 2.092.549.161.1 ± 1.078.341.763.9 ± 1.487.440.1
Uniform selection
Uniform random composer71.3 ± 2.892.447.475.0 ± 1.994.551.664.1 ± 8.684.842.070.0 ± 2.393.044.9
Oracle references
Heuristic Selector75.4 ± 4.494.253.878.0 ± 3.596.056.168.3 ± 7.688.348.375.2 ± 4.594.653.3
Oracle Selector85.0 ± 2.697.869.888.4 ± 2.199.474.680.0 ± 2.995.063.383.2 ± 3.297.067.1
Task-adaptive composition
STITCH79.3 ± 2.396.059.883.8 ± 1.598.364.772.8 ± 5.190.055.077.0 ± 2.395.856.3

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer dc5K (Part 2/4)

Weakness 1 & Question 1 (cont.): Statistical Reliability of the Results

We additionally evaluated the method on Terminal Bench 4 (66 tasks x 3 trials) for stronger empirical evidence. The results, along with standard deviation, are as follows.

MethodOverall: Pass@1Pass@3Pass³Easy: Pass@1Pass@3Pass³Medium: Pass@1Pass@3Pass³Hard: Pass@1Pass@3Pass³
Mini-SWE-agent49.5 ± 2.384.818.284.8 ± 11.4100.054.545.5 ± 9.1100.00.018.2 ± 9.154.50.0
SWE-agent27.8 ± 3.560.63.051.5 ± 5.290.99.121.2 ± 6.959.10.010.6 ± 11.431.80.0
Codex CLI47.5 ± 2.378.821.283.3 ± 13.9100.054.539.4 ± 9.581.89.119.7 ± 10.554.50.0
MemoHarness41.9 ± 3.274.215.274.2 ± 9.5100.036.433.3 ± 11.477.34.518.2 ± 9.145.54.5
MetaHarness45.5 ± 4.077.313.674.2 ± 13.1100.031.843.9 ± 17.286.49.118.2 ± 7.945.50.0
Our fixed harnesses
State Guard52.0 ± 1.783.322.784.8 ± 18.4100.054.548.5 ± 5.295.59.122.7 ± 7.954.54.5
Check Replay50.5 ± 5.381.824.286.4 ± 12.0100.059.142.4 ± 14.690.99.122.7 ± 12.054.54.5
First-Failure Localizer53.0 ± 5.284.822.783.3 ± 18.4100.050.042.4 ± 18.986.49.133.3 ± 6.968.29.1
Experiment Keeper49.5 ± 3.881.821.280.3 ± 14.6100.045.547.0 ± 11.490.913.621.2 ± 6.954.54.5
Execution Supervisor55.1 ± 3.286.425.883.3 ± 13.9100.054.553.0 ± 13.990.918.228.8 ± 9.568.24.5
Uniform selection
Uniform random composer51.6 ± 2.983.822.583.8 ± 14.6100.053.046.5 ± 11.692.49.824.5 ± 8.459.14.5
Oracle references
Heuristic Selector57.6 ± 2.686.428.886.4 ± 12.0100.059.153.0 ± 13.990.918.233.3 ± 6.968.29.1
Oracle Selector68.7 ± 5.392.445.595.5 ± 4.5100.086.468.2 ± 12.0100.031.842.4 ± 6.977.318.2
Task-adaptive composition
STITCH62.6 ± 4.489.436.490.9 ± 12.0100.072.760.6 ± 9.595.522.736.4 ± 12.072.713.6

As we can see, STITCH consistently outperforms existing baselines, including Codex (+8.25 pp), on an extended sample size with statistical significance (Extended 400 tasks on SWE-Bench Verified and the additional TerminalBench 4 benchmark, each task with 3 trials).

We address the cases where no primitives are selected by the composer in our following response to Weakness 2 & Question 2, with concrete examples and empirical results on additional baselines.

We thank the reviewer for the feedback and have added the experiments above in Section 5.2 of the revised paper, marked in blue for your review.

AuthorsOpenReview ↗

Response to Reviewer dc5K (Part 3/4)

Weakness 2 & Question 2: All-Primitives and Router/Oracle Baselines

Weakness 2: The actual use of composition is limited, and important baselines are missing. On roughly half of the tasks, the composer selects no primitive, while only 10 SWE-bench tasks and no Terminal-Bench tasks use two or more primitives (App. F). This suggests that STITCH mainly functions as a task-specific router rather than a mechanism composition framework. While this is still a reasonable approach, the paper lacks two important comparisons: a harness with all primitives enabled and a simpler task-specific routing baseline.

Question 2: Missing baselines: How does STITCH compare with a harness that enables all primitives, and with a simpler per-task router or an oracle that selects among the single-primitive harnesses? These comparisons would help clarify how much of the improvement comes from composition rather than task-specific selection alone.

As suggested, we added all three baselines mentioned by the reviewer:

  • Oracle: select the best-performing single-primitive harness for each task, serving as an upper-bound baseline.
  • Simple per-task router: select the best-performing single primitive for each type of task (Categorized by repository name for SWE Bench and difficulty level for Terminal Bench). This is also a baseline with oracle information.
  • All On: A fixed harness with the maximum number of compatible primitives enabled.
Method (Luna)SWE: Pass@1Pass@3Pass³Terminal: Pass@1Pass@3Pass³
Seed harness72.2 ± 3.194.048.849.5 ± 2.384.818.2
Codex CLI77.9 ± 2.295.059.047.5 ± 2.378.821.2
Best fixed harness72.8 ± 3.493.849.855.1 ± 3.286.425.8
All on67.3 ± 0.888.244.252.0 ± 1.783.322.7
Simple per-task router75.4 ± 4.494.253.857.6 ± 2.686.428.8
Oracle Selector85.0 ± 2.697.869.868.7 ± 5.392.445.5
STITCH79.3 ± 2.396.059.862.6 ± 4.489.436.4

As shown in the table above,

  • Oracle shows a +5.9-point advantage over the second-best method, demonstrating the potential of task-specific harnesses.
  • A simple per-task router is 2.6 points over the best single-primitive baseline despite using oracle information. Our application-scope-based composer design is +5 points higher compared to the simple per-task alternative.
  • All On shows lower performance compared to the single primitive baselines. This empirical evidence shows that redundant mechanisms in harnesses not only do not improve performance, but may lead to cognitive burden for the model and thus performance degradation (-5.5 pp compared to the best fixed harness on SWE Bench Verified), supporting sparse harness mechanism activation.

To demonstrate why sparsely selected primitives are intentional and preferable when the base harness already suffices, we provide a concrete example of when no primitives are selected according to application scopes:

Task "rstan-to-pystan" in Terminal Bench requires translating an R analysis script into Python, executing the analysis, and saving numerical results. None of the five available primitives meets its application conditions. First-Failure Localizer requires executable reference and candidate programs, but the task explicitly forbids running the original R script. Check Replay has no supplied correctness-check command to replay. Experiment Keeper requires a numerical objective for comparing improved candidates, whereas this task asks to reproduce a specified analysis. Execution Supervisor is unnecessary because the analysis terminates without leaving a persistent service. State Guard has no required destructive operation to guard against: source files are read, and results are written separately. The composer therefore selects no additional primitive, and both base harness attempts succeed. This exemplifies how the proposed composer applies harness mechanisms only when necessary to avoid the model cognitive burden from unnecessary mechanisms.

As suggested, these baselines and the accompanying discussion are now included in Section 5.2 and Appendices F and G.1 of the revised paper, highlighted in blue.

Reviewer_XUrZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1(−1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes STITCH, a framework that builds task specific agent harnesses from reusable primitives. An LLM selects suitable primitives, and a deterministic compiler connects them into an executable harness. Experiments on SWE-bench Verified and Terminal-Bench 2 report improvements over fixed harnesses with low composition overhead.

Critical strengths

The design is practical, and the results on two benchmarks are encouraging.

Critical weaknesses
  1. It is not obvious why we need to compile a separate harness for every task. A single fixed pipeline could reuse the same selectors and primitives, and just use conditionals to toggle which parts execute. The paper doesn’t include this comparison.

  2. The 638× cost number assumes building the whole harness from the ground up. A real world generator could reuse the existing runtime and only emit task specific logic. I don’t see how much better the proposed approach is against this setup.

  3. The contribution seems fairly incremental. The experiments haven’t convincingly shown enough benefits over existing harness designs to merit acceptance.

Questions for authors
  1. Using the same selector, primitives, and execution budget, how does STITCH compare with a fixed workflow that conditionally enables components?

  2. How much cost does STITCH save against a code generation baseline that reuses the runtime and generates only task specific logic?

3 comments
AuthorsOpenReview ↗

Response to Reviewer XUrZ (Part 1/3)

We thank Reviewer XUrZ for the feedback. We are glad that you found the design of STITCH practical and the results on two benchmarks encouraging. We address each weakness and question below:

Weakness 1 & Question 1: Comparison with "a Fixed Workflow that Conditionally Enables Components"

Weakness 1: It is not obvious why we need to compile a separate harness for every task. A single fixed pipeline could reuse the same selectors and primitives, and just use conditionals to toggle which parts execute. The paper doesn't include this comparison.

Question 1: Using the same selector, primitives, and execution budget, how does STITCH compare with a fixed workflow that conditionally enables components?

Since the reviewer asked for a comparison with "the same selector, primitives, and execution budget" and "a fixed workflow that conditionally enables components" is the proposed methodology and idea of STITCH, the comparison could only come from the implementation difference in the compiler module, an integral part of this pipeline that combines the selected components into an executable task-specific harness.

We conduct an ablation study on the deterministic compiler part with an alternative agent-based integrator implementation: given the raw code of the primitives, the agent is asked to integrate the mechanism into the base harness

BenchmarkAgent integration: first passAgent integration: within 50 callsInitial integration (k)Feedback debugging (k)Total integration (k)
SWE-bench2/65/6162.9253.2416.1
Terminal-Bench0/55/5189.1441.9631.1

As shown in the table above, the agent-based integrator integrated the mechanisms into the base harness with a 10/11 success rate, despite its 631.1k tokens of cost, while the proposed deterministic compiler design integrates the mechanisms with negligible cost and 0 failures in all of our experiments.

We thank the reviewer for the feedback and have added the experiment above in Appendix H of the revised paper, marked in blue for your review.

Weakness 2 & Question 2: Cost Savings Against a Code Generator

Weakness 2: The 638× cost number assumes building the whole harness from the ground up. A real world generator could reuse the existing runtime and only emit task specific logic. I don't see how much better the proposed approach is against this setup.

Question 2: How much cost does STITCH save against a code generation baseline that reuses the runtime and generates only task specific logic?

Following the reviewer’s advice, we test the cost savings of STITCH compared to a code generator that reuses a base harness runtime and generates only the task-specific logic. Specifically, we provide the descriptions of the mechanisms that underlie the primitives and ask the code generator to implement them on the base harness. The total cost of the generator, as well as its decomposition into code generation and debugging, is as follows

Benchmark libraryInitial generation (k)debugging (k)Generation total (k)STITCH (k)Token ratio
SWE-bench127.6247.9375.52.044183.7×
Terminal-Bench96.8403.3500.12.042244.9×

As shown above, compared to a test-time code-generation baseline that only incrementally implements task-specific logic instead of from scratch, STITCH is 244.9 times more cost-efficient, due to the high debugging cost (4 times that of initial code generation on Terminal Bench) of code generation baselines.

Following the reviewer's suggestion, this comparison is now included in Appendix H of the revised paper, highlighted in blue for your review.

AuthorsOpenReview ↗

Response to Reviewer XUrZ (Part 2/3)

Weakness 3: Significance of the Contribution

Weakness 3: The contribution seems fairly incremental. The experiments haven't convincingly shown enough benefits over existing harness designs to merit acceptance.

To demonstrate the empirical benefit of the proposed method, we expanded our experiments to the full SWE Verified benchmark with 3 trials per task, with a gentle note that "a fixed workflow that conditionally enables components" is our technical contribution described in a simplified way, formally defined in Section 3, with no similar designs in existing works, as discussed in Section 2, or products like Codex CLI or Claude Code.

MethodOverall: Pass@1Pass@3Pass³Django: Pass@1Pass@3Pass³SymPy: Pass@1Pass@3Pass³Other repos: Pass@1Pass@3Pass³
Mini-SWE-agent (Seed Harness)72.2 ± 3.194.048.875.1 ± 1.094.853.265.6 ± 9.288.345.071.7 ± 4.295.245.5
SWE-agent52.2 ± 1.780.824.054.5 ± 1.282.127.749.4 ± 6.973.323.350.7 ± 1.982.020.4
Codex CLI77.9 ± 2.295.059.079.6 ± 1.895.461.370.0 ± 6.090.051.779.0 ± 3.296.459.3
MemoHarness68.4 ± 2.791.842.571.9 ± 0.994.245.161.7 ± 12.683.338.367.3 ± 4.492.241.3
MetaHarness71.6 ± 1.993.048.573.6 ± 1.294.250.363.9 ± 8.285.043.372.3 ± 2.894.648.5
Our fixed harnesses
Change-surface Tracer70.8 ± 2.493.045.073.0 ± 1.294.246.862.8 ± 7.985.040.071.3 ± 2.294.644.9
Compatibility-envelope Gate70.8 ± 3.291.047.875.0 ± 1.593.154.362.8 ± 11.181.740.069.5 ± 2.692.243.7
Contract-case Explorer72.3 ± 2.893.847.875.5 ± 2.795.450.968.3 ± 7.688.348.370.5 ± 1.994.044.3
Discriminating-oracle Runner71.2 ± 4.091.547.874.4 ± 2.693.650.363.9 ± 10.883.343.370.5 ± 3.892.246.7
Domain-trace Template71.4 ± 2.993.546.575.5 ± 1.996.550.965.0 ± 10.983.341.769.5 ± 2.194.043.7
Environment-capability Runner72.5 ± 3.193.248.878.0 ± 3.596.056.163.3 ± 11.786.738.370.1 ± 1.892.844.9
All on67.3 ± 0.888.244.272.8 ± 2.092.549.161.1 ± 1.078.341.763.9 ± 1.487.440.1
Uniform selection
Uniform random composer (9-way)71.3 ± 2.892.447.475.0 ± 1.994.551.664.1 ± 8.684.842.070.0 ± 2.393.044.9
Oracle references
Heuristic Selector75.4 ± 4.494.253.878.0 ± 3.596.056.168.3 ± 7.688.348.375.2 ± 4.594.653.3
Oracle Selector85.0 ± 2.697.869.888.4 ± 2.199.474.680.0 ± 2.995.063.383.2 ± 3.297.067.1
Task-adaptive composition
STITCH79.3 ± 2.396.059.883.8 ± 1.598.364.772.8 ± 5.190.055.077.0 ± 2.395.856.3

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer XUrZ (Part 3/3)

Weakness 3 (cont.): Significance of the Contribution

We additionally tested our method on TerminalBench 4, with 3 trials per task.

Method (66 tasks; 3 runs)Overall: Pass@1Pass@3Pass³Easy: Pass@1Pass@3Pass³Medium: Pass@1Pass@3Pass³Hard: Pass@1Pass@3Pass³
Mini-SWE-agent49.5 ± 2.384.818.284.8 ± 11.4100.054.545.5 ± 9.1100.00.018.2 ± 9.154.50.0
SWE-agent27.8 ± 3.560.63.051.5 ± 5.290.99.121.2 ± 6.959.10.010.6 ± 11.431.80.0
Codex CLI47.5 ± 2.378.821.283.3 ± 13.9100.054.539.4 ± 9.581.89.119.7 ± 10.554.50.0
MemoHarness41.9 ± 3.274.215.274.2 ± 9.5100.036.433.3 ± 11.477.34.518.2 ± 9.145.54.5
MetaHarness45.5 ± 4.077.313.674.2 ± 13.1100.031.843.9 ± 17.286.49.118.2 ± 7.945.50.0
Our fixed harnesses
State Guard52.0 ± 1.783.322.784.8 ± 18.4100.054.548.5 ± 5.295.59.122.7 ± 7.954.54.5
Check Replay50.5 ± 5.381.824.286.4 ± 12.0100.059.142.4 ± 14.690.99.122.7 ± 12.054.54.5
First-Failure Localizer53.0 ± 5.284.822.783.3 ± 18.4100.050.042.4 ± 18.986.49.133.3 ± 6.968.29.1
Experiment Keeper49.5 ± 3.881.821.280.3 ± 14.6100.045.547.0 ± 11.490.913.621.2 ± 6.954.54.5
Execution Supervisor55.1 ± 3.286.425.883.3 ± 13.9100.054.553.0 ± 13.990.918.228.8 ± 9.568.24.5
Uniform selection
Uniform random composer51.6 ± 2.983.822.583.8 ± 14.6100.053.046.5 ± 11.692.49.824.5 ± 8.459.14.5
Task-adaptive composition
STITCH62.6 ± 4.489.436.490.9 ± 12.0100.072.760.6 ± 9.595.522.736.4 ± 12.072.713.6

As shown in the table above, the proposed method outperforms the strongest baseline by +7.5 points on extended full SWE-Bench Verified and additional Terminal Bench 4. We are happy to include further methodological and empirical comparisons with any baselines the reviewer suggests.

The expanded results above have been incorporated into Section 5.2 of the revised paper, marked in blue for your reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.