- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes STITCH, a framework for constructing task-specific agent harnesses through reusable Harness Primitives. The primitives are developed offline from recurring failures in development trajectories, and each primitive is described by an application scope and a composition contract. At test time, an LLM-based composer selects a subset of primitives for a given task, while a deterministic compiler handles dependency resolution, graph construction, and validity checks. The resulting harness is then used by the task-solving actor.
The central idea is to obtain task-specific adaptation without generating or debugging an entire harness from scratch at test time. Experiments on SWE-bench Verified and Terminal-Bench 2 show consistent improvements over the seed harness and several fixed-harness baselines, with relatively small online composition overhead.
Critical strengths
The paper addresses a relevant and increasingly important problem. The observation that a fixed harness can be suboptimal across heterogeneous tasks is intuitive, but the paper gives it a useful mechanism-level formulation and connects it to concrete execution failures.
The separation between primitive selection and primitive implementation is a sensible design choice. Having the language model produce a composition intent while delegating dependency handling and graph wiring to a deterministic compiler is more robust than asking the model to generate an entire executable harness. The scope and contract abstraction also provides a reasonably clear interface for reusable harness mechanisms.
The experimental results are promising. STITCH improves Pass@1 from 73.0% to 80.5% on SWE-bench and from 60.0% to 72.2% on Terminal-Bench, and the cross-model and cross-domain experiments provide some evidence that the approach is not entirely tied to one actor model. The library-size analysis and the sparse primitive-selection pattern are also useful for supporting the proposed task-adaptive formulation.
Critical weaknesses
The main limitation is the experimental positioning. The paper discusses test-time harness evolution methods such as TTHE and compositional systems such as HarnessX, but does not provide a direct comparison with either of them. These are substantially closer baselines than fixed harnesses or fixed individual primitives. The current experiments therefore show that STITCH is effective relative to the selected baselines, but do not establish whether frozen primitive composition is preferable to trace-driven harness evolution in terms of success rate, adaptability, or total cost.
It is also difficult to isolate where the reported gains come from. Some improvement may be due to the individual mechanisms themselves rather than the task-conditioned composer. The random-composer and fixed-primitive baselines are useful, but an oracle selector and ablations of the composer, composition contracts, and deterministic compiler would make the contribution of task-specific selection much clearer. The reported 638× cost reduction is also somewhat optimistic, since it relies on an idealized code-generation lower bound and excludes the offline cost of developing and maintaining the primitive library.
Questions for authors
Can the authors provide a direct comparison with TTHE and HarnessX under matched task sets, actor models, execution budgets, and total token/API costs? In particular, it would be useful to separate the benefit of selecting frozen primitives from the benefit of allowing the harness to evolve from execution traces.
The proposed Composer–Compiler design seems potentially well suited to a plugin-oriented runtime such as DeepSeek Harness. Would implementing the primitives and compiler on such a runtime change the reliability, composition overhead, or ability to recompose the harness during execution? A comparison could clarify whether the contribution is tied to Mini-SWE-agent or reflects a more general runtime abstraction.
How much of the final performance gain comes from the primitive library itself, and how much comes from the task-conditioned selector? An oracle selector, a learned or heuristic selector, and ablations removing contract-guided compilation would help answer this.
How were the application scopes and composition contracts created? Please clarify the amount of human involvement, the number of candidate primitives considered and rejected, and the total offline development cost. This is important for assessing whether the proposed library can scale beyond the small collections evaluated in the paper.
6 comments
Response to Reviewer jor1 (Part 2/4)
Weakness 2 & Question 3: Ablations on Composer and Compiler
Weakness 2: It is also difficult to isolate where the reported gains come from. Some improvement may be due to the individual mechanisms themselves rather than the task-conditioned composer. The random-composer and fixed-primitive baselines are useful, but an oracle selector and ablations of the composer, composition contracts, and deterministic compiler would make the contribution of task-specific selection much clearer.
Question 3: How much of the final performance gain comes from the primitive library itself, and how much comes from the task-conditioned selector? An oracle selector, a learned or heuristic selector, and ablations removing contract-guided compilation would help answer this.
As suggested, we first added two baselines mentioned by the reviewer:
- Oracle Selector: select the best-performing single-primitive harness for each task, serving as an upper bound baseline.
- Heuristic Selector: select the best-performing single primitive for each type of task (Categorized by repository name for SWE Bench and difficulty level for Terminal Bench). This is also a baseline with oracle information
| Method | SWE: Pass@1 | Pass@3 | Pass³ | Terminal: Pass@1 | Pass@3 | Pass³ |
|---|---|---|---|---|---|---|
| Seed harness | 72.2 ± 3.1 | 94.0 | 48.8 | 49.5 ± 2.3 | 84.8 | 18.2 |
| Codex CLI | 77.9 ± 2.2 | 95.0 | 59.0 | 47.5 ± 2.3 | 78.8 | 21.2 |
| Best fixed harness | 72.8 ± 3.4 | 93.8 | 49.8 | 55.1 ± 3.2 | 86.4 | 25.8 |
| All on | 67.3 ± 0.8 | 88.2 | 44.2 | 52.0 ± 1.7 | 83.3 | 22.7 |
| Heuristic Selector | 75.4 ± 4.4 | 94.2 | 53.8 | 57.6 ± 2.6 | 86.4 | 28.8 |
| Oracle Selector | 85.0 ± 2.6 | 97.8 | 69.8 | 68.7 ± 5.3 | 92.4 | 45.5 |
| STITCH | 79.3 ± 2.3 | 96.0 | 59.8 | 62.6 ± 4.4 | 89.4 | 36.4 |
As shown in the table above, Oracle Selector shows a +21.2 points advantage over Codex CLI, demonstrating the potential of task-specific harnesses.
Heuristic Selector, on the other hand, is 2.6 points over the best single-primitive baseline despite using oracle information. Our application-scope-based composer design is +5 points higher compared to heuristic alternatives, showing the performance gain mainly comes from composing task-specific harnesses rather than individual mechanisms.
As requested by the reviewer, we conduct ablation studies on the contract-guided compilation by comparing it with an alternative agent-based integrator implementation: given the raw code of the primitives, the agent is asked to integrate the mechanism into the base harness.
| Benchmark | Agent integration: first pass | Agent integration: within 50 calls | Initial integration (k tokens) | Feedback debugging (k tokens) | Total integration (k tokens) |
|---|---|---|---|---|---|
| SWE-bench | 2/6 | 5/6 | 162.9 | 253.2 | 416.1 |
| Terminal-Bench | 0/5 | 5/5 | 189.1 | 441.9 | 631.1 |
As shown in the table above, the agent-based integrator integrated the mechanisms into the base harness with a 10/11 success rate, despite its 631.1k token cost, while the proposed deterministic compiler design integrates the mechanisms with negligible cost and 0 failures in all of our experiments.
As suggested, these baselines and the compiler ablation are now included in Appendices G.1 and H of the revised paper, highlighted in blue for your review.
Response to Reviewer jor1 (Part 3/4)
Weakness 2: Cost Reduction and Offline Library Cost
Weakness 2: The reported 638× cost reduction is also somewhat optimistic, since it relies on an idealized code-generation lower bound and excludes the offline cost of developing and maintaining the primitive library.
We provide the offline cost of developing the primitives as follows, which is about $400 in total API cost.
| Pipeline stage | Estimated inference tokens |
|---|---|
| Trajectory collection | ≈110M |
| Primitive proposal | ≈120M |
| Primitive implementation | ≈400M |
| Primitive validation and revision | ≈570M |
| Total | ≈1.2B |
To make the test-time cost comparison more practical, we compare the cost of STITCH and a test-time code-generation baseline that adds mechanisms to the base harness instead of generating from scratch. We provide the specifications of the primitives and ask the agent to add the corresponding mechanism to the base harness.
| Benchmark library | Initial generation (k tokens) | debugging (k tokens) | Generation total (k tokens) | STITCH (k tokens/task) | Token ratio |
|---|---|---|---|---|---|
| SWE-bench | 127.6 | 247.9 | 375.5 | 2.044 | 183.7× |
| Terminal-Bench | 96.8 | 403.3 | 500.1 | 2.042 | 244.9× |
As shown above, compared to a test-time code-generation baseline that only incrementally implements task-specific logic, STITCH is 244.9 times more cost-efficient due to the high debugging cost (4 times that of initial code generation) of code-generation baselines.
We appreciate this comment; the cost analyses above have been added to Appendices H and I of the revised paper, marked in blue for your reference.
Question 2: Implementation on a Plugin-Oriented Runtime
Question 2: The proposed Composer–Compiler design seems potentially well suited to a plugin-oriented runtime such as DeepSeek Harness. Would implementing the primitives and compiler on such a runtime change the reliability, composition overhead, or ability to recompose the harness during execution? A comparison could clarify whether the contribution is tied to Mini-SWE-agent or reflects a more general runtime abstraction.
We agree that a plugin-oriented runtime such as DeepSeek Harness can be well-suited for the Composer–Compiler design, as the plugin marketplace can be viewed as a natural source of the primitive library.
Without empirical evidence available, our analysis of implementing the Composer–Compiler design on a plugin-oriented runtime is as follows:
- Composition overhead. The Composer design carries over directly. However, a production plugin marketplace may contain thousands of plugins, far more than the small libraries evaluated in our paper, so their application scopes can no longer all fit into a single Composer prompt. An additional retrieval stage that first shortlists candidate plugins by their application scopes would therefore be needed. Given a reliable retriever, the Composer only reads a fixed-size shortlist, so the cost of its LLM call should stay roughly constant as the library grows.
- Compilation reliability and mid-execution recomposition. On a plugin-oriented runtime, compiling a composition reduces to installing the selected plugins through the runtime's existing plugin interface, which is typically well-supported and well-tested. We therefore expect compilation to be at least as reliable as in our Mini-SWE-agent implementation, and recomposing the harness during execution becomes a matter of loading or unloading plugins. Composition contracts would remain useful for checking that co-selected plugins do not conflict with each other.
So, for whether the contribution is tied to Mini-SWE-agent: The proposed method is not tied to specific harnesses and uses nothing specific to Mini-SWE-agent. As formally analyzed in Proposition 1, the contribution is a runtime abstraction that resolves the task-mechanism mismatch problem with selective mechanism activation. What would make a difference, however, is whether the base harness is cleanly implemented, which we believe is the case for DeepSeek Harness, and would make extending the harness with new mechanisms easier. Nonetheless, this is a separate issue of code extensibility.
Response to Reviewer jor1 (Part 1/4)
We thank Reviewer jor1 for the thoughtful review and positive assessment. We appreciate your recognition that the paper addresses a relevant and increasingly important problem with a useful mechanism-level formulation. We address each weakness and question below:
Weakness 1 & Question 1: Comparison with TTHE and HarnessX
Following the reviewer's advice, we additionally add TTHE as a baseline. As shown in the following table,
As we can see from the table above, the proposed method outperforms the trace-driven harness evolution baseline TTHE by +8.5 points. MemoHarness is also a trace-driven harness evolution baseline, which STITCH outperforms by +10 points.
We thank the reviewer for the feedback and have added the experiment above in Appendix G.2 of the revised paper, marked in blue for your review.