- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes SSA agent, a simple and customizable harness that achieves better performance in various coding tasks. It also defines a metric to evaluate the quality of the agent trajectories and conducts a comprehensive analysis of 154k trajectories generated by SSA.
Critical strengths
- The paper proposes a new agent harness that offers feedback-rich tool boundaries, balanced reasoning with tool-calling, and aligns the models' tool-use preferences.
- The author proposes novel metrics to quantify the quality of the agent trajectories by measuring the distance between the agent's current repository state and a verified correct solution.
- The author conducts a comprehensive and rigorous evaluation and result analysis on various coding tasks and agent trajectories. They also detail the experiment configs, methodologies, benchmark issues, and prompts to ensure the reproducibility of the results.
Critical weaknesses
-
The writing is a bit hard to follow. Specifically:
-
In the abstract, the author argues that SSA illustrates the impact of harness-model alignment, but I did not find any studies that compare the performance difference when the harness and model are aligned versus when they are not.
-
In Table 1/ Table 2, I don't understand why the authors don't report the exact number (though I know they reported it in the appendix) but use such loose metrics ("reproduced", "exceeded," etc). As a reviewer, I am not required to read the appendix. Therefore, at first glance, the results in the main sections read as confusing, as I don't know to what extent SSA outperforms the baseline or what the baseline number is that they are comparing with.
-
The definition of the solution distance is a bit hard to follow, maybe because I don't have enough background knowledge. Generally, it would be better to provide some higher-level, intuitive explanation for and .
-
-
Some of the related work is missing. There are a bunch of studies that explore how agent harness will affect the agents' performance (e.g.HAL https://arxiv.org/pdf/2510.11977, MetaHarness https://arxiv.org/pdf/2603.28052), which seems to directly overlap with the research questions this paper wants to answer.
-
It seems like the SSA harness and the agent quality trajectories are independent and disconnected: The solution distance metric neither motivated nor evaluated any SSA design choice. All the trajectories come from SSA, and there's nothing to compare against.
-
SSA contains prompts that are highly customized based on the model families, and I don't fully understand how they are generated. Are we using an automated pipeline to do this, or purely handcrafted? If we are doing this manually, it may raise the concern that every time we switch the model in SSA, we need to first handcraft a prompt to align the model's tool-use preferences.
Questions for authors
I have listed my questions in the weaknesses section.
4 comments
Author response - Part 2/2
5. The harness and the trajectory metrics disconnected. We would like to clarify that the solution distance metrics did not motivate the SSA design and we do not claim so in the paper. Instead, the harness choices came from inspecting failures and successive refinements as listed in Section-2. The code space methodology and solution distance metric are novel mechanism developed to capture the diverse model behaviors for solving tasks even when harness is fixed (SSA here). The proposed metrics could be applied to any other harness (not shown in the current paper). The metrics can, however, evaluate SSA's choices, and the new ablations use it that way. In each of three cases below, it explains something pass@1 cannot.
The tool-call nudge. Pass@1 shows only that nothing changed (80.8% vs. 80.7% for Opus 4.6). We replayed every resolved trajectory from the two Opus 4.6 arms, with and without the nudge (5 runs × 500 SWE-Bench-Verified tasks per arm; about 2k resolved trajectories each). In each trajectory, we located the half-fix point: the tool call at which half of the verified fix is first present in the code (D(t) ≤ 0.5). Calls before it are mostly exploration and the first edits toward the fix, and calls after it complete the fix and verify it.
- Where the extra calls go. With the nudge, Opus 4.6 makes 11 more calls before that point (31.5 vs. 20.5, median) and 7 more after it (22.5 vs. 15.5). The split between work before and after the fix stays at about 58/42 in both arms, so the nudge stretches both phases in proportion.
- What does not change. Backtracking is unchanged (peak 0.010 vs. 0.011), as is the number of edits that move the code toward the solution.
- Grok under serving overload. The aggregate score showed only what looked like a weaker model. The trajectories located the cause: 26 of 60 sampled runs ended in the middle of a tool call, compared with none of 60 reference runs. Fixing how the harness handles empty responses recovered 15–19 points.
- Kimi-K3's editor. The restrictive run had no crashes, and nothing looked wrong in aggregate. Its trajectories, however, contained 210 rejected editor calls across 51 of the 113 tasks.
On "nothing to compare against": the ablation arms now provide harness-vs-harness comparisons for the same model. Keeping the harness fixed in the main analysis is deliberate, because that is what lets differences between trajectories be attributed to the model.
6. Are the family-specific prompts handcrafted, and does every new model need one? They are handcrafted and we are open-sourcing all of them via this work. Important to note, that the family-specific part is small, and is central to our argument.
- The adapters are small. The GPT adapter adds the
apply_patchconventions to the prompt (21 lines on Deep-SWE, 33 on SWE-Bench-Verified). The Claude prompt differs from the generic one by six lines, which are the nudge. The Grok adapter swaps the tool set and leaves the prompt unchanged. - New models needed no new prompts. Every model added after the main sweep ran on one of the two existing prompts: Opus 5.0/5.5, GPT-6, Kimi-K3, Muse Spark 1.3, Qwen 3.8, Gemini 3.8 Flash and Grok 4.7. These models reached their published numbers. Nevertheless, for newer tasks and new models, the fundamental issue of harness-model gaps may come back and would need fixes using the SSA principles mentioned in Section-2.
I thank the reviewers for their detailed responses. However, I am still confused about the main takeaway/contribution of this paper. As I understand it, the approach involves developing a general-purpose agent scaffold, introducing model-specific modifications to address observed failure modes, and demonstrating that the resulting system can reproduce the performance reported with native harnesses. While this is a useful engineering effort, the research contribution seems somewhat incremental. In particular, it remains unclear what advantages SSA offers over using each model's native harness, or what generalizable insights can be drawn from the model-specific modifications. As also noted by Reviewer 9uxk, how it motivates the agent design choices remains unclear. I therefore maintain my assessment.
Author response
"While this is a useful engineering effort, the research contribution seems somewhat incremental."
The research contribution is two-fold - 1) designing and detailing the design of a SOTA agent harness on the back of a large scale empirical study that covers several frontier model families. 2) providing a theoretical framework to evaluate these trajectories beyond pass@1 metrics.
"it remains unclear what advantages SSA offers over using each model's native harness"
The native harnesses used in obtaining published scores aren't public and understood. SSA is an harness that is able to reproduce the benchmark numbers (and in many cases exceed them) with all its details public (design, tools, prompts).
"what generalizable insights can be drawn from the model-specific modifications"
The generalizable insight is that the harness needs to align tools with the model's preferences to get the best out of the model. Please see Sec 2.3 in the paper.
"how it motivates the agent design choices remains unclear".
We are assuming "it" means "model-specific modifications" from the previous sentence in your response. The impact of model-specific modifications is discussed in Sec 2.2 and Sec 2.3.
Author response - Part 1/2
Thank you for the specific and constructive suggestions. Our responses follow the order of your review.
1. Comparison of aligned and misaligned harness–model pairs. We have now added controlled ablations in Section-4.3 and Figure-3 for every SSA refinement mechanism (Section 2). Each one runs the same model and benchmark with the aligned harness and with a misaligned variant, and reports the paired difference with a 95% confidence interval. The full results are in Table R2.1 of our reply to Reviewer 9uxk. The new added ablation figure carrying forest plot in the revised manuscript is also attached with this response.
In summary, aligning the interface with a model family's preferences helps. Giving GPT models their preferred apply_patch interface improves accuracy on all four model and benchmark pairs (+0.6 to +2.5 points), significantly for GPT-OSS 120B (+2.5 [+0.6, +4.2]). On Grok 4.20, hiding line numbers in file read tool gave additional +2.2. The reasoning nudge likewise has to fit the family. For Claude, the quantitative tool-call target raises tool use by about 50% without changing accuracy. For Gemini 3 Flash, removing that target gained 2.8 points. For Grok 4.20, a qualitative nudge did about 1.5 points better than the quantitative one. Large effects come from how the harness handles failures: unsurfaced parse failures cost GPT-OSS 120B 22.6 points, empty responses under serving overload cost Grok 4.7 15–19 points, and an editor that rejected Kimi-K3's scratch files with an error it could not act on cost 10.6 points. In each of these cases the model is unchanged, and only the fidelity of the harness differs.
2. Tables 1 and 2 use loose marks instead of numbers. Agreed, and we have reverted our initial choice of only using markers, and added numerical values from Appendix C.
3. The solution distance intuition. We have added a plain-language explanation before the formal definitions.
4. Missing related work (HAL, Meta-Harness). Thank you for these pointers. We are happy to add both these references in the paper. As a summary:
We see our work as complementary in three ways: