acceptodds
Under review as a conference paper at ICLR 2027

Dissecting model behavior through agent trajectories

Abstract

AI agent performance is not just a modeling problem, it is fundamentally a systems problem. The advanced capabilities of models are realized through agent harnesses. Therefore, a gap between model assumptions and harness behavior can easily prevent the model's full capabilities from translating into agent performance. We formalize this as the `intent-execution' gap: the mismatch between what the model intends and what the harness executes, and vice versa. We argue that minimizing this intent-execution gap is as important as other aspects of harness design such as tools and execution loops. To illustrate the impact of this harness-model alignment, we develop a simple and customizable harness called Simple Strands Agent (SSA). SSA aims to find the bulk of common patterns which generalize across different model families (such as Claude, Gemini, GPT, Grok, Qwen), as well as a small number of model-specific preferences. We make two contributions: (i) we reproduce or improve on the pass@1 performance reported by diverse model-provider families on popular agentic benchmarks (SWE-Pro, SWE-Verified, Terminal-Bench-2, and Deep-SWE), and (ii) building on an analysis of 154k trajectories generated by SSA, we look beyond the pass@1 numbers which tend to be relatively even across frontier models. By representing agent trajectories in code state-spaces, we compute solution distance and backtracking to observe model-level differences in problem-solving behavior. Finer-grained metrics such as edit frequency, testing activity, and phase-transitions reveal how individual models allocate effort across different stages of problem solving.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agent, code state space, backtracking, swe-bench, terminal-bench, deep-swe, agent harness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_38fWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SSA agent, a simple and customizable harness that achieves better performance in various coding tasks. It also defines a metric to evaluate the quality of the agent trajectories and conducts a comprehensive analysis of 154k trajectories generated by SSA.

Critical strengths
  1. The paper proposes a new agent harness that offers feedback-rich tool boundaries, balanced reasoning with tool-calling, and aligns the models' tool-use preferences.
  2. The author proposes novel metrics to quantify the quality of the agent trajectories by measuring the distance between the agent's current repository state and a verified correct solution.
  3. The author conducts a comprehensive and rigorous evaluation and result analysis on various coding tasks and agent trajectories. They also detail the experiment configs, methodologies, benchmark issues, and prompts to ensure the reproducibility of the results.
Critical weaknesses
  1. The writing is a bit hard to follow. Specifically:

    1. In the abstract, the author argues that SSA illustrates the impact of harness-model alignment, but I did not find any studies that compare the performance difference when the harness and model are aligned versus when they are not.

    2. In Table 1/ Table 2, I don't understand why the authors don't report the exact number (though I know they reported it in the appendix) but use such loose metrics ("reproduced", "exceeded," etc). As a reviewer, I am not required to read the appendix. Therefore, at first glance, the results in the main sections read as confusing, as I don't know to what extent SSA outperforms the baseline or what the baseline number is that they are comparing with.

    3. The definition of the solution distance is a bit hard to follow, maybe because I don't have enough background knowledge. Generally, it would be better to provide some higher-level, intuitive explanation for DD and BB.

  2. Some of the related work is missing. There are a bunch of studies that explore how agent harness will affect the agents' performance (e.g.HAL https://arxiv.org/pdf/2510.11977, MetaHarness https://arxiv.org/pdf/2603.28052), which seems to directly overlap with the research questions this paper wants to answer.

  3. It seems like the SSA harness and the agent quality trajectories are independent and disconnected: The solution distance metric neither motivated nor evaluated any SSA design choice. All the trajectories come from SSA, and there's nothing to compare against.

  4. SSA contains prompts that are highly customized based on the model families, and I don't fully understand how they are generated. Are we using an automated pipeline to do this, or purely handcrafted? If we are doing this manually, it may raise the concern that every time we switch the model in SSA, we need to first handcraft a prompt to align the model's tool-use preferences.

Questions for authors

I have listed my questions in the weaknesses section.

4 comments
AuthorsOpenReview ↗

Author response - Part 1/2

Thank you for the specific and constructive suggestions. Our responses follow the order of your review.

1. Comparison of aligned and misaligned harness–model pairs. We have now added controlled ablations in Section-4.3 and Figure-3 for every SSA refinement mechanism (Section 2). Each one runs the same model and benchmark with the aligned harness and with a misaligned variant, and reports the paired difference with a 95% confidence interval. The full results are in Table R2.1 of our reply to Reviewer 9uxk. The new added ablation figure carrying forest plot in the revised manuscript is also attached with this response.

In summary, aligning the interface with a model family's preferences helps. Giving GPT models their preferred apply_patch interface improves accuracy on all four model and benchmark pairs (+0.6 to +2.5 points), significantly for GPT-OSS 120B (+2.5 [+0.6, +4.2]). On Grok 4.20, hiding line numbers in file read tool gave additional +2.2. The reasoning nudge likewise has to fit the family. For Claude, the quantitative tool-call target raises tool use by about 50% without changing accuracy. For Gemini 3 Flash, removing that target gained 2.8 points. For Grok 4.20, a qualitative nudge did about 1.5 points better than the quantitative one. Large effects come from how the harness handles failures: unsurfaced parse failures cost GPT-OSS 120B 22.6 points, empty responses under serving overload cost Grok 4.7 15–19 points, and an editor that rejected Kimi-K3's scratch files with an error it could not act on cost 10.6 points. In each of these cases the model is unchanged, and only the fidelity of the harness differs.

2. Tables 1 and 2 use loose marks instead of numbers. Agreed, and we have reverted our initial choice of only using markers, and added numerical values from Appendix C.

3. The solution distance intuition. We have added a plain-language explanation before the formal definitions.

  • D(t). The fraction of a known-correct fix that is still missing from the agent's working tree at time t. It starts at 1 when nothing has been done, and reaches 0 once a complete verified fix is in place.
  • B(t). The progress the agent once had and has since lost: the gap between the best D it has reached so far and its current D. A run that never undoes useful work has B = 0 throughout.

4. Missing related work (HAL, Meta-Harness). Thank you for these pointers. We are happy to add both these references in the paper. As a summary:

  • HAL (Kapoor et al., 2025) standardizes agent evaluation across models, scaffolds and benchmarks. It shows that scaffold choice changes both accuracy and behavior.
  • Meta-Harness (Lee et al., 2026) treats the harness as something to optimize, and searches over harness code automatically.

We see our work as complementary in three ways:

  • Rather than measuring or optimizing harness variance, we attribute it to individual design choices, and find that faithful failure handling matters far more than interface shape.
  • We show that a single fixed harness with small adapters reproduces provider-reported numbers across seven model families.
  • We characterize the resulting trajectories in code state space, which neither work does.
AuthorsOpenReview ↗

Author response - Part 2/2

5. The harness and the trajectory metrics disconnected. We would like to clarify that the solution distance metrics did not motivate the SSA design and we do not claim so in the paper. Instead, the harness choices came from inspecting failures and successive refinements as listed in Section-2. The code space methodology and solution distance metric are novel mechanism developed to capture the diverse model behaviors for solving tasks even when harness is fixed (SSA here). The proposed metrics could be applied to any other harness (not shown in the current paper). The metrics can, however, evaluate SSA's choices, and the new ablations use it that way. In each of three cases below, it explains something pass@1 cannot.

The tool-call nudge. Pass@1 shows only that nothing changed (80.8% vs. 80.7% for Opus 4.6). We replayed every resolved trajectory from the two Opus 4.6 arms, with and without the nudge (5 runs × 500 SWE-Bench-Verified tasks per arm; about 2k resolved trajectories each). In each trajectory, we located the half-fix point: the tool call at which half of the verified fix is first present in the code (D(t) ≤ 0.5). Calls before it are mostly exploration and the first edits toward the fix, and calls after it complete the fix and verify it.

  • Where the extra calls go. With the nudge, Opus 4.6 makes 11 more calls before that point (31.5 vs. 20.5, median) and 7 more after it (22.5 vs. 15.5). The split between work before and after the fix stays at about 58/42 in both arms, so the nudge stretches both phases in proportion.
  • What does not change. Backtracking is unchanged (peak 0.010 vs. 0.011), as is the number of edits that move the code toward the solution.
  • Grok under serving overload. The aggregate score showed only what looked like a weaker model. The trajectories located the cause: 26 of 60 sampled runs ended in the middle of a tool call, compared with none of 60 reference runs. Fixing how the harness handles empty responses recovered 15–19 points.
  • Kimi-K3's editor. The restrictive run had no crashes, and nothing looked wrong in aggregate. Its trajectories, however, contained 210 rejected editor calls across 51 of the 113 tasks.

On "nothing to compare against": the ablation arms now provide harness-vs-harness comparisons for the same model. Keeping the harness fixed in the main analysis is deliberate, because that is what lets differences between trajectories be attributed to the model.

6. Are the family-specific prompts handcrafted, and does every new model need one? They are handcrafted and we are open-sourcing all of them via this work. Important to note, that the family-specific part is small, and is central to our argument.

  • The adapters are small. The GPT adapter adds the apply_patch conventions to the prompt (21 lines on Deep-SWE, 33 on SWE-Bench-Verified). The Claude prompt differs from the generic one by six lines, which are the nudge. The Grok adapter swaps the tool set and leaves the prompt unchanged.
  • New models needed no new prompts. Every model added after the main sweep ran on one of the two existing prompts: Opus 5.0/5.5, GPT-6, Kimi-K3, Muse Spark 1.3, Qwen 3.8, Gemini 3.8 Flash and Grok 4.7. These models reached their published numbers. Nevertheless, for newer tasks and new models, the fundamental issue of harness-model gaps may come back and would need fixes using the SSA principles mentioned in Section-2.

I thank the reviewers for their detailed responses. However, I am still confused about the main takeaway/contribution of this paper. As I understand it, the approach involves developing a general-purpose agent scaffold, introducing model-specific modifications to address observed failure modes, and demonstrating that the resulting system can reproduce the performance reported with native harnesses. While this is a useful engineering effort, the research contribution seems somewhat incremental. In particular, it remains unclear what advantages SSA offers over using each model's native harness, or what generalizable insights can be drawn from the model-specific modifications. As also noted by Reviewer 9uxk, how it motivates the agent design choices remains unclear. I therefore maintain my assessment.

AuthorsOpenReview ↗

Author response

"While this is a useful engineering effort, the research contribution seems somewhat incremental."

The research contribution is two-fold - 1) designing and detailing the design of a SOTA agent harness on the back of a large scale empirical study that covers several frontier model families. 2) providing a theoretical framework to evaluate these trajectories beyond pass@1 metrics.

"it remains unclear what advantages SSA offers over using each model's native harness"

The native harnesses used in obtaining published scores aren't public and understood. SSA is an harness that is able to reproduce the benchmark numbers (and in many cases exceed them) with all its details public (design, tools, prompts).

"what generalizable insights can be drawn from the model-specific modifications"

The generalizable insight is that the harness needs to align tools with the model's preferences to get the best out of the model. Please see Sec 2.3 in the paper.

"how it motivates the agent design choices remains unclear".

We are assuming "it" means "model-specific modifications" from the previous sentence in your response. The impact of model-specific modifications is discussed in Sec 2.2 and Sec 2.3.

Reviewer_9uxkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper provides SSA (Simple Strands Agent), an optimized agentic framework that aims to improve a wide range of models' performance on 4 agentic benchmarks. The paper also developed solution distance, roughly the edit distance of half-finished code modifications and any final correct solution. The authors also analyzed the models behaviors on the 154k agentic traces they collected.

Critical strengths
  • A large number (154k) of agentic traces were collected in the research, including frontier models such as Claude series and GPT Codex series. My estimation of the API cost in Claude models alone is at least $500k. The authors provide a wide range of statistics on the collected traces.
Critical weaknesses
  • The writing quality is very, very poor. The author failed to surface the most valuable information in the main text, while leaving a 100-page plus appendix. For example, the main result table shows the test results of Claude Opus model 4.5 and 4.6, but the appendix also show that the authors later tested newer models such as Claude Opus 5.0 "after the main sweep" and the results were not added back to the main table.
  • The authors only provide textual and empirical reasons to core design choices in their agent harness. There are no quantified or ablation tests, despite the large volume of analysis.
Questions for authors
  1. Do you have plans to release your agent traces collected while evaluating the wide range of models in agentic benchmarks?
  2. The "solution distance" concept measure the distance between code modifications in a half-finished trajectory and any final correct solution. This includes the solution this half-finished trajectory finally reaches. So its endpoint distance is 0 by construction and the curve mostly measures progress toward its own endpoint. Backtracking then reduces to "edits later undone". Is my understanding valid?
4 comments
AuthorsOpenReview ↗

Author response - Part 1/2

Thank you for the detailed and candid review. The following changes have been done to the main text of the paper to address the concerns of the reviewer:

  • Quantified results for every model. Tables 1 and 2 will give, for every model and benchmark, the gap between SSA and the published number with a confidence interval, in addition to the ✓/▲ marks. The original version of Table 1/2 was a design choice that authors took to put the message that most of the model-benchmark pairs are reproduced/exceeded by SSA and relegate exact numbers to Appendix C. Additionally, since the models are being continuously released while the work progresses, we have made a decision in the original text to only update latest benchmark Deep-SWE with recent models, and keep SWE and TB2 benchmarks with the previous generation of models. As the reviewer has rightfully noted, that some latest models do exist in Appendix for SWE-benchmarks and we have brought them (Opus 5.0, Opus 4.8, GPT 5.5) to Table1/2 in the revision.
  • The ablations for the SSA refinements. Section 4.3 and Figure-3 are added on page-8 which presents the ablation results for the SSA refinements of Section-2. It will also include two results that were previously only in the appendix; (i) the 22.6-point effect of surfacing parse failures for GPT-OSS 120B (Appendix K), (ii) and the effect of resource budgets on Terminal-Bench-2 (Appendix D). The ablation results are also summarized below in the Table R2.1.

Table R2.1. Measured effect of each Section 2 mechanism (the same rows as the new forest plot in the revised paper). For conciseness, SBV = SWE-Bench-Verified, DSW = Deep-SWE, SBP = SWE-Bench-Pro, TB2 = Terminal-Bench-2. Δ = pass@1 with the mechanism minus without, in points.

Harness mechanismModel / benchmarkΔ [95% CI]
Failure handling
Parse failures surfaced as retryable errorsGPT-OSS 120B / SBV+22.6 (App. K)
Empty responses under serving overload (same harness at 90 vs. 240 concurrent agents)Grok 4.7 / DSW+15.2 to +18.6 (three tool configurations)
Editor accepts scratch files outside the workspaceKimi-K3 / DSW+10.6 (5 runs vs. 1)
Memory and time budgets lifted13 models / TB2+5 to +10 (App. D)
Feedback-rich tool boundaries
Structured execution feedbackOpus 4.6 / SBV−0.1 [−1.0, +0.7]
Middle instead of head-only clippingOpus 4.6 / SBV+0.8 [−0.0, +1.6]
… on instances where clipping firesOpus 4.6 / SBV+1.2 [+0.2, +2.2]
… character budget, with overflow fixesGPT-OSS 120B / SBV+5.6
Duplicate edit matches rejectedOpus 4.6 / SBV+0.4 [−0.3, +1.2]
Partial-line edit matches rejectedOpus 4.6 / SBV+0.3
Reasoning nudge
Quantitative tool-call target (Claude)Opus 4.6 / SBV+0.1 [−0.6, +0.8]
Qualitative instead of quantitative (Grok)Grok 4.20 / SBV+1.5
Model tool preferences
apply_patch instead of generic editor (GPT)GPT-OSS 120B / SBV+2.5 [+0.6, +4.2]
GPT-5.5 / DSW+2.5 [−2.3, +7.1]
GPT-5.6-Sol / DSW+1.6 [−1.4, +4.8]
Split view/edit tools vs. one editor (Grok)Grok 4.7 / DSW+0.7 [−3.2, +4.4]
Line numbers hidden in file views (Grok)Grok 4.20 / SBP+2.2
Grok 4.7 / DSW+1.6 [−2.3, +5.5]

The above two changes directly addresses both of the critical weakness points raised by the reviewer.

Trajectories release: Yes, with SSA our main goal is to establish reproducibility in the community which have not been done (to the best of authors knowledge) yet for advanced benchmarks such as SWE, TB2. We have plans to release the trajectories as well in the future.

AuthorsOpenReview ↗

Author response - Part 2/2

Solution Distance: For the endpoint, the reviewer's understanding is correct. A resolved run's own patch is one of the reference solutions, so its final distance is zero by construction (as stated in Appendix B.1.6).

Recalling Appendix B.1.6. A resolved run's final patch has passed the test oracle, so it is, by definition, a member of the solution space, and its correct endpoint is D(t_end) = 0. As the appendix puts it, this "is not an extra credit for solved runs, it is the endpoint-consistency condition implied by the definition of the solution space." Unresolved runs receive no self-anchor, because their patches are not verified.

We have also reported the alternative reviewer have in mind, strict leave-one-run-out (LOO) scoring, where a run is scored only against solutions found by other runs. It explains why LOO misrepresents the endpoint. The empirical set of observed solutions Si\mathcal{S}_i "is a finite sample, not a dense cover of the true solution space. Nearby correct solutions can differ in variable names, hunk boundaries, guard placement, helper structure, or surrounding context, and the nearest textual variant may simply be absent from the other observed runs." Under LOO, some runs that solved their task would therefore end at D>0D > 0, and that positive distance "can reflect sparsity in the empirical reference set rather than a real failure to reach the solution set." Appendix B.1.8 (item 3) adds that this sparsity grows with task difficulty: the reference set is dense on easy instances and thin on hard ones.

For reviewer point "curve mostly measures progress toward its own endpoint" depends on the benchmark. To illustrate, we recomputed every main-text quantity under leave-one-run-out (LOO) scoring, where each run is compared only with other runs' verified patches.

Resolved runs, SWE-Bench-VerifiedOpus 4.6, self-anchoredOpus 4.6, LOOGemini 3.1 Pro, self-anchoredGemini 3.1 Pro, LOO
Backtracking peak0.0140.0150.0470.050
Timing IQR of T(0.5)0.1540.1590.3560.372
Runs with ≥ 1 backtracking edit9.6%10.6%24.5%27.0%
Mean endpoint distance0.0000.0680.0010.105

Across all 27 models, the LOO and self-anchored values have rank correlations between 0.87 and 0.995 for every shape and backtracking statistic (all correlations significant with p < 1e-8). The only quantity that changes materially is the endpoint itself, as expected.

Deep-SWE: your reading is correct. Deep-SWE is the extreme case of the sparsity described in Appendix B.1.8. Almost every correct patch on this benchmark is distinct, and a run's own patch is the nearest reference at 84–96% of edit events. Under LOO, resolved runs end at a distance of about 0.68. On Deep-SWE, the curve therefore measures progress toward the run's own eventual solution. We still find this informative about when a model commits to its solution in a normalized time.

Backtracking. Yes, B(t)B(t) amounts to verified progress that was later undone. We think this is the right quantity to measure. It is computed by replaying every edit, including undos and git reverts (Appendix B.1.5), and Appendix B.1.7 relates it exactly to the per-edit changes in DD. It captures something pass@1 and edit counts cannot: under those metrics, a run that finds a correct change and then retracts it looks identical to a run that never found it.

Reviewer_9uxkOpenReview ↗

Official Comment by Reviewer 9uxk

Thank you for the response. I have read the revision and the replies to all three reviewers. My rating is unchanged, and further tables will not change it, for one reason: the paper no longer has a result. The new ablations show that the Section 2 design principles have no measurable effect on pass@1, and the authors say as much to Reviewer 38fW ("failure handling matters far more than interface shape"). What remains is a harness that reproduces model-card numbers once its bugs are fixed, and trajectory statistics that, by the authors' own statement, motivated and evaluated no design decision. The title, abstract and conclusion still claim the original thesis. The presentation problem is the same at every scale: 124 pages, and now an 18-row ablation table, around a null central finding, with no attempt to compress to what matters. A paper whose claim matches this evidence would be a different submission.

AuthorsOpenReview ↗

Author response

"the paper no longer has a result"

We respectfully disagree with this assessment. The main contributions listed in page-2 mentions (i) SSA reproducing frontier models scores across coding benchmarks, (ii) a novel code state-space theory to evaluate model trajectories, (iii) Expanding the metrics for model trajectories beyond aggregated pass@1, and (iv) quantifying git leakage in swe-bench-pro causing inflated scores for several models.

"The new ablations show that the Section 2 design principles have no measurable effect on pass@1"

This is an incorrect observation. It is evident from Figure-3 in the revised paper (also Table R2.1 in rebuttal) that individual factors contribute towards improving pass@1, for example, bash-tool middle clipping raised Opus 4.6 score on SBV by +1.2. Note that, in an agent, these mechanisms work together due to 10's if not 100's of sequential tool calls and each bad-tool response impacting the sequential calls. The compounding result is visible from SSA achieving published (closed-source harness) scores for frontier and several open-source models.

"trajectory statistics that, by the authors' own statement, motivated and evaluated no design decision"

We would like to clarify that, code state-space theory provides a novel mechanism to evaluate agent trajectories beyond pass@1. Since trajectories are a function of harness and model, and we keep the harness fixed (SSA), we effectively evaluate diverse model behaviors via code state-spaces. A vanilla pass@1 could not differentiate Opus 4.6 and Gemini 3.1 Pro on SWE-Bench verified (due to similar scores), but how these model explore and develop solutions over time is represented in Figure-5 which is strikingly different. As explained in Section 4.4, Opus 4.6 explores and makes sharp transition from D=1 to D=0, while Gemini does it in a more evenly spread slow descent to the solution space.

"124 pages, and now an 18-row ablation table, around a null central finding, with no attempt to compress to what matters"

One of the key contributions of the work is transparent reproduction of the frontier model scores across 5 coding benchmarks. First, due to large volume of the involved data, and experiments (67 model-benchmark pairs), we have dedicated 32 pages for full prompts, developed tools, vllm inference configs disclosure. Second, due to novelty of code state-spaces interpretation for agent trajectories, we have dedicated 15 pages in Appendix-B to provide tutorials, examples, ablations for better understanding the concept. Lastly, we aimed to present the minimal diverse set of models and benchmarks to show the impact of each component in ablation Figure-3.

Reviewer_yieyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how agent harnesses influence coding-agent performance and whether conventional success rates adequately reflect model capabilities. The authors identify two key phenomena: mismatches between model intentions and harness execution can limit performance, while models achieving similar success rates may exhibit substantially different problem-solving behaviors. To investigate these phenomena, they develop SSA, a harness designed to minimize the intent-execution gap, and introduce trajectory-level metrics, including solution distance, backtracking, and tool-use patterns, to characterize model behavior. Experiments involving 38 models and approximately 154,000 trajectories demonstrate that SSA matches or exceeds published performance in 61 of 66 model–benchmark pairs. Further analysis reveals distinct model-specific strategies in editing, testing, and solution progression, highlighting behavioral differences overlooked by conventional accuracy metrics.

Critical strengths
  1. The paper provides an interesting observation that different models exhibit substantially different problem-solving behaviors, particularly in how they interact with tools, process feedback, and adjust their solutions. These differences are not captured by conventional success-rate metrics.

  2. The paper demonstrates that the alignment between models and agent harnesses has a tangible impact on coding performance. By adapting tool interfaces and feedback mechanisms to model-specific preferences, the proposed SSA harness achieves competitive results across multiple models and benchmarks, highlighting the practical value of harness-model alignment.

Critical weaknesses
  1. While the paper identifies interesting behavioral differences across models, the underlying causes remain unclear. It is difficult to determine whether these differences arise from intrinsic model capabilities, model-specific training characteristics, benchmark properties, or stochastic variations during execution. Without further controlled experiments, it is unclear whether the observed patterns represent stable, generalizable model behaviors or are specific to the evaluated settings.
Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Author Response - Part 1/2

Thank you for the positive assessment of the paper and dedicating time to review. The main concern raised is whether the behaviors we report are stable properties of the models, or products of the setting and of chance? Note that, every model in the paper was run five times on every instance, and we have reported mean pass@1 also also 95% confidence intervals. That also lets us answer most of this from the existing trajectories.

Stochastic Variation

We tested this in two ways.

  • Variance decomposition. We split the variance of each trajectory-level signature into four parts:

    • model: signature is model behavior
    • instance: some tasks inhibit/facilitate observed signatures
    • model × instance: property of model and task pair
    • run-to-run: behaves differently even with same model and task. This is pure chance and also reviewer's main ask
  • Split-half reliability. We (i) ranked the models first using odd runs 1, 3 and 5 only, and (ii) ranked them again using even runs 2 and 4 only, and compared the two rankings. If the model differences were stochastic, the two halves would disagree.

Table R1.1. Share of variance from each source (SBV / SBP / DSW, the four shares sum to 1 per benchmark). For conciseness, SBV = SWE-Bench-Verified (27 models), SBP = SWE-Bench-Pro (26), DSW = Deep-SWE (17).

SignatureModelInstanceModel × instanceRun-to-run
Read share0.47 / 0.34 / 0.470.20 / 0.25 / 0.260.16 / 0.22 / 0.140.17 / 0.20 / 0.13
Edit share0.18 / 0.21 / 0.210.17 / 0.38 / 0.340.30 / 0.23 / 0.230.35 / 0.18 / 0.22
Scratch-testing share0.37 / 0.23 / 0.250.23 / 0.35 / 0.380.21 / 0.24 / 0.190.19 / 0.18 / 0.19
Suite-testing share0.48 / 0.35 / 0.290.09 / 0.15 / 0.200.21 / 0.26 / 0.260.22 / 0.23 / 0.24
Verify-phase share0.40 / 0.41 / 0.500.17 / 0.17 / 0.180.19 / 0.20 / 0.150.24 / 0.21 / 0.17
Implement-phase position0.30 / 0.40 / 0.450.18 / 0.13 / 0.160.21 / 0.22 / 0.160.31 / 0.25 / 0.24
T(0.5) timing (resolved runs)0.37 / 0.34 / 0.230.12 / 0.17 / 0.250.23 / 0.24 / 0.250.28 / 0.25 / 0.27
Any backtracking edit (yes/no)0.22 / 0.15 / 0.190.12 / 0.12 / 0.110.16 / 0.17 / 0.070.51 / 0.56 / 0.64

The picture differs by signature:

  • The model dominates how much an agent reads, runs the test suite and spends in verification. For these, the model is the largest of the four sources on every benchmark. This is the message of the paper as well as written in abstract "Finer-grained metrics such as edit frequency ... reveal how individual models allocate effort"
  • The model and the task matter about equally for edit share and scratch testing. Some tasks simply require more editing or ad hoc testing.
  • Backtracking is mostly chance for a single run, but not for a model. Whether one run backtracks is largely chance (run-to-run 0.51–0.64), but our work concern how often a model backtracks, and run-to-run noise averages out over a model's runs while the model effect does not.
LevelModelInstanceModel × instanceRun-to-run
One trajectory0.22 / 0.15 / 0.190.12 / 0.12 / 0.110.16 / 0.17 / 0.070.51 / 0.56 / 0.64
One model on one task (5-run rate)0.37 / 0.26 / 0.360.19 / 0.21 / 0.210.26 / 0.29 / 0.130.18 / 0.24 / 0.31

For a single trajectory, chance is the largest source (0.51–0.64). Once we average a model's five runs on a task, chance falls to 0.18–0.31, and the model becomes the largest source on SWE-Bench-Verified and Deep-SWE (0.37, 0.36) and is close to the largest on SWE-Bench-Pro (0.26). Backtracking on any one run is hard to predict, but how often a model backtracks is a property of the model.

AuthorsOpenReview ↗

Author Response - Part 2/2

Split Half reliability: A large chance-component in individual runs does not mean the model averages are unreliable. To test the averages directly, we split each model's runs into two disjoint halves (odd and even runs), ranked the models on each half separately, and correlated the two rankings. What we aim to test is if the differences between models were chance, the two halves would disagree.

Table R1.2. Agreement between model rankings. All correlations are statistically significant (p < 0.001)

SignatureSplit-half ρ (SBV / SBP / DSW)SBV ↔ SBP ρ (24 models)
Read share1.00 / 1.00 / 1.000.87
Edit share0.97 / 0.99 / 0.990.75
Scratch-testing share1.00 / 1.00 / 0.980.65
Suite-testing share0.99 / 1.00 / 0.990.75
Verify-phase share1.00 / 0.99 / 0.990.87
Implement-phase position0.99 / 1.00 / 0.990.89
T(0.5) timing (resolved runs)1.00 / 0.99 / 0.980.74
Any backtracking edit (yes/no)0.98 / 0.96 / 0.930.68

The two halves order the models almost identically, with ρ = 0.93–1.00 for every signature on every benchmark. Backtracking shows how this fits with the large run-to-run share in Table R1.1. Whether a single run backtracks is close to a coin flip. A model's backtracking rate over roughly 2,000 trajectories is nonetheless very stable, in the same way that a weighted coin is unpredictable on any one flip but easy to tell apart from a fair coin over thousands of flips. We will therefore state explicitly that the signatures characterize a model in aggregate, not any single run.

The harness, and controlled experiments. The new harness ablations (described in our reply to Reviewer 9uxk) are controlled experiments of the kind you ask for. We have added them in Figure-3 and Section-4.3.

Capability versus training. We agree that our data cannot separate intrinsic capability from training-specific habits. For closed models, neither weights nor training data are available. We thank the reviewer to point this out and we have also updated the conclusion section to add this limitation and called out explicitly.

Lastly, we would like to emphasize that what the analysis does establish is that the signatures are stable, transfer across benchmarks, and are invisible to pass@1. Those are the preconditions for any causal study.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

We thank all the reviewers for their careful and constructive reviews. It is encouraging that reviewers are finding the paper has "interesting observations" and that the study being "comprehensive and rigorous." Based on the overall feedback, we are making the following changes to the current manuscript to further enhance its presentation of results.

Summary of changes

  1. Ablation section for the harness design. Added Section-4.3 and Figure-3 on page-8 of the revised paper to present the ablation results for the SSA design choices (Section-2).
  2. Update Table-1/2 in the main text to bring the numerical values of pass@1 (originally present in Appendix C) along with the existing reproduction check-marks.
  3. Adding 2 new references suggested by Reviewer 38fW, and also intuitive definition of solution distance D(t)D(t) and backtracking B(t)B(t) in Section-3 for a better understanding of the concept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.