acceptodds
Under review as a conference paper at ICLR 2027

Analyzing and Mitigating Cost-Inefficient Behaviors in Coding Agents

Abstract

Although effective, coding agents often incur substantial monetary costs. Their recurring *cost-inefficient behaviors* remain underexplored. We conduct a systematic study of behavioral cost inefficiencies in coding agents, analyzing 1,200 trajectories from Claude Code and Mini-SWE-Agent across four configurations on SWE-bench Verified. We identify three cost-inefficient behaviors: *subsumed retrieval*, *similar script generation*, and *test re-execution*. We then evaluate three mitigation strategies: *structure-aware retrieval*, *agent-synthesized skills*, and *developer-designed skills*, over 10k trajectories on held-out SWE-bench Verified and Pro tasks. Our main findings are: (1) The three behaviors affect 79.00%–98.00% of coding tasks and account for up to 22.75% of task cost. (2) Structure-aware retrieval can introduce retrieval overhead and alter agent delegation, causing inconsistent improvements in retrieval efficiency and cost increases of up to 28.14%. (3) Agent-synthesized skills tend to produce low-level, trace-specific guidance, limiting their effectiveness and generality. (4) In contrast, developer-designed skills provide high-level, trace-agnostic guidance, reducing cost by up to 41.73%, roughly twice the maximum gain from agent-synthesized skills.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Coding Agents, Agent Trajectory Analysis, Cost Efficiency, Cost-Inefficient Behaviors, Behavioral Optimization, Structure-Aware Retrieval, Agent Skills

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8xRGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies cost-inefficient behaviors in coding agents. By analyzing trajectories of Claude Code and Mini-SWE-Agent on SWE-bench, it identifies three recurring behaviors: redundant code retrieval, regeneration of near-duplicate scripts, and repeated test execution without code changes. It then evaluates three mitigation approaches. Structure-aware retrieval does not reliably reduce cost and can even increase it. Prompt-level skills are more effective, and developer-designed skills outperform agent-synthesized ones, which the authors attribute to their higher-level, more general guidance.

Critical strengths
  1. This paper studies an important problem about cost inefficiency in coding agents. The behavior analysis is well-structured and informative, and for each the paper analyzes both how it is detected and why it arises across agent architectures.

  2. The negative results on structure-aware retrieval are interesting. Although the code-graph tool reduces redundant retrieval, it does not necessarily lower the overall cost and sometimes even increases it because of longer outputs and changes in subagent delegation.

Critical weaknesses

Minor concern: The comparison between DevSkills and SynSkills does not fully isolate the benefit of human abstraction. While SynSkills are generated from individual trajectories and then consolidated, DevSkills are informed by the broader RQ1 analysis across configurations. SynSkills also use each configuration's backbone model, which may limit synthesis quality. These differences make it difficult to determine whether the performance gap comes from human-designed guidance or from the synthesis setup.

Questions for authors
  1. Would using a stronger model or providing the aggregated RQ1 analysis improve SynSkills and narrow the gap with DevSkills?

  2. One DevSkill, "Break Loops," encourages agents to submit their best solution when stuck. Could some of the cost savings come from early termination rather than directly reducing the three identified behaviors? An ablation without this skill would help clarify its contribution.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8xRG: Stronger SynSkills with Pooled Trajectories (Q1)

We sincerely thank the reviewer for the encouraging assessment and insightful questions. We especially appreciate your recognition of the importance of studying cost inefficiency in coding agents. Your comments motivated us to further investigate the effectiveness of automated skill synthesis and the mechanisms behind DevSkills' cost savings.

Q1

Would using a stronger model or providing the aggregated RQ1 analysis improve SynSkills and narrow the gap with DevSkills?

Related weakness: The comparison between DevSkills and SynSkills does not fully isolate the benefit of human abstraction. While SynSkills are generated from individual trajectories and then consolidated, DevSkills are informed by the broader RQ1 analysis across configurations. SynSkills also use each configuration's backbone model, which may limit synthesis quality. These differences make it difficult to determine whether the performance gap comes from human-designed guidance or from the synthesis setup.

Summary. With a stronger synthesizer (Sonnet 5.5) and pooled trajectories across agent configurations, SynSkills match or exceed DevSkills' savings in three Sonnet 4.6 settings, but increase cost on MSA(Q35+), where DevSkills remain effective. These results highlight that both SynSkills and DevSkills benefit from our RQ1 results and DevSkills has stronger cross-model consistency.

RQ1-guided synthesis. Importantly, all SynSkills experiments, including those in the original paper, this response, and our response to Rt4B Q3, use our RQ1 diagnoses as input, including flagged cost-inefficient behavior types, their frequencies, and flagged step indices (Appendix A.4.1, Lines 1281–1325). Thus, SynSkills already benefit from our behavioral analysis, even though their effectiveness leaves room for improvement.

Setup. In the paper, SynSkills were synthesized separately for each configuration using its own backbone model and RQ1 trajectories. Here, we reran the pipeline using Sonnet 5.5 (high reasoning effort) and pooled trajectories from all four configurations to enable unified, cross-configuration synthesis. We retained the original prompt templates and experiment settings. The resulting shared skill set contains 54 rules (1,880 words).

We evaluated the four settings where DevSkills had significantly lower cost than the original SynSkills. MSA(MM3) was excluded due to model drift (recent Pass@1: 94.00% vs. 75.17% originally) and frequent observed benchmark-gaming behaviors, making a reliable comparison infeasible.

SettingApproachΔCostΔPass@1, ppΔSubRetrvΔSimScrptΔReTest
MSA (S46), Verified-200SynSkills (paper)-22.32%-0.50-11.38%-45.10%-36.82%
SynSkills (54 rules)-39.26%-0.50-48.14%-63.69%-32.56%
DevSkills-41.11%+0.17-49.97%-58.83%-34.46%
CC, Verified-200SynSkills (paper)-1.38%-0.50-33.20%-41.85%+12.72%
SynSkills (54 rules)-24.94%-2.00-62.93%-46.29%-35.95%
DevSkills-13.94%-1.50-53.04%-54.93%-4.70%
MSA (S46), Pro-100SynSkills (paper)-6.13%+0.00+2.50%+79.32%-28.25%
SynSkills (54 rules)-38.56%-2.00-30.51%-64.14%-58.73%
DevSkills-33.00%+0.00-43.57%-38.36%-26.98%
MSA (Q35+), Verified-200SynSkills (paper)-0.70%+2.50-5.05%-17.63%-8.37%
SynSkills (54 rules)+21.78%+1.67+20.37%+27.53%-7.81%
DevSkills-11.34%+2.67-17.25%-0.49%-7.89%

On the three Sonnet 4.6 settings, the new SynSkills reduce cost by 24.94–39.26%, substantially improving over the original SynSkills (1.38–22.32%). They match DevSkills on both MSA(S46) settings and outperform them on CC, without significant Pass@1 changes. However, on MSA(Q35+), SynSkills increase cost by 21.78%, whereas DevSkills reduce it by 11.34%. A compact 10-rule SynSkills variant exhibits the same pattern (see Rt4B, Q3).

Overall, these results demonstrate that our RQ1 diagnoses can guide effective automated skill synthesis, with further improvements from stronger synthesis and pooled evidence. Nevertheless, DevSkills deliver more consistent cost reductions across model backbones, including where stronger SynSkills fail.

AuthorsOpenReview ↗

Response to Reviewer 8xRG: Break Loops Ablation Study (Q2)

Q2

One DevSkill, "Break Loops," encourages agents to submit their best solution when stuck. Could some of the cost savings come from early termination rather than directly reducing the three identified behaviors? An ablation without this skill would help clarify its contribution.

Summary. Removing "Break Loops" yields comparable cost savings, API calls, and reductions in the three identified behaviors, suggesting that early termination is not the primary source of DevSkills' savings.

Setup. We removed "Break Loops" while retaining the other six DevSkills principles and all experimental settings. We evaluated MSA(S46) on Verified-200 (the largest reported saving: 41.73%) and CC on Verified-200 (a reported −1.50 pp Pass@1 change). Due to evaluation costs, each ablation uses one run, compared against three-run DevSkills results, including two additional MSA(S46) runs conducted for AhRp, Q2.

SettingApproachPass@1Cost saving vs. baselineAPI callsSubRetrvSimScrptReTest
MSA (S46)DevSkills72.67%-41.11%31.971.92 (-49.97%)1.44 (-58.83%)1.40 (-34.46%)
MSA (S46)DevSkills w/o Break Loops72.50%-43.05%31.101.98 (-48.52%)1.22 (-65.17%)1.31 (-38.46%)
CCDevSkills73.50%-13.94%20.430.35 (-53.04%)0.15 (-54.93%)1.45 (-4.70%)
CCDevSkills w/o Break Loops75.00%-17.12%20.070.33 (-57.01%)0.16 (-53.48%)1.36 (-10.61%)

Findings. Removing "Break Loops" yields comparable cost savings (43.05% vs. 41.11% on MSA; 17.12% vs. 13.94% on CC), API calls, and reductions in all three behaviors. Pass@1 differs by at most 1.50 pp. If "Break Loops" were driving savings through early termination, removing it would be expected to increase API calls and cost; neither occurs. These results suggest that early termination is not the primary source of savings, although the single-run ablations limit stronger causal claims.

Thank you again for these thoughtful suggestions. The additional experiments have provided valuable insights into skill synthesis and the sources of cost savings, helping us refine our conclusions. We will incorporate the Sonnet 5.5 SynSkills results and the "Break Loops" ablation into RQ3 and the appendix. We hope these new results adequately address your concerns.

AuthorsOpenReview ↗

Revised Paper Uploaded

Thank you again for your encouraging review and insightful questions. We have uploaded a revised version of the paper. The main changes related to your comments are:

  • Q1 (stronger synthesizer and pooled RQ1 analysis): The Sonnet 5.5 SynSkills results with pooled trajectories (54-rule and compact 10-rule variants) are now in Sec. 4.3, with Finding 5, the Abstract, and the Introduction revised accordingly. Sec. 4.3 also clarifies that all SynSkills use our RQ1 diagnoses as input. Synthesis details are in Appendix A.4.
  • Q2 ("Break Loops" ablation): The ablation without this principle is in Appendix A.4.2 (Table 20).

We would greatly appreciate it if you could let us know whether the revision and our responses address your concerns. As the discussion period is coming to an end, any feedback would be very helpful, and we are happy to clarify anything further.

Reviewer_Rt4BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates recurring behavioral patterns that contribute to coding-agent inference costs. The authors analyze 1,200 trajectories from Claude Code and three Mini-SWE-Agent model configurations, identifying three common sources of inefficiency: subsumed retrieval, redundant script generation, and repeated test execution.

To mitigate these inefficiencies, the authors evaluate three approaches: CodeGraph-based retrieval, automatically synthesized behavioral skills, and seven developer-designed principles. Experiments span 200 held-out SWE-bench Verified tasks and 100 SWE-bench Pro tasks, yielding over 10,000 trajectories.

The results reveal that reducing redundant actions does not necessarily translate into lower inference costs. For instance, CodeGraph reduces repeated retrieval but can increase overall costs due to verbose outputs and shifts in model delegation behavior. Developer-designed skills achieve the largest reported cost reduction of up to 41.73%, outperforming the automated skill-synthesis pipeline.

Critical strengths
  1. A comprehensive empirical study connecting agent behavior to operational costs. The paper provides a large-scale empirical analysis across two agent frameworks, multiple model backbones, and different benchmarks, offering reasonably broad coverage of coding-agent behaviors. Rather than focusing solely on aggregate token usage or inference costs, it identifies specific sources of inefficiency, including redundant retrieval, repeated script generation, and unnecessary test execution. These findings offer actionable insights into where researchers and practitioners should focus their efforts when designing and optimizing agent harnesses.

  2. Simple behavioral interventions yield meaningful practical improvements. The paper demonstrates that relatively simple, developer-designed behavioral principles can substantially reduce inference costs. The seven shared principles achieve cost reductions in six of eight settings under the paper’s robustness criterion, with savings reaching up to 41.73%. Most observed changes in task success rates are small, although one setting shows a notable degradation. Importantly, the authors release the complete set of principles, making the findings directly actionable and potentially valuable to the broader coding-agent community.

Critical weaknesses
  1. Limited comparison of retrieval methods.

The paper only evaluates CodeGraph as a retrieval method, without comparing it against alternatives such as embedding-based search or LSP-based navigation. These alternatives might provide more targeted information with less output overhead. Since the increased costs observed in the paper are partly caused by CodeGraph’s verbose outputs and changes in agent behavior, it is unclear whether the same conclusions would hold for other retrieval methods. A broader comparison would help clarify whether this is a general issue with retrieval or a limitation of the specific CodeGraph implementation.

  1. Repeated actions are not always wasteful.

The paper treats repeated actions as potential sources of inefficiency, but some of these actions may actually be necessary. For example, if a subagent reads code and only returns a summary, the main agent may still need to read the original code before making changes. Similarly, agents may need to revisit code when working with long contexts or rerun tests after making changes to the environment. These are reasonable behaviors rather than wasted computation. It would be useful for the paper to distinguish between necessary and unnecessary repetition, rather than treating the cost of repeated actions as an indication of inefficiency.

Questions for authors
  1. Detector validity. Have the authors manually evaluated the detectors to measure false positives and false negatives? In particular, how often are flagged repeated actions actually useful, such as rereading code inspected by a subagent, verifying an edit, or rerunning tests after an environment change?
  2. Alternative retrieval methods. How would other retrieval methods, such as embedding-based search or LSP-based navigation, compare with CodeGraph? Are the observed cost increases specific to CodeGraph’s implementation, or do they reflect a broader limitation of retrieval-based approaches?
  3. Fair comparison between DevSkills and SynSkills. DevSkills contain only seven principles, while SynSkills contain 23–41 rules. Could the authors provide a more controlled comparison with similar prompt lengths, rule counts, and levels of abstraction? Would DevSkills still outperform SynSkills under these conditions?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Rt4B: Detector False Positives and False Negatives (Q1)

We sincerely thank the reviewer for the detailed feedback and for recognizing the practical value of our behavioral analysis and optimization strategies. Your suggestions motivated us to further evaluate detector validity, retrieval alternatives, and skill comparisons. We address each concern below.

Q1

Detector validity. Have the authors manually evaluated the detectors to measure false positives and false negatives? In particular, how often are flagged repeated actions actually useful, such as rereading code inspected by a subagent, verifying an edit, or rerunning tests after an environment change?

Related weakness 2: repeated actions are not always wasteful. The paper treats repeated actions as potential sources of inefficiency, but some of these actions may actually be necessary. For example, if a subagent reads code and only returns a summary, the main agent may still need to read the original code before making changes. Similarly, agents may need to revisit code when working with long contexts or rerun tests after making changes to the environment. These are reasonable behaviors rather than wasted computation. It would be useful for the paper to distinguish between necessary and unnecessary repetition, rather than treating the cost of repeated actions as an indication of inefficiency.

Summary. Two authors independently annotated 300 flagged actions. Among agreed labels, 13.24% of SubRetrv and 6.21% of ReTest actions were judged necessary rather than cost-inefficient (false positives). We further audited 40 randomly sampled trajectories, and found no missed instances of the three identified behaviors (false negatives).

Annotation criteria. As detailed in our response to AhRp, Q1, we exclude justified cross-agent re-reads and test reruns following environment changes.

Regarding the long-context repetitions raised in Weakness 2, we retain cases where agents revisit code or rerun tests despite the relevant information remaining in their available context, as these reflect limitations in context utilization. As a sensitivity check, excluding re-reads more than 32k tokens apart reduces SubRetrv cost by at most 1.13 percentage points.

Regarding the other example in Q2, re-reading code to verify an edit, successful edits change the code and therefore do not trigger SubRetrv. However, failed or silent edits may prompt agents to re-read unchanged code, triggering SubRetrv. We retain these cases because they reflect limitations in agents' ability to reliably modify code (Lines 204–209).

Human validation. We sampled 150 flagged SubRetrv and 150 flagged ReTest pairs from the RQ1 trajectories, stratified by configuration. Following the above annotation protocol, the false-positive results are reported in our response to AhRp, Q1 (93.67% agreement; Cohen's κ = 0.82). For false negatives, we additionally audited 40 randomly sampled trajectories (10 per configuration) and found no missed instances of the three identified behaviors.

Impact of justified repetitions. Cross-agent re-reads account for 50.15% of CC's originally flagged SubRetrv actions. Excluding them reduces CC's attributed SubRetrv cost from 5.01% to 1.28%. Excluding such ReTest pairs that involved intervening package installations or Git operations modifying the working tree removes 4.15–9.83% of originally flagged ReTest actions across configurations.

After refining the detectors, the main RQ1 finding remains: the three behaviors affect 68.67–98.00% of tasks and account for 3.47–22.77% of task cost. Detailed results are provided in our response to AhRp, Q1.

AuthorsOpenReview ↗

Response to Reviewer Rt4B: LSP-Based Retrieval (Q2)

Q2

Alternative retrieval methods. How would other retrieval methods, such as embedding-based search or LSP-based navigation, compare with CodeGraph? Are the observed cost increases specific to CodeGraph’s implementation, or do they reflect a broader limitation of retrieval-based approaches?

Related weakness 1: The paper only evaluates CodeGraph as a retrieval method, without comparing it against alternatives such as embedding-based search or LSP-based navigation. Since the increased costs are partly caused by CodeGraph's verbose outputs and changes in agent behavior, it is unclear whether the same conclusions would hold for other retrieval methods.

Summary. We evaluated LSP-based navigation as an alternative to CodeGraph. Despite substantially shorter outputs, LSP achieves no significant cost reductions and significantly increases cost on MSA(Q35+) (+15.89%). These results suggest that verbose outputs are CodeGraph-specific, while subagent displacement and prompt overhead represent broader challenges for retrieval-based approaches.

Setup. We enabled CC's built-in LSP tool (Pyright) and integrated LSP into MSA via a CLI wrapper, preserving their original toolsets (same as CodeGraph setting). Unlike CodeGraph's verbose code chunks, LSP returns concise symbol-level information. Given rebuttal time and cost constraints, we evaluated four of eight settings spanning different architectures, backbones, and benchmarks, each with one run against three baseline runs. MSA(MM3) was excluded due to model drift (see 8xRG, Q1).

The table compares LSP and CodeGraph against the baselines, using the updated detectors (AhRp, Q1; Rt4B, Q1). Tool calls are averaged per task, and output tokens per retrieval-tool call.

SettingApproachTasks using toolTool callsOutput tokens per tool callTask tokens (M)ΔSubRetrvΔCostΔPass@1, pp
CC, Verified-200CodeGraph91.50%2.235,1990.91 (-1.70%)-78.84%+8.30%-0.83
LSP87.50%1.716350.96 (+3.59%)-16.00%+3.73%+0.00
CC, Pro-100CodeGraph88.51%3.605,2731.70 (-0.36%)-33.83%+12.19%+1.92
LSP54.00%1.166321.69 (+3.67%)+14.53%+6.80%+1.35
MSA (S46), Verified-200CodeGraph95.00%5.078761.09 (-0.37%)-25.33%-3.03%+0.00
LSP96.00%2.151161.11 (+1.03%)-9.04%-6.85%+3.00
MSA (Q35+), Verified-200CodeGraph93.17%6.311,1221.63 (-1.15%)-24.25%-3.29%+5.33
LSP39.50%1.011571.94 (+17.58%)+21.16%+15.89%+0.83

Results and mechanisms. LSP outputs are 7–8× shorter than CodeGraph's, yet no setting shows significant cost savings or Pass@1 degradation. Cost increases significantly on MSA(Q35+) (+15.89%). Two mechanisms help explain the results:

  • Subagent displacement. In CC, both LSP and CodeGraph nearly eliminate delegation to cheaper subagents, shifting work toward the more expensive main agent.
  • Prompt overhead and tool adoption. In MSA, LSP instructions incur prompt costs even when unused. MSA(S46) widely adopts LSP, with little change in total tokens or cost. In contrast, MSA(Q35+) rarely adopts it, leaving prompt overhead largely uncompensated and increasing task cost.

Conclusion. These results extend Finding 4 beyond CodeGraph: even retrieval tools with concise outputs may increase cost due to subagent displacement or insufficient adoption to offset prompt overhead. Generalization to other methods, such as embedding-based search, remains untested.

AuthorsOpenReview ↗

Response to Reviewer Rt4B: Length-Matched SynSkills (Q3)

Q3

Fair comparison between DevSkills and SynSkills. DevSkills contain only seven principles, while SynSkills contain 23–41 rules. Could the authors provide a more controlled comparison with similar prompt lengths, rule counts, and levels of abstraction? Would DevSkills still outperform SynSkills under these conditions?

Summary. Using a stronger synthesizer and pooled trajectories to encourage higher-level abstraction, we generated compact SynSkills with comparable rule counts and prompt lengths. These skills narrow or reverse the gap with DevSkills in three Sonnet 4.6 settings, but still increase cost on MSA(Q35+), where DevSkills remain effective. Both SynSkills and DevSkills benefit from our RQ1 results and DevSkills has more consistent cross-model effectiveness.

Setup. As in the original paper and our response to 8xRG Q1, SynSkills receive our RQ1 diagnoses, including flagged behavior types, frequencies, and step indices (Appendix A.4.1, Lines 1281–1325). To encourage higher-level abstraction, we pooled RQ1 trajectories across all agent configurations, slightly refined the skill-merging prompt to prioritize lessons that generalize across tasks, and used Sonnet 5.5 (high reasoning effort) for synthesis. We constrained the output to at most 10 rules and 500 tokens, comparable to DevSkills' 7 principles (486 tokens), while keeping the remaining pipeline settings unchanged.

We evaluated four representative settings spanning agent architectures, model backbones, and benchmarks (see 8xRG Q1). MSA(MM3) was excluded due to model drift and observed benchmark-gaming behaviors.

The table reports changes relative to the baselines, with behavior statistics computed using the updated detectors (AhRp, Q1; Rt4B, Q1).

SettingApproachΔCostΔPass@1, ppΔSubRetrvΔSimScrptΔReTest
MSA (S46), Verified-200SynSkills (paper)-22.32%-0.50-11.38%-45.10%-36.82%
SynSkills (10 rules)-39.59%+0.50-49.44%-66.46%-36.82%
DevSkills-41.11%+0.17-49.97%-58.83%-34.46%
CC, Verified-200SynSkills (paper)-1.38%-0.50-33.20%-41.85%+12.72%
SynSkills (10 rules)-21.53%+0.00-58.99%-11.32%-20.80%
DevSkills-13.94%-1.50-53.04%-54.93%-4.70%
MSA (S46), Pro-100SynSkills (paper)-6.13%+0.00+2.50%+79.32%-28.25%
SynSkills (10 rules)-42.27%+4.00-48.29%-86.28%-42.98%
DevSkills-33.00%+0.00-43.57%-38.36%-26.98%
MSA (Q35+), Verified-200SynSkills (paper)-0.70%+2.50-5.05%-17.63%-8.37%
SynSkills (10 rules)+10.65%+1.67+4.23%+27.99%-14.19%
DevSkills-11.34%+2.67-17.25%-0.49%-7.89%

Findings. On the three Sonnet 4.6 settings, compact SynSkills reduce cost by 21.53–42.27%, substantially improving over the original SynSkills (1.38–22.32%). They achieve comparable savings to DevSkills on MSA(S46) Verified-200 and greater savings on CC Verified-200 and MSA(S46) Pro-100, without reductions in Pass@1. However, on MSA(Q35+), SynSkills increase cost by 10.65%, whereas DevSkills reduce it by 11.34%.

The unconstrained 54-rule SynSkills exhibit a similar pattern (8xRG, Q1). Since both variants use Sonnet 5.5 and pooled RQ1 trajectories with flagged behaviors in RQ1 but differ substantially in length, their consistent results suggest that rule count and prompt length are not the primary factors. The improvements over the original SynSkills are more likely associated with stronger models and pooled evidence.

Conclusion. These results further demonstrate the potential of our RQ1 diagnoses to guide effective automated skill synthesis. While stronger synthesis and pooled evidence substantially improve SynSkills, DevSkills achieve more consistent cost reductions across model backbones, including where stronger SynSkills fail.

We greatly appreciate these constructive suggestions, which have helped us strengthen our evaluation and clarify the generalizability and limitations of our findings. We will refine the detectors and RQ1 results, add the human validation protocol and analysis, incorporate the LSP evaluation into RQ2, and include the controlled SynSkills comparison in RQ3, with supporting details in the appendix. We hope these additional experiments and analyses adequately address your concerns.

Reviewer_AhRpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper characterizes subsumed retrieval, similar script generation, and test re-execution in 1,200 coding-agent trajectories, then evaluates retrieval tooling and synthesized/developer skills on held-out Verified and Pro tasks. Developer skills deliver substantial cost reductions in several settings, while CodeGraph can increase cost through verbose output and changed delegation.

Critical strengths
  1. The large, separated discovery/evaluation study connects concrete behaviors to end-to-end cost and task success across multiple configurations. The negative retrieval result and cross-benchmark skill evaluation offer actionable evidence beyond simply counting tokens.
Critical weaknesses
  1. The detectors identify repetition more directly than avoidable waste. Main agents may need code seen only by a subagent, and unchanged-patch test reruns may recover information or follow environment changes. Consequently, the reported behavior-attributed cost is not a validated estimate of removable cost.
  2. Some robustness claims transfer variance from just three baseline runs to unreplicated interventions. Validation on nine cells is useful but does not establish equal variance elsewhere or task-success non-inferiority; the strongest conclusions should include uncertainty on the cost–success tradeoff.
Questions for authors
  1. On an independently annotated sample, what fraction of flagged actions is actually unnecessary given the acting agent’s available information and environment state? How do prevalence and cost attribution change after excluding justified repetition?
  2. Do paired task-level intervals and additional independent repeats preserve the main DevSkills savings and success rates, particularly the 41.73% result? Please distinguish absence of significant degradation from evidence of non-inferiority.
4 comments
AuthorsOpenReview ↗

Response to Reviewer AhRp: Human Validation of Detectors (Q1)

We sincerely thank the reviewer for the constructive feedback and for recognizing the value of our large-scale empirical study. Your comments motivated us to strengthen both the validity of our behavior detection and the statistical rigor of our evaluation. We address your questions below.

Q1

On an independently annotated sample, what fraction of flagged actions is actually unnecessary given the acting agent’s available information and environment state? How do prevalence and cost attribution change after excluding justified repetition?

Related weakness 1: The detectors identify repetition more directly than avoidable waste. Main agents may need code seen only by a subagent, and unchanged-patch test reruns may recover information or follow environment changes. Consequently, the reported behavior-attributed cost is not a validated estimate of removable cost.

Summary. Two authors independently annotated 300 flagged actions. Among agreed labels, 86.76% of SubRetrv and 93.79% of ReTest actions were cost-inefficient. After refining the detectors to exclude the two types of justified repetition, the three behaviors still affect 68.67–98.00% of tasks and account for 3.47–22.77% of task cost.

Annotation criteria. Following the reviewer's examples, we exclude two types of justified repetition:

  • SubRetrv: Cross-agent re-reads in Claude Code (CC), where the main agent needs code previously accessed only by a subagent since it only returns a summary.
  • ReTest: Reruns following package installation or Git operations that change the working tree.

We retain two ReTest patterns reflecting agent limitations:

  • Incorrect test runners or configurations: Agents repeatedly attempt tests without first understanding the repository's testing framework, retrieving relevant information only after multiple failures. This reflects limitations in repository understanding (Lines 293–299, 1137–1147 in the paper).
  • Test-signal recovery: Agents rerun tests because previous outputs were truncated, ambiguous, or poorly captured, reflecting limitations in extracting and retaining relevant test information (Lines 293–299, 1148–1165).

Human validation. We sampled 150 flagged SubRetrv and 150 flagged ReTest pairs from the RQ1 trajectories, stratified by configuration. Following the above standards, Two authors independently assessed whether each repetition was useful and thus not cost-inefficient (false positive), achieving 93.67% agreement (Cohen's κ = 0.82). We report the proportion of flagged actions judged cost-inefficient among agreed labels.

SubRetrvReTest
Agreed labels136145
Judged justified (false positives)189
Judged cost-inefficient118 (86.76%)136 (93.79%)

All 18 justified SubRetrv actions were cross-agent re-reads, while all 9 justified ReTest pairs involved intervening package installation or Git operations modifying the working tree. For our false-negative audit, see our response to Rt4B, Q1.

Updated results. We refined the detectors accordingly and fixed a minor bug in ReTest cost calculation. The updated rows in the following table reflect the corrected cost attribution and behavior prevalence for RQ1. Each cell reports task prevalence / attributed cost share.

BehaviorDetectorCCMSA (S46)MSA_MM3MSA (Q35+)
SubRetrvPaper64.33% / 5.01%87.33% / 8.42%92.33% / 7.88%89.33% / 11.41%
Updated40.00% / 1.28%87.33% / 8.42%92.33% / 7.88%89.33% / 11.41%
SimScrptUnchanged20.67% / 1.02%51.33% / 9.57%68.00% / 8.85%57.67% / 7.91%
ReTestPaper49.67% / 0.83%69.00% / 3.17%83.00% / 5.39%66.00% / 3.43%
Updated49.00% / 1.17%67.33% / 3.16%83.00% / 5.02%64.33% / 3.46%
TotalPaper79.00% / 6.86%97.33% / 21.16%98.00% / 22.12%96.67% / 22.75%
Updated68.67% / 3.47%97.33% / 21.15%98.00% / 21.75%96.67% / 22.77%

The main RQ1 finding persists: cost-inefficient behaviors remain widespread (occur in 68.67–98.00% of tasks and account for 3.47–22.77% of task cost, compared with 79.00–98.00% and 6.86–22.75% in the paper). The largest change occurs in CC, where excluding justified cross-agent re-reads reduces SubRetrv prevalence from 64.33% to 40.00% and attributed cost from 5.01% to 1.28%. Changes in MSA are minor.

We agree that behavior-attributed cost is not directly removable cost. Avoiding flagged actions may require better repository understanding, information retention, or test-output interpretation. We therefore treat this metric as a diagnostic indicator of optimization opportunities, not a prediction of achievable savings.

AuthorsOpenReview ↗

Response to Reviewer AhRp: Paired CIs with Independent Repeats (Q2)

Q2

Do paired task-level intervals and additional independent repeats preserve the main DevSkills savings and success rates, particularly the 41.73% result? Please distinguish the absence of significant degradation from evidence of non-inferiority.

Related weakness 2: Some robustness claims transfer variance from just three baseline runs to unreplicated interventions. Validation on nine cells is useful but does not establish equal variance elsewhere or task-success non-inferiority; the strongest conclusions should include uncertainty on the cost–success tradeoff.

Summary. Yes. Paired task-level confidence intervals and two additional independent runs confirm substantial DevSkills cost savings, including the reported 41.73%. Pass@1 shows no significant degradation, although non-inferiority is not established in every setting.

Validation. We computed paired 95% CIs using 10,000 bootstrap resamples over tasks and added two additional DevSkills runs under MSA(S46) on Verified-200 with identical configurations.

RunPass@1CostCost saving [95% CI]ΔPass@1, pp [95% CI]ΔSubRetrvΔSimScrptΔReTest
Baseline (3 runs)72.50%$0.639--3.843.492.14
DevSkills 1 (paper)73.00%$0.372-41.73% [-34.08, -48.96]+0.50 [-2.67, +3.67]1.86 (-51.52%)1.18 (-66.16%)1.31 (-38.69%)
DevSkills 275.00%$0.396-36.74% [-28.26, -44.93]+2.50 [-0.50, +5.67]2.01 (-47.74%)1.49 (-57.34%)1.59 (-25.45%)
DevSkills 370.00%$0.371-40.38% [-31.33, -48.90]-2.50 [-5.83, +0.67]1.89 (-50.64%)1.64 (-52.92%)1.30 (-39.25%)
DevSkills (3 runs)72.67%$0.380-41.11% [-35.29, -46.84]+0.17 [-2.17, +2.50]1.92 (-49.97%)1.44 (-58.83%)1.40 (-34.46%)

Across three runs, DevSkills reduces cost by 41.11% [35.29, 46.84] while changing Pass@1 by +0.17 pp [-2.17, +2.50], so the saving comes with no detectable loss in task success. Each run alone saves 36.74–41.73%, and every cost CI excludes 0. Under the updated detectors, all three behaviors consistently decrease as well.

Pass@1 non-inferiority. Using a 5 pp margin fixed before the reruns, non-inferiority holds for runs 1–2 and the pooled estimate, but not run 3 alone (lower bound: −5.83 pp). Across all eight DevSkills settings, none show significant Pass@1 degradation, while five establish non-inferiority. The remaining three are inconclusive, primarily due to wide intervals.

Other Approaches. Paired CIs also preserve the main SynSkills and CodeGraph findings. SynSkills significantly reduces cost in three settings (8.86–22.32%), without significant Pass@1 degradation. CodeGraph yields no significant cost reductions but increases cost in four settings (8.30–28.14%). Pass@1 non-inferiority holds in six of eight settings for each intervention; the others remain inconclusive.

We appreciate these valuable suggestions, which have helped us strengthen the empirical evidence and clarify the limitations of our findings. We hope our additional analysis and experiments adequately address your concerns. We will update the detectors, revise the paper's main claims to report cost savings and Pass@1 changes together, each with a paired 95% CI. The appendix will provide both intervals for all 24 intervention settings.

AuthorsOpenReview ↗

Revised Paper Uploaded

Thank you again for your constructive review. We have uploaded a revised version of the paper incorporating the analysis and experiments from our responses. The main changes related to your comments are:

  • Q1 (detector validity and justified repetition): We refined the SubRetrv and ReTest detectors (Sec. 3.1 and 3.3), and updated all results accordingly (Table 1, Abstract, Introduction, and so on). The human validation based on the refined detectors, the false-negative audit, and extra clarifications are in Appendix A.1.3.
  • Q2 (uncertainty and non-inferiority): A change is now considered robust only if its paired task-level 95% CI excludes zero and it also passes the noise-floor test (Sec. 2). The main claims now report cost savings together with Pass@1 changes and their CIs, including the two additional DevSkills runs (Sec. 4.3). Paired CIs for all intervention settings and the Pass@1 non-inferiority analysis are in Appendix A.1.5.

We would greatly appreciate it if you could let us know whether the revision and our responses address your concerns. As the discussion period is coming to an end, any feedback would be very helpful, and we are happy to clarify anything further.

Area_Chair_Zp76OpenReview ↗

Reminder from AC

Dear Reviewers,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.