- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies cost-inefficient behaviors in coding agents. By analyzing trajectories of Claude Code and Mini-SWE-Agent on SWE-bench, it identifies three recurring behaviors: redundant code retrieval, regeneration of near-duplicate scripts, and repeated test execution without code changes. It then evaluates three mitigation approaches. Structure-aware retrieval does not reliably reduce cost and can even increase it. Prompt-level skills are more effective, and developer-designed skills outperform agent-synthesized ones, which the authors attribute to their higher-level, more general guidance.
Critical strengths
-
This paper studies an important problem about cost inefficiency in coding agents. The behavior analysis is well-structured and informative, and for each the paper analyzes both how it is detected and why it arises across agent architectures.
-
The negative results on structure-aware retrieval are interesting. Although the code-graph tool reduces redundant retrieval, it does not necessarily lower the overall cost and sometimes even increases it because of longer outputs and changes in subagent delegation.
Critical weaknesses
Minor concern: The comparison between DevSkills and SynSkills does not fully isolate the benefit of human abstraction. While SynSkills are generated from individual trajectories and then consolidated, DevSkills are informed by the broader RQ1 analysis across configurations. SynSkills also use each configuration's backbone model, which may limit synthesis quality. These differences make it difficult to determine whether the performance gap comes from human-designed guidance or from the synthesis setup.
Questions for authors
-
Would using a stronger model or providing the aggregated RQ1 analysis improve SynSkills and narrow the gap with DevSkills?
-
One DevSkill, "Break Loops," encourages agents to submit their best solution when stuck. Could some of the cost savings come from early termination rather than directly reducing the three identified behaviors? An ablation without this skill would help clarify its contribution.
5 comments
Response to Reviewer 8xRG: Break Loops Ablation Study (Q2)
Q2
One DevSkill, "Break Loops," encourages agents to submit their best solution when stuck. Could some of the cost savings come from early termination rather than directly reducing the three identified behaviors? An ablation without this skill would help clarify its contribution.
Summary. Removing "Break Loops" yields comparable cost savings, API calls, and reductions in the three identified behaviors, suggesting that early termination is not the primary source of DevSkills' savings.
Setup. We removed "Break Loops" while retaining the other six DevSkills principles and all experimental settings. We evaluated MSA(S46) on Verified-200 (the largest reported saving: 41.73%) and CC on Verified-200 (a reported −1.50 pp Pass@1 change). Due to evaluation costs, each ablation uses one run, compared against three-run DevSkills results, including two additional MSA(S46) runs conducted for AhRp, Q2.
| Setting | Approach | Pass@1 | Cost saving vs. baseline | API calls | SubRetrv | SimScrpt | ReTest |
|---|---|---|---|---|---|---|---|
| MSA (S46) | DevSkills | 72.67% | -41.11% | 31.97 | 1.92 (-49.97%) | 1.44 (-58.83%) | 1.40 (-34.46%) |
| MSA (S46) | DevSkills w/o Break Loops | 72.50% | -43.05% | 31.10 | 1.98 (-48.52%) | 1.22 (-65.17%) | 1.31 (-38.46%) |
| CC | DevSkills | 73.50% | -13.94% | 20.43 | 0.35 (-53.04%) | 0.15 (-54.93%) | 1.45 (-4.70%) |
| CC | DevSkills w/o Break Loops | 75.00% | -17.12% | 20.07 | 0.33 (-57.01%) | 0.16 (-53.48%) | 1.36 (-10.61%) |
Findings. Removing "Break Loops" yields comparable cost savings (43.05% vs. 41.11% on MSA; 17.12% vs. 13.94% on CC), API calls, and reductions in all three behaviors. Pass@1 differs by at most 1.50 pp. If "Break Loops" were driving savings through early termination, removing it would be expected to increase API calls and cost; neither occurs. These results suggest that early termination is not the primary source of savings, although the single-run ablations limit stronger causal claims.
Thank you again for these thoughtful suggestions. The additional experiments have provided valuable insights into skill synthesis and the sources of cost savings, helping us refine our conclusions. We will incorporate the Sonnet 5.5 SynSkills results and the "Break Loops" ablation into RQ3 and the appendix. We hope these new results adequately address your concerns.
Revised Paper Uploaded
Thank you again for your encouraging review and insightful questions. We have uploaded a revised version of the paper. The main changes related to your comments are:
- Q1 (stronger synthesizer and pooled RQ1 analysis): The Sonnet 5.5 SynSkills results with pooled trajectories (54-rule and compact 10-rule variants) are now in Sec. 4.3, with Finding 5, the Abstract, and the Introduction revised accordingly. Sec. 4.3 also clarifies that all SynSkills use our RQ1 diagnoses as input. Synthesis details are in Appendix A.4.
- Q2 ("Break Loops" ablation): The ablation without this principle is in Appendix A.4.2 (Table 20).
We would greatly appreciate it if you could let us know whether the revision and our responses address your concerns. As the discussion period is coming to an end, any feedback would be very helpful, and we are happy to clarify anything further.
Response to Reviewer 8xRG: Stronger SynSkills with Pooled Trajectories (Q1)
We sincerely thank the reviewer for the encouraging assessment and insightful questions. We especially appreciate your recognition of the importance of studying cost inefficiency in coding agents. Your comments motivated us to further investigate the effectiveness of automated skill synthesis and the mechanisms behind DevSkills' cost savings.
Q1
Summary. With a stronger synthesizer (Sonnet 5.5) and pooled trajectories across agent configurations, SynSkills match or exceed DevSkills' savings in three Sonnet 4.6 settings, but increase cost on MSA(Q35+), where DevSkills remain effective. These results highlight that both SynSkills and DevSkills benefit from our RQ1 results and DevSkills has stronger cross-model consistency.
RQ1-guided synthesis. Importantly, all SynSkills experiments, including those in the original paper, this response, and our response to Rt4B Q3, use our RQ1 diagnoses as input, including flagged cost-inefficient behavior types, their frequencies, and flagged step indices (Appendix A.4.1, Lines 1281–1325). Thus, SynSkills already benefit from our behavioral analysis, even though their effectiveness leaves room for improvement.
Setup. In the paper, SynSkills were synthesized separately for each configuration using its own backbone model and RQ1 trajectories. Here, we reran the pipeline using Sonnet 5.5 (high reasoning effort) and pooled trajectories from all four configurations to enable unified, cross-configuration synthesis. We retained the original prompt templates and experiment settings. The resulting shared skill set contains 54 rules (1,880 words).
We evaluated the four settings where DevSkills had significantly lower cost than the original SynSkills. MSA(MM3) was excluded due to model drift (recent Pass@1: 94.00% vs. 75.17% originally) and frequent observed benchmark-gaming behaviors, making a reliable comparison infeasible.
On the three Sonnet 4.6 settings, the new SynSkills reduce cost by 24.94–39.26%, substantially improving over the original SynSkills (1.38–22.32%). They match DevSkills on both MSA(S46) settings and outperform them on CC, without significant Pass@1 changes. However, on MSA(Q35+), SynSkills increase cost by 21.78%, whereas DevSkills reduce it by 11.34%. A compact 10-rule SynSkills variant exhibits the same pattern (see Rt4B, Q3).
Overall, these results demonstrate that our RQ1 diagnoses can guide effective automated skill synthesis, with further improvements from stronger synthesis and pooled evidence. Nevertheless, DevSkills deliver more consistent cost reductions across model backbones, including where stronger SynSkills fail.