- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TokenEconomizer is an external controller that reduces coding-agent costs by learning from completed tasks. It selectively injects repository-specific operational micro-skills, prunes unnecessary tools, and intervenes in unproductive failure loops through existing agent interfaces. The paper evaluates three coding agents across five agent–model configurations and four benchmarks totaling 579 tasks. On SWE-Bench, it reports 18.4–30.9% lower cache-aware inference costs, including controller overhead, with broadly comparable success rates. Component ablations and evaluations on sequential web development and security auditing provide additional evidence of effectiveness.
Critical strengths
1.Practical problem and deployable approach.
The method targets recurring costs in coding workflows without requiring model training or changes to the agent’s internal execution loop. Its implementation across three agents strengthens its practical relevance.
2.Consistent empirical savings.
Costs decrease in all 30 SWE-Bench repository–configuration combinations, while aggregate success rates remain close to the baseline. Including controller overhead, reporting three seeds, and providing component ablations make these results more informative.
Critical weaknesses
1.The benefit of adaptation is insufficiently isolated.
Tool pruning provides the largest individual savings, but many removed tools appear unnecessary for unattended benchmark execution. Comparisons against a fixed minimal tool configuration, a short static repository manual, and a simple restart/budget policy are missing. Consequently, the additional value of the proposed adaptive mechanisms remains unclear.
2.Evidence for realistic continual development is limited.
SWE-Bench tasks reset to separate base commits, so they primarily test operational knowledge reuse rather than development on an evolving workspace. Web-Bench addresses this distinction, but covers only five projects and excludes projects where the default agent fails early. This limits the breadth of the continual-workflow claim.
Questions for authors
1.How much improvement remains over fixed minimal tools, static operational instructions, and a simple restart/budget baseline? 2.How were development tasks separated from evaluation tasks, and how sensitive are the Web-Bench results to including the excluded projects? 3.Appendix I.1(c) shows failed commands being counted as micro-skill confirmations. How common is this issue, and does requiring successful execution change the results?
2 comments
Thank you for the additional experiments. The comparison with a fixed, lean tool set addresses my main concern. Adaptive tool pruning increased cost savings by 6.7 and 4.6 percentage points across the two tested agent configurations, with nearly unchanged Pass@1. The full system achieved further savings while maintaining comparable success rates. These results provide convincing evidence that the adaptive policy adds value beyond a simpler static configuration in the tested settings. I am therefore raising my rating to 3.
Author Response to Reviewer KPcC
We appreciate your recognition of TE's practical value and contributions. Here we provide our responses to your concerns, with some new experiments also presented in the global response (G1–G3).
W1 / Q1. Isolating the benefit of adaptation.
We added the fixed-tool and manual baselines (G1):
CC = Claude Code + Sonnet 5; Codex = Codex + GPT-5.6-terra; sklearn + sympy, three seeds (std in G1).
W2. Limited evidence for continual development.
To our knowledge, no benchmark yet offers long development sessions in large real repositories with standardized grading. Real interactive sessions are a natural next test for us, where TE can run as a plugin for these mainstream agents.
Q2. Development tasks and excluded Web-Bench projects.
Q3. Failed commands counted as confirmations.
See G3: 340 of 7,956 confirmations (4.3%) came from a failed command, and in 148 (1.9%) the skill was not retired afterward. Where a wrong skill stayed active, TE added only 0.1–0.3 failed commands per task and still cut cost. A straightforward next step, left as future work, is to require successful execution.
We hope these responses have effectively addressed your concerns, and we are open for further discussions.