acceptodds
Under review as a conference paper at ICLR 2027

TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents

Abstract

Advances in frontier coding agents have sharply increased the token costs of AI-assisted software engineering. Their token efficiency is typically evaluated on isolated tasks, whereas real-world developers use them to continuously build, update, and maintain large-scale projects. This mismatch motivates us to evaluate agents over sequences of realistic coding tasks, where we find that agents exhibit recurring environment-dependent token costs that existing methods struggle to reduce. To minimize these costs, we introduce TokenEconomizer, an external controller that cuts environment-specific token costs over a sequence of tasks. These cuts come from determining (i) which operational micro-skills to inject, (ii) which persistently irrelevant tools to prune, and (iii) when to intervene in unproductive failure loops. Across Claude Code, Codex, and GitHub Copilot CLI paired with frontier models, TokenEconomizer reduces the average token cost by 18.4–30.9% with comparable task success. Savings hold in every repository under every tested agent–model setting and generalize to diverse domains such as sequential web development and security auditing, where costs fall by 6.4–21.2% depending on task structure. Our work establishes adaptive black-box cost control as a promising direction for reducing token costs of coding agents in realistic settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agents, Token Efficiency, Cost Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KPcCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TokenEconomizer is an external controller that reduces coding-agent costs by learning from completed tasks. It selectively injects repository-specific operational micro-skills, prunes unnecessary tools, and intervenes in unproductive failure loops through existing agent interfaces. The paper evaluates three coding agents across five agent–model configurations and four benchmarks totaling 579 tasks. On SWE-Bench, it reports 18.4–30.9% lower cache-aware inference costs, including controller overhead, with broadly comparable success rates. Component ablations and evaluations on sequential web development and security auditing provide additional evidence of effectiveness.

Critical strengths

1.Practical problem and deployable approach.

The method targets recurring costs in coding workflows without requiring model training or changes to the agent’s internal execution loop. Its implementation across three agents strengthens its practical relevance.

2.Consistent empirical savings.

Costs decrease in all 30 SWE-Bench repository–configuration combinations, while aggregate success rates remain close to the baseline. Including controller overhead, reporting three seeds, and providing component ablations make these results more informative.

Critical weaknesses

1.The benefit of adaptation is insufficiently isolated.

Tool pruning provides the largest individual savings, but many removed tools appear unnecessary for unattended benchmark execution. Comparisons against a fixed minimal tool configuration, a short static repository manual, and a simple restart/budget policy are missing. Consequently, the additional value of the proposed adaptive mechanisms remains unclear.

2.Evidence for realistic continual development is limited.

SWE-Bench tasks reset to separate base commits, so they primarily test operational knowledge reuse rather than development on an evolving workspace. Web-Bench addresses this distinction, but covers only five projects and excludes projects where the default agent fails early. This limits the breadth of the continual-workflow claim.

Questions for authors

1.How much improvement remains over fixed minimal tools, static operational instructions, and a simple restart/budget baseline? 2.How were development tasks separated from evaluation tasks, and how sensitive are the Web-Bench results to including the excluded projects? 3.Appendix I.1(c) shows failed commands being counted as micro-skill confirmations. How common is this issue, and does requiring successful execution change the results?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer KPcC

We appreciate your recognition of TE's practical value and contributions. Here we provide our responses to your concerns, with some new experiments also presented in the global response (G1–G3).

W1 / Q1. Isolating the benefit of adaptation.

We added the fixed-tool and manual baselines (G1):

MethodCC Pass@1CC cost vs. defaultCodex Pass@1Codex cost vs. default
Default75.0–74.2–
Fixed tools74.7−7.5%73.3−9.2%
Fixed tools + manual74.4−5.8%73.3−3.2%
TE, tool pruning only74.4−14.2%73.6−13.8%
TE (full)76.4−26.6%75.0−24.2%

CC = Claude Code + Sonnet 5; Codex = Codex + GPT-5.6-terra; sklearn + sympy, three seeds (std in G1).

  • Fixed minimal tools. One tool list per repository, chosen in advance by an LLM judge. It saves 7.5% / 9.2%; TE's pruning alone saves 14.2% / 13.8%, because TE sees which tools actually go unused and judges again for each task.
  • Short manual. A manual of similar length to TE's injected micro-skills, written from the same past sessions but without TE's checks. Adding it reduces the fixed list's saving: wrong or rarely needed lines are re-sent on every turn and never removed.
  • Restart/budget policy. We did not add a separate baseline, because TE's failure loop intervention is already a simple restart-and-budget policy: its redirect restarts the agent once with its edits kept, and its last-resort stop is a turn budget. It adds only two things:
    • A learned budget. No turn limit is set by hand: it is learned from each repository's earlier successful tasks, and results are not very sensitive to its margin (G2).
    • A judge before each restart. In the ablation with only failure loop intervention, the judge was consulted 429 times and found the agent stuck in only 188 (44%).

W2. Limited evidence for continual development.

  • What we claim. TE reuses operational experience from earlier tasks in a repository: which commands work, which tools are used, and how long successful tasks take. None of this relies on the agent's earlier edits staying in the workspace, so SWE-Bench streams test what TE is built for.
  • SWE-Bench streams also evolve. Tasks are real issues from one repository in commit order, so the codebase changes between tasks through the project's own commits, as a shared repository does between sessions. This is a harder test than a workspace only the agent edits: each stream spans 1.9–8.5 years, over which layouts, test commands, and dependencies drift. TE still saves cost in all 30 repository–setting pairs.
  • Web-Bench tests the evolving workspace directly. Each task builds on code the agent wrote earlier. Its five projects cover Web standards and frameworks, under all five agent–model settings. TE cuts cost by 16.0–21.2% on tasks both methods reached. Counting every task each method attempted, it still saves 11.7–18.8% per task (see our reply to yLW4), with comparable Pass@2.

To our knowledge, no benchmark yet offers long development sessions in large real repositories with standardized grading. Real interactive sessions are a natural next test for us, where TE can run as a plugin for these mainstream agents.

Q2. Development tasks and excluded Web-Bench projects.

  • Development tasks. As mentioned in G2, some also appear in the SWE-Bench evaluation streams, because only a limited number of tasks in one repository are close enough in time. Development only set generic thresholds; each evaluation stream starts with no skills, tool history, or turn budget. Web-Bench and the security benchmarks were never used to choose any value, and results are not very sensitive to the values.
  • Excluded projects. A Web-Bench project stops at the first task that fails both attempts. If default execution fails within the first few tasks, the stream ends before TE's learned parts take effect (on SWE-Bench, TE needs up to six tasks to become cheaper than default, Figure 4c), and costs over so few tasks would be very noisy. So we did not run TE on these projects.

Q3. Failed commands counted as confirmations.

See G3: 340 of 7,956 confirmations (4.3%) came from a failed command, and in 148 (1.9%) the skill was not retired afterward. Where a wrong skill stayed active, TE added only 0.1–0.3 failed commands per task and still cut cost. A straightforward next step, left as future work, is to require successful execution.

We hope these responses have effectively addressed your concerns, and we are open for further discussions.

Thank you for the additional experiments. The comparison with a fixed, lean tool set addresses my main concern. Adaptive tool pruning increased cost savings by 6.7 and 4.6 percentage points across the two tested agent configurations, with nearly unchanged Pass@1. The full system achieved further savings while maintaining comparable success rates. These results provide convincing evidence that the adaptive policy adds value beyond a simpler static configuration in the tested settings. I am therefore raising my rating to 3.

Reviewer_yLW4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TokenEconomizer, which accumulates execution experience across tasks to learn repository-specific operational micro-skills, prune tool interfaces, and intervene in failure loops, thereby reducing the cumulative cost of coding agents. The method operates through external interfaces without modifying model weights or the agent’s internal loop. Experiments cover three agent harnesses, five model configurations, and multiple workloads, reporting cost reductions of 18.4%–30.9% on SWE-Bench, with task success rates generally close to default execution.

Critical strengths

The research question has practical value. Compared with isolated tasks, task sequences provide a more appropriate setting for studying the reuse of environment knowledge and cost amortization.

The cost accounting includes auxiliary controller calls and distinguishes cached from uncached inputs. This avoids apparent savings obtained by shifting task costs to the controller.

Cost reductions are relatively consistent across configurations: all 30 repository–configuration combinations on SWE-Bench show savings. The appendix also provides concrete decision processes and failure cases, which help clarify the method’s limitations.

Critical weaknesses
  1. The lack of baselines for basic experience maintenance and reasonable tool configuration leaves the method’s incremental contribution unclear.

This is my main concern. The paper treats repeated discovery of environment knowledge as an important source of avoidable cost, but actual development workflows commonly preserve such knowledge through project instructions, test and build scripts, and persistent operational notes. The representative micro-skills in the appendix, such as test entry points and reproduction procedures, are precisely the kinds of information these project resources typically cover.

The comparisons in §4.1 include default execution, reduced reasoning effort, RTK, and RepoMap, but not simple ongoing experience maintenance. RepoMap provides structural information and does not replace operational instructions. The paper also does not adequately explain which existing project resources the default agent loads, or whether the baseline can preserve and update experience between tasks.

Tool pruning raises a similar concern. It is the largest individual source of savings in Table 2a, yet the tools removed in Appendix I.2 include capabilities such as scheduling and interactive questions that are clearly inapplicable to tasks without human interaction. The experiments do not establish how much of this benefit requires adaptation from historical traces rather than a reasonable tool configuration established beforehand.

I do not assume that project documentation would necessarily eliminate the reported gains. Indeed, Gloaguen et al., cited by the paper, also show that context files can increase costs. This makes a direct comparison necessary. A simple experience manual using the same historical digests and a comparable length budget, together with a fixed toolset suited to the evaluation tasks, would provide direct and feasible controls. The current results support an improvement over default configurations, but do not establish how much additional value comes from the more complex verification, gating, and adaptation mechanisms.

  1. The micro-skill confirmation mechanism may mistake the occurrence of a behavior for evidence that a fact holds.

Appendix I.1(c) presents a noteworthy failure: an incorrect pytest instruction was activated because the confirmation detector matched the command itself rather than its execution outcome. Earlier failed attempts were therefore counted as supporting evidence.

The subsequent retirement of the skill is useful to see. However, this case also shows that repeated confirmations, non-overlapping edited files, and greater weight on contradictions all depend on the detectors having the correct semantics in the first place. If confirmation detectors are too permissive or contradiction detectors miss counterexamples, the amount of evidence cannot guarantee that a skill is correct.

I suggest auditing a sample of activated skills and their corresponding trajectories, reporting whether the confirmations genuinely support the skill claims, how long incorrect skills remain injected, and whether any counterexamples go undetected for an extended period. This could largely use existing records without rerunning all experiments.

Furthermore, the need rate counts use of the method described by a skill as evidence of “need,” but use does not directly establish a net benefit from injection. Injection itself may also increase the frequency with which the method is used. This metric should therefore be treated as a proxy for relevance rather than direct validation of skill utility.

  1. The evidence for “preserving quality,” and the interpretation of some cost results, require further analysis.

Table 1 reports means and standard deviations across three seeds, but does not sufficiently characterize quality differences relative to default execution. Similar average success rates do not establish that failure intervention is without cost: redirection may help some tasks, while early termination may sacrifice difficult tasks that could otherwise have been completed. Success rates decline in some settings in Table 2a, so differences of only a few percentage points are not, by themselves, convincing evidence that cost savings do not come at the expense of success.

I suggest reporting quality differences on matched tasks along with an analysis of uncertainty. For the stopping mechanism, a limited subset could record the points at which termination would have occurred while allowing execution to continue to the original limit, measuring subsequent success and additional cost. This would more directly establish whether the savings primarily come from failure trajectories that are unlikely to recover.

The Web-Bench results also need a clearer qualification. Appendix D excludes projects on which default execution fails early, while the costs in Table 2 are compared only over tasks reached by both methods. This measures the cost of jointly reached tasks, but is not equivalent to the cost of completing an entire project. Reporting the numbers of tasks actually reached and completed by each method, total expenditure, and costs on jointly reached tasks would help avoid conflating these different quantities.

Questions for authors

Which project instructions does the default agent load automatically, and is it allowed to preserve or update operational notes and scripts across tasks? Are comparisons with a simple ongoing experience manual and a task-appropriate fixed toolset available? If so, how much additional benefit does the full method provide over these baselines?

Have the confirmation and contradiction detectors of activated skills been audited? How many validate the facts stated by the skills rather than merely matching commands or filenames? Beyond Appendix I.1(c), how common are cases in which incorrect skills are activated or not retired promptly?

Can the existing results separately show the effects of redirection and early termination on success rates and costs? Is there evidence that terminated trajectories would remain unlikely to succeed if allowed to continue? For Web-Bench, could you provide the numbers of tasks actually reached and completed by each method, together with total expenditure, to contextualize the costs reported on jointly reached tasks?

Do the success labels used to estimate historical budgets contain information available only from the official evaluation and unavailable during execution? Would the budget estimates differ substantially if they instead used test results visible to the agent?

Do the development runs described in Appendix C overlap in tasks with the formal evaluation? Which results were used to determine the parameters, and were the formal evaluation tasks first run only after those parameters had been frozen?

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer yLW4 (1/3)

Thank you for the thorough and constructive review. Here we respond by topic, pairing each weakness with its related questions; some of our new experiments are also in the global response (G1–G3).

W1 / Q1. Baselines for experience maintenance and tool configuration.

What default execution loads and keeps. Each agent loads its own system prompt and tool definitions, but no project instruction file, as the SWE-Bench repositories have none by default. In the standardized evaluation pipeline, each task runs in a fresh container, so notes or scripts that the agent writes during one task are not available in the next. The agents' built-in memory also does not run in this setup (G1): Claude Code's memory-writing step is off by default, and Codex writes memory only after a session has been idle for 6 hours, which never happens when tasks run back to back. Default execution therefore keeps no agent-native memories across tasks.

New baselines (G1). We added the two controls you suggested, a fixed toolset and a simple manual, plus the agents' built-in memory, which updates project memory across tasks:

MethodCC Pass@1CC cost vs. defaultCodex Pass@1Codex cost vs. default
Default75.0–74.2–
Built-in memory74.2+8.7%74.4+10.3%
Fixed tools74.7−7.5%73.3−9.2%
Fixed tools + built-in memory74.2−2.2%73.6+1.5%
Fixed tools + ungated manual74.4−5.8%73.3−3.2%
TE, tool pruning only74.4−14.2%73.6−13.8%
TE (full)76.4−26.6%75.0−24.2%

CC = Claude Code + Sonnet 5; Codex = Codex + GPT-5.6-terra; sklearn + sympy, three seeds (std in G1).

  • Fixed tools. One tool list per repository, chosen once before the first task and kept for every task. An LLM judge chooses it from each tool's description, the repository's README and file listing, and a description of the job (fixing issues in this repository without a human), as an operator could do beforehand. It saves 7.5% / 9.2%, while TE's pruning alone saves 14.2% / 13.8%. A list fixed in advance therefore gets part of the saving, but not all of it. TE saves more because it does not rely on descriptions alone: it also sees which tools the agent actually leaves unused in this repository, and it judges again for each task.
  • Fixed tools + ungated manual. This is the simple ongoing manual you suggested, added on top of fixed tools. TE's own proposer writes it from the same session digests, with the same prompt and schedule, and it is of similar length to TE's injected micro-skills. The only difference is that it skips TE's checks: every line is injected right away, with no confirmation by later tasks, no need gate, and no retirement when contradicted. Adding it reduces the saving of fixed tools (from 7.5% / 9.2% to 5.8% / 3.2%), because wrong or rarely needed lines are re-sent on every turn and never removed.
  • Built-in memory. It raises cost by 8.7% / 10.3% with no gain in Pass@1, and adding it to fixed tools largely cancels their saving (−2.2% / +1.5%). It fits a different scenario: interactive work, where it carries context from a user's earlier conversations (preferences, corrections, project decisions) into later ones. Here, with no user, writing notes takes extra model calls and nothing checks whether a note is correct or still needed. TE instead learns from real traces (working commands, used tools, turn counts) and also updates what it has learned during later sessions.

Additional benefit of full TE. Full TE saves 26.6% / 24.2%, which is 15–19 points more than the best baseline (fixed tools), with Pass@1 at or slightly above default. A reasonable configuration set beforehand thus accounts for 7.5–9.2 points of this saving; the rest comes from what TE learns during the stream. The ungated manual differs from TE's micro-skills only in the gating mechanisms. Adding the ungated manual to fixed tools raises cost, whereas TE's gated micro-skills lower it (Table 2a), which shows that the gating is beneficial.

AuthorsOpenReview ↗

Author Response to Reviewer yLW4 (2/3)

W2 / Q2. Micro-skill confirmations.

We agree that counting confirmations only works if the pattern-matching detectors are largely reliable, so we audited them (details in G3).

  • Why a wrong confirmation can occur. Each skill has two detectors, which the proposer writes as patterns: one confirms that the skill held in a task, and one contradicts it. The confirmation pattern often matches only the command, so whether the command worked is left to the contradiction pattern, which is checked first. The proposer sees which commands earlier tasks ran but not their outputs, so the contradiction pattern has to guess what a failure looks like. If the guess misses, a failed command is counted as a confirmation, as in Appendix I.1(c).
  • Whether confirmations support the skill. In our main Claude Code runs, 340 of 7,956 confirmations (4.3%) came from a failed command, and in 148 (1.9%) the skill was not retired afterward. Most come from one repository, sympy, whose test environment lacks pytest: a skill that recommends pytest is confirmed whenever the agent runs the command, even though the command fails. We also ran a broader blind audit, which is not limited to failed commands and covers any confirmation that does not truly support its skill. We randomly sampled 3 skills from each of the 36 Claude Code streams (6 repositories × 2 models × 3 seeds) and 4 confirmations from each skill, giving 432 confirmations from 108 skills. Two stronger LLM judges (Claude Opus 5.5 and GPT-5.6-sol) both found a missed contradiction in only 16 (3.7%).
  • Wrong skills: how common and how long they stay. Of the 108 audited skills, 79 were injected. Both judges found contradictions for 4 of these 79 (5.1%), and 6 more were flagged by only one judge. TE retires a skill quickly once it detects a contradiction: of the 86 retired skills in the Claude Code runs we re-examined, 35 were retired before they were ever injected, and the other 51 were injected for a median of 5.5 tasks. We acknowledge that a wrong skill that remains undetected stays until the stream ends: these 4 skills were injected on 9 to 56 tasks and never retired, even though evidence against them had appeared before most of these tasks. However, the measured cost in the main case is small. On sympy, the default agent also tries pytest by itself, and where the wrong pytest skill stayed active, TE added only 0.1–0.3 failed commands per task and still cut cost on every sympy stream.
  • Next-step improvement. The audit above shows that the current detectors are largely reliable. A straightforward next step, which we leave as future work, is to make them stricter without changing the method: a confirmation would count only if its command succeeded, and any failure of a command named in the skill (a missing tool, bad arguments, no tests collected) would count as a contradiction. The proposer could also be shown error messages from earlier tasks, so it can write better patterns. This would affect only the few confirmations that came from a failed command (4.3%).
  • Need rate. Among the tasks where a skill's situation arose, the need rate is the fraction in which the agent either made the mistake the skill prevents or used the fact it states. We agree that it shows how often a skill is relevant, not how much it helps. Its role is only to keep rarely relevant skills out of the prompt, since an injected skill is re-sent on every turn. We will revise the text to describe it as a measure of relevance, to avoid this confusion.
AuthorsOpenReview ↗

Author Response to Reviewer yLW4 (3/3)

W3 / Q3. Fine-grained analysis of failure loop intervention, and Web-Bench accounting.

Redirects and stops, separately. We examined every intervention in the ablation with only failure loop intervention (Sonnet 5 and Opus 4.8, three seeds, 1,662 runs):

  • Frequency. 166 runs (10%) were redirected and 56 (3.4%) were stopped: 22 by the judge after a redirect and 34 by the last-resort rule.
  • Mistaken interventions. A panel of three stronger models (Claude Fable 5.2, GPT 6 Astra, Gemini 4 Argon; majority vote) reviewed the judge's 188 interventions and found 26 (14%) mistaken. Compared with default on the same tasks, 12 of these still passed, 6 failed under default too, and 8 turned a pass into a failure (0.5% of all runs).
  • Stopped runs. To see what happens without a stop, we compared these 56 runs with default runs of the same task still running at that point. Of these, 23% still passed because the fix was already in place (e.g., the loop occurred during verification); in 57%, default failed too, so the stop only saved cost; in 20%, default passed, so the stop lost the task (including last-resort stops, which the judge panel above did not review). In real deployment, the stop will be replaced by a notice to the user, who decides whether to stop.
  • Redirects. A redirect can also help: in example (h) of Appendix I.3, the redirected run passes in 31 turns where default needs 42.
  • Net effect. Pass@1 gains from redirects generally offset or outweigh losses from stops: across both models, the ablation passes more tasks than default out of 1,662, with Pass@1 of 76.2 ± 0.9 vs. 74.7 ± 1.3 on Sonnet 5 and 72.6 ± 0.5 vs. 74.0 ± 0.6 on Opus 4.8 (Table 2a).

Matched tasks. All methods run the same 277 tasks with three seeds, so the differences in Table 1 are already on matched tasks: +1.3, −1.0, +0.9, +1.6, and +2.6 points across the five settings.

Web-Bench. We agree that cost on tasks both methods reached is not the cost of completing a project. The table below adds the quantities you asked for:

AgentMethodTasks reachedPass@2 (%)Cost per task, tasks both reachedTotal cost ÷ tasks reached
Claude CodeDefault73.869.30.3550.359
TE78.073.30.280 (−21.2%)0.291 (−18.8%)
CodexDefault68.264.00.2950.302
TE71.366.50.248 (−16.0%)0.266 (−11.7%)
Copilot CLIDefault65.760.80.2720.280
TE63.759.30.221 (−18.7%)0.230 (−17.9%)

Means over models and seeds, out of 102 tasks; costs in USD. Both methods reached 70.2 (Claude Code), 63.8 (Codex), and 59.0 (Copilot CLI) tasks. The last column also counts tasks that only one method reached.

TE's cost per task is lower on both measures for all three agents. On Claude Code and Codex it also reaches and passes more tasks; on Copilot CLI slightly fewer. We plan to add this table to the paper.

Q4. Success labels for the turn budget.

Yes, in our experiments the label comes from the official grader, whose tests the agent cannot see. That said, we do not think this gives TE an unrealistic advantage:

  • The agent never receives the label. TE reads it only after a task has finished and uses it only for later tasks. While a task runs, nothing from its hidden tests reaches the agent or the judge.
  • Real use has the same kind of signal. A CI result, a merged or rejected change, or the user's acceptance is also not visible while the agent works and arrives afterwards. The budget estimation needs no more, since it uses only earlier tasks.
  • A weaker signal would make TE more cautious. The label only decides which earlier runs enter the budget. If it came from the tests the agent runs itself, some failed runs would be counted as successes. Failed runs are longer on average (Section 2.2), so the budget would mostly grow and TE would step in later.
  • The budget has a limited role. Exceeding it only triggers a check by the judge, and the last-resort stop needs 2.5× the budget while the agent still repeats the same failure. Across margins from 1.00 to 1.50, savings stay at 24.3–26.4% and Pass@1 at 75.0–76.7 (G2).

Q5. Development runs and evaluation tasks.

Please see G2. In short:

  • Overlap. Yes, some development tasks also appear in the evaluation streams, because only a limited number of tasks in one repository are close enough in time.
  • What was used. All values were chosen in development runs on the six SWE-Bench repositories and frozen before the main runs, which produced all reported results. Web-Bench and the security benchmarks were never used to choose any value.
  • Why this does not drive the results. Development only set generic thresholds; each evaluation stream starts with no skills, tool history, or turn budget, and results are not very sensitive to the values.

We hope our responses have effectively addressed your concerns, and we are open for further discussions.

Reviewer_gMJsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes TokenEconomizer, a new method that reduces the token cost of black-box coding agents using cross-task experience. Specifically, TokenEconomizer reduces costs by inserting micro-skills, pruning tools, and intervening in failure loops without changing the LLM backend. Micro-skills are summarized from historical traces, tools are re-audited by LLMs based on recent tool-call histories, and failure loops are actively detected and addressed using LLM judges. Evaluations across diverse agent harnesses and LLM backends show that TokenEconomizer effectively reduces costs while maintaining or even improving performance.

Critical strengths
  • This paper explores an interesting and important direction.
  • Results clearly show that TokenEconomizer can effectively cut costs while maintaining/improving performance.
Critical weaknesses
  • It’s unclear whether TokenEconomizer’s dynamic tool pruning is necessary. It would be helpful to add a baseline where we manually prune each agent harness’s tool set in advance, use the resulting fixed tool set across all repositories, and evaluate its performance.
  • “Force stop” is used to interrupt failure loops, but this seems impractical for real deployments, since we would not want to stop the agent without providing any output to the user. A softer intervention may be more appropriate.
  • TokenEconomizer’s design involves many hyperparameter choices without a clear analysis of how different values affect performance. This may limit its generalizability, as finding suitable hyperparameters for different scenarios could be difficult.
Questions for authors
  • Will there be any false positives for the LLM judge used in the failure loop intervention?
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer gMJs

Thank you for your helpful comments. We address each concern below; some new results are also presented in our global response (G1–G3).

W1. Is dynamic tool pruning necessary?

We added a new baseline (G1): one tool list per repository, chosen before the first task and kept for every task. To avoid hindsight from TE's runs, we did not pick the list by hand. It is chosen by the same kind of LLM judge TE uses, from the tool descriptions and the repository's README and file listing. One list per repository is a stronger version of your suggested baseline (which is shared across repos), since each list is tailored to its repository.

MethodCC Pass@1CC cost vs. defaultCodex Pass@1Codex cost vs. default
Default75.0–74.2–
Fixed tool list74.7−7.5%73.3−9.2%
TE, tool pruning only74.4−14.2%73.6−13.8%
TE (full)76.4−26.6%75.0−24.2%

CC = Claude Code + Sonnet 5; Codex = Codex + GPT-5.6-terra; sklearn + sympy, three seeds (std in G1).

At similar Pass@1, TE's pruning saves clearly more than the fixed list (14.2% vs. 7.5% on Claude Code, 13.8% vs. 9.2% on Codex). A list chosen in advance can only guess from descriptions which tools the agent will call, while TE sees which tools the agent actually leaves unused and judges again for each task. TE also restores the full tool list every few tasks, so an incorrectly removed tool comes back.

W2. Forced stop vs. a softer intervention.

We agree that a softer step should come first, and TE actually works this way:

  • Redirect first. When the judge finds the agent stuck, TE first redirects it: it summarizes the agent's recent actions and asks for a different approach, keeping all edits. A run is stopped only if the agent is judged stuck again afterwards, or as a last resort when it exceeds 2.5× its learned turn budget while still repeating the same failure.
  • Stops are rare. In the guard-only ablation (Sonnet 5 and Opus 4.8, 1,662 runs), 166 runs (10%) were redirected and only 56 (3.4%) were stopped.
  • A stop keeps the agent's work. The edits made so far are kept and graded like any other run.

The stop exists because a benchmark run has no user to hand control to. In interactive use, it can simply be replaced by a notice to the user, who decides whether to stop.

W3. Hyperparameters and generalizability.

Please see G2 for details. In short:

  • No tuning per scenario. As we emphasized in our submitted paper, one configuration, fixed before the main runs, is used for all agents, models, and repositories. Web-Bench and the security benchmarks reuse it with only the changes their task structure requires. TE reduces cost in every setting.
  • Thresholds adapt by themselves. For example, the turn budget is not a fixed number but 1.25× the 90th percentile of turn counts in that repository's earlier successful tasks.
  • Low sensitivity. Changing four key values one at a time keeps savings between 23.2% and 26.4% and Pass@1 between 74.4 and 77.2.

Q1. False positives of the LLM judge.

Yes, some occur, and we measured them. In the ablation run where only failure loop intervention is switched on, the judge intervened 188 times (166 redirects, 22 stops). A panel of three stronger models (Claude Fable 5.2, GPT 6 Astra, Gemini 4 Argon; majority vote) judged 26 of these (14%) to be false positives. Compared with default execution of the same tasks:

  • 12 still passed (e.g., the redirect only interrupted a repeated verification step);
  • 6 failed under default execution too;
  • 8 turned a pass into a failure (0.5% of all runs).

False positives are limited by design: the judge is consulted only after a rule-based check fires, it must be at least 0.7 confident, and a first stuck verdict only redirects, so a single wrong verdict never stops a run. Redirects can also turn a failing run into a passing one. For example, with the guard alone, Pass@1 on Sonnet 5 rises from 74.7 to 76.2 (Table 2a).

We hope these responses have effectively addressed your concerns, and we are open for further discussions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Author Response (2/2)

G2. How hyperparameters were chosen, and how sensitive TE is to them (gMJs W3, yLW4 Q5, KPcC Q2)

One fixed configuration. All values were determined in development runs (our pilot runs) on the six SWE-Bench repositories and frozen before the main runs. The same values (Appendix C) are used for all three agents, all five agent–model settings, and every repository; nothing is tuned per repository or stream. Only two changes follow from how other benchmarks are built: Web-Bench streams (20–21 tasks) are too short to learn a turn budget from past tasks, so the intervention starts from a fixed 30-turn budget; and every security task is a different repository, so there is no later task for a micro-skill to help, and micro-skills are off.

Overlap with evaluation tasks. Some development tasks also appear in the evaluation streams. Our setting needs many tasks from one repository that are close in time (Appendix D), and even these large repositories have a limited number of such tasks, so a separate development set would have shortened the evaluation streams. As is common for ML hyperparameters, we chose the values in development runs and then froze them for all experiments. Four facts suggest the values are not overfitting to specific tasks:

  1. Nothing learned carries over. Each evaluation stream starts with no skills, no tool-usage history, and no turn budget; all are learned only from earlier tasks in that stream. Development only set generic thresholds, such as how many confirmations a skill needs.
  2. Unseen benchmarks. Web-Bench and the two security benchmarks were never used in hyperparameter selection, yet TE cuts cost there by 6.4–21.2%.
  3. Every setting. With the same values, TE saves cost in all 30 repository–setting pairs on SWE-Bench (Figure 4a).
  4. Stable results. Values fit to specific tasks would work well only at the chosen setting. Instead, results change little around it (see below).

Sensitivity. On sklearn (60 tasks, Claude Code + Sonnet 5, three seeds), we selected several key hyperparameters and perturbed one value at a time. Cells show value: Pass@1 / cost vs. default (%); bold = value from our main experiments.

HyperparameterLowerUsedHigher
Confirmations needed to activate a skill2: 75.6 / −26.13: 76.7 / −25.84: 76.7 / −25.4
Minimum need rate to inject a skill0.2: 74.4 / −23.20.4: 76.7 / −25.80.6: 77.2 / −24.9
Tasks a tool must go unused before pruning2: 75.6 / −26.03: 76.7 / −25.84: 76.7 / −25.9
Turn-budget margin β (loop intervention)1.00: 76.1 / −24.31.25: 76.7 / −25.81.50: 75.0 / −26.4

Overall, results are not very sensitive to these values: savings stay between 23.2% and 26.4%, and Pass@1 between 74.4 and 77.2 (25.8% and 76.7 at the chosen values). The largest drop comes from a need rate of 0.2, which lets in rarely needed skills (cf. G1).

G3. Confirmations from failed commands (yLW4 W2/Q2, KPcC Q3)

This is the edge case we already reported in Appendix I.1(c).

Cause. The proposer gives each skill two patterns: one that confirms the skill held in a task, and one that contradicts it. The proposer only sees summaries of earlier tasks, which list commands but not their outputs. So the confirmation pattern often just matches the command, while the contradiction pattern must guess what a failure looks like. If that guess misses, a failed command counts as a confirmation.

How often. In the Claude Code streams of our main runs (Sonnet 5 and Opus 4.8), 340 of 7,956 confirmations (4.3%) came from a failed command, and in 148 cases (1.9%) the skill was not retired afterward. We also checked all kinds of confirmations that do not truly support a skill: we randomly sampled 3 skills from each of these 36 streams (6 repositories × 2 models × 3 seeds) and 4 confirmations from each skill, giving 432 confirmations from 108 skills. In a blind audit by two stronger LLM judges (Claude Opus 5.5 and GPT-5.6-sol), both judges found a missed contradiction in only 16 (3.7%).

Impact. Most cases come from one quirk in one repository: sympy's test environment lacks pytest, which the default agent itself tries 0.7–1.2 times per task. When a wrong pytest skill stayed active, TE added only 0.1–0.3 failed commands per task, still cut cost on every sympy stream (−20.5% to −38.3%), and left sympy Pass@1 unchanged within noise.

Next-step improvement. A straightforward next step, which we leave as future work, is to make the detectors stricter, with the method unchanged: (i) count a confirmation only if its command succeeded; (ii) count any failure of a command named in the skill (missing tool, bad arguments, no tests collected) as a contradiction; (iii) show the proposer error messages from earlier tasks, so it can write better patterns.

AuthorsOpenReview ↗

Global Author Response (1/2)

G1. New baselines (gMJs W1, yLW4 W1/Q1, KPcC W1/Q1)

We thank the reviewers for their constructive feedback. We originally left out the agents' built-in memory because it does not run in our containerized benchmark setup by default: Claude Code's memory-writing step is off by default, and Codex only writes memory for interactive sessions that have been idle for 6 hours. Both can be enabled with small changes, so we now include them, along with the fixed-tool and experience-manual baselines the reviewers asked for. Due to budget limits, we run all of them on two SWE-Bench repositories (sklearn and sympy, 60 tasks each) with Claude Code + Sonnet 5 and Codex + GPT-5.6-terra, three seeds each. Every extra model call a baseline makes counts toward its cost, as TE's own calls do.

Baselines.

  1. Built-in memory. The agent's own memory, kept across tasks in each repository. For Claude Code, we turn on its end-of-task memory-writing step and keep everything else at defaults. For Codex, a small model extracts notes after each session, and a consolidation agent periodically merges them into a memory summary that later sessions load. Codex only starts this after a session has been idle for 6 hours, which never happens when tasks run back to back. We therefore trigger both steps ourselves, using Codex's own prompts and models, on a simulated clock where each task takes one hour. Following Codex's schedule (6 idle hours before extraction, at most one merge per 6 hours), the memory is updated once every 6 tasks.
  2. Fixed tools. One tool list per repository, chosen once before the first task and kept for every task. It is chosen the same way TE's per-task tool judge decides which tools are relevant: the task model reads each tool's description, a description of the job (fixing issues in this repository without a human), and the repository's README and file listing, then removes the tools it judges irrelevant (on Codex, whole capability packages such as sub-agents and skills). The difference is that TE makes this call for every task, using that task's issue text and usage evidence from past sessions, while the fixed list is decided once in advance, as an operator could do.
  3. Fixed tools + built-in memory. Tests whether the agent's own memory can replace TE's micro-skills once tools are pruned.
  4. Fixed tools + ungated manual. The control yLW4 asked for. TE's own proposer writes a manual from the same session digests, with the same prompt, schedule, and length cap (TE's average manual size), but without TE's checks: every line is injected right away, with no confirmation, no need gate, and no retirement when contradicted.
MethodCC Pass@1CC Cost vs. Default (%)Codex Pass@1Codex Cost vs. Default (%)
Default75.0 ± 0.8–74.2 ± 0.8–
(1) Built-in memory74.2 ± 1.4+8.7 ± 2.474.4 ± 1.3+10.3 ± 1.6
(2) Fixed tools74.7 ± 1.0−7.5 ± 1.173.3 ± 0.8−9.2 ± 0.7
(3) Fixed tools + built-in memory74.2 ± 1.7−2.2 ± 1.973.6 ± 1.3+1.5 ± 2.0
(4) Fixed tools + ungated manual74.4 ± 1.0−5.8 ± 1.273.3 ± 1.4−3.2 ± 1.3
TE, tool pruning only74.4 ± 0.5−14.2 ± 1.573.6 ± 1.0−13.8 ± 1.1
TE (full)76.4 ± 1.3−26.6 ± 1.675.0 ± 1.4−24.2 ± 1.2

CC = Claude Code + Sonnet 5; Codex = Codex + GPT-5.6-terra. Mean ± std over three seeds.

Takeaways.

  • Built-in memory raises cost (+8.7% / +10.3%) with no gain in Pass@1. Writing memory takes extra model calls, the notes are re-sent on every turn of later sessions, and nothing checks whether a note is correct or ever needed again.
  • Learned tool pruning beats a fixed list (gMJs W1): TE's pruning alone saves 14.2% / 13.8%, the fixed list 7.5% / 9.2%. Judging from descriptions alone, a fixed list can only guess which tools the agent will call; TE also sees which tools the agent actually leaves unused, and judges again for each task's issue.
  • Unchecked notes cost tokens. Adding memory or the ungated manual to fixed tools shrinks their savings. The ungated manual has TE's length budget, so its extra cost comes from content: wrong or rarely needed lines lead to wasted actions and are never removed. TE injects a skill only once it is confirmed and needed, and retires it when contradicted, so wrong skills that slip through cost little (G3). Since these checks are the only difference, they are what make learned experience pay off.
  • Full TE saves 24–27%, versus at most 9% for any baseline, with Pass@1 at or slightly above default.
AuthorsOpenReview ↗

Summary of Changes in the Revised Paper

Following the suggestions from the reviewers, we have uploaded a revised version of the paper, which adds the new results and clarifications from our responses. All results from the original submission are unchanged, and the main text is now 10 pages. The main changes are:

  • Additional baselines (Section 4.4). Built-in memory, fixed tools, and an ungated manual, compared with TE's tool pruning alone and full TE.
  • Hyperparameters (Appendix C). More details on how the values were chosen, and a sensitivity analysis.
  • Reliability of micro-skill confirmations (Appendix J). How often a failed command counts as a confirmation, and its measured impact.
  • Analysis of failure loop interventions (Appendix K). How often runs are redirected or stopped, and how many interventions are mistaken.
  • Web-Bench accounting (Appendix D). Number of tasks reached, Pass@2, and cost per task for default execution and TE.
  • Clarifications. Clearer descriptions of the default execution setup and the need rate, and consistent terminology throughout.

Our responses to each reviewer have already been posted below, and we look forward to the reviewers' feedback and further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.