acceptodds
Withdrawn from ICLR 2027

Tokenometrica: Trace-Grounded Accounting and Scaling Laws for Agentic Token Demand

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM agents are metered and billed in tokens, yet a token total says neither where the tokens went nor how they will grow as tasks get longer. We introduce Tokenometrica, a framework that treats every agent run as an observable metered trace and partitions each model call, through an exact accounting identity, into new prompt, replayed context, completion, reasoning, tool-schema, tool-argument, and tool-observation tokens. Token-demand functions over families of traces are stated with ordinary in named scale variables under an explicit quality floor. We prove a replay law: under full-history replay, a -round loop costs tokens, with a closed-form depth beyond which replay dominates. Under random termination, expected token demand is finite if and only if , and the bill inherits half the tail index of the interaction count. We also give an exact log-mean Divisia identity that attributes an empirical scaling exponent to mechanisms. We apply Tokenometrica to 26,437 public trajectories from -bench, -bench, and SWE-agent, covering nine LLMs including a reasoning model; the reconstructed meters match provider-reported usage within 0.6% median error. Replayed context and repeatedly exposed tool schemas account for 89–96% of tokens in every system. Model-generated tokens are under 3% for non-reasoning agents, and reasoning adds only 5.3% for o4-mini because it is never replayed. On SWE-agent the token elasticity in rounds rises from 1.06 to 1.69 as runs lengthen, and replay explains 93% of it. Replay-aware forecasts cut the error of interaction-count extrapolation by about half. Live -bench runs confirm the architectural saving that counterfactual accounting predicts, but expose a behavioral cost: with less in context an agent takes more calls, which replay makes expensive, so masking keeps half its predicted saving and every tool-loading variant raised the bill. A 15-round circuit breaker makes each resolved SWE-agent issue cheaper.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, token accounting, scaling laws, context replay, tool use, inference efficiency, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1ZjEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Tokenometrica, a framework for attributing agent token consumption to new prompts, replayed context, tool schemas, observations, visible outputs, tool arguments, and reasoning. It derives quadratic token growth under full history replay with bounded positive history increments, extends this analysis to random termination, and applies an existing log mean Divisia identity to attribute empirical scaling exponents to individual components. The evaluation covers 26,437 trajectories across τ-bench, τ²-bench, and SWE-agent. It finds that replayed context and repeated schema exposure dominate token consumption in these systems. Further experiments examine conditional cost prediction, retrospective stopping policies, and architectural counterfactuals. Live τ-bench experiments show that behavioral changes can offset or reverse the savings predicted when decision sequences are held fixed.

Critical strengths
  1. The investigated question that how to attribute agent token consumption to new prompts, replayed context, tool schemas is important.

  2. The authors analyze 26,437 trajectories and validate their accounting against provider reported usage.

  3. The framework separates replayed context, tool schemas, and other sources of token consumption.

  4. The live experiments demonstrate how behavioral changes can offset predicted token savings.

Critical weaknesses
  1. The presentation needs significant improvement. There are large space between Figure1 and line 89.

  2. The experiments do not verify a common quality floor across trajectory depths. Section 4.2 compares truncated prefixes of long runs without measuring whether these prefixes satisfy the quality requirement in Definition 2.

  3. The masking experiments do not establish an asymptotic order reduction. Algorithm 1 masks old observations but retains assistant outputs and user messages, whose repeated exposure can still produce quadratic token growth.

  4. The forecasting method requires the final interaction count, which limits its practical usefulness. Section 4.3 uses the realized number of interactions as an input, so the evaluation does not establish whether the method can predict an unknown future bill.

Questions for authors
  1. Which evaluated intervention satisfies the linear branch of Theorem 1?

  2. How is the quality floor in Definition 2 verified for the prefix families in Figure 2? If it is not verified, should these results and the report cards be explicitly described as conditional accounting laws without an established common quality floor?

  3. Is there an evaluated configuration that meets a prespecified quality floor while reducing token demand, with uncertainty reported?

Reviewer_u9uoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposed a specification (framework) for recording and analyzing the output tokens of agentic LLMs. Authors categorize and calculate different types of tokens in public output traces. As an agent may call the underlying model iteratively to complete a task and tokens will thus accumulate, authors analyze the dynamics of accumulation and propose the scaling law that can help forecast the growth of tokens. The main finding is that the accumulated number of replayed tokens fed to the model grows quadratically and will start to dominate the total number of accumulated tokens over some points.

Critical strengths
  1. The specification for recording and analyzing output tokens of the agent seems an important direction and is probably underexplored.
  2. The authors analyzed 26,437 public traces, which makes the results of quantitative analysis robust.
Critical weaknesses
  1. This paper is hard to follow. Many condensed terms and notations are packed without adequate introduction.
  2. This work analyses the output traces and provides some insights. However, it is a bit hard for me to tell the actionability of these insights. For example, can you explain the high-level implication of "Under random termination, expected token demand is finite if and only if E[M^2] < ∞ "?
  3. The main finding that replayed tokens accumulate quadratically seems not a novel perspective. Quadratic accumulation is probably the reason why the community developed token caches. Furthermore, given existing techniques like token caches, I am not certain about the value of theoretical analysis in the main body especially if the delivered claims/findings do not involve these real-world factors.
Questions for authors
  1. The analyzed models are not the newest ones. Have you tried more recent models?
  2. The discussions on related literature are insufficient in my opinion. Are their any previous works that analyze the output traces? [1] also compared failed and successful traces and showed that the failed ones are significantly longer, which is similar to your finding in lines 268-269.

[1] Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories

Reviewer_2noKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposed a framework for analyzing the agent token consumption and its growth with interaction length. The framework splits the tokens for each llm call in an agent trajectory into components such as replayed context, tool schemas and the completion tokens. The paper combines theoretical analysis with experiments on 26437 trajectories and live tests for cost-saving.

Critical strengths
  1. The experiments are comprehensive and reproducible, covering 3 benchmarks and 9 llms. And the main findings generally hod across different settings, which are useful as reference measurements for the community.
  2. The live experiments part is interesting, providing useful evidence beyond fixed-trajectory estimates. The failure cases where the interventions fail to produce the saving are also useful and can provide insights.
Critical weaknesses
  1. The paper is poorly presented. A lot of definitions that the main claims depend on are only described in the appendix. Many numbers and formulas are packed into each paragraph, making the paper very hard to read and follow. I believe the presentation is below the ICLR acceptance standard.
  2. The theory is thin, and the experiments cannot effectively test the 'order vs. coefficient' claim, because at the observed run lengths the lower exponent under masking can come from a smaller coefficient alone.
Questions for authors

I'm curious on the authors' thoughts about how to distinguish an order change from a coefficient change at the observed run lengths? For example, could you fit a model with separate linear and quadratic terms for each architecture and report whether the quadratic coefficient under masking is significantly different from zero?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.