- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces Tokenometrica, a framework for attributing agent token consumption to new prompts, replayed context, tool schemas, observations, visible outputs, tool arguments, and reasoning. It derives quadratic token growth under full history replay with bounded positive history increments, extends this analysis to random termination, and applies an existing log mean Divisia identity to attribute empirical scaling exponents to individual components. The evaluation covers 26,437 trajectories across τ-bench, τ²-bench, and SWE-agent. It finds that replayed context and repeated schema exposure dominate token consumption in these systems. Further experiments examine conditional cost prediction, retrospective stopping policies, and architectural counterfactuals. Live τ-bench experiments show that behavioral changes can offset or reverse the savings predicted when decision sequences are held fixed.
Critical strengths
-
The investigated question that how to attribute agent token consumption to new prompts, replayed context, tool schemas is important.
-
The authors analyze 26,437 trajectories and validate their accounting against provider reported usage.
-
The framework separates replayed context, tool schemas, and other sources of token consumption.
-
The live experiments demonstrate how behavioral changes can offset predicted token savings.
Critical weaknesses
-
The presentation needs significant improvement. There are large space between Figure1 and line 89.
-
The experiments do not verify a common quality floor across trajectory depths. Section 4.2 compares truncated prefixes of long runs without measuring whether these prefixes satisfy the quality requirement in Definition 2.
-
The masking experiments do not establish an asymptotic order reduction. Algorithm 1 masks old observations but retains assistant outputs and user messages, whose repeated exposure can still produce quadratic token growth.
-
The forecasting method requires the final interaction count, which limits its practical usefulness. Section 4.3 uses the realized number of interactions as an input, so the evaluation does not establish whether the method can predict an unknown future bill.
Questions for authors
-
Which evaluated intervention satisfies the linear branch of Theorem 1?
-
How is the quality floor in Definition 2 verified for the prefix families in Figure 2? If it is not verified, should these results and the report cards be explicitly described as conditional accounting laws without an established common quality floor?
-
Is there an evaluated configuration that meets a prespecified quality floor while reducing token demand, with uncertainty reported?