acceptodds
Under review as a conference paper at ICLR 2027

TokenCast: Forecasting Token Consumption During LLM Agent Execution

12 views

Abstract

When a large language model (LLM) agent executes the same task, token consumption can vary by over an order of magnitude across runs. The agent chooses its next steps based on tool feedback and intermediate results, while the growing context steadily inflates the input size of every subsequent call. The total consumption of a task is therefore hard to predict before execution and the prediction must be revised as the run unfolds. In this paper, we propose TokenCast, which learns a composable cost representation for each execution segment, recording its own consumption and the context growth it introduces. Composing adjacent segments yields a cumulative estimate that captures the extra input cost incurred when context from earlier segments is re-read by every later call. As execution unfolds, newly observed evidence refreshes the forecast, requiring no additional LLM calls and incurring a mean cumulative prediction time of 32.8 ms per run on SWE-bench Verified. Across 4 task suites and 6 agent models, TokenCast's mean absolute error reduction against the strongest comparator averages 14.5% over 96 evaluated combinations. In offline budget-control replay, TokenCast uses 21.3% fewer tokens on average than a fixed-budget policy at matched trace completion. The code is available at https://anonymous.4open.science/r/TokenCast-1F8B.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, token consumption prediction, online forecasting, budget management

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ax9GOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TokenCast predicts token consumption before and during LLM agent execution. The method decomposes execution into call count, context growth, and residual cost, then uses LightGBM to forecast remaining consumption. Across four benchmarks and six models, the authors report an average 14.5% reduction in mean absolute error against the strongest comparator in each setting. An offline budget-control experiment also reports 21.3% fewer tokens than a fixed-budget policy at matched trace completion.

Critical strengths
  • The method explicitly accounts for the repeated consumption of accumulated context across agent calls. This is a sensible modeling choice: an early action can affect both its immediate token cost and the input cost of subsequent calls. The decomposition gives the predictor a clear connection to how agent execution consumes tokens.

  • The evaluation covers several benchmarks, models, and execution frameworks. The component ablations help explain which parts of the method contribute to the reported forecasting gains. The additional transfer and reasoning-configuration experiments are also valuable in examining robustness beyond the primary evaluation settings.

  • The predictor requires no additional LLM calls. The reported mean cumulative prediction time of 32.8 ms per SWE-bench Verified run supports the feasibility of updating forecasts throughout execution.

Critical weaknesses

W1. The connection between forecasting accuracy and practical resource efficiency remains insufficiently evaluated.

The paper demonstrates improvements in token-consumption prediction, but the evidence that these improvements translate into better resource-management decisions remains limited. The offline budget-control experiment is useful, yet it does not evaluate an online policy that responds to forecasts during actual execution.

I recognize that implementing a GPU scheduler is outside the primary scope of this prediction paper. Nevertheless, evaluating one concrete downstream policy, whether for token-budget management or resource allocation, would help establish the practical significance of the forecasting improvements. Token-count reductions alone should not be interpreted as evidence of improved GPU utilization or memory efficiency.

W2. The reported token savings do not establish that task success is preserved.

The 21.3% saving comes from replaying recorded executions. The authors clearly distinguish trace completion from task resolution, which is helpful. Nevertheless, matching trace completion does not establish that the policies solve the same number of SWE-bench tasks: reaching a recorded terminal state can include an unsuccessful outcome.

This distinction matters because early stopping may change which tasks succeed, even when the overall trace-completion rate remains unchanged. The replay also does not establish the additional cost of retrying stopped tasks. Reporting task resolution alongside token consumption would therefore be necessary to support a claim of cost reduction at preserved task success. A live budget-control experiment would provide stronger evidence about the effects of intervention during execution.

W3. The benefit over simple adaptive budget control remains unclear.

An agent execution framework already records cumulative token consumption and recent call costs. These observations could support inexpensive stopping rules, for example by estimating the cost of another call from recent consumption and context growth. The current evaluation does not establish how much TokenCast improves budget decisions over such rules.

The forecasting baselines and component ablations are useful, but they do not replace this decision-level comparison. All stopping policies should be tuned using the same validation procedure and compared at matched task success. The analysis should also identify unnecessary early stops: an estimated remaining cost above the available budget does not imply that the actual run would exceed that budget.

W4. The evidence for advance budget planning is mixed.

At Task Start, TokenCast's average MAE reduction against the strongest comparator is −2.2%, although it outperforms the comparators in several individual configurations. This does not undermine the substantial improvements reported during execution, but it weakens the case for relying on TokenCast before a task begins.

The paper would benefit from separating the value of initial forecasts from that of online updates. For budget control, it is also important to establish when the improvements become actionable and how much of the total token budget has already been consumed by that point.

Questions for authors
  1. Can the forecasts improve a concrete resource-allocation policy, with benefits measured in execution time, throughput, or memory use? If the intended application is limited to token-budget management, please clarify that scope.

  2. In the budget-control replay, what are the task-resolution rates under TokenCast and the fixed-budget policy? How many stopped runs would have succeeded within the fixed budget?

  3. How does TokenCast compare with simple adaptive stopping rules based on recent call costs and context growth, under the same tuning procedure and at matched task success?

  4. At which execution checkpoints do the forecasts begin to improve budget decisions, and how many tokens have already been consumed at those checkpoints?

4 comments
AuthorsOpenReview ↗

Follow-up on our response to your review

Dear Reviewer Ax9G,

Thank you again for your careful review and constructive suggestions. We have posted our responses, including online resource-scheduling experiments, evaluations of task resolution and consumption with early stopping and retries, comparisons with the Simple Adaptive baseline, and experiments on initial forecasts and decision timing. The results are included in Sec. 4.2 and 4.3, with detailed online evaluations in Appendices D.6 and D.7 of the revised manuscript.

When you have a chance, could you please let us know whether our responses adequately address your concerns? We would be happy to clarify any remaining points.

Best regards,

The Authors

Thank you for the additional experiments and for answering my questions. The scheduling results clarified how TokenCast’s forecasts affect task completion time. The early stopping experiments also addressed my question about how token savings affect task resolution. After reviewing these results, I have raised my score from 1 to 2.

AuthorsOpenReview ↗

Thank you for your response

Thank you for reviewing our additional experiments and updating your assessment. We are glad that the new results helped address your questions about scheduling efficiency and task resolution.

We would also appreciate your guidance on any remaining concerns or aspects that you feel need further improvement. In your view, what further improvements would help our work meet the standards you would expect of a strong paper? We would be happy to address your suggestions during the remaining discussion period.

AuthorsOpenReview ↗

Thank you for your response

Dear Reviewer Ax9G,

Thank you again for taking the time to review the additional experiments and update your assessment! We appreciate your comments on how TokenCast models the cost of accumulated context, its forecasting accuracy across benchmarks and models, and its ability to update forecasts without extra LLM calls.

In the original paper, we identified runtime decisions and resource scheduling as directions for future work. We’ve continued to work on those ideas, and your suggestions helped us take them further in this revision. We tested the forecasts in a live scheduler and examined how token savings affect task resolution. These experiments strengthened the paper by giving us concrete evidence of how TokenCast can help in practice. Your feedback has also given us more confidence to keep working in this direction!

Best regards,

The Authors

Reviewer_Z8rJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TokenCast predicts token consumption before and during agent execution. It represents a segment by its call count, context-length change, and cost relative to its starting context. An exact composition rule accounts for the repeated input cost that earlier context growth creates in later calls. The predictor combines direct and compositional estimates using LightGBM models, cross-fitting, and a correction model. Evaluation covers 11,712 traces from 240 tasks, four benchmarks, and six agent models. The strongest results concern forecasts updated during execution. An offline stopping experiment also reports lower token use at matched trace completion.

Critical strengths
  • The paper targets a useful quantity that response-length prediction alone does not capture. Future agent calls are not known in advance, and their costs depend on the context accumulated by earlier actions. Equation 1 makes this dependence explicit through the product of context growth and subsequent call count. Appendix B.3 correctly distinguishes an exact accounting identity from errors introduced by prediction.

  • The dynamic prediction results are consistent across the reported benchmark and model combinations. Section 4.2.1 reports average MAE reductions of 30.4% at In-call Update and 27.8% at Task Update against the best comparator in each setting. It also reports the weaker results at Task Start and Call Start. This breakdown is more informative than the overall 14.5% average alone.

  • The data split avoids an important source of leakage. All executions of the same task remain in one partition, including repeated runs. The suffix model is trained on predicted boundaries produced out of fold, which addresses a plausible mismatch between training and inference. The appendix also includes direct regression and component controls, rather than relying only on comparisons with unrelated estimators.

Critical weaknesses
  • The method description assigns the compositional path to task-level forecasting and describes separate call-level predictors. Yet removing composition changes call-level normalized MAE from 0.68 to 0.74 in Table 18. Appendix D.6 explicitly acknowledges that this intervention changes more than the task-level readout. Consequently, that table cannot establish how much of the benefit comes from the cross-term in Equation 1. Table 17 is a useful cleaner comparison, but it covers one benchmark and model. The same-feature direct baseline should be reported more broadly, with a precise explanation of which learned quantities call-level predictors actually use.

  • Table 3 gives 82.0% Task Start coverage for a nominal 90% interval. Zero-shot transfer to an unseen domain is worse than Self-Prediction, and the unseen-model Task Update experiment changes the target to remaining output tokens. These are informative results, but they leave the main remaining-total-consumption target under model shift insufficiently tested. The small number of distinct tasks in three benchmarks, 32 each, also makes uncertainty over tasks important. Repeated trajectories and three split seeds do not substitute for reporting that uncertainty.

Questions for authors

Plz see the weakness section.

Reviewer_knSDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies token-consumption forecasting for sequential LLM agents, where uncertain execution paths and repeated processing of retained context jointly determine the total cost. TokenCast represents an execution segment by its call count, net input-length change, and cost residual, with an exact composition rule for adjacent segments. The task-level predictor combines direct and compositional LightGBM forecasts using cross-fitting and residual correction; separate quantile models provide calibrated intervals. Evaluation covers 11,712 traces from 240 tasks, four benchmarks, six agent models, and two harnesses. The reported improvements are concentrated at In-call Update and Task Update. An offline stopping-policy replay additionally reports 21.3% average token savings across seven budgets while matching the fixed-budget trace-completion rate.

Critical strengths

The paper addresses a useful problem that differs meaningfully from single-response length prediction. The representation explicitly accounts for how context growth in one segment increases the input cost of later calls. The composition identity is transparent, and the resulting predictor requires no additional LLM calls, making the approach potentially useful for API-based agents.

The online forecasting results are promising across several task families and agent models. The reported mean MAE reductions against the strongest comparator in each benchmark–model configuration—30.4% at In-call Update and 27.8% at Task Update—are more compelling than the aggregate improvement alone. Keeping all runs of a task in the same data partition is also appropriate. Table 17 provides useful evidence that direct and compositional forecasts are complementary, while task-level prediction overhead is small in the measured SWE-bench Verified setting.

Critical weaknesses

Table 17 supports the compositional path, but does not establish the contribution of the composition rule separately from its representation, auxiliary supervision, and model capacity. As the authors acknowledge, removing composition in Table 18 also changes normalized call-level MAE from 0.68 to 0.74, affecting more than the task-level readout. A controlled comparison would better isolate the benefit of the composition rule.

Section 3.3 and Appendix B.2 describe staged fitting, but leave key details unclear: the prefix–suffix partition, predicted ending-boundary variables, and next-request input estimation at Task Update. How internal cross-fitting and the generation of out-of-fold pipeline inputs for correction training are nested within the outer training splits also needs clarification. These gaps hinder reconstruction from the manuscript and interpretation of the ablations.

Questions for authors

Could the authors provide a controlled ablation that isolates the contribution of the composition rule while keeping the input features, auxiliary supervision, and model capacity fixed? What changes in the no-composition ablation explain its effect on call-level prediction in Table 18?

Could the authors provide end-to-end pseudocode specifying the prefix–suffix partition, predicted ending-boundary variables, and next-request input estimation? How are cross-fitting and the outer data splits coordinated to generate the correction model's out-of-fold training inputs?

1 comment
AuthorsOpenReview ↗

Follow-up on our response to your review

Dear Reviewer knSD,

Thank you again for your careful review and constructive suggestions. We have posted our responses, including controlled ablations, an explanation of the call-level error change, and clarified training procedures with complete pseudocode. We also provide comparisons with the Simple Adaptive baseline, online decision experiments, and transfer results. The core ablations and implementation details are included in Sec. 3.3 and 4.4 and Appendices B.2 and D.5 of the revised manuscript.

When you have a chance, could you please let us know whether our responses adequately address your concerns? We would be happy to clarify any remaining points.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (2/2)

(4) With retries, TokenCast resolves more tasks using fewer tokens

We evaluated task resolution and total consumption with retries allowed. With up to two retries, TokenCast resolves 61.1% of tasks compared with 57.6% for Fixed Budget and uses 15.8% fewer tokens, including retry and forecasting overhead. We use the same validation procedure to select the strongest combination of execution signals as Simple Adaptive. Averaged across four retry limits, TokenCast resolves 53.1% of tasks compared with 49.0% for Simple Adaptive, uses 14.3% fewer tokens, and reduces the average number of unnecessary early stops from 16 to 8. (Secs. 4.2 and 4.3 and Appendices C.3 and D.6)

(5) Better Task Start forecasts and clearer timing for budget decisions

Task Start forecasts improved when we increased the training set from 96 to 192 tasks per fold. Adding tasks of the same type or different types reduces normalized MAE from 0.80 to 0.68 and 0.64, respectively. The strongest baselines in these settings achieve 0.75 and 0.73. By the end of the third call, runs have used only 12.8% of the budget on average, and a single stopping decision at that checkpoint saves 16.3% of consumption. (Sec. 4.2 and Appendices D.2 and D.4)

We understand that you have limited time to review the paper and the discussion. We hope this summary helps explain the revisions and the work we have done to address the reviewers' comments. Regardless of the final decision, we sincerely thank you for your time and for reviewing our work during your busy schedule.

Best regards,

The Authors

AuthorsOpenReview ↗

General Response (1/2)

Dear Area Chair,

Thank you for taking the time to review our paper. During the discussion, we conducted additional experiments in response to the reviewers' comments and included the results and clarifications in the revised manuscript. All three reviewers have now responded. We would like to summarize their feedback and the main revisions for your consideration.

The final ratings are as follows:

ReviewerFinal rating
knSD3
Z8rJ3
Ax9G2

Both reviewers who initially recommended rejection raised their ratings after reviewing our responses and additional experiments. Reviewer knSD raised their rating from 2 to 3, stating that their concerns were mostly addressed. Reviewer Z8rJ thanked us for addressing the concerns and maintained a rating of 3. Reviewer Ax9G acknowledged that our responses addressed their questions and raised their rating from 1 to 2, without raising new questions or identifying unresolved concerns.

The three reviewers acknowledged TokenCast's motivation, method design, novel aspects, and experimental results. They recognized the practical value of forecasting consumption over an agent's entire execution. Modeling context accumulation and repeated processing costs connects the forecasts to how agents actually execute tasks. They also acknowledged the results across tasks and models, along with the low prediction overhead and the absence of additional LLM calls.

In their Weaknesses, the reviewers did not question the work's novelty, research focus, or the soundness of its core approach. Their requests mainly concerned additional ablations and downstream experiments, broader evaluation, and clearer details. The main issues were:

a) A controlled ablation of the composition rule was needed with features, auxiliary supervision, and model capacity held fixed. The change in call-level error in the original ablation also needed an explanation. (knSD, Z8rJ)

b) The coverage evaluation included few independent tasks, and further evaluation across models and domains was needed. (Z8rJ)

c) Live downstream experiments were needed to test whether the forecasts improve resource scheduling and execution efficiency. (Ax9G)

d) Early stopping required analysis of task resolution, unnecessary stops, and retry costs, along with comparisons against simple adaptive stopping rules. (Ax9G)

e) Task Start results were relatively weak, and it was unclear when the forecasts became useful for budget decisions. (Ax9G)

We added experiments addressing these comments and updated Sections 4.2, 4.3, and 4.4 and the corresponding appendices. The main revisions and results are summarized below.

(1) Composition reduces forecasting error

We added ablations on two datasets and two models, keeping features, auxiliary supervision, and model capacity fixed. Full composition gives the lowest Task Update error in all four settings. Its mean normalized MAE is 0.63, compared with 0.73 for direct prediction and 0.77 without the context accumulation term. A separate control confirms that the original change in call-level error came from prefix-recall features removed by the same ablation. Restoring them reduces the error from 0.74 to 0.68. (Sec. 4.4 and Appendix D.5)

(2) TokenCast outperforms Self-Prediction on unseen models and domains

We expanded each coverage dataset to 128 tasks. With equal task weights for calibration, mean Task Start coverage reaches 89.3%, close to the nominal 90%. In zero-shot evaluation, TokenCast outperforms Self-Prediction on unseen models and in all six new domains, reducing mean MAE by 13.9% and 10.5%, respectively. Adding 10 target tasks reduces mean MAE across models from 24.5k to 18.5k. Adding five target tasks reduces mean MAE across domains from 109.8k to 86.7k. (Sec. 4.2 and Appendices D.1 and D.3)

(3) TokenCast reduces completion time and memory use in live scheduling

We used TokenCast to schedule GPU requests during live execution. Across the two models, it reduces mean completion time by about 28% and peak KV cache usage by about 11% compared with Simple Adaptive. The share of tasks resolved within the time limit rises from 68.8% to 77.1% on Qwen3.8-27B and from 45.8% to 51.0% on Llama-3.2-3B-Instruct. (Sec. 4.3 and Appendix D.7)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.