- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TokenCast predicts token consumption before and during LLM agent execution. The method decomposes execution into call count, context growth, and residual cost, then uses LightGBM to forecast remaining consumption. Across four benchmarks and six models, the authors report an average 14.5% reduction in mean absolute error against the strongest comparator in each setting. An offline budget-control experiment also reports 21.3% fewer tokens than a fixed-budget policy at matched trace completion.
Critical strengths
-
The method explicitly accounts for the repeated consumption of accumulated context across agent calls. This is a sensible modeling choice: an early action can affect both its immediate token cost and the input cost of subsequent calls. The decomposition gives the predictor a clear connection to how agent execution consumes tokens.
-
The evaluation covers several benchmarks, models, and execution frameworks. The component ablations help explain which parts of the method contribute to the reported forecasting gains. The additional transfer and reasoning-configuration experiments are also valuable in examining robustness beyond the primary evaluation settings.
-
The predictor requires no additional LLM calls. The reported mean cumulative prediction time of 32.8 ms per SWE-bench Verified run supports the feasibility of updating forecasts throughout execution.
Critical weaknesses
W1. The connection between forecasting accuracy and practical resource efficiency remains insufficiently evaluated.
The paper demonstrates improvements in token-consumption prediction, but the evidence that these improvements translate into better resource-management decisions remains limited. The offline budget-control experiment is useful, yet it does not evaluate an online policy that responds to forecasts during actual execution.
I recognize that implementing a GPU scheduler is outside the primary scope of this prediction paper. Nevertheless, evaluating one concrete downstream policy, whether for token-budget management or resource allocation, would help establish the practical significance of the forecasting improvements. Token-count reductions alone should not be interpreted as evidence of improved GPU utilization or memory efficiency.
W2. The reported token savings do not establish that task success is preserved.
The 21.3% saving comes from replaying recorded executions. The authors clearly distinguish trace completion from task resolution, which is helpful. Nevertheless, matching trace completion does not establish that the policies solve the same number of SWE-bench tasks: reaching a recorded terminal state can include an unsuccessful outcome.
This distinction matters because early stopping may change which tasks succeed, even when the overall trace-completion rate remains unchanged. The replay also does not establish the additional cost of retrying stopped tasks. Reporting task resolution alongside token consumption would therefore be necessary to support a claim of cost reduction at preserved task success. A live budget-control experiment would provide stronger evidence about the effects of intervention during execution.
W3. The benefit over simple adaptive budget control remains unclear.
An agent execution framework already records cumulative token consumption and recent call costs. These observations could support inexpensive stopping rules, for example by estimating the cost of another call from recent consumption and context growth. The current evaluation does not establish how much TokenCast improves budget decisions over such rules.
The forecasting baselines and component ablations are useful, but they do not replace this decision-level comparison. All stopping policies should be tuned using the same validation procedure and compared at matched task success. The analysis should also identify unnecessary early stops: an estimated remaining cost above the available budget does not imply that the actual run would exceed that budget.
W4. The evidence for advance budget planning is mixed.
At Task Start, TokenCast's average MAE reduction against the strongest comparator is −2.2%, although it outperforms the comparators in several individual configurations. This does not undermine the substantial improvements reported during execution, but it weakens the case for relying on TokenCast before a task begins.
The paper would benefit from separating the value of initial forecasts from that of online updates. For budget control, it is also important to establish when the improvements become actionable and how much of the total token budget has already been consumed by that point.
Questions for authors
-
Can the forecasts improve a concrete resource-allocation policy, with benefits measured in execution time, throughput, or memory use? If the intended application is limited to token-budget management, please clarify that scope.
-
In the budget-control replay, what are the task-resolution rates under TokenCast and the fixed-budget policy? How many stopped runs would have succeeded within the fixed budget?
-
How does TokenCast compare with simple adaptive stopping rules based on recent call costs and context growth, under the same tuning procedure and at matched task success?
-
At which execution checkpoints do the forecasts begin to improve budget decisions, and how many tokens have already been consumed at those checkpoints?
4 comments
Thank you for the additional experiments and for answering my questions. The scheduling results clarified how TokenCast’s forecasts affect task completion time. The early stopping experiments also addressed my question about how token savings affect task resolution. After reviewing these results, I have raised my score from 1 to 2.
Thank you for your response
Thank you for reviewing our additional experiments and updating your assessment. We are glad that the new results helped address your questions about scheduling efficiency and task resolution.
We would also appreciate your guidance on any remaining concerns or aspects that you feel need further improvement. In your view, what further improvements would help our work meet the standards you would expect of a strong paper? We would be happy to address your suggestions during the remaining discussion period.
Thank you for your response
Dear Reviewer Ax9G,
Thank you again for taking the time to review the additional experiments and update your assessment! We appreciate your comments on how TokenCast models the cost of accumulated context, its forecasting accuracy across benchmarks and models, and its ability to update forecasts without extra LLM calls.
In the original paper, we identified runtime decisions and resource scheduling as directions for future work. We’ve continued to work on those ideas, and your suggestions helped us take them further in this revision. We tested the forecasts in a live scheduler and examined how token savings affect task resolution. These experiments strengthened the paper by giving us concrete evidence of how TokenCast can help in practice. Your feedback has also given us more confidence to keep working in this direction!
Best regards,
The Authors
Follow-up on our response to your review
Dear Reviewer Ax9G,
Thank you again for your careful review and constructive suggestions. We have posted our responses, including online resource-scheduling experiments, evaluations of task resolution and consumption with early stopping and retries, comparisons with the Simple Adaptive baseline, and experiments on initial forecasts and decision timing. The results are included in Sec. 4.2 and 4.3, with detailed online evaluations in Appendices D.6 and D.7 of the revised manuscript.
When you have a chance, could you please let us know whether our responses adequately address your concerns? We would be happy to clarify any remaining points.
Best regards,
The Authors