- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MIA, a serving method that uses model advice to manage KV caches during tool execution. It addresses uncertainty about when an agent will resume after a tool call. The model supplies semantic hints about tool duration. MIA combines these hints with runtime history. It calibrates uncertainty and updates the remaining-time estimate as waiting continues. The scheduler uses these estimates for cache decisions. The evaluation uses trace replay across different hardware configurations. Validation replays two agent trace collections. MIA improves time to first token across the primary comparisons.
Critical strengths
- MIA calibrates duration predictions and conditions remaining time on elapsed waiting. Calibration improves job completion in the ablation. Removing calibration worsens job completion time. The scheduler does not treat raw hints as exact deadlines. The ablation supports the practical value of calibration in the tested setting.
- Auxiliary inference uses the same serving resources. Its cost is included, including abandoned requests. The evaluation reports transfer volume, preemptions and latency. This makes the net benefit more credible. The operational measurements help explain how the policy changes resource use.
Critical weaknesses
The no-hint variant uses a tool-history prior. The reported comparisons do not include a matched lightweight predictor using tool arguments. The measured completion-time contribution of hints is modest in this setting. The results do not fully separate semantic model advice from cheaper prediction and scheduling improvements. That distinction matters because advice consumes additional inference resources.
Questions for authors
How does MIA compare with a lightweight predictor using the same names, arguments and runtime history? How are prior and calibrator fitting data separated from evaluation?
2 comments
Response to Reviewer 2fmv (2/2)
(Continued from Part 1.)
2. Incremental semantic benefits and end-to-end cost
The measured completion-time contribution of hints is modest in this setting. The results do not fully separate semantic model advice from cheaper prediction and scheduling improvements. That distinction matters because advice consumes additional inference resources.
We explain the incremental benefits of MIA through end-to-end evaluation, comparison with other methods, and the technical design.
The end-to-end latency experiments already include the actual resources consumed by Hint generation and demonstrate a beneficial trade-off. To avoid unnecessary work, MIA acquires a Hint only for tool invocations that remain unfinished after three seconds; the many quickly completed invocations do not trigger additional generation. Hint acquisition overlaps asynchronously with tool execution and accepts an average of 23.7 output tokens per request. Auxiliary requests share inference resources with business requests, and this usage is included in the measured TTFT and JCT of the complete run. MIA retains a performance advantage after paying this generation cost, demonstrating the trade-off between semantic acquisition and hierarchical KV-cache scheduling benefits.
Comparisons with other methods also demonstrate the additional benefit of LLM Hints over inexpensive prediction. In the main experiments, the improvement over Continuum, which predicts TTLs from tool-level duration statistics, demonstrates the overall effectiveness of MIA. The lightweight-predictor comparison further shows that model semantics improve prediction quality and serving performance even when both approaches share the same scheduling backend.
MIA jointly designs return-time prediction and cache scheduling for agentic sessions. LLM Hints provide semantic information for predicting return-time distributions. The backend uses those distributions to coordinate cross-session offloading and prefetching, accounting for HBM pressure, restoration costs, and competing sessions to produce the final decisions. These components form an integrated method, and the ablations in the paper show performance contributions from both LLM Hints and the other modules.
3. Initialization of the prior and calibrator
How are prior and calibrator fitting data separated from evaluation?
Offline initialization and evaluation use different datasets. Terminal-Bench experiments initialize their prior from six Trace Commons sessions, while Trace Commons experiments initialize from six Terminal-Bench sessions. This setup evaluates cross-dataset generalization and avoids fitting the initial prior on evaluation traces. The lightweight-predictor experiments also train on the Common sessions described above to maintain comparability.
Runtime distribution updates follow causal time order: an invocation's observed duration enters subsequent statistics and calibration only after the tool has completed and the system has received its result. This matches the information available during live execution.
Response to Reviewer 2fmv (1/2)
Thank you for the questions on lightweight predictors, semantic benefits and inference cost, and initialization of the prior and calibrator. We compare predictors under the same backend, explain the contributions of semantic prediction and scheduling and their cost-benefit trade-off, and clarify initialization data and causal online updates. We address each point below.
1. Lightweight prediction with the same inputs and runtime backend
To our knowledge, prior work has not directly trained a lightweight predictor for this hierarchical KV-cache scheduling task. We believe that training a small model to predict tool execution-time distributions is challenging, and that limited semantic extraction can constrain its generalization.
To test the benefit of LLM Hints over lightweight prediction, we implement predictors that directly consume the current tool context, aliging with our LLM-Hint input. TF-IDF features are reduced to 64 dimensions by SVD and passed to an MLP with two equally sized hidden layers, producing a return distribution over 20 time bins. We evaluate hidden widths of 64, 128, 256, and 512. The models are trained and validated on Trace Commons using observed tool durations as supervision, then tested on Terminal-Bench. We replace LLM-based prediction with these predictors while retaining the same runtime backend, tool statistics, online calibration, and cache scheduling mechanisms. Each MLP configuration is trained with three seeds (23, 101, and 202); reported end-to-end results are averaged over the corresponding runs.
MIA achieves mean TTFT and JCT of 7.681 and 992.225 seconds. The four lightweight predictors achieve mean JCT between 1000.381 and 1010.522 seconds. MIA obtains the lowest mean TTFT and JCT in this comparison. Under the same scheduling backend, the semantic information supplied by LLM Hints therefore translates into observable end-to-end benefits, including the cost of generating those Hints.
Table 1. End-to-end comparison of MIA and lightweight predictors. Terminal-Bench, A100, Qwen3.5-35B-A3B, JPS 0.030.
We also measure the MAE of MIA and MLP predictions of the remaining wait after a given elapsed time. Here, both MIA and MLP denote the final output of the complete calibration pipeline. At elapsed waits of 5, 10, 30, and 60 seconds, MIA has lower MAE than all four lightweight predictors. These are stages at which the system must continue deciding when to retain or restore a waiting session's cache, demonstrating the contribution of semantics to the timing distributions used by the scheduler.
Table 2. MAE of the conditional-median remaining-time prediction, in seconds; lower is better.
(Continued in Part 2.)