acceptodds
Under review as a conference paper at ICLR 2027

Model as Inference Advisor: Leveraging LLM Semantics for Agentic KV Cache Scheduling

Abstract

Agentic workloads powered by Large Language Models (LLMs) are becoming increasingly critical; however, their long contexts and concurrent executions generate massive key-value (KV) caches, imposing severe GPU memory pressure. To alleviate this bottleneck, hierarchical KV cache offloading and scheduling are imperative. Existing scheduling policies typically rely on request histories or agent importance, without fully distinguishing task-specific return behavior within the same tool. This semantic ambiguity can cause scheduling mismatches that underutilize hierarchical storage resources and degrade response latency. To bridge this gap, we propose Model as Inference Advisor (MIA), a novel inference system that leverages the LLM itself to expose performance-relevant task semantics as structured hints, enabling highly efficient hierarchical cache management. Experiments on Terminal-Bench demonstrate that MIA reduces mean time to first token (TTFT) and mean job completion time (JCT) by up to 28.7% and 9.1%, respectively, relative to the adapted vLLM baseline.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Agentic LLM Serving, KV Cache Management, Hierarchical KV Cache

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2fmvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MIA, a serving method that uses model advice to manage KV caches during tool execution. It addresses uncertainty about when an agent will resume after a tool call. The model supplies semantic hints about tool duration. MIA combines these hints with runtime history. It calibrates uncertainty and updates the remaining-time estimate as waiting continues. The scheduler uses these estimates for cache decisions. The evaluation uses trace replay across different hardware configurations. Validation replays two agent trace collections. MIA improves time to first token across the primary comparisons.

Critical strengths
  1. MIA calibrates duration predictions and conditions remaining time on elapsed waiting. Calibration improves job completion in the ablation. Removing calibration worsens job completion time. The scheduler does not treat raw hints as exact deadlines. The ablation supports the practical value of calibration in the tested setting.
  2. Auxiliary inference uses the same serving resources. Its cost is included, including abandoned requests. The evaluation reports transfer volume, preemptions and latency. This makes the net benefit more credible. The operational measurements help explain how the policy changes resource use.
Critical weaknesses

The no-hint variant uses a tool-history prior. The reported comparisons do not include a matched lightweight predictor using tool arguments. The measured completion-time contribution of hints is modest in this setting. The results do not fully separate semantic model advice from cheaper prediction and scheduling improvements. That distinction matters because advice consumes additional inference resources.

Questions for authors

How does MIA compare with a lightweight predictor using the same names, arguments and runtime history? How are prior and calibrator fitting data separated from evaluation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2fmv (1/2)

Thank you for the questions on lightweight predictors, semantic benefits and inference cost, and initialization of the prior and calibrator. We compare predictors under the same backend, explain the contributions of semantic prediction and scheduling and their cost-benefit trade-off, and clarify initialization data and causal online updates. We address each point below.

1. Lightweight prediction with the same inputs and runtime backend

The no-hint variant uses a tool-history prior. The reported comparisons do not include a matched lightweight predictor using tool arguments.

To our knowledge, prior work has not directly trained a lightweight predictor for this hierarchical KV-cache scheduling task. We believe that training a small model to predict tool execution-time distributions is challenging, and that limited semantic extraction can constrain its generalization.

To test the benefit of LLM Hints over lightweight prediction, we implement predictors that directly consume the current tool context, aliging with our LLM-Hint input. TF-IDF features are reduced to 64 dimensions by SVD and passed to an MLP with two equally sized hidden layers, producing a return distribution over 20 time bins. We evaluate hidden widths of 64, 128, 256, and 512. The models are trained and validated on Trace Commons using observed tool durations as supervision, then tested on Terminal-Bench. We replace LLM-based prediction with these predictors while retaining the same runtime backend, tool statistics, online calibration, and cache scheduling mechanisms. Each MLP configuration is trained with three seeds (23, 101, and 202); reported end-to-end results are averaged over the corresponding runs.

MIA achieves mean TTFT and JCT of 7.681 and 992.225 seconds. The four lightweight predictors achieve mean JCT between 1000.381 and 1010.522 seconds. MIA obtains the lowest mean TTFT and JCT in this comparison. Under the same scheduling backend, the semantic information supplied by LLM Hints therefore translates into observable end-to-end benefits, including the cost of generating those Hints.

Table 1. End-to-end comparison of MIA and lightweight predictors. Terminal-Bench, A100, Qwen3.5-35B-A3B, JPS 0.030.

PredictorMLP parametersMean TTFT (s)Mean JCT (s)
vLLM—8.3161022.234
MIA—7.681992.225
SVD64 + MLP [64, 64]9,6207.9201010.522
SVD64 + MLP [128, 128]27,4127.8361004.543
SVD64 + MLP [256, 256]87,5727.8231001.685
SVD64 + MLP [512, 512]306,1967.7161000.381

We also measure the MAE of MIA and MLP predictions of the remaining wait after a given elapsed time. Here, both MIA and MLP denote the final output of the complete calibration pipeline. At elapsed waits of 5, 10, 30, and 60 seconds, MIA has lower MAE than all four lightweight predictors. These are stages at which the system must continue deciding when to retain or restore a waiting session's cache, demonstrating the contribution of semantics to the timing distributions used by the scheduler.

Table 2. MAE of the conditional-median remaining-time prediction, in seconds; lower is better.

Elapsed wait (s)MIAMLP-64MLP-128MLP-256MLP-512
529.51632.87831.90331.28631.073
1043.56250.11551.28447.71250.413
3052.16259.09559.57258.21860.638
6057.36480.20078.10772.49277.885

(Continued in Part 2.)

AuthorsOpenReview ↗

Response to Reviewer 2fmv (2/2)

(Continued from Part 1.)

2. Incremental semantic benefits and end-to-end cost

The measured completion-time contribution of hints is modest in this setting. The results do not fully separate semantic model advice from cheaper prediction and scheduling improvements. That distinction matters because advice consumes additional inference resources.

We explain the incremental benefits of MIA through end-to-end evaluation, comparison with other methods, and the technical design.

The end-to-end latency experiments already include the actual resources consumed by Hint generation and demonstrate a beneficial trade-off. To avoid unnecessary work, MIA acquires a Hint only for tool invocations that remain unfinished after three seconds; the many quickly completed invocations do not trigger additional generation. Hint acquisition overlaps asynchronously with tool execution and accepts an average of 23.7 output tokens per request. Auxiliary requests share inference resources with business requests, and this usage is included in the measured TTFT and JCT of the complete run. MIA retains a performance advantage after paying this generation cost, demonstrating the trade-off between semantic acquisition and hierarchical KV-cache scheduling benefits.

Comparisons with other methods also demonstrate the additional benefit of LLM Hints over inexpensive prediction. In the main experiments, the improvement over Continuum, which predicts TTLs from tool-level duration statistics, demonstrates the overall effectiveness of MIA. The lightweight-predictor comparison further shows that model semantics improve prediction quality and serving performance even when both approaches share the same scheduling backend.

MIA jointly designs return-time prediction and cache scheduling for agentic sessions. LLM Hints provide semantic information for predicting return-time distributions. The backend uses those distributions to coordinate cross-session offloading and prefetching, accounting for HBM pressure, restoration costs, and competing sessions to produce the final decisions. These components form an integrated method, and the ablations in the paper show performance contributions from both LLM Hints and the other modules.

3. Initialization of the prior and calibrator

How are prior and calibrator fitting data separated from evaluation?

Offline initialization and evaluation use different datasets. Terminal-Bench experiments initialize their prior from six Trace Commons sessions, while Trace Commons experiments initialize from six Terminal-Bench sessions. This setup evaluates cross-dataset generalization and avoids fitting the initial prior on evaluation traces. The lightweight-predictor experiments also train on the Common sessions described above to maintain comparability.

Runtime distribution updates follow causal time order: an invocation's observed duration enters subsequent statistics and calibration only after the tool has completed and the system has received its result. This matches the information available during live execution.

Reviewer_Mc3COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MIA, which uses model-generated descriptions and soft duration windows for tool calls to guide hierarchical KV-cache offloading and prefetching. A runtime combines these hints with tool-duration histories, calibrates residuals, conditions on elapsed waiting time, and makes resource-aware cache decisions. Evaluation uses forced-decoding replay of 40 Terminal-Bench traces and 28 longer Trace Commons traces on two model sizes and two GPU platforms. On Terminal40, the reported improvements over a Mooncake-backed vLLM baseline are 1.6–28.7% in mean TTFT and 1.7–9.1% in mean JCT, including auxiliary hint-generation work.

Critical strengths
  1. Useful separation of semantic advice and runtime control. Sections 3.2–3.4 give a plausible way to distinguish calls to the same tool without letting the model prescribe physical cache operations. Conditioning a total-duration forecast on elapsed waiting time is appropriate, and fallback behavior limits dependence on unavailable or invalid hints. This is a useful systems interface for agent serving.

  2. The evaluation charges the main additional cost and includes meaningful controls. Section 4.1 uses a hierarchical-cache baseline rather than an HBM-only straw man; Appendix A includes auxiliary requests in the shared service workload. Table 2 retains the backend when hints are removed, and Table 3 reports lower restoration volume and fewer preemptions. These results support an actual serving benefit in the evaluated replay setting.

Critical weaknesses
  1. The central scheduler and predictor are insufficiently specified for reproduction. Section 3.4 describes expected benefit, harm to competing sessions, feasible eviction choices, and admission checks without giving the objective, candidate-search procedure, or pseudocode. Sections 3.3 and Appendix A similarly leave the mixture-weight functions, shrinkage parameters, recency weighting, and historical-prior provenance unspecified. The stated horizon and benefit threshold do not determine the algorithm. This matters because much of the improvement can arise from the backend rather than semantic advice.

  2. The incremental evidence for model-generated semantics is limited. Table 2 shows that removing hints increases TTFT from 5.971 to 6.359 seconds and JCT from 910.568 to 921.260 seconds, at one operating point. There is no matched command-argument or lightweight non-LLM predictor to establish that an LLM is needed for this gain, nor direct forecast calibration/error results. Appendix B.1 repeats one fixed arrival schedule, and Appendix A does not identify a disjoint source for the fitted historical prior. Consequently, the semantic contribution and its robustness to unseen tasks or arrival patterns remain uncertain.

Questions for authors
  1. Please specify the cache-decision objective, candidate selection, transfer-cost model, and calibration/shrinkage functions, including all parameters needed to reproduce Section 3.4.

  2. What data initialize the historical prior, and are tasks, repositories, and observed durations disjoint from the evaluated traces? Please distinguish offline prior fitting from causal online updates.

  3. Does the hint advantage persist against a command-argument-aware non-LLM predictor and across independent arrival seeds? A matched comparison retaining the same scheduler, together with forecast accuracy/calibration, would materially affect my assessment.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Mc3C (1/2)

Thank you for the questions on reproducibility, lightweight predictors, and historical data sources. We provide predictor and scheduler equations, parameters, and pseudocode, compare lightweight predictors under the same backend, and clarify the data splits for prior initialization and causal online updates. We address each point below.

1. Key predictor and scheduler computations (W1, Q1)

The central scheduler and predictor are insufficiently specified for reproduction...

Please specify the cache-decision objective... Section 3.4.

The initial paper presented MIA's main mechanisms, equations, and parameters, but yet should organize more implementation details systematically. The revised Appendix A provides the parameters, candidate-comparison procedure, and pseudocode needed to reproduce the interface from return distributions to offloading and prefetching decisions.

The following table and pseudocode summarize the main parameters and algorithms. Further details are included in the revised appendix.

Table 1. Main prediction parameters in the frozen configuration.

ComponentSetting and interpretation
Tool-prior shrinkagew(n) = n/(n+16); more observations increase the tool-group weight
Residual-group shrinkageTool residuals: n/(n+16); finer semantic residuals: 0.5 n/(n+16)
Semantic mixturelambda = 0.75 n/(n+16), where n is the global residual count
Soft-window scaleMidpoint and half-width in log(1+t) space; minimum half-width: 0.25
Online-history capacityAt most 256 recent observations per group
Recent/initial-history mixtureWith initial history, recent-distribution weight: n_recent/(n_recent+32)
Recency weightingExponential decay in observation order; half-life: 64 records
Hint acquisitionAcquire after 3 s if the tool remains unfinished; retain the current prediction while pending

Pseudocode 1: Return-distribution estimation and online calibration.

# M(A,B,w)=(1-w)A+wB; w(n)=n/(n+16); ECDFs use the recency weights above.
# g/u/s: global/tool-combination/semantic-and-confidence group; n_x: its count.
Predict(call, hint=[lo,hi], a):                   # a: elapsed wait
    F = M(ECDF_g, ECDF_u, w(n_u))
    if valid Hint window and historical residuals exist:
        mu = [log(1+lo)+log(1+hi)]/2
        sigma = max(0.25, [log(1+hi)-log(1+lo)]/2)
        R = M(R_g, R_u, w(n_ru))                 # normalized residual CDFs
        R = M(R, R_s, 0.5*w(n_rs))
        S = Distribution(max(0, exp(mu+sigma*Z)-1)), Z ~ R
        F = M(F, S, 0.75*w(n_rg))
    F = 0.99*F + 0.01*Exp(mean=max(0.1, E_F[T]))
    return G(r|a) = [F(a+r)-F(a)]/[1-F(a)]       # r>=0, F(a)<1

OnToolResultReceived(call, T, hint):
    Append(T, histories={g,u})                   # completed, received results only
    if valid Hint window:
        Append((log(1+T)-mu_hint)/sigma_hint, residuals={g,u,s})
    UpdateRecentHistory(capacity=256, half_life=64, seed_mass=32)

Pseudocode 2: Offloading and prefetching.

# r~G_j: remaining return time; c: seconds/byte; b_j(V): missing bytes after V.
# Q/C: transfer/compute availability relative to now.
# tau/epsilon/beta: discount time scale/benefit threshold/risk coefficient.
L_j(V) = E_{r~G_j}[exp(-r/tau)*[max(r,Q)+c*b_j(V)-max(r,C)]_+]
Delta_j(V,V0) = L_j(V0)-L_j(V)                  # positive means lower delay
Score(Delta) = sum_j Delta_j - beta*sum_j[-Delta_j]_+ - epsilon
# [x]_+ = max(0,x)

Offload(k):
    Q0 = native reclaimable-block queue; V0 = Q0[:k]
    E = empty, uncached blocks in Q0
    for each candidate session j with reclaimable cached blocks:
        S_j = first max(0,k-|E|) reclaimable blocks of j in Q0 order
        V_j = (E + S_j + remaining Q0 blocks)[:k]
    if enabled, add feasible composite candidates preserving known prefixes
    recompute missing bytes b_j(V) for all affected sessions and candidates
    reject plans harming running sessions or changing unpriced sessions' caches
    V_star = argmax_{feasible V} Score({Delta_j(V,V0)})
    if Score({Delta_j(V_star,V0)}) > 0 and calibration-stability checks pass:
        use V_star
    else: use V0

Prefetch(i):
    s = c*max(current_missing_bytes, 3*page_bytes)
    t_check = now + max(0, G_i^(-1)(0.5)-s-delta) # delta: lead-time margin
    at t_check, recheck missing bytes and actual allocation; H=s+h
    D(r) = max(max(r,Q)+c*b_i, C)                # demand-restore readiness
    P(r) = max(r, Q_p+c*b_i, C)                 # prefetch readiness
    Delta_i = E_{r~G_i}[exp(-r/tau)*(D(r)-P(r))*1(r<=H)]
    compute Delta_j for affected j!=i using L_j above
    if Score(Delta)>0 and stability, foreground, capacity, transfer checks pass:
        execute prefetch

# h: holding time; Q_p: prefetch-transfer availability.
# Stability checks recompute the same candidate's Score under calibrators
# each omitting one initial fitting session; plans adding competing-session
# losses must retain positive benefit.

(Continued in Part 2.)

AuthorsOpenReview ↗

Response to Reviewer Mc3C (2/2)

(Continued from Part 1.)

2. Comparison with lightweight predictors using tool arguments (W2, Q3)

The incremental evidence for model-generated semantics is limited... There is no matched command-argument or lightweight non-LLM predictor to establish that an LLM is needed for this gain, nor direct forecast calibration/error results.

Does the hint advantage persist against a command-argument-aware non-LLM predictor and across independent arrival seeds? ...

To our knowledge, prior work has not directly trained a lightweight predictor for this hierarchical KV-cache scheduling task. We believe that training a small model to predict tool execution-time distributions is challenging, and that limited semantic extraction can constrain its generalization.

To test the benefit of LLM Hints over lightweight prediction, we implement predictors that directly consume the current tool context, aliging with our LLM-Hint input. TF-IDF features are reduced to 64 dimensions by SVD and passed to an MLP with two equally sized hidden layers, producing a return distribution over 20 time bins. We evaluate hidden widths of 64, 128, 256, and 512. The models are trained and validated on Trace Commons using observed tool durations as supervision, then tested on Terminal-Bench. We replace LLM-based prediction with these predictors while retaining the same runtime backend, tool statistics, online calibration, and cache scheduling mechanisms. Each MLP configuration is trained with three seeds (23, 101, and 202); reported end-to-end results are averaged over the corresponding runs.

MIA achieves mean TTFT and JCT of 7.681 and 992.225 seconds. The four lightweight predictors achieve mean JCT between 1000.381 and 1010.522 seconds. MIA obtains the lowest mean TTFT and JCT in this comparison. Under the same scheduling backend, the semantic information supplied by LLM Hints therefore translates into observable end-to-end benefits, including the cost of generating those Hints.

Table 2. End-to-end comparison of MIA and lightweight predictors. Terminal-Bench, A100, Qwen3.5-35B-A3B, JPS 0.030.

PredictorMLP parametersMean TTFT (s)Mean JCT (s)
vLLM—8.3161022.234
MIA—7.681992.225
SVD64 + MLP [64, 64]9,6207.9201010.522
SVD64 + MLP [128, 128]27,4127.8361004.543
SVD64 + MLP [256, 256]87,5727.8231001.685
SVD64 + MLP [512, 512]306,1967.7161000.381

We also measure the MAE of MIA and MLP predictions of the remaining wait after a given elapsed time. Here, both MIA and MLP denote the final output of the complete calibration pipeline. At elapsed waits of 5, 10, 30, and 60 seconds, MIA has lower MAE than all four lightweight predictors. These are stages at which the system must continue deciding when to retain or restore a waiting session's cache, demonstrating the contribution of semantics to the timing distributions used by the scheduler.

Table 3. MAE of the conditional-median remaining-time prediction, in seconds; lower is better.

Elapsed wait (s)MIAMLP-64MLP-128MLP-256MLP-512
529.51632.87831.90331.28631.073
1043.56250.11551.28447.71250.413
3052.16259.09559.57258.21860.638
6057.36480.20078.10772.49277.885

3. Sources of initialization, validation, and online-update data (W2, Q2)

... Appendix A does not identify a disjoint source for the fitted historical prior. Consequently, the semantic contribution and its robustness to unseen tasks or arrival patterns remain uncertain.

What data initialize the historical prior, and are tasks, repositories, and observed durations disjoint from the evaluated traces? Please distinguish offline prior fitting from causal online updates.

Offline initialization and evaluation use different datasets. Terminal-Bench experiments initialize their prior from six Trace Commons sessions, while Trace Commons experiments initialize from six Terminal-Bench sessions. This setup evaluates cross-dataset generalization and avoids fitting the initial prior on evaluation traces. The lightweight-predictor experiments also train on the Common sessions described above to maintain comparability.

Runtime distribution updates follow causal time order: an invocation's observed duration enters subsequent statistics and calibration only after the tool has completed and the system has received its result. This matches the information available during live execution.

Reviewer_A7DPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In agentic serving, a session’s KV cache sits idle while the agent waits for a tool call, and hierarchical caching must decide when to offload it from HBM and when to prefetch it back. Existing schedulers rely on runtime signals such as tool names, which cannot distinguish, for example, a quick status check from a full rebuild issued through the same shell tool. MIA has the model generate a structured semantic hint for each pending tool call, which the runtime calibrates against observed behavior to estimate return time and guide offloading and prefetching. On replayed Terminal-Bench traces, MIA reduces mean TTFT by up to 28.7% and mean JCT by up to 9.1% compared with vLLM using hierarchical caching.

Critical strengths
  • Natural and well motivated idea. The same tool can hide very different return behaviors, and the model issuing the call already knows which case it is. Keeping the model as an advisor while the runtime retains cache control, with fallback to native policies, is a sound design.
  • Honest end-to-end accounting. Auxiliary Hint inference shares the serving engine and is included in all measurements. Mechanism-level counters show that scheduling behavior actually changes.
Critical weaknesses
  • The value of semantic hints for return time prediction is assumed rather than shown. MIA pools calibration residuals by operation/scope/mode, assuming that calls with the same semantic labels have similar durations. However, calls with the same label (e.g., a full rebuild) can vary widely depending on repository size or cache state, and the paper reports no direct comparison of prediction accuracy with and without hints.
  • Cross-session scheduling under contention is underspecified and narrowly evaluated. Even with accurate per-session predictions, deciding which sessions to offload or restore under limited HBM is a cross-session trade-off. The paper describes this policy only qualitatively.
Questions for authors
  • What is the return-time prediction accuracy (e.g., median absolute error or calibration) of the statistical prior alone versus prior plus hints, broken down by operation type? How much does duration vary within a single semantic group?
  • How does MIA compare with an oracle scheduler that knows true return times? This would show how much of the achievable improvement MIA captures.
4 comments
AuthorsOpenReview ↗

Response to Reviewer A7DP (1/2)

Thank you for the specific questions about the value of semantic information and cross-session cache scheduling. We provide additional analyses of semantic labels and observed durations, a same-history ablation of Hint-based prediction, cross-session scheduling ablations, and a matched-cost Oracle using true return times. We address each point below.

1. How semantic labels distinguish return times within the same tool (W1)

The value of semantic hints for return time prediction is assumed rather than shown ... assuming that calls with the same semantic labels have similar durations. However, calls with the same label (e.g., a full rebuild) can vary widely depending on repository size or cache state...

We share the reviewer's observation, and therefore designed MIA's semantic description as hierarchical labels supplemented by a soft time window, enabling finer distinctions. This design reflects our observation that the same task can have different execution modes and timing characteristics. Beyond operation (which distinguishes reading, testing, building, etc.), scope distinguishes one target, a subset, and the whole workload, helping describe differences such as partial and full builds; completion mode distinguishes synchronous completion from a background launch. The soft window provides a further invocation-specific time scale. For example, the arguments of a repeated full build may include a previous execution time. Thus, even when both invocations use Bash, the runtime can distinguish a local-target build from a full build and calibrate the relationship between these semantics and observed durations.

MIA takes semantic information expressed in the current context, distinguishes the intended work, and produces calibrated timing estimates. For example, the mean duration of build/whole/sync is approximately 28.2 times that of build/one/sync. Grouping only by Bash or build therefore mixes different time scales, whereas scope further distinguishes the target extent. The P90 values show shorter overall observed durations for build/one/sync and a longer tail for build/whole/sync; this information is retained in the return distributions, contributing to offloading decisions.

Table 1. Semantic groups and observed durations for Bash builds, in seconds. Terminal-Bench, A100, Qwen3.5-35B-A3B, JPS 0.030.

Semantic groupMean duration (s)P90
build/one/sync6.3186.942
build/whole/sync178.412440.788

2. How Hints change return-time prediction accuracy (W1, Q1)

... the paper reports no direct comparison of prediction accuracy with and without hints.

What is the return-time prediction accuracy (e.g., median absolute error or calibration) of the statistical prior alone versus prior plus hints, broken down by operation type? How much does duration vary within a single semantic group?

Compared with the tool-only implementation, Hints reduce return-time prediction error. We conduct an online prediction ablation that removes only the current invocation's Hint and its associated window, comparing tool-only with tool + Hint. The prediction target is the conditional median of the remaining wait, evaluated by MAE in seconds. At elapsed waits of 5, 10, 30, and 60 seconds, adding the current Hint reduces MAE by 8.75%, 3.91%, 5.75%, and 23.94%, respectively. These results demonstrate the contribution of Hints, including a larger improvement at the 60-second waiting point, when offloading decisions are particularly relevant.

Table 2. Online prediction ablation.

Elapsed wait (s)Tool-only MAETool + Hint MAEMAE reduction
533.69130.7438.75%
1045.17443.4053.91%
3055.34152.1625.75%
6075.41757.36523.94%

Not every tool invocation triggers MIA scheduling in the online experiment. To obtain broader coverage, we also evaluate offline semantic predictions for all tool invocations in Terminal-Bench. Hints reduce aggregate MAE both across all tools and within Bash invocations.

Table 3. Aggregate offline prediction results. MAE is in seconds.

ScopeTool-only MAEPrior + Hint MAEMAE reduction
All tool batches4.7734.4267.26%
Single Bash invocations7.1066.5547.76%

(Continued in Part 2.)

AuthorsOpenReview ↗

Response to Reviewer A7DP (2/2)

(Continued from Part 1.)

We further analyze Bash invocations by coarse task category and complete operation/scope/completion-mode labels. Labels with fewer than five invocations are omitted. MAE decreases for most of the displayed groups.

It should be noted that the offline experiment directly comparing time-prediction MAE cannot capture MIA's full performance advantage because online calibration is disabled. In addition, tool invocations that finish within three seconds do not trigger Hint acquisition, so many short-running requests are filtered out of Hint acquisition. Therefore, offline prediction error cannot fully capture the benefits of online scheduling.

Table 4. Offline prediction error by coarse Bash category. MAE is in seconds.

LabelTool-only MAEPrior + Hint MAE
read2.6850.789
search2.5782.441
write2.5810.449
test7.7387.583
build24.38724.821
network10.31211.030
other2.8521.822

Table 5. Offline prediction error by complete Bash semantic group. MAE is in seconds.

LabelTool-only MAEPrior + Hint MAE
read/one/sync2.7350.619
read/subset/sync2.6120.856
read/whole/sync2.8521.673
search/subset/sync2.6561.367
search/whole/sync2.4773.820
write/one/sync2.5810.449
test/one/sync2.6962.692
test/subset/sync7.9097.748
build/one/sync7.4008.842
build/whole/sync177.272168.631
network/one/sync11.14811.758
network/whole/sync9.84010.861
other/one/sync2.2611.228
other/whole/sync5.9754.962

3. How multiple sessions jointly determine offloading and prefetching (W2)

Cross-session scheduling under contention is underspecified and narrowly evaluated... The paper describes this policy only qualitatively.

MIA coordinates multiple sessions (instead of making decisions from individual session's prediction). It includes two explicit cross-session mechanisms: offloading priority based on cross-session benefit (E), and cross-session cache protection during prefetching (P). The scheduler operates over sessions sharing a finite HBM budget. Under memory pressure, it compares the native reclaim order with alternative eviction orders, checks their effects on reusable KV prefixes across affected sessions, and evaluates the aggregate benefit using return distributions and restoration costs. Prefetching similarly weighs the delay saved for the target session against the restoration costs imposed on other sessions by the resulting HBM occupancy. Existing occupancy, reclaimable capacity, and in-flight transfers constrain both decisions. Thus, the current session's return estimate is evaluated jointly with competing sessions and live cache state.

We ablate E and P. Removing E or P increases TTFT by 3.54% or 1.63%, respectively; removing both increases TTFT by 4.82%. These ablations directly demonstrate the contribution of cross-session coordination beyond return-time prediction.

Table 6. Cross-session scheduling ablations. TTFT and JCT are means in seconds.

MethodMean TTFT (s)Mean JCT (s)ΔTTFTΔJCT
MIA7.681992.2250.00%0.00%
MIA − E7.9531001.095+3.54%+0.89%
MIA − P7.806997.695+1.63%+0.55%
MIA − E − P8.0511001.339+4.82%+0.92%

4. Comparison with an Oracle using true return times (Q2)

How does MIA compare with an oracle scheduler that knows true return times? This would show how much of the achievable improvement MIA captures.

MIA with LLM Hints approaches the performance of the matched-cost Oracle. This demonstrates the effectiveness of the complete Hint-based prediction pipeline, including the calibration of timing errors.

We construct a matched-cost Oracle that retains MIA's execution logic and charges its inference work, while replacing the final return distribution used for scheduling with the true return time. This evaluates the value of more accurate timing information while retaining the cost of acquiring advice.

MIA achieves mean TTFT/JCT of 7.681/992.225 seconds, compared with 7.560/982.051 seconds for the matched-cost Oracle. Replacing MIA's forecast with true return times further reduces TTFT and JCT by 1.58% and 1.03%. This provides a true-return-time reference for the same MIA scheduling backend and shows that the Hint-based pipeline already performs well.

Table 7. MIA and the matched-cost Oracle in the same setting. TTFT and JCT are means in seconds.

MethodMean TTFT (s)Mean JCT (s)
vLLM8.3161022.234
MIA7.681992.225
Matched-cost Oracle7.560982.051

Thank you for the detailed response and the effort for the additional experiments. Table 1 shows that semantic groups with different mean durations, however, my concern about variance within a group still remains. Table 5 shows that hints do not improve prediction for the build/one/sync and network/whole/sync group. I appreciate authors' effort but I maintain my current rating.

Thank you for your follow-up and consideration. Our additional results illustrate how multi-level semantic groups refine single-level operation categories, distinguishing duration patterns relevant to offloading decisions—particularly whether a tool call leaves sufficient time to release and restore its cache. And as noted in our response, Table 5 evaluates offline prediction with online calibration disabled and thus does not fully capture MIA’s online scheduling benefits; even so, hints reduce prediction error in most groups.

We appreciate your feedback and would be happy to address any further questions.

Comments on the paper

1 comment

We thank the reviewers for their feedback.

We have posted detailed responses to each reviewer, including:

  1. Semantic labels and variation in return times within the same tool — Reviewer A7DP.
  2. Return-time prediction accuracy with and without Hints — Reviewer A7DP.
  3. Cross-session offloading and prefetching, including policy ablations — Reviewer A7DP.
  4. Comparison with a matched-cost Oracle using true return times — Reviewer A7DP.
  5. Predictor and scheduler specifications, hyperparameters, and reproducibility — Reviewer Mc3C.
  6. Comparison with lightweight predictors using tool arguments under the same scheduling backend — Reviewers Mc3C and 2fmv.
  7. Prior and calibrator initialization, separation of fitting and evaluation data, and causal online updates — Reviewers Mc3C and 2fmv.
  8. Incremental semantic benefits, the contributions of prediction and scheduling, and the inference-cost trade-off — Reviewer 2fmv.

We have also uploaded a revised PDF with the following updates:

  • Additional reproducibility details and experiments. Appendix A now provides further equations, hyperparameters, and pseudocode for the predictor and scheduler. Appendix B presents additional experimental results and analyses motivated by the reviewers’ questions.
  • More precise experimental results. We fixed a bug in our Continuum reproduction that had degraded its performance on RTX4090 and updated its results in the main experimental figure. This correction does not affect our analysis and conclusions in initial manuscript.

We hope these responses and updates further clarify our research and address the reviewers’ concerns :)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.