acceptodds
Under review as a conference paper at ICLR 2027

Risk-Controlled Speculative Prefetching for Tool-Augmented Dialogue

Abstract

Speculative decoding accelerates LLM inference by predicting future tokens. We apply the same principle to future user actions in tool-augmented dialogue: PRESTO (Predictive Speculative Tool Orchestration) predicts diverse hypotheses for the next user query, pre-executes read-only tools while the user reads the current response, and serves cached results on exact-key match when the actual query arrives, or discards with no degradation. We make three contributions. (1) The proxy reversal and its constructive converse. Offline embedding commit rate is a gameable proxy: predictors that maximize it degrade in-loop. The fix is coverage-WTA, adaptation supervised by the in-loop objective (labeling winners by realized cache hits); it beats the zero-shot predictor at equal planner budget, and no-training retrieval even when that retrieval is given a larger prefetch budget, in both -bench domains, reaching 70.1%/32.2% turn-level cache hits vs. 59.4%/23.7% zero-shot. (2) The entity-revelation boundary: an a-priori, per-domain ceiling on speculation (88.4% retail, 47.3% airline) that explains most remaining misses and scopes the magnitude, not the applicability, of adaptation's win. (3) A risk-controlled speculation gate: conformal risk control over a learned score operationalizing that boundary lifts fired precision to 82-87% (retail) and 50-59% (airline) over 69-70%/29-33% always-fire at the certified operating point; served-result correctness comes from exact-key matching, not the gate. A  600K-parameter MCL predictor (<1ms) enables the study: offline-stronger frontier predictors (82.6% vs. 54.4% commit) serve 0% in-loop at short prediction-time slack, where MCL serves 32-35%; with a 15s speculation window, MCL hides 32.6% of retail per-turn tool latency. Speculative prefetching must be evaluated, adapted, and gated in-loop.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative execution, tool prefetching, tool-augmented dialogue, LLM agents, multiple choice learning, conformal risk control, in-loop evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v8ZuOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PRESTO, a speculative prefetching framework for multi-turn tool-augmented dialogue. It predicts the user's next-turn tool needs and executes read-only calls in advance, so that a completed exact-key cache hit can eliminate the corresponding tool-execution wait. PRESTO combines a Multiple Choice Learning (MCL) predictor, a grounding pipeline using kNN retrieval and a planner LLM, and a gate based on Learn-then-Test (LTT) with Clopper-Pearson bounds intended to control the joint risk of firing and missing. The authors evaluate the method on their PRESTOBench and on external τ-bench retail and airline tasks. The main empirical finding is that optimizing an embedding-based commit proxy can reduce actual cache coverage. The proposed coverage-WTA adaptation instead uses realized cache hits to assign training targets. On two-fold τ\tau-bench replay with a 15-second speculation window, it improves turn-level hit rates over zero-shot MCL from 59.4% to 70.1% in retail and from 23.7% to 32.2% in airline. The paper also includes transfer to human-authored MultiWOZ dialogue, architecture ablations, and failure analysis.

Critical strengths
  1. A substantive study of objective mismatch. The paper goes beyond reporting prediction accuracy and measures realized exact-key cache hits. The comparison between proxy-supervised adaptation, zero-shot prediction, and coverage-WTA demonstrates a consequential failure mode and a constructive response.
  2. The entity-revelation analysis provides a useful diagnostic of speculative feasibility under the paper's grounding assumptions. Distinguishing argument availability from prediction quality helps explain why the achievable cache coverage differs across domains. Relative to the reported ceilings of 88.4% for retail and 47.3% for airline, coverage-WTA closes approximately 37% and 36% of the gap left by zero-shot MCL, respectively. This is a useful complement to prediction accuracy and could inform domain selection in subsequent work. Its value is as a diagnostic under the specified argument-availability assumptions, rather than an unconditional upper bound for all predictors.
Critical weaknesses
  1. Insufficient empirical differentiation from Speculative Actions. As noted, Speculative Actions is PRESTO's most directly related predecessor, reporting 55% next-action prediction accuracy and system-level latency reductions at ICLR 2026. PRESTO's differentiating narrative (“predicting user utterances, not tool chains or agent actions”) is conceptually clear, but conceptual difference is not empirical advantage. The paper never compares the two speculation targets under a unified τ\tau-bench replay and latency model. If a Speculative Actions-style predictor achieves comparable or higher hit rates within PRESTO's framework, then the contributions of MCL and coverage-WTA need repositioning; if MCL is indeed superior, the paper should show that comparison. Without it, the question “why MCL rather than a simpler fast-model predictor?” remains unanswered.
  2. The end-to-end serving benefit of the complete method is not yet established. Section 5.4 reports tool-latency savings under simulated delays and fixed speculation windows, including a 32.6% reduction for retail Arm B at W=15s. However, this does not establish the user-visible response-time improvement of coverage-WTA combined with the learned gate. The 9.6 ms “full pipeline” measurement in Appendix A.15 covers encoding, MCL, and the cosine gate, while grounding/planning and the remaining agent-serving stages are outside that measurement. Although A.17 and A.19 provide additional cost and timing analyses, I could not find an integrated evaluation reporting complete response latency with and without prefetching, including both hits and misses. This leaves the practical magnitude of the acceleration unclear.
Questions for authors
  1. Evaluation request: For the complete coverage-WTA plus learned-gate configuration, what are the mean and p50/p95 response latencies with and without prefetching, measured from arrival of the user's request to completion of the response? Please include all requests, including hits, misses, and turns on which the gate abstains, and report absolute times and relative reductions. Which stages finish before request arrival, and which remain on the response critical path? Please distinguish directly measured timings from simulated delays, and these results from the Arm B tool-latency savings in Section 5.4.
  2. Evaluation request: Tables 1 and 7 report offline results for the fine-tuned Qwen2.5-0.5B baseline. How does this baseline perform in the τ-bench replay under matched training data, candidate count, grounding, hardware, and prefetch budget, particularly at short speculation windows? Please report its in-loop hit-rate/latency tradeoff relative to MCL and specify the model variant and inference configuration. This comparison would help establish whether the practical advantage demonstrated against Sonnet also holds against a compact language-model predictor.
  3. Clarification request: Table 1 lists one API call for the diverse-forced Sonnet predictor, while Section 5.4 and Appendix A.19 describe five calls per turn in the monetary comparison. Which configuration is used in each result, and do the cost comparisons include the common grounding/planning stages for both MCL and Sonnet? If different configurations were used, please provide the cost corresponding to the predictor evaluated in-loop.
5 comments
AuthorsOpenReview ↗

Response to Reviewer v8Zu (1/2)

Thank you for a careful review. You asked for an empirical comparison with Speculative Actions (SA), a matched in-loop Qwen2.5-0.5B baseline, a resolution of the Sonnet one-call vs five-call inconsistency, and a complete-response latency evaluation. We ran the first two under one replay harness, answer the third from source, and ran the fourth as a pre-registered live serving experiment (part 2 of this reply). All are in Appendix C.7 with every null; details we omit here are there.

Common harness. A held-out set of conversation prefixes from the no-prefetch test conversations (328 trajectories, 1,267 prefetch opportunities; retail 257 eligible turns over 57 scenarios, airline 101 over 25). Every arm sees only the prefix and shares the Haiku planner (which turns a predicted utterance into tool calls), its grounding check, a budget of 5 read-only calls per opportunity and the replay timing model. Endpoint: coverage, the fraction of eligible turns on which the tool call the agent actually needed was already cached (exact key match) within the 15 s window; scenario-cluster bootstrap CIs, exact scenario-sign tests, Holm over six contrasts. C is coverage-WTA, P the matched proxy-supervised control, Q Qwen2.5-0.5B, SA the read-only SA adaptation.

(a) Speculative Actions. The SA arm follows the SA paper's procedure, restricted to read-only tools, on the same Haiku backbone: one predicted-user-query request, then up to five sequential action decisions, executing the first tool call of each response and feeding back its observation. Coverage at W=15 s: retail C 56/257 (21.8%) vs SA 16/257 (6.2%), +15.56 pp [+9.27, +22.41], Holm p 6.58e-5; airline C 30/101 (29.7%) vs SA 14/101 (13.9%), +15.84 pp [+4.76, +27.05], Holm p .26. An independent replay on 72 known scenarios agrees: +21.31 pp (Holm .0146) retail, +20.34 pp (Holm .0127) airline.

One confound, disclosed in Section 5.3 and C.7: the SA adaptation mostly does not issue calls (34 unique calls over 922 retail opportunities and 51 over 345 airline, against 1,147 and 383 for C), so its low coverage partly reflects abstention rather than worse prediction of the needed call, and these data cannot separate the two. What survives: a grounded set of predicted utterances yields more distinct cached calls ready in time than one predicted query plus sequential decisions, significantly in retail after adjustment and in both domains on known scenarios. We claim nothing about native SA or its serving stack.

(b) Qwen2.5-0.5B in-loop. Arm Q is Qwen2.5-0.5B-Instruct with LoRA, fine-tuned on the same training-side context/next-turn pairs MCL uses, generating K=5 sampled candidates that the shared planner grounds under the same budget. Training data, K, grounding, budget and timing model are matched; compute and architecture are not (configuration in C.7). Coverage at W=15 s: retail C 56 vs Q 44 of 257, +4.67 pp [+0.36, +9.65], Holm .26; airline C 30 vs Q 17 of 101, +12.87 pp [+3.90, +21.15], Holm .26. The known-scenario replay gives +11.48 and +11.86 pp, both Holm .125. MCL leads on every point estimate, but no Qwen contrast rejects after adjustment: MCL, at about 600K parameters, is not significantly better than the matched 0.5B model at these sample sizes, as C.7 states.

(c) Sonnet one call vs five. The two statements count different stages; we failed to say so. The Sonnet predictor (prompted for diverse candidates) issues one request producing five candidate utterances; the common Haiku planner is then called once per distinct nonempty candidate, up to five times, which MCL incurs identically. Table 1's "1 call" is the predictor request; Section 5.4 and C.6 (formerly A.19) count planner calls. Our saved replay logs record call counts, not token charges, so the submitted dollar figure cannot be tied to the evaluated configuration; we withdraw it. The Table 1 caption now counts predictor requests only, and C.6 gives per-stage calls.

AuthorsOpenReview ↗

Response to Reviewer v8Zu (2/2)

End-to-end latency with and without prefetching (new live experiment)

You asked for complete response latency with and without prefetching, all requests pooled, with the gate in the loop. We ran this as a live serving experiment during the discussion period, with contrasts and pass criterion written down before launch (design fixed on a disjoint 20-scenario development run).

Setup. 62 test-split tau-bench scenarios (47 retail, 15 airline), five replicate conversations per arm per scenario, five arms: N (no prefetching); C (the complete method as served: coverage-WTA predictor, learned speculation gate, planner, window-aware admission); planner-interface variants C_lookup and C_handoff; and P_handoff (proxy-supervised control; unregistered). Each scenario-replicate is a block: its five arms share the opening customer turn and run in a balanced rotating order. Agent gpt-oss-120b, planner Claude Haiku 4.5, simulated customer Claude Opus 5; 15 s speculation window, injected per-tool delays of 2-5 s (retail) and 3-10 s (airline). Response latency is wall-clock from the customer request's arrival to completion of the agent's response, over every request: hits, misses and turns on which the gate abstained. Predictor, gate, planner and agent calls are measured; tool delays are injected because sandbox tools return instantly. (The 9.6 ms figure you cite, now C.2, covers encoding, MCL and the offline gate only.) No parameter was fit on live outcomes. A conversation with an error or an incomplete request is retained and excluded from its contrast as an incomplete pair, never retried or relabeled.

Why the gate abstains on most turns. It maximizes retained hits subject to fired misses <= .10 over eligible turns and does not optimize net latency; in the original gated serving traces it abstained on 456 of 508 eligible opportunities. The complete method therefore rarely reuses the cache, bounding any end-to-end effect.

Pre-declared grading. Primary: C minus N on mean per-request response latency over complete scenario-replicate pairs, predicted negative. Secondary: C_handoff minus N. Cluster = scenario, scenarios weighted equally; two-sided sign-flip randomization p over the 62 cluster statistics, Holm over the two contrasts, 95% scenario-cluster bootstrap CI. Pass = predicted sign with Holm p < .05.

Result. 308 of 310 blocks complete; 2 failed at the shared opening turn before any arm ran.

contrastdifference (s)95% CIHolm ppairsarm vs N mean (s)
C minus N (primary)-0.30[-0.63, +0.07].0912886.14 vs 6.45
C_handoff minus N (secondary)-0.49[-0.92, -0.13].01352845.97 vs 6.46

Both have the predicted sign; the secondary passes the pre-declared criterion, the primary does not (Holm p = .091). Registered descriptives (no p): C minus N is -0.53 s [-0.80, -0.27] in retail and +0.43 s [-0.55, +1.60] in airline; it has the same sign in each of the three dispatch runs (-0.63, -0.68, -0.19 s); it is -0.18 s [-0.66, +0.39] on the 36 scenarios touched by earlier harness development and -0.47 s [-0.85, -0.08] on the other 26; the per-request median is unchanged (2.08 s in both arms) and the p95 moves from 25.1 s to 24.2 s. Beyond the two opening failures, the primary excludes 20 blocks (17 with an error, 3 with an incomplete request) and the secondary 24.

Where the reduction comes from. Entirely tool wait: non-tool time per request is unchanged (N 2.55 s, C 2.56 s), so the predictor, gate and planner add no measurable foreground cost (C_handoff's is 0.09 s higher). 9.1% of C requests were served at least one cached tool result, saving 5.9 s each; C's executed tool wait per request is 0.45 s lower than N's (3.21 vs 3.66 s). Removing all tool wait would save 3.66 s per request, so the live system realizes about 12% of the available reduction, against the 32.6% of eligible-turn tool wait hidden in the Section 5.4 replay; the gap is requests with no tool call, gate abstention and cache misses.

What this does and does not show. A direct measurement of the complete method's user-visible benefit against no prefetching, distinct from the Section 5.4 replay. The benefit is 0.3 to 0.5 s per request on a 6.4 s mean (5 to 8%), negative for every arm, significant after adjustment for the secondary contrast but not the primary, concentrated in retail and in requests that call tools, with no measurable overhead on requests that do not. Section 5.4, the abstract and C.7 state this result.

Scope statement. The same six-contrast family includes the matched proxy-supervised control; on scenarios disjoint from training, coverage-WTA does not outperform it (neither domain significant after adjustment; C.7), as the revised abstract, Contribution 1 and Conclusion state.

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

Reviewer_P3ivOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper uses a mechanism similar to LLM speculative decoding for multi turn tool augmented dialogue. It predicts potential tool calls for the next user turn, executes them in advance, and caches the results. The authors identify a mismatch between the offline performance of predictions based on embeddings and actual cache hit rates, which they call Proxy Reversal. To address this, they propose Coverage-WTA, which uses realized cache hits to guide training. They also introduce an entity revelation boundary to characterize the upper limit of speculative prefetching and a Conformal Risk Control gate to limit unproductive speculation. Experiments show that Coverage-WTA improves cache hit rates from 59.4% to 70.1% on retail tasks and from 23.7% to 32.2% on airline tasks in τ-bench. The conformal gate further improves prefetch precision, while simulated experiments show lower latency for tool calls.

Critical strengths
  1. The paper identifies the Proxy Reversal phenomenon, where better offline prediction performance can lead to worse in loop cache hit rates.

  2. The authors propose Coverage-WTA to address this issue by aligning the training objective with actual cache hit performance.

  3. Extensive experiments validate the effectiveness of the proposed method, demonstrating improvements over baselines across both retail and airline domains.

Critical weaknesses
  1. Potential Violation of the i.i.d. Assumption in the Conformal Risk Guarantee:

    Section 5.5 applies Clopper–Pearson (CP) bounds to pooled turn level outcomes. However, the standard CP guarantee assumes a binomial failure count, typically requiring i.i.d. Bernoulli observations. Since turns within the same task may be dependent, this assumption may not hold.

  2. Misalignment Between the Risk Definition and Actual Speculation Cost:

    Section 5.5 defines the risk as R = Pr(fire ∧ miss). However, this risk only accounts for turns where speculation is triggered but no cache hit occurs. It ignores wasted prefetches on turns with at least one cache hit and does not consider the varying costs of different tool calls. Therefore, controlling this risk does not necessarily guarantee a bound on wasted computation or positive net latency savings.

Questions for authors
  1. Does the conformal risk guarantee still hold when turn level observations are not i.i.d. as mentioned in Weakness 1? If the guarantee relies on task level independence, how does it establish a turn weighted rather than a task weighted risk guarantee?

  2. Why is the risk defined as Pr(fire ∧ miss) rather than a cost-sensitive measure of wasted computation? How does the proposed risk guarantee relate to the actual computational cost or net latency savings?

6 comments
AuthorsOpenReview ↗

Response to Reviewer P3iv (1/2)

We accept both points. Page numbers refer to the revised PDF.

1. The i.i.d. assumption behind the Clopper-Pearson certificate

Withdrawn. The submitted paper applied Clopper-Pearson to pooled turn outcomes with groups split by conversation but sharing scenarios. Turns within a task are dependent, so the interval is not a distribution-free bound. The revision no longer calls it a certificate anywhere: Section 5.5 "Calibration and the independent unit" (p9) treats the operating points as descriptive; the Table 3 caption (p10) reads "Descriptive ... operating points" and states pooled-turn CP does not provide a task-independent certificate; Contribution 3 (p3) describes the gate as a learned gate whose threshold is selected to target Pr[fire and miss] <= epsilon and states up front that the operating points are observed rather than guaranteed, with a formal bound requiring calibration at an independent unit; the abstract (p1, "measured operating points rather than a guarantee"), A.1 (p14), C.5 (formerly A.18, p28) and Limitations (p10) carry the same scoping. Section 5.5 (p10) also states that a fixed testing order does not repair within-task dependence. The title keeps "risk-controlled" as the gate's design target, the quantity its threshold is selected against; the paper no longer claims a certificate for it.

Task-level guarantee we can state. This is the statement we would put in place of the withdrawn claim. It is rebuttal-only: the revision claims no certificate, and we have not run it on new data.

Setting. Condition on all development artifacts (predictor, planner, features, candidate thresholds). The sampling unit is a whole cluster: a scenario with all its trials (claim about unseen scenarios) or a conversation (claim about fresh conversations from the fixed benchmark mixture). Clusters are i.i.d.; within-cluster dependence is arbitrary. For cluster i, N_i = eligible turns, M_i = fired-and-missed turns, 0 <= M_i <= N_i <= B. Our simulator caps every task at 30 actions and each opportunity consumes two, so B = 15 (read from the runtime code).

Target. R(t) = E[M_i] / E[N_i], the long-run fraction of eligible turns that fire and miss: turn-weighted, not the mean of per-task rates; clusters with N_i = 0 are kept.

Test. X_i = epsilon N_i - M_i. Under H_t: R(t) >= epsilon, E[X_i] <= 0 and -(1 - epsilon) B <= X_i <= epsilon B, so for lambda in [0, 1 / ((1 - epsilon) B)] the product E_n = prod_{i <= n} (1 + lambda X_i), or any mixture over development-fixed lambda, is a nonnegative supermartingale. Ville: P(exists n: E_n >= 1 / alpha) <= alpha. Crossing rejects R(t) >= epsilon time-uniformly; not crossing is inconclusive. No Bernoulli or independent-turn assumption. With epsilon = 0.10, B = 15: E_n = 0.9 prod(1 + X_i / 13.5) + 0.1 prod(1 + X_i / 27).

Threshold selection. Candidates are tested in a development-fixed order; we certify the contiguous prefix whose e-processes have crossed. If the first unsafe candidate j* is ever selected, its e-process must have crossed, which Ville bounds by alpha, so no per-grid Bonferroni. Four (domain, family) combinations at alpha = 0.0125 each sum to 0.05; each e-process crosses at 1/alpha = 80.

Guaranteed: under cluster i.i.d., with probability >= 1 - alpha no selected threshold has turn-weighted fire-and-miss rate >= epsilon. Not guaranteed: it does not retro-validate the pooled-turn CP intervals or archived operating points; it certifies no operating point in the paper; it says nothing about useful firing, cost or net latency; it does not extend to scenario families or live users outside the sampled mixture.

Fresh-conversation diagnostic (not a certificate). 50 fit plus 50 held-out conversations per domain, W = 15; the conversations are fresh draws, but their scenarios come from coverage-WTA's training half (the same 100 conversations as the known-scenario replay in C.7), so this is independence across conversations, not unseen scenarios. Retail: 61 eligible turns, 10 fires, 9 hits, 1 fired miss, 1/61 = 1.64% (bootstrap 95% CI [0, 5.56]%). Airline: 59 eligible, 12 fires, 5 hits, 7 fired misses, 7/59 = 11.86% (CI [3.28, 20.51]%), above the nominal 0.10. A paired e-test of learned-gate hit retention against epsilon-thinning (firing at random with a probability fixed on the fitting conversations so that the fired-miss rate stays below epsilon) gave a p-value bound of 1.0 in both domains, so this sample does not show the learned gate retains more hits than random thinning. The learned gate does not currently meet the target in airline.

AuthorsOpenReview ↗

Response to Reviewer P3iv (2/2)

2. Risk is not cost

What the gate controls. Section 5.5 "What the gate controls" (p9) now states: R = Pr(F = 1, H = 0), the fired-miss rate over eligible turns. It is not the conditional miss rate Pr(H = 0 | F = 1), does not count unused calls on turns that have another hit, does not weight expensive and cheap calls differently, and is not a compute or latency guarantee. A.1 (p14) repeats this and adds that neither statement supplies a net-latency guarantee. Section 5.4 (p9) now says tool-wait savings are not response-time savings.

Workload accounting (rebuttal-only). On the 100 diagnostic conversations (the 50 held-out conversations per domain) we counted every planner call over all 393 prefetch opportunities (retail 205, airline 188), including ineligible turns, scoring each unique call with the replay harness's per-tool service-delay constants (synthetic seconds, not measured work or prices):

Retail learnedRetail always-onAirline learnedAirline always-on
Fires (all opportunities)3920561188
Unique planner calls61229107307
Ready and useful calls1624821
Unused calls4520599286
Synthetic tool-seconds, total / unused249 / 171927 / 818628 / 5901867 / 1763

Of the learned gate's unused calls, 40/45 (retail) and 79/99 (airline) fall on ineligible opportunities the fired-miss rate does not see; 17 (airline) fall on the 7 miss turns. The retail miss and two airline misses produced zero grounded calls yet incurred planning. Counting every planner emission rather than unique calls gives 160 emissions / 727 tool-s (retail) and 175 / 933 (airline).

What this shows. The gate trades retained hits (it keeps 9 of the 16 eligible-turn hits always-on firing would have produced in retail and 5 of 16 in airline) for lower modeled speculative workload. It is not a cost guarantee; the cost analogue of the e-process (X_i = c A_i - W_i, W_i <= C_max) is written down but not computed. We do not derive a net latency claim from the gate; the paper's claim is control of the fired-miss rate only. End-to-end response latency with and without prefetching, with this gate in the loop, is reported in our reply to Reviewer v8Zu and in Appendix C.7: on 62 live test scenarios the complete method's mean response latency is 0.30 s lower (95% CI [-0.63, +0.07], Holm p = .091), entirely from hidden tool wait, with non-tool time per request unchanged; the gate reduces wasted speculation, it does not deliver a latency guarantee, and the live number is consistent with that reading.

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

Reviewer_vDocOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper propose PRESTO: while the user is reading the agent's reply to turn t, a small MCL predictor (~600K parameters, under 1 ms) guesses K = 5 possible next user queries. Each guess is turned into tool calls via retrieval plus a Haiku planner, and the read-only ones are run in the background and cached. When the real turn t+1 arrives, any actual tool call whose canonical key exactly matches a cached one is served from the cache. The three claimed contributions are: (1) Proxy reversal / coverage-WTA: offline commit rate is a misleading metric. Training directly on cache hits in the real loop works better; (2) Entity-revelation boundary: a tool call can only be prefetched if its arguments already appeared earlier in the conversation. (3) Conformal (Learn-then-Test) gate: decides whether to speculate at all, with a certified error bound.

Critical strengths
  1. A practical, well-motivated problem with a lossless design. The paper hides tool latency in multi-turn agents by using the user's reading time, applying the idea behind speculative decoding to user actions. Results are served only on an exact canonical-key match for read-only tools, and any miss falls back to normal execution. So a wrong guess costs only background compute, never answer quality.

  2. It catches a real evaluation pitfall and fixes it. The authors show that an offline proxy (commit rate based on embedding similarity) can rank predictors in the wrong order once they run in the real loop. The authors' propose coverage-WTA, which supervises the predictor directly with realized cache hits and needs no extra labels.

Critical weaknesses
  1. The entire paper is poorly structured and poorly written. A lot of terms are not defined when they are mentioned for the first time in the paper. For example, in the abstract: What is a WTA? What is always-fire? What is MCL (although later defined in intro)? Don’t use a term without explaining it in the paper. The actual goal of this paper is to improve the decision on whether to execute a speculatively prefetched too-call, not to improve the speculative prefetching itself or to improve the prediction of user intent right? But this is never discussed in the abstract and Line 017-028 directly goes into the technical details, which made no sense at that point. Line 017-028 is so awkwardly formulated that it reads more like a short casual conversation rather than a professional scientific paper written by a human researcher. The story runs in reverse. §5.2–5.3 present PRESTOBench results, and then §5.4 explains that this metric inverts in the real loop. A reader has to unlearn the first half of the results section. Terms include proxy reversal, entity-revelation boundary, coverage-WTA, arms P/C/A/A′/B, G0–G3, SLR, PDR, CPA, pf/turn, "fire" vs. "commit," and "banked." "Gate 2" (A.5–A.7, Tables 10–11) is never defined. There are three different PDRs (pivot, tool-match, result-usable). SLR is defined but used only once. The paper reads like a rebuttal. It often argues against anticipated objections rather than explaining the work: "It is not:" (l.200), "we claim neither," "the honest form of the moat." This hurts readability and makes it hard to tell the core contribution from defensive asides.

  2. The main paper can't stand alone. Key evidence lives only in the appendix, including A.17 (is prediction necessary?), A.18 (the coverage-WTA method itself), and A.21. The appendix keeps material that the paper's own §5.4 argument makes obsolete (A.5–A.13) next to essential new material.

  3. Coverage-WTA has no formal definition. It appears only in (Line358–362). Missing details include how the winning head is chosen when several heads hit, what "assigned in call space" means, and how masked pairs enter the loss.

  4. Most of the paper's evidence rests on a metric the paper itself rejects. A commit counts as a success if any centroid has cosine ≥ 0.3 with the embedding of the actual next utterance (Alg. 1, Line216–218). That is a low bar in all-mpnet space. The "persistence" baseline, which simply predicts that the user repeats their last utterance, already reaches 49.8% (Table 1). §5.4 also says this metric "inverts" in the real loop. Yet these results all depend on it: Tables 1, 5, 8, 10–16, Fig. 2, Table 9, and the scaling.

Questions for authors

See the weakness points above,

3 comments

Thank you for the detailed reading. We agree the draft assumed context the reader did not have. Page numbers refer to the revised PDF; where a change is a reframing rather than a fix, we say so.

1. Undefined terms

Every term you listed is now defined at first use:

  • Abstract and Introduction (p1-3): coverage-WTA ("a coverage-based winner-take-all adaptation"), MCL ("multiple-choice-learning"), the proxy reversal (p1-2), the entity-revelation boundary (p2; in full in Section 5.4, p7) and always-fire ("prefetch on every eligible turn", Contribution 3, p3).
  • Section 5.1 (p5): SLR ("Speculative Latency Ratio; the fraction of tool latency the user experiences, 1=no savings, 0=perfect speculation"; used in 5.2, p6) and the three PDRs (pivot, tool-match, result-usable) as different outcomes.
  • Sections 4.2-5.5: "fire" (start background work before the request) versus "commit" (sandbox cosine match) in 4.2 "Commit Phase" (p5); the five arms P/C/A/A'/B in text at the start of 5.4 (p6); the feature groups G0-G3 in 5.5 "The learned score" (p9).
  • Appendix, at the point of use: Gate 2 in the Table 8 caption (B.1, p19), CPA in the Table 5 footnote (A.4, p16), pf/turn in C.4 (p27), "banked" as "previously recorded (banked)" in C.5 (p29).

2. Abstract (original lines 17-28)

Rewritten (p1), same structure, every term defined at first use. It now says what the system does at decision time, which the submission's did not: "After the real request arrives, an exact matching call reuses a valid cached result; other calls execute normally", with the two decisions (whether to speculate before the request; whether to reuse a cached result after it) spelled out in "Our Approach" (p2). Two scoping statements are new: on unseen scenarios coverage-WTA does not outperform a matched proxy-supervised control, and the gate's operating points are "measured operating points rather than a guarantee". The live measurement is stated directly (a 5 to 8 percent reduction in mean response latency on 62 test scenarios, all of it hidden tool wait; Appendix C.7).

3. Story in reverse

Section 5 now opens (p5) with the proxy reversal (persistence, the repeat-last-utterance baseline, tops offline commit rate yet is at or below the predictor-free floor in exact-key hits, Table 2) and names exact-key cache hits as the primary outcome. Section 5.2 is retitled "PRESTOBench Sandbox Results" with a one-sentence caveat that its rankings need not carry over.

4. Defensive asides

Removed, not softened: the "It is not:" list in 4.1 (p4); the "moat" paragraph in 5.3 (now one sentence, p6); "we claim neither", "information-theoretically impossible" and "the indictment is sharper" in 5.4 (p6-7); and the "additive, not a reversal of the paper's thesis" paragraph.

5. Main text cannot stand alone (A.17/A.18/A.21, now C.4/C.5/C.9)

Coverage-WTA's definition and loss are now in 5.4 (p8, Eq. 4; item 7), with C.5 (p28) a method record. The "Is prediction necessary?" comparison (C.4) is summarized in 5.4 (p7). C.9 stays in the appendix for the page limit; the main text (p9) states what is there.

6. Obsolete appendix A.5-A.13

Now Appendix B, "Offline Proxy-Metric Results (Commit Rate)" (p17-25), with a lead-in stating that they use the commit-rate proxy that Section 5.4 shows does not predict exact-key cache hits. Numbers unchanged; the remaining appendices are now Appendix C.

7. Coverage-WTA formal definition

Section 5.4 (p8, Eq. 4) states the rule in words and then as a loss: an example is labeled when a read-only call the planner proposes from the masked actual next utterance matches a call the agent actually made after it, and is assigned to a head by a fixed rule on that call's canonical key, so assignment depends on the tool call rather than on which head currently predicts best; unlabeled examples are masked out, with no WTA fallback. C.5 (p28) gives the exact procedure and the planner work the labels require.

8. Proxy-metric dependence (Tables 1, 5, 8, 10-16, Fig. 2, Table 9)

Their cells are unchanged; all are cosine >= 0.3 commit-rate results. What changed is their status: Section 5 (p5-6) labels them sandbox results whose ranking need not carry over, the Appendix B lead-in (p17) says they are not evidence of useful prefetching, Section 4.2 (p5) restricts the cosine rule to the sandbox, and Section 5.5 (p9) reports that cosine tau = 0.3 fires on every evaluated turn, so it is not a selective gate. The primary results (Tables 2 and 3) and the new Appendix C.7 (p31-32) use exact-key cache hits, every null retained.

Not fixed in this revision. Section 5.4 still follows 5.2-5.3 (moving it would renumber Tables 1 and 2); sandbox table cells unchanged; the Table 2 caption still writes tau=0.3; C.4 and C.9 remain in the appendix; the task-level e-process (reply to P3iv) is not in the paper; the A.3 Table 4 caption keeps its offline ~15 USD compute-budget figure.

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

Dear Reviewer vDoc,

With the discussion closing soon, we want to check whether the revised PDF addresses the readability concerns. Every term you listed (WTA, always-fire, MCL, SLR, the PDRs, fire vs commit, the five arms, the feature groups) is now defined at first use, and the abstract is rewritten to say what the system does at decision time. Coverage-WTA has a formal definition in the main text (Eq. 4), including head assignment and how masked pairs enter the loss. Section 5 now opens with the proxy reversal and names exact-key cache hits as the primary outcome, so the commit-rate results are presented as the offline proxy the paper later rejects; the obsolete appendices (A.5 to A.13 in the submission) are grouped as Appendix B with a lead-in stating what they do and do not show, and the material the main argument depends on (former A.17, A.18, A.21) is referenced from the sections that use it. If the main text still does not stand alone for you, naming the paragraph that fails would let us fix it before the deadline.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and new results

We thank all three reviewers. A revised PDF is uploaded (main text 10 pages, Limitations on p10; appendix pages change because Appendix A was split). This note lists what changed so the revision can be read efficiently; each reviewer thread has the details.

New experiments (Appendix C.7, all nulls reported). (1) A pre-registered live serving experiment measuring complete response latency with and without prefetching on 62 test scenarios, requested by v8Zu: the complete method reduces mean response latency by 0.3 to 0.5 s per request (5 to 8%), entirely hidden tool wait; the secondary contrast passes the pre-declared criterion, the primary does not (Holm p = .091). (2) Matched in-loop baselines under one replay harness: coverage-WTA beats a read-only Speculative Actions adaptation (significant in retail after adjustment, with an abstention confound disclosed) and leads a matched Qwen2.5-0.5B LoRA model on every point estimate without reaching significance. (3) The same family includes a matched proxy-supervised MCL control; on unseen scenarios coverage-WTA does not outperform it. Contribution 1, the abstract, Section 5.4 and the Conclusion now carry this scope.

Withdrawn. The gate's operating points are now reported as measured rather than certified (P3iv): the certificate claim is withdrawn from Contribution 3, Section 5.5, Table 3, the abstract and the appendices, and the title's "risk-controlled" names the gate's design target, not a guarantee. The Sonnet cost figure is withdrawn and the one-call vs five-call inconsistency is explained (v8Zu).

Readability (vDoc). Abstract rewritten with every term defined at first use; Section 5 opens with the proxy reversal and names exact-key cache hits as the primary outcome; defensive asides removed; the coverage-WTA loss is in the main text (Eq. 4); the offline commit-rate appendices are grouped as Appendix B with a lead-in on what they do and do not show.

The numeric cells of all 24 submitted tables are unchanged; new numbers appear only in Appendix C.7 and in the live-latency sentences of the abstract, Section 5.4 and the Conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.