- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces PRESTO, a speculative prefetching framework for multi-turn tool-augmented dialogue. It predicts the user's next-turn tool needs and executes read-only calls in advance, so that a completed exact-key cache hit can eliminate the corresponding tool-execution wait. PRESTO combines a Multiple Choice Learning (MCL) predictor, a grounding pipeline using kNN retrieval and a planner LLM, and a gate based on Learn-then-Test (LTT) with Clopper-Pearson bounds intended to control the joint risk of firing and missing. The authors evaluate the method on their PRESTOBench and on external τ-bench retail and airline tasks. The main empirical finding is that optimizing an embedding-based commit proxy can reduce actual cache coverage. The proposed coverage-WTA adaptation instead uses realized cache hits to assign training targets. On two-fold -bench replay with a 15-second speculation window, it improves turn-level hit rates over zero-shot MCL from 59.4% to 70.1% in retail and from 23.7% to 32.2% in airline. The paper also includes transfer to human-authored MultiWOZ dialogue, architecture ablations, and failure analysis.
Critical strengths
- A substantive study of objective mismatch. The paper goes beyond reporting prediction accuracy and measures realized exact-key cache hits. The comparison between proxy-supervised adaptation, zero-shot prediction, and coverage-WTA demonstrates a consequential failure mode and a constructive response.
- The entity-revelation analysis provides a useful diagnostic of speculative feasibility under the paper's grounding assumptions. Distinguishing argument availability from prediction quality helps explain why the achievable cache coverage differs across domains. Relative to the reported ceilings of 88.4% for retail and 47.3% for airline, coverage-WTA closes approximately 37% and 36% of the gap left by zero-shot MCL, respectively. This is a useful complement to prediction accuracy and could inform domain selection in subsequent work. Its value is as a diagnostic under the specified argument-availability assumptions, rather than an unconditional upper bound for all predictors.
Critical weaknesses
- Insufficient empirical differentiation from Speculative Actions. As noted, Speculative Actions is PRESTO's most directly related predecessor, reporting 55% next-action prediction accuracy and system-level latency reductions at ICLR 2026. PRESTO's differentiating narrative (“predicting user utterances, not tool chains or agent actions”) is conceptually clear, but conceptual difference is not empirical advantage. The paper never compares the two speculation targets under a unified -bench replay and latency model. If a Speculative Actions-style predictor achieves comparable or higher hit rates within PRESTO's framework, then the contributions of MCL and coverage-WTA need repositioning; if MCL is indeed superior, the paper should show that comparison. Without it, the question “why MCL rather than a simpler fast-model predictor?” remains unanswered.
- The end-to-end serving benefit of the complete method is not yet established. Section 5.4 reports tool-latency savings under simulated delays and fixed speculation windows, including a 32.6% reduction for retail Arm B at W=15s. However, this does not establish the user-visible response-time improvement of coverage-WTA combined with the learned gate. The 9.6 ms “full pipeline” measurement in Appendix A.15 covers encoding, MCL, and the cosine gate, while grounding/planning and the remaining agent-serving stages are outside that measurement. Although A.17 and A.19 provide additional cost and timing analyses, I could not find an integrated evaluation reporting complete response latency with and without prefetching, including both hits and misses. This leaves the practical magnitude of the acceleration unclear.
Questions for authors
- Evaluation request: For the complete coverage-WTA plus learned-gate configuration, what are the mean and p50/p95 response latencies with and without prefetching, measured from arrival of the user's request to completion of the response? Please include all requests, including hits, misses, and turns on which the gate abstains, and report absolute times and relative reductions. Which stages finish before request arrival, and which remain on the response critical path? Please distinguish directly measured timings from simulated delays, and these results from the Arm B tool-latency savings in Section 5.4.
- Evaluation request: Tables 1 and 7 report offline results for the fine-tuned Qwen2.5-0.5B baseline. How does this baseline perform in the τ-bench replay under matched training data, candidate count, grounding, hardware, and prefetch budget, particularly at short speculation windows? Please report its in-loop hit-rate/latency tradeoff relative to MCL and specify the model variant and inference configuration. This comparison would help establish whether the practical advantage demonstrated against Sonnet also holds against a compact language-model predictor.
- Clarification request: Table 1 lists one API call for the diverse-forced Sonnet predictor, while Section 5.4 and Appendix A.19 describe five calls per turn in the monetary comparison. Which configuration is used in each result, and do the cost comparisons include the common grounding/planning stages for both MCL and Sonnet? If different configurations were used, please provide the cost corresponding to the predictor evaluated in-loop.
5 comments
Response to Reviewer v8Zu (2/2)
End-to-end latency with and without prefetching (new live experiment)
You asked for complete response latency with and without prefetching, all requests pooled, with the gate in the loop. We ran this as a live serving experiment during the discussion period, with contrasts and pass criterion written down before launch (design fixed on a disjoint 20-scenario development run).
Setup. 62 test-split tau-bench scenarios (47 retail, 15 airline), five replicate conversations per arm per scenario, five arms: N (no prefetching); C (the complete method as served: coverage-WTA predictor, learned speculation gate, planner, window-aware admission); planner-interface variants C_lookup and C_handoff; and P_handoff (proxy-supervised control; unregistered). Each scenario-replicate is a block: its five arms share the opening customer turn and run in a balanced rotating order. Agent gpt-oss-120b, planner Claude Haiku 4.5, simulated customer Claude Opus 5; 15 s speculation window, injected per-tool delays of 2-5 s (retail) and 3-10 s (airline). Response latency is wall-clock from the customer request's arrival to completion of the agent's response, over every request: hits, misses and turns on which the gate abstained. Predictor, gate, planner and agent calls are measured; tool delays are injected because sandbox tools return instantly. (The 9.6 ms figure you cite, now C.2, covers encoding, MCL and the offline gate only.) No parameter was fit on live outcomes. A conversation with an error or an incomplete request is retained and excluded from its contrast as an incomplete pair, never retried or relabeled.
Why the gate abstains on most turns. It maximizes retained hits subject to fired misses <= .10 over eligible turns and does not optimize net latency; in the original gated serving traces it abstained on 456 of 508 eligible opportunities. The complete method therefore rarely reuses the cache, bounding any end-to-end effect.
Pre-declared grading. Primary: C minus N on mean per-request response latency over complete scenario-replicate pairs, predicted negative. Secondary: C_handoff minus N. Cluster = scenario, scenarios weighted equally; two-sided sign-flip randomization p over the 62 cluster statistics, Holm over the two contrasts, 95% scenario-cluster bootstrap CI. Pass = predicted sign with Holm p < .05.
Result. 308 of 310 blocks complete; 2 failed at the shared opening turn before any arm ran.
| contrast | difference (s) | 95% CI | Holm p | pairs | arm vs N mean (s) |
|---|---|---|---|---|---|
| C minus N (primary) | -0.30 | [-0.63, +0.07] | .091 | 288 | 6.14 vs 6.45 |
| C_handoff minus N (secondary) | -0.49 | [-0.92, -0.13] | .0135 | 284 | 5.97 vs 6.46 |
Both have the predicted sign; the secondary passes the pre-declared criterion, the primary does not (Holm p = .091). Registered descriptives (no p): C minus N is -0.53 s [-0.80, -0.27] in retail and +0.43 s [-0.55, +1.60] in airline; it has the same sign in each of the three dispatch runs (-0.63, -0.68, -0.19 s); it is -0.18 s [-0.66, +0.39] on the 36 scenarios touched by earlier harness development and -0.47 s [-0.85, -0.08] on the other 26; the per-request median is unchanged (2.08 s in both arms) and the p95 moves from 25.1 s to 24.2 s. Beyond the two opening failures, the primary excludes 20 blocks (17 with an error, 3 with an incomplete request) and the secondary 24.
Where the reduction comes from. Entirely tool wait: non-tool time per request is unchanged (N 2.55 s, C 2.56 s), so the predictor, gate and planner add no measurable foreground cost (C_handoff's is 0.09 s higher). 9.1% of C requests were served at least one cached tool result, saving 5.9 s each; C's executed tool wait per request is 0.45 s lower than N's (3.21 vs 3.66 s). Removing all tool wait would save 3.66 s per request, so the live system realizes about 12% of the available reduction, against the 32.6% of eligible-turn tool wait hidden in the Section 5.4 replay; the gap is requests with no tool call, gate abstention and cache misses.
What this does and does not show. A direct measurement of the complete method's user-visible benefit against no prefetching, distinct from the Section 5.4 replay. The benefit is 0.3 to 0.5 s per request on a 6.4 s mean (5 to 8%), negative for every arm, significant after adjustment for the secondary contrast but not the primary, concentrated in retail and in requests that call tools, with no measurable overhead on requests that do not. Section 5.4, the abstract and C.7 state this result.
Scope statement. The same six-contrast family includes the matched proxy-supervised control; on scenarios disjoint from training, coverage-WTA does not outperform it (neither domain significant after adjustment; C.7), as the revised abstract, Contribution 1 and Conclusion state.
Dear Reviewers,
This is a friendly reminder that the rebuttal discussion period is approaching its deadline.
Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.
As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.
Thank you for your time and effort.
Best regards,
Area Chair
Response to Reviewer v8Zu (1/2)
Thank you for a careful review. You asked for an empirical comparison with Speculative Actions (SA), a matched in-loop Qwen2.5-0.5B baseline, a resolution of the Sonnet one-call vs five-call inconsistency, and a complete-response latency evaluation. We ran the first two under one replay harness, answer the third from source, and ran the fourth as a pre-registered live serving experiment (part 2 of this reply). All are in Appendix C.7 with every null; details we omit here are there.
Common harness. A held-out set of conversation prefixes from the no-prefetch test conversations (328 trajectories, 1,267 prefetch opportunities; retail 257 eligible turns over 57 scenarios, airline 101 over 25). Every arm sees only the prefix and shares the Haiku planner (which turns a predicted utterance into tool calls), its grounding check, a budget of 5 read-only calls per opportunity and the replay timing model. Endpoint: coverage, the fraction of eligible turns on which the tool call the agent actually needed was already cached (exact key match) within the 15 s window; scenario-cluster bootstrap CIs, exact scenario-sign tests, Holm over six contrasts. C is coverage-WTA, P the matched proxy-supervised control, Q Qwen2.5-0.5B, SA the read-only SA adaptation.
(a) Speculative Actions. The SA arm follows the SA paper's procedure, restricted to read-only tools, on the same Haiku backbone: one predicted-user-query request, then up to five sequential action decisions, executing the first tool call of each response and feeding back its observation. Coverage at W=15 s: retail C 56/257 (21.8%) vs SA 16/257 (6.2%), +15.56 pp [+9.27, +22.41], Holm p 6.58e-5; airline C 30/101 (29.7%) vs SA 14/101 (13.9%), +15.84 pp [+4.76, +27.05], Holm p .26. An independent replay on 72 known scenarios agrees: +21.31 pp (Holm .0146) retail, +20.34 pp (Holm .0127) airline.
One confound, disclosed in Section 5.3 and C.7: the SA adaptation mostly does not issue calls (34 unique calls over 922 retail opportunities and 51 over 345 airline, against 1,147 and 383 for C), so its low coverage partly reflects abstention rather than worse prediction of the needed call, and these data cannot separate the two. What survives: a grounded set of predicted utterances yields more distinct cached calls ready in time than one predicted query plus sequential decisions, significantly in retail after adjustment and in both domains on known scenarios. We claim nothing about native SA or its serving stack.
(b) Qwen2.5-0.5B in-loop. Arm Q is Qwen2.5-0.5B-Instruct with LoRA, fine-tuned on the same training-side context/next-turn pairs MCL uses, generating K=5 sampled candidates that the shared planner grounds under the same budget. Training data, K, grounding, budget and timing model are matched; compute and architecture are not (configuration in C.7). Coverage at W=15 s: retail C 56 vs Q 44 of 257, +4.67 pp [+0.36, +9.65], Holm .26; airline C 30 vs Q 17 of 101, +12.87 pp [+3.90, +21.15], Holm .26. The known-scenario replay gives +11.48 and +11.86 pp, both Holm .125. MCL leads on every point estimate, but no Qwen contrast rejects after adjustment: MCL, at about 600K parameters, is not significantly better than the matched 0.5B model at these sample sizes, as C.7 states.
(c) Sonnet one call vs five. The two statements count different stages; we failed to say so. The Sonnet predictor (prompted for diverse candidates) issues one request producing five candidate utterances; the common Haiku planner is then called once per distinct nonempty candidate, up to five times, which MCL incurs identically. Table 1's "1 call" is the predictor request; Section 5.4 and C.6 (formerly A.19) count planner calls. Our saved replay logs record call counts, not token charges, so the submitted dollar figure cannot be tied to the evaluated configuration; we withdraw it. The Table 1 caption now counts predictor requests only, and C.6 gives per-stage calls.