- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper looks at a privacy risk created when tool-using LLM agents speculate, meaning they send likely future tool calls before the agent has decided it needs them, to hide latency. When a speculated branch is dropped, its calls have already reached outside providers. The paper calls these "ghost tool calls." Because a provider cannot be made to forget a call it has logged, the paper argues that privacy has to be decided at issue time, not at commit time (Proposition 1). It also argues that limiting speculation to read-only tools, as SIA does, is not the same as preventing observation (Proposition 2). The proposed fix is "Speculative Tool Privacy Contracts." Each proposed call gets labels (execution mode, observer, sensitivity, destination, confidence, audit), and a fail-closed monitor chooses one of five actions before dispatch: allow, rewrite, shadow, defer, or block. Theorem 1 says, by construction, that under complete mediation, correct labels, and correct actions, no forbidden field leaves before commitment. The evaluation uses the Speculative Actions speculator on tau^2-bench retail and airline and tau-Knowledge banking, plus generated corpora (30 and 150 tasks), a 65-task AgentDojo subset, 15 HotpotQA questions, and small live DuckDuckGo and Brave studies. Before the first committed write, the issued trace reveals the user's intent on 0.675 of tasks, against 0.211 for the committed-only trace. 31% of intent-bearing ghost calls imply an intent the task never had. Shadow+reissue cuts ghost-only recovery from 0.298 to 0.004 while serving the same 0.92 promoted calls per task. On a multi-tool live-search workload, the protection costs a median 6.8 points of the sequential baseline's latency. The authors frame their contribution as the problem, the trace model, and the measurements, not the mechanism.
Critical strengths
-
A new and timely problem: The paper argues convincingly that speculative tool dispatch is a separate privacy surface: abandoned calls leave a permanent trace with third parties. It grounds this in deployed or recommended practice (incident.io, the Stream voice-agent guide, Chrome omnibox prefetch; Table 4). The re-analysis of the published Speculative Actions results (Figure 4: 1,381 of 3,245 speculative calls are ghosts, and 24 hand a customer ID and email to the human-handoff tool in conversations that never escalate) is a good concrete motivating example.
-
A clear conceptual framing: The difference between issue time and commit time (Proposition 1), and the point that read-only does not mean observation-free (Proposition 2), are stated cleanly. The trace model (issued trace S, committed-only counterfactual C, observer projections O_i, marginal exposure Delta_i) defines a reasonable security goal. That goal explicitly does not try to hide what the committed plan itself must disclose (Section 3.2).
-
Uses third-party benchmarks and a third-party speculator: Using tau^2-bench with the speculator released by Speculative Actions, rather than only self-built corpora, reduces the worry that the setup was designed to make the defense look good (Appendix D, "How the corpora were built").
-
Many robustness checks: multiple adversaries (Opus 4.7, Haiku-4.5, GPT-4o-mini, a 7B model, keyword, TF-IDF, naive Bayes), a label-noise sweep (Table 12), destination anonymization and an enterprise-style corpus (C.19), a second speculator (D.4), and an argument-reading adversary for the committed-only baseline (D.5). A full artifact is promised (Appendix F).
Critical weaknesses
-
The technical contribution is thin, and the main defense is close to "don't speculate sensitive calls.": The paper itself says Theorem 1 "is a specification rather than a deep result: it holds by construction under A1-A3" (Section 4). The best-performing deployable policy, Shadow+reissue, leaves the provider "exactly the trace" that Drop leaves (a read-only rule extended with argument labels), and "withholding actions are interchangeable for the observer" (Section 5, Q2). So the practical recommendation is: label calls, and do not send protected ones until commit. That is a natural extension of SIA's read-only rule with a sensitivity predicate. The finding that "the leak is separable from the speedup" then follows almost by definition: speedup is kept only for calls the labeler marks benign (Section 5, "When is withholding free?"). Given the authors' own framing (problem + measurement), the paper has to be judged as a measurement paper. On that basis, the size and realism of the threat (points 2 and 3 below) matter a great deal.
-
The justification depends on labels that the paper shows are unreliable or not portable: Theorem 1 requires A2 (correct labels). In Table 22, the shipped rule labeler gets argument-class accuracy 0.310 and macro-F1 0.095 against an Opus gold annotator, because it emits a constant INTENT-REVEALING label per adapter. As a result, on the generated corpora "Rewrite, Shadow and Drop act on every speculative call" (Appendix B, Table 10 caption). Their large gains there are essentially "suppress or transform all speculation," not selective protection. The trained probe that gives selective protection drops to macro-F1 0.337 on the enterprise corpus and "protects only 0.8% of calls" on tau^2 without retraining (Appendix C.23). On the held-out 80-task check, it closes 72% of the gap but stays above No-Spec (0.450 vs 0.338, Table 24). On retail, the "free" privacy comes from the fact that protected calls are mostly writes, which speculation never serves anyway (Table 25). The general claim, "withholding sensitive speculative calls ... keeping most of speculation's benefit" (contributions, Section 1), is therefore supported mainly where the benefit was never at risk. Where intent rides in reads, the paper reports 64% (banking) and about 50% (generated corpora) of reuse kept, with labelers that must be built per schema.
-
Utility and latency evidence is weak: (a) On tau^2, the main policies are replayed over Naive's captured frontiers, so task success is the same across rows by construction (Table 1: 0.575 everywhere). Replay is checked live only on a 30-task retail slice at one guess (Tables 27 and 28). Airline and banking replay is "checked against live runs on retail only" (Table 30 caption). (b) Banking task success is 0.05 (Appendix D.2), so utility there tells us very little. (c) Wall-clock savings are small, and the baseline is sequential: "No-Spec is a sequential reference, not an optimized parallel committed-plan baseline" (Table 21). Naive saves 12.1% and Shadow+reissue 7.7%, so the contract gives up about half the saving (C.20). On the single-tool DuckDuckGo workload, the saving is 5.4%, and Shadow's p95/p99 are worse than No-Spec's (Table 16). (d) On tau^2, latency is reported only as round trips, because in-process tools make wall-clock timing uninformative. Together, these make the privacy-utility trade-off look more favorable than the evidence firmly supports.
-
The threat model leaves out the biggest observer and an important channel: The hosted planner or speculator provider "sees the prompt and every candidate branch before the monitor runs" and is out of scope (Section 3.2). For most deployed agents, which use hosted LLM APIs, this observer sees strictly more than any tool provider, so it limits how much the contract protects in practice. The decision channel (call counts, timing) is not protected. It is only measured on the 150-task corpus, and at domain level call counts alone reach 0.30-0.45 against a 0.33 chance (C.10). Complete mediation (A1) is "the deployment's to provide, not a hardened mechanism we ship" (Appendix A).
-
The privacy metric is narrow and somewhat fragile: Privacy is closed-set attribute inference over a benchmark-specific label menu, scored by an LLM at default temperature. The retail result uses a deterministic schema classifier that by design ignores reads. Empty traces count as misses, which lets suppression policies score below chance and caused the banking failure in D.4. The authors call recovery a "lower bound." But the comparisons between policies, and the claim that a policy reduces leakage "to 0.4%," depend on this specific scoring. The open-set probes are small and weak (open-set Jaccard 0.025 for Opus, Table 14) and are not run on tau^2.
-
The paper is hard to read: The main text packs a very large number of corpora, policies (5 in the main text plus 9 more), adversaries, widths, and metrics into dense prose, often with forward references to more than 30 appendix tables. Key definitions (ghost-only vs speculative recovery, the meaning of "served," which numbers are replayed and which are live) are spread across captions. Many results appear only in the appendix (for example, airline and banking tables). I had trouble telling which single experiment supports each claim in the abstract. Some abstract claims mix numbers from different settings: the 68% vs 21% uses the schema classifier before the first write, the 29.8%-to-0.4% is ghost-only replay, and the 6.8 points is a separate synthetic multi-tool workload.
Post Rebuttal: Thank you for your response. I like the topic of this paper on speculative calls to tools that may or may not be needed. The author response has satisfied some of my concerns. However, readability, the metric etc. are still an issue in my opinion. Also the response seems quite dryly written. Given this I will maintain my score.
Questions for authors
See weaknesses
3 comments
Response to Reviewer XLP6 (2/2)
W3. Utility and latency.
(a) Live runs and replay. Shadow + reissue answers a withheld call locally and sends the real call at commit, so the agent's committed calls see real results. We ran it live on 30 airline and 30 banking tasks (D.15):
| 30 tasks per domain | same abandoned calls | reads served live / replay | recovery live / replay | task success live / replay |
|---|---|---|---|---|
banking, Shadow + reissue | 30/30 | 5 / 5 | 0.067 / 0.067 | 0.067 / 0.067 |
airline, Shadow + reissue | 30/30 | 29 / 29 | 0.000 / 0.000 | 0.433 / 0.433 |
On the same tasks, Naive's live recovery is 0.367 over the 30 banking tasks and 0.438 over the 16 airline tasks that need a write. Tools ran in the sandbox. Every Shadow + reissue model request was answered from the request cache, because answering a withheld call locally changes nothing the agent, the user simulator, or the speculator sees. Drop and SIA refuse calls instead, and their abandoned calls differ from replay on 3 and 13 of 30 banking tasks. An audit against τ²'s tool code supports replay's premise on every domain and width (D.16).
(b) Banking's 5% task success. We measure disclosure and reuse from the agent's calls, independently of the task grader, so banking's low task success does not affect them (Table 1).
(c) Latency. On the multi-tool workload with live DuckDuckGo search, speculation cuts median latency by 12.1% without protection and 7.7% with the contract (Table 21), a paired cost of 6.8 points [1.2, 10.4], p = 0.013 (App. C.20). The contract sends 4.25 provider calls per task, compared with 4.24 for No-Spec and 12.72 for Naive. With simulated tool latencies of 0.1–2.5 s, unprotected speculation still saves 7–18% against a parallel committed baseline (§5 Q3; App. C.20). On τ², 71% of tool-calling turns issue a single call whose result later turns need, so few calls could run in parallel after the agent commits to them (§5 Q3). Table 16's higher tail latency is for plain Shadow, the suppression baseline, not Shadow + reissue.
(d) τ². Its tools run in-process, so Table 1 reports reuse instead of wall-clock time.
W4. Threat model. The hosted planner sees the prompt with or without speculation. If it also hosts the speculator, it writes the guesses itself, so speculation adds the tool results to its view (§3.2). Our security goal covers the extra disclosures that abandoned calls make to tool providers. These include the "financial hardship" summary sent to the escalation endpoint (§5 Q1), as well as queries sent to a knowledge-base backend or search API. On the decision channel, an observer of tool names and counts gains +0.140 from Naive's ghost calls and nothing detectable from the contract's (−0.013 [−0.035, +0.004]; D.8). Complete mediation is a deployment property, as for any reference monitor (Fig. 3).
W5. The metric. The 0.4% figure counts an empty trace as a miss, and the definitions box now says so. Five scoring methods give the same ordering without counting empty traces as misses.
| scoring | contract | no protection |
|---|---|---|
| blind annotator, runs with an intent-revealing ghost call (Table 1) | 0% / 2% / 2% | 51% / 91% / 88% |
| calibrated held-out observer, retail / banking (Table 32) | −0.009 / 0.000 bits | +0.365 / +0.668 bits |
| Opus 4.7, issued vs. committed-only, retail (App. D.5) | −0.026 (p = 0.21) | +0.482 (p = 6.8e-12) |
| observer of tool names and counts, retail (Table 38) | −0.013 | +0.140 |
| calibrated observer, second (prefetch) speculator, retail / banking (Table 34) | −0.005 / −0.004 bits | +0.947 / +0.203 bits |
The retail headline uses a deterministic schema classifier. Opus, which reads arguments, matches it (0.294 against 0.298; per-task agreement 0.91–0.97, Table 29). On the generated corpora, the ranking holds across six adversaries (App. C.9).
W6. Readability. §5 now has a definitions box, Table 1 shows all three domains with 95% intervals on every row, Figure 2 has a new cross-domain row, and the abstract ties its headline numbers to their settings, including the cut point. Sources:
| abstract number | setting | where |
|---|---|---|
| 68% vs 21% | τ² retail, schema classifier, one guess, before the first committed write | Table 35 (114 tasks); Fig. 2 top right (the 107 that commit a write) |
| 31% false intents | τ² retail, intent-bearing ghost calls, one guess | §5 Q1 |
| 29.8% → 0.4% (73.2% → 2.2%) | τ² retail, abandoned calls only, replay at one (two) guesses | Table 1; Fig. 2 top left |
| 37–51%, up to 65% | the Speculative Actions authors' GPT-5 and Gemini runs, τ-bench retail | Table 36 |
| 12.1% → 7.7%, 6.8 points | multi-tool workload with live DuckDuckGo search, against sequential No-Spec | Table 21; App. C.20 |
If these answers resolve your concerns, we would be grateful if you would consider raising your score.
Kind Reminder
Dear Reviewer XLP6,
Thank you again for your feedback. We would appreciate hearing whether our additional experiments and analysis address your concerns. If so, we would be grateful if you could reconsider your assessment. We are happy to clarify any remaining questions.
Best regards, The Authors
Response to Reviewer XLP6 (1/2)
The paper's case rests on how much speculative calls reveal about users and whether this happens in realistic settings. The revision adds those measurements, together with the label and replay checks you asked for.
W1. How much the calls reveal about users.
Testing the contract's safeguards. The contract defers sensitive calls until commit.
Dropis one configuration. None of the speculating systems we survey decides what an issued call discloses (§2, Table 4). We tested policies that omit individual checks or handle withheld calls differently.SIA), the annotator finds disclosures in 73% of banking and 22% of airline runs (Figure 2).Drop's refusals change the speculator's later calls.Dropserves 2 banking reads, compared with 5 forShadow + reissue(D.15).W2. The labels.