acceptodds
Under review as a conference paper at ICLR 2027

Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Tools

Abstract

Tool-augmented agents are beginning to execute likely future tool calls speculatively to hide latency. When the agent abandons a speculated branch, those calls have already reached external services, carrying what the runtime guessed the user wanted. We call them ghost tool calls. Because no provider can be made to forget what it has logged, privacy must be decided when a speculative call goes out, not when its branch is dropped. We put the decision there: Speculative Tool Privacy Contracts label each proposed call and settle it before dispatch. What speculation gives away is both large and early. On a third-party agent benchmark, an observer who sees the agent's outgoing calls names what the user is deliberating about on 68% of tasks before the user confirms anything, against 21% from the calls the agent actually ends up making. Of the ghost calls that carry an intent at all, 31% name one the user never had. The leak recurs on two further domains and, independently, under a second speculator. The leak is separable from the speedup. Holding back only the sensitive speculative calls until the agent commits takes what an observer learns from the abandoned calls alone from 29.8% of tasks to 0.4%, while still allowing the agent to finish as many tasks and reuse every speculative call it would have. This costs latency: on live search, a median 6.8 points of the sequential baseline. The closest existing defense, restricting speculation to read-only calls, does not reach it: that helps only when the intent shows in which write the agent picks, and still leaks when the intent sits in what the agent reads.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Speculative Tool Execution, LLM Agents, Privacy, Information Flow Control, Agent Safety, Tool-Augmented Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XLP6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at a privacy risk created when tool-using LLM agents speculate, meaning they send likely future tool calls before the agent has decided it needs them, to hide latency. When a speculated branch is dropped, its calls have already reached outside providers. The paper calls these "ghost tool calls." Because a provider cannot be made to forget a call it has logged, the paper argues that privacy has to be decided at issue time, not at commit time (Proposition 1). It also argues that limiting speculation to read-only tools, as SIA does, is not the same as preventing observation (Proposition 2). The proposed fix is "Speculative Tool Privacy Contracts." Each proposed call gets labels (execution mode, observer, sensitivity, destination, confidence, audit), and a fail-closed monitor chooses one of five actions before dispatch: allow, rewrite, shadow, defer, or block. Theorem 1 says, by construction, that under complete mediation, correct labels, and correct actions, no forbidden field leaves before commitment. The evaluation uses the Speculative Actions speculator on tau^2-bench retail and airline and tau-Knowledge banking, plus generated corpora (30 and 150 tasks), a 65-task AgentDojo subset, 15 HotpotQA questions, and small live DuckDuckGo and Brave studies. Before the first committed write, the issued trace reveals the user's intent on 0.675 of tasks, against 0.211 for the committed-only trace. 31% of intent-bearing ghost calls imply an intent the task never had. Shadow+reissue cuts ghost-only recovery from 0.298 to 0.004 while serving the same 0.92 promoted calls per task. On a multi-tool live-search workload, the protection costs a median 6.8 points of the sequential baseline's latency. The authors frame their contribution as the problem, the trace model, and the measurements, not the mechanism.

Critical strengths
  1. A new and timely problem: The paper argues convincingly that speculative tool dispatch is a separate privacy surface: abandoned calls leave a permanent trace with third parties. It grounds this in deployed or recommended practice (incident.io, the Stream voice-agent guide, Chrome omnibox prefetch; Table 4). The re-analysis of the published Speculative Actions results (Figure 4: 1,381 of 3,245 speculative calls are ghosts, and 24 hand a customer ID and email to the human-handoff tool in conversations that never escalate) is a good concrete motivating example.

  2. A clear conceptual framing: The difference between issue time and commit time (Proposition 1), and the point that read-only does not mean observation-free (Proposition 2), are stated cleanly. The trace model (issued trace S, committed-only counterfactual C, observer projections O_i, marginal exposure Delta_i) defines a reasonable security goal. That goal explicitly does not try to hide what the committed plan itself must disclose (Section 3.2).

  3. Uses third-party benchmarks and a third-party speculator: Using tau^2-bench with the speculator released by Speculative Actions, rather than only self-built corpora, reduces the worry that the setup was designed to make the defense look good (Appendix D, "How the corpora were built").

  4. Many robustness checks: multiple adversaries (Opus 4.7, Haiku-4.5, GPT-4o-mini, a 7B model, keyword, TF-IDF, naive Bayes), a label-noise sweep (Table 12), destination anonymization and an enterprise-style corpus (C.19), a second speculator (D.4), and an argument-reading adversary for the committed-only baseline (D.5). A full artifact is promised (Appendix F).

Critical weaknesses
  1. The technical contribution is thin, and the main defense is close to "don't speculate sensitive calls.": The paper itself says Theorem 1 "is a specification rather than a deep result: it holds by construction under A1-A3" (Section 4). The best-performing deployable policy, Shadow+reissue, leaves the provider "exactly the trace" that Drop leaves (a read-only rule extended with argument labels), and "withholding actions are interchangeable for the observer" (Section 5, Q2). So the practical recommendation is: label calls, and do not send protected ones until commit. That is a natural extension of SIA's read-only rule with a sensitivity predicate. The finding that "the leak is separable from the speedup" then follows almost by definition: speedup is kept only for calls the labeler marks benign (Section 5, "When is withholding free?"). Given the authors' own framing (problem + measurement), the paper has to be judged as a measurement paper. On that basis, the size and realism of the threat (points 2 and 3 below) matter a great deal.

  2. The justification depends on labels that the paper shows are unreliable or not portable: Theorem 1 requires A2 (correct labels). In Table 22, the shipped rule labeler gets argument-class accuracy 0.310 and macro-F1 0.095 against an Opus gold annotator, because it emits a constant INTENT-REVEALING label per adapter. As a result, on the generated corpora "Rewrite, Shadow and Drop act on every speculative call" (Appendix B, Table 10 caption). Their large gains there are essentially "suppress or transform all speculation," not selective protection. The trained probe that gives selective protection drops to macro-F1 0.337 on the enterprise corpus and "protects only 0.8% of calls" on tau^2 without retraining (Appendix C.23). On the held-out 80-task check, it closes 72% of the gap but stays above No-Spec (0.450 vs 0.338, Table 24). On retail, the "free" privacy comes from the fact that protected calls are mostly writes, which speculation never serves anyway (Table 25). The general claim, "withholding sensitive speculative calls ... keeping most of speculation's benefit" (contributions, Section 1), is therefore supported mainly where the benefit was never at risk. Where intent rides in reads, the paper reports 64% (banking) and about 50% (generated corpora) of reuse kept, with labelers that must be built per schema.

  3. Utility and latency evidence is weak: (a) On tau^2, the main policies are replayed over Naive's captured frontiers, so task success is the same across rows by construction (Table 1: 0.575 everywhere). Replay is checked live only on a 30-task retail slice at one guess (Tables 27 and 28). Airline and banking replay is "checked against live runs on retail only" (Table 30 caption). (b) Banking task success is 0.05 (Appendix D.2), so utility there tells us very little. (c) Wall-clock savings are small, and the baseline is sequential: "No-Spec is a sequential reference, not an optimized parallel committed-plan baseline" (Table 21). Naive saves 12.1% and Shadow+reissue 7.7%, so the contract gives up about half the saving (C.20). On the single-tool DuckDuckGo workload, the saving is 5.4%, and Shadow's p95/p99 are worse than No-Spec's (Table 16). (d) On tau^2, latency is reported only as round trips, because in-process tools make wall-clock timing uninformative. Together, these make the privacy-utility trade-off look more favorable than the evidence firmly supports.

  4. The threat model leaves out the biggest observer and an important channel: The hosted planner or speculator provider "sees the prompt and every candidate branch before the monitor runs" and is out of scope (Section 3.2). For most deployed agents, which use hosted LLM APIs, this observer sees strictly more than any tool provider, so it limits how much the contract protects in practice. The decision channel (call counts, timing) is not protected. It is only measured on the 150-task corpus, and at domain level call counts alone reach 0.30-0.45 against a 0.33 chance (C.10). Complete mediation (A1) is "the deployment's to provide, not a hardened mechanism we ship" (Appendix A).

  5. The privacy metric is narrow and somewhat fragile: Privacy is closed-set attribute inference over a benchmark-specific label menu, scored by an LLM at default temperature. The retail result uses a deterministic schema classifier that by design ignores reads. Empty traces count as misses, which lets suppression policies score below chance and caused the banking failure in D.4. The authors call recovery a "lower bound." But the comparisons between policies, and the claim that a policy reduces leakage "to 0.4%," depend on this specific scoring. The open-set probes are small and weak (open-set Jaccard 0.025 for Opus, Table 14) and are not run on tau^2.

  6. The paper is hard to read: The main text packs a very large number of corpora, policies (5 in the main text plus 9 more), adversaries, widths, and metrics into dense prose, often with forward references to more than 30 appendix tables. Key definitions (ghost-only vs speculative recovery, the meaning of "served," which numbers are replayed and which are live) are spread across captions. Many results appear only in the appendix (for example, airline and banking tables). I had trouble telling which single experiment supports each claim in the abstract. Some abstract claims mix numbers from different settings: the 68% vs 21% uses the schema classifier before the first write, the 29.8%-to-0.4% is ghost-only replay, and the 6.8 points is a separate synthetic multi-tool workload.

Post Rebuttal: Thank you for your response. I like the topic of this paper on speculative calls to tools that may or may not be needed. The author response has satisfied some of my concerns. However, readability, the metric etc. are still an issue in my opinion. Also the response seems quite dryly written. Given this I will maintain my score.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer XLP6 (1/2)

The paper's case rests on how much speculative calls reveal about users and whether this happens in realistic settings. The revision adds those measurements, together with the label and replay checks you asked for.

W1. How much the calls reveal about users.

  • Before the first committed write, the calls already sent reveal intent on 68% of retail tasks, while the same classifier recovers only the 21% constant guess from the calls committed by then (p = 1.1e-14). Opus 4.7, which reads every argument and lookup, finds similar recovery rates (69% against 21%), and even a naive Bayes observer trained on the deployment's own traffic finds 21 points more in the issued trace (p = 7.0e-5; App. D.5).
  • At two guesses per step, abandoned calls alone reveal intent on 73% of tasks. The contract holds this at 2.2% (Fig. 2).
  • 31% of intent-bearing ghost calls name an intent the user never had (64.9% in the speculator authors' GPT-5-high runs), and these accumulate over a customer's sessions (D.6, D.7). In the speculator authors' own runs, 24 ghost calls send a customer's ID and email to the human-handoff tool in 17 conversations that never escalate (App. C.1).
  • Judged call by call by a blind annotator, 51–91% of unprotected runs send an intent-revealing abandoned call, against 0–2% under the contract (Figure 2, bottom left).
  • Who receives them. Ghost calls reach parties the user is not talking to: the escalation endpoint, the knowledge-base backend, a search API. App. C.1 gives examples of retention. An MCP gateway advertises logging every message with its full payload, and Workspace administrators cannot delete audit logs. A retailer can also receive calls that describe actions the user never intended.

Testing the contract's safeguards. The contract defers sensitive calls until commit. Drop is one configuration. None of the speculating systems we survey decides what an issued call discloses (§2, Table 4). We tested policies that omit individual checks or handle withheld calls differently.

  • Without checking whether the agent has committed to a call, our access-control baseline leaks on the static-plan corpus (0.967) and blocks the agent's own calls on τ² (task success 0.000).
  • Scrubbing calls at commit time gives the same leakage as no protection (0.298), as Proposition 1 predicts.
  • With read-only speculation (SIA), the annotator finds disclosures in 73% of banking and 22% of airline runs (Figure 2).
  • Rewriting a call while allowing its speculative branch to continue raises leakage from 0.300 to 0.533.
  • In live runs, Drop's refusals change the speculator's later calls. Drop serves 2 banking reads, compared with 5 for Shadow + reissue (D.15).

W2. The labels.

  • Table 22 covers the generated corpora, as you note. Selectivity there comes from the trained probe (Table 23, Fig. 7). On τ², schema-specific rules apply.
  • Two blind annotations check the rules. The first used Table 22's generic taxonomy and flags almost nothing on retail and airline, because it does not read intent from tool names (D.13). The second asks the threat-model question, whether a call reveals an intended action or need. It agrees with our rules (κ 0.80, 0.97, 0.59), and our rules protect 99–100% of the visible calls it flags (D.14). With its intent-or-personal-data labels in place of ours, the schema classifier's leakage is the same on all three domains.
  • A default that uses no domain lists withholds every abandoned call that names a write intent (D.12). One labeled call per tool reproduces the retail rules (D.11).
  • Held out on the generated corpora (C.23), none of the five thresholds from 0.05 to 0.4 shows a significant gap to no speculation (not an equivalence; of ten reported, 0.5 shows +0.113). At 0.3 the labeler closes 86% of the gap and still serves 44.5% of promoted calls.
  • §1 now reports reuse by domain. The contract retains 62% on airline and 64% on banking. A local copy restores all banking reuse (Figure 2, bottom middle).
AuthorsOpenReview ↗

Response to Reviewer XLP6 (2/2)

W3. Utility and latency.

(a) Live runs and replay. Shadow + reissue answers a withheld call locally and sends the real call at commit, so the agent's committed calls see real results. We ran it live on 30 airline and 30 banking tasks (D.15):

30 tasks per domainsame abandoned callsreads served live / replayrecovery live / replaytask success live / replay
banking, Shadow + reissue30/305 / 50.067 / 0.0670.067 / 0.067
airline, Shadow + reissue30/3029 / 290.000 / 0.0000.433 / 0.433

On the same tasks, Naive's live recovery is 0.367 over the 30 banking tasks and 0.438 over the 16 airline tasks that need a write. Tools ran in the sandbox. Every Shadow + reissue model request was answered from the request cache, because answering a withheld call locally changes nothing the agent, the user simulator, or the speculator sees. Drop and SIA refuse calls instead, and their abandoned calls differ from replay on 3 and 13 of 30 banking tasks. An audit against τ²'s tool code supports replay's premise on every domain and width (D.16).

(b) Banking's 5% task success. We measure disclosure and reuse from the agent's calls, independently of the task grader, so banking's low task success does not affect them (Table 1).

(c) Latency. On the multi-tool workload with live DuckDuckGo search, speculation cuts median latency by 12.1% without protection and 7.7% with the contract (Table 21), a paired cost of 6.8 points [1.2, 10.4], p = 0.013 (App. C.20). The contract sends 4.25 provider calls per task, compared with 4.24 for No-Spec and 12.72 for Naive. With simulated tool latencies of 0.1–2.5 s, unprotected speculation still saves 7–18% against a parallel committed baseline (§5 Q3; App. C.20). On τ², 71% of tool-calling turns issue a single call whose result later turns need, so few calls could run in parallel after the agent commits to them (§5 Q3). Table 16's higher tail latency is for plain Shadow, the suppression baseline, not Shadow + reissue.

(d) τ². Its tools run in-process, so Table 1 reports reuse instead of wall-clock time.

W4. Threat model. The hosted planner sees the prompt with or without speculation. If it also hosts the speculator, it writes the guesses itself, so speculation adds the tool results to its view (§3.2). Our security goal covers the extra disclosures that abandoned calls make to tool providers. These include the "financial hardship" summary sent to the escalation endpoint (§5 Q1), as well as queries sent to a knowledge-base backend or search API. On the decision channel, an observer of tool names and counts gains +0.140 from Naive's ghost calls and nothing detectable from the contract's (−0.013 [−0.035, +0.004]; D.8). Complete mediation is a deployment property, as for any reference monitor (Fig. 3).

W5. The metric. The 0.4% figure counts an empty trace as a miss, and the definitions box now says so. Five scoring methods give the same ordering without counting empty traces as misses.

scoringcontractno protection
blind annotator, runs with an intent-revealing ghost call (Table 1)0% / 2% / 2%51% / 91% / 88%
calibrated held-out observer, retail / banking (Table 32)−0.009 / 0.000 bits+0.365 / +0.668 bits
Opus 4.7, issued vs. committed-only, retail (App. D.5)−0.026 (p = 0.21)+0.482 (p = 6.8e-12)
observer of tool names and counts, retail (Table 38)−0.013+0.140
calibrated observer, second (prefetch) speculator, retail / banking (Table 34)−0.005 / −0.004 bits+0.947 / +0.203 bits

The retail headline uses a deterministic schema classifier. Opus, which reads arguments, matches it (0.294 against 0.298; per-task agreement 0.91–0.97, Table 29). On the generated corpora, the ranking holds across six adversaries (App. C.9).

W6. Readability. §5 now has a definitions box, Table 1 shows all three domains with 95% intervals on every row, Figure 2 has a new cross-domain row, and the abstract ties its headline numbers to their settings, including the cut point. Sources:

abstract numbersettingwhere
68% vs 21%τ² retail, schema classifier, one guess, before the first committed writeTable 35 (114 tasks); Fig. 2 top right (the 107 that commit a write)
31% false intentsτ² retail, intent-bearing ghost calls, one guess§5 Q1
29.8% → 0.4% (73.2% → 2.2%)τ² retail, abandoned calls only, replay at one (two) guessesTable 1; Fig. 2 top left
37–51%, up to 65%the Speculative Actions authors' GPT-5 and Gemini runs, τ-bench retailTable 36
12.1% → 7.7%, 6.8 pointsmulti-tool workload with live DuckDuckGo search, against sequential No-SpecTable 21; App. C.20

If these answers resolve your concerns, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer XLP6,

Thank you again for your feedback. We would appreciate hearing whether our additional experiments and analysis address your concerns. If so, we would be grateful if you could reconsider your assessment. We are happy to clarify any remaining questions.

Best regards, The Authors

Reviewer_VKPqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at the privacy risks of speculative execution in tool-augmented agents. Agents send tool calls when they think they might need before actually confirming them. When a call gets dropped, third-party services still log it. These "ghost calls" end up leaking the model's guesses about user intent, including things the user never even meant, and there is no way to undo that exposure once sent. Therefore, the authors argue privacy decisions have to happen right when a call goes out, not later. They label each call before it leaves to decide whether to allow, rewrite, substitute locally, defer, or block it. Their main fix, Shadow + reissue, simply holds back any sensitive speculative call until the agent commits, then sends it for real.

Critical strengths

The experimental setup reflects a real-world scenario with clear practical demand.

Critical weaknesses
  1. Ghost call volume and content depend directly on how often and how the speculator mispredicts. The core evaluation tests only Haiku, with GPT-4o-mini as a secondary speculator. Stronger models could predict more accurately and produce fewer ghost calls, or reveal more specific user intent when they fail. The paper leaves this sensitivity unexplored.

  2. The adversary is tested strictly on single-task traces. In practice, a provider observes interactions over months under the same account or ID. Accumulating ghost calls over time risks building a persistent and potentially inaccurate user profile, a threat the current single-session setup overlooks.

Questions for authors
  1. How do ghost call rates and privacy leakage change when using frontier models for speculation?

  2. When aggregating ghost calls across multiple tasks for a simulated user, does privacy leakage scale linearly or saturate, and do single-task contract guarantees remain secure under long-term accumulation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer VKPq

Frontier speculators still abandon 37–51% of their calls, and up to 65% of their intent-bearing ghost calls describe an intent the user never had. Across a customer's sessions, false intents accumulate without protection. The contract keeps every session's write intents equal to the committed plan's (228/228), and none of the 30 returning customers receives an intent-revealing ghost call.

Q1 / W1. Frontier speculators. Ghost calls stay, and the leak shifts toward false intents. The strongest predictor, GPT-5 high, abandons as large a share of its calls as nano and names a false intent more often. Fig. 4 (App. C.1) already analyzed the Speculative Actions authors' 683 published τ-bench retail runs under GPT-5 (nano, mini, high) and Gemini 2.5 Flash (low, medium, high). The revision scores what their abandoned calls reveal (§5 Q1, App. D.6, Figure 2 bottom right). The models and prompts are those of the authors' published runs.

speculatoragent's calls it predictedcalls abandonedghost calls per taskintent-bearing ghosts naming a false intenttasks given a false intent: no protection → contract
GPT-5 nano28.8%37.3%1.5535.8%12% → 0%
GPT-5 mini34.6%51.4%3.2158.7%17% → 0%
GPT-5 (high)43.1%38.4%2.3564.9%20% → 0%
Gemini 2.5 Flash (low/med/high)28.6–31.0%38–44%1.53–1.7654–60%18–23% → 0%
  • GPT-5 high predicts more of the agent's calls than nano (43.1% against 28.8%), yet it abandons a similar share of its guesses (38.4% against 37.3%) and leaves more ghost calls per task (2.35 against 1.55).
  • 64.9% of GPT-5 high's intent-bearing ghost calls name an intent the user never had (nano: 35.8%), and 20% of its tasks receive a false intent (nano: 12%). The true intent is also disclosed early. Across configurations, the full speculative trace reveals it on 42–55% of tasks, compared with 22% for a constant guess (D.6).
  • The same tool labels remove every false write-intent claim in all six configurations. Under the schema classifier, labeling the write tools gives this result by construction.

All six configurations disclose intent. GPT-5 mini produces the most ghost calls. Model and effort vary together, so these runs do not isolate model strength.

Q2 / W2. Ghost calls aggregated across a user's tasks (App. D.7, Table 37). τ² retail's 114 tasks come from 53 simulated customers, 30 of whom return for two to five tasks. Each customer is one account and each task one session. We average over 50 random session orders.

  • Linear or saturating? Events grow linearly, and distinct false intents stay flat. Without protection, on the 30 returning customers, Opus 4.7 counts 0.25 false-intent events per customer after one session and 1.0 after four. Distinct false intents per customer stay at 0.3–0.6 from the first session on, and the same abandoned calls reveal true intents (table). The provider's profile becomes both more complete and persistently wrong. Only 8 and 5 customers reach four and five sessions, so we do not claim a growth law.
  • Do the single-task guarantees hold under accumulation? Yes, for protected fields. Theorem 1 applies to each speculative decision, and its proof is an induction over any sequence of them (App. B, §3.2). Under its assumptions, no session releases a protected field before commit, so neither does their union. Under the schema classifier, the write intents the provider receives equal the committed plan's in 228/228 sessions, and the contract's abandoned calls attribute no intent at any horizon.
per returning customer, from abandoned callsno protection, 1 guessno protection, 5 guessescontract
customers given an intent they never had26–40%30–49%0%
false-intent events per session0.321.90
false-intent events over 50 sessions (projection)15.994.80
true intents revealed by session 5 (5 customers)1.70 of 2.602.60 of 2.600
  • Without protection, call by call, only 11 of 228 sessions match the committed plan (5 at five guesses): in 46 the provider receives a write intent the plan never commits, and in 171 the plan's own intents arrive before commit.
  • Two other observers. Opus 4.7 finds no false write intents in the contract's abandoned calls at any session count. The blind annotator finds an intent-revealing abandoned call for 25 of 30 returning customers without protection, and for none with the contract.
  • The guarantee does not cover linking allowed, harmless calls across sessions. Per session, an observer of tool names and counts detects no gain from the contract's traffic on τ² (D.8).

If these answers resolve your concerns, we would be grateful if you would consider raising your score.

Reviewer_sH4dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies privacy leakage caused by speculative tool execution in LLM agents. The authors point out that speculative tool calls may reveal sensitive information or inferred user intent to external providers, even if these calls are never used by the agent. The paper calls such abandoned calls Ghost Tool Calls and proposes Speculative Tool Privacy Contracts to control disclosure at issue time. The framework supports five actions: ALLOW, REWRITE, SHADOW, DEFER, and BLOCK. The authors evaluate privacy leakage, defense effectiveness, and performance costs across several agent benchmarks and speculators.

Critical strengths

The paper identifies a privacy risk in speculative tool execution that has received limited attention. The authors distinguish state mutation from external observation and explain why read-only restrictions and rollback cannot prevent information that has already been disclosed. This problem is relevant to the growing use of speculative execution for reducing agent latency.

The authors study Ghost Tool Calls on several third-party agent benchmarks and synthetic datasets. They compare different speculators, disclosure channels, and defense policies. The evaluation also includes paired replay, statistical tests, and experiments with real tool providers. These results provide useful evidence of the privacy risk and the effectiveness of some defenses.

Critical weaknesses

W1. The method is not fully specified. The authors define the Contract structure and five actions, but do not clearly explain how to identify sensitive calls, select the right defense action, or ensure that local substitutes return valid results. These key steps largely depend on external configurations or specific implementations. In addition, Shadow + reissue mainly delays sensitive calls until commitment. It also achieves the same privacy results and speculative reuse as the simpler Drop policy in the main experiments. It is therefore unclear what new technical capability the Contract framework provides beyond existing call filtering or deferred execution.

W2. The method does not fully address the trade-off between privacy and speculative execution efficiency. The paper reports a good privacy–utility trade-off on Retail. However, this is largely because sensitive intent is carried by write-tool identity, while writes must already be re-executed at commitment. When sensitive information is carried by read arguments, the defense must give up some speculative reuse. The paper therefore mainly measures this trade-off rather than providing a new way to address it. The reported benefits also depend on the task setting.

W3. The defense strongly depends on correct privacy labels. The security guarantee relies on assumptions such as label correctness and complete mediation. However, the rule-based labeler performs poorly on some datasets, and the trained labeler does not transfer well to new tool schemas. On held-out tasks, the practical labeler reduces privacy leakage but does not reach the No-Spec level. This leaves a gap between the results under ideal labels and the privacy protection that the system can provide in practice.

Questions for authors
  1. How does the system assign privacy labels to a new tool call and decide which defense action to use? Which parts are handled by the proposed method, and which require manually defined, schema-specific policies? For SHADOW, how is the local substitute built? How does the system ensure that its results are valid, and what happens if they differ from the real tool results?

  2. What is the main difference between Shadow + reissue and simply deferring sensitive speculative calls until commitment? Since Drop achieves the same privacy results and speculative reuse in the experiments, what additional capabilities does the Contract framework provide? What is the technical contribution beyond extending existing information-flow control to the speculative dispatch boundary?

  3. When sensitive intent is carried by read arguments, how does the method preserve the latency benefit of speculation while protecting privacy? Can the authors separate the benefits of their defense from those caused by the benchmark setting, such as the need to re-execute writes at commitment?

1 comment
AuthorsOpenReview ↗

Response to Reviewer sH4d

The method specifies the decision function, the five actions, and the default of deferring or blocking calls with missing labels (§4). Appendix B gives the semantics, a worked contract, and pseudocode. Each deployment supplies labels and a local function that answers withheld calls.

Q1 / W1 / W3. A new tool gets its labels from its schema, and one labeled call per tool suffices on retail.

  • A default that protects any tool not declared read-only, plus any free-text argument, withholds every abandoned call that names a write intent on all three domains (D.12). Our rules add protection for reads that reveal intent. On airline, the default lets an intent-revealing read through in 22 of 100 runs (mostly flight searches), while our rules let through only two escalations.
  • With one labeled call for each of the 15 retail tools, a probe produces exactly the same provider-visible trace as our retail rules on all 228 runs (App. D.11).
  • Choosing the action. If the choice of tool reveals intent, withhold the call. If free text reveals intent, withhold or rewrite it. A rewrite must end its speculative branch. A missing label never maps to Allow (§4).
  • A blind, intent-focused annotator agrees with our rules (κ 0.80 retail, 0.97 airline, 0.59 banking), and the rules protect 99–100% of the visible calls it flags (D.14).
  • The rule labeler's poor scores (Table 22) come from the generated corpora, where it emits a constant label per adapter. On τ², the rules are schema-specific and fixed per tool on retail.
  • Held out on the generated corpora, a learned labeler shows no significant gap to no speculation at the five thresholds from 0.05 to 0.4 (not an equivalence), and at 0.3 it still serves 44.5% of promoted calls early (C.23).

Q1. The local answer feeds only speculation. The committed plan always uses the real result. The substitute is any local function of the tool and its arguments that never contacts the provider (App. B). Shadow + reissue sends the real call at commit and uses that result for the committed plan. An inaccurate local answer affects only speculation, where calls still pass through the monitor. On τ² we run the same tool code in the sandbox to produce the local answer, so it is exact. In live airline and banking runs, the agent made the same sequence of calls (D.15).

Q2. Deferring sensitive calls. We agree that deferring a sensitive call until the agent commits to it can prevent early disclosure. Drop implements this within our contract. The contract specifies how to identify those calls, when to release them, and how to respond to the speculator while a call is withheld.

In replay, Drop and Shadow + reissue leave the same provider-visible trace. Live, refusing a call changes what the speculator does next.

  • When to release a call. A policy needs to distinguish a speculative call from one the agent has committed to. Our access-control baseline lacks this distinction. On the static-plan corpus, its leakage matches Naive (0.967). On τ², it blocks the agent's committed calls, reducing task success to 0.000.
  • Which calls to withhold. Read-only calls can reveal sensitive intent through their arguments. The blind annotator finds an intent-revealing abandoned call in 73% of SIA's banking runs and 22% of its airline runs, including flight searches. Under the contract, the corresponding rate is 2% in each domain (Figure 2, bottom left).
  • How to answer a withheld call. In live runs on 30 banking tasks, Drop's refusals changed the speculator's subsequent calls. Drop served 2 reads early, compared with 5 for Shadow + reissue, which matched replay on all 30 tasks (D.15). With an authoritative local copy, plain Shadow also serves the reads Drop withholds (D.10).

The speculating systems we survey issue calls without checking what they disclose (§2, Table 4). IFC systems could use these labels, but the committed-call IFC systems we compare have no notion of a call that may later be abandoned (§2, Table 46).

Q3 / W2. Privacy and reuse across domains (App. D.9). On retail, the privacy gain comes from withholding writes, which τ² re-runs at commit anyway, and all 209 promoted reads are kept. Serving a sensitive read early without disclosing it to the provider requires a local answer. Figure 2 (bottom middle) compares how many reads each policy serves early.

reads served earlyretailairlinebanking
no protection20911659
our rules209 (100%)72 (62%)38 (64%)
default labels (D.12)20811640

If these answers resolve your concerns, we would be grateful if you would consider raising your score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

All three reviewers agree that the problem matters in practice. sH4d notes that it has received little attention, and XLP6 calls it new and timely. We have uploaded a revised PDF with the main changes marked in blue and added the analyses requested in the reviews. All references below are to the revision. The related-work table moved from Table 36 to Table 46. We corrected the cut point in the abstract, §1, and §5 Q1, corrected one count in §5 Q3, and specified which domains the reuse claim covers in the abstract and §1. The results from the submission still hold.

Main results

  • Before the agent commits its first write (which τ² allows only after the user confirms), the calls it has already sent reveal the user's intent on 68% of retail tasks. From the calls it has committed by then, the same classifier recovers only 21%, the constant guess (p = 1.1e-14).
  • An LLM annotator (Opus 4.7) blind to our rules judged every call on its own, with no label menu and no rule for empty traces. Without protection, the provider receives an intent-revealing abandoned call in 51% of retail, 91% of airline, and 88% of banking runs. Under the contract, these rates are 0%, 2%, and 2% (Table 1 and Figure 2, bottom left).

New analyses and revisions

  • Frontier speculators (VKPq). The GPT-5 and Gemini 2.5 Flash runs counted in Fig. 4 are now scored for what they reveal (§5 Q1, App. D.6).
  • Returning customers (VKPq). Without protection, false-intent events accumulate across a customer's sessions. With the contract, every session shows the provider exactly the committed plan's write intents (228/228; App. D.7).
  • Labels (sH4d, XLP6). One labeled call per tool reproduces our retail rules (D.11). A default with no domain lists covers every write intent (D.12). A blind, intent-focused annotation agrees with our rules (κ 0.80 / 0.97 / 0.59; D.14).
  • Beyond deferral (sH4d, XLP6). Read-only speculation (SIA) leaks through banking's knowledge-base queries and airline's flight searches. Run live, Drop's refusals change what the speculator does next. A local copy keeps the reuse Drop gives up (§5 Q2, D.10, D.15).
  • Live runs (XLP6). On 30 airline and 30 banking tasks run live, Shadow + reissue matches replay on every measure (D.15).
  • Readability (XLP6). A definitions box in §5, Table 1 with all three domains and 95% intervals on every row, and a new row in Figure 2 with results across domains.

Terms used below. Naive runs speculation without protection. Shadow + reissue is our deployable policy. It answers a protected speculative call locally and sends the real call when the agent commits to it. Drop defers sensitive arguments and blocks sensitive destinations. SIA allows speculation only for read-only tools.

AuthorsOpenReview ↗

AC summary

No reviewer disputes the paper's main result. Speculative tool calls reveal user intent to third parties before the agent commits, and no commit-time control can take that back. All three reviewers accept the problem. sH4d calls it a risk "that has received limited attention", XLP6 calls it "a new and timely problem", and VKPq notes its "clear practical demand." The objections still open concern the defense prototype and its labels. They leave the problem and the measurement, which §1 states as the contribution, untouched.

The measurement uses no labels, and it holds on third-party benchmarks with a third-party speculator.

  • On τ² retail, before the first committed write, the calls already sent reveal intent on 68% of tasks. The committed calls alone reveal 21% (p = 1.1 × 10⁻¹⁴).
  • Without protection, a blind annotator finds an intent-revealing abandoned call in 51% of retail, 91% of airline, and 88% of banking runs (Table 1). Under the contract, these rates fall to 0%, 2%, and 2%.
  • Frontier speculators still abandon 37–51% of their calls, and up to 64.9% of their intent-bearing ghost calls name an intent the user never had (D.6).
  • Read-only speculation, the closest prior defense, still leaks on 73% of banking runs (§5).

Each open objection was answered.

  • FIDES (sH4d) checks committed calls. This paper is about abandoned ones. A FIDES-style policy stays inside Naive's interval (C.18). The fix sH4d proposes, adding an "execution mode" to IFC, is exactly the gap the paper identifies, and §5 Q2 shows that a mode label alone does not close it.
  • Labels (sH4d) are a deployment input, as in any access-control system. Theorem 1 states its guarantee relative to them (A2). A default with no domain lists already withholds every ghost call that names a write intent (D.12).
  • Frontier speculators and accumulation (VKPq) are measured. D.6 scores the frontier runs, and our reply of 18 Nov adds a live run. Without protection, the provider's record of 25 of 30 returning customers contains an intent-revealing abandoned call. Under the contract, none does (D.7).
  • The mechanism is thin (XLP6) by design. XLP6 writes that "the paper has to be judged as a measurement paper", and we agree. The live runs, label checks, and cross-domain results XLP6 asked for are in the revision (D.14, D.15, and Table 1). XLP6 has not responded.

The follow-ups move from one objection to the next without disputing a single result. sH4d moved from labels to FIDES, and VKPq from model strength to live runs. We ask the Area Chair to weigh what remains against the contributions the paper actually claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.