- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TEMPO studies injection-independent time-of-check-to-time-of-use (TOCTOU) attacks against tool-using LLM agents. An adversary modifies shared state after validation but before execution, causing subsequent actions to use different data. The paper introduces three attack primitives, evaluates multiple victim models and defenses, and reproduces attacks with an independent-process adversary in LangGraph. Its central finding is that content-level safeguards cannot ensure state consistency, while execution-level defenses depend on object binding and protection scope.
Critical strengths
The paper distinguishes content safety from state consistency. Matched experiments show that content-hardening measures reduce prompt-injection success but provide little protection against the corresponding TOCTOU attacks. This offers useful evidence that agent security requires safeguards at the execution layer.
Critical weaknesses
-
The causal interpretation of the defense results is insufficiently isolated. Section 4.4 attributes cross-model variation primarily to execution timing, but the appendices document additional mechanisms, including missing-snapshot allow paths, snapshot re-baselining, and independent refusal by use tools. Some successful attacks under CAS involve retries that adopt attacker-modified state after CAS correctly rejects the initial attempt. These outcomes also depend on snapshot management and authorization policies. Although the paper discloses these mechanisms, their contributions are not sufficiently separated to support the timing-based explanation.
-
Key statistics supporting the defense conclusions are inconsistent. Table 6 reports a locking ASR of 66.7%, whereas the text describes 66.7% protection and attributes all residual failures to delayed-context attacks. These statements do not align under the balanced three-primitive evaluation. Table 16 also leaves unresolved whether the guard-trigger count is 300 or 450. Since defense analysis is a central contribution, consistent accounting is necessary to assess the conclusions reliably.
Questions for authors
-
Could you provide a complete successful attack trace under CAS, identifying each state modification, validation, snapshot update, and final use? Did the final action use a state different from the one checked during the last successful validation?
-
Could you reconcile the locking results and guard-trigger counts using consistent per-run records, and separately report genuine guard interventions, missing-snapshot allow paths, snapshot re-baselining, and independent tool refusals?
2 comments
Thank you for the detailed clarifications and additional results. My concerns have been largely addressed, and I am inclined to raise my overall rating to 3.
Response to Reviewer so5V: timing controls, CAS trace, and locking accounting
We thank the reviewer; both weaknesses are correct.
W1. The weakness is correct: §4.4 attributed cross-model variation to execution timing while the appendices separately documented missing-snapshot paths, snapshot re-baselining, and independent refusal by the use tool. A controlled experiment separates them. Same task, same mutation, same initial state; only the write time varies — pre_check, check_use (after the check returns, before the guard), post_use (after the action record is written) — crossed with four defenses, 30 runs per cell, 360 runs total, zero crashes and zero censoring:
All 12 cells reach ASR 100%. The data support the causal statement: "The variance is not in the rule logic; it is in what happens after a rejection: whether the victim retries, whether the retry re-baselines the snapshot, and whether the surrounding tool independently authorizes the action." Timing is necessary for the window to exist; it does not decide the outcome.
W2. The weakness is correct: the locking figure was described as a protection rate while the table reports ASR, and the guard-trigger count was left ambiguous. The reconciliation, the per-family split and the category accounting are under Q2.
Q1. Complete trace, check_use/cas, trial 0:
Direct answer. The final action used version 1, the state of the last successful validation — so it is not different from it; it differs from the first validation (version 0). CAS correctly rejected the first attempt; the retry re-validated on the attacker's state, so the comparison that mattered was v1 == v1. The failure is snapshot re-baselining, not a missing atomicity check.
Q2. Locking results and accounting.
Here H = 16 genuine holds; Rj = 300 runs with at least one rejection; H/Rj = 16/300 = 5.33% is a post-selection diagnostic. NRP = BP × (1 − ASR), with BP the benign-task pass rate.
Residual failures are not in delayed-context:
Guard trigger: the guard fires in all 450 runs; 300 is the number of runs with at least one rejection. Appendix B's "300 of those 450" therefore mislabels its denominator, and H/Rj is conditioned on a post-treatment event — a diagnostic, never compared across levels.
Category accounting, per run:
The 16 tool refusals are the refused subset of the 284 re-baselined runs (268 + 16), not an additional category on top of the 284. The guard rejects at least one use in 66.7% of runs (300/450). Of those 300 rejected runs, 16 (5.33%) end in a genuine hold and another 16 (5.33%) are refused by the use tool; 268 are exploited after a retry. That conversion — 66.7% rejection coverage → 5.33% genuine holds — is the quantitative form of "the guard fires but does not help".