- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies stale tool results in asynchronous LLM agents. It combines versioned results with decision-time scope watermarks to detect outdated evidence. WATERMARK then prioritizes refreshes within a budget. Controlled experiments show accuracy gains, while natural-code experiments highlight the need for atomic refresh when mixing snapshots can produce inconsistent answers.
Critical strengths
- Clear distinction between snapshot freshness and result arrival order. Schedule equivalence provides a useful framework for testing order sensitivity.
- A simple controller with measurable gains. WATERMARK-I improves accuracy by 3.09 percentage points over version-only ranking, without LLM fine-tuning.
- Shows how partial refresh can mix incompatible snapshots. Atomic refresh provides a practical extension, tested on natural code changes.
Critical weaknesses
- Influence ranking is mainly tested on threshold, band, and argmax queries. It remains unclear how the influence rule would be defined for less structured agent tasks. And the authors could explain how to choose between partial and atomic refresh.
- The interactive evaluation lacks a canonical-order baseline (e.g. by scope ID, while preserving the versions, watermarks, and refresh budget) with the same evidence and refresh budget. This would help isolate benefits beyond reducing order sensitivity.
- The interactive evidence remains limited. As reported in Appendix D, no same-schedule repeats were completed, so model variability is not cleanly separated from delivery-order effects. The DeepSeek accuracy-gain interval also includes zero. Fixed-schedule repetitions would make the attribution of answer changes to delivery order more convincing.
Questions for authors
- How can real-world tools provide scope versions and current watermarks at reasonable cost? How would coarse versions or content hashes affect performance?
- Section 3.3 distinguishes invariance of the selected refresh set from empirical end-to-end consistency. What additional runtime or model assumptions would be sufficient to make the final answer invariant as well? The main controlled result uses WATERMARK-I, whereas the interactive experiments use WATERMARK-M. Does the calibration-free policy retain its benefits in the interactive setting?
- What happens if watermarks lag, or the environment changes between refresh and commit? Would another version check be needed?
2 comments
Response to Reviewer 3XC9 (2/2): versions, final-answer invariance, lag and commit, partial vs. atomic refresh
Q1. How can real tools supply versions cheaply? Coarse versions and hashes? (Appendix J, Tables R6-R7)
Adapters can keep a revision column next to writes, use repository revision IDs, or use a strong HTTP ETag where offered. Lookup and hashing count as explicit operations, so their cost stays visible.
Missing versions fall back safely: an unknown source becomes a refresh candidate. Accuracy then declines gradually as metadata gets sparser, coarser or later, and it stays well above no-refresh even with no versions at all or a single revision domain. Hashes change nothing, because validation only needs equality.
Q2 (first part). What makes the final answer invariant, not just the refresh set? (Section 3.3, Appendix H)
The refresh cut is already identical for schedule-equivalent inputs, because it never looks at arrival time. The final answer is identical too when three conditions hold:
- (1) the same refreshed read state;
- (2) a decision view without arrival fields, so the model cannot see the order or timing of results;
- (3) a deterministic readout, or a stochastic one with shared randomness.
With independent sampling, the output distributions are identical. Table R3 tests this on the model that is sensitive to delivery order. With arrival fields visible, answers still vary with the schedule, so a stable cut alone is not enough. With arrival fields masked, the schedule effect is no longer detectable.
Q3. What if watermarks lag, or the world changes between refresh and commit? (Tables R6 and R8)
Lag is a freshness trade-off: accuracy goes down as lag grows, but stays above no-refresh.
For a change between refresh and commit: yes, strict acceptance needs a version check at commit, and it must be atomic. We make validation and acceptance one atomic operation on the store. In our SQLite test it rejects every schedule with an intervening change and never accepts a stale state. A separate check followed by a write does accept stale states. Tools that only support read-based validation get best-effort freshness through budgeted revalidation.
W1. Partial versus atomic refresh, and less structured queries (Section 3.3, Appendix I)
The rule: independent consistency domains can share the budget partially. Sources tied by a snapshot invariant, meaning they jointly determine an answer and must come from the same snapshot, are refreshed together as one domain. The task adapter declares these domains; for natural code it is the commit-level source set. If a coherent refresh does not fit the budget, the controller keeps the complete older cut.
For less structured queries, the calibration-free Watermark-FD perturbs observed values and checks how often the answer changes. On bilinear, Boolean-DNF and relational-XOR queries it beats random and version-age selection at the same refresh cost (Tables R4-R5).
Thank you again for suggesting these experiments; they made the paper stronger. We think the core idea is simple and useful: before acting, an agent should check which state a result describes, not just when it arrived. We hope this work encourages others to build on it. If our answers address your remaining concerns, we would be grateful if you would consider raising your score. We are happy to answer any follow-up questions.
Response to Reviewer 3XC9 (1/2): canonical-order baseline, fixed-schedule repetitions, interactive Watermark-I
Thank you for the constructive review. You proposed two sharp tests, and we ran both.
In short, the answers are clear. Sorting the results makes no reliable difference on any model. The gain comes from choosing what to refresh. In live agents, the fit-free Watermark-I keeps that gain on two of the three models.
This comment covers W2, W3 and the second part of Q2. The next one covers Q1, the first part of Q2, Q3 and W1. All numbers are in the revised PDF (Section 5.3, Appendices G-H).
Setup of the new interactive experiments (Table R1)
Four arms share the same cases, values, versions, watermarks and call limits:
We ran three new models, with repeated runs of every schedule.
W2. Canonical-order baseline (Table R2)
Sorting alone does not produce the gain. On all three models, Canonical is indistinguishable from Plain. Watermark-I does better than Canonical on two of the three models.
W3. Fixed-schedule repetitions (Table R3)
We repeated each fixed schedule to measure run-to-run noise, then compared it with disagreement across schedules. For one model, delivery order changes answers well beyond that noise, and a whole-session randomization test confirms it. The other two show no excess disagreement. These repeats are the variability control for the single-run DeepSeek differences in the submission.
Q2 (second part). Does the calibration-free policy keep its benefit in live agents?
Yes, on two of the three models, where it matches or beats the calibrated Watermark-M. In every session the controller enforces its frozen cut, which carries the cut invariance of Section 3.3 into agent execution.