acceptodds
Under review as a conference paper at ICLR 2027

Snapshot Illusion: Cursor Exhaustion Does Not License Global Claims

Abstract

Tool-using language-model agents often treat cursor exhaustion as evidence for an exact global claim. Yet complete pagination certifies a traversal, not that its pages describe one database state. We formalize this Snapshot Illusion as an identifiability failure: if two executions induce the same trace and contract but different target answers, no trace-only rule can soundly choose an exact answer. The result covers counts, sets, none/all, uniqueness, sums, and extrema. Across 7,220 versioned SnapshotBench episodes, five open-weight families make unsupported exact claims in all 719/719 valid controlled dynamic responses; matched eligible controls and calculator normalization rule out traversal and arithmetic as the main cause. The rate is 238/300 on public NYC 311 histories, 67/420 for seven advanced APIs on frozen traces, and 86/120 under ordinary six-model interaction. SnapGuard, a certificate-checking release monitor, has 0/880 observed unsafe structured releases and 755/880 safe resolutions. On the same saved natural responses, blinded review finds that a heuristic adapter leaves 18/120 unsafe releases and yields 90/120 valid qualifications. Target-bound snapshots permit exact release in 106/108 eligible runs, but only 58/120 scheduled runs are correct. Capability narrows the empirical gap; coherent evidence closes the logical one but does not ensure correct computation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
tool-using language models, language-model agents, snapshot consistency, pagination, evidence identifiability, runtime verification, AI safety, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_hsYVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors discuss a type of failure mode in agentic systems in which the agent may draw some conclusion after having enumerated data/evidence while the underlying data/evidence may change during the period of time the agent examines said data. The authors use pagination with cursors as a recurring example -- an agent that does not find some matching rows may conclude none exist even if the rows may have changed before that cursor reaches the end. The paper informally appears to address, how can the agent know if its observations in fact match reality?

The authors present a good deal of formal development that, as far as I can tell, is basically "A fact is known if and only if it holds in every execution compatible with one's observations."

Critical strengths
  • It is straightforward, intuitive, and arguably "right" to split up the validity of tools from the identifiability of evidence from the validity of inference, to then find issues with evidence.

  • The evaluation appears rigorous and well-controlled. The observations separating open-weight and API-based models is quite compelling.

Critical weaknesses
  • Based on how the paper defines eligibility, the types of failures observed in this manuscript are based on the nature of the contract assumed, and not necessasrily upon the model under test. It does not seem that the authors really ever consider an evaluation under a realistic deployment contract, nor does it assess or analyze what these contracts would look like in production. To this end, the authors claiming something like "agents make unsupported claims at 100%" is perhaps overstated.
Questions for authors
  • Can you provide an example contract under which traces would be eligible with realistic backends (e.g., soft-delete or append-only audit logging)? How does this affect the practical significance of your findings?

  • Could you separate instruction following from claim qualification by adding a condition to reason about whether evidence is identified? Consider: same prompt, same tools, but a system prompt that indicates the eligibility criteria without telling the model the answer?

Reviewer_Q6LiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The core problem which the paper or the authors are trying to address is the following: AI agents nowadays pull information from underlying data sources. This data source could be a database, a file system, etc. Now, it is possible that the data source goes through continuous change even while the agent is trying to access the information from the data source. In this case, whether the agent has correctly read the information or not depends on the sequence in which the reading and the updates to the system were made. The authors try to formalize this problem statement and provide a solution to avoid this issue.

Critical strengths
  1. Identified issue is timely, relevant and practical.
  2. Comprehensive evaluation consisting of both synthetic and real-world traces, multiple model families etc.
Critical weaknesses

As has been the case with most of the papers I've reviewed in this cycle, it has been very, very difficult to understand this paper. The writing is very, very difficult to follow. Unfortunately, it just appears that the authors have used too much AI to write the paper. That has led to essentially the standard problem nowadays: it is very difficult to follow and understand what is being presented in the paper.

Request the authors to please spend some time, maybe rewriting the paper from scratch without AI and then using AI to improve the grammar and presentation, or revising the current draft comprehensively (so that a reader like me, who is reasonably well-versed with the agentic AI literature, can understand and make sense of the paper).

Questions for authors

See Weaknesses

Comments on the paper

1 comment
Area_Chair_uTpFOpenReview ↗

Authors response?

Dear authors,

Author-reviewer discussions end on November 18th. Please submit your responses to the reviewers reviews as soon as possible, so the reviewers would be able to respond to you.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.