- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors discuss a type of failure mode in agentic systems in which the agent may draw some conclusion after having enumerated data/evidence while the underlying data/evidence may change during the period of time the agent examines said data. The authors use pagination with cursors as a recurring example -- an agent that does not find some matching rows may conclude none exist even if the rows may have changed before that cursor reaches the end. The paper informally appears to address, how can the agent know if its observations in fact match reality?
The authors present a good deal of formal development that, as far as I can tell, is basically "A fact is known if and only if it holds in every execution compatible with one's observations."
Critical strengths
-
It is straightforward, intuitive, and arguably "right" to split up the validity of tools from the identifiability of evidence from the validity of inference, to then find issues with evidence.
-
The evaluation appears rigorous and well-controlled. The observations separating open-weight and API-based models is quite compelling.
Critical weaknesses
- Based on how the paper defines eligibility, the types of failures observed in this manuscript are based on the nature of the contract assumed, and not necessasrily upon the model under test. It does not seem that the authors really ever consider an evaluation under a realistic deployment contract, nor does it assess or analyze what these contracts would look like in production. To this end, the authors claiming something like "agents make unsupported claims at 100%" is perhaps overstated.
Questions for authors
-
Can you provide an example contract under which traces would be eligible with realistic backends (e.g., soft-delete or append-only audit logging)? How does this affect the practical significance of your findings?
-
Could you separate instruction following from claim qualification by adding a condition to reason about whether evidence is identified? Consider: same prompt, same tools, but a system prompt that indicates the eligibility criteria without telling the model the answer?
Authors response?
Dear authors,
Author-reviewer discussions end on November 18th. Please submit your responses to the reviewers reviews as soon as possible, so the reviewers would be able to respond to you.
Thank you.