- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks at agents that complete a task but change records the user did not ask to change, which the official evaluators fail to notice. The authors argue that pre-execution checks are "effect blind" because the request does not say what must stay unchanged, or because the agent's observations omit records that decide the outcome. The paper proposes intent-consistent completion (ICC), which adds both independently evaluated forbidden-effect and preservation constraints to official success, plus LEDGER, a method that extracts an effect contract along with a record-level effect prediction in contexts isolated from the executor and checks prior records by reading off their receipt identifiers. LEDGER then reconciles these in code, deciding if it should execute, revise or stop. Evaluation uses three backbones on 21 AppWorld-Amazon order tasks, 8 AppWorld-Splitwise continuations, 97 AgentDojo user tasks and 43 SWE-bench tasks, with Self-critique, GuardAgent and AGrail serving as its baselines.
Critical strengths
- The problem the paper targets is real, but official success hides it, as the paper shows. On AppWorld-Amazon, Self-critique, with the best official success of all methods, has just 28.6% ICC, since its runs drop items from the cart or buy unrelated ones. This makes ICC a good metric in its own right, independent of LEDGER.
- The explanation that observations can miss records is supported by both an ablation and paired tests. With no verified prior records, 13 of 24 Splitwise runs record one expense twice. On Amazon, the gain over the No check baseline is also nearly entirely one-sided over paired runs. Reading receipts gives a check some information it would otherwise lack, so LEDGER's contribution goes beyond a change to the prompt.
Critical weaknesses
- For the unstated-preservation source, LEDGER does not overcome the ambiguity formalized in Equation 3; it resolves it with a default, given in Section 2.2 as K "preserves every record the request does not address", said by the Limitations to match the evaluated constraints. None of the evaluated tasks has an unrequested adjacent change that the user would want, so ICC cannot tell a method inferring intent from one applying this convention, and it treats any harmless unrequested change as a violation.
- The central claim, "overcoming effect blindness" in the title and abstract, is therefore supported only where observations miss records.
- As far as I can tell, the headline margins over baselines are mainly due to the extracted contract. Contract critic, a free-text reviewer that gets LEDGER's contract without prediction or code reconciliation, isn't significantly different from LEDGER on Amazon ICC, with 11 against 8 discordant pairs, and even gets more ICC runs on Splitwise. Prediction and reconciliation do reduce cart loss further.
- The margins over the strongest baseline are measured against baselines given no scope statement specific to the request, so they mostly reflect giving the check a request-specific contract.
- The supporting evidence is also thin: the Splitwise margin comes to just three of 24 runs, and on AgentDojo LEDGER clearly loses to No check in a paired comparison, mostly through abstentions from contracts that omit a requested write, so the general conclusion that a safe check must compare effects with the authorized request rests mainly on the two AppWorld settings.
Questions for authors
- How does ICC behave on tasks where an unrequested adjacent change is acceptable, and has a human judged a sample of runs that ICC flags?
- Could Contract critic be reported in Table 2 with the same ICC and violation metrics? What about simply telling the executor, in its prompt, to preserve unrelated records?
- Would AppWorld's built-in state-diff evaluation already flag the lost cart and wish-list records in the 21 Amazon tasks, and how do your forbidden-effect and preservation constraints differ from its collateral-damage checks?
- Would running Progent, Task Shield, or a simple hand-written preservation rule for each adapter on the AppWorld settings show if an LLM-extracted contract with effect prediction is needed over simpler request-derived policies?
- What fraction of AgentDojo abstentions would a fix for contracts omitting a requested write recover, and would it change the AppWorld results? Work on out-of-scope actions in benign tasks, such as "Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks", may also help place ICC in context.