- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DEG checks whether an agent's claimed write actually happened to the intended target. It connects the target, write and post-state evidence, then tracks when that evidence becomes stale. Across six settings, it reports no observed false certificates and better task completion with the full system. However, the paper suffers several wekaness. The case for the graph representation is weaker than the case for tracking these evidence relations. Another weakness is that the end-to-end comparison does not cleanly isolate certification.
Critical strengths
-
IMHO, the distinction between a successful tool call and a verified effect is useful. The attribution, reachability, stale-evidence interventions make the failure modes concrete. I also appreciate the explicit limit that certifying a write does not certify the whole task.
-
The paper goes beyond its own checker with an execution audit. It also reports uncertainty around zero observed errors. That matters here, since the two AppWorld settings produce only four and seven certificates. Zero observed errors alone would be easy to overread.
Critical weaknesses
-
The case for the graph representation is weaker than the case for tracking these evidence relations. Appendix B.3's EffectRecord baseline gives the same certificates and outcomes. Although the graph's remaining advantage is efficiency, I would want a comparison with a record implementation that also memoizes checks and invalidates only affected evidence. Otherwise, some savings may come from avoiding redundant calls rather than from the graph itself.
-
Another weakness is that the end-to-end comparison does not cleanly isolate certification. Appendix B.1 gives full DEG a native candidate pool plus graph-guided execution, while the candidate-pool oracle only selects from that pool. This makes the completion gains hard to attribute. For Gemini, Table 8 also shows a small difference between the selector and full DEG: 153 versus 154 passed sub-tests out of 262.
Questions for authors
-
Clarification: what are the total candidate-generation, inference as well as tool budgets for full DEG versus the strongest selector? Does the gain remain when those budgets match?
-
Additional evidence: how much of the graph's efficiency gain remains against a memoized, scope-aware EffectRecord baseline? This would help establish the representation's contribution.