- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CAVI, which decides whether an agent should act, ask for clarification, or declare a request non-actionable by comparing the outcomes of possible interpretations. It provides a conditional soundness result, a synthetic benchmark, and evaluations with model-generated interpretations and executable agent sandboxes. The reported results suggest that outcome-based checking can reduce unnecessary clarification while limiting incorrect actions.
Critical strengths
Comparing consequences rather than surface interpretations is a useful way to decide when clarification matters. The targeted outcome-abstraction experiment provides concrete support: PolicyOmega asks unnecessary questions on 29/80 tasks versus 78/80 for FullState, and 42 paired cases isolate the abstraction effect on identical represented spaces and concrete outcomes.
The paper explicitly separates its conditional theorem from end-to-end correctness. The parser coverage audits, transaction checks, and disclosed proposal–intent mismatch make the limits of certification visible. These diagnostics are useful beyond aggregate task success.
Critical weaknesses
The decision rule appears to discard a consequential form of ambiguity. Appendix A.1 allows a completion to be consistent with the request but non-executable, with F(z) empty. Suppose a request could refer to order A or B, A cannot be cancelled, and B can. With exact coverage, F(z_A) is empty and F(z_B) contains one outcome, so the stated rule returns CERTAIN_ACTION. Yet cancelling B is not justified if the user meant A. The theorem about equivalence of reachable outcomes can still hold, but it does not resolve this permission-to-act question. The paper needs either an additional condition or a clear justification for excluding blocked interpretations from the decision.
Several central evaluation quantities do not reconcile under their stated definitions. Section 6 defines NCR as correct clarifications among 720 gold-clarification calls, but Table 3 reports 1,554 clarification decisions and only 73 correct clarifications. With only 1,440 calls outside that gold class, at least 114 of those decisions must be correct clarifications. Appendix B also reports grounded coverage as both 91/105 and 90/105. Finally, the 44/60 executed gold-act tasks in Section 6.1 need reconciliation with the 57 successes across FS, HU, and SD in Table 7. These discrepancies affect the safety–interaction assessment, rather than only presentation.
Questions for authors
How does CAVI handle a mixture of executable and non-executable interpretations that are all consistent with the request? Does the implementation represent non-execution as a distinct outcome, require universal executability, or apply another rule absent from the formal definition?
Please provide the parser’s three-way confusion matrix, including terminal-failure routing, and reconcile the NCR, coverage, and executed-task counts identified above. In particular, does 44/60 refer to eventual execution or a more restricted stage of the trajectory?