- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ReCASH-PM, a framework that separates state acquisition, request representation, and action selection to audit whether remembered requests warrant execution. Across 512 dialogue prefixes, two model families, and 13 conditions, structured state improves overall accuracy by 10.55–24.02 percentage points while reducing authorized-action recall to 0–3.41%. Field-based rules, alternative information views, and closure checks help diagnose these failures. Additional evaluations on ABCD, EAL, ToolSandbox, and PM-Bench examine action prediction and execution. The results show that higher accuracy can conceal missed authorized actions; the revised interface partially restores recall but increases false execution, with no consistent gains on complete execution tasks.
Critical strengths
- The paper introduces ReCASH-PM, which separately examines state acquisition from dialogue, request-state representation, and action selection. The actions are FIRE for immediate execution, DEFER for waiting, VERIFY for verification, and RETIRE for closure. The most notable finding is that structured state can improve four-class accuracy while almost entirely suppressing actions that should be executed.
- The primary experiments use human reference labels and paired bootstrap resampling with dialogue-level clusters.
- The authors explicitly acknowledge that Revised does not meet the prespecified criterion for joint improvement and that the actual execution experiments show no consistent gains. This transparency deserves recognition.
Critical weaknesses
1 The central mechanism remains unresolved Original and Revised jointly change the schema, writer instructions, and output budget, while fields-only and summary-only expose different information (§5.2). These comparisons therefore do not isolate representation effects. Moreover, acquired fields frequently rule out FIRE on authorized cases (§5.1); historical results show that Gemma marks authorization as unknown in all 88 FIRE cases (Table 25). Although the authors acknowledge the limits of causal attribution, the evidence still cannot distinguish state-acquisition failures from failures in using the resulting representation. 2 The scientific contribution beyond the closest prior work remains unclear EAL-Bench already separates authorization errors in memory from their propagation to actions and uses memory interventions to localize failures. StateMem studies evolving state with length- and cost-matched controls, while PIS implements intention lifecycle logic through explicit state and code. This paper adds a four-action taxonomy, occurrence-specific closure evidence, and comparisons on shared dialogue prefixes. However, it remains unclear what new diagnostic conclusions these additions enable. The principal findings substantially overlap with existing work on authorization fidelity and the tradeoff between missed and unauthorized actions.
Questions for authors
- How can the effects of errors in state acquisition be distinguished from the effects of state representation on decisions?
- If the executor still cannot correctly assess authorization when given accurate memory, how can the results support conclusions about the role of memory?