- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes deployment accounting for stateful agents (named CAPO), which jointly reports successful commitment, wrong commitment, native success, and oracle availability. It combines outcome-based selection with controlled RL comparisons on agentic tasks. The main finding is that candidate quality, selection, and coverage can explain different apparent improvements, while the proposed CAPO credit does not outperform the simpler control on average.
Critical strengths
-
Evaluating success, wrong commitments, and abstention together is practically useful for LLM agents. From this perspective, the authors' comparisons on identical candidate pools help distinguish improved generation from the incremental benefit of selection.
-
The matched D0 control and transparent reporting of negative results strengthen the empirical analysis. The authors appropriately acknowledge that the proposed CAPO is unnecessary for the strongest average policy, and that reduced errors can result from increased abstention.
Critical weaknesses
-
The methodological contribution is limited. The proposed framework combines execution-based consensus, abstention, and standard controlled comparisons. An empirical method can still be valuable, but the paper needs stronger evidence that its stateful formulation provides substantive insight beyond applying these existing evaluation principles.
-
The overall writing of this paper is poor, without a comprehensive, readable flow. The experimental design and results look like a draft report rather than an academic paper.
-
The experimental evidence is too narrow to establish a robust deployment method. ALFWorld uses four continuations from one shared warm start, and the principal success-difference intervals in Table 5 include zero. Favorable low-error CAPO results rely on a single seed, while WebShop uses a restricted, low-success configuration. More fundamentally, the selector depends on imperfect action-derived signatures, but their false splits and omitted state changes are not quantitatively audited.
-
Many necessary citations are missing. e.g., datasets adopted in Section 4.
Questions for authors
-
How much of the reported 9–13-point deployment gain comes from empty filtering versus plurality selection? Table 4 suggests a smaller incremental benefit from voting; please report these contributions separately with uncertainty.
-
Do the main policy rankings and low-error operating points persist across independent warm starts and all continuation seeds? Can thresholds selected on validation data retain their performance on held-out tasks?
-
Can a factorial comparison of invalid-action penalties and normalization isolate the reward-processing explanation for D0’s gains? The current control establishes that auxiliary credit is unnecessary, but does not identify the mechanism behind the improvement.
-
Is committing the complete state of a selected sandbox execution implemented and tested, or assumed by the evaluation? Please clarify the supported deployment setting and report selection overhead and inference cost under comparable budgets.