acceptodds
Under review as a conference paper at ICLR 2027

What Gets Deployed? Deployment Accounting for Stateful Agents

Abstract

Agentic RL studies usually report single-run success or oracle pass@k, yet practical systems may generate several sandboxed executions and release only one. These metrics can therefore misidentify both the best system and the reason it improves. We introduce deployment accounting: a protocol that jointly reports native success, successful commitment, wrong commitment, abstention, and oracle availability, and that decomposes policy differences into candidate quality and the selector's incremental value on identical pools. We instantiate a label-independent outcome selector and conduct a controlled 7B-scale study on ALFWorld and a restricted WebShop setup. Selection adds roughly 9–13 success points to the same ALFWorld weights across the evaluated learned policies. Over four continuation seeds, a simple penalty-free outcome-RL control reaches 72.5% committed success in seen rooms and 71.3% in unseen rooms, compared with 62.9% and 60.5% for the original GRPO pipeline. A commit-aware variant reaches 79.2% in the designated unseen-room run and provides favorable low-error operating points, but a matched zero-credit control shows that its auxiliary credit is not required for the strongest average policy. Same-pool analyses attribute most average gains to better candidates rather than a larger voting increment; WebShop shows that fewer wrong commitments can instead come from more abstention. These results establish deployment accounting as a practical methodology for separating learning, selection, and coverage effects before attributing improvements in stateful-agent systems.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Stateful Agents, Inference-Aware Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Sqs8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes deployment accounting for stateful agents (named CAPO), which jointly reports successful commitment, wrong commitment, native success, and oracle availability. It combines outcome-based selection with controlled RL comparisons on agentic tasks. The main finding is that candidate quality, selection, and coverage can explain different apparent improvements, while the proposed CAPO credit does not outperform the simpler control on average.

Critical strengths
  1. Evaluating success, wrong commitments, and abstention together is practically useful for LLM agents. From this perspective, the authors' comparisons on identical candidate pools help distinguish improved generation from the incremental benefit of selection.

  2. The matched D0 control and transparent reporting of negative results strengthen the empirical analysis. The authors appropriately acknowledge that the proposed CAPO is unnecessary for the strongest average policy, and that reduced errors can result from increased abstention.

Critical weaknesses
  1. The methodological contribution is limited. The proposed framework combines execution-based consensus, abstention, and standard controlled comparisons. An empirical method can still be valuable, but the paper needs stronger evidence that its stateful formulation provides substantive insight beyond applying these existing evaluation principles.

  2. The overall writing of this paper is poor, without a comprehensive, readable flow. The experimental design and results look like a draft report rather than an academic paper.

  3. The experimental evidence is too narrow to establish a robust deployment method. ALFWorld uses four continuations from one shared warm start, and the principal success-difference intervals in Table 5 include zero. Favorable low-error CAPO results rely on a single seed, while WebShop uses a restricted, low-success configuration. More fundamentally, the selector depends on imperfect action-derived signatures, but their false splits and omitted state changes are not quantitatively audited.

  4. Many necessary citations are missing. e.g., datasets adopted in Section 4.

Questions for authors
  1. How much of the reported 9–13-point deployment gain comes from empty filtering versus plurality selection? Table 4 suggests a smaller incremental benefit from voting; please report these contributions separately with uncertainty.

  2. Do the main policy rankings and low-error operating points persist across independent warm starts and all continuation seeds? Can thresholds selected on validation data retain their performance on held-out tasks?

  3. Can a factorial comparison of invalid-action penalties and normalization isolate the reward-processing explanation for D0’s gains? The current control establishes that auxiliary credit is unnecessary, but does not identify the mechanism behind the improvement.

  4. Is committing the complete state of a selected sandbox execution implemented and tested, or assumed by the evaluation? Please clarify the supported deployment setting and report selection overhead and inference cost under comparable budgets.

Reviewer_h2ucOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces deployment accounting for stateful agents that generate multiple executions and commit one result. It jointly evaluates successful commitment, wrong commitment, and abstention, while comparing selectors on identical candidate pools. Experiments on ALFWorld and WebShop distinguish candidate quality, selection gains, and coverage effects. CAPO is evaluated as one training hypothesis, and the matched D0 control shows that its auxiliary credit is unnecessary for the strongest average performance.

Critical strengths
  1. Problem is well-defined. Jointly reporting successful commitment, wrong commitment, abstention, and oracle availability captures distinctions that single-run accuracy or pass@k cannot resolve. The same-pool selector comparisons make the protocol operational and help separate candidate generation from selection.
  2. Faithful of unfavorable findings. The manuscript retains negative ablations, uncertain comparisons, and unsuccessful extensions, with providing useful analysis.
Critical weaknesses
  1. The grouping can miss state changes or separate equivalent outcomes, but the paper does not quantify how these errors affect the selected results.
  2. Unclear source of D0’s gains. The paper provides a plausible analysis of how group normalization amplifies the influence of invalid-action penalties, but does not experimentally establish whether this mechanism explains D0’s improvement over GRPO. Targeted ablations would help clarify the source and magnitude of the gain.
Questions for authors
  1. D0 achieves the strongest average performance in the main study, but the additional model experiments do not include this method. Can you provide more experiments with D0? Can targeted penalty and normalization ablations test the proposed explanation for D0’s improvement over GRPO?
  2. How often do outcome-grouping errors occur, and do the main conclusions remain unchanged when grouping is based on actual simulator-state changes?
Reviewer_BvhxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies agents that run several sandboxed attempts and release only one. It proposes "deployment accounting", a reporting protocol, instantiated with a selector that samples 8 attempts, drops those with no effect, then commits the most common outcome or abstain. It then reports successful commits, wrong commits, and abstentions together with single-run success and pass@k. The authors compare GRPO, a proposed commit-aware credit (CAPO), and a matched control without that credit (D0). Selection adds up to 13 points across policies, D0 is the strongest of the multi-seed policies on average, while CAPO’s lower WebShop error comes mostly from abstaining.

Critical strengths
  1. The penalty analysis is an interesting hypothesis, though the authors note it is untested. Eq. 3 shows that group standardization cancels the penalty size in all-failure groups, so a -0.1 penalty for invalid actions that affects less than 1% of rows can still produce a large learning signal. D0 beating GRPO in every seed on both splits (though the paired intervals include zero) is also a useful finding.

  2. Analysis of the commit / wrong-commit / abstain breakdown is useful. In WebShop, CAPO’s 8-point drop in wrong purchases comes almost entirely from abstaining more, with no gain in success. In Table 9, GRPO’s wrong-commit rate rises from 27.6% to 36.4% with the growth of k, so more samples can reinforce a wrong mode.

Critical weaknesses

The paper confirms its components are known (Sec. 8.2). The commit / wrong / abstain breakdown is risk-coverage from selective prediction. Outcome voting is execution-based selection, as in AlphaCode. With CAPO not beating D0, the contribution rests on the controlled findings. The paper calls itself "a methodology and set of controlled findings," but Figure 1, Section 3 and Algorithm 1 are built around CAPO, which is shown to not beat D0. It is hard to tell whether this is a method paper, an evaluation protocol, or an empirical study, and novelty limited. The findings come from four continuation seeds off one shared warm start of one 7B model, and every ΔC8\Delta C_8 in Table 5 includes zero.

"Selection adds 9-13 points" compares 8 samples to 1. Table 4 shows that much of this comes from just discarding empty outcomes, selecting a random nonempty outcome already raises GRPO from 53.6% to 59.4% in seen rooms. Voting itself adds only up to 6 points. The paper's own Table 4 shows this, but the abstract and Discussion still present 9-13 points as the effect of selection.

Because of limited novelty and weak evidence behind the controlled findings, I am leaning towards rejection.

Questions for authors
  1. How does the k=8 release rule compare with a learned verifier, a reward model, or an LLM judge?

  2. Could you run penalized GRPO with all-failure groups excluded? This could test the Eq. 3 explanation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.