- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper considers a special class of long-horizon agentic tasks in which an agent modifies an environment through multiple tool-use turns and final success is determined by a collection of executable acceptance checks. Let the terminal outcome be . The checks can also be executed on intermediate environment states, although their results are used only during training.
For task , the paper uses a set of binary checks and defines progress as the fraction currently passed:
In the experiments, only checks that fail in the initial state are included in the progress calculation, so . Consequently, can be a subset of the complete acceptance conditions: final success implies , but need not imply success. A turn receives the progress-difference reward
and a trajectory receives the score
=R+c(\Phi_T-\Phi_0).$$ ProCredit uses this signal at two levels. It standardizes $S$ among trajectories sampled for the same task to obtain a trajectory-level advantage. It also computes an undiscounted onward return for each turn, $$G_{i,t}=R_i+c(\Phi_{i,T_i}-\Phi_{i,t-1}),$$ centers these returns over all sampled turns for the task, and adds the resulting turn-level advantage to the trajectory-level advantage. All tokens within a turn receive the same combined advantage. The paper evaluates ProCredit using Qwen3.5 base models at 4B, 9B, and 35B-A3B scales. ProCredit has the highest reported mean task-goal completion rate among the included methods in all six AppWorld model-size/test-split combinations. The largest gains occur at 4B and become smaller at larger scales. A smaller ToolSandbox study also favors ProCredit over GRPO, and BFCL evaluation suggests that the measured function-calling capabilities are retained. The ablations report that adding final progress only to the trajectory score does not consistently help, whereas the complete turn-level construction does, and that undiscounted returns outperform $\gamma=0.95$ in this setup.Critical strengths
1. The method exploits a useful source of objective intermediate supervision
When executable checks can be evaluated on intermediate environment states, their changes provide more information than a single binary outcome: they can distinguish partially completed failures, supply signal in some all-fail groups, and differentiate state-changing turns from pure queries. This is practically useful and avoids training a separate reward model. The method is also simple to implement in environments that already expose suitable checks.
2. The paper provides substantial empirical evidence that its complete training rule can help
The AppWorld experiments cover three model scales and two test splits, and the paper includes outcome-only and progress-based alternatives, three independent runs, several credit granularities, and analyses of all-fail groups and discounting. In particular, the consistent ordering between ProCredit and GRPO- is meaningful evidence that the complete turn-level estimator contributes beyond using the paper's trajectory score alone. The second environment and retention evaluation are also useful, even though they do not establish broad generality.
Critical weaknesses
1. The paper does not clearly establish the research problem, its importance, or its status in prior work
The method applies not to general reinforcement learning with verifiable rewards, but to a narrower problem class: the terminal objective must be decomposable into multiple binary criteria, the criteria must be executable on intermediate environment states. This problem class should be defined explicitly before introducing ProCredit. The paper currently moves directly from examples such as AppWorld to the claim that acceptance checks can be reused as progress, without establishing how common this structure is and which important applications possess it.
More importantly, the paper does not explain whether identifying this problem class is itself new. Binary executable checks are a special case of a broader decomposed-evaluation setting in which a task is assessed by multiple checklist or rubric items, potentially with graded rather than binary scores. The paper briefly mentions CM2, VPR, GiGPO, and other related approaches, but it does not organize this literature around the underlying problem class or state what was already known about checklist-, rubric-, milestone-, and verifier-based intermediate supervision. Thus, it remains unclear whether the scientific contribution is (i) recognizing a new problem class, or (ii) proposing a new mathematical model for a known class. Without this positioning, the novelty and importance of the work cannot be assessed.
The paper should define the problem class independently of its proposed solution, explain its practical scope, review prior formulations of decomposable verifiers and rubric/checklist rewards, and identify the precise unresolved problem that ProCredit addresses.
2. The paper's reward construction is one particular modeling choice, but the most direct alternative is neither discussed nor evaluated
A more direct formulation would define progress as the fraction of all acceptance checks currently passed and redistribute the final all-check score across turns without changing the trajectory objective. The first turn would receive reward , the second would receive , and, in general, turn would receive . These rewards telescope exactly:
Thus, maximizing the total intermediate reward is exactly the same optimization problem as evaluating the fraction of all acceptance checks passed once at the end of the trajectory. Success is simply the event , and no separate score of the form is needed to recover the terminal all-check objective. This construction also retains checks that already pass initially, so damaging a required preservation condition reduces the score rather than being invisible to the progress signal.
ProCredit instead removes initially passing checks, separately retains the binary outcome, and optimizes a score of the form . These choices have substantive consequences. Removing initially passing checks can make damage to required conditions invisible to the progress signal. Adding creates a separate completion bonus, and determines how success probability is traded against partial completion.
The paper lacks discussing why choosing its current formulation instead of the more direct one.
3. The GRPO- ablation does not isolate which component of the turn-level estimator causes the gain
The paper explicitly concludes that “the gain comes from crediting progress to the turn where it occurs.” Its main evidence is the comparison between ProCredit and GRPO-, an ablation introduced by this paper. The two methods use the same filtered checks and the same trajectory score, while ProCredit additionally uses
This comparison shows that adding the complete turn-level estimator is useful under the paper's configuration. It does not isolate temporal placement of progress as the cause.
It is also unclear from the paper why helps in theory. Also, why use both and ? Why not use $A^{turn} $ alone?
Questions for authors
I don't have questions for authors.