- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Agentic Procedural Policy Optimization (APPO), a framework that shifts branching and credit assignment from coarse-grained tool-call boundaries to fine-grained decision points in agentic reasoning trajectories. APPO introduces a future-aware Branching Score that combines token entropy with discounted suffix policy shifts, together with dual-group advantage estimation and procedure-level advantage scaling. Experiments across mathematical reasoning, knowledge-intensive reasoning, and deep-search benchmarks demonstrate improvements over existing agentic reinforcement learning baselines.
Critical strengths
- The paper identifies an important limitation of existing agentic RL methods, namely their reliance on coarse-grained branching and credit assignment at tool-call or workflow boundaries. Motivating finer-grained exploration through a pilot study provides a reasonable starting point for the proposed approach.
- APPO combines a future-aware Branching Score, dual-group advantage estimation, and procedure-level advantage scaling within a unified framework. The paper also includes component-wise ablations to investigate the contributions of these mechanisms.
- The authors evaluate APPO across a broad range of mathematical reasoning, knowledge-intensive reasoning, and deep-search benchmarks, using multiple model backbones and competitive baselines. The reported improvements across different tasks and model scales suggest the potential usefulness of the proposed method.
Critical weaknesses
-
The paper lacks sufficient theoretical justification for the proposed method. In particular, there is no rigorous analysis explaining why the discounted suffix policy-shift signal can identify important procedural decisions or improve credit assignment. The proposed mechanisms remain largely heuristic.
-
The formulation of the Branching Score raises concerns. Multiplying two z-score-normalized quantities may assign high scores to tokens with both low entropy and low policy shift. Moreover, when the current policy equals the behavior policy, the policy-shift signal becomes constant, potentially making the normalization undefined. These issues undermine the reliability of the core algorithm.
-
The claimed procedure-level credit assignment is not convincingly demonstrated. The proposed advantage estimation still relies on trajectory-level rewards, and changes in token likelihood do not necessarily reflect the causal importance of intermediate decisions. Therefore, it remains unclear whether APPO genuinely achieves fine-grained credit assignment.
-
The experimental evaluation is insufficient to fully support the claims. The paper lacks comprehensive multi-seed evaluations and statistical significance analysis, making it difficult to assess the robustness of the reported improvements. Furthermore, the ablation studies do not adequately investigate alternative Branching Score formulations or simpler advantage-scaling strategies, making it difficult to isolate the contributions of the proposed components.
Questions for authors
-
Could the authors provide stronger theoretical justification for why the discounted suffix policy-shift signal can reliably identify important procedural decisions and improve credit assignment?
-
How do the authors address the potential issues in Eq. (5), particularly the zero-variance normalization when the current policy equals the behavior policy and the possibility of assigning high scores to low-entropy, low-policy-shift tokens?
-
Could the authors clarify how the proposed procedure-level advantage estimation differs fundamentally from trajectory-level credit assignment, given that it still relies on trajectory-level rewards?
-
Could the authors provide additional ablations comparing alternative Branching Score formulations and simpler advantage-scaling strategies to better isolate the contributions of the proposed components?
2 comments
Thank you for the authors' response. I will maintain my initial score.
Thank you for the detailed assessment. We address the suffix signal, the behavior of Eq. (5), the status of the credit-assignment claim, and the ablation evidence.
The suffix signal within the score. Local entropy measures uncertainty at a candidate position. The discounted suffix signal summarizes how the likelihood of the realized continuation changes between the behavior policy and the current policy, with the form log Ω_{n,i} = Σ_{j≥i} γ^{j−i} log ρ_{n,j}, where the discount concentrates weight on nearby tokens and clipping bounds extreme ratios. The two signals describe different aspects of a candidate, namely how under-determined the position is and how strongly the observed continuation responds to the policy update. Positions in a reasoning trace are therefore ranked by local uncertainty together with the behavior of their continuation, not by local uncertainty alone. The branching criterion is a design choice whose justification is empirical, and the paper presents it as such rather than as a derived optimum. Section 2.2 defines the criterion, the pilot in Figure 2 compares branch accuracy under entropy and under the combined score, and the ablation in Table 4 shows that entropy-only selection reduces the five-task average from 58.1 to 56.3 on Qwen2.5-7B and from 57.9 to 57.0 on Llama3.1-8B.
Behavior of Eq. (5). You identify two properties of the product correctly. First, since both factors are within-rollout z-scores, the product is positive both when the two normalized terms are jointly above their means and when they are jointly below them. The score is a ranking over positions, and it does not assert that every selected position has above-average entropy and above-average suffix shift. Second, when the current policy coincides with the behavior policy every ratio equals one and the suffix signal is constant across positions. Eq. (5) computes the standard deviation within the rollout and adds a constant of 1e-6 to the denominator, so the standardized suffix value remains defined in this case, and all candidate positions then receive the same standardized suffix value. The product consequently carries no information about which position is preferable, and the retained positions follow the order in which candidate positions are enumerated along the sequence. The normalization is described in Section 2.2.
Procedure-level credit and outcome rewards. APPO optimizes with trajectory-level outcome rewards and makes the effective advantage vary across positions within a trajectory. The initial-rollout and branch groups supply outcome-based comparisons, and the suffix-based factor scales the resulting advantage at each position, so two positions in the same rollout carry different effective weights even though they share the same outcome reward. The reward direction comes from the outcome comparison, while the position weighting comes from the observed continuation. The distinction you draw is the one that matters here. A change in suffix likelihood is not a causal estimate of the effect of an individual token on the final outcome. What APPO provides is an optimization signal that is resolved at the level of positions inside a trajectory while keeping outcome-based supervision, and the ablations in Table 4 measure the effect of each part of this signal on task performance. The advantage structure is given in Section 2.3, and Appendix A summarizes the components and points to the pilot study and ablation results that examine them.
Alternative formulations and robustness. The component ablation in Table 4 reports three comparisons on the five-task average.
These isolate the branching criterion, the future-aware scaling and the dual-group estimation in turn. Appendix D visualizes alternative score combinations, including additive combinations and the suffix signal alone, as qualitative comparisons. Uncertainty is currently characterised through the consistent direction and magnitude of the ablation gaps across the two backbones.
We wish that the above would address your concerns!