- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TASPO, a credit assignment framework that reconciles outcome-based rewards with process supervision in agentic policy optimization. The paper is motivated by two key observations:
-
Limitations of outcome-based reinforcement learning and conflicts with process supervision. Methods such as GRPO rely on final outcomes and assign the same advantage to all tokens within a trajectory, failing to distinguish their individual contributions to the final result. This motivates the introduction of process supervision. However, methods such as OPSD optimize the likelihood between current model's action and the teacher model, which differs from the outcome-based optimization objective of GRPO and may lead to conflicting optimization directions. To reconcile these objectives, TASPO uses process supervision as the weight of credit, while outcome rewards determine the polarity and mean of the advantage.
-
Context and granularity mismatches in process supervision. Process supervision in OPSD may rely on information that is not applicable to the states encountered in the target trajectory. To address this issue, TASPO introduces trajectory-aligned process supervision to ensure that the guidance is relevant to the target trajectory's actual interaction history. Furthermore, since agents interact with their environments by executing complete actions, token-level supervision may assign different credit to tokens belonging to the same action. TASPO therefore performs credit assignment at the action level, aligning the granularity of supervision with the executable decisions made by the agent.
Experiments on ALFWorld, Search-QA, and WebShop with three backbone models show an average improvement of 12.4 percentage points over GRPO across nine settings.
Critical strengths
-
The paper identifies an important challenge in reconciling process supervision with outcome-based reinforcement learning, particularly the potential misalignment between their optimization objectives. TASPO addresses this issue through a simple credit reweighting mechanism that incorporates process guidance while preserving the polarity and mean of outcome-based advantages.
-
The paper provides extensive experiments across multiple agent benchmarks and backbone models, demonstrating consistent improvements over GRPO. The ablation studies further examine key design choices and provide empirical support for the effectiveness of the proposed method.
Critical weaknesses
A critical motivation of this paper is that directly combining process supervision, such as OPSD, with outcome-based RL, such as GRPO, introduces conflicting optimization objectives. While this argument is plausible, the empirical evidence supporting the proposed mechanism is limited.
It seems that there isn't any direct evidence can directly characterize the alleged gradient conflict between GRPO and OPSD. The reported performance improvements do not establish that mitigating such conflicts is the primary reason for TASPO's advantage.
Figure 3(b) appears to be a conceptual illustration rather than a quantitative measurement of actual training gradients.
Given the central role of this argument in the paper's motivation, I believe more direct evidence of the claimed optimization objectives' conflict would substantially strengthen the contribution.
Questions for authors
- What does "TASPO" mean? I cannot find an explicit explanation of the method's name in the paper.
- Are Figures 1(b) and 1(c) based on real experimental data? The two figures are intended to illustrate the context and granularity mismatches, but the paper does not clearly explain how they were produced. Besides, Figure 1(c) appears a little different in visual style from the other experimental figures. If AI tools were used to generate or modify the figure, should you clarify their use in the AI Use Statement?
- Why are the StepOPSD results missing for Qwen2.5-7B-Instruct in Table 1?
- Could you provide more detailed statistics on the generation and utilization of privileged information during training? For example, it would be helpful to report the number of candidate guidance items extracted, the number and proportion retained after trajectory alignment, and the fraction of trajectories for which valid PI is available. Furthermore, Section 4.2 states that "As the policy improves, successful rollouts provide richer evidence for aligned guidance, enabling progressively finer credit allocation." Is this claim supported by empirical measurements?
- The baseline specification in Table 4. OPSD and GRPO+OPSD are labeled as using a "Self" teacher, whereas TASPO is labeled "Self + PI". But according to [1], OPSD should use a self-teacher conditioned on privileged information. Could you elaborate on how OPSD is implemented in your setting?
[1] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
5 comments
Response to Reviewer 1rno(2/3)
Q1. What does "TASPO" mean? I cannot find an explicit explanation of the method's name in the paper.
TASPO stands for "Trajectory-Aligned Supervision for Policy Optimization." This was the initial working title for the project, describing the method itself: performing policy optimization through process supervision that is aligned with target trajectories. We later concluded that the title should highlight our core contribution rather than the specific procedural steps—namely, how to reconcile process supervision with outcome-based credit (where the outcome determines the sign and total magnitude, while process supervision governs the relative distribution within the trajectory). Consequently, we updated the title to its current form: "Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization," while retaining the original acronym. The relationship between the name and the title is as follows: "Trajectory-Aligned Supervision" corresponds to the "process supervision" discussed in the paper (Section 3.2), while "for Policy Optimization" reflects how this approach is integrated into the optimization process by redistributing outcome-based credit (Sections 3.1 and 3.4).
Q2. Are Figures 1(b) and 1(c) based on real experimental data? [...] If AI tools were used to generate or modify the figure, should you clarify their use in the AI Use Statement?
We thank the reviewer for pointing this out, both figures were generated directly from our experimental records:
- Figure 1(b) compares measured performance under different PI sources: Generic Skill, Random Success, and Nearest Success.
- Figure 1(c) compares action-level and token-level credit allocation using LLM-as-judge assessments of actions from recorded trajectories.
- Figure 3(b) and its gradient measurements are explained in Part 1/3.
AI use. We used AI tools to generate and refine the visual presentation from our experimental data and author-specified instructions, including the layout, colors, and labels. The experimental data were obtained by us; the LLM-based assessments in Figure 1(c) and AI-assisted figure preparation are distinct uses of AI. The revised AI Use Statement(line1717-1720) discloses AI assistance in preparing the figures.
Q3. Why are the StepOPSD results missing for Qwen2.5-7B-Instruct in Table 1?
The StepOPSD entries are reported results from its paper (Zhang et al., 2026), not matched reruns, because we could not locate a usable official implementation to rerun it under our budget. That paper reports Qwen2.5-3B and Qwen3-1.7B on ALFWorld and Search-QA only, with no Qwen2.5-7B or WebShop results, hence the empty cells. In the revised version, Table 1 (line 275) marks these entries as reported results, and Appendix C.1 lists their source configurations.
Response to Reviewer 1rno(3/3)
Q4. Could you provide more detailed statistics on the generation and utilization of privileged information during training? Furthermore, Section 4.2 states that "As the policy improves, successful rollouts provide richer evidence for aligned guidance, enabling progressively finer credit allocation." Is this claim supported by empirical measurements?
We track four statistics: the number of candidate guidance items extracted per rollout group (), the number retained per target after alignment (), the retention ratio (), and the valid-PI rate, the fraction of nonzero-advantage trajectories () that receive valid aligned PI (the complement of the no-PI fallback rate reported in our response to Reviewer ikMb). The table compares the first 10 and the last 10 policy updates.
| Backbone | Environment | (%) | Valid-PI rate (%) | ||
|---|---|---|---|---|---|
| Qwen2.5-3B | ALFWorld | 4.1 → 6.8 | 3.2 → 5.9 | 78 → 87 | 83.2 → 93.6 |
| WebShop | 5.3 → 7.4 | 4.2 → 6.7 | 79 → 91 | 77.4 → 89.1 | |
| Qwen2.5-7B | ALFWorld | 4.7 → 7.2 | 3.8 → 6.5 | 81 → 90 | 86.8 → 94.5 |
| WebShop | 5.7 → 7.9 | 4.8 → 7.4 | 84 → 94 | 82.6 → 91.4 | |
| Qwen3-1.7B | ALFWorld | 2.9 → 5.4 | 2.0 → 4.6 | 69 → 85 | 72.7 → 91.0 |
| WebShop | 4.1 → 6.2 | 3.1 → 5.5 | 76 → 89 | 71.9 → 87.3 |
Across all six settings, both the amount of aligned guidance and the fraction of trajectories that receive it increase over training. This supports the statement in Section 4.2: as training progresses, successful rollouts provide richer evidence, and credit redistribution applies to more trajectories. We have added these statistics to Appendix D.8(line1560-1608) and revised the sentence in Section 4.2(line373-377) to state this more precisely.
Q5. OPSD and GRPO+OPSD are labeled as using a "Self" teacher, whereas TASPO is labeled "Self + PI". But according to [1], OPSD should use a self-teacher conditioned on privileged information. Could you elaborate on how OPSD is implemented in your setting?
Yes, the OPSD self-teacher is conditioned on PI, consistent with Self-Distilled Reasoner [1]. In Table 4, "Self" referred to the teacher model (the same base model as the student), not to the absence of PI. In our agent adaptation, each task produces 8 on-policy rollouts; the first environment-verified successful trajectory supplies its complete interaction record (actions and observations) as shared PI for the group's teacher evaluations. If no rollout succeeds, the group receives no distillation signal.
The distinction lies in how PI is constructed and used. OPSD conditions its self-teacher on a raw successful trajectory and directly optimizes a distillation objective. TASPO instead extracts guidance from successful siblings, aligns it with each target's realized history, and uses action-level likelihood shifts to redistribute the original outcome advantage.
We have revised Table 4 (line 449) to distinguish raw-PI and aligned-PI conditioning explicitly.
Again, we thank the reviewer for the valuable feedback. Please let us know if there are any further questions or suggestions.
[1] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
Response to Reviewer 1rno(1/3)
We thank the reviewer for the careful reading and for recognizing the challenge we address and the breadth of our experiments. This part addresses the weakness; Parts 2/3 and 3/3 answer the questions. Revisions addressing your comments are highlighted in dark green; the shared PI-availability analysis in Appendix D.8 is highlighted in blue.
We agree that performance improvements alone do not establish the mechanism. Figure 3(b) visualizes measured objective gradients from training. We clarify its construction and provide complementary evidence at three levels.
1. Token-level coefficient disagreement. On ALFWorld with Qwen2.5-3B-Instruct, we measure disagreement between the outcome and distillation surrogate coefficients using AiΔ<0, where Δ=logπ(y∣PI)−logπ(y). Excluding ∣Δ∣<0.05, the disagreement rate is 31%, averaged over 150 training updates, and 38% when weighted by ∣Ai∣∣Δ∣. Among these conflicting tokens, 63% occur in failed trajectories and receive positive teacher support. TASPO does not optimize a separate distillation term; its positive weights introduce no coefficient-sign reversals relative to the outcome advantage.
2. Gradient-level measurements. Let gU, d, and gT denote the GRPO, OPSD, and TASPO objective gradients. This diagnostic uses raw successful-trajectory PI for OPSD.
Figure 3(b) projects the full-batch gradients at update 80 onto the plane spanned by gU and d. Dots show per-trajectory projections, and arrow lengths are normalized for display. In this plane, the joint gradient gJ=gU+λd, with λ=0.1, deviates from gU by 35°, compared with 6° for TASPO. Additionally, 23% of per-trajectory OPSD gradients point more than 90° away from the outcome reference.
To examine whether this pattern extends beyond one update, we measure gradient directions in full parameter space at 15 checkpoints, from update 10 to 150 in increments of 10:
Early and late denote updates 10–50 and 100–150, respectively. The aggregate distillation gradient has weak average alignment with the outcome gradient, while approximately one-third of trajectory-level distillation gradients oppose the batch outcome reference. TASPO exhibits smaller mean angular deviations in both windows.
3. Training-level sensitivity to the distillation coefficient. Separately, Appendix D.2 compares GRPO+OPSD and TASPO under the same aligned-PI construction protocol, using three training seeds. Each method constructs PI from its own on-policy trajectories. We test three distillation coefficients around λ0=0.1:
Among the tested coefficients, λ0 achieves the highest GRPO+OPSD success rate. Larger coefficients are accompanied by larger mean gradient deviations. TASPO exceeds all three configurations under the matched PI-construction protocol.
Together, these results document objective disagreement and smaller gradient deviations under TASPO, alongside improved performance in the controlled comparison. They support our motivation for constrained redistribution, but do not by themselves establish gradient rotation as the primary cause of the performance gap.
Revision. We have clarified the discussion in Section 4.2(line372-377), added the token-level coefficient disagreement analysis and gradient diagnostics to Appendix B.6(line1049-1073), and included the matched-PI distillation-coefficient study in Appendix D.2(line1262-1284).