- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DepGPO, which redistributes GRPO trajectory advantages across steps using execution dependencies traced from verifier-inspected resources. It assigns credit to relevant writes and supporting reads. Experiments with two model sizes and two training sources on Terminal-Bench 2.0/2.1 report consistent improvements over GRPO and several step-level credit assignment methods.
Critical strengths
- A well-motivated approach to step-level credit assignment. Using execution dependencies is an interesting alternative to state-based methods or learned process reward models.
- Substantial improvements across eight experimental settings (Table 1). The controlled GRPO comparison and component ablations (Table 2) provide useful supporting evidence.
Critical weaknesses
1. The validity of dependency-based credit assignment.
- Dependency on verifier-inspected resources does not necessarily imply a useful or causal contribution. For example, correct intermediate actions in failed trajectories may receive stronger negative advantages.
- Stdout dependencies rely on matching reused values (Section 3.2), which may not reflect information actually used by the agent and may miss meaningful dependencies.
- It is unclear whether the full dependency construction is necessary. Table 2 includes a direct-write ablation, but simpler alternatives such as recency-based weighting (e.g., emphasizing the last few steps) could provide useful additional comparisons.
2. Verifier-tracing assumptions and experimental validation.
- The method requires instrumented verifier resource-access traces during training. The paper could better clarify its scope of applicability, particularly for settings with black-box verifiers.
- Table 1 reports variation across evaluations of fixed checkpoints, rather than independent training seeds (Appendix B.5). This limits the evidence for training stability.
- GiGPO and GraphGPO perform substantially worse than GRPO in several settings (Table 1). Although sparse state recurrence is discussed, it is unclear whether this fully explains the poor baseline performance.
3. Other clarity issues.
- Figure 1(a) motivates the zero-advantage problem, which DepGPO itself does not address. Section 4.1 instead filters candidate tasks using mixed-outcome SFT rollouts.
- Figures 1(c)-(d) are difficult to interpret. The meaning of "supports one" in Figure 1(c) and the y-axis in Figure 1(d) are insufficiently explained.
- Fig 2 is also difficult to follow. The GiGPO illustration lacks an adequate explanation, while the red and blue arrow directions in the DepGPO example appear inconsistent with the success/failure illustrations.
Overall, the method is promising, but I am not yet fully convinced. My assessment is borderline, and I am open to reconsidering it if the main concerns are addressed.
Questions for authors
- Are the reported improvements (Table 1) and training stability (Fig 5) reproducible across independent training seeds?
- Can the authors further explain the poor performance of GiGPO and GraphGPO relative to GRPO?
- Have the authors examined whether the inferred dependencies reflect meaningful action contributions, particularly in failed trajectories?
6 comments
Response to Reviewer GJDa (2/3)
W2: What assumptions does DepGPO make about access to verifier traces?
DepGPO does not require access to the verifier's source code, internal decision logic, or intermediate correctness labels. It only requires observing which environment resources the verifier inspects during training. In our implementation, these resource-access records, including files, ports, and processes, are collected during verifier execution and used to construct the endpoint set for dependency tracing.
Importantly, the verifier's internal logic can remain opaque as long as its resource accesses are observable. For example, a verifier may internally check a service's behavior without exposing its validation rules; DepGPO only needs the recorded resource accesses to identify execution dependencies associated with that check.
Furthermore, verifier tracing is performed only after a training trajectory is completed. The resulting dependency information is used to redistribute the original trajectory advantage, rather than being provided to the agent during action generation. No verifier instrumentation is required for policy inference.
For fully inaccessible verifiers that expose only a scalar reward and no observable resource accesses, the current implementation cannot construct verifier-guided dependencies. In such cases, the documented unit-factor fallback recovers the original GRPO advantage assignment rather than introducing unsupported credit.
Therefore, DepGPO requires training-time observability of verifier resource accesses, not white-box access to the verifier itself. We will clarify this requirement and its applicability to instrumentable black-box verifiers in the revised manuscript.
W2 & Q1: Are the performance and stability improvements reproducible across independent training seeds?
The improvements in training stability are observed across both model sizes and training datasets, rather than only in the Qwen3.5-9B + SETA setting shown in Figure 5. We further summarize the training logs from all four configurations to provide a more comprehensive view of training stability.
We report three diagnostics: (1) Late Reward, the mean training reward over the final 20% of updates (30 of 150); (2) Reward Degradation, the difference between the highest 30-update moving-average reward and the final 30-update average; and (3) Late Gradient Norm, the mean gradient norm over the final 30 updates.
Table R6. Training stability across four model–dataset configurations. Each cell reports GRPO / DepGPO.
| Configuration | Late Reward ↑ | Reward Degradation ↓ | Late Gradient Norm |
|---|---|---|---|
| 9B + SETA | 0.02 / 0.54 | 0.52 / 0.00 | 0.01 / 0.58 |
| 9B + TMAX | 0.00 / 0.53 | 0.74 / 0.18 | 0.00 / 0.35 |
| 27B + SETA | 0.01 / 0.58 | 0.57 / 0.13 | 1.12 / 0.41 |
| 27B + TMAX | 0.00 / 0.81 | 0.63 / 0.01 | 0.00 / 0.23 |
DepGPO consistently maintains higher late-stage rewards and substantially smaller reward degradation across all four configurations. The gradient norms provide complementary information about optimization dynamics, although larger gradients do not necessarily indicate better performance. These statistics summarize the training behavior already observed in our main experiments, for which only representative diagnostic curves are shown in Figure 5 due to space constraints.
We also clarify that the mean ± SD in Table 1 is computed over three evaluation rounds of a fixed checkpoint, rather than independent RL training seeds. To directly assess training-seed reproducibility, we additionally consider three independent RL runs for GRPO and DepGPO under Qwen3.5-9B + SETA, using the same SFT checkpoint, training tasks, hyperparameters, and 150-update budget. Each trained policy is evaluated on Terminal-Bench 2.0/2.1, with five attempts per task. Results are aggregated as mean ± sample SD across independently trained policies.
Table R7. Independent training-seed comparison
| Method | Late Reward ↑ | Reward Degradation ↓ | TB2.0 Pass@1 (%) ↑ | TB2.1 Pass@1 (%) ↑ |
|---|---|---|---|---|
| GRPO | 0.03 ± 0.01 | 0.50 ± 0.05 | 11.19 ± 0.90 | 10.91 ± 0.83 |
| DepGPO | 0.53 ± 0.03 | 0.03 ± 0.02 | 25.19 ± 0.83 | 24.67 ± 1.16 |
In this three-seed comparison, DepGPO achieves a substantially higher late-stage reward (0.53 vs. 0.03) and much smaller reward degradation (0.03 vs. 0.50). Its average Pass@1 also exceeds GRPO by 14.00 and 13.76 percentage points on TB2.0 and TB2.1, respectively. These results would indicate that the improvements in both training stability and task performance persist across different training seeds. We will include the corresponding multi-seed experiments and additional stability analyses in the revised appendix.
Response to Reviewer GJDa (3/3)
W2 & Q2: Why do GiGPO and GraphGPO sometimes perform worse than GRPO?
The weaker performance of GiGPO and GraphGPO may arise from the difficulty of obtaining reliable step-level credit signals in terminal environments, where persistent file modifications and long-range command dependencies make state-based comparisons challenging.
- GiGPO relies on recurring environment states for step-level comparisons. However, Figure 1(b) shows that only 4 of 25 anchor states recur across the two analyzed tasks with the most anchors. Importantly, GiGPO reduces to GRPO only when its step-level advantages are entirely zero. With sparse but nonzero anchor groups, it still combines trajectory-level advantages with additional local estimates. These estimates may be noisy due to limited comparisons and differing interaction histories, potentially interfering with the trajectory-level signal and explaining why GiGPO can perform worse than GRPO rather than simply matching it.
- GraphGPO derives credit from state-transition graph structure, which may not adequately capture file-content dependencies and information flow between commands. Consequently, its structural estimates may assign importance to steps that are not closely connected to the verified outcome.
Thus, introducing step-level credit does not necessarily improve upon GRPO when the additional credit estimates are unreliable. DepGPO instead traces execution dependencies without requiring repeated states, providing more direct evidence for allocating the learning signal.
All methods share the same SFT checkpoint, training-task pool, and agent scaffold within each configuration, while retaining their respective algorithmic components. We will clarify these differences and the limitations of state-based credit assignment in the revised manuscript.
W3: Clarifications to Figures 1 and 2
We appreciate the reviewer’s suggestions and will improve the figures to make their intended meanings clearer.
Figure 1(a). This panel illustrates the zero-advantage problem in group-based methods such as GRPO, where all-success or all-failure groups provide no learning signal. It motivates our subsequent mixed-outcome task selection in Section 4.1 and Appendix B.2, rather than dependency-based credit assignment itself.
Figures 1(c)–(d). In Figure 1(c), “supports one” indicates that a command supports a downstream verifier-relevant operation. Figure 1(d) presents the cumulative distribution of the fraction of commands connected to verifier-inspected resources , where the y-axis indicates the proportion of trajectories at or below a given fraction. Across 51 passing reference trajectories, the median fraction is 0.57, dropping to 0.18 for the 18 trajectories with at least five commands. In contrast, GRPO assigns the same advantage to all steps, regardless of their connections to the verified outcome, as illustrated by the pink reference guides. These results highlight how uniform credit assignment can distribute learning signals across many commands without recorded dependencies to the final outcome, particularly in longer trajectories. These interpretations are discussed in the main text, and we will make them clearer in the figure labels and captions.
Figure 2. The left side of the GiGPO illustration shows its requirement for recurring states to construct step-level groups, while the right side highlights the difficulty of finding repeated states in terminal tasks, which limits such comparisons. For DepGPO, the success/failure illustrations emphasize how individual steps influence the final outcome, whereas the concrete example traces backward from the verifier to identify contributing commands. Both illustrate the same credit-assignment principle from different perspectives. We will unify the arrow directions and clarify the corresponding labels to avoid confusion.
We appreciate the reviewer's careful consideration of our work. We hope the clarifications, additional analyses, and controlled comparisons address the main concerns about credit validity, applicability, and reproducibility. We would be grateful if these responses could motivate a reassessment of the paper's overall rating.
Response to Reviewer GJDa (1/3)
We thank the reviewer for the thoughtful assessment and for recognizing the motivation and empirical improvements of DepGPO. We appreciate the detailed suggestions concerning credit validity, experimental reproducibility, and presentation clarity. We address these points below.
A failed trajectory does not imply that every intermediate action is incorrect, and DepGPO does not make this assumption. Its dependency weights reflect how strongly each step is connected to the verifier-inspected outcome, while the sign of the learning signal comes from the group-relative trajectory advantage.
For failed trajectories, our primary goal is to identify the operations most closely connected to the unsuccessful final outcome, rather than independently determining the correctness of every intermediate step. This distinction is important because a terminal task may involve many exploratory, preparatory, or unrelated operations, while only a subset directly shapes the final state evaluated by the verifier. Under GRPO, every step in a relatively unsuccessful trajectory receives the same negative advantage. DepGPO instead concentrates the negative signal on the recorded execution paths leading to the failed result, reducing unnecessary penalties on unrelated steps. Our existing failed-trajectory example in Appendix F.1 illustrates this behavior.
A correct intermediate action may still receive negative credit if it contributes to a failed execution path. However, this does not undermine the purpose of dependency-aware attribution: the method focuses the available outcome supervision on operations relevant to the final result, rather than treating every action in a failed trajectory as equally responsible. The same mechanism assigns stronger positive learning signals to relevant operations in successful trajectories. Furthermore, all-failure groups produce zero group-relative advantages and therefore do not introduce negative updates.
The empirical results further support the effectiveness of this selective allocation. Removing verifier-based filtering reduces performance by 6.59/6.89 percentage points on Terminal-Bench 2.0/2.1 (Table 2). The task-level analysis in Appendix C also shows improvements on 31 versus 5 regressed tasks on TB2.0 and 33 versus 2 on TB2.1, with the remaining tasks tied. Although these results do not establish exact causal attribution for every action, they demonstrate the practical benefits of using execution dependencies to guide outcome-based supervision.
Our implementation matches typed values (e.g., file paths and ports) between command outputs and subsequent commands, excluding values already provided in the task description. A match receives credit only if it leads to a downstream write with positive verifier-relevant contribution.
Although value matching may introduce spurious edges or miss implicit dependencies, typed matching and downstream verification help reduce unsupported attribution. We will clarify these rules and their limitations in the revised manuscript.
Temporal proximity does not necessarily indicate contribution: early commands may create essential intermediate files, while later steps may perform redundant checks. DepGPO instead uses execution dependencies to identify relevant steps regardless of their positions.
We introduce two controls to examine this distinction:
Both controls use Qwen3.5-9B + SETA, sharing the SFT checkpoint, training tasks, GRPO objective, and 150 updates. Results are Pass@1 (%), mean ± sample SD across three evaluation rounds, with five attempts per task.
Table R5. Recency and randomized weighting comparisons.
Recency Weighting improves over GRPO but trails Full DepGPO by 10.63/9.73 points, showing that temporal preference alone is insufficient. Randomly Permuted DepGPO trails by 12.21/11.68 points despite preserving the step-factor distribution, demonstrating the importance of aligning credit with execution dependencies, rather than merely using non-uniform weights.