acceptodds
Under review as a conference paper at ICLR 2027

Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents

20 views

Abstract

Terminal-using agents benefit from reinforcement learning (RL) in coding, debugging, and other multi-step terminal tasks. In these tasks, later commands often depend on information or intermediate results produced by earlier commands. However, existing trajectory-level and step-level credit assignment methods do not explicitly trace the read-write dependencies through which commands affect the final outcome. Consequently, training signals could still be assigned to irrelevant operations, weakening learning from relevant steps. In this paper, we propose Dependency-Aware Group Policy Optimization (DepGPO), which uses execution dependencies between commands to guide credit assignment for terminal agents. Specifically, we construct a command dependency graph from execution traces and trace backward from the resources inspected by the task verifier. We then assign credit to relevant writes and their supporting reads along these paths, and use it to redistribute trajectory advantages across steps. Extensive comparative experiments and ablation studies demonstrate that DepGPO improves task performance and training stability on complex terminal tasks.

Primary area:
reinforcement learning
Keywords:
Large Language Models, LLM Agents, Reinforcement Learning, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GJDaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DepGPO, which redistributes GRPO trajectory advantages across steps using execution dependencies traced from verifier-inspected resources. It assigns credit to relevant writes and supporting reads. Experiments with two model sizes and two training sources on Terminal-Bench 2.0/2.1 report consistent improvements over GRPO and several step-level credit assignment methods.

Critical strengths
  • A well-motivated approach to step-level credit assignment. Using execution dependencies is an interesting alternative to state-based methods or learned process reward models.
  • Substantial improvements across eight experimental settings (Table 1). The controlled GRPO comparison and component ablations (Table 2) provide useful supporting evidence.
Critical weaknesses

1. The validity of dependency-based credit assignment.

  • Dependency on verifier-inspected resources does not necessarily imply a useful or causal contribution. For example, correct intermediate actions in failed trajectories may receive stronger negative advantages.
  • Stdout dependencies rely on matching reused values (Section 3.2), which may not reflect information actually used by the agent and may miss meaningful dependencies.
  • It is unclear whether the full dependency construction is necessary. Table 2 includes a direct-write ablation, but simpler alternatives such as recency-based weighting (e.g., emphasizing the last few steps) could provide useful additional comparisons.

2. Verifier-tracing assumptions and experimental validation.

  • The method requires instrumented verifier resource-access traces during training. The paper could better clarify its scope of applicability, particularly for settings with black-box verifiers.
  • Table 1 reports variation across evaluations of fixed checkpoints, rather than independent training seeds (Appendix B.5). This limits the evidence for training stability.
  • GiGPO and GraphGPO perform substantially worse than GRPO in several settings (Table 1). Although sparse state recurrence is discussed, it is unclear whether this fully explains the poor baseline performance.

3. Other clarity issues.

  • Figure 1(a) motivates the zero-advantage problem, which DepGPO itself does not address. Section 4.1 instead filters candidate tasks using mixed-outcome SFT rollouts.
  • Figures 1(c)-(d) are difficult to interpret. The meaning of "supports one" in Figure 1(c) and the y-axis in Figure 1(d) are insufficiently explained.
  • Fig 2 is also difficult to follow. The GiGPO illustration lacks an adequate explanation, while the red and blue arrow directions in the DepGPO example appear inconsistent with the success/failure illustrations.

Overall, the method is promising, but I am not yet fully convinced. My assessment is borderline, and I am open to reconsidering it if the main concerns are addressed.

Questions for authors
  1. Are the reported improvements (Table 1) and training stability (Fig 5) reproducible across independent training seeds?
  2. Can the authors further explain the poor performance of GiGPO and GraphGPO relative to GRPO?
  3. Have the authors examined whether the inferred dependencies reflect meaningful action contributions, particularly in failed trajectories?
6 comments
AuthorsOpenReview ↗

Response to Reviewer GJDa (1/3)

We thank the reviewer for the thoughtful assessment and for recognizing the motivation and empirical improvements of DepGPO. We appreciate the detailed suggestions concerning credit validity, experimental reproducibility, and presentation clarity. We address these points below.


W1 & Q3: Can dependency-based credit assignment provide meaningful supervision in failed trajectories?

A failed trajectory does not imply that every intermediate action is incorrect, and DepGPO does not make this assumption. Its dependency weights reflect how strongly each step is connected to the verifier-inspected outcome, while the sign of the learning signal comes from the group-relative trajectory advantage.

For failed trajectories, our primary goal is to identify the operations most closely connected to the unsuccessful final outcome, rather than independently determining the correctness of every intermediate step. This distinction is important because a terminal task may involve many exploratory, preparatory, or unrelated operations, while only a subset directly shapes the final state evaluated by the verifier. Under GRPO, every step in a relatively unsuccessful trajectory receives the same negative advantage. DepGPO instead concentrates the negative signal on the recorded execution paths leading to the failed result, reducing unnecessary penalties on unrelated steps. Our existing failed-trajectory example in Appendix F.1 illustrates this behavior.

A correct intermediate action may still receive negative credit if it contributes to a failed execution path. However, this does not undermine the purpose of dependency-aware attribution: the method focuses the available outcome supervision on operations relevant to the final result, rather than treating every action in a failed trajectory as equally responsible. The same mechanism assigns stronger positive learning signals to relevant operations in successful trajectories. Furthermore, all-failure groups produce zero group-relative advantages and therefore do not introduce negative updates.

The empirical results further support the effectiveness of this selective allocation. Removing verifier-based filtering reduces performance by 6.59/6.89 percentage points on Terminal-Bench 2.0/2.1 (Table 2). The task-level analysis in Appendix C also shows improvements on 31 versus 5 regressed tasks on TB2.0 and 33 versus 2 on TB2.1, with the remaining tasks tied. Although these results do not establish exact causal attribution for every action, they demonstrate the practical benefits of using execution dependencies to guide outcome-based supervision.


W1: How reliable are standard-output dependencies?

Our implementation matches typed values (e.g., file paths and ports) between command outputs and subsequent commands, excluding values already provided in the task description. A match receives credit only if it leads to a downstream write with positive verifier-relevant contribution.

Although value matching may introduce spurious edges or miss implicit dependencies, typed matching and downstream verification help reduce unsupported attribution. We will clarify these rules and their limitations in the revised manuscript.


W1: Could simpler recency-based weighting explain the improvements?

Temporal proximity does not necessarily indicate contribution: early commands may create essential intermediate files, while later steps may perform redundant checks. DepGPO instead uses execution dependencies to identify relevant steps regardless of their positions.

We introduce two controls to examine this distinction:

  • Recency Weighting: Assigns increasing weights to later steps, fi,t=2t/(Ti+1)f_{i,t}=2t/(T_i+1), without dependency information.
  • Randomly Permuted DepGPO: Shuffles DepGPO's step factors among traced steps within each trajectory, preserving their values and sparsity but breaking dependency alignment. Untraced steps retain factor one; token normalization is recomputed.

Both controls use Qwen3.5-9B + SETA, sharing the SFT checkpoint, training tasks, GRPO objective, and 150 updates. Results are Pass@1 (%), mean ± sample SD across three evaluation rounds, with five attempts per task.

Table R5. Recency and randomized weighting comparisons.

MethodTB2.0TB2.1
GRPO11.09 ± 0.2611.01 ± 0.67
Recency Weighting14.61 ± 1.2514.91 ± 1.24
Randomly Permuted DepGPO13.03 ± 0.9012.96 ± 1.01
Full DepGPO25.24 ± 1.0124.64 ± 2.36

Recency Weighting improves over GRPO but trails Full DepGPO by 10.63/9.73 points, showing that temporal preference alone is insufficient. Randomly Permuted DepGPO trails by 12.21/11.68 points despite preserving the step-factor distribution, demonstrating the importance of aligning credit with execution dependencies, rather than merely using non-uniform weights.

AuthorsOpenReview ↗

Response to Reviewer GJDa (2/3)

W2: What assumptions does DepGPO make about access to verifier traces?

DepGPO does not require access to the verifier's source code, internal decision logic, or intermediate correctness labels. It only requires observing which environment resources the verifier inspects during training. In our implementation, these resource-access records, including files, ports, and processes, are collected during verifier execution and used to construct the endpoint set EE for dependency tracing.

Importantly, the verifier's internal logic can remain opaque as long as its resource accesses are observable. For example, a verifier may internally check a service's behavior without exposing its validation rules; DepGPO only needs the recorded resource accesses to identify execution dependencies associated with that check.

Furthermore, verifier tracing is performed only after a training trajectory is completed. The resulting dependency information is used to redistribute the original trajectory advantage, rather than being provided to the agent during action generation. No verifier instrumentation is required for policy inference.

For fully inaccessible verifiers that expose only a scalar reward and no observable resource accesses, the current implementation cannot construct verifier-guided dependencies. In such cases, the documented unit-factor fallback recovers the original GRPO advantage assignment rather than introducing unsupported credit.

Therefore, DepGPO requires training-time observability of verifier resource accesses, not white-box access to the verifier itself. We will clarify this requirement and its applicability to instrumentable black-box verifiers in the revised manuscript.


W2 & Q1: Are the performance and stability improvements reproducible across independent training seeds?

The improvements in training stability are observed across both model sizes and training datasets, rather than only in the Qwen3.5-9B + SETA setting shown in Figure 5. We further summarize the training logs from all four configurations to provide a more comprehensive view of training stability.

We report three diagnostics: (1) Late Reward, the mean training reward over the final 20% of updates (30 of 150); (2) Reward Degradation, the difference between the highest 30-update moving-average reward and the final 30-update average; and (3) Late Gradient Norm, the mean gradient norm over the final 30 updates.

Table R6. Training stability across four model–dataset configurations. Each cell reports GRPO / DepGPO.

ConfigurationLate Reward ↑Reward Degradation ↓Late Gradient Norm
9B + SETA0.02 / 0.540.52 / 0.000.01 / 0.58
9B + TMAX0.00 / 0.530.74 / 0.180.00 / 0.35
27B + SETA0.01 / 0.580.57 / 0.131.12 / 0.41
27B + TMAX0.00 / 0.810.63 / 0.010.00 / 0.23

DepGPO consistently maintains higher late-stage rewards and substantially smaller reward degradation across all four configurations. The gradient norms provide complementary information about optimization dynamics, although larger gradients do not necessarily indicate better performance. These statistics summarize the training behavior already observed in our main experiments, for which only representative diagnostic curves are shown in Figure 5 due to space constraints.

We also clarify that the mean ± SD in Table 1 is computed over three evaluation rounds of a fixed checkpoint, rather than independent RL training seeds. To directly assess training-seed reproducibility, we additionally consider three independent RL runs for GRPO and DepGPO under Qwen3.5-9B + SETA, using the same SFT checkpoint, training tasks, hyperparameters, and 150-update budget. Each trained policy is evaluated on Terminal-Bench 2.0/2.1, with five attempts per task. Results are aggregated as mean ± sample SD across independently trained policies.

Table R7. Independent training-seed comparison

MethodLate Reward ↑Reward Degradation ↓TB2.0 Pass@1 (%) ↑TB2.1 Pass@1 (%) ↑
GRPO0.03 ± 0.010.50 ± 0.0511.19 ± 0.9010.91 ± 0.83
DepGPO0.53 ± 0.030.03 ± 0.0225.19 ± 0.8324.67 ± 1.16

In this three-seed comparison, DepGPO achieves a substantially higher late-stage reward (0.53 vs. 0.03) and much smaller reward degradation (0.03 vs. 0.50). Its average Pass@1 also exceeds GRPO by 14.00 and 13.76 percentage points on TB2.0 and TB2.1, respectively. These results would indicate that the improvements in both training stability and task performance persist across different training seeds. We will include the corresponding multi-seed experiments and additional stability analyses in the revised appendix.

AuthorsOpenReview ↗

Response to Reviewer GJDa (3/3)

W2 & Q2: Why do GiGPO and GraphGPO sometimes perform worse than GRPO?

The weaker performance of GiGPO and GraphGPO may arise from the difficulty of obtaining reliable step-level credit signals in terminal environments, where persistent file modifications and long-range command dependencies make state-based comparisons challenging.

  • GiGPO relies on recurring environment states for step-level comparisons. However, Figure 1(b) shows that only 4 of 25 anchor states recur across the two analyzed tasks with the most anchors. Importantly, GiGPO reduces to GRPO only when its step-level advantages are entirely zero. With sparse but nonzero anchor groups, it still combines trajectory-level advantages with additional local estimates. These estimates may be noisy due to limited comparisons and differing interaction histories, potentially interfering with the trajectory-level signal and explaining why GiGPO can perform worse than GRPO rather than simply matching it.
  • GraphGPO derives credit from state-transition graph structure, which may not adequately capture file-content dependencies and information flow between commands. Consequently, its structural estimates may assign importance to steps that are not closely connected to the verified outcome.

Thus, introducing step-level credit does not necessarily improve upon GRPO when the additional credit estimates are unreliable. DepGPO instead traces execution dependencies without requiring repeated states, providing more direct evidence for allocating the learning signal.

All methods share the same SFT checkpoint, training-task pool, and agent scaffold within each configuration, while retaining their respective algorithmic components. We will clarify these differences and the limitations of state-based credit assignment in the revised manuscript.


W3: Clarifications to Figures 1 and 2

We appreciate the reviewer’s suggestions and will improve the figures to make their intended meanings clearer.

Figure 1(a). This panel illustrates the zero-advantage problem in group-based methods such as GRPO, where all-success or all-failure groups provide no learning signal. It motivates our subsequent mixed-outcome task selection in Section 4.1 and Appendix B.2, rather than dependency-based credit assignment itself.

Figures 1(c)–(d). In Figure 1(c), “supports one” indicates that a command supports a downstream verifier-relevant operation. Figure 1(d) presents the cumulative distribution of the fraction of commands connected to verifier-inspected resources EE, where the y-axis indicates the proportion of trajectories at or below a given fraction. Across 51 passing reference trajectories, the median fraction is 0.57, dropping to 0.18 for the 18 trajectories with at least five commands. In contrast, GRPO assigns the same advantage to all steps, regardless of their connections to the verified outcome, as illustrated by the pink reference guides. These results highlight how uniform credit assignment can distribute learning signals across many commands without recorded dependencies to the final outcome, particularly in longer trajectories. These interpretations are discussed in the main text, and we will make them clearer in the figure labels and captions.

Figure 2. The left side of the GiGPO illustration shows its requirement for recurring states to construct step-level groups, while the right side highlights the difficulty of finding repeated states in terminal tasks, which limits such comparisons. For DepGPO, the success/failure illustrations emphasize how individual steps influence the final outcome, whereas the concrete example traces backward from the verifier to identify contributing commands. Both illustrate the same credit-assignment principle from different perspectives. We will unify the arrow directions and clarify the corresponding labels to avoid confusion.


We appreciate the reviewer's careful consideration of our work. We hope the clarifications, additional analyses, and controlled comparisons address the main concerns about credit validity, applicability, and reproducibility. We would be grateful if these responses could motivate a reassessment of the paper's overall rating.

Reviewer_UrEuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses credit assignment for terminal agents when only trajectory-level rewards are available. The authors propose Dependency-Aware Group Policy Optimization, which constructs command dependency graphs from file accesses and standard-output reuse. These graphs guide the redistribution of advantages across steps while retaining the GRPO optimization procedure. Experiments show that DepGPO outperforms the evaluated baselines on Terminal-Bench 2.0 and 2.1.

Critical strengths
  1. This paper addresses an important problem in reinforcement learning, where rewards are often available only at the end of a trajectory. Using execution traces to improve credit assignment is a reasonable and concrete approach.

  2. The proposed method achieves substantial performance improvements over the evaluated baselines across multiple experimental settings.

Critical weaknesses
  1. The proposed method primarily constructs dependencies through file reads and writes and the reuse of standard outputs, using manually defined rules. It remains unclear whether these rules can adequately capture dependencies in more complex terminal tasks involving other types of operations, such as service management, permission changes, or database interactions.

  2. Some design choices in the method require further justification. For example, the authors state: “We apply a conservative rule to indirect contributions: the consuming command must write to a file other than the input file and must not overwrite the input lines being counted.” It is unclear why these conditions should determine credit assignment, as modifying a file in place versus writing to a new file may reflect the agent’s operational preferences rather than differences in actual contribution to task completion.

Questions for authors
  1. The proposed method relies on manually defined rules for file reads and writes and standard-output reuse, which may limit its applicability to a narrow range of terminal workflows. How broadly can the method apply to tasks whose key dependencies fall outside these rules, such as service management, permission changes, or database interactions?

  2. The method assigns indirect credit only when the consuming command writes to a file other than the input file and does not overwrite the input lines being credited. Why are these two restrictions necessary? An agent may read intermediate content and either modify the original file in place or write the revised content to a new file. These choices may reflect different operational preferences, even when the original content plays a similar role in producing the final result. Why should the earlier write receive different credit in these cases?

4 comments
AuthorsOpenReview ↗

Response to Reviewer UrEu (1/2)

We thank the reviewer for recognizing the motivation and empirical improvements of DepGPO. We appreciate the constructive questions about the generality of execution dependencies and the design of our indirect-credit rules. We clarify these two aspects below.


W1 & Q1: Can DepGPO handle complex terminal workflows beyond file read–write operations?

Our experimental data sources already cover a diverse range of terminal workflows, including the types of tasks raised in this question. SETA includes system administration, DevOps, security, and networking tasks, while TMAX contains explicit examples involving service management, permission control, and database operations.

The following examples are drawn from the publicly released task collections used as our training data sources.

Task typeDatasetRepresentative task
Service managementTMAXConfiguring systemd services, managing service processes, and verifying network endpoints (task_000002_81d660f3)
Permission changesTMAXManaging service access permissions and ACL configurations (task_000016_6eb64181)
Database operationsTMAXQuerying and processing SQLite databases to generate verified reports (task_000012_58fdcccc)
System administration and securitySETASystem configuration, user management, security, and networking tasks across its released task categories

These examples demonstrate that the task distributions underlying our experiments are not limited to file editing, but also involve heterogeneous operations and verification requirements.

More importantly, DepGPO is not tied to specific task categories or commands. Different tasks naturally involve different intermediate operations and final verification criteria. Our approach applies the same credit-assignment procedure by identifying observable execution dependencies and tracing their connections to verifier-inspected resources.

For instance, service-management tasks may involve configuration files, processes, and network ports, while database tasks may involve database files, scripts, and generated outputs. The current implementation supports file read–write dependencies, standard-output reuse, and verifier-inspected file, port, and process resources. Thus, these workflows can share the same credit-assignment framework without requiring separately designed rules for each task type.

We also acknowledge that task coverage does not imply complete dependency coverage. Certain operations, such as permission-only modifications or internal database transactions, may not be fully observable under the current tracer. When no usable relative dependency credit is available, DepGPO falls back to the original GRPO advantage assignment rather than introducing unsupported attribution.

Therefore, the credit-assignment framework is not restricted to file editing; its applicability is determined by the execution evidence available for each workflow. To make the task diversity of our experimental settings more transparent, we will include a detailed breakdown of the task categories and their distributions in the SETA and TMAX datasets used in our experiments in the revised appendix.

AuthorsOpenReview ↗

Response to Reviewer UrEu (2/2)

W2 & Q2: Why does DepGPO restrict indirect credit for same-file edits and overwritten lines?

The two restrictions are intended to make indirect credit assignment conservative, rather than to favor one file-editing operation over another. Importantly, they apply only to the propagation of indirect credit, not to the credit assigned to in-place edits themselves.

DepGPO tracks file modifications throughout the trajectory, recording which command last wrote each tracked line. When a later command modifies the same file, the ownership of the modified lines is updated, while unchanged lines retain their original attribution. After the trajectory completes, credit is assigned according to the recorded dependencies and the resources inspected by the verifier.

For example, suppose one command creates a 100-line config.yaml, and a later command modifies 20 lines in place. If the verifier inspects the final file, the first command can still receive direct credit for its 80 surviving lines, while the later command receives direct credit for the 20 modified lines. Thus, DepGPO naturally handles successive edits to the same file without discarding earlier contributions.

The additional restrictions concern whether an earlier write should also receive indirect credit through a subsequent write. Our implementation tracks file reads at file level and writes at line level, so a recorded read does not always establish which individual lines were used.

We therefore adopt two conservative conditions:

  1. Different output file: A later command must write to a file other than the input file. This provides a clearer record of information flowing from an intermediate file to a downstream output, rather than merely recording successive versions of the same file.

  2. No overwrite of credited input lines: The later command must not overwrite the earlier lines being credited. Otherwise, we cannot reliably distinguish whether those lines contributed to the new output or were simply replaced.

These conditions do not imply that overwritten content is necessarily irrelevant. Rather, we avoid propagating credit when the available trace cannot reliably distinguish content reuse from replacement. This may exclude some genuine dependencies, but reduces the risk of assigning credit to discarded intermediate content.

To examine the practical effect of these choices, we introduce a Relaxed Indirect Contribution variant. It retains the original dependency graph, verifier-relevant endpoints, and direct-credit computation, but removes both additional restrictions on indirect attribution. Specifically, an earlier write becomes eligible for indirect credit whenever a later write command reads its file while the earlier command still owns the corresponding tracked records, even if the later command writes back to the same file or overwrites those records. The later command must still have a qualifying dependency path to a verifier-inspected resource. All remaining components, including supporting-read credit, step-factor normalization, and the policy objective, remain unchanged.

We compare this variant with Full DepGPO and the existing Direct Writes Only ablation under the same Qwen3.5-9B + SETA configuration.

Table R4. Effect of conservative indirect-credit restrictions

MethodTB2.0 Pass@1 (%)TB2.1 Pass@1 (%)
Direct Writes Only23.45 ± 0.9123.00 ± 1.04
Relaxed Indirect Contribution24.79 ± 1.1324.27 ± 1.12
Full DepGPO25.24 ± 1.0124.64 ± 2.36

The Direct Writes Only ablation demonstrates the importance of capturing intermediate contributions beyond writes that persist directly in verifier-inspected resources. The relaxed variant performs close to Full DepGPO, with differences of only 0.45 and 0.37 percentage points. This suggests that the conservative restrictions have a relatively modest effect on overall performance, while providing a clearer criterion for avoiding uncertain indirect attribution.

In summary, these restrictions do not imply that in-place edits are less valuable than writing to a new file. Both operations can contribute equally to task completion, and DepGPO assigns direct credit to their resulting writes when they remain relevant to verification. The restrictions apply only to additional indirect credit for earlier writes: cross-file propagation without overwriting provides clearer evidence of their continued use, whereas same-file replacement may simply discard earlier content. Thus, the difference in credit reflects the reliability of the observed dependency, rather than the intrinsic importance of the operation.


We appreciate the reviewer's constructive feedback and hope these clarifications and additional comparisons address the main concerns. If our responses resolve your concerns, we would be grateful if you could reconsider your assessment and rating.

Thank you for your response. While it addresses some of my concerns, I will maintain my original score.

AuthorsOpenReview ↗

Thank You for Your Careful Review and Engagement in the Discussion

We thank the reviewer for the follow-up and respect your assessment. However, we would appreciate more specific feedback on which concerns remain unresolved.

We have directly addressed both original concerns through additional task-category analysis (Appendix B.7), clarification of the indirect-credit rules (Appendix A.3), and a new controlled experiment (Appendix G.2).

Could the reviewer clarify which aspects of these responses remain insufficient and why? This would help us address any remaining technical concerns and ensure that the additional evidence is properly considered in the final assessment score.

Reviewer_ddEbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a dependency aware group policy optimization (DepGPO), which is a credit assignment method that uses agents for terminal reinforcement learning. The author discovered the limitations of existing trajectory level and step level methods: they do not explicitly capture the read-write dependencies of terminal commands on task results and propose DepGPO to address this issue. Experimental results have shown consistent improvement across multiple reinforcement learning baselines. The ablation study and training analysis further investigated the effectiveness of the proposed components and their impact on optimizing stability.

Critical strengths

S1. Well-motivated and meaningful contribution. The paper identifies the specific limitations of existing credit allocation methods in terminal agents and proposes a practical method for reallocating learning signals using observable execution dependencies.
S2. Strong empirical results. DepGPO has achieved consistent improvements across multiple model and dataset configurations. The ablation supports the effectiveness of its main components, while the additional computational overhead remains moderate.

Critical weaknesses

W1. It’s difficult to distinguish whether the experimental effect comes from credit sparsity. Although ablation studies provide useful evidence for the effectiveness of individual components, they do not fully determine whether these improvements depend specifically on assigning dependency credits to the corresponding steps. There is an ambiguity that even if the weights and command dependencies are not accurately aligned, sparse or non-uniform advantage redistribution itself may improve optimization. Randomly arranging the weights of each step within each trajectory and reporting the experimental results under this setting will help clarify this ambiguity and provide more direct evidence for the central motivation.
W2. The generality of the training stability conclusions remains limited. The paper presents a series of findings indicating that DepGPO can alleviate post training degradation. However, detailed stability analysis is mainly based on a model and dataset setup, which may result in conclusions that do not have high generalizability. Therefore, the evidence supports improving stability in the current setting, but a broader claim about continuous stable optimization during training runs will benefit from further experimental evidence.

Questions for authors

Q1. Can experiments be conducted to randomly arrange or sparsify the step weights obtained from the dependency relationships in each trajectory? This comparison can help clarify whether performance improvement specifically comes from the correspondence between execution dependencies and assigned credits, rather than solely from uneven or sparse advantage weights.
Q2. Is the post stability improvement consistent across different models (data configuration or independent training run)? Clarifying the scope of this observation will help evaluate the generality of the stability statement.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ddEb (1/2)

We thank the reviewer for recognizing the motivation and empirical contributions of our work. We address the two concerns regarding dependency-aligned credit assignment and training stability through additional controlled experiments and analyses.


W1 & Q1: Are the improvements due to dependency-aligned credit assignment or merely sparse and non-uniform advantage weights?

We thank the reviewer for this insightful question. The key concern is whether DepGPO benefits from assigning credit to dependency-relevant steps, or simply from introducing sparse and non-uniform advantage weights.

To distinguish these two effects, we introduce a Randomly Permuted DepGPO baseline. We choose this control because it directly isolates dependency alignment while preserving the other properties of DepGPO's credit redistribution.

Specifically, we first compute the original dependency-based step factors and randomly permute them among the traced steps within each trajectory. This preserves the factor values, sparsity, and mean, but breaks their correspondence with the recorded execution dependencies. Untraced steps retain their default factors, and token-level normalization is recomputed after permutation. All other settings, including the SFT checkpoint, training tasks, optimization objective, and training budget, remain identical to Full DepGPO.

Table R1. Effect of dependency alignment

MethodTB2.0 Pass@1 (%)TB2.1 Pass@1 (%)
GRPO11.09 ± 0.2611.01 ± 0.67
GiGPO11.24 ± 0.3911.39 ± 1.04
Randomly Permuted DepGPO13.03 ± 0.9012.96 ± 1.01
Full DepGPO25.24 ± 1.0124.64 ± 2.36

Randomly Permuted DepGPO achieves modest improvements over GRPO and GiGPO, suggesting that non-uniform advantage redistribution can provide some benefit even without accurate dependency alignment.

However, Full DepGPO substantially outperforms the randomized variant by 12.21 and 11.68 percentage points on Terminal-Bench 2.0 and 2.1, respectively. Since the two variants preserve the same step-factor distribution within each trajectory, this gap highlights the importance of assigning larger weights to steps identified by execution dependencies, rather than simply making the learning signal sparse.

This comparison complements the component ablations in Table 2, which establish the contributions of verifier filtering, graded write credit, and supporting-read credit. The new permutation control further isolates the importance of aligning these credits with the corresponding interaction steps.

Taken together, these results support our central motivation: DepGPO benefits not merely from redistributing trajectory advantages, but from using execution dependencies to determine where the learning signal should be assigned.

AuthorsOpenReview ↗

Response to Reviewer ddEb (2/2)

W2 & Q2: Does DepGPO's improved training stability generalize across different models, training data, and independent runs?

Our stability analysis extends beyond the Qwen3.5-9B + SETA setting presented in Figure 5. We examined training dynamics across all four model–data configurations, covering Qwen3.5-9B and Qwen3.6-27B trained on SETA and TMAX. The main paper presents one representative configuration due to space constraints.

To evaluate stability more systematically, we consider three complementary diagnostics over the full 150-update training process:

  • Late-stage reward (↑): Average training reward over the final 20% of updates (121–150).
  • Reward degradation (↓): Difference between the highest 30-update rolling-average reward over the entire training run and the final 30-update average.
  • Late-stage gradient norm: Average gradient norm over updates 121–150, providing complementary information about ongoing policy updates.

These metrics jointly characterize sustained training performance, degradation from the best training stage, and optimization behavior, rather than relying on the final reward alone.

Table R2. Training stability across model–data configurations

ConfigurationMethodLate reward ↑Reward degradation ↓Late gradient norm
9B / SETAGRPO0.020.520.01
GiGPO0.040.170.01
DepGPO0.540.000.58
9B / TMAXGRPO0.000.740.00
GiGPO0.030.390.02
DepGPO0.530.180.35
27B / SETAGRPO0.010.571.12
GiGPO0.020.450.10
DepGPO0.580.130.41
27B / TMAXGRPO0.000.630.00
GiGPO0.020.390.02
DepGPO0.810.010.23

The recorded GRPO–DepGPO comparisons show a consistent improvement across all four configurations. DepGPO achieves higher late-stage rewards and substantially smaller reward degradation, indicating that it better preserves training performance as optimization continues. These results support the generality of the observed stability improvements across model scales and training sources.

The gradient statistics provide additional context. In several settings, GRPO exhibits near-zero late-stage gradients alongside severe reward degradation. However, the 27B/SETA setting also shows that nonzero gradients do not necessarily imply effective learning: GRPO retains a relatively large gradient norm despite poor late-stage rewards. We therefore assess stability through reward preservation and optimization behavior together, rather than gradient magnitude alone.


To further examine reproducibility across independent runs, we also consider a controlled comparison using three RL training seeds under Qwen3.5-9B + SETA. Each run starts from the same SFT checkpoint and uses identical training tasks, optimization settings, and 150-update budget. This experiment evaluates both late-stage training stability and final benchmark performance.

Table R3. Stability and performance across independent RL seeds

MethodLate reward ↑Reward degradation ↓TB2.0 Pass@1 (%) ↑TB2.1 Pass@1 (%) ↑
GRPO0.03 ± 0.010.50 ± 0.0511.19 ± 0.9010.91 ± 0.83
DepGPO0.53 ± 0.030.03 ± 0.0225.19 ± 0.8324.67 ± 1.16

Mean ± SD is computed across independently trained policies. Each policy follows the original benchmark evaluation protocol; these SDs are distinct from the evaluation-round SDs reported in Table 1.

The independent-run comparison is intended to test whether the observed reduction in reward degradation and the final benchmark improvements persist across different training seeds. Together with the cross-configuration analysis, it provides a complementary assessment of generality and run-to-run reproducibility.

We will include the additional training curves in the revised Appendix D to make these comparisons more transparent.


We thank the reviewer again for these suggestions. The additional comparisons directly examine the importance of dependency alignment and extend the evidence for training stability, strengthening the two central claims of DepGPO.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for overseeing our submission and the three reviewers (ddEb, UrEu, and GJDa) for their careful assessments and constructive feedback. During the rebuttal, we provided detailed responses, additional experiments, and manuscript revisions, with all changes highlighted in blue. Below, we summarize the main concerns and how we have addressed them.

1. Dependency-Aligned Credit Assignment and Attribution Validity · ddEb, GJDa

Concern. Do the improvements arise from dependency-aligned credit rather than generic weighting or recency? How reliable is attribution in failed trajectories?

  • Controlled comparisons. Full DepGPO achieves 25.24%/24.64% on TB2.0/2.1, outperforming Recency Weighting (14.61%/14.91%) and Randomly Permuted DepGPO (13.03%/12.96%). The latter preserves the step-factor distribution and sparsity while breaking dependency alignment, directly supporting the importance of assigning credit to the corresponding steps.
  • Attribution reliability. We clarified stdout matching rules and their limitations. Our failed-trajectory analysis further illustrates how DepGPO concentrates negative learning signals on verifier-relevant operations rather than uniformly penalizing unrelated steps, without assuming independent step correctness.

Supporting evidence: Appendices A.2–A.4, F.1, and G.1.

2. Training Stability and Reproducibility · ddEb, GJDa

Concern. Are the stability improvements consistent across model configurations and independent training seeds?

  • Cross-setting stability. Across all four model–data configurations (9B/27B × SETA/TMAX), DepGPO maintains higher late-stage rewards and smaller reward degradation than GRPO.
  • Independent training seeds. Across three independent RL runs, DepGPO achieves 25.19 ± 0.83% / 24.67 ± 1.16% Pass@1, compared with 11.19 ± 0.90% / 10.91 ± 0.83% for GRPO. Reward degradation also decreases from 0.50 to 0.03. We explicitly distinguish training-seed variability from repeated checkpoint evaluations.

Supporting evidence: Appendices B.5 and D.6–D.7.

3. Task Coverage and Dependency Construction · UrEu, GJDa

Concern. Does DepGPO apply to diverse terminal workflows, and are its indirect-credit restrictions and verifier-access assumptions justified?

  • Task coverage. The existing SETA and TMAX datasets already include database operations, permissions/security, and service/process management tasks. We further analyzed the category distributions of tasks actually encountered during RL training across all four configurations, providing quantitative evidence of their coverage.
  • Dependency rules and verifier access. We clarified that surviving in-place file modifications can receive direct credit, while conservative indirect-credit restrictions reduce uncertain attribution. The relaxed variant achieves 24.79%/24.27%, compared with 25.24%/24.64% for Full DepGPO. We also clarified that training requires observable verifier resource accesses, not access to its internal logic.

Supporting evidence: Appendices A.3, B.6–B.7, and G.2.

4. Baseline Behavior and Presentation Clarity · GJDa

Concern. Why can GiGPO and GraphGPO underperform GRPO, and how should the pilot analyses and illustrations be interpreted?

  • Baseline analysis. We clarified how sparse recurring states may introduce noisy local advantage estimates in GiGPO, while state-transition graphs may not fully capture the long-range file dependencies important to terminal tasks.
  • Figure clarifications. We revised Figures 1 and 2 to clarify the role of mixed-outcome task selection, the interpretation of dependency statistics, and the differences between state-based grouping and dependency-based credit assignment.

Supporting evidence: Figures 1–2, Section 4.1, and Appendices B.2 and D.2.


We hope the additional experiments and clarifications provide stronger evidence for the effectiveness of dependency-aware credit assignment and address the key questions raised during the review. We hope the revised manuscript will help the reviewers further assess the contributions and limitations of our work.

With the rebuttal period nearing its end, we would be glad to clarify any remaining questions or concerns from the reviewers. We greatly appreciate the time and attention devoted to evaluating our work.

Best regards,
The Authors

Dear Reviewers,

The authors have posted their rebuttals. As the author-reviewer discussion period is coming to a close soon, please read their responses and engage in the discussion. Thank you for your time and effort.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.