- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies credit assignment for multi-turn LLM agents. The authors argue that final trajectory rewards can give misleading supervision to intermediate decisions. For example, a failed trajectory may still contain useful actions, and a successful trajectory may contain unnecessary or harmful actions.
The paper proposes Dynamic Causal Tree (DCT), which compares different actions under the same interaction history by sampling alternative continuations. It also separates causal relevance from the credit used for policy optimization.
Experiments on multi-hop QA show improvements across several model sizes. For Qwen2.5-3B, DCT achieves 41.0% EM, compared with 36.8% for Tree-GRPO and 31.8% for GRPO.
Critical strengths
-
I think the paper gives a useful way to understand the credit assignment problem in agentic RL. The distinction between terminal outcome and decision-level contribution is important, especially for long trajectories. I also find the separation between causal relevance and optimization credit interesting, since these two things are often mixed together.
-
The experiments are reasonably convincing. Apart from the final QA performance, the authors also evaluate the quality of decision attribution using held-out interventions. The difference between replaceable and protected decisions in Table 2 is good evidence that the method learns some useful information about which actions are important. The credit ablations are also helpful.
Critical weaknesses
-
One concern is the gap between the theoretical causal definition and the actual estimation. Causal relevance is defined over feasible alternative actions, but DCT only samples a small number of branches. With a limited tree budget, some important alternatives may never be tested. The theoretical bound in Proposition 1 relates estimation error to policy update error, but does not directly guarantee that DCT will obtain accurate credit estimates. I think this limitation should be more clearly discussed.
-
The experimental validation is mainly limited to retrieval-based QA. In this setting, it is relatively easy to replay the same history and sample another continuation. I am less sure whether the method is practical for GUI agents or coding agents, where resetting an environment and executing counterfactual actions can be expensive. Although rollout budgets are matched, a comparison of wall-clock training cost would help understand the practical trade-off.
Questions for authors
-
How sensitive is DCT to the number of alternative actions sampled at each history? Can the authors provide some analysis of the stability or variance of the estimated causal credit under small rollout budgets?
-
Have the authors considered environments where the exact interaction state cannot be easily restored? How much additional computational cost would be introduced by DCT in such cases?
-
DCT does not consistently outperform Tree-GRPO on Single-Hop QA. Can the authors further explain under what task conditions causal tree construction is beneficial, and when it may introduce unnecessary exploration?
2 comments
Thank you for your thoughtful comments and the time you've dedicated to reviewing our work. Our responses discuss the gap between population definitions and finite-budget estimation, provide a new sampling-budget and credit-stability analysis, and clarify the applicability and limitations of DCT beyond retrieval-based QA.
As the discussion period is nearing its end, we would be grateful if you have time to review our response and share any further feedback. We would be happy to clarify any remaining concerns.
We thank the reviewer for the detailed comments.
R1: We view this as a trade-off between estimation accuracy and rollout cost. To increase opportunities for discovering useful low-probability actions, DCT adds same-history trials through shared-prefix branching. Uniform history exploration also keeps low-relevance histories eligible for further comparisons (Section 5.2). These mechanisms mitigate missed-comparison risks, although coverage still depends on policy support and budget.
For sampled alternatives, DCT aggregates continuation returns for value and credit estimation (Eqs. 12–13). Appendix B.4 bounds the remaining estimation error under its stated conditions, and Proposition 1 relates this error to expected-update deviation. The audit in Q1 and the fixed-tree credit ablations (Table 5) support the practical usefulness of this finite-budget approximation.
R2: The table below evaluates alternative coverage and credit stability across control-sampling budgets J. Avg. Alt. counts distinct controls based on normalized action content and reasoning, with invalid outputs distinguished by token identity; Missing measures the fraction without distinct controls. Mean Std. and Mean Var. average credit-estimate variability over five independent repetitions per (h,a), excluding repetitions without distinct controls. Ref. RMSE uses an independent reference with 32 target continuations and 32 control draws per pair.
Even at J=2, DCT obtains 1.96 distinct controls on average, with only 1.25% missing comparisons. Increasing J to 8 reduces mean credit variance by 50.6% and reference RMSE by 24.9%. These results indicate that small budgets already provide broad alternative coverage, while additional controls improve credit estimation stability.
R3: We have considered this issue. Compared with retrieval-based QA, where interaction histories are relatively easy to reconstruct, GUI and coding agents involve more complex external states, making state restoration and counterfactual execution potentially more expensive. Recent studies provide practical support for addressing these challenges.
TreeCUA [a] demonstrates replay-based state reconstruction in GUI agents, while CRR [b] investigates counterfactual credit assignment and its computational trade-offs in coding agents. The table below reports CRR increases training cost from 188 to 321 H100-hours while improving pass@1 from 36.4% to 41.7%.
Notably, CRR retains a 5.0-point improvement over GRPO under equal wall-clock time. DCT similarly adopts a counterfactual approach to decision-level credit assignment. Together, these studies provide practical support for extending DCT to GUI and coding agents. In future work, we plan to explore DCT in these environments, considering both causal credit effectiveness and computational trade-offs.
[a] Jiang et al. "TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution." Proceedings of International Conference on Machine Learning. 2026.
[b] Li et al. "Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agents." Proceedings of Advances in Neural Information Processing Systems. 2026.
R3: Tree-GRPO uses intra- and inter-tree group-relative advantages to leverage terminal rewards. DCT instead employs same-history causal comparisons to uncover decision-level contributions that may be obscured by subsequent outcomes.
In Single-Hop QA, success often depends on a single effective retrieval, making terminal feedback relatively informative and leaving less room for additional causal refinement. Nevertheless, DCT remains competitive across model scales.
As shown in Fig. 5(b) of our paper, DCT outperforms Tree-GRPO by 2.2, 4.6, and 10.6 EM points on questions requiring two, three, and four actions, respectively. These results highlight DCT's advantages in multi-step tasks, where interdependent decisions create greater credit ambiguity and more opportunities for causal exploration.