acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Causal Tree: Interventional Credit Assignment for LLM Agent

Abstract

Reinforcement learning improves multi-turn decision making in LLM agents, but trajectory-level rewards obscure the contribution of individual decisions. Failed trajectories can suppress credit for useful decisions, while successful ones can reinforce harmful decisions. We formalize this mismatch through same-history interventions, separate causal relevance from policy-update credit, and bound how credit error distorts the expected policy update. We propose Dynamic Causal Tree (DCT), which selectively constructs shared-prefix branches to approximate exhaustive intervention comparisons. The resulting tree supports a dual readout: pairwise branch contrasts estimate causal relevance, while policy-weighted aggregation yields signed credit. Experiments on multi-hop QA across model scales show consistent gains over trajectory-level and tree-based RL baselines, while independent intervention tests support more faithful decision attribution.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Credit Assignment, Causal Inference.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dFYxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies credit assignment for multi-turn LLM agents. The authors argue that final trajectory rewards can give misleading supervision to intermediate decisions. For example, a failed trajectory may still contain useful actions, and a successful trajectory may contain unnecessary or harmful actions.

The paper proposes Dynamic Causal Tree (DCT), which compares different actions under the same interaction history by sampling alternative continuations. It also separates causal relevance from the credit used for policy optimization.

Experiments on multi-hop QA show improvements across several model sizes. For Qwen2.5-3B, DCT achieves 41.0% EM, compared with 36.8% for Tree-GRPO and 31.8% for GRPO.

Critical strengths
  1. I think the paper gives a useful way to understand the credit assignment problem in agentic RL. The distinction between terminal outcome and decision-level contribution is important, especially for long trajectories. I also find the separation between causal relevance and optimization credit interesting, since these two things are often mixed together.

  2. The experiments are reasonably convincing. Apart from the final QA performance, the authors also evaluate the quality of decision attribution using held-out interventions. The difference between replaceable and protected decisions in Table 2 is good evidence that the method learns some useful information about which actions are important. The credit ablations are also helpful.

Critical weaknesses
  1. One concern is the gap between the theoretical causal definition and the actual estimation. Causal relevance is defined over feasible alternative actions, but DCT only samples a small number of branches. With a limited tree budget, some important alternatives may never be tested. The theoretical bound in Proposition 1 relates estimation error to policy update error, but does not directly guarantee that DCT will obtain accurate credit estimates. I think this limitation should be more clearly discussed.

  2. The experimental validation is mainly limited to retrieval-based QA. In this setting, it is relatively easy to replay the same history and sample another continuation. I am less sure whether the method is practical for GUI agents or coding agents, where resetting an environment and executing counterfactual actions can be expensive. Although rollout budgets are matched, a comparison of wall-clock training cost would help understand the practical trade-off.

Questions for authors
  1. How sensitive is DCT to the number of alternative actions sampled at each history? Can the authors provide some analysis of the stability or variance of the estimated causal credit under small rollout budgets?

  2. Have the authors considered environments where the exact interaction state cannot be easily restored? How much additional computational cost would be introduced by DCT in such cases?

  3. DCT does not consistently outperform Tree-GRPO on Single-Hop QA. Can the authors further explain under what task conditions causal tree construction is beneficial, and when it may introduce unnecessary exploration?

2 comments

We thank the reviewer for the detailed comments.

W1: Gap Between the Theoretical Causal Definition and Actual Estimation.

R1: We view this as a trade-off between estimation accuracy and rollout cost. To increase opportunities for discovering useful low-probability actions, DCT adds same-history trials through shared-prefix branching. Uniform history exploration also keeps low-relevance histories eligible for further comparisons (Section 5.2). These mechanisms mitigate missed-comparison risks, although coverage still depends on policy support and budget.

For sampled alternatives, DCT aggregates continuation returns for value and credit estimation (Eqs. 12–13). Appendix B.4 bounds the remaining estimation error under its stated conditions, and Proposition 1 relates this error to expected-update deviation. The audit in Q1 and the fixed-tree credit ablations (Table 5) support the practical usefulness of this finite-budget approximation.

Q1: Sensitivity to Alternative-Action Sampling Budgets and Credit Stability.

R2: The table below evaluates alternative coverage and credit stability across control-sampling budgets JJ. Avg. Alt. counts distinct controls based on normalized action content and reasoning, with invalid outputs distinguished by token identity; Missing measures the fraction without distinct controls. Mean Std. and Mean Var. average credit-estimate variability over five independent repetitions per (h,a)(h,a), excluding repetitions without distinct controls. Ref. RMSE uses an independent reference with 32 target continuations and 32 control draws per pair.

Even at J=2J=2, DCT obtains 1.96 distinct controls on average, with only 1.25% missing comparisons. Increasing JJ to 8 reduces mean credit variance by 50.6% and reference RMSE by 24.9%. These results indicate that small budgets already provide broad alternative coverage, while additional controls improve credit estimation stability.

JJAvg. Alt.Mean Std. ↓Mean Var. ↓Ref. RMSE ↓Missing (%) ↓
21.960.19430.083650.36431.25
43.940.16470.046810.31370.00
87.890.14360.041340.27360.00

W2 & Q2: Applicability and computational cost when GUI or coding states are difficult to restore.

R3: We have considered this issue. Compared with retrieval-based QA, where interaction histories are relatively easy to reconstruct, GUI and coding agents involve more complex external states, making state restoration and counterfactual execution potentially more expensive. Recent studies provide practical support for addressing these challenges.

TreeCUA [a] demonstrates replay-based state reconstruction in GUI agents, while CRR [b] investigates counterfactual credit assignment and its computational trade-offs in coding agents. The table below reports CRR increases training cost from 188 to 321 H100-hours while improving pass@1 from 36.4% to 41.7%.

MetricGRPOCRR
H100-hours188321
Wall-clock (h)23.540.1
Pass@1 (%)36.441.7

Notably, CRR retains a 5.0-point improvement over GRPO under equal wall-clock time. DCT similarly adopts a counterfactual approach to decision-level credit assignment. Together, these studies provide practical support for extending DCT to GUI and coding agents. In future work, we plan to explore DCT in these environments, considering both causal credit effectiveness and computational trade-offs.

[a] Jiang et al. "TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution." Proceedings of International Conference on Machine Learning. 2026.

[b] Li et al. "Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agents." Proceedings of Advances in Neural Information Processing Systems. 2026.

Q3: DCT does not consistently outperform Tree-GRPO on Single-Hop QA. Can the authors further explain under what task conditions causal tree construction is beneficial, and when it may introduce unnecessary exploration?

R3: Tree-GRPO uses intra- and inter-tree group-relative advantages to leverage terminal rewards. DCT instead employs same-history causal comparisons to uncover decision-level contributions that may be obscured by subsequent outcomes.

In Single-Hop QA, success often depends on a single effective retrieval, making terminal feedback relatively informative and leaving less room for additional causal refinement. Nevertheless, DCT remains competitive across model scales.

As shown in Fig. 5(b) of our paper, DCT outperforms Tree-GRPO by 2.2, 4.6, and 10.6 EM points on questions requiring two, three, and four actions, respectively. These results highlight DCT's advantages in multi-step tasks, where interdependent decisions create greater credit ambiguity and more opportunities for causal exploration.

Thank you for your thoughtful comments and the time you've dedicated to reviewing our work. Our responses discuss the gap between population definitions and finite-budget estimation, provide a new sampling-budget and credit-stability analysis, and clarify the applicability and limitations of DCT beyond retrieval-based QA.

As the discussion period is nearing its end, we would be grateful if you have time to review our response and share any further feedback. We would be happy to clarify any remaining concerns.

Reviewer_H2bCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Causal Tree for credit assignment in multi-turn LLM agents. It compares alternative actions at the same history, uses their outcomes to estimate decision-level credit, and adaptively allocates further rollouts. Experiments show improvements over Tree-GRPO on multi-hop QA, without using tree search at inference time.

Critical strengths
  1. The paper makes a useful distinction between whether a decision changes the outcome and whether its credit gives the correct policy update. Keeping counterfactual branches as evidence while updating only the original anchors is also a sensible design.
  2. The multi-hop results are consistent across several model sizes and two model families. The random-expansion and fixed-tree credit ablations help explain which parts of the method matter.
Critical weaknesses
  1. Under the stated assumptions, population signed credit is Q−V. Appendix B.4 distinguishes fixed-design estimation from adaptive sampling and fallback, which may introduce bias.
  2. Tree-GRPO, random-expansion and fixed-tree credit controls already provide useful evidence. The remaining gap is a direct comparison with ordinary same-history Monte Carlo advantage estimation under matched actual costs. This would help separate credit construction from adaptive rollout allocation.
  3. The paper matches nominal continuation budgets, but these need not match actual rollouts, generated tokens or training time. Independent training-seed variability is also not clear.
Questions for authors
  1. Can you compare DCT with fixed-budget same-history Monte Carlo advantage estimation, keeping anchor batches and actual continuation costs matched?
  2. Can you report actual continuation counts, generated-token costs, wall-clock time, training-seed variation, and the numerical optimization and search settings needed to reproduce the results?
2 comments

We thank the reviewer for the detailed comments.

W1: Under the stated assumptions, population signed credit is Q−VQ-V. Appendix B.4 distinguishes fixed-design estimation from adaptive sampling and fallback, which may introduce bias.

R1: We acknowledge that adaptive sampling and fallback can introduce bias. As with biased advantage estimation, the approximation should be assessed through both its update error and practical benefit. Appendix B.4 analyzes the estimation error under its stated conditions, and Proposition 1 bounds its effect on the expected update. Importantly, with intervention trees and budgets fixed, DCT credit improves average EM from 37.6 for trajectory-level credit to 40.7 (Table 5). This supports the practical value of the approximation within the reported settings.

W2 & Q1: Comparison with ordinary fixed-budget same-history Monte Carlo advantage estimation under matched actual costs.

R2: We compare DCT with MC (Fixed-budget), a same-history Monte Carlo advantage estimator, under nominal rollout budgets of 4 and 8 equivalent full trajectories per prompt.

The table below reports that DCT achieves 37.6 average EM at budget 4, compared with 34.9 for MC, a gain of 2.7 points. At budget 8, DCT reaches 41.0 EM versus 36.6 for MC, widening the gap to 4.4 points. These results suggest that DCT's targeted causal exploration and backward necessity testing yield more informative decision-level credit than fixed-budget Monte Carlo advantage estimation, with greater benefits under larger rollout budgets.

BudgetMethodHotpot2WikiMuSiQBamb.Avg.
4MC39.241.317.142.134.9
4DCT (Ours)43.744.916.745.137.6
8MC41.142.519.543.436.6
8DCT (Ours)46.948.819.249.141.0

W3 & Q2: Actual continuation counts, generated-token costs, wall-clock time, training-seed variation, and numerical configurations.

R3.1: Actual Computational Cost.

The table below reports training costs for 180 updates on eight A800 GPUs. Compared with Tree-GRPO, DCT requires 23.1% more generated tokens and 56.7% longer training time, while improving average Multi-Hop EM from 36.8 to 41.0. This additional cost arises from targeted causal exploration and backward necessity testing, which provide more informative decision-level credit during training.

MethodTokens (M)Time (h)GPU-hours
Tree-GRPO66.3216.03128.25
DCT (Ours)81.6725.13201.00

R3.2: Training-Seed Variation.

We evaluate DCT with three independent training seeds (42, 43, 44). As shown in Table 5, DCT achieves 48.8±0.248.8\pm0.2 average EM on Single-Hop QA and 41.0±0.641.0\pm0.6 on Multi-Hop QA. The relatively small variations across seeds suggest that DCT's performance is stable across independent training runs.

MethodNQTriviaPopQASingle-Hop Avg.Hotpot2WikiMuSiQBamb.Multi-Hop Avg.
DCT (Ours)49.2±0.249.2_{\pm0.2}56.7±0.356.7_{\pm0.3}40.6±0.340.6_{\pm0.3}48.8±0.248.8_{\pm0.2}46.9±0.546.9_{\pm0.5}48.8±0.648.8_{\pm0.6}19.2±0.419.2_{\pm0.4}49.1±0.849.1_{\pm0.8}41.0±0.641.0_{\pm0.6}

R3.3: Numerical Configurations.

We summarize our experimental configurations, including optimization hyperparameters, rollout allocation, tree expansion, and credit estimation settings. The same configuration is used for both Single-Hop and Multi-Hop QA.

ConfigurationValue
Learning rate1×10−61\times10^{-6}
Training batch size512 prompts
Training updates180
PPO clipping coefficient0.20.2
KL coefficient0.0010.001
Sampling temperature1.01.0
Top-pp0.950.95
Maximum interaction rounds33
Maximum generated tokens per action40964096
Anchor trees (MM)22
Continuation budget per anchor (BB)33
Maximum trajectories per question88
Forward expansion budget (BfwdB_{\mathrm{fwd}})3
Exploration coefficient (ε\varepsilon)0.20.2

Thank you for your thoughtful comments and the time you've dedicated to reviewing our work. Our responses clarify the estimation-bias analysis and provide additional comparisons with fixed-budget Monte Carlo estimation, computational costs, training-seed results, and implementation settings.

As the discussion period is nearing its end, we would be grateful if you have time to review our response and share any further feedback. We would be happy to clarify any remaining concerns.

Reviewer_B8doOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Causal Tree, a method for improving credit assignment in multi-turn reinforcement learning training. It works by constructing shared-prefix trees by sampling alternative actions from selected interaction histories and evaluating their downstream outcomes under a fixed policy. It uses these comparisons to estimate causal relevance and decision credit, along with adaptive tree expansion and reference-path refinement to select informative interventions. Experiments across five model backbones show improvements over other RL baselines on multi-hop tasks.

Critical strengths
  • Comparing alternative actions from the same interaction history is a reasonable way to obtain more informative feedback about individual decisions in multi-turn RL, and the paper shows this approach is more beneficial on multi-hop QA than on simpler single-hop tasks.
Critical weaknesses
  • the paper is unusually difficult to follow (the writing is heavily polished by text tools), especially the intro and sec 4 and 5, many key concepts are introduced via highly condensed statements with no context or intuitive explanation, or clear connections between them. Thus, a huge effort is required to reconstruct what problem is to be addressed and how and why the proposed components should work. Some concrete examples:

In abs and intro, terms such as "same-history interventions," "shared-prefix branches," "causal relevance," "optimization consistency," and "exhaustive intervention comparisons" appear before explaining their meaning and relevance. E.g., the statement "These are attribution failures rather than properties of trajectory length" is difficult to understand as there is no explanation of how the preceding discussion relates to attribution rather than trajectory length. Similarly, the sentence beginning "These signals improve optimization, but finer granularity and better outcome prediction do not establish causal attribution..." has no context for interpreting what it is talking about (causal attribution). Same for the two important concepts: causal relevance and optimization consistency

the sec 3 often introduces several related symbols without clearly distinguishing their roles. for instance, the latent state (S_t) vs the V = (S,O,H,A,R) in Causal representation. The notation involving actions is also difficult to follow, with (A_t) representing an action and (A_t^\pi) an advantage.

sec 4.1 and 4.2 are particularly difficult to read because many claims are introduced without first explaining their implications. E.g., For example, the statement "The terminal-reward estimator remains unbiased, whereas estimated decision credit may distort the expected update" is not sufficiently explained at that point. How does it motivate Proposition 1? also what is this about "History-dependent baselines can loosen the bound without changing the update; causal relevance therefore requires a separate statistic."?

sec 4.2 has many concepts such as "causal relevance," "optimization consistency," "relevance faithfulness," and "zero sets," but the discussion moves quickly between them with no explanation of their relationships.Also, can the authors clarify this "Causal credit mechanisms should preserve relevance evidence and yield valid update weights. These population-level requirements are ideal targets for finite estimators using sampled interventions."? it reads quite abstract.

sec 5 includes "anchor decisions," "anchor trajectories," "successful references," and "reference refinement", which is again difficult to follow. The connections between forward discovery, backward testing, and tree-wide credit estimation are also not explained.

important metrics in sec 6.3 like Alternative Coverage, Recovery Rate, CF-EM, PNS, UTR, and RDR, are used to interpret results but with no definitions in the main text. authors should explain what these metrics measure and why they matter in the main text, not the appendix, otherwise the analysis is very hard to appreciate.

i think the underlying idea of the paper is simpler than the language (which is hard to read) used to describe it: the proposed DCT samples alternative actions at selected turns and compare their downstream outcomes, and uses estimated credit to train on the original trajectories. but the paper introduces these ideas using many abstract concepts without first explaining their purpose

the overall paper organization is poor beyond the poor writing; thus, it makes it hard for readers to assess and appreciate the claimed contributions. and the conclusion is incomplete, and ethics and reproducibility statements are absent. I strongly suggest that the authors carefully revisit the manuscript, and the problem is not merely writing.

  • the proposed DCT estimates the value of an action by sampling continuations after that action and observing if they succeed or fail. It thus raises a question of whether the differences between a few sampled outcomes do reflect real differences in action quality. this matters since DCT uses estimated relevance to decide where to spend additional sampling effort. also, since the alternative actions are sampled from the current policy, what if the policy rarely chooses a useful action at that training point? It thus would be useful to know how accurate the estimate action values and relevance scores are throughout the training.

  • the paper needs to provide a report of the extra computational overhead induced by DCT, compared to that from the closest competitive baseline, Tree-GRPO, which beats DCT in the single-hop setting. Knowing this trade-off between performance gains and cost would make the practical value of DCT easier to assess. Also, since this paper involves a multi-turn setting, it is fairer to use recent multi-turn RL methods (based on GRPO) that pay particular attention to the credit assignment in long-horizon interactions, unlike GRPO, which originates from single-turn setting. the paper misses details required for reproduction, like the values for important settings such as the forward- expansion budget, the exact exploration coefficient, etc.

Questions for authors
  • When non-good alternative actions are sampled, DCT uses the original terminal reward as a fallback. Could the authors report how frequently this happens across training and decision turns? This would help clarify how much of the training signal comes from the proposed interventional estimation mechanism and whether it changes as the policy improves.

  • the paper assumes that intervening on an action keeps the past env context fixed, but what if the env is partially observable env where the same visible history may correspond to different situations, could the authors clarify this?

See more in the weaknesses above.

4 comments

W1: Clarity and organization.

R1: We thank the reviewer for the detailed and constructive suggestions. Although reviewers H2bC and dFYx acknowledged our theoretical contributions and empirical evidence, the suggestions you provided are highly valuable for refining the structure of the manuscript and enhancing its overall quality. In response to your suggestions, we have addressed the necessary clarifications and improvements across two areas: (i) definitions and terminology, and (ii) manuscript organization. We will incorporate these revisions into the final version.

R1.1: Definitions and terminology

Notation and roles (Sec. 3). Our formulation follows the classical SCM representation of POMDPs [a]. Here, S,O,H,AS,O,H,A collect variables across interaction turns, whereas St,Ot,Ht,AtS_t,O_t,H_t,A_t denote the corresponding variables at turn tt. We will clarify this convention at lines 103–104. To distinguish actions from advantages, we will use adv⁡tπ\operatorname{adv}_t^\pi for the advantage function throughout the revised manuscript.

[a] Buesing et al. "Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search." Proceedings of International Conference on Learning Representations. 2019.

Metrics (Sec. 6.3). We will move these metric definitions and explanations to Section 6.1 in the revised manuscript.

R1.2: Manuscript organization

Abstract and introduction: We assess decision contribution through a counterfactual question: how would replacing an action change success probability? Same-history interventions implement this comparison while retaining the preceding history and continuation policy (lines 45–46). This explains why helpful decisions can be obscured by failure and harmful ones reinforced by success (lines 32–37). We will introduce this definition before underthinking/overthinking.

Process rewards provide intermediate feedback, while critics and rollouts estimate returns (lines 38–44). Return estimates can support attribution through replacement comparisons; their accuracy alone does not establish the replacement effect. These comparisons assess whether decisions affect outcomes (causal relevance), while using them for training requires credit that preserves the expected policy gradient (optimization consistency) (lines 47–57). We will make this connection explicit before introducing the two concepts.

Sec. 4.1 and 4.2: Terminal reward gives an unbiased policy-gradient estimate, whereas estimated credit may change the expected update. Proposition 1 bounds this deviation (lines 131–141); optimization consistency requires preserving the original expected gradient.

However, baselines can change credit's zero set—where credit vanishes—without changing the expected update (lines 186–189), while opposing replacement effects can cancel in advantage (lines 148–157). Causal relevance therefore separately asks whether any feasible replacement changes the expected outcome. Relevance faithfulness requires zero relevance scores exactly when none does. These are population targets for finite estimators (Appendix B.3). We will clarify these links before Definition 2 using the key explanations from B.2–B.3.

Sec. 5: The tree collects comparisons for relevance and credit estimation, while policy updates use the original trajectories, called anchors (lines 202–205, 252–262). Successful paths, called references, guide where further comparisons are collected. Forward discovery seeks such a reference for a failed anchor; backward testing compares alternatives along it and may refine its suffix. Tree-wide estimation then combines this evidence to estimate relevance and assign credit to anchor decisions. We will present this workflow before Section 5.1, drawing on A.1–A.2 and C.3.

W2: Reliability of sampled estimates and coverage of useful alternatives.

R2: We view this as a trade-off between estimation accuracy and rollout cost [b]. More continuations can reduce variance but require additional rollouts. Our repeated-sampling audit shows that larger control-sampling budgets improve credit stability. Please refer to our R2 to review dFYx for the coverage results, stability measurements, and protocol. Table 2's independent interventions provide complementary evidence that the sampled comparisons are informative.

For useful actions rarely sampled by the policy, shared-prefix branching provides additional trials without regenerating preceding interactions. Uniform history exploration keeps histories with low estimated relevance eligible for further trials (Section 5.2), although discovery still depends on policy probability and budget. We therefore use bounded, relevance-guided expansion while retaining uniform exploration, balancing intervention evidence against rollout cost.

[b] Kazemnejad et al. "VinePPO: Refining Credit Assignment in RL Training of LLMs." Proceedings of International Conference on Machine Learning. 2025.

W3: Training overhead relative to Tree-GRPO, single-hop performance, missing multi-turn credit baselines, and reproduction settings.

W3.1: Training overhead relative to Tree-GRPO.

R3.1: For the detailed computational cost comparison with Tree-GRPO, please refer to our R3.1 to Reviewer H2bC.

W3.2: Limited improvements on Single-hop QA.

R3.2: Single-hop QA typically requires only one retrieval followed by an answer, resulting in shallow interaction trees and limited ambiguity in decision-level credit. Thus, fine-grained credit assignment offers smaller gains over trajectory-level supervision. Similar observations are reported in Tree-GRPO, where improvements are also less pronounced on single-hop tasks.

W3.3: Missing multi-turn credit assignment baselines.

R3.3: We additionally compare DCT with two recent multi-turn credit assignment methods, GiGPO [c] and TIPS [d]. On Qwen2.5-3B, DCT achieves the highest average EM on both Single-Hop QA and Multi-Hop QA. The larger gains on multi-hop tasks further support the effectiveness of DCT's same-history causal credit assignment for complex multi-step interactions.

MethodNQTriviaPopQASingle-Hop Avg.Hotpot2WikiMuSiQBamb.Multi-Hop Avg.
GiGPO42.059.542.448.036.937.012.664.137.7
TIPS43.4658.8042.7648.331.4029.258.7320.8022.5
DCT (Ours)49.256.740.648.846.948.819.249.141.0

[c] Feng et al. "Group-in-Group Policy Optimization for LLM Agent Training." Proceedings of Advances in Neural Information Processing Systems. 2025.

[d] Xie et al. "TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs." Proceedings of International Conference on Learning Representations. 2026.

W3.4: Missing reproduction settings.

R3.4: Please refer to our response to Reviewer H2bC (R3.3: Numerical Configurations), where we provide the detailed training hyperparameters and implementation settings.

Q1: When non-good alternative actions are sampled, DCT uses the original terminal reward as a fallback. Could the authors report how frequently this happens across training and decision turns?

R4: DCT uses the original anchor's terminal reward as fallback credit when fewer than two same-history action draws are available (K(h)<2K(h)<2). We examine fallback frequency across training updates and decision turns.

Updates 1–60Updates 61–120Updates 121–180Turn 1Turn 2Turn 3Overall
6.65%4.35%3.55%0.65%3.86%6.95%4.85%

The result shows a decline in fallback frequency from 6.65% to 3.55% across training stages. Fallback is more frequent at deeper decision turns, where same-history samples may be less abundant. The low overall frequency suggests that DCT primarily leverages same-history comparisons for decision-level credit, while terminal-reward fallback ensures that credit remains available when comparison samples are insufficient.

Q2: The paper assumes that intervening on an action keeps the past env context fixed, but what if the env is partially observable env where the same visible history may correspond to different situations, could the authors clarify this?

R5: Our formulation permits latent states, consistent with SCM-based POMDP evaluation [a]. DCT’s tree branches compare alternative actions within one replayed pre-decision context, without requiring history to identify the state. Our audited QA implementation restores the truncated input and execution progress, uses fixed, session-independent retrieval, and resamples continuations under the frozen policy.

Thank you for your detailed and constructive comments. In our responses, we clarify the manuscript’s terminology and conceptual connections, outline specific revisions, and provide additional baseline comparisons and fallback statistics.

As the discussion period is nearing its end, we would be grateful if you have time to review our response and share any further feedback. We would be happy to clarify any remaining concerns.

Dear Reviewer B8do,

Thank you very much for your great efforts in reviewing this paper!

As the end of the discussion period is approaching, we are eagerly anticipating your feedback to ascertain if our responses have adequately addressed your concerns. Your feedback will be instrumental in enhancing the quality of our manuscript. We would appreciate the opportunity to address any additional concerns you may have before the discussion period ends.

Thank you once again for your time and consideration!

Sincerely,

The Submission 40614 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.