acceptodds
Under review as a conference paper at ICLR 2027

Time and Cause: Reliability-Adaptive Hybrid Credit Assignment in Reinforcement Learning

Abstract

Assigning credit to decisions with delayed consequences remains a central challenge in reinforcement learning. Causal credit assignment can reduce policy-gradient variance, but its benefits depend on accurately learned underlying causal mechanisms. In this paper, we study two challenges of this approach: unreliable learned contributions can introduce persistent policy-gradient bias, while ignoring the delay between actions and outcomes can make the relevant causal relationships more difficult to learn. We introduce ybrid elay-aware redit ssignment (HDCA), which addresses both challenges by (i) adaptively interpolating between unbiased temporal and learned causal policy-gradient estimators based on uncertainty in the estimated causal mechanisms and (ii) conditioning credit on the time gap between an action and its outcome. We support these design choices with theoretical analyses that characterize their benefits. Across a diverse suite of existing and newly designed credit-assignment benchmarks, HDCA achieves the best or tied-best final performance while improving sample efficiency and training stability. Mechanistic evaluations of gradient quality and credit-assignment accuracy provide evidence for the sources of these improvements. Finally, targeted experiments empirically validate the theoretical analyses underlying HDCA.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Credit Assignment, Causal credit assignment, Policy gradients, Delayed rewards

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Anz7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HDCA extends COCOA-reward with delay-conditioned contribution coefficients, learned kernel smoothing, and uncertainty-based mixing of temporal and causal gradient estimates. The paper reports strong results on credit-assignment benchmarks and symbolic Montezuma’s Revenge with behavior-cloning initialization, supported by theoretical analysis, DP gradient diagnostics, and ablations.

Critical strengths
  1. The diagnostic tasks expose concrete failure modes, and the enumerable MDPs allow comparisons against DP ground truth for gradient bias, variance, and credit error (Figs. 2, 5, 6, 13).

  2. The appendix is candid about the approximations behind Eq. (8) and the limits of variance reduction. The component ablations and oracle-mixing comparisons provide useful evidence beyond learning curves.

Critical weaknesses
  1. The benefit of the particular adaptive mixing rule is not sufficiently isolated. Three things contribute to this.

    • (a) Figure 14 tests only one fixed hybrid weight, α=0.5\alpha = 0.5. Prop E.5.6 shows α^\hat\alpha can never go below (M−1)/(2M−1)=0.44(M-1)/(2M-1) = 0.44 at M=5M = 5, so that control sits at the floor of the admissible range, and no larger constant is tried.

    • (b) The causal-only variants in the same figure also reduce MM from 5 to 1, so ensemble size changes together with mixing and the two effects cannot be separated.

    • (c) The HCR evidence does not fill the gap. App. I.4 measures expected assigned credit after mixing. REINFORCE's expected credit at (s,a,z)(s, a, z) is E[Rt+z∣s,a]−E[Rt+z∣s]E[R_{t+z} \mid s, a] - E[R_{t+z} \mid s], which is the ground truth Cz⋆(s,a)C^\star_z(s, a), so its HCR is zero in expectation, and HDCA's HCR falls with the REINFORCE weight 1−α^1 - \hat\alpha whatever the hindsight model does. Lower HCR therefore establishes better credit for the combined estimator, but does not show a more accurate hindsight model (Sec. 4.2, B.2). A diagnostic of the causal component before mixing would directly test that explanation.

    These experiments support the overall method, but leave the added value of adaptivity open. I may have misread the HCR definition in (c); if so, I would reconsider this point.

  2. Theorem 3.1.1 omits conditions required by its appendix counterpart.

    • (a) Theorem E.5.4 assumes signal–error orthogonality and cross-pair decoupling and optimizes over deterministic mixing vectors. These restrictions matter for the claimed global MSE optimum and should appear in the main statement.

    • (b) The practical input-dependent rule additionally relies on the approximations listed on p. 34, so the gap between what is proved and what is run is wider than the main text suggests.

The larger experiment also needs clearer reporting: the Reproducibility Statement says all Section 4 results use five seeds with 95% bootstrap intervals, but Fig. 4 and Sec. 4.4 do not say so themselves, and the task-success criterion on the y-axis is never defined. This limits how much weight I can place on the evidence beyond the small discrete benchmarks.

Questions for authors
  1. How does adaptive HDCA compare with fixed α∈{0.7,0.8,0.9,1}\alpha \in \{0.7, 0.8, 0.9, 1\}, keeping M=5M = 5, delay conditioning, smoothing, seeds, and budgets matched?

  2. Can you report causal-component HCR before mixing at the same policies and checkpoints as mixed HCR? Under App. I.4, how is REINFORCE's expected credit defined, and does its population HCR equal zero?

  3. Please clarify the scope of Theorem 3.1.1 and include the assumptions above. Were Figs. 9–11 run with tabular policies, as B.5 states, or the MLP in Table 4? If tabular, those experiments do not test the isotropic approximation in the main MLP setting.

  4. For Montezuma, please define the task-success criterion in Fig. 4 and confirm that the five-seed, 95% bootstrap convention of the Reproducibility Statement applies to it.

  5. What’s the relation to upside-down reinforcement learning (Schmidhuber, 2019; Srivastava et al., 2019), where the desired reward is an input viewed as a command, and the net learns actions that satisfy the command in hindsight? The hindsight model behind your contribution coefficient looks like the same object, trained the same way, and Appendix A does not discuss this line.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Anz7 (1/2)

We thank the reviewer for the careful reading, including of the appendix, and for the constructive feedback.

Weakness 1, Questions 1 and 2: Isolating the adaptive mixing rule.

Response:

(a) Fixed weights (Question 1). We agree that a single constant at the floor was not a sufficient control. App. C.2 (Fig. 19) now compares HDCA with nine constants, α∈{0.1,…,0.9}\alpha \in \lbrace 0.1, \ldots, 0.9 \rbrace, on five tasks, with M=5M = 5, delay conditioning, kernel smoothing, seeds, and budget matched; with α=1\alpha = 1 in App. C.1, this covers all requested values (0.70.7, 0.80.8, 0.90.9, 11). High causal weights (0.70.7–0.90.9) learn quickly, but several later lose performance, and the best constant changes with the task and training stage. HDCA attains the highest final performance on all five tasks without choosing a constant. In a related control (App. B.6), an annealed schedule approaches HDCA only when its decay rate is fitted, per task, to the oracle weights.

(b) Ensemble size confounded with mixing. Agreed. App. C.1 (Fig. 18) now adds two causal-only variants with HDCA's ensemble (M=5M = 5), with and without kernel smoothing; the smoothed one keeps all of HDCA's hindsight components and removes only the temporal term. Ensembling and smoothing help causal-only performance on several tasks but do not consistently recover HDCA's learning speed or final performance, so ensemble size does not explain the gain.

(c) HCR after mixing (Question 2). The reviewer's reading is correct. In the centered form of App. I.4, REINFORCE's expected credit at (s,a,z)(s,a,z) is E[Rt+z∣St=s,At=a]−E[Rt+z∣St=s]=Cz⋆(s,a)\mathbb{E}[R_{t+z} \mid S_t=s, A_t=a] - \mathbb{E}[R_{t+z} \mid S_t=s] = C^\star_z(s,a), so its population HCR is zero, and mixing alone can lower HCR. App. B.3 now states this and reports the requested diagnostic: Fig. 7 shows the HCR of the causal component before mixing, at the same policies and checkpoints, on all six conveyor tasks. The two curves nearly coincide (a gap of about 0.50.5 dB or less on five tasks and about 11–1.51.5 dB on Poisoned-apples), so HDCA's low HCR is already present in the learned causal component rather than produced by the REINFORCE weight.

Weakness 2: Theorem 3.1.1 omits conditions required by its appendix counterpart.

Response: Please see Common Concern 1: (a) for the assumptions now in the theorem, (b) for how far they hold and the relaxation in Theorem E.6.2, and (c) for the gap between the proved and the implemented rule.

Weakness 3: Reporting for Fig. 4 and Sec. 4.4.

Response: Please see our answer to Question 4 below.

AuthorsOpenReview ↗

Response to Reviewer Anz7 (2/2)

Question 3: Scope of Theorem 3.1.1; were Figs. 9–11 run with tabular policies, as B.5 states, or with the MLP in Table 4?

Response: For the scope, please see Common Concern 1. Thank you for catching the second point, and we apologize for the confusion. The sentence "Since all policies here are tabular softmax" was poorly worded. We did not mean that these experiments used tabular policies; we meant that we approximate the policy scores as isotropic, a property that holds exactly for tabular softmax policies, so that VΔV_\Delta and VεV_\varepsilon share the same scale. All results in that section, including Figs. 9–11 (Figs. 10–12 in the revision), use the same MLP policy as the main experiments (Table 4; Table 5 in the revision). We have corrected the sentence, and Section 3.1 now states this approximation explicitly as assumption (A3). We also test it directly in App. B.7 with the same MLP policy: oracle weights computed with the isotropic approximation and with the exact score Gram matrix differ by less than 0.010.01 in mean at every checkpoint, and the resulting MSE stays within a few percent on five tasks; on Poisoned-apples, the isotropic weights do slightly better early in training (Fig. 15).

Question 4: Montezuma's Revenge: the success criterion in Fig. 4 and the statistical protocol.

Response: Section 4.4 now defines a successful episode: the agent collects the key, unlocks a door, and exits the starting room. Fig. 4 follows the convention of the rest of Section 4 (mean over five seeds with 95% bootstrap intervals), which its caption now states.

Question 5: Relation to upside-down reinforcement learning.

Response: App. A now discusses this connection. The learned object is indeed closely related: our hindsight model h(a∣s,u,z)h(a \mid s, u, z) predicts actions from retrospectively labeled outcomes and is trained by supervised learning on the agent's own experience, as a UDRL behavior function is. The two differ in conditioning and in use:

  • Conditioning. UDRL conditions on a desired cumulative reward and horizon (a command); HDCA conditions on an individual rewarding outcome and its delay.
  • Role. In UDRL, the model is the policy. In HDCA, it is never used to act: its posterior, divided by the current policy probability, gives a contribution coefficient that weights rewards in a separate policy-gradient estimator. HDCA thus separates two kinds of knowledge: which action to take, held by the policy, and how each action changes the probability of a later outcome, held by the hindsight model and used only to make the policy's learning more efficient. In UDRL, a single model plays both roles, and its aim is to act on commands rather than to make the learning of a separate policy more efficient.
  • Consequence. UDRL's basic return command aggregates rewards over a horizon (although its broader framework supports richer commands), whereas HDCA attributes each reward event to earlier actions. HDCA's contributions (reliability-adaptive mixing with an unbiased temporal estimator, and delay conditioning) concern how far that attribution can be trusted, a question that does not arise when the model is the policy.
AuthorsOpenReview ↗

Gentle Reminder: Following up on our rebuttal and paper revisions

Dear Reviewer Anz7,

Thank you once again for your time and constructive feedback on our submission.

As the ICLR author-reviewer discussion period is drawing to a close, we wanted to kindly follow up to ensure you have had a chance to review our rebuttal responses and the revised manuscript. Based on your suggestions, we have updated the paper (with changes highlighted in the revision) and provided additional experiments and clarifications to address your concerns.

Please let us know if our responses have satisfactorily resolved your questions, or if there are any remaining points we can clarify before the discussion phase ends. We truly appreciate your thoughtful consideration and look forward to any further feedback.

Best regards,

The Authors

Thank you for the thorough revision; it resolves my concerns, and Fig. 19 shows no single constant matches HDCA across tasks. I'm raising my score to 3.

Reviewer_qVTuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The study proposes Hybrid Delay-aware Credit Assignment (HDCA), which extends COCOA's hindsight policy-gradient estimator in three ways: a) it conditions the hindsight distribution on the delay between action and outcome; b) it uses an ensamble of models to estimate the hindsight distribution, allowing to also report on the uncertainty of the estimation, relying less on it, and more on Monte-Carlo returns when it is uncertain; c) it shares information across neighbouring delays through kernel smoothing with a learned bandwidth, interpolating between delay-specific and pooled hindsight. HDCA is evaluated on COCOA's key-door conveyor task, on five conveyor tasks proposed anew by the author, on two diagnostic tasks and a symbolic Montezuma's Revenge. Results are reported for HDCA and baselines that vary depending on the experiment.

Critical strengths

The methodology is sound as it identifies drawbacks of a current method and tries to address them. Credit accuracy is measured directly. The Hindsight Credit Reconstruction error against ground-truth credit computed by dynamic programming, and the policy-gradient SNR and bias ratio (Fig. 2). Diagnostic tasks isolate the identified problems well. Evaluations are faid: baselines use COCOA's defaults, the learning rate is searched for all methods, a 12-configuration sensitivity analysis is reported (App. I.5), 5 seeds with bootstrap intervals, and code is supplied.

Critical weaknesses

Major

1. Methods

I have major doubts about the soundness of the method:

  • Since the method is only tested with a mix of its own estimator and the canonical Monte-Carlo returns, it is not clear how good the new estimator is. When tested alone (Figure 1.): "... plateaus below full HDCA on Ordered-keys and Multicolored-keys, and exhibits late-training declines on XOR-keys and Poisoned-apples. Its uncertainty bands are also particularly wide on several tasks.", or a COCOA-reward without mixing "stays roughly 20-30 percentage points below HDCA at every ensemble size".
  • It is not clear how the uncertainty, and so the balance between Monte-Carlo returns and hindsight credit, varies across tasks and over training. The main text never shows the mixing coefficient α^t,z\hat\alpha_{t,z}. Appendix B.1 discusses the mixing dynamics and the uncertainty surrogates for one task only (Colored-keys, Figure 5), and defers the others to Appendix B.6. How much weight does the hindsight term receive on each task, and does HDCA outperform COCOA where it relies mostly on REINFORCE?

2. Claims

  • Optimal mixing. Theorem 3.1.1 is stated in the main text without assumptions ("Fix a pair (t, z)", l. 210). Its proof (Theorem E.5.4) needs signal–error orthogonality and cross-pair decoupling, i.e. uncorrelated errors across source–delay pairs (lines 1671–1676), which the paper calls "a simplification" (line 1680). How much this assumption holds in reality is unclear and not investigated. Pairs from the same trajectory share policy scores and rewards, so decoupling is unlikely to hold.
  • Temporal methods. "These methods ... retain a temporal heuristic for credit assignment, rather than explicitly measuring each action's contribution to a future outcome" (lines 36–38). The advantage Qπ(s,a)−Vπ(s)Q^\pi(s, a) - V^\pi(s) is the action's contribution to the expected return, and GAE estimates it; hindsight methods estimate the same policy gradient differently (unbiased in the oracle case, Theorem F.1.3). The difference is in the estimator's bias and variance, not in whether contribution is measured.
  • Diversity and scale. "A diverse suite of existing and newly designed credit-assignment benchmarks" (lines 21–22): one task from prior work and five conveyor variants designed by the authors, all with small action spaces, full observability and easy exploration (lines 361–363), plus a symbolic Montezuma's Revenge with behaviour-cloned policies and warm-started value and hindsight models (lines 462–470). These isolate credit assignment well, but don't show HDCA beyond small gridworlds.

3. Evidence

  • Evidence is limited and mostly on environments designed by the authors themselves. "A diverse suite of existing and newly designed credit-assignment benchmarks" (lines 21–22): one task from prior work and five conveyor variants designed by the authors, all with small action spaces, full observability and easy exploration (lines 361–363), plus a symbolic Montezuma's Revenge with behaviour-cloned policies and warm-started value and hindsight models (lines 462–470). These isolate credit assignment well, but don't show HDCA beyond small gridworlds.
  • Key baselines are missing, e.g., actor-critic with GAE(λ), HCA, CCA.
  • Compute is not reported even if HDCA trains five hindsight models and their bandwidth networks, against one hindsight model in COCOA. The paper should report wall-clock and parameter counts, at least.
  • On Montezuma's Revenge, COCOA-reward is better for most of training and HDCA overtakes it late (Fig. 4).

4. Writing

  • The main text has no related work section: it is only in the appendix ("for a detailed review of prior work, see Appendix A", line 46). The introduction positions HDCA against COCOA and HCA (lines 41–47), but not against the wider credit assignment literature (e.g. return decomposition, value-based credit), which should be in the main text.
  • Parts of the method are in the background. Section 2 defines HDCA's own components: "We refer to the component of HDCA's gradient estimator that assigns credit using learned action–outcome contribution coefficients as the causal estimator", and the ensemble with its epistemic and aleatoric uncertainty measures, which are the core of the mixing rule (Eq. 8). They belong in Section 3.1, where they are used; Section 2 should keep only what HDCA builds on (the MDP, REINFORCE, COCOA's contribution coefficient and estimator, Eq. 1–2).

Minor

5. Writing

  • The main text heavily depends on a 62-page appendix for most definitions and every proof; one paragraph points to six appendix sections (lines 346–354), and the key assumptions should move up.
  • The pairwise notation is heavy (GRγ,(t,z)G_R^{\gamma,(t,z)}, Eq. 3), and several symbols are overloaded.
  • Small slips: "we set γ = 1 and write GR:=GR1G_R := G_R^1 ,." (line 100); "We call wtw_t and hth_t as pooled-delay" (lines 139).

6. Inconsistencies and missing details

  • UU denotes both the outcome representation (lines 115–117) and the ensemble spread U2(x)U^2(x) (lines 162–163); KK both the uniform future time KtK_t (line 135) and the kernel KσK_\sigma (line 312).
  • "Equation (8) has no tuned parameters" (line 243) is true of the rule itself; the ensemble size, the regularisation toward initialisation and the bandwidth networks are still choices.
Questions for authors

Additional questions after the weaknesses:

  • How much bias remains when the learned coefficient is unreliable, given the lower bound on α^\hat\alpha? Would a rule that can reach α=0\alpha = 0 do better on Coupon-or-vault?
  • How large are the cross-pair correlations of ε\varepsilon and Δ\Delta in practice, i.e. how far is assumption (A2) from holding?
  • How does HDCA compare with an actor-critic using GAE(λ) on these tasks?
  • What is HDCA's compute and memory overhead relative to COCOA?
  • Can you show HDCA on a delayed-reward benchmark from prior work at a larger scale?
9 comments
AuthorsOpenReview ↗

Response to Reviewer qVTu (1/3)

We thank the reviewer for the thorough review and for recognizing the diagnostic tasks and the fairness of the evaluation.

Weakness 1 (Methods): (i) the estimator is only tested mixed with Monte-Carlo returns; (ii) how the balance between the two varies across tasks and over training.

Response:

(i) First, a clarification: HDCA is not a standalone estimator that we then mix with Monte-Carlo returns. The hybrid in Eq. (6) is the proposed method: its premise (P1) is that a learned causal estimator should not be trusted uniformly, so mixing is part of the design. We agree, though, that the learned component should be evaluated on its own, and the revision does so:

  • Causal component before mixing (new App. B.3, Fig. 7). At the same policies and checkpoints, its HCR is within about 0.50.5 dB of the full hybrid on five tasks and 11–1.51.5 dB on Poisoned-apples, so the low credit error comes from the learned delay-conditioned coefficients, not from mixing.
  • Ablations (App. C.1, C.2, C.4): causal-only variants (α=1\alpha = 1, M∈{1,5}M \in \lbrace 1, 5 \rbrace, with and without smoothing), fixed mixtures α∈{0.1,…,0.9}\alpha \in \lbrace 0.1, \ldots, 0.9 \rbrace, and an ensembled COCOA-reward.

These ablations also show why mixing is still needed when the causal component is accurate on average: its errors on rarely observed outcomes steer the on-policy data away from them (P1), leading to plateaus and late declines. Fixed mixtures with α=0.7\alpha = 0.7–0.90.9 show the same pattern, and ensembling alone does not close the gap; only adaptive mixing reaches the best final performance on all five tasks (Figs. 18–19; App. C.4).

(ii) The mixing dynamics are now reported across tasks, mainly in App. B.1 (Fig. 5: bias, variance, mixing coefficient, and HCR on three tasks) and App. B.6 (Fig. 10: the REINFORCE weight 1−α^1-\hat{\alpha} against the oracle α⋆\alpha^\star on four tasks), and Section 4.3 now summarizes them.

On every task, the hindsight term receives most of the weight. The REINFORCE weight is largest at the start of training (about 0.10.1–0.20.2), when the ensemble is untrained and hindsight data are sparse, falls below about 0.050.05 within the first few thousand episodes (with brief spikes later), and follows the trend of the oracle α⋆\alpha^\star (Fig. 10). By construction, it never exceeds 5/95/9 per pair (Prop. E.5.6). So, in our experiments, HDCA never relies mostly on REINFORCE; it uses it as a temporary, input-dependent safeguard. This small early weight is what separates HDCA from the causal-only and high-α\alpha variants, which learn quickly but plateau or decline later (Figs. 18–19).

Weakness 2 (Claims): optimal mixing; temporal methods; diversity and scale.

Response:

Optimal mixing. Please see Common Concern 1. In short, (A1)–(A2) are now in Theorem 3.1.1 and (A3) where Eq. (8) is derived; Theorem E.6.2 removes the need for orthogonality between pairs sharing a source timestep, where, as the reviewer anticipated, decoupling fails; and App. B.7 measures how far each assumption holds on all six tasks.

Temporal methods. We agree: the advantage measures the action's contribution, and hindsight methods estimate the same policy gradient (Theorem F.1.3), so the difference lies in the estimator, not the target. Our point concerned the sampled update, which credits every earlier action for every later reward; temporal methods reduce the resulting variance through recency (γ,λ<1\gamma, \lambda < 1), at the cost of bias, whereas hindsight methods offer a trade-off that does not rely on recency. Lines 36–38 now say that these methods' sampled updates "need not isolate each action's contribution to individual future rewards."

Diversity and scale. The suite now includes three tasks from prior work (Short/Long Key-door and Stochastic-interleaving from COCOA, SkillLuck from CSCA), alongside our five conveyor tasks, each built on a distinct credit-assignment structure (App. I.2), two diagnostic tasks, and symbolic Montezuma's Revenge. We kept the tasks small and fully observable on purpose. That way, credit assignment can be studied separately from exploration and representation learning, and the exact ground truth needed for our diagnostics is available. Small, controlled tasks like these are common practice in the credit-assignment literature (e.g., CCA, HCA, COCOA, CSCA). Our symbolic Montezuma's Revenge task goes a step further, with the full Atari action space, trajectories of up to 256 decisions, and a multi-step goal (collect the key, unlock a door, and exit the room) that is considerably more challenging than the smaller MDPs. Extending HDCA to high-dimensional observations remains future work (Section 5).

AuthorsOpenReview ↗

Response to Reviewer qVTu (2/3)

Weakness 3 (Evidence): benchmarks; missing baselines; compute; Montezuma's Revenge.

Response:

Benchmarks. Please see "Diversity and scale" above.

Baselines. We added GAE (λ∈{0.9,0.95}\lambda \in \lbrace 0.9, 0.95 \rbrace), CCA, and CSCA on all conveyor tasks (Figs. 3 and 17); HCA+ was already included. They show the same task dependence as the other baselines: GAE is the strongest (see Q3), CCA progresses substantially only on Poisoned-apples, and CSCA (our re-implementation; App. I.1) only on Colored-keys.

Compute. The new App. B.9 reports FLOPs per training iteration and parameter counts (see Q4). We use FLOPs because they do not depend on hardware or on implementation details such as whether ensemble members run in parallel.

Montezuma's Revenge. We agree that COCOA-reward was ahead for most of the original 2M-sample runs, so we extended all runs to 5M samples (Fig. 4). The first 2M samples match the original runs: COCOA-reward improves faster early and peaks at about 0.80.8 task success near 1.6M samples, but then declines to about 0.50.5. HDCA rises sharply from about 1.2M samples, overtakes COCOA-reward at about 1.7M, and remains the best method for the rest of training (about two-thirds of the run), finishing near 0.970.97; the next best, Q-critic, finishes near 0.80.8.

Weakness 4 (Writing): (i) no related-work section in the main text; (ii) parts of the method are in the background.

Response:

(i) Space prevents a full related-work section in the main text. The introduction now positions HDCA against temporal methods (TD learning, eligibility traces, GAE), reward redistribution (RUDDER, CSCA), and hindsight methods (HCA, CCA, COCOA). App. A gives the detailed review, including Align-RUDDER, value transport (TVT), model-based credit assignment, and uncertainty-weighted estimators.

(ii) Agreed. The ensemble, its mean and variance, and the uncertainty measures are now defined in Section 3.1, where they are used.

Minor (Writing): (i) dependence on a long appendix; (ii) heavy pairwise notation; (iii) small slips.

Response:

(i) We agree that the appendix is long. However, the main text is self-contained: it presents the method, the main results, and the assumptions of Theorems 3.1.1 and 3.2.1, and refers to the relevant appendix at each point. The appendix provides the proofs, diagnostics, and ablations, which give readers a deeper understanding of the method but are not needed to follow its main message.

(ii) The (t,z)(t,z) indexing is needed because the mixing coefficient is defined per source–delay pair, and we use it only where this matters (Eqs. (3)–(8), Theorem 3.1.1).

(iii) Both are corrected.

Inconsistencies: (i) overloaded symbols; (ii) "Equation (8) has no tuned parameters".

Response:

(i) The outcome representation is the italic UU and the ensemble disagreement the calligraphic U\mathcal{U} (Table 2); KtK_t and KσK_\sigma are distinguished by their subscripts.

(ii) The statement refers to the mixing rule itself, which adds no parameter beyond the ensemble; the ensemble size, the regularization toward initialization, and the bandwidth networks remain design choices. To avoid confusion, we have reworded the sentence in Section 3.1 to make this explicit.

AuthorsOpenReview ↗

Response to Reviewer qVTu (3/3)

Question 1: How much bias remains when the learned coefficient is unreliable, given the floor on α^\hat{\alpha}? Would a rule that can reach α=0\alpha = 0 do better on Coupon-or-vault?

Response: Intuitively, for each source–delay pair, the hybrid keeps a fraction αt,z\alpha_{t,z} of the learned estimator's bias and replaces the rest with unbiased REINFORCE (Theorem E.7.1; a data-dependent α^\hat{\alpha} adds a covariance term). At the 4/94/9 floor, mixing therefore removes up to 5/95/9 of an unreliable pair's bias, and the remainder vanishes as the hindsight models become consistent (Theorem E.10.2).

On Coupon-or-vault, the floor does not appear to be the limiting factor: HDCA already matches or slightly exceeds Advantage, a temporal method, at every number of gates (Fig. 2a). Where a lower floor helps, the batch-size-corrected variant (App. C.3) lowers it toward zero as the batch grows, and it outperforms standard HDCA for batch sizes from 128 to 512 on the four tasks we tested.

Question 2: How large are the cross-pair correlations of ε\varepsilon and Δ\Delta in practice?

Response: Please see Common Concern 1(b), which reports these correlations on all six conveyor tasks.

Question 3: How does HDCA compare with an actor-critic using GAE(λ\lambda)?

Response: We added GAE with λ=0.9\lambda = 0.9 and 0.950.95 on all conveyor tasks (Figs. 3 and 17; App. I.1). Like other temporal methods, it does well when its recency prior matches the task and falls behind when credit is long-range or depends on the structure of earlier decisions. It matches HDCA on Short Key-door and XOR-keys and exceeds it on Multicolored-keys (about 99% vs. 93% at the end of training), but learns Ordered-keys much later, reaches only about 25% on Colored-keys, and stays near zero on Poisoned-apples. No baseline is competitive on all tasks, whereas HDCA is best or near-best on every task.

Question 4: What is HDCA's compute and memory overhead relative to COCOA?

Response: Per training iteration, HDCA uses 7.5×7.5\times the FLOPs of COCOA-reward and 6.7×6.7\times its auxiliary parameters (148k vs. 22k), while the policy and its cost are identical (App. B.9, Table 1). Most of the overhead comes from the MM-member hindsight ensemble. Since both methods evaluate all O(L2)\mathcal{O}(L^2) source–outcome pairs, these ratios do not grow with the horizon (they are the same at L=23L = 23 and L=43L = 43). The ensemble members are independent and could run in parallel, and MM offers a direct trade-off between compute and performance (App. C.4).

Question 5: HDCA on a delayed-reward benchmark from prior work at a larger scale.

Response: Beyond the COCOA tasks (Short and Long Key-door, with L=23L = 23 and 4343, and Stochastic-interleaving), we added SkillLuck from CSCA and extended Montezuma's Revenge (up to 256 decisions per trajectory) to 5M samples. There, HDCA overtakes COCOA-reward at about 1.7M samples and stays the best method to the end (see Weakness 3), suggesting that adaptive mixing remains useful with longer horizons and richer state spaces. We have not evaluated HDCA on pixel-based benchmarks; this remains future work (Sec. 4.4, App. I.3, App. B.8).

Reviewer_9ZASOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HDCA extends COCOA in three ways: 1) It mixes REINFORCE with the learned hindsight estimator for each source–delay pair. The weight is computed from an ensemble of M = 5 hindsight models. 2)It conditions the hindsight posterior on the delay z between action and outcome. 3) It kernel-smooths across delays with a learned bandwidth.

Critical strengths
  • The problem is well motivated.
  • The writing is clear and easy to follow. Not the type of submissions full of LLM generated texts.
Critical weaknesses
  • A very important recent work, Counterfactual Shapley Credit Assignment (RLC 26, https://arxiv.org/pdf/2607.16999), is missing from the discussion, which showed that COCOA cannot do credit assignment causally and proposed a principled causal credit assignment framework without relying on learned causal models. (So line 12-13 is also inaccurate)
  • When framing the credits being causal, it's better to use SCMs. For MDP, identification is satisfied automatically but this should be careful when claiming.
  • The main-text theorems rely on assumptions that appear only in the appendix and that fail structurally. Theorem 3.1.1 is stated without assumptions in the main text. App. E.5.3 needs (A1), learned error orthogonal to the oracle signal, and (A2), zero correlation across all pairs. The optimization is over deterministic α only. (A2) fails for pairs that share a source. Their errors \eps_{t,z} = R_{t+z}(g_{A_t} − \mu_h) all contain the same score g_{A_t}, which is the general multi-reward case. Theorem 3.2.1 assumes the action's influence is localized at a single delay, with occurrences at other delays independent of the action. Thus, they actually presuppose an exogenous, additively factored reward composed of independent action-driven and action-independent factors.
Questions for authors
  1. Please state (A1) and (A2) in the main text. How large are the cross-pair correlations in your tasks?
  2. How does HDCA compare with a tuned constant \alpha, and with an \alpha that includes the |B| correction? How does it behave as batch size grows?
  3. Can you build a task where ensemble members agree on a biased posterior, and show how HDCA behaves given the 4/9 floor?
  4. Could you test HDCA in SkillLuck env from the CSCA paper fig 1? just calculate the credits in equations would suffice
6 comments
AuthorsOpenReview ↗

Response to Reviewer 9ZAS (1/3)

We thank the reviewer for a detailed and technically precise review.

Weakness 1: CSCA is missing, and lines 12–13 are inaccurate.

Response: We agree that CSCA should be discussed. The revision (i) discusses it in the introduction and App. A; (ii) adds it as a baseline on all conveyor tasks and SkillLuck (Figs. 3 and 17), using our own re-implementation since no official code was available (App. I.1); and (iii) narrows the claim of lines 12–13, in both the abstract and the introduction, to hindsight-based causal credit assignment and learned action–outcome contributions, rather than causal credit assignment in general. We also state explicitly (footnote 1, App. A) that we use "causal" in COCOA's interventional, contribution-analysis sense, not in the counterfactual SCM sense formalized by CSCA. The two target different quantities, as our answer to Q4 shows on SkillLuck.

Weakness 2: Framing credit as causal should use SCMs.

Response: We agree this needs care. App. A now writes the MDP as an SCM, and Proposition D.0.6 shows that pt,z(u∣s,a)=P(Ut+z=u∣St=s,do(At=a))p_{t,z}(u \mid s,a) = P(U_{t+z}=u \mid S_t=s, \mathrm{do}(A_t=a)), with later actions drawn from πθ\pi_\theta. The contribution coefficient is thus a ratio of interventional distributions, identified from on-policy data. Remark D.0.7 states its scope: it is policy-dependent and population-level, not a counterfactual attribution. As the reviewer notes, identification in an MDP comes with conditions, which we now state: the hindsight model must condition on the policy's full input, and the data must be on-policy.

Weakness 3: (i) Theorem 3.2.1 assumes influence localized at a single delay; (ii) this presupposes an exogenous, additively factored reward.

Response:

(i) Localization. We agree that Theorem 3.2.1 covers a specific regime, and Section 3.2 says so: it is the regime where the benefit of delay conditioning is clearest and has a closed form, and the benefit shrinks as influence spreads. The new Remark F.4.4 extends the comparison to arbitrary per-delay coefficients: delay conditioning helps when influence is concentrated at one delay or changes sign across delays, and pooling helps when it is spread evenly with a common sign. However, please note that HDCA itself does not assume localization, and this is exactly why it uses kernel smoothing: the learned bandwidth (Sec. 3.3) interpolates between the delay-conditioned and pooled estimators, so HDCA can move toward pooling wherever neighboring delays carry similar contributions (App. G, App. B.5).

(ii) Exogenous, additive reward. We agree with the reviewer's description of the condition (at delays z≠z⋆z \neq z^\star, the probability of observing uu does not depend on whether aa was taken), but respectfully disagree that it presupposes such a reward. The condition, wt,z⋆(s,a,u)=0w^\star_{t,z}(s,a,u)=0, or equivalently P(At=a∣St=s,Ut+z=u)=πθ(a∣s)P(A_t=a \mid S_t=s, U_{t+z}=u)=\pi_\theta(a \mid s), is a pointwise statement about one tuple (t,s,a,u)(t,s,a,u). Occurrences of uu at other delays may still depend on the state, on later actions, and on each other, and need not enter the reward additively. An additive model with an independent distractor reward is one way to satisfy the condition: it is sufficient, but not necessary. The impression may have come from the original Assumption F.3.1, which bundled this condition with conditional independence across delays and a constant reward. To avoid this misreading by other readers, we revised App. F.3 to separate them: the two extra conditions are needed only for the oracle variance comparison, not for the detectability results (Theorem F.4.1).

Question 1: State (A1) and (A2) in the main text. How large are the cross-pair correlations?

Response: Please see Common Concern 1: part (a) for the main-text statement and part (b) for the measured correlations.

AuthorsOpenReview ↗

Response to Reviewer 9ZAS (2/3)

Question 2: Tuned constant α\alpha, an α\alpha with the ∣B∣|B| correction, and behavior as the batch grows.

Response:

Tuned constant α\alpha. App. C.2 (Fig. 19) compares HDCA with nine constants, α∈{0.1,…,0.9}\alpha \in \lbrace 0.1, \ldots, 0.9 \rbrace, using the same ensemble, delay conditioning, and kernel smoothing (α=1\alpha = 1 is in App. C.1). No constant works well across tasks and training stages: small temporal weights learn quickly, but several later lose performance, while larger ones slow learning. HDCA attains the highest final performance on all five tasks without choosing a constant. This is expected, because the errors of the two estimators change differently during training: the learned estimator's error is largest early and shrinks as the hindsight models improve, whereas REINFORCE's variance does not shrink in the same way (Fig. 5). A constant cannot follow this change, but the uncertainty-based weight does (Fig. 10).

We also considered two idealized alternatives, which rely on ground truth from dynamic programming and are therefore not practical. A constant re-tuned at every checkpoint reaches a gradient MSE similar to that of the oracle mixing coefficient of Eq. (7) (App. B.7, Fig. 16). An exponential schedule fitted to the oracle weights comes close to HDCA in performance, but its decay rate differs across tasks, so it would need per-task tuning (App. B.6).

α\alpha with the ∣B∣|B| correction. The corrected coefficient (App. C.3) divides the aleatoric term by ∣B∣|B|, adds no hyperparameter, reduces to Eq. (8) at ∣B∣=1|B| = 1, and lowers the floor on the causal weight from (M−1)/(2M−1)(M-1)/(2M-1) to (M−1)/(M−1+∣B∣M)(M-1)/(M-1+|B|M). To see how it behaves as the batch grows, we trained it alongside standard HDCA, REINFORCE, and COCOA-reward with ∣B∣∈{32,128,256,512,1024}|B| \in \lbrace 32, 128, 256, 512, 1024 \rbrace on four conveyor tasks, keeping the total sample budget fixed (Figs. 20 and 21). We observe the following:

  • Both HDCA variants have a higher AUC than REINFORCE and COCOA-reward at every batch size and on every task.
  • At ∣B∣=32|B| = 32, the two variants are close (standard HDCA slightly ahead); from ∣B∣=128|B| = 128 to 512512, the corrected variant is better in both best performance and AUC.
  • At ∣B∣=1024|B| = 1024, REINFORCE reaches a higher best performance than standard HDCA, whose learned term carries an error that batching does not reduce. The corrected variant closes this gap, matching REINFORCE's best performance with a clearly higher AUC. Its temporal weight grows with ∣B∣|B|, to about 0.80.8 late in training at ∣B∣=1024|B| = 1024, so with very large batches it relies mostly on REINFORCE, whose variance batching has already reduced.

Question 3: A task where the ensemble agrees on a biased posterior, given the 4/9 floor.

Response: We agree this is the failure mode that disagreement-based mixing cannot detect, and Section 3.1 now states it as a limitation. We did not build a dedicated task, because the behavior follows from Eq. (8): if all members agree on a biased posterior, disagreement vanishes, α^→1\hat{\alpha} \to 1, and HDCA reduces to its learned causal estimator, inheriting (not correcting) its bias, which Theorem E.8.1 bounds linearly in the posterior error.

Three observations suggest this case was not dominant in our experiments: (i) members are initialized independently and weakly regularized toward their initialization, which preserves disagreement on poorly supported inputs; (ii) on the conveyor tasks, the surrogate V^Δ\widehat{V}_ \Delta broadly tracks the true error VΔV_\Delta throughout training (App. B.6, Fig. 11); and (iii) on Coupon-or-vault (Fig. 2a), which targets rare outcomes early in training, where a learned posterior is most likely to be wrong, HDCA's gradient bias stays well below COCOA-reward's as the task gets harder, and HDCA matches or slightly exceeds Advantage at every number of gates.

The floor matters in the opposite case, where the ensemble does disagree: with M=5M = 5 it caps the REINFORCE weight at 5/95/9 per pair. The batch-size-corrected rule (see Q2) lowers this floor to about 0.090.09 at our default ∣B∣=8|B| = 8, and toward zero as the batch grows, so the rule can rely almost entirely on the temporal estimator wherever the ensemble disagrees.

AuthorsOpenReview ↗

Response to Reviewer 9ZAS (3/3)

Question 4: SkillLuck from the CSCA paper (Fig. 1); calculating the credits would suffice.

Response: We did both: we computed the credits in closed form (table below) and trained all methods on our adaptation of the task (App. B.8, Fig. 17). Let p=π(skill)p = \pi(\text{skill}) at the branch decision and q=π(correct final action∣skill)q = \pi(\text{correct final action} \mid \text{skill}). The only nonzero reward is at the final step: 11 with probability qq on the skill branch, and with probability 1/21/2 on the luck branch regardless of the final action.

Table: Oracle contribution coefficients and expected credits C⋆=E[R∣s,a]−E[R∣s]C^\star = \mathbb{E}[R \mid s,a] - \mathbb{E}[R \mid s] in SkillLuck.

DecisionOracle coefficient (r=1r=1)Expected credit C⋆C^\star
Branch, skillqpq+(1−p)/2−1\frac{q}{pq+(1-p)/2}-1(1−p)(q−12)(1-p)(q-\frac{1}{2})
Branch, luck1/2pq+(1−p)/2−1\frac{1/2}{pq+(1-p)/2}-1−p(q−12)-p(q-\frac{1}{2})
Intermediate actions, either branch0000
Final action, luck branchNo delayed term00
Final action, skill branchNo delayed term1−q1-q (correct), −q-q (wrong)

The table shows that, with oracle coefficients, the inert intermediate actions receive exactly zero credit, and so does the final action on the luck branch, in expectation. The branch decision receives the interventional advantage: at the uniform initial policy, both branches are worth 1/21/2, so this credit is zero, and it favors skill exactly when the final action becomes better than chance (q>1/2q > 1/2). This is also where HDCA and CSCA differ. HDCA's credit is interventional and policy-dependent, the same kind of quantity the policy gradient itself is built on (Theorem F.1.3, Remark D.0.7), whereas CSCA's credit is counterfactual and trajectory-specific, which lets it separate skill from luck within a single trajectory. App. A now states this distinction.

The learning curves follow the same picture. The final skill action is learned from its immediate reward, so qq rises above 1/21/2, and the branch credit then points to skill. Accordingly, HDCA learns to choose the skill branch nearly as fast as Advantage and GAE, while COCOA-reward stays at chance and HCA+ improves only slowly, in line with the CSCA paper's results for these two methods. Our CSCA re-implementation (φ\varphi-PPO, without official code and without Prioritized Trajectory Replay; App. I.1) also learns to prefer skill, but more slowly; we report it with this caveat.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful and constructive reviews. Major changes in the revised PDF are highlighted in blue. The main changes are:

  • Assumptions of Theorem 3.1.1 are now stated in the main text, measured on all six conveyor tasks (new App. B.7), and relaxed for pairs that share a source timestep (new App. E.6); see Common Concern 1.
  • New baselines: GAE (λ∈{0.9,0.95}\lambda \in \lbrace 0.9, 0.95 \rbrace), CCA, and CSCA on all conveyor tasks (Figs. 3 and 17).
  • New task: SkillLuck from the CSCA paper (App. B.8).
  • Isolating adaptive mixing: nine constant mixtures with all of HDCA's hindsight components (App. C.2), causal-only variants with HDCA's ensemble (App. C.1), and the HCR of the causal component before mixing (new App. B.3).
  • Batch-size-corrected mixing rule, tested for batch sizes from 32 to 1024 (new App. C.3).
  • Compute comparison with COCOA-reward (new App. B.9).
  • Montezuma's Revenge extended from 2M to 5M samples, with the success criterion and statistical protocol stated in Sec. 4.4.
  • Positioning: CSCA, the SCM view of the contribution coefficient, and upside-down RL (Sec. 1, App. A, App. D).

Common Concern 1: Assumptions of Theorem 3.1.1 and their validity (Reviewers 9ZAS, qVTu, Anz7)

We thank the reviewers for this precise diagnosis; we agree with it.

(a) Main statement. Theorem 3.1.1 now states (A1) signal–error orthogonality and (A2) cross-pair orthogonality, and that the optimum is over deterministic per-pair weights.

(b) Pairs that share a source timestep. As Reviewers 9ZAS and qVTu point out, (A2) need not hold for such pairs, because εt,z\varepsilon_{t,z} and εt,z′\varepsilon_{t,z'} both contain the score gAtg_{A_t} of the same action. We quantified how much this matters and extended the theory to cover it.

  • Measurement (new App. B.7). We measured each term on all six conveyor tasks at checkpoints throughout training, using exact ground truth from dynamic programming. Across source timesteps, (A2) holds closely (Fig. 14): normalized correlations are below 0.020.02 for temporal residuals (εt,z\varepsilon_{t,z}) and about 0.10.1 or less for learned errors (Δt,z\Delta_{t,z}). Within a source timestep, as expected, the correlations are larger: about 0.60.6 between temporal residuals and 0.40.4–0.60.6 between learned errors. (A1) holds closely per pair after initialization; summed over pairs, the learned-error term is sometimes non-negligible, but negative. This negative term is conservative: it means that the learned error partly cancels the noise of the oracle causal gradient, so the learned estimator is more accurate than Eq. (7) assumes. By ignoring it, Eq. (7) gives the learned estimator less weight than the exact optimum would, i.e., it errs toward the unbiased temporal estimator.

  • Theory (new App. E.6). To cover the same-source case, Theorem E.6.2 drops orthogonality within a source timestep entirely, keeping (A1) and orthogonality across source timesteps. It shows that the oracle mixing coefficient of Eq. (7) is still the unique minimizer of an upper bound on the global gradient MSE. Section 3.1 therefore presents it as exactly MSE-optimal under (A1)–(A2), and as upper-bound-optimal when same-source orthogonality is relaxed.

  • Cost of the simplification (Fig. 16). Finally, we measured what ignoring the cross terms costs in practice. After initialization, the hybrid estimator with the oracle mixing coefficient of Eq. (7) has a gradient MSE three to six orders of magnitude below that of REINFORCE on all six tasks, and close to the exact optimum (which keeps every cross term) on four of them. On Ordered-keys, the excess peaks at about 30% during training and is close to zero by the end; on Poisoned-apples, it is noisy early and near zero afterwards.

(c) Proved vs. implemented rule (Anz7). We agree the main text understated this gap. Section 3.1 now presents Eq. (8) as a practical approximation to the oracle mixing coefficient and explicitly states the isotropic-score approximation as assumption (A3). As shown in the new App. B.7 (Fig. 15), with the MLP policy used in all experiments, oracle weights computed with the isotropic approximation and with the exact score Gram matrix differ by less than 0.010.01 in mean at every checkpoint. The remaining approximations (the ensemble mean for the oracle posterior, and the ensemble spread for the learned error) were already compared with their oracle counterparts in the initial submission (now App. B.6, Figs. 10–12).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.