- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HDCA extends COCOA-reward with delay-conditioned contribution coefficients, learned kernel smoothing, and uncertainty-based mixing of temporal and causal gradient estimates. The paper reports strong results on credit-assignment benchmarks and symbolic Montezuma’s Revenge with behavior-cloning initialization, supported by theoretical analysis, DP gradient diagnostics, and ablations.
Critical strengths
-
The diagnostic tasks expose concrete failure modes, and the enumerable MDPs allow comparisons against DP ground truth for gradient bias, variance, and credit error (Figs. 2, 5, 6, 13).
-
The appendix is candid about the approximations behind Eq. (8) and the limits of variance reduction. The component ablations and oracle-mixing comparisons provide useful evidence beyond learning curves.
Critical weaknesses
-
The benefit of the particular adaptive mixing rule is not sufficiently isolated. Three things contribute to this.
-
(a) Figure 14 tests only one fixed hybrid weight, . Prop E.5.6 shows can never go below at , so that control sits at the floor of the admissible range, and no larger constant is tried.
-
(b) The causal-only variants in the same figure also reduce from 5 to 1, so ensemble size changes together with mixing and the two effects cannot be separated.
-
(c) The HCR evidence does not fill the gap. App. I.4 measures expected assigned credit after mixing. REINFORCE's expected credit at is , which is the ground truth , so its HCR is zero in expectation, and HDCA's HCR falls with the REINFORCE weight whatever the hindsight model does. Lower HCR therefore establishes better credit for the combined estimator, but does not show a more accurate hindsight model (Sec. 4.2, B.2). A diagnostic of the causal component before mixing would directly test that explanation.
These experiments support the overall method, but leave the added value of adaptivity open. I may have misread the HCR definition in (c); if so, I would reconsider this point.
-
-
Theorem 3.1.1 omits conditions required by its appendix counterpart.
-
(a) Theorem E.5.4 assumes signal–error orthogonality and cross-pair decoupling and optimizes over deterministic mixing vectors. These restrictions matter for the claimed global MSE optimum and should appear in the main statement.
-
(b) The practical input-dependent rule additionally relies on the approximations listed on p. 34, so the gap between what is proved and what is run is wider than the main text suggests.
-
The larger experiment also needs clearer reporting: the Reproducibility Statement says all Section 4 results use five seeds with 95% bootstrap intervals, but Fig. 4 and Sec. 4.4 do not say so themselves, and the task-success criterion on the y-axis is never defined. This limits how much weight I can place on the evidence beyond the small discrete benchmarks.
Questions for authors
-
How does adaptive HDCA compare with fixed , keeping , delay conditioning, smoothing, seeds, and budgets matched?
-
Can you report causal-component HCR before mixing at the same policies and checkpoints as mixed HCR? Under App. I.4, how is REINFORCE's expected credit defined, and does its population HCR equal zero?
-
Please clarify the scope of Theorem 3.1.1 and include the assumptions above. Were Figs. 9–11 run with tabular policies, as B.5 states, or the MLP in Table 4? If tabular, those experiments do not test the isotropic approximation in the main MLP setting.
-
For Montezuma, please define the task-success criterion in Fig. 4 and confirm that the five-seed, 95% bootstrap convention of the Reproducibility Statement applies to it.
-
What’s the relation to upside-down reinforcement learning (Schmidhuber, 2019; Srivastava et al., 2019), where the desired reward is an input viewed as a command, and the net learns actions that satisfy the command in hindsight? The hindsight model behind your contribution coefficient looks like the same object, trained the same way, and Appendix A does not discuss this line.
4 comments
Response to Reviewer Anz7 (2/2)
Question 3: Scope of Theorem 3.1.1; were Figs. 9–11 run with tabular policies, as B.5 states, or with the MLP in Table 4?
Response: For the scope, please see Common Concern 1. Thank you for catching the second point, and we apologize for the confusion. The sentence "Since all policies here are tabular softmax" was poorly worded. We did not mean that these experiments used tabular policies; we meant that we approximate the policy scores as isotropic, a property that holds exactly for tabular softmax policies, so that and share the same scale. All results in that section, including Figs. 9–11 (Figs. 10–12 in the revision), use the same MLP policy as the main experiments (Table 4; Table 5 in the revision). We have corrected the sentence, and Section 3.1 now states this approximation explicitly as assumption (A3). We also test it directly in App. B.7 with the same MLP policy: oracle weights computed with the isotropic approximation and with the exact score Gram matrix differ by less than in mean at every checkpoint, and the resulting MSE stays within a few percent on five tasks; on Poisoned-apples, the isotropic weights do slightly better early in training (Fig. 15).
Question 4: Montezuma's Revenge: the success criterion in Fig. 4 and the statistical protocol.
Response: Section 4.4 now defines a successful episode: the agent collects the key, unlocks a door, and exits the starting room. Fig. 4 follows the convention of the rest of Section 4 (mean over five seeds with 95% bootstrap intervals), which its caption now states.
Question 5: Relation to upside-down reinforcement learning.
Response: App. A now discusses this connection. The learned object is indeed closely related: our hindsight model predicts actions from retrospectively labeled outcomes and is trained by supervised learning on the agent's own experience, as a UDRL behavior function is. The two differ in conditioning and in use:
- Conditioning. UDRL conditions on a desired cumulative reward and horizon (a command); HDCA conditions on an individual rewarding outcome and its delay.
- Role. In UDRL, the model is the policy. In HDCA, it is never used to act: its posterior, divided by the current policy probability, gives a contribution coefficient that weights rewards in a separate policy-gradient estimator. HDCA thus separates two kinds of knowledge: which action to take, held by the policy, and how each action changes the probability of a later outcome, held by the hindsight model and used only to make the policy's learning more efficient. In UDRL, a single model plays both roles, and its aim is to act on commands rather than to make the learning of a separate policy more efficient.
- Consequence. UDRL's basic return command aggregates rewards over a horizon (although its broader framework supports richer commands), whereas HDCA attributes each reward event to earlier actions. HDCA's contributions (reliability-adaptive mixing with an unbiased temporal estimator, and delay conditioning) concern how far that attribution can be trusted, a question that does not arise when the model is the policy.
Gentle Reminder: Following up on our rebuttal and paper revisions
Dear Reviewer Anz7,
Thank you once again for your time and constructive feedback on our submission.
As the ICLR author-reviewer discussion period is drawing to a close, we wanted to kindly follow up to ensure you have had a chance to review our rebuttal responses and the revised manuscript. Based on your suggestions, we have updated the paper (with changes highlighted in the revision) and provided additional experiments and clarifications to address your concerns.
Please let us know if our responses have satisfactorily resolved your questions, or if there are any remaining points we can clarify before the discussion phase ends. We truly appreciate your thoughtful consideration and look forward to any further feedback.
Best regards,
The Authors
Thank you for the thorough revision; it resolves my concerns, and Fig. 19 shows no single constant matches HDCA across tasks. I'm raising my score to 3.
Response to Reviewer Anz7 (1/2)
We thank the reviewer for the careful reading, including of the appendix, and for the constructive feedback.
Weakness 1, Questions 1 and 2: Isolating the adaptive mixing rule.
Response:
(a) Fixed weights (Question 1). We agree that a single constant at the floor was not a sufficient control. App. C.2 (Fig. 19) now compares HDCA with nine constants, α∈{0.1,…,0.9}, on five tasks, with M=5, delay conditioning, kernel smoothing, seeds, and budget matched; with α=1 in App. C.1, this covers all requested values (0.7, 0.8, 0.9, 1). High causal weights (0.7–0.9) learn quickly, but several later lose performance, and the best constant changes with the task and training stage. HDCA attains the highest final performance on all five tasks without choosing a constant. In a related control (App. B.6), an annealed schedule approaches HDCA only when its decay rate is fitted, per task, to the oracle weights.
(b) Ensemble size confounded with mixing. Agreed. App. C.1 (Fig. 18) now adds two causal-only variants with HDCA's ensemble (M=5), with and without kernel smoothing; the smoothed one keeps all of HDCA's hindsight components and removes only the temporal term. Ensembling and smoothing help causal-only performance on several tasks but do not consistently recover HDCA's learning speed or final performance, so ensemble size does not explain the gain.
(c) HCR after mixing (Question 2). The reviewer's reading is correct. In the centered form of App. I.4, REINFORCE's expected credit at (s,a,z) is E[Rt+z∣St=s,At=a]−E[Rt+z∣St=s]=Cz⋆(s,a), so its population HCR is zero, and mixing alone can lower HCR. App. B.3 now states this and reports the requested diagnostic: Fig. 7 shows the HCR of the causal component before mixing, at the same policies and checkpoints, on all six conveyor tasks. The two curves nearly coincide (a gap of about 0.5 dB or less on five tasks and about 1–1.5 dB on Poisoned-apples), so HDCA's low HCR is already present in the learned causal component rather than produced by the REINFORCE weight.
Weakness 2: Theorem 3.1.1 omits conditions required by its appendix counterpart.
Response: Please see Common Concern 1: (a) for the assumptions now in the theorem, (b) for how far they hold and the relaxation in Theorem E.6.2, and (c) for the gap between the proved and the implemented rule.
Weakness 3: Reporting for Fig. 4 and Sec. 4.4.
Response: Please see our answer to Question 4 below.