- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Explanatory World Models (EWM), which leverage LLM-generated event dependencies to improve temporal credit assignment in reinforcement learning. A Conservative Hierarchical Operator recursively redistributes rewards along the inferred explanation graph while preserving the undiscounted episode return. The explanation graphs can also be cached and reused.
Experiments on Crafter, WordCraft, KuaiRand, and an industry-scale recommender demonstrate promising performance improvements in delayed-reward tasks and selected recommendation metrics.
Critical strengths
-
The paper introduces explanation-based credit assignment using LLM-generated event dependencies and a recursive reward redistribution operator. The explanation graphs can be cached and reused without additional LLM calls during recursion, making the framework practically appealing.
-
The method is evaluated across long-horizon RL and recommendation tasks, showing promising results in several settings. The ground-truth oracle comparison on WordCraft, anonymization ablations, and analyses of session length and per-step reward provide useful insights into when and why the method may be effective.
Critical weaknesses
-
CHO preserves the undiscounted total episode reward, but this does not necessarily preserve discounted returns or per-step advantage estimates. In particular, retrospectively assigning terminal rewards to selected earlier actions may remove direct learning signals from other actions that are equally important for the final outcome. The paper does not sufficiently clarify whether standard PPO/GAE updates using redistributed rewards remain consistent with the original task objective, or under what additional conditions this consistency holds. This raises a fundamental concern about the theoretical justification of the proposed redistribution mechanism.
-
The paper presents explanatory dependencies and hierarchical recursion as central contributions, but the main Crafter experiments use only Depth=1, while the deeper results in the appendix are limited to static attribution examples. Moreover, although WordCraft includes an oracle based on ground-truth dependencies, the paper does not directly measure the agreement between LLM-generated dependency graphs and the true dependencies, nor does it examine the stability of explanations across repeated generations. Consequently, the current experiments primarily demonstrate downstream performance improvements without sufficiently validating the accuracy, stability, or independent contribution of the proposed explanation structures.
Questions for authors
-
CHO preserves the undiscounted episode return, whereas PPO uses discounted returns and advantage estimates for policy optimization. Could the authors clarify under what conditions policy gradient estimates based on redistributed rewards remain consistent with the original task objective? In particular, when multiple actions jointly contribute to an outcome, how does the method prevent important actions from losing their learning signals if the explanation graph assigns most of the reward to earlier events?
-
The main Crafter experiments use Depth=1, while the Depth=2 results in the appendix are static examples. Could the authors compare Depth=1, 2, and 3 under the same cached explanation graphs and training configurations to demonstrate whether deeper recursive credit assignment provides additional benefits for policy learning?
-
Since WordCraft provides ground-truth dependencies, could the authors directly evaluate the agreement between LLM-generated dependency graphs and the true dependencies, using metrics such as edge-level Precision, Recall, and F1? Additionally, how stable are the generated dependency edges and attribution weights across repeated LLM calls on the same trajectory, and how does this variability affect downstream policy learning?
-
Although the paper compares against Uniform, LaRe, and a ground-truth dependency oracle on WordCraft, the specific contribution of LLM-generated explanation structures remains unclear. Could the authors keep the CHO operator fixed while comparing the original explanation graphs with alternatives such as randomly shuffled dependency edges? This would help isolate the value of LLM-based attribution from the redistribution mechanism itself.
-
Figure 5 shows that some EWM variants improve AUC and Log Loss, while LaRe achieves the largest mean improvements across the four reported ranking metrics. Could the authors clarify which explanation variants outperform LaRe on which metrics and revise the broader claims in the abstract and conclusion accordingly?
-
Section 5.2 uses A2C rather than PPO. Furthermore, the appendix attributes Vanilla's performance to optimizing only immediate recommendation rewards, although the baseline actually uses discounted returns, a value network, and GAE. Could the authors correct this interpretation and consistently identify the baseline used in the corresponding experiments? Additionally, the main improvement in this experiment comes from explanation-derived action gating rather than CHO-based reward redistribution. Could the authors more clearly distinguish the contributions of these mechanisms and avoid interpreting improvements from action constraints as direct evidence for the effectiveness of reward redistribution?
3 comments
I thank the authors for their detailed response. I have carefully reviewed the rebuttal and find that the questions I raised have been addressed. The additional clarifications help clarify several aspects of the submission.
Response
Thank you for engaging with our response and for the time you have given our paper. We are glad we were able to address the questions you raised in your review. As our response has helped clarify aspects of our submission, would you consider updating your assessment and raising your score? If not, are there outstanding questions you have that we could try and address? We remain available as the end of the rebuttal period draws near to help clarify any remaining questions about our submission.
Thank you for the detailed review. We address each point below.
Consistency with the task objective. CHO conserves the undiscounted return of every trajectory under any explanation graph: Σₜ r̃ₜ = Σₜ rₜ.
On jointly contributing actions:
Depth. The same Crafter sweep has depth-2 runs. With LLM explanations at γ=0.99, final return for d1 vs d2 is:
L & D=1 & D=2 \\\\ \hline 150 & 3.44 & 3.25 \\\\ 300 & 3.56 & 3.65 \\\\ 500 & 4.01 & 3.15 \\\\ 1000 & 3.93 & 3.84 \end{array}$$ Terminal PPO scores 0.29–0.49 in the same cells. With the ground-truth Crafter tech tree, d1, d2 and d3 are statistically indistinguishable. Depth 1 therefore already captures most of the benefit in Crafter, whose dependency chains are short relative to the horizon. We will report these results and narrow the text: hierarchy is a capability of CHO that costs no extra LLM calls, not a demonstrated source of gains in Crafter. **Agreement with true dependencies.** We have existing analyses and will add them: * WordCraft: cause-level precision/recall/F1 against true recipe parents is 0.30/0.43/0.34 (hard) and 0.22/0.33/0.26 (very hard). * Crafter (1,000 trajectories, earlier offline pipeline): 72.0% of LLM-cited steps exactly match a ground-truth achievement unlock, 8.8% fall within ±2 steps, and 19.2% are spurious. So the explanations are far from perfect at the edge level. Yet on WordCraft-hard, EWM-LLM approaches the oracle. We read this as evidence that CHO tolerates imperfect explanations as long as most of the credited mass lands on genuinely causal events. We did not measure stability across repeated calls, and we will list it as a limitation. In training, explanations are cached per trajectory, so a given trajectory always receives the same credit. **Isolating the explanation from CHO.** The existing evidence that the content of the explanation matters, with redistribution held fixed: * Anonymisation (MovieLens, KuaiRand): the same LLM, prompt structure and redistribution with item identities removed degrade EWM. On MovieLens, anonymised EWM loses its advantage; on KuaiRand, it falls below the dense baseline. * Uniform redistribution: this conservative relocation with no explanation underperforms EWM on MovieLens. * Ground-truth oracle on WordCraft: this is the upper bound of the same operator. **KuaiRand claims.** The two metric families measure different things: AUC and log loss evaluate discrimination and calibration over all candidate pairs, while HR/NDCG/MRR evaluate the rank of the held-out item among sampled candidates. On AUC, five of six explanation variants beat LaRe (LB: +1.99pp vs +0.36pp over Uniform); on log loss, four of six do. On the ranking metrics LaRe leads, with HP closest on all four (e.g. HR@20 +2.50 vs +2.82pp) while also beating LaRe on AUC and log loss. We will report this per metric and restrict the abstract and conclusion accordingly. **A2C and gating.** Thank you; these are errors on our part. * The online KuaiRand experiment uses A2C. "PPO" in the introduction and conclusion is wrong and will be corrected. * The appendix sentence attributing Vanilla's behaviour to optimising immediate rewards is also wrong. We will remove it. * We will state clearly that this experiment evaluates explanation-derived action gating, not reward redistribution. The recommendation evidence for CHO itself is the MovieLens PPO experiment.