acceptodds
Under review as a conference paper at ICLR 2027

Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

Abstract

Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and PPO online. Finally, in an offline evaluation of an industry generative recommender, considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, credit assignment, decision making, reward redistribution, recommender systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2PNNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Explanatory World Models (EWM), which leverage LLM-generated event dependencies to improve temporal credit assignment in reinforcement learning. A Conservative Hierarchical Operator recursively redistributes rewards along the inferred explanation graph while preserving the undiscounted episode return. The explanation graphs can also be cached and reused.

Experiments on Crafter, WordCraft, KuaiRand, and an industry-scale recommender demonstrate promising performance improvements in delayed-reward tasks and selected recommendation metrics.

Critical strengths
  1. The paper introduces explanation-based credit assignment using LLM-generated event dependencies and a recursive reward redistribution operator. The explanation graphs can be cached and reused without additional LLM calls during recursion, making the framework practically appealing.

  2. The method is evaluated across long-horizon RL and recommendation tasks, showing promising results in several settings. The ground-truth oracle comparison on WordCraft, anonymization ablations, and analyses of session length and per-step reward provide useful insights into when and why the method may be effective.

Critical weaknesses
  1. CHO preserves the undiscounted total episode reward, but this does not necessarily preserve discounted returns or per-step advantage estimates. In particular, retrospectively assigning terminal rewards to selected earlier actions may remove direct learning signals from other actions that are equally important for the final outcome. The paper does not sufficiently clarify whether standard PPO/GAE updates using redistributed rewards remain consistent with the original task objective, or under what additional conditions this consistency holds. This raises a fundamental concern about the theoretical justification of the proposed redistribution mechanism.

  2. The paper presents explanatory dependencies and hierarchical recursion as central contributions, but the main Crafter experiments use only Depth=1, while the deeper results in the appendix are limited to static attribution examples. Moreover, although WordCraft includes an oracle based on ground-truth dependencies, the paper does not directly measure the agreement between LLM-generated dependency graphs and the true dependencies, nor does it examine the stability of explanations across repeated generations. Consequently, the current experiments primarily demonstrate downstream performance improvements without sufficiently validating the accuracy, stability, or independent contribution of the proposed explanation structures.

Questions for authors
  1. CHO preserves the undiscounted episode return, whereas PPO uses discounted returns and advantage estimates for policy optimization. Could the authors clarify under what conditions policy gradient estimates based on redistributed rewards remain consistent with the original task objective? In particular, when multiple actions jointly contribute to an outcome, how does the method prevent important actions from losing their learning signals if the explanation graph assigns most of the reward to earlier events?

  2. The main Crafter experiments use Depth=1, while the Depth=2 results in the appendix are static examples. Could the authors compare Depth=1, 2, and 3 under the same cached explanation graphs and training configurations to demonstrate whether deeper recursive credit assignment provides additional benefits for policy learning?

  3. Since WordCraft provides ground-truth dependencies, could the authors directly evaluate the agreement between LLM-generated dependency graphs and the true dependencies, using metrics such as edge-level Precision, Recall, and F1? Additionally, how stable are the generated dependency edges and attribution weights across repeated LLM calls on the same trajectory, and how does this variability affect downstream policy learning?

  4. Although the paper compares against Uniform, LaRe, and a ground-truth dependency oracle on WordCraft, the specific contribution of LLM-generated explanation structures remains unclear. Could the authors keep the CHO operator fixed while comparing the original explanation graphs with alternatives such as randomly shuffled dependency edges? This would help isolate the value of LLM-based attribution from the redistribution mechanism itself.

  5. Figure 5 shows that some EWM variants improve AUC and Log Loss, while LaRe achieves the largest mean improvements across the four reported ranking metrics. Could the authors clarify which explanation variants outperform LaRe on which metrics and revise the broader claims in the abstract and conclusion accordingly?

  6. Section 5.2 uses A2C rather than PPO. Furthermore, the appendix attributes Vanilla's performance to optimizing only immediate recommendation rewards, although the baseline actually uses discounted returns, a value network, and GAE. Could the authors correct this interpretation and consistently identify the baseline used in the corresponding experiments? Additionally, the main improvement in this experiment comes from explanation-derived action gating rather than CHO-based reward redistribution. Could the authors more clearly distinguish the contributions of these mechanisms and avoid interpreting improvements from action constraints as direct evidence for the effectiveness of reward redistribution?

3 comments

Thank you for the detailed review. We address each point below.

Consistency with the task objective. CHO conserves the undiscounted return of every trajectory under any explanation graph: Σₜ r̃ₜ = Σₜ rₜ.

  • γ=1: every policy has the same expected return, so the optimal policies are unchanged.
  • γ<1: neither objective equals the task objective. Terminal-reward PPO already optimises E[γ^(T−1) R], which favours short episodes regardless of cause; redistribution replaces this with a bias that depends on where the explanation places reward. A bound, (1−γ^(L−1))·E[Σₜ|rₜ|], exists but is informative only for short horizons.
  • Per-step estimates: because EWM explains in hindsight, r̃ₜ can depend on later actions, so PPO/GAE gradient estimates are biased even at γ=1 (RUDDER avoids this by using the trajectory prefix only). EWM trades this bias for lower variance; we state this explicitly.

On jointly contributing actions:

  • The explanation graph allows several causes per phenomenon, each with its own weight, so jointly necessary actions can all be cited.
  • An uncited action is not cut off from learning. Its reward-to-go still includes all reward deposited at cited events after it.
  • The only actions that lose signal are those after the last cited cause, which is the intended effect.
  • Empirically, EWM's advantage over terminal-reward PPO holds on every Crafter horizon.

Depth. The same Crafter sweep has depth-2 runs. With LLM explanations at γ=0.99, final return for d1 vs d2 is:

L & D=1 & D=2 \\\\ \hline 150 & 3.44 & 3.25 \\\\ 300 & 3.56 & 3.65 \\\\ 500 & 4.01 & 3.15 \\\\ 1000 & 3.93 & 3.84 \end{array}$$ Terminal PPO scores 0.29–0.49 in the same cells. With the ground-truth Crafter tech tree, d1, d2 and d3 are statistically indistinguishable. Depth 1 therefore already captures most of the benefit in Crafter, whose dependency chains are short relative to the horizon. We will report these results and narrow the text: hierarchy is a capability of CHO that costs no extra LLM calls, not a demonstrated source of gains in Crafter. **Agreement with true dependencies.** We have existing analyses and will add them: * WordCraft: cause-level precision/recall/F1 against true recipe parents is 0.30/0.43/0.34 (hard) and 0.22/0.33/0.26 (very hard). * Crafter (1,000 trajectories, earlier offline pipeline): 72.0% of LLM-cited steps exactly match a ground-truth achievement unlock, 8.8% fall within ±2 steps, and 19.2% are spurious. So the explanations are far from perfect at the edge level. Yet on WordCraft-hard, EWM-LLM approaches the oracle. We read this as evidence that CHO tolerates imperfect explanations as long as most of the credited mass lands on genuinely causal events. We did not measure stability across repeated calls, and we will list it as a limitation. In training, explanations are cached per trajectory, so a given trajectory always receives the same credit. **Isolating the explanation from CHO.** The existing evidence that the content of the explanation matters, with redistribution held fixed: * Anonymisation (MovieLens, KuaiRand): the same LLM, prompt structure and redistribution with item identities removed degrade EWM. On MovieLens, anonymised EWM loses its advantage; on KuaiRand, it falls below the dense baseline. * Uniform redistribution: this conservative relocation with no explanation underperforms EWM on MovieLens. * Ground-truth oracle on WordCraft: this is the upper bound of the same operator. **KuaiRand claims.** The two metric families measure different things: AUC and log loss evaluate discrimination and calibration over all candidate pairs, while HR/NDCG/MRR evaluate the rank of the held-out item among sampled candidates. On AUC, five of six explanation variants beat LaRe (LB: +1.99pp vs +0.36pp over Uniform); on log loss, four of six do. On the ranking metrics LaRe leads, with HP closest on all four (e.g. HR@20 +2.50 vs +2.82pp) while also beating LaRe on AUC and log loss. We will report this per metric and restrict the abstract and conclusion accordingly. **A2C and gating.** Thank you; these are errors on our part. * The online KuaiRand experiment uses A2C. "PPO" in the introduction and conclusion is wrong and will be corrected. * The appendix sentence attributing Vanilla's behaviour to optimising immediate rewards is also wrong. We will remove it. * We will state clearly that this experiment evaluates explanation-derived action gating, not reward redistribution. The recommendation evidence for CHO itself is the MovieLens PPO experiment.

I thank the authors for their detailed response. I have carefully reviewed the rebuttal and find that the questions I raised have been addressed. The additional clarifications help clarify several aspects of the submission.

Thank you for engaging with our response and for the time you have given our paper. We are glad we were able to address the questions you raised in your review. As our response has helped clarify aspects of our submission, would you consider updating your assessment and raising your score? If not, are there outstanding questions you have that we could try and address? We remain available as the end of the rebuttal period draws near to help clarify any remaining questions about our submission.

Reviewer_bgYMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Explanatory World Models (EWM), a framework for addressing long-horizon credit assignment in reinforcement learning. Specifically, it uses a pretrained language model to identify earlier events that contributed to an observed outcome, instead of relying primarily on temporal proximity to distribute rewards. A hierarchical operator is then used to redistribute rewards assigned to events while preserving the undiscounted episode return. This approach is evaluated on Crafter, WordCraft, KuaiRand, MovieLens, and an industry-scale generative recommender.

Critical strengths
  • The problem is clearly motivated, and the idea of using explanations to identify relevant earlier decisions is both intuitive and interesting; the redistribution operator provides a useful mechanism for incorporating these explanations into RL.

  • The empirical results show substantial improvements across multiple environments and demonstrate interesting applications of explanation-based attribution

Critical weaknesses
  • It is difficult to understand exactly how an EWM is instantiated and integrated with varying envs, this practical construction is not clear, thus making the reproducibility and computational cost hard to assess. For example, what constitutes an event that can be explained in each environment? Is it a state, an action, an observed transition, or a higher-level achievement? How are these events represented to the language model, and how does the model construct an explanation graph and assign numerical weights? Also, there is no access to the complete prompts, which is important for reproducing and interpreting the performance difference in the six EWM explanation variants. A concrete end-to-end example (e.g., showing how a trajectory is extracted into events, how the LLM correctly identifies them, and the resulting graph) that goes beyond fig 7 would improve the paper

  • this also raises a further concern on the reliability of the attribution since the effectiveness of the proposed method relies on LLM identifying earlier events that genuinely contributed to the observed outcome, and there is no such systematic evaluation of failure modes by LLM; how sensitive performance is to errors in attribution. Again, it raises a generic question: what properties an EWM's explanations must satisfy to provide useful and reliable credit assignment? Knowing this would help establish when EWM provides trustworthy supervision. If possible, the authors could compare LLM-identified events with known dependencies in WordCraft or Crafter and examine whether important events are missed or irrelevant events are included.

  • (minor) A central contribution of this work is the hierarchical operator; however, the main Crafter training comparison uses redistribution depth 1, and it would be useful to compare different redistribution depths; and I also find it informative to include PPO with the original, non-delayed rewards as a reference on Crafter, and it would help understand how much performance is recovered by EWM relative to learning with more immediate feedback.

Questions for authors
  • Is hierarchical redistribution actually necessary? Since the main crafter exp uses depth 1 or 2, I wonder how sensitive the recursive redistribution is to the depth.

  • Can the authors discuss whether explanation-derived attribution remains reliable or necessary as the RL policy's behaviour changes?

  • What are the practical requirements and limitations of the explanatory model since the current implementation relies on a frontier language model?

  • Can the authors clarify what "We evaluate a goal objective that provides a single reward when the target entity is created" means, in connection with "We aggregate all rewards at the end of the episode," in sec 4.2? Does the environment terminate immediately when the target is created?

2 comments

Thank you for the positive and constructive review.

How an EWM is instantiated. We agree this was underspecified, and we will add the following.

  • An "event" table per environment. In every case the LLM explains phenomena by citing earlier trajectory steps by index:
    • Crafter: the evidence is the step log (action plus resulting inventory or environment change). Phenomena are every step with a state change (e.g. collecting wood, placing a table, making a pickaxe) plus one terminal reward event.
    • WordCraft: each step picks one entity, and every N-th pick triggers a mix. Phenomena are mixes that create a new entity plus the terminal reward event.
    • Recommendation: the phenomenon is the session outcome (e.g. the user's return activity); candidate causes are the items in the session, cited by position.
  • How the graph is built. The LLM sees the action/event log and the realised outcome. In one call it returns strict JSON listing, for each phenomenon, the earlier step indices that caused it, each with an importance in [0,1] (1–5 causes per phenomenon).
    • Invalid, missing or future-pointing indices are discarded.
    • CHO thresholds the weights at η, normalises them locally and recurses up to depth D with no further calls.
  • The full prompts (Crafter, WordCraft, the six recommendation variants) are in the appendix.
  • An end-to-end example extending Fig. 7: raw trajectory → event list → LLM JSON → graph → redistributed rewards.
  • LLM call counts. A 5M-step Crafter run takes roughly 33k calls at L=150 and 6k at L=1000, since one call covers one episode.

Reliability of attributions. We have two existing agreement analyses against ground truth, which we will add to the paper:

  • WordCraft: cause-level precision/recall/F1 against true recipe parents is 0.30/0.43/0.34 (hard) and 0.22/0.33/0.26 (very hard).
  • Crafter (1,000 trajectories, earlier offline pipeline): 72.0% of cited steps exactly match a true achievement unlock, 8.8% are within ±2 steps, and 19.2% are spurious.

Errors are common at the edge level, yet EWM-LLM approaches the oracle on WordCraft-hard. We take this as a first answer to your question about which properties matter: precision on the steps that receive most of the mass matters more than recall of every dependency. The anonymisation ablations show the opposite case: when the LLM lacks the semantic knowledge to explain (item identities removed), EWM degrades, falling below the dense baseline on KuaiRand. We will add a paragraph on this and on the failure modes we observed. For example, in Crafter, recall is uneven across achievements: some, such as making a wood sword, are almost never cited even though they occur in nearly every trajectory.

Depth and the dense-reward reference.

  • Depth: in our existing Crafter sweep, final return for d1 vs d2 is 3.44/3.25, 3.56/3.65, 4.01/3.15 and 3.93/3.84 at L=150/300/500/1000, against 0.29–0.49 for terminal-reward PPO. With the ground-truth tech tree, d1, d2 and d3 are statistically indistinguishable. Depth 1 already captures most of the gain in Crafter; we will add these results. Note that the relevance of depth depends on the environment chosen and current available environments are limited on this aspect due to the problematics of RL on sparse rewards.
  • Dense reward: an existing PPO run on the original per-achievement reward (L=300) reaches 4.98, against 3.56 (d1) and 3.65 (d2) for EWM and 0.32 for terminal-reward PPO. EWM thus recovers about 70% of dense-reward performance from a single terminal signal. We will add this reference to the Crafter figure.

Is hierarchy necessary? In Crafter, no (see Depth above). With the ground-truth tree, d2 helped in some WordCraft-hard oracle runs. We will present hierarchy as a capability rather than a demonstrated necessity.

Reliability and necessity as the policy changes.

  • Reliability: each episode is explained on its own, so attribution follows the current policy. In Crafter, EWM's return holds up as L grows while LaRe's falls (Fig. 3), so explanations stay useful on longer trajectories.
  • Necessity: we did not test switching EWM off. We expect it to matter most early, when the critic cannot yet link terminal reward to early actions. Once the critic predicts returns well, bootstrapping can carry credit, and annealing redistribution towards the raw reward would also remove the hindsight bias in per-step estimates. Our operator already supports a redistribution fraction; we will discuss this as future work.

Practical requirements. EWM needs events that can be described in text, a model with the relevant prior knowledge, and one LLM call per episode. We will add these to the limitations.

WordCraft reward and termination. Thank you; the episode terminates as soon as the target entity is created and pays a single +1, so that reward is already terminal. We will clarify.

AuthorsOpenReview ↗

Follow-up on reliability of attributions

We have two additions to our earlier answer. Shuffled-edge control (WordCraft). To isolate the explanation from the operator, we kept CHO and the amount of redistributed reward fixed, and replaced the LLM's cited causes with randomly chosen earlier events. With the LLM's graphs, 98% of the redistributed return lands on true recipe parents; with shuffled edges, about 10% does. Unlike the cause-level precision/recall in our first response, which counts every cited cause equally, this measure is weighted by the credit each cause receives after thresholding, so low-importance spurious citations contribute little. This measures where credit lands rather than final return, and is consistent with our earlier point that what matters most is precision on the steps that receive most of the credit. Failure modes in Crafter. Expanding on the failure modes we mentioned: the main failure is omission rather than spurious attribution. 81% of cited steps are within ±2 steps of a true achievement unlock, so most cited causes are right, while some achievements (e.g. making a wood sword) are almost never cited.

We will add both to the paper, alongside the agreement analyses.

Reviewer_7jr7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Explanatory World Models (EWM), which use Claude Sonnet 4.5 to retrospectively analyze complete trajectories and outcomes and identify earlier events that plausibly contributed to the observed reward. The proposed Conservative Hierarchical Operator then recursively redistributes the reward to these events while preserving the undiscounted episodic return. Experiments on Crafter, WordCraft, KuaiRand, and recommendation tasks demonstrate the empirical effectiveness of the proposed framework.

Critical strengths
  1. Important Problem. Credit assignment under long horizons and sparse rewards is a fundamental challenge in reinforcement learning and long-term recommendation. Leveraging the prior knowledge of large language models for retrospective attribution is an interesting direction.

  2. Intuitive Framework. The method converts LLM-generated explanations into an attribution graph and uses CHO to perform hierarchical reward redistribution. The overall framework is intuitive and can be integrated with existing reinforcement learning algorithms.

Critical weaknesses
  1. The Correctness of LLM-Generated Attributions Is Not Validated. The method relies on Claude Sonnet 4.5 to generate retrospective explanations, but the paper provides no direct validation that the identified events are the actual causes of the observed outcomes.

  2. The Effects of Explanation Quality and Reward Relocation Are Not Properly Isolated. CHO preserves the undiscounted episodic return, but the actual training algorithms use discounted returns. Moving rewards to earlier events therefore changes the discounted objective even when the total reward remains unchanged. It is consequently unclear whether the observed improvements arise from accurate EWM explanations or simply from relocating delayed rewards to earlier time steps.

  3. The Crafter and WordCraft experiments mainly compare EWM with PPO and LaRe. Important long-horizon credit-assignment and reward-redistribution methods, such as RUDDER and Align-RUDDER, are omitted.

Questions for authors

Please refer to weaknesses

2 comments

Thank you for the review.

Correctness of the attributions. We have direct measurements against ground truth and will add them to the paper:

  • WordCraft (true recipe parents): cause-level precision/recall/F1 of 0.30/0.43/0.34 (hard) and 0.22/0.33/0.26 (very hard).
  • Crafter (true achievement unlocks): 72.0% of cited steps are exact matches, 8.8% are within ±2 steps and 19.2% are spurious.
  • The attributions are informative but imperfect. The policy-learning results suggest CHO tolerates this level of error: on WordCraft-hard, EWM-LLM approaches the ground-truth oracle that uses the same operator.

Explanation quality vs. reward relocation. This is an important distinction. CHO conserves each trajectory's undiscounted return, so with γ=1 every policy's expected return is unchanged. With γ<1, and through hindsight dependence, it does change the per-step targets PPO sees. Both effects are explained in point 1 of our general response, and we add them to the paper in a new paragraph, under Section 3.2.

Several existing comparisons hold relocation fixed and vary the explanation:

  • LaRe also redistributes the terminal reward to earlier steps while conserving the return. EWM exceeds it on every Crafter horizon, by about 2×.
  • Uniform redistribution moves reward earlier with no explanation and underperforms EWM on MovieLens.
  • Anonymisation keeps the same LLM, prompt structure and operator but removes item semantics. This degrades EWM (below the dense baseline on KuaiRand), so the gains depend on the content of the explanation.

RUDDER and Align-RUDDER. We chose LaRe as the representative return-decomposition baseline: it is the most recent method in the family, and like EWM it uses an LLM. We agree that RUDDER and Align-RUDDER are the canonical references. We will expand the related-work comparison, in particular RUDDER's return-equivalence result, which our new proposition builds on, and Align-RUDDER's use of demonstrations versus our use of prior knowledge. We will state that a direct empirical comparison is left for future work.

AuthorsOpenReview ↗

Follow-up: RUDDER baselines and an explanation-isolation control.

Since our first response, we have run two additional experiments.

RUDDER and Align-RUDDER Crafter: same architecture, terminal reward, 5M-step budget and 3 seeds. Since no expert demonstrations exist, Align-RUDDER uses the agent’s highest-return episodes as demonstrations. Across L ∈ {150, 300, 500, 1000}, both stay close to terminal-reward PPO (RUDDER 0.30–0.68, Align-RUDDER 0.32–0.38), far below EWM (≈3.6–3.9, Fig. 3). WordCraft 3-mix: same setup as Fig. 4 (6M steps, 3 seeds). RUDDER reaches 0.71 ± 0.21 (one of three seeds solves the task). Align-RUDDER does not learn (0.00), since it rarely obtains the successful episodes it needs as demonstrations. EWM-LLM reaches ≈0.95.

Explanation quality vs. reward relocation. In addition to the LaRe, uniform and anonymisation comparisons, we ran a control on WordCraft that isolates the explanation from the operator. We kept CHO and the amount of redistributed reward fixed, and replaced the LLM’s cited causes with randomly chosen earlier events. With the LLM’s graphs, 98% of the redistributed return lands on true recipe parents; with shuffled edges, about 10% does. Unlike the cause-level precision/recall in our first response, which counts every cited cause equally, this measure is weighted by the credit each cause receives after thresholding, so low-importance spurious citations contribute little. The same amount of reward is moved in both cases, so the gains depend on which events are credited, not on relocation alone. This measures where credit lands rather than final return. We will add both baselines to Figs. 3 and 4, alongside the related-work comparison described in our first response, and the shuffled-edge control to the paper.

Comments on the paper

1 comment

We thank the reviewers for their careful reading. All three reviewers ask the same two questions: 1) what redistribution does to the RL objective; and 2) whether the gains come from the explanations or just from moving reward earlier. We answer both below.

1) What does redistribution do to the RL objective? For every trajectory and every explanation graph, CHO conserves the undiscounted return: Σₜ r̃ₜ = Σₜ rₜ.

  • With γ=1: Every policy keeps its expected undiscounted return, so the set of optimal policies is unchanged. We will state this as a proposition in Section 3.2.
  • With γ<1: Neither objective equals the task objective. Terminal-reward PPO with γ<1 already optimises E[γ^(T−1) R], which favours short episodes regardless of what caused the outcome. Redistribution replaces this bias with one that depends on where the explanation places reward.
  • Per-step estimates: EWM explains in hindsight, so r̃ₜ can depend on actions taken after step t. PPO/GAE gradient estimates are therefore biased even at γ=1; RUDDER avoids this by redistributing from the trajectory prefix only. EWM trades this bias for a lower-variance signal concentrated on cited causes, and our experiments test that trade-off. We will state this explicitly in the paper.

2) Do the gains come from the explanations, or just from moving reward earlier? Several comparisons move reward earlier just as EWM does but change or remove the explanation:

  • LaRe learns a redistribution that also conserves the return, and EWM beats it on every Crafter horizon. Moreover, uniform redistribution moves reward earlier with no explanation and underperforms EWM on MovieLens.
  • Item anonymisation keeps the same LLM, prompt structure and operator but removes item semantics. It degrades EWM on MovieLens and KuaiRand.
  • The ground-truth oracle on WordCraft is the upper bound of the same operator, and EWM-LLM approaches it on the hard split.

Overall, the evidence indicates that the LLM’s explanations are doing the heavy lifting: the gains come from what the explanations say, not merely from moving reward earlier.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.