acceptodds
Withdrawn from ICLR 2027

Replay-Conditioned Explanations: Disentangling Current, Historical, and Constraint Effects in Memory-Based Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Explaining recorded decisions of memory-based reinforcement-learning policies requires distinguishing current observations, retained history, and action constraints. We introduce replay-conditioned explanations, which bind each target decision to its policy and replay context and separately evaluate current-input replacement, historical intervention with suffix replay, and changes to action legality. The contributions of this paper are an executable query contract and an empirical characterization of the gap between computational validity and reference-dependent fault diagnosis. Reconstruction checks and optional exact Shapley attribution make the evaluated computation explicit. We validate query correctness on analytical mechanisms, reward-trained recurrent policies with 8- and 64-step horizons, and a native POPGym task with a three-step memory dependency. Public TimeSHAP agrees with exact event-level attribution under matched queries. In the independent-reference study, recovery falls from 100% with clean paired references to approximately 52% under the tested retrieval rule. A gate calibrated on clean episodes reduces clean-context modifications from approximately 48% to 4–5%, while recovery falls to 16–17%. Greedy search matches exhaustive recovery with fewer policy evaluations on these tasks. These results support explicit query specification and reference checks, while showing that computational correctness alone does not establish diagnostic reliability. Broader benchmark generalization and new industrial validation remain open.

Primary area:
interpretability and explainable AI
Keywords:
Explainable Reinforcement Learning, Memory‑based Reinforcement Learning, Intervention semantics, Replay‑conditioned explanation, Policy diagnosis, Action constraints

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kMH2OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I am not familiar with this area and cannot assess the submission as a specialist. As I understand it, the paper explains one recorded action of a memory policy by changing one declared input: the current observation, the past followed by a memory replay, or the legal actions. It does not propose a new explanation method. The checks agree with the formulas. Recovery is complete against a matched clean record and much weaker against retrieved records. The trained tasks are a short symbol sequence and a three-step recall exercise.

Critical strengths

The contribution is a procedure that explains one recorded action by changing one declared input, and by replaying memory when the past changes. It reports both full recovery against a matched clean record and much weaker recovery against retrieved records.

Critical weaknesses

The evaluation is limited to a symbolic sequence and a three-step memory task. I do not see what this procedure contributes to current robot reinforcement learning or large-model reinforcement learning. A single-factor replacement is also difficult to interpret: if the alternative input would have produced the same action in that situation, the action probability need not change, and the factor appears irrelevant.

Questions for authors

I am not familiar with this area and cannot assess the submission as a specialist.

Reviewer_GySmOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

In this paper, the authors try to explain single decisions of simple RL agents with memory by splitting the cause of the decision into what the agent sees in the environment right now, what it remembers from earlier steps in its memory, and which steps or actions are allowed in the future. The paper doesn't really invent a new explanation method. Instead, it gives a recipe for each of these three causes separately, with existing tools and some sanity checks. It then tests whether this recipe can find adversarial faults.

Critical strengths
  • Separating the three causes of what makes an agent take a decision is a good direction and the fact that action masking can hide what the network actually prefers is interesting
  • The fault injection setup to check if explnation recovers them seems like a good evaluation design
Critical weaknesses
  • The novelty seems limited because this couterfactual editing the current input, replaying memory, and then changing allowed set of actions separately seem established in the literature
  • Most of the experiments are very toy, with tiny GRUs and a three-step memory task, and no realistic agents are being used in practice today.
  • No comparison with existing literature and how this connects to or helps real-world use cases is also not talked about.
  • The paper is very, very hard to read and follow with an appendix that looks like internal notes.
Questions for authors
  • Did the authors try this work on transformer-based agents or tasks with longer memory?
  • In the paper authors suggest that the diagnostic work only in the presence of a clean comparison. In practice, how would such a good comparison example be obtained>
Reviewer_jBFrOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a framework for disentangling three categories of inputs in a specific reinforcement learning setting. In particular, when the policy is a function of the current observation, history, and the set of legal actions, their method uses counterfactual queries performed on each of the three categories to determine their contributions. In a set of simple synthetic environments, and a short POPGym task, the paper verifies that their method correctly identifies contribution errors in a learned model.

Critical strengths

None

Critical weaknesses
  • The paper's writing is extremely difficult to follow. There is a general lack of high-level coherence, making it difficult to follow the line of reasoning the author's are trying to convey, especially since the method and results are built on heuristics and intuition instead of theoretical or strong empirical results. The general vernacular and notation is also non-standard, as it does not seem to follow standard reinforcement learning or POMDP vocabulary or notation. If the work is not meant to be directed towards the RL literature or readers, it does not make clear what prior work or area of research it is meant to be part of it. If both of these are not true, and the paper is meant to be an introduction to a new problem statement, it also does not make that clear and lacks the necessary motivation to introduce a new problem setting.

  • The experiments are quite simple and the empirical contribution is generally weak given the lack of theoretical results. The paper only experiments on a few synthetic tasks, and a single POPGym environment. The experimental protocol is also unclear, and lacks a simple explanation or motivation of what the tasks are.

Questions for authors
  • It is not clear to me that the separation of "current observation", "retained history", and "action restriction" is a useful one. While I can understand the motivation to separate "current observation" and "history" contributions, "action restriction" does not seem like a standard policy input in traditional Reinforcement learning. Can you further motivate this separation and why it is important to make the separation between these three inputs in particular?

  • Why is the vocabulary and problem statement decidedly different from a standard memory-RL or POMDP setup? It would greatly benefit the readability of the paper to use a standard POMDP setup for example, and build the framework on top of that. For example, the "recording" seems to be analogous to a trajectory. In general, the language and notation in section 3, equations 1 and 2 are extremely difficult to follow. There are several technical words that are not typically used in RL that would require further contextualization if the authors decide that those words are necessary to the framing of the paper. For example, "recording", "version bundle", "schema", "semantic names", "reconstructed event logs", "diagnostic contexts", "reference plausibility", "physically inconsistent replacement". All of these terms appear within just a few lines, with no proper context, explanations or definitions in between.

  • Based on details from the appendix and certain portions of the main paper (Section 5.4), it seems like some of the motivation and intuitions come from a video game. If so, it would help the paper to be explicit and concrete about what video game in particular is of interest to the authors. If for some reason the game cannot be named, it can be described at a high level to help contextualize the paper's motivations and intuitions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.