acceptodds
Under review as a conference paper at ICLR 2027

Measuring the Local Action Utility of Visual History in Dreamer

Abstract

An agent can see the same image yet choose a different action because of its visual history. Does that change help it succeed? We measure this link by holding the current image fixed, changing past images, and evaluating the resulting first-action probabilities with the same continuing agent. A second measurement asks whether different current cues lead to different later outcomes when the first action is held fixed. Both use real simulator continuations and an explicit pairing of the recurrent policy's stochastic states. Controlled tasks distinguish information that is stored, information that changes actions, and information that improves outcomes. In Craftax-Classic, an exploratory DreamerV3 cohort motivates a prespecified confirmation on five new training seeds. Recent history improves local target success through the first-action choice by 5.13 and 5.36 percentage points under two masked continuation conditions. Every new model has a positive estimate, and joint model/world intervals remain positive after adjustment for three primary endpoints. An exploratory analysis links this gain to a shift toward the nominated tree; any-wood and native-reward gains remain unresolved. The separate continuation-square endpoint is also unresolved. The confirmed benefit concerns recent history with a visible current target, erased-history recipients, and masked later observations. A full-panel extension with clean recipient history and unmasked future images leaves its benefit unresolved. The measurements connect visual input to local action utility under explicit execution conditions.

Primary area:
reinforcement learning
Keywords:
world models, visual history, recurrent policies, memory, action utility, causal intervention, DreamerV3, Craftax-Classic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KJWzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether visual history actually helps an agent choose better actions. The authors keep the current image fixed, change the past images, and separate the first action from what the agent does afterward. They test the measurements on simple tasks and then apply them to Dreamer in Craftax. Recent history helps the agent collect wood from a specific tree, but improvements in overall reward and normal play remain unclear. The measurement idea is useful and the experiments are careful, but the confirmed finding is narrow, and its value for broader task performance remains unclear.

Critical strengths

The paper asks a meaningful question. Remembering past information, or changing an action because of it, does not necessarily help an agent succeed. Measuring what happens after the action gives a more practical view of whether that information is useful.

Critical weaknesses
  1. The practical benefit is still limited. History helps the agent collect wood from the selected tree, but it does not clearly help it collect wood overall or earn more reward. It may mainly change which tree the agent chooses. The study also focuses on one local decision in one game, with the target usually visible just one frame earlier, so it says little about longer-term memory.

  2. It is unclear how much the findings carry over to normal play. The main experiments hide relevant information in both the past and later observations. With full history and normal observations, the benefit is uncertain. Editing the images may also confuse the model, so the results could partly reflect damage from the edits rather than useful information in the original history.

Questions for authors
  1. Why does the improvement for the selected tree not lead to clearer gains in collecting any wood or earning reward? Does history mainly shift the agent from another tree to this one, or is there evidence that it helps with the broader task?

  2. What evidence separates the benefit of useful history from the harm caused by edited images? Are there controls that keep the inputs closer to normal observations while making comparable changes? How strongly do the current results support a benefit during normal play?

Reviewer_p4nuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces two behavioral intervention measurements for recurrent agents. The history-source gain changes past visual observations while holding the current image fixed, then evaluates how the resulting first-action distribution changes a local outcome using simulator continuations from a retained recipient state. The continuation-square measurement changes the current cue while holding the first action fixed, asking whether cue-dependent behavior remains in later actions. These measurements are intended to distinguish information being stored, information changing action selection, and information improving an outcome.

The authors calibrate the measurements on small tasks with known information paths and apply them to DreamerV3 agents in Craftax-Classic. In a prespecified confirmation cohort of five new training seeds, recent visual history improves the nominated-tree target-success measure by 5.13 and 5.36 percentage points under two masked continuation conditions, with adjusted model/world intervals just above zero. However, the continuation-square endpoint is unresolved in that cohort; any-wood success and native reward gains are unresolved; and a clean-recipient, unmasked extension also has intervals spanning zero. Thus the main evidence establishes a local first-action utility under explicit masked execution conditions, not a demonstrated improvement in full-episode return, task completion, sample efficiency, or a practical training intervention.

Critical strengths
  1. Precise causal/intervention framing. The paper cleanly separates changing the information available to the source policy from changing the recipient’s later execution. Holding the current image fixed while changing history, and separately fixing the first action while changing the current cue, is a useful way to distinguish first-action utility from later cue-dependent behavior. This is more informative than treating any history sensitivity or decodability as evidence that memory helps.

  2. A potentially useful measurement vocabulary and careful local evidence. The distinction among information being stored, information changing an action, and information improving an outcome could become a useful framework for recurrent RL analysis. The calibration tasks, paired simulator continuations, explicit stochastic-state coupling, and independent five-seed confirmation provide credible evidence for the narrower claim that recent history can improve a nominated local target-success measure under the specified masked conditions.

Critical weaknesses
  1. Practical significance and task-level competence are not established. The approximately five percentage-point gain is a local target-success effect for obtaining the nominated tree’s first wood and surviving a 16-action masked continuation. Any-wood success and native reward gains are unresolved, and the paper does not show improved full-episode return, task completion, sample efficiency, or a broader competence measure. The clean-recipient/unmasked extension also leaves target success, any-wood success, and native reward unresolved. Consequently, it is unclear whether the measure tracks the performance that matters for an RL agent or mainly captures a local redistribution toward a chosen target.

  2. The setting and validation are narrow for a broad conference claim. The main evidence uses one controlled Craftax-Classic decision geometry, two cohorts of five DreamerV3 models, a visible current target, erased-history recipients, masked later observations, a nominated-tree endpoint, and a coupling-dependent continuation estimator. The manuscript includes useful calibration, masking/content controls, and some history assays, but I did not find a direct feed-forward Dreamer baseline, a systematic shorter-history comparison, or a clean matched shuffled/corrupted-history comparison that would establish how the proposed quantity compares with simpler alternatives. There is also no demonstrated intervention in training or model selection based on the measure. The contribution may be valuable as a measurement proposal, but the current experiments do not yet establish broad generalization or practical utility.

Questions for authors

The following questions are intended to be material to the assessment rather than a list of general future-work suggestions:

  1. Relation to competence: Across anchors and models, does the history-source gain correlate with episode return, full-task completion, or another task-level success measure? In particular, when history is removed in states where the method predicts positive utility, does native execution show a measurable harm in full episodes rather than only in the nominated-tree local endpoint?

  2. Outcome and intervention sensitivity: How does the conclusion change under native reward, any-wood success, different continuation horizons, different target definitions, and unmasked future observations? Can the authors separate loss of useful historical information from harm caused by the masking/replacement edits themselves? The paper’s current controls leave this distinction unresolved.

  3. Baselines and simpler history manipulations: What happens with a feed-forward/no-history policy or an otherwise comparable policy with recurrent state reset, a systematic shorter-history sweep, and matched shuffled/corrupted-history controls? The small calibration reset controls and content edits are informative, but they do not fully establish how much the proposed measure adds over these simpler comparisons.

  4. Generalization: Does the effect replicate across additional tasks, decision geometries, environment conditions, recurrent architectures, or model families? The five confirmation training seeds are useful, but the paper itself notes that they are not enough to identify a broad training-population effect, and the clean-recipient/unmasked condition remains unresolved.

  5. Actionability: Can the measure be used prospectively to select states, history representations, or training/model-selection interventions that improve held-out full-task performance? A convincing demonstration that the measurement leads to an improvement would substantially strengthen the paper’s practical significance.

  6. Clarity of the contribution: Could the authors revise the introduction to state, in the first paragraph, that the paper measures whether visual history changes a recurrent Dreamer policy’s first action beneficially, rather than recognizing the action to take? It would also help to state explicitly how “local action utility” relates—and does not relate—to task-level competence, return, and memory usefulness.

Reviewer_htuuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explore whether visual history in recurrent world-model agents provides measurable benefits for decision-making, beyond merely being retained in latent representations or influencing action probabilities. The authors introduce two complementary measurements: history-source gain and continuation square. The framework is first validated using controlled tasks with known information pathways and is then applied to DreamerV3 agents in Craftax-Classic. An independent confirmation across five new training seeds finds that retaining recent visual history improves target-specific local success under two masked-continuation conditions.

Critical strengths
  1. The paper addresses an important question: information being represented does not mean that it improves decision outcomes. The proposed history-source gain explicitly evaluates the utility of history-induced action changes using real simulator continuations rather than relying on representation probes or gradient-based sensitivity measures.

  2. The authors validate the measurements on controlled policies with known information pathways, explicitly account for stochastic-state coupling, and use paired simulator continuations to reduce estimation noise. I particularly appreciate the distinction between exploratory analyses and the independently specified confirmation cohort. The positive history-source gains across five new DreamerV3 training seeds provide credible evidence for the narrowly defined local effect.

Critical weaknesses
  1. The demonstrated history utility is highly conditional and does not establish improvements in natural control performance. The primary history-source gain is evaluated by changing first-action probabilities while retaining a recipient whose history has been erased and whose subsequent observations are deliberately masked. Formally:
\mathbb{E}\left[ (p_{\mathrm{clean}}-p_T)^\top Q_c \right].$$ This is a well-defined counterfactual quantity, but it is not equivalent to the performance difference of a complete agent executing with versus without visual history. The confirmed gains concern collecting wood from a specific tree. The any-wood gains gains a little points. The full-panel extension using a clean-history recipient and unmasked future observations yields a target-success gain. So I think this paper therefore demonstrates a conditional local action-source benefit rather than a general improvement in control utility. 2. The experiment attributes action changes to visual history, but the relevant target is already visible in the current observation, and nearly every evaluated anchor contains a visible target exposure immediately before the decision. Consequently, the result primarily concerns very recent visual context rather than longer-term memory. Also, replacing or erasing historical image regions may introduce artificial visual patterns or misleading information. Although the authors investigate alternative masking and replacement schemes, these controls do not fully separate the loss of useful historical information from responses to edited observations.
Questions for authors
  1. Why does visual history improve success for the nominated tree but not show clear improvements in collecting any wood or overall reward?
  2. How can the authors distinguish the loss of useful historical information from unintended effects introduced by editing past images? Do the existing controls sufficiently address this concern?

Comments on the paper

1 comment

Dear Reviewers, 

We’re about halfway through the discussion period. Please read the authors’ rebuttal and assess whether it adequately addresses the key concerns raised in your review. We encourage you to leave a follow-up comment for the authors, indicating which concerns have been resolved and whether any important issues remain or require further clarification. Please update your review and scores as appropriate in light of the rebuttal and discussion.

Thank you for your time and participation. 

Best regards, 

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.