acceptodds
Under review as a conference paper at ICLR 2027

Measuring What Must Persist: A MiniGrid Diagnostic of State-Formation Scaffolding

Abstract

Partially observable RL agents must decide what information should persist as internal state. We treat this as a measurement problem on a controlled MiniGrid cue_recall diagnostic, using a slot module (ASF) as an instrument: keep/forget gates, cue-decode readouts, and causal memory interventions—not as a general-purpose memory architecture. Unless noted, training-run success is the last 100 episodes of a 50k-step PPO log (3-seed); frozen-policy and distillation numbers are 200-episode evals. With privileged cue-identity losses and event-gated writes, ASF reaches 84.7% vs. 29.7% feedforward PPO and 35.0% GRU; the training auxiliary cue-head decodes the cue from memory at 100%, and zero-masking memory at decision drops success by 21–64pp across privileged Claims A–D. The same template, with delay, distractors, and slot budget varied one at a time, yields 262 selected runs in the submission inventory across four memory pressures. Retention loss (−34.3pp at b=40) and cue-identity auxiliary losses (−11.3pp vs. Claim A full) are load-bearing; the compression loss is not. Removing both cue losses and privileged writes collapses ASF to 3-way chance (30.7%), alongside a retuned GRU (31.3%): the 84.7% result measures scaffolding, not slots versus recurrences. Replacing simulator cue_id with a pixel readout—no probe labels at train or test—yields 90.0% from-scratch ASF. Distilling that student's write vector, with no discrete cue_id at test and no privileged teacher, is within 0.3pp of a 200-episode eval of the student (92.8% vs. 92.5%; n=3, not an equivalence test). Writing observation tokens remains at 3-way chance. The load-bearing object is an identity-shaped write channel; PPO does not invent it (23.0% from-scratch write-head). Under our protocols the identity write channel is required for the observed gains, but its effect is architecture-dependent and high-variance: the same identity write yields 72.7%±13.4 (88/63/67) in a Transformer (w=16) vs. 37.0% in a GRU. We report this isolation rather than claiming a new memory module.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, partial observability, memory, MiniGrid, diagnostic evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_PQMxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper uses a MiniGrid cue-recall task to study how training supervision and memory-write mechanisms affect delayed decision-making.

Critical strengths

The ablations explicitly distinguish gains from privileged supervision from those attributable to the memory architecture

Critical weaknesses
  1. Formatting: The title and section headings do not follow ICLR template.
  2. Limited baseline evaluation: Comparisons primarily involve feedforward PPO and PPO+GRU, although Transformer results are also included.
  3. Narrow contribution: Evidence centers on one cue-recall task family. Removing the privileged write pipeline reduces ASF to chance-level performance, while broader task results remain preliminary or negative. The diagnostic findings therefore have limited demonstrated generality.
Questions for authors
  1. Can the authors include additional baselines, such as PPO+LSTM, a Transformer with a context window covering the full cue-to-decision interval, and an external-memory agent, under matched supervision and training budgets?
  2. Can the diagnostic conclusions be replicated beyond the cue-recall task family?
Reviewer_MibQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes ASF, a slot-based memory for RL tasks. ASF holds episodic and semantic slots, and allows writes and reads to them during an episode. The authors train ASF using PPO.

ASF is evaluated on a cue-recall task, a delay robustness task, and a distractor sensitivity task. The agent success rate is reported across these tasks.

Critical strengths

None

Critical weaknesses

This paper appears entirely AI-generated and is nearly impossible for human readers to understand. The structure of the paper is nonsensical and it invents new terminology without defining it. I cannot recommend acceptance without a full rewrite of this paper. In its current state, it is unreadable and offers no benefit to the academic community.

Questions for authors

The paper is not in a state such that I can ask meaningful questions.

Reviewer_uhcLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies which forms of supervision and memory-writing support enable delayed cue recall in a controlled MiniGrid task. It uses a slot-memory module, ASF, as a diagnostic instrument. Privileged ASF achieves 84.7% success, while removing its supporting pipeline reduces performance to approximately chance. Observation-derived cue identities and write-vector distillation recover high performance.

Critical strengths
  1. The paper asks a useful question: whether successful memory-based control comes from the architecture or from the information and supervision supplied to it. The authors acknowledge the privileged nature of their strongest comparisons, distinguish trained auxiliary heads from independent probes, and report negative findings.
  2. One observation is interesting: a linear probe on the frozen GRU hidden state decodes the cue at 91.4% during the blank phase, yet control stays at chance. This could motivate a proper study of the gap between decodability and control in memory-based RL.
Critical weaknesses
  1. Presentation and completeness. The manuscript does not read as a finished paper. Section A.7 ("Algorithm") is an empty heading. There are unresolved references to "Appendix 4" (Sec. 5 and the AI-use statement) and "Section B". A diagnostic is "under development but not yet stable" (Sec. 6). Workflow artifacts appear throughout the text: directory paths (runs/poster_fair/, runs/followup/), a find command used to count results files, "post-submission follow-ups," "supplementary investigation log," "smokes," "dagger band," "re-anchor." The prose is heavily hedged and repetitive, e.g., "84.7" appears 22 times.

  2. Limited mechanistic insight and unclear supervision. The privileged setting writes simulator-derived cue identity into memory, directly supplying the key variable needed to determine the correct action. High performance is therefore partly an expected consequence of this design. Removing the supporting pipeline reduces ASF to 30.7%, near the retuned GRU’s 31.3%, but multiple components change simultaneously, preventing attribution to cue-identity writes specifically. Furthermore, the abstract claims that the pixel student uses “no probe labels at train or test,” while Table 11 indicates that it uses cue loss. These statements could be compatible if targets are derived from observations, but their source and the classifier’s training procedure are insufficiently specified.

3, Incomplete reporting of central diagnostic results. The claimed 21–64 percentage-point zero-mask decrease across Claims A–D lacks a corresponding breakdown: among those claims, only Claim A has tabulated intervention results. The pixel student’s 100% probe accuracy and 62–75-point zero-mask decrease appear only as brief statements in Appendix B. Similarly, the GRU’s 67% success at b=0b=0 lacks per-seed results or uncertainty estimates and is absent from Table 15’s purportedly comprehensive configuration list. Section 5.3 references an appendix table of active-slot usage across delays that is not provided, and Appendix A.3 lists shuffle and wrong-cue checks without reporting their outcomes. These omissions limit verification of the paper’s central mechanism claims.

Questions for authors
  1. Pixel-student supervision: What observation representation, cue extractor, supervision targets, and frozen/trainable components produce the 90.0% and 92.8% results? In particular, how is Table 11’s “cue loss: yes” consistent with the claim of “no probe labels at train or test”?
  2. Validity and completeness of memory interventions: What tensors are modified during zero-masking, and how is the operation defined under Equation 11 when K=1K=1? Please provide the missing intervention results for Claims B–D and the pixel students, together with the shuffle and wrong-cue checks.
  3. Memory failure versus optimization failure: Given that GRU achieves only 67% success at b=0b=0, what do PPO, GRU, and ASF achieve on this Markov setting under the same 50k-step budget? What evidence distinguishes failures of memory from failures of optimization or action mapping?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.