acceptodds
Withdrawn from ICLR 2027

LOOPREPLAY: Reliability- and Diameter-Adaptive Sequence Replay for Rapid Recovery After Reward Relocation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Under conditions where rewards are sparse and spatially distributed, three failures arise in sequence replay. Sweeping stalls due to iterative states, low-confidence stochastic edges contaminate propagation values, and a fixed replay horizon becomes mismatched with the scale of the learned graph. LOOPREPLAY introduces an event-based controller. This controller regulates access based on reward outcomes and signed prediction errors, enforces non-cyclic scans prioritized by empirical transition confidence, uses residual depletion as a termination condition, and determines propagation depth based on the diameter of the learned graph. All rules were fixed prior to running the validation experiment, which consisted of 12 conditions. This experiment covered three topologies (ring, bridge, cross), two sizes (9, 17), two slip probabilities (0.08, 0.22), and a balanced reward-shifting geometry. Compared with the stronger of prioritized replay and Dyna in each condition, LOOPREPLAY improved the average success AUC by +0.071 (95% hierarchical confidence interval [0.024, 0.126]) at matched replay compute per online step. The gain arises after the reward moves, where post-relocation success rose from 0.72 to 0.88. LOOPREPLAY remained within 0.05 AUC of the best baseline across all 12 conditions, and it also outperformed standard Dyna-Q, prioritized sweeping, and backward trajectory replay. The completion rate for backward sweeps exceeded 95% across all conditions, and the selection rate for low-confidence edges was less than 10%. The same rule improved all six conditions at the unseen size of 25. These results indicate that sequence reliability and the learned structural scale are joint control variables in replay-driven credit assignment.

Primary area:
applications to neuroscience & cognitive science
Keywords:
hippocampal replay, experience replay, model-based planning, reinforcement learning, credit assignment, replay horizon calibration, dynamic reward contexts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jGPcOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In reinforcement learning (RL), propagating sparse rewards across long trajectories via one-step Temporal Difference (TD) learning is notoriously slow. While replaying ordered sequences of transitions can rapidly carry value across multiple steps, doing so in non-stationary, stochastic environments (where rewards relocate and actions occasionally "slip") exposes severe control burdens.

LOOPREPLAY solves this by introducing an event-triggered graph controller that pairs one specific operator to each failure mode, alongside a strict context-isolation invariant. Tested across 12 held-out navigation conditions (varying topologies, sizes, and slip noise) and 6 unseen larger-scale conditions, LOOPREPLAY improves post-relocation success AUC from ~0.72 to ~ 0.87 while using slightly fewer replay updates per step.

Critical strengths

While all methods achieve similar acquisition performance before rewards move, LOOPREPLAY drastically accelerates credit reassignment after reward relocation.

OOPREPLAY achieves these performance gains without inflating compute budgets; its replay updates per online step remained comparable to or lower than the baselines

Standard model-free replay (PER, Backward Replay) and simple tabular models entangle transition dynamics with reward values. When a reward moves, an agent faces a dilemma: if it flushes its memory buffer to forget the old reward, it also forgets the topology of the maze; if it keeps its buffer, old reward transitions contaminate backups.LOOPREPLAY's Context-Safe Structural Fallback (N~≠zd\tilde{\mathcal{N}}_{\neq z}^d) acts like a lightweight, algorithmic version of the Successor Representation (Dayan, 1993) or Tolman's cognitive map. By advancing context ID zz upon reward omission (ω\omega) and stripping reward labels from cross-context edges, it preserves 100% of the structural connectivity learned in episodes 1–100 while achieving zero reward leakage in episodes 101–200.

Perhaps the most elegant insight of the paper is treating the learned graph diameter DD as a joint controller for both sweep depth and replay interval (H(D)H(D)). In physics and signal processing, maximum power transfer requires impedance matching. In sequence replay, the "characteristic length scale" of credit assignment is the diameter DD of the state space.

Furthermore, by tying the replay interval to H(D)H(D) (e.g., sweeping 16 steps every 16 online actions instead of 8 steps every 8 actions), LOOPREPLAY amortizes its compute budget: it executes fewer, deeper, coherent end-to-end sweeps on large graphs without increasing the per-step computational cost.

Critical weaknesses

The current evaluation is restricted to a controlled tabular setting that relies on exact undirected graph diameters (DD), exact visit counts, and explicit context identifiers. To scale the method to deep reinforcement learning, exact quantities need to be replaced with approximate latent graphs, learned feature representations, and estimated context identifiers.

Questions for authors
  1. Section 3 and Algorithm 1 specify that LOOPREPLAY computes the exact undirected diameter DD of the learned transition graph at each online step, while Section 8 notes that approximate diameters would be required for deep reinforcement learning with function approximation. Question for Authors: What is the computational overhead of calculating the exact graph diameter at every online step as the state space scales? Furthermore, how sensitive is the diameter-gated horizon rule H(D)H(D) to inaccuracies when using approximate or heuristic diameter estimators in larger, continuous environments?

  2. Forward sweeps terminate early when the mean absolute residual falls below ϵ=0.025\epsilon = 0.025 after at least two edges, and Table S3 details sensitivity tests for various constants. Question for Authors: How sensitive is the overall performance and compute savings to the exact choice of the stopping tolerance ϵ\epsilon, and does this threshold require re-tuning if the reward magnitude or discount factor (γ\gamma) changes?

  3. The experimental evaluation focuses on a single reward position shift between episodes across three controlled grid topologies (Ring, Bridge, Cross). How do the event gate, context isolation, and backward-sweep mechanisms behave in environments featuring multiple simultaneous goals, continuous reward gradients, or continuously moving (rather than abruptly relocated) targets?

Reviewer_VaduOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LOOPREPLAY which introduces an event-based controller. The controller regulares access based on reward outcomes and signed prediction errors, enforces non-cyclic scans prioritized by empirical transition confidence, uses residual depletion as a termination condition, and determines propagation depth based on the diameter of the learned graph. It tries to fix failures under the conditions that RL rewards are sparse and spatially distributed, which includes sweeping stalls due to iterative states, low-confidence stochastic edges contaminate propagation values, and a fixed replay horizon becomes mismatched with the scale of the learned graph. LOOPREPLAY is a rule-based replay controller for tabular Q-learning in small stochastic grid graphs with 4 actions (covers 3 ring, bridge and cross topologies) with sparse terminal rewards, and a single reward position shift between episodes. Comparing to the stronger of prioritized replay and Dyna under each condition,LOOPREPLAY improved the average success AUC by +0.071 (95% hierarchical confidence interval [0.024,0.126]) at matched replay compute per online step.

Critical strengths

1, the experiment is carefully designed, including rules and criteria before the hold-out matrix, seeds, per-cell results for baselines, etc. 2, acyclic advancement, confidence ranking and the diameter-calibrated horizon are each tied to a falsification signature with failure mode, this is a good template for operate level metrics.

Critical weaknesses

1, the result metrics select weakest baseline of prioritized replay and Dyna to compare with. 2, the writing can be improved with more background as the problem is not well-known, and also cite and run Dyna-Q+ as a baseline.

Questions for authors

1, can you include Dyna-Q+ baseline?

Reviewer_18NaOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

LOOPREPLAY is a hand-built controller for sequence replay in tabular Q-learning on small stochastic grid graphs where the reward relocates mid-run.

Critical strengths

None

Critical weaknesses

The writing and presentation (including figures) of the paper is frankly abysmal. Sentences, while individually grammatically and factually correct, are disjoint in paragraphs, overly dense, and do not materialize into coherent arguments, claims, or narrative structures. Numbers and constants appear from nowhere, with virtually no justification. It is completely unclear what the paper is presenting, what the authors are claiming to be contributions, or even what problem(s) the authors seek to address.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.