acceptodds
Under review as a conference paper at ICLR 2027

Self-Confirming Superposition Traps in Reinforcement Learning

Abstract

Reinforcement learning (RL) trains representations on data selected by the agent's policy and updates that policy using the returns they support. We show that this loop can sustain a lower-return policy even when representation fitting is globally optimal for the policy's own data. Specifically, we identify a self-confirming superposition trap, in which every optimal code shares features that rarely co-activate under the current policy but interfere after an alternative action. The resulting control error lowers that action's return and reinforces its avoidance, although choosing it would earn more after representation adaptation at the same capacity. In a tied two-step model, we characterize which feature counts and representation dimensions admit a trap. We also show separately that continued visitation, equal feature frequencies, and independent controller learning need not prevent it. Because rarely chosen actions contribute little to the fitting objective, optimal fitting can tolerate errors large enough to reverse their return advantage. We derive a bound on this distortion in a broader finite-action model and use it to prove a sufficient replay condition: retaining enough training weight on the best separately adapted action preserves its return advantage despite residual representation error. Experiments with neural PPO show that agents initialized toward different actions develop different interference patterns and opposite mean return rankings at the same capacity, with corresponding differences in their final policies. Motivated by the replay condition, we test whether preserving access to neglected training states can improve control. Providing such access reduces measured interference and improves sequential return, with return gains even when the encoder is frozen. We extend these tests to MiniGrid and DMControl, where training-state access, replay reweighting, and overlap penalties can improve control. In DreamerV3–Crafter, protecting world-model fitting weight also improves cumulative training scores at unchanged capacity. Our code is available at https://anonymous.4open.science/r/self-confirming-superposition-traps-SCST.

Primary area:
reinforcement learning
Keywords:
Superposition, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DiR8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper shows that, with limited capacity (or representation dimensionality), a policy might get "trapped" with representations that are optimal for the policy's own visiation distribution, but discourage visiting states from which it could achieve higher cumulative reward. They show that this "trap" exists across a variety of controlled experiments, and proceed with some experiments on MiniGrid and DMControl.

Critical strengths

None.

Critical weaknesses
  1. The paper is extremely hard to parse. It is filled with vague terminolgy (e.g., "Cross-talk allocation").
  2. It is very unclear what the neglected branch in DMControl and other tasks is. The authors mention that they "rebalance replay across task phases", but don't elaborate on it.
  3. The entire problem seems to be just an exploration problem framed as a representation problem. More concretely, if the agent can visit high reward regions, shouldn't reward give some learning signal to improve? So the experience itself is the underlying problem?
Questions for authors
  1. Isn't increasing the representation dimension a straightforward fix?
  2. Please explain the experiments on DMControl.
  3. How is this problem different from exploration?
Reviewer_ny9BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how feature superposition can lead RL agents to learn bad policies. The authors give theoretical proofs and test some ways to reduce this problem, such as replay and better state coverage.

Critical strengths
  1. The theory is quite solid. The authors show that the problem can happen even when the representation is globally optimal.
  2. The paper has many experiments, including PPO, MiniGrid, DMControl, and DreamerV3.
  3. The idea of linking feature superposition with policy learning is interesting.
Critical weaknesses
  1. I am not sure how new the main idea is. It is already well known that RL policies affect the data they collect, and bad data can lead to bad policies. Similar issues have been studied before, such as policy confounding and primacy bias.

  2. I am not convinced that superposition is the main cause of the failures in real RL tasks. The improvements could also come from better data coverage or better training. The paper does not clearly separate these factors.

  3. The proposed solutions are not very new. Replay and balanced sampling have been widely used in RL. I think the paper needs stronger comparisons with existing methods.

  4. The theory mainly uses simple models. It is unclear how common this problem is in larger RL systems. Some experiments also use very few seeds, and the improvements are not always consistent.

Questions for authors

I think the theory is interesting and the paper is well organized. But the main idea seems close to existing work. More importantly, the experiments do not clearly show that superposition is the cause of the problem. I would like to see stronger evidence and better comparisons with existing methods.

Reviewer_K7a9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper shows that interference between features can cause suboptimal policies to remain stable, even when representations are learned optimally under the current policy’s data distribution and true rewards are evaluated accurately. The authors refer to this phenomenon as a “superposition trap” and analyze the conditions under which it arises, its stability, and the conditions under which replay-based learning can mitigate it. They also evaluate related phenomena and mitigation effects in existing algorithms.

Critical strengths

S1. The paper provides a toy example that supports its central claim: the problem can arise even when optimization is successful.

S2. The analysis is concrete and detailed, including constructions in which the trap persists even with independent controllers and positive visitation probabilities.

Critical weaknesses

W1. The setup and explanation of the toy example are difficult to follow.

W2. The paper offers limited practical guidance on how to apply the proposed mitigation conditions during training, particularly regarding which experiences should be given greater weight and by how much.

W3. The lack of direct comparisons with existing replay reweighting methods, such as prioritized experience replay (PER), makes it difficult to assess the additional benefits of the proposed intervention.

W4. The core analysis relies on a two-step model and specific dynamics, limiting its generalizability to broader reinforcement learning settings.

Questions for authors

Q1. Please address weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.