acceptodds
Withdrawn from ICLR 2027

SELF-DISCOVERING RL IN THE ERA OF EXPERIENCE: IS LEARNING HISTORY AN ASSET OR A BURDEN?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The pursuit of recursive self-improvement (RSI) toward general intelligence is divided between macro-level language model scaling and the interaction-driven principles of Sutton and Silver’s Era of Experience. Yet, any self-improving architecture ultimately rests upon its underlying optimization engine: if general intelligence requires learning from grounded interaction, the reinforcement learning (RL) update rule itself must be capable of cumulative adaptation. While algorithm self-discovery has produced Disco103—a self-discovered rule that surpassed PPO to achieve state-of-the-art benchmark performance—its internal update machinery remains an uninspected black box. Specifically, how its persistent recurrent state uses accumulated learning history across shifting conditions is entirely unknown. We present the first causal mechanistic audit of a self-discovered RL rule, structured directly around the five pillars of the Era of Experience: extended horizon, grounded reward scales, continuing streams, within-lifetime change, and exploration depth. By surgically pinning, freezing, and transplanting recurrent states while holding meta-parameters fixed, we test when learning history acts as an asset or a burden. Three findings organize the audit: (1) Recurrent history actively expands usable reward scales, sustaining a six-decade window on Catch versus three under zero-pinning (difference: 3 [2, 3] decades). (2) Decoupling historical content from its maintenance reveals that the penalty of mismatched history stems from perpetual clamping; allowing imported state to evolve naturally attenuates this burden (interaction: 0.393 [0.079, 0.861]). (3) Under environmental change, controlling replay retention reverses the apparent adaptation advantage over DQN,demonstrating that external data turnover can confound internal plasticity. Validated through capability thresholds and ported to a second rule (OPEN), this work grounds macro-RSI ambitions in micro-level learning dynamics, establishing a foundational audit standard for next-generation, self-evolving RL algorithms.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, algorithm discovery, meta learning, recursive self-improvement, continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kYHhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims that a self-improving learning rule must have 5 properties, inspired by the Era of Experience paper. Namely, 1) the ability to learn over long horizons, 2) be robust to varying reward scales, 3) the ability to learn from continuing non-episodic streams of interaction, 4) adapt to changes in the learning environment, and 5) explore in sparse reward settings.

The authors study how the recurrent hidden state in Disco-103's meta-network influences the inner agent's learning properties with respect to the criteria above by using causal interventions, for example, setting the hidden state to 0.

Critical strengths

None

Critical weaknesses

The paper is very difficult to read, and each individual line is borderline incomprehensible. This makes it very difficult to flag technical points, simply because I’m unable to follow what is being said.

Below, I provide examples of such text in the paper, and also raise some major technical concerns. Hence, I recommend a Reject rating.

  1. How does a study on the learning history of Disco-103 relate to recursive self-improvement? What insights does this paper provide for recursive self-improvement?

  2. The paper does not sufficiently describe background knowledge (Section 2)

    1. the Disco 103 architecture

    2. In Eq. (1), what is u_t?

    3. What are h_t and c_t in the recurrent state?

    4. What constitutes “experiences, predictions” in \tau?

  3. Experiments:

    1. They miss a lot of key details, with some details scattered all across the paper

      1. Ex: In Section 4.2, what exactly is inherited, and what are the donors? In Section 4.3, what is Tracking?
    2. It is very difficult to follow the experiments. Further, the authors do not clearly state the intended takeaway/insight.

  4. The paper contains many terms that are not defined:

    1. Ex: replay turnover
  5. Finally, many statements are difficult to interpret, for example:

    1. [Line 26] Decoupling historical content from its maintenance reveals that the penalty of mismatched history stems from perpetual clamping; allowing imported state to evolve naturally attenuates
      this burden

    2. [Line 203] What scalar information can the categorical value head preserve, and can similar behavior conceal different uses of that representation?

    3. [Line 388] A self-discovered rule must not treat past states as permanent physical constraints

    4. [Line 406] Establish capability before attributing changes in task competence to memory.

    5. [Line 412] OPEN (Goldie et al., 2024) demonstrates protocol portability with condition-dependent effects: one of six endpoint live-minus-zero contrasts has an adjusted interval above zero

    6. [Line 452] Separating algorithmic demands from perceptual complexity makes individual mechanisms experimentally accessible.

Questions for authors

I believe the paper requires substantial revision to improve clarity and make its technical contributions assessable. Can the authors clearly state the technical contributions of this paper, how they relate to recursive self-improvement, and what novel insights they produce compared to Oh et al. 2025?

Reviewer_LUYuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits the persistent recurrent state of the released Disco103 learning rule with its meta-parameters frozen. State interventions and replay controls examine reward-scale robustness, adaptation, and other capabilities. The contribution is an empirical study of when retained learning history helps or hinders learning.

Critical strengths
  1. The paper addresses an interesting question: when learning history helps or hinders further learning.
  2. The experiments cover several aspects of learning and include useful controls to examine the role of history.
  3. The inclusion of negative and inconclusive results helps clarify the limits of the findings.
Critical weaknesses
  1. The causes of the observed effects remain unclear. Resetting, freezing, and transferring the recurrent state show that it affects learning, but do not explain what it remembers or how that information is used. For example, does it help with small rewards by adjusting their scale or by tracking learning progress? The additional tests in Appendix I.2 do not resolve this. Explaining one central effect would better support the paper's claim to open the recurrent-state black box.

  2. Some conclusions about adaptation need narrower wording. Matching replay capacity reverses the Disco–DQN ranking in action-reversal Catch, but equal capacity does not ensure equally old data or equal reuse. This shows that replay settings matter, without fully explaining the performance difference. In the low-reward experiment, a state transferred from a different reward scale benefits from further updates. Yet fixed states also work well in several conditions, so updating history is not always necessary.

  3. The supplement does not enable independent reproduction of the experiments. It provides analysis and plotting code and plotted values, but explicitly omits training code, per-run logs, and checkpoints. The scripts also require data archives absent from the package. The training implementation and underlying data are needed to repeat the experiments and verify the statistical results.

  4. Minor comments. A brief Disco103 overview explaining how it generates training targets and distinguishing η, r, and θ would improve readability. The numerical checks of the PyTorch reimplementation are useful; repeating one key state-intervention experiment in both PyTorch and the original JAX version would further strengthen confidence in their agreement over training.

Questions for authors
  1. Can further analysis explain why the recurrent state helps with small rewards, for example, whether it adjusts for reward scale or tracks learning progress?
  2. How might differences in the age and reuse of replay data affect the interpretation of the Disco–DQN adaptation comparison, even after matching buffer capacity?
  3. Given that fixed states work well in several experiments, under what conditions does continuing to update the recurrent state help? How should this qualify the paper's broader conclusions about revising learning history?
  4. How does termination masking affect A2C's return and advantage calculations, and why do synthetic boundaries restore performance?
  5. How do the findings relate to offline-to-online and continual RL, and what additional insight comes from intervening on the learned update rule's recurrent state?
Reviewer_rhP8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the internal state accumulated by Disco103 during training affects subsequent learning. The authors use state intervention experiments to examine the role of past experience and point out that replay settings can also affect assessments of an algorithm’s adaptability.

Critical strengths
  1. This work examines the dependence of self-discovered reinforcement learning rules on past experience, offering a useful perspective for understanding and evaluating such learning systems.
  2. By comparing factors such as internal state, history inheritance methods, and experience replay, the work reveals differences that a single performance score can easily obscure.
Critical weaknesses
  1. Interventions on the full state do reveal that learning performance depends on it, but the authors provide limited explanation of how the historical information in that state changes specific update computations.
  2. The continuing-interaction and sparse-exploration experiments mainly characterize capabilities and have not directly established their dependence on the lifetime state. Their connection to the paper’s central investigation of mechanisms involving learning history is therefore relatively indirect.
  3. The appendix is excessively long, making it harder to identify the key points and increasing the reading burden.
  4. In addition, descriptions in many parts of the paper are not sufficiently rigorous. For example, numerical results are sometimes reported to three decimal places and sometimes to five. The excessive use of structured supplementary phrases also disrupts the flow of the writing, as illustrated by the sentence structure in lines 16–17 of the abstract and the parenthetical addition in line 27.
Questions for authors
  1. What evidence can distinguish the state’s roles in scale calibration, update magnitude regulation, and other functions? Is there any analysis that directly shows how state changes affect the outputs of the update rule?
  2. At the low reward scale, what are the paired effect and confidence interval for cross-natural versus the standard Live condition trained from scratch? How do the authors define the scope of their conclusions about revisability, and which specific conclusions are supported by the OPEN extension?
  3. Could the authors provide minimal learning-iteration pseudocode specifying where state updates and interventions occur, and clarify how random states are sampled, which quantities are frozen in the EMA control, and the units and conversion used for replay capacity matching?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.