acceptodds
Under review as a conference paper at ICLR 2027

Learning from the Unexpected: A Von Economo Neuron-Inspired Efference Copy Architecture for Continual Reinforcement Learning

Abstract

Efficient continuous learning requires computing the orthogonal subspaces in the learnt parameter space, separating parameters to be updated for the new task from the ones that should be retained. This implies a continuous one-to-one mapping between the input and generated output to distinguish which inputs are caused of the model output, which is also required for animal self-recognition in the mirror. We derived a novel reinforcement learning framework inspired by von Economo neurons (VEN), a type of neuron that is only found in the small number of species with documented mirror self-recognition. Morphology of VEN suggests that it compares efference copy motor plan to the sensory input, helping to yield truly unpredictable residual signals for learning. Unlike the reward-prediction error of standard TD learning, these residual isolate self-caused from externally induced change. We distilled VEN architecture into an explicit efference copy circuit for reinforcement learning. In Atari-2600 game and control suite environments, VEN augmented DDQN learned much more efficiently than capacity-matched controls under self-position perturbation, actively probed their own position in the game, and performed continuous learning across sequentially presented tasks. This architecture provides a computational account of how preserved latent state information could support continuous learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continuous learning, reinforcement learning, von Economo neuron

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PC2cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose an RL framework inspired by Von Economo neurons, which learning a forward model and augments the Critic's obervation with the three most recent outputs of this model. This allows the agent to model the residual between what the agent expected to happen and the true environment outcome, improving performance in tasks with dynamics not controlled by the agent. The authors test their method in Atari games, use the DDQN as a baseline as well as different ablations of their method. Results show improved performance over DDQN on the tested scenarios.

Critical strengths
  1. The paper is well motivated in the introduction
  2. The idea of flipping which bar is controlled by the agent was interesting and section 5.1 provides ablations enough ablations to interpret the results
  3. In the experiments presented by the authors, their approach improves over the DDQN baseline.
Critical weaknesses

Major:

  1. The paper doesn't have an AI use statement, something that was mandatory under the policy of the conference.

  2. In section 4.2 there are references to terms that were not previously presented: the auxiliary loss functions, the flip-matrix, action reversal. The same happens in section 4.3 where the authors refer to the motor-cortex counterpart of the VEN-enhanced network without introducing the term or ever referring to it again in the paper. More examples of terms that are never defined: binarized protocol, rehearsal. This is odd writing and bad structure that makes it very hard to read the paper.

  3. While the paper reports improvement over the DDQN, it does not compare to other self-supervised methods such as SPR, CURL or methods that explicitly model the distinction between agent and environment.

  4. The authors miss relevant work in the literature that also studies self-other distinction in RL. The proposed method should be presented in the context of previous work, detailing which limitations it tackles and, if possible, comparing to them.To name a few of these works:

[1] Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., & Bachman, P. (2020). Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations.

[2] Amos, B., Dinh, L., Cabi, S., Rothörl, T., Colmenarejo, S. G., Muldal, A., Erez, T., Tassa, Y., Freitas, N. & Denil, M. (2018). Learning awareness models. In International Conference on Learning Representations.

[3] Gmelin, K., Bahl, S., Mendonca, R., & Pathak, D. (2023). Efficient rl via disentangled environment and agent representations. In International Conference on Machine Learning.

[4] Serra Nunes, M., Dehban, A., Demiris, Y., & Santos-Victor, J. (2026). Ego-Foresight: Self-supervised Learning of Agent-Aware Representations for Improved RL. In International Conference on Learning Representations.

Minor:

  1. Extremely small font in the figures 1, 3, 4

  2. Related work writing is dense and telegraphic, making it hard to parse.

Questions for authors
  1. In Figure 1a what is the meaning of V1, V2, V4, MT, M1? As stated above, in this paper there is terminology and labels that lack sufficient introduction or definition. I assume that the Motor M1 is the QQ head described in the text since it is the only function that receives the output of VV, but this cannot be left for the readers to disentangle.
  2. I’m not sure I understood the intuition behind the approach. The visual encoder only receives gradients from LTD\mathcal{L}_{TD}, meaning that the VV network doesn’t have an influence in feature extraction. VV network outputs serve merely as an augmentation to the QQ network’s observation, which should learn the self-other split by implicitly computing the residual ete_t? This would in turn allow the QQ network to provide better return estimates. I think this is the idea described at the end of section 4.3, but in my opinion it could be made more clear.
  3. As theta is updated during training, the loss function of equation (9) has a moving target. How is this handled? Is VϕV_{\phi} robust to this? The authors refer to the target QQ network with periodically copied weights, but nothing is mentioned in regards to the endoder.
  4. I also need more clarification regarding the action swap setting regarding there being “no visual queue” and the only way of revealing the swap being the comparison between predicted and observed embeddings.
1 comment
AuthorsOpenReview ↗

Response to Reviewer PC2c (final revision)

We thank you for the careful reading and the Nov 12 follow-up. Your concerns about presentation and framing were fair, and the revision was restructured around them. On the follow-up, point by point:

SPR row. The row is now defined in the text (Sec. 5.1) with the citation: the SPR-style variant of DDQN uses the same prediction signal as a training-time loss (1- and 2-step, without SPR's target networks or projection heads), and it fails on the combined flip (−0.8±1.0 vs. VEN +6.5±1.1, Table 1; loss specification in Appendix Table 2). What separates VEN is an explicit mechanism-contrast sentence in Related Work: in prior work the predictive or contrastive signal only shapes training; VEN sends the prediction itself into the value head, where the residual z − ẑ can be formed implicitly.

Gradients. The direct/indirect qualification is removed. Sec. 4.3 now has an explicit gradient-routing paragraph: L_TD updates all three modules (θ and ψ directly, φ through the un-detached ẑ input); L_VEN updates θ (through V_φ's input) and φ, and never reaches ψ. Q2's mechanistic reading is further supported by interventions: zeroing ẑ in the trained head changes Q by 0.6–0.8 and flips 50–75% of decisions; under transfer to an unseen flip the prediction's share of the advantage rises to 44%; and a counterfactual action sweep moves ẑ by 31.5% of its norm and flips the greedy action at 21.9% of steps.

Figures. Regenerated to improve visibility. Figure 1 is now with modules labeled by their symbols (f_θ, V_φ, g_ψ) rather than neurobiology labels.

Also delivered from the original review: a Statement on generative AI use; a protocols subsection (Sec. 4.4) defining the flip matrix, action reversal, grayscale/binarized rendering, and rehearsal at first use, with mixture semantics explicit (h = 50/50 mixture; hv = 25% per orientation); the terminology table (Appendix Table 1); and Related Work rewritten in full prose, citing the four works you listed (SPR; awareness models; disentangled agent–environment representations; Ego-Foresight).

Two corrections to our Nov 9 replies. (i) The CURL-style contrastive control we said the revision "will add" was not finished in time; the limitations say so explicitly, and we apologize for promising more than the revision delivers. (ii) the "DDQN+ahist −7.5 vs. VEN +7.1" numbers quoted under Q4 were hv-flip numbers, not action-reversal numbers (the a-history hv cell is −7.8±1.1; a reversal-protocol a-history control remains untested).

Additional corrections in the final revision. A final internal review tightened wording: the architecture is now described as an action-conditioned latent forward model with a target-side stop-gradient (the stop-gradient applies to the prediction target only, Eq. 8); the residual is said to estimate — not isolate — the externally caused component of change; the abstract opens from the introduction's own delineation framing; and continual evidence is explicitly scoped to Pong flip-orientation sequences.

What remains: positioning without empirical comparison to CURL/DEER/Ego-Foresight, and Continual World as future work. We hope the defined terms, citations, mechanism contrasts, and open corrections address your concerns and merit your current score or higher.

Reviewer_rSLcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a Von Economo Neuron (VEN) Inspired efference copy architecture for continuous reinforcement learning. It states that the morphology of VEN will compare the efference copy motor plan to the sensory input, helping to yield truly unpredictable residual signals for learning. Unlike the reward-prediction error of standard TD learning, these residuals isolate self-caused from externally induced change. Experiments demonstrate substantial improvements under observation flips and action reversals, alongside improved task retention under rehearsal. Overall the paper is interesting, well-written and easy to follow.

Critical strengths
  • The method that using VEN-inspired efference copy architecture for continuous reinforcement learning is interesting and novel.
  • The paper demonstrates substantial performance improvements in challenging observation and action perturbation settings.
  • The authors report informative negative results, including catastrophic forgetting without rehearsal and limited additional benefits when integrating the module into DreamerV3.
Critical weaknesses
  • Fig 1, Fig. 3 and Fig 4 are too small, it is suggested that the user can reorganised the plot to make it more readable.
  • The training environments for continuous reinforcement learning is not stronger enough to support the claim, for example, some standard continuous RL benchmark would be helpful, such as: https://arxiv.org/pdf/2105.10919
  • More continous RL baseline methods would also support the improvements of the proposed approach.
Questions for authors

My concerns are in the above section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer rSLc (final revision)

We thank you for assessing the method as "interesting and novel," the improvements as "substantial," and especially for reading our negative results as informative.

Figures. Figures were regenerated in new versions to improve the readability.

Environments and scope. We narrowed claims rather than over-reaching: the title is scoped to robust RL under sensorimotor perturbations, and Continual World is named explicitly as future work in the limitations. Within the axis the paper does claim: five control environments with per-dimension state flips (Table 3; 5–9 seeds; all five individually significant, t ≥ 3.3), and, as a robustness check beyond visual perturbations, a state-input flip curriculum on CartPole (Table 5): phase-1 VEN reaches 143.1±49.7, then retains the source orientation (per-seed 84.8–114.4) while acquiring the flipped one (77.6–114.1); DDQN stays at floor (22.5–25.0 source).

Baselines. EWC (λ = 1, 100) and L2-SP run through the identical protocol on both architectures, n = 3, on all three transitions plus 0%-replay arms (Table 6). The results are negative and reported as such: on DDQN, EWC λ=1 is within seed noise of plain retraining, λ=100 hurts, L2-SP is catastrophic on the single-flip transition (−11.2±1.5 vs. +1.3±0.7); on VEN, the regularizers hurt increasingly with strength (−4.4 → −7.7 → −14.2 on no-flip→hv). The paper states plainly that at the strengths tested, parameter regularization does not help in such perturbation tasks.

Protocol transparency. Seed replication surfaced that continual-learning dynamics depend materially on the phase-2 retraining schedule, so the revision separates the two protocol families: grayscale policy-preserving (ε = 0.05 fixed) primary in the main text (Table 4); binarized exploration-restart (ε: 1.0→0.05 over 2M steps) in Appendix Tables 4–5, with the explicit statement that numbers across families are not comparable. The VEN-over-DDQN retention ordering holds under both. Long-horizon 8M/10M continuations are in Appendix Table 3; the DreamerV3 null result stands (39 runs, Table 7).

Delivered vs. promised. From our Nov 9 plan: figures, per-orientation evaluation of every arm, seed-replicated EWC/L2-SP including the VEN+EWC composition, and rescoped claims. Remaining, stated in the paper: Continual World and a CURL-style control are future work; with 3 Atari seeds only 3 of 7 transfer games reach individual significance (Table 3 caption).

Additional corrections in the final revision. The abstract now opens from the introduction's own framing (methods that learn continually must first delineate the old solution) and says the residuals estimate — not isolate — the externally caused component of sensory change; the no-flip→hv 0%-replay contrast (+6.9±4.1 vs. −10.8±0.3, p = 0.017) now cites only the binarized-family table it comes from (Appendix Table 5); per-transition numbers absent from any table were removed; and the capacity claim is scoped to the 302,454-parameter family.

We hope the narrowed claims, added baselines with honest negative results, the CartPole robustness check, and the protocol transparency address your concerns and merit your current score or higher.

Reviewer_YqHxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces an extension to DDQN network where they predict next observation embedding from current embedding, action pair and the Q-network uses predicted and observed embeddings. They introduce this change to help the agent understand when its actions don't have the expect effects. The paper claims this supports continual learning by helping the agent reuse learned action outcome relationships across tasks. The efficacy of this approach is verified via comparing comparing against DDQN on Pong and other environments, and do careful ablations to check prediction module's contribution.

Critical strengths
  • It's an interesting finding and simple change that finds improved performance under control reversals!
  • Careful experimentation (across envs) and ablations done to find contrubtion of prediction features
Critical weaknesses
  • VEN suffers from forgetting old tasks with 0% replay similar to its baseline, so the claim for continual learning is not grounded. The 50% replay result can support improved retention with rehearsal
  • Baseline results are not fair and have no access to action history while VEN implcitly receives via predictions.
Questions for authors
  • How many seeds are used for results in Table 2?
  • Add comparison against DDQN with one-hot action history in the Q-head input
1 comment
AuthorsOpenReview ↗

Response to Reviewer YqHx (final revision)

We thank you again for a review that was constructive and, in hindsight, precise about the paper's two weakest points — the over-broad continual-learning claim and the missing action-history controls. Both shaped the revision directly.

Seeds. Table 1 of the revision (the flip-matrix table, your Table 2) now reports mean ± s.d. across seeds with n stated in the caption: n = 3 in every cell, and on the decisive hv column n = 5 for VEN-s, DDQN, a-history and z+a-history, n = 6 for VEN and VEN-det. The earlier within-run ± values are gone.

Action-history controls. DDQN+a-history and DDQN+z+a-history now run in all four flip conditions (Table 1), and the paper states the honest reading explicitly: z+a-history matches or exceeds VEN on no-flip, h, and v (+11.3±1.8, +9.2±0.6, +9.6±0.8); VEN leads only on the combined hv condition (+6.5±1.1 vs. +4.3±1.0; p = 0.005, n = 6 vs. n = 5), where every history and auxiliary-loss variant falls to +4.3 or below and plain DDQN sits at −9.4±0.7. The content–gradient factorial (Table 2) adds that wrong efference is worse than none (shuffled −3.2 vs. constant −2.5).

Continual claim, rescoped exactly as you worded it. The title ends "…for Robust Reinforcement Learning under Sensorimotor Perturbations," and the abstract claims only faster new-task acquisition and retention under rehearsal. Table 4 (grayscale, policy-preserving retraining; 1,848 evaluation episodes per architecture): with rehearsal, VEN retains the old task at +20.9±0.3 while DDQN falls to −3.8±12.8; without rehearsal both architectures forget the source (−20.6±1.4 vs. −20.9±0.2), and our rehearsal-free claim is acquisition only (+18.8±3.3 vs. +4.5±13.1 pooled; +6.9±4.1 vs. −10.8±0.3 on no-flip→hv, p = 0.017, binarized exploration-restart family, Appendix Table 5). Untrained configurations sit at the floor for both, and the cross-scene evaluation (Appendix Table 6, 100 episodes per cell) shows no architecture generalizes zero-shot to an unseen orientation.

Delivered vs. promised. All items of our Nov 9 plan are in the revision: four-condition controls, n ≥ 3 cells with stated seed counts, the train-X/eval-Y evaluation, and EWC/L2-SP on both architectures as honest negative results (Table 6). The optional second game was not added due to time limitations (we are still working on them). Nov 9 interim numbers are superseded by the final tables.

Additional corrections in the final revision. A final internal pass corrected wording: "capacity-matched controls" became "controls of matched or larger capacity" (and "no variant in the 302,454-parameter family can win by capacity; the raw history controls carry more"); VEN-det is described as costing "little except on the combined flip" (it exceeds VEN on no-flip, h, v); continual evidence is scoped to Pong flip-orientation sequences; per-transition continual numbers absent from any table were removed; and Fig. 2's caption marks its curves as single illustrative seeds, cross-seed means in Table 1.

With seed-replicated statistics, information-matched controls, and claims narrowed to what the data support, we believe the revision addresses both of your concerns in full and hope it merits your current score or higher.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.