- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors propose an RL framework inspired by Von Economo neurons, which learning a forward model and augments the Critic's obervation with the three most recent outputs of this model. This allows the agent to model the residual between what the agent expected to happen and the true environment outcome, improving performance in tasks with dynamics not controlled by the agent. The authors test their method in Atari games, use the DDQN as a baseline as well as different ablations of their method. Results show improved performance over DDQN on the tested scenarios.
Critical strengths
- The paper is well motivated in the introduction
- The idea of flipping which bar is controlled by the agent was interesting and section 5.1 provides ablations enough ablations to interpret the results
- In the experiments presented by the authors, their approach improves over the DDQN baseline.
Critical weaknesses
Major:
-
The paper doesn't have an AI use statement, something that was mandatory under the policy of the conference.
-
In section 4.2 there are references to terms that were not previously presented: the auxiliary loss functions, the flip-matrix, action reversal. The same happens in section 4.3 where the authors refer to the motor-cortex counterpart of the VEN-enhanced network without introducing the term or ever referring to it again in the paper. More examples of terms that are never defined: binarized protocol, rehearsal. This is odd writing and bad structure that makes it very hard to read the paper.
-
While the paper reports improvement over the DDQN, it does not compare to other self-supervised methods such as SPR, CURL or methods that explicitly model the distinction between agent and environment.
-
The authors miss relevant work in the literature that also studies self-other distinction in RL. The proposed method should be presented in the context of previous work, detailing which limitations it tackles and, if possible, comparing to them.To name a few of these works:
[1] Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., & Bachman, P. (2020). Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations.
[2] Amos, B., Dinh, L., Cabi, S., Rothörl, T., Colmenarejo, S. G., Muldal, A., Erez, T., Tassa, Y., Freitas, N. & Denil, M. (2018). Learning awareness models. In International Conference on Learning Representations.
[3] Gmelin, K., Bahl, S., Mendonca, R., & Pathak, D. (2023). Efficient rl via disentangled environment and agent representations. In International Conference on Machine Learning.
[4] Serra Nunes, M., Dehban, A., Demiris, Y., & Santos-Victor, J. (2026). Ego-Foresight: Self-supervised Learning of Agent-Aware Representations for Improved RL. In International Conference on Learning Representations.
Minor:
-
Extremely small font in the figures 1, 3, 4
-
Related work writing is dense and telegraphic, making it hard to parse.
Questions for authors
- In Figure 1a what is the meaning of V1, V2, V4, MT, M1? As stated above, in this paper there is terminology and labels that lack sufficient introduction or definition. I assume that the Motor M1 is the head described in the text since it is the only function that receives the output of , but this cannot be left for the readers to disentangle.
- I’m not sure I understood the intuition behind the approach. The visual encoder only receives gradients from , meaning that the network doesn’t have an influence in feature extraction. network outputs serve merely as an augmentation to the network’s observation, which should learn the self-other split by implicitly computing the residual ? This would in turn allow the network to provide better return estimates. I think this is the idea described at the end of section 4.3, but in my opinion it could be made more clear.
- As theta is updated during training, the loss function of equation (9) has a moving target. How is this handled? Is robust to this? The authors refer to the target network with periodically copied weights, but nothing is mentioned in regards to the endoder.
- I also need more clarification regarding the action swap setting regarding there being “no visual queue” and the only way of revealing the swap being the comparison between predicted and observed embeddings.
Response to Reviewer PC2c (final revision)
We thank you for the careful reading and the Nov 12 follow-up. Your concerns about presentation and framing were fair, and the revision was restructured around them. On the follow-up, point by point:
SPR row. The row is now defined in the text (Sec. 5.1) with the citation: the SPR-style variant of DDQN uses the same prediction signal as a training-time loss (1- and 2-step, without SPR's target networks or projection heads), and it fails on the combined flip (−0.8±1.0 vs. VEN +6.5±1.1, Table 1; loss specification in Appendix Table 2). What separates VEN is an explicit mechanism-contrast sentence in Related Work: in prior work the predictive or contrastive signal only shapes training; VEN sends the prediction itself into the value head, where the residual z − ẑ can be formed implicitly.
Gradients. The direct/indirect qualification is removed. Sec. 4.3 now has an explicit gradient-routing paragraph: L_TD updates all three modules (θ and ψ directly, φ through the un-detached ẑ input); L_VEN updates θ (through V_φ's input) and φ, and never reaches ψ. Q2's mechanistic reading is further supported by interventions: zeroing ẑ in the trained head changes Q by 0.6–0.8 and flips 50–75% of decisions; under transfer to an unseen flip the prediction's share of the advantage rises to 44%; and a counterfactual action sweep moves ẑ by 31.5% of its norm and flips the greedy action at 21.9% of steps.
Figures. Regenerated to improve visibility. Figure 1 is now with modules labeled by their symbols (f_θ, V_φ, g_ψ) rather than neurobiology labels.
Also delivered from the original review: a Statement on generative AI use; a protocols subsection (Sec. 4.4) defining the flip matrix, action reversal, grayscale/binarized rendering, and rehearsal at first use, with mixture semantics explicit (h = 50/50 mixture; hv = 25% per orientation); the terminology table (Appendix Table 1); and Related Work rewritten in full prose, citing the four works you listed (SPR; awareness models; disentangled agent–environment representations; Ego-Foresight).
Two corrections to our Nov 9 replies. (i) The CURL-style contrastive control we said the revision "will add" was not finished in time; the limitations say so explicitly, and we apologize for promising more than the revision delivers. (ii) the "DDQN+ahist −7.5 vs. VEN +7.1" numbers quoted under Q4 were hv-flip numbers, not action-reversal numbers (the a-history hv cell is −7.8±1.1; a reversal-protocol a-history control remains untested).
Additional corrections in the final revision. A final internal review tightened wording: the architecture is now described as an action-conditioned latent forward model with a target-side stop-gradient (the stop-gradient applies to the prediction target only, Eq. 8); the residual is said to estimate — not isolate — the externally caused component of change; the abstract opens from the introduction's own delineation framing; and continual evidence is explicitly scoped to Pong flip-orientation sequences.
What remains: positioning without empirical comparison to CURL/DEER/Ego-Foresight, and Continual World as future work. We hope the defined terms, citations, mechanism contrasts, and open corrections address your concerns and merit your current score or higher.