acceptodds
Withdrawn from ICLR 2027

Connectome-based Reinforcement Learning for Embodied Neural Circuit Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning policies are typically represented by generic neural networks, whose architectures are largely independent of biological neural organization. In contrast, animal behavior emerges from efficient, structured neural circuits through continuous interaction with the body and environment. Whether an experimentally grounded connectome can serve directly as a reinforcement learning policy remains largely unexplored. To address this question, we propose Connectome-as-Policy, which formulates a connectome-derived Caenorhabditis elegans salt chemotaxis subcircuit as the actor in a closed-loop embodied learning system. The framework combines a fixed circuit topology, continuous neural dynamics, a simulated worm body, and a two-dimensional salt field. Reinforcement learning optimizes synaptic strengths and neuronal parameters while preserving the predefined connectivity. Across six reinforcement learning algorithms, the connectome policy learns stable chemotactic navigation and concentration regulation. Compared with conventional neural backbones, it achieves competitive performance with substantially fewer trainable parameters while enabling circuit-level interpretability. Robustness analyses reveal smooth behavioral transitions and stable performance under varying concentrations and sensory noise. Additional reinforcement learning benchmarks demonstrate comparable performance across standard control tasks. These results suggest that biological circuit priors offer a compact and interpretable foundation for embodied reinforcement learning.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Reinforcement learning, C. elegans chemotaxis, embodied neural control, biologically grounded policy, closed-loop simulation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_dHybOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Connectome-as-Policy, a reinforcement learning framework that uses the connectome derived from C. elegans to define a fixed topology for the policy network, with standard reinforcement learning algorithms learning the synaptic weights and the parameters of the neural dynamics (membrane capacitance and potentials). The author benchmarked this algorithm on a chemotactic task plus several standard RL environments and compare it to standard alternatives (mlp, lstm).

Critical strengths

None

Critical weaknesses

In my opinion this paper does not meet the standard of acceptance in several ways.

  1. The authors failed to delineate a clear problem their algorithm is trying to solve, the claim that "an experimentally identified connectome can function directly as a trainable reinforcement learning policy therefore remains largely unexplored" appears unjustified given the same literature the authors cite (Sakamoto et al. (2021); Churchland & Garcia-Ojalvo 2026; Chung & Kim 2026) where many prior works explore connectome-constrained training, including in a RL setting.
  2. Even assuming the stated motivation, the proposed experiments do no support meaningful advancements: in the main-text Figure 2,3,4 author explore standard reinforcement learning algorithm (all yielding equivalent performances), a comparison that is completely independent of the proposed contribution since all share the same fixed topology. In the one test the author offer against other possible strategies (Fig.5 in the Appendix) the proposed worm-based strategy yields equal-or-inferior performances to the other baselines (e.g random sparse) but the author themselves state that two runs are insufficient "to claim statistical superiority of any topology", which makes the inclusion of this result confusing in the first place.
  3. One of the authors' claimed contributions is to "enable circuit-level interpretability", however the only related analysis (Appendix B, Fig. 2) fails to support such claim as it is limited to a visualisation of a small fraction of neuron, with little to no interpretation or discussion of what exactly this method is offering in terms of interpretability of the circuit-performed computation.
  4. The motivation behind applying a worm-derived fixed connectome on generic RL tasks (that include controlling a lunar lander) is puzzling.
  5. The paper is often imprecise and unpolished, with many overlapping/cut labels (e.g. Figure 2; Appendix Fig. 4,5,7 ) or very small/illegible panels (e.g. Fig. 3, Appendix Fig. 6).
Questions for authors

In general, the experiments do not provide convincing evidence for the merit of the proposed algorithm and I disclose that a substantial redesign of the manuscript would be needed to affect my assessment. Some quantitative questions include:

  1. What is the takeaway message of Figure 2? All explored RL algorithm appear roughly equivalent. And how do the authors justify the fact that performances seem to degrade over iteration/episode (decreasing avg. rewards and increasing errors)?
  2. How is the Connectome-as-Policy applied to generic, i.e. non worm-related, RL environments?
  3. Several of the reported returns in Fig. 6a show heavily non-monotonic trends, irrespective of the method: does this indicate a failure of training? There seems to be inconsistency between the reported values of Fig.6b and the traces in Fig.6a, for example LunarLander reports positive peak returns while the corresponding panel a shows strictly negative (< -100) values.
  4. How do the authors assess whether the claimed robustness (Fig. 5) of their technique relates to other approaches? What counts as a robust strategy in the first place?
  5. What is the mechanistic insight offered by the proposed method that other alternatives lack?
  6. The authors used a much simplified C. elegans connectome (~300 parameters), while recent work have used the exact full connectome (~3000 synapses): why was the simplified connectome chosen when the complete one is available? Do the result change in a meaningful why by scaling the connectome?
Reviewer_foE3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper uses the salt chemotaxis circuit in C. elegans to demonstrate the potential for circuit/connectome-based neural architectures to be trained with deep reinforcement learning. Their main contribution seems to be the specific choice of circuit here, rather than using the connectome as a policy (since this has been done before). The authors test several standard deep reinforcement learning optimizers on what seems to be a custom made neuromechanical embodiment and circuit architecture, showing all algorithms can successfully train the topologically constrained circuit to perform chemotaxis. The evaluations primarily concern the learning performance of each algorithm on a reasonable salt reward signal that locally tracks changes of salt concentration in time, with additional bonuses as the agent gets close to the source. They also test the architecture on a few standard continuous control tasks, and show that it achieves reasonable performance.

Critical strengths

Using several RL algorithms that span different methodological assumptions is a good idea, since the point of the paper is to show that the neural circuit is amenable to reinforcement learning in general and not only under a single optimizer. Also, the transfer experiments in which the neural circuit is used on robotic control tasks is also a good idea to understand the extent to which neural circuit mechanisms, even from simple creatures like C. elegans, can function in complex tasks outside the animal's ethology.

But these strengths are abstract ones, in that the choice to pursue these directions are good choices. However, how they were specifically carried out in the paper was insufficient for these to weigh as positive considerations for an acceptance.

Critical weaknesses

There are numerous material weakness with the paper. However, they fall all into two categories: 1) communication of content, whether via writing or figures, and 2) the scientific significance of the content.

1). Communication of content

a) Figures are very difficult to read. They are inconsistently sized, missing legends in some cases, and offer very little to the readers interpretation of the results. For example, figure 2 reported very similar results for all six algorithms. Yet this was not interpreted in the paper. In fact, the text that references this figure says that PPO exhibits the most stable learning wheras SAC converges the quickest, but this is not clear from the figure. Figure 3 is also very unclear. Panel a) depicts weights changing after 1 episode and holding fixed for three episodes. This isn't explained or interpreted anywhere in the paper. The corresponding snapshots show a big learned difference in behavior. I assume the point is that the last three rows show different behaviors by modifying those highlighted connections, but this is a poor way to show that, and it should have been explained in text. Panels don’t have legible legends, especially in c), and are way too small. And again, the authors again don't point to these figures to demonstrate any specific claim in text. Similarly, figure 4) shows different trajectories for different policies, but the authors do not write what the reader is supposed to gain from looking at them. The differences are minor. Is that the point? Or are the minor differences the interesting bits? Neither explanation occurs in the paper.

b) Likewise, the text in the document is very redundant, doesn't illuminate any significant scientific interpretation the reader should takeaway, and is very vague on important technical details. For example, the explanations of PPO and SAC are poor in addition to being irrelevant to the content of the paper. The specific implementation details of the segment-spring model was not described, only mentioned. Further, how the dorsal-ventral steering outputs of the policy were adapted to the robotic control tasks was also not explained. Overall, the paper lacked in motivating why this specific C. elegan behavior is interesting, and thus why any of the results are interesting. The paper presented itself as a proof of concept, but this is not suitable since several published works already demonstrate how neural circuits or full connectome can trained with reinforcement learning. The scientific results themselves were very basic, almost trivial. The authors reported that their chemotaxis architecture, which is already validated in neuroscience as being computationally sufficient for this behavior, correctly learns chemotaxis from reinforcement learning. They did not offer any nuanced interpretation of the differences or similarities of the different RL algorithms and the role of different components of the neural circuit via ablation. Overall, it was full of unimportant details and lacked the important ones.

2). The main contribution of the paper, which is emphasized many times, is that "biological connectomes can serve as effective policy priors for reinforcement learning". However, several prior works have demonstrated this concept in Drosophila and C. elegans using full connectomes. However, the authors present their paper as if this were a new concept, both in the title and name of their proposed method, Connectome-as-Policy. Even worse, the authors proceed to not even study the full connectome as a policy substrate, but rather a specific subcircuit for a specific behavior: chemotaxis. Then, the scientific content of the paper consists of several very basic simulations demonstrating that the six RL algorithms successfully learned chemotaxis behaviors starting from a neural circuit prior specifically for chemotaxis. I don't see why this is significant at all. At the very least, the authors could have included a general, end-to-end policy with no connectivity constraints, and tested if the neural circuit prior afforded differences in sample complexity or learned behavior. Without such a baseline, we simply have 6 agents with the same circuit trained other variations of off- and on-policy RL algorithms, all resulting in qualitatively the same behavior. Similarly, the transfer experiment was very low-effort. The authors did not motivate or explain the choice of control tasks, nor explain how the architecture was adapted. The experiments report a non-standard metric of "Peak" reward, and even on this metric, only show the neural circuit prior achieves best performance on one of four tasks. So again, no scientific significance to the reported results.

Questions for authors

Although I have many questions for the authors, none of the answers would materially change my assessment. The paper is not in a state suitable for publication.

Reviewer_7rU1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses a fixed C. elegans chemotaxis subcircuit as an actor with continuous neural dynamics. Reinforcement learning adjusts circuit parameters within a simulated brain–body–environment loop. Six RL algorithms are evaluated on salt-concentration regulation, followed by comparisons with MLP, LSTM, and LTC backbones on four control environments. The intended contribution is a compact, biologically grounded and interpretable policy representation.

Critical strengths
  1. A compact circuit policy in a natural sensorimotor setting

    The native sensorimotor task gives the circuit prior a clear interpretation, and preserving topology while learning continuous parameters is a coherent experimental setup.

    Evaluating the same policy family with several optimizers and on additional control tasks is a useful attempt to test applicability. The reported actor size of approximately 350–430 parameters is attractive if its performance and parameter accounting can be established under adequate controls.

  2. Repeated-run evidence in the chemotaxis setting

    The chemotaxis comparison reports ten independent runs, standard deviations, and an ANOVA rather than relying exclusively on example trajectories. The text also acknowledges that most algorithm differences are not statistically significant.

Critical weaknesses
  1. A connectome-specific contribution is not established

    The experiments do not identify a benefit attributable to the connectome. Comparing six optimizers on one fixed circuit shows that the circuit is trainable; it does not distinguish its biological topology from another compact sparse dynamical network. Table 2 compares against much larger backbones, without a matched-size sparse/reconfigured circuit control. Hasani et al. (ICML 2020), already cited by this submission, preserved a C. elegans circuit topology while optimizing neuronal and synaptic parameters for control. The present paper needs to distinguish its contribution from that established approach and explain what the particular chemotaxis circuit contributes. Named neurons alone also do not establish the claimed circuit-level functional interpretability.

    The native chemotaxis setting is interesting, but the advance beyond prior biologically structured neural-circuit policies is not yet convincingly demonstrated.

    Claims of biological plasticity should be distinguished from using a biological architecture with an engineering RL optimizer. The latter can be valuable without establishing the former.

  2. The cross-task evidence and implementation description are insufficient

    The cross-task evidence supports only a limited performance conclusion. Table 2 emphasizes peak training returns without uncertainty; Figure 6 also shows last-50 returns but without seed dispersion for the backbone comparison. Worm obtains a CartPole peak of 63 versus 83 for MLP, and a LunarLander peak of 12.1 versus 45.5 for LSTM. These results and the short learning curves do not establish competitive performance at useful task proficiency. This is not a requirement to achieve state of the art: the issue is whether the evaluation can support the claimed performance–parameter trade-off. CartPole-v1 has a discrete action space, despite the description of all four tasks as continuous control, and the observation/action adapters for the eight-neuron circuit are unspecified. It is also unclear whether the counts include adapters and whether actor and critic budgets are consistently handled.

    The available methods description is insufficient to verify the implementation. The supplied 13-page PDF ends with references, although Section 3.6 and Section 4 repeatedly refer to supplementary hyperparameters, Appendix Table 5, evolutionary comparisons, and parameter ablations. No separate supplement was present in the local packet. Consequently, the exact adjacency and synapse constraints, ODE integration and differentiation, policy distributions, recurrent-state handling, and task adapters cannot be checked. Equation (1) additionally lists membrane potentials among optimized parameters without distinguishing dynamic state from a learned initial condition. These details matter to both reproducibility and the claim of biological constraint preservation.

Questions for authors
  1. What is the substantive advance over ordinary neural circuits and related circuit policies? Do any existing experiments compare the biological wiring with a rewired or matched-size sparse circuit using the same dynamics and optimization budget, or test a specific functional circuit interpretation?
  2. Please provide final held-out evaluation returns with variability across the existing seeds, the training budget in environment steps, and exact actor/adapter/critic parameter accounting. How are inputs and outputs mapped for each Gym task, particularly the discrete CartPole action?
  3. Please identify the missing supplementary file and the sections specifying topology, parameter bounds, integration, gradient computation and state handling. In Equation (1), is V a dynamic membrane state, a trainable initial condition, or another parameter?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.