acceptodds
Withdrawn from ICLR 2027

Cortico-cerebellar modularity as an architectural inductive bias for efficient temporal learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The cerebellum and cerebral cortex establish closely integrated loops that are involved in temporal processing; however, the manner in which their distinct circuit architectures influence learning remains ambiguous. This study investigates whether the integration of recurrent dynamics with a cerebellar-inspired feedforward pathway alters the learning of temporal computations. We enhance a recurrent neural network (RNN) by incorporating an expansion–compression module that feeds back into the recurrent dynamics, with both pathways being trained concurrently towards the same task objective. In the context of curriculum-trained temporal cognition tasks, this cortico-cerebellar RNN (CB-RNN) demonstrates more efficient and adaptive learning compared to parameter-matched recurrent networks across single-task, multitask, and task-switching scenarios. Limiting the plasticity of recurrent weights retains much of this learning advantage, suggesting that learning can increasingly be carried by the feedforward pathway while recurrent dynamics provide a substrate for computation. However, causal ablation indicates that the resulting recurrent representations remain reliant on cerebellar input. Mechanistic analyses reveal that this interaction is organised differently depending on task demands. Collectively, these findings illustrate that recurrent–feedforward modularity can enhance temporal learning without imposing a rigid functional decomposition between pathways, thereby allowing a task-dependent division of labour to emerge through joint optimisation.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Recurrent neural networks, modular neural networks, inductive biases, sequence learning, temporal credit assignment, learning dynamics, biologically inspired neural networks, cortico-cerebellar networks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wimxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether integrating recurrent dynamics with a cerebellar-inspired feedforward pathway enhances temporal learning. The proposed CB-RNN combines an Elman RNN with an expansion-compression module that modulates recurrent activity, with both pathways jointly optimized under a shared task objective. The authors report improved learning efficiency over parameter-matched RNN baselines across delayed match-to-sample and parity tasks under single-task, multitask, and task-switching settings, and suggest that the architecture enables an emergent, task-dependent division of labor.

Critical strengths
  1. The paper is clearly written and well organized. Exploring cortico-cerebellar architectures as an inductive bias for brain-inspired computing is an interesting and potentially valuable research direction.

  2. The proposed CB-RNN demonstrates substantial improvements in learning efficiency over parameter-matched RNN baselines across the evaluated temporal learning tasks and training settings.

Critical weaknesses
  1. Although the proposed architecture is inspired by cortico-cerebellar circuits, DMS and Parity are generic temporal learning tasks without a clear link to specific cortico-cerebellar computations. Moreover, the evaluation is limited to two relatively simple binary sequence tasks, and the learning advantage is substantially reduced with a GRU backbone. It remains unclear whether the proposed architecture captures a general computational principle or primarily benefits specific tasks and recurrent architectures.

  2. Although the architecture is motivated by the cerebellar expansion–compression circuit, the specific contributions of these operations remain insufficiently characterized. Increasing the expansion-layer size improves performance, but this may reflect increased feedforward capacity rather than a specific architectural benefit. Parameter-matched controls and representational analyses are needed to determine whether expansion and compression play distinct computational roles in temporal learning.

  3. The experiments use fixed optimization settings without reporting systematic searches over learning rates, recurrent-weight initialization, or optimizer choices. Given the sensitivity of recurrent networks to these factors, it is unclear whether the reported efficiency gains reflect the proposed architectural inductive bias or suboptimal training of the RNN-only baselines. Comparable hyperparameter tuning across architectures is needed to establish the robustness of the performance advantage.

  4. The authors interpret their mechanistic findings as evidence for an emergent, task-dependent division of labor between cortical and cerebellar pathways. However, the architecture contains two recurrent pathways: the internal recurrence of the RNN and the feedback loop through the cerebellar module. The latter introduces additional nonlinear transformations into the recurrent state update, potentially enhancing computational depth and expressivity. Thus, the observed advantages may reflect enhanced recurrent dynamics rather than functional specialization between modules. A unified causal analysis and appropriate architectural controls are needed to distinguish the effects of inter-module feedback from an emergent division of labor.

Questions for authors
  1. Are the learning-efficiency gains robust to hyperparameter choices? Can the authors compare RNN and CB-RNN under different learning rates, weight initializations, and optimizers to rule out suboptimal baseline training?

  2. How much of the performance gain comes from the expansion–compression structure rather than increased computational depth? Can the authors vary network depth and redistribute parameters between the expansion and compression layers while keeping the total parameter count fixed?

  3. Can the authors evaluate a temporal cognitive task with established cortico-cerebellar involvement? The current DMS and Parity tasks are relatively simple and do not clearly reflect cortico-cerebellar computations. A more biologically motivated task would help establish the generality and neuroscientific relevance of the proposed architecture.

  4. Does the learning advantage extend to other recurrent architectures, such as LSTMs? Given the reduced benefit observed with GRUs, it remains unclear whether the proposed architecture offers a general advantage or mainly compensates for limitations of simple RNNs.

Reviewer_tQ8YOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Summary

  • The paper investigates whether architectural features inspired by cortico-cerebellar interactions can improve temporal learning in recurrent neural networks.
  • The proposed CB-RNN augments a cortical RNN with a feedforward expansion–compression module that transforms the current input and previous recurrent state into a modulatory contribution to each recurrent update.
  • The authors evaluate the model on N-bit delayed match-to-sample and N-bit parity tasks, using single-task, multitask, and task-switching curricula. They report faster curriculum progression and higher maximum task difficulty than approximately parameter-matched recurrent-only baselines.
  • Restricted-plasticity experiments show that much of the learning-speed advantage persists when recurrent weights are frozen after initial training, although permanent freezing can reduce the final difficulty reached.
Critical strengths

Strengths

  • Research direction: Understanding how cerebellar interactions influence cortical computation is a worthwhile question. The paper explores this through an RNN model augmented with feedforward cerebellar modulation and examines how the contributions of its recurrent and feedforward pathways vary with task demands.
  • Experimental comparisons: The principal comparisons approximately match total parameter counts, and the authors report results across multiple training runs. These choices make the observed performance differences more informative than comparisons based on a single run or unmatched model capacity.
Critical weaknesses

Weaknesses:

  • Lack of novelty and insufficient baselines: My main concern is that the biological framing carries more explanatory weight than the model and experiments support. The proposed cerebellar pathway implements a multilayer feedforward transformation of the current input and previous recurrent state. Computationally, this closely resembles a recurrent network with a deeper state transition, an established architectural direction. Consequently, improvement over a shallow recurrent baseline does not establish that cerebellar-specific organization explains the benefit.

    The authors should identify which biologically motivated features are essential to their hypothesis and test whether those features provide an advantage over generic nonlinear recurrent updates. A particularly relevant baseline would be a deep-transition RNN with comparable parameter count, input access, and training conditions. The existing parameter-matched recurrent-only and modular recurrent controls do not resolve this question. Without that comparison, the demonstrated contribution appears to concern nonlinear augmentation of recurrent computation, while the claims about cortico-cerebellar interactions remain insufficiently supported.

  • Limited task suite: The evaluation is confined to two synthetic binary sequence tasks. These provide controlled tests of memory and nonlinear updating, but offer limited evidence for the broader biological interpretation. More ecologically grounded temporal tasks would help establish whether the proposed organization captures useful aspects of cerebellar modulation beyond these narrow benchmarks.

  • Readability: The Abstract, Results, and Discussion are dense, making the central contribution and the relationship between analyses difficult to follow. The Abstract and Discussion would benefit from simpler wording and a clearer distinction between demonstrated results and biological interpretations. Some material in Sections 4.3 and 4.5 could be condensed or moved to the appendix, allowing the main text to explain the principal comparisons and their implications more coherently.

Questions for authors

Please refer to my weaknesses section above.

Reviewer_6bVzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents an augmented model of the Elman RNN, termed cortico-cerebellar RNN (CB-RNN). CB-RNN includes an additional module that non-linearly transforms inputs and previous step hidden states, modelled after the circuits of the cerebellar cortex. The authors tested CB-RNN against a vanilla RNN and two reservoir networks with (partially) frozen recurrent weights on two simple bit-memory tasks and their combined tasks. CB-RNN was demonstrated to beat all other competitor networks in almost all settings, with much shorter re-training time after a task switch.

Critical strengths

The brain-inspired modelling, notably the addition of the cerebellar module, is justified. The authors aligned the cerebellum's connectivities (projections matching the close loop from the cortex to deep nuclei back to the cortex) and functions (the DMS and Parity tasks the RNNs are tested on are loosely related to the working memory). Whilst this work may not present much interest to the ML audience, it does prompt an interesting discussion on the bias induced by the architecture of cerebellar computation on task-dependent pattern of reliance.

The experiment results show clear advantages over competitor networks, and the breadth of analysis makes it convincing.

The authors clearly discuss their findings, including limitations and future directions. Moreover, a reproducibility statement is included and the code is open-sourced.

Critical weaknesses

The architecture of CB-RNN is strikingly similar to Deep Transition RNN with Skip Connections (DTS-RNN, Pascanu et al., ICLR 2014), which also incorporated a MLP transforming input and hidden state for the hidden state update. Without having the DTS-RNN as a baseline model for comparison, it is hard to justify the cerebellar advantage. Given this, I had given a rating of 2.

In the reservoir computing comparisons, my rough estimate is that the majority (>90%) of trainable parameters lie in the cerebellar modules.

A restructuring of the figures could be considered for better readability. For example, I would have appreciated a single Figure with panel a of Figure 2, and panels of Figures 3 and 4.

The analysis left some open questions. I have left these in the questions section - I would be happy to raise the rating to 3 if the authors can answer them.

Questions for authors
  • Could the authors include a comparison with the DTS-RNN model on the same tasks?
  • Could the authors provide a mechanistic explanation of the advantages of the cerebellar module - for example, it could me stated what happens in the cerebellar module during training - how does the division of labour emerge during training? What does GC and PC units encode? Do they correspond to some neuroscientific observations?
  • Could the authors establish why CB-RNN recovers faster after the task switch?
  • As a food for thought, I would be interested to know what determines which task relies on the CB, in the context of division of labour.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.