acceptodds
Under review as a conference paper at ICLR 2027

Feedback-gated weight modulation enables state-dependent computation in recurrent neural networks

Abstract

Recurrent neural networks (RNNs) are useful models of sequential processing in cortical circuits but omit a defining feature of cortical organization: top-down feedback that dynamically regulates synaptic integration in lower order regions. In cortical pyramidal neurons, feedforward drive tends to arrive at proximal dendrites, whereas feedback carrying contextual and behaviorally relevant information often targets the distal apical dendrites. Nonlinear interactions between these streams can selectively reshape neuronal responses. To emulate this organization, we introduce a new RNN architecture, Gating on Weights with Feedback (GaWF), in which a standard RNN receives feedback gating signals computed from the subsequent layer's activations. These gates modulate individual weights on both the input and hidden weight matrices. This stands in contrast to gated RNN architectures, such as LSTMs and GRUs, which dynamically control information flow through gates computed on feedforward inputs and local hidden states that then act on unit-level representations. We evaluated GaWF on a new dynamic cluttered target-tracking task that requires simultaneous inference of target identity and location. Both the target and distractors could change dynamically. GaWF consistently outperformed conventional gated RNNs and other recurrent and state-space models, achieving higher held-out accuracy with less overfitting. Its gates produced sparse weight matrices that track the current target. Further mechanistic analyses revealed a functional division: input-weight gating predominantly tracked target location, whereas recurrent-weight gating represented target identity. Neither LSTMs nor GRUs segregated their gating signals based on the properties of the tracked target; instead, their gates remained largely undifferentiated, which may have contributed to their overfitting. These findings identify top-down connection-specific feedback gating as an effective and interpretable mechanism for context-dependent recurrent computation and highlight the functional importance of the interaction between hierarchical feedback in cortical circuits and dendritic computation.

Primary area:
applications to neuroscience & cognitive science
Keywords:
recurrent neural networks, gating, dendritic integration, top-down feedback, state-dependent computation, sequence modeling, neuroscience-inspired architectures

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9S2WOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a recurrent architecture, termed Gating on Weights with Feedback (GaWF), and find that relative to parameter-matched RNNs, LSTMs, GRUs, and SSMs it can outperform them on a cluttered target-tracking MNIST task.

Critical strengths

I enjoyed the parameter-matched controls (Fig. 2) and detailed analyses of the internals of the GaWF (Fig. 11), including the topographic matching in Fig. 5.

Critical weaknesses
  • The differences between the GaWF and RNN on test accuracy in Fig. 2a appear to be quite small.

  • There is only one task as far as I can tell, and it is quite toy and synthetic (based on MNIST images).

More details on these weaknesses with my questions, in the "Questions" section below.

Questions for authors
  • The task is synthetic (and there really is only one, CM-MNIST), what about on real images? Say you take ImageNet images and add clutter? Could you compare to the multi-layer Reciprocal Gated Cells (RGC) with feedback (Nayebi et al. NeurIPS 2018: https://arxiv.org/abs/1807.00053) and CORNet-S ConvRNNs (Kubilius et al. NeurIPS 2019: https://arxiv.org/abs/1909.06161) on these images?

I suggest this since both of these models have top-down feedback and have been applied to much larger scale datasets such as ImageNet. In particular, if you happen to be short on time during the rebuttal stage, the Reciprocal Gated Cell (RGC) of Nayebi et al. 2018 does segregate its feedback signal in the cell architecture itself, so it may be the most relevant of the two comparisons, though comparing to both would be ideal.

  • The models used are quite small, what happens if it is increased while keeping parameter count across models the same -- do the model differences remain the same? Or is this not possible on the current MNIST task because it will overfit otherwise given the smallness of the task? This overfitting issue can be alleviated by training on larger naturalistic datasets like ImageNet. The reason I ask this is because I worry any benefit from GaWF may be due to a "law of small numbers" effect, and may not hold on larger models and larger, more naturalistic, datasets (which also end-up producing brain-like responses in the primate ventral visual pathway, as ImageNet does, cf. Yamins et al. PNAS 2014).

  • In Appendix A.2 pg. 13, the initializations of the encoders are said to be the same, but I am not sure if this means the RNNs/SSMs are initialized the exact same. If that's the case, this is quite suboptimal for various choices of these models -- as their initialization schemes (especially for SSMs) really matter (e.g. HiPPO). What happens if you do the optimal initialization schemes, along with optimal parameters, for each model family?

2 comments

Thank you for the constructive review and for the positive comments on the parameter-matched controls and the analyses of the GaWF internals.

Effect size, model scale, and a second task family. We also examined whether the benefits of the GaWF design extend to a different, harder task and are retained in a multi-layer architecture. In preliminary experiments, we trained a three-layer GaWF with DQN on Atari Breakout from pixels for 3M agent steps, with four-frame action repeat and frame stacking and the minimal four-action set. In this multi-layer GaWF, the first two layers receive feedback from the previous hidden state of the adjacent higher layer, and the third receives the previous step's four action values from the readout. GaWF, RNN, LSTM, and GRU each used three recurrent layers, approximately 7.55M parameters (about 13 times the CM-MNIST budget), and 10 seeds. GaWF has the highest mean of the four models in final 100-episode training return (mean ± SEM, ε = 0.05):

GaWFRNNLSTMGRU
128.74 ± 2.17119.02 ± 13.8199.00 ± 2.8249.01 ± 5.13

Initialization. Appendix A.2 (p. 13) states that all architectures instantiate the same encoder class under the same PyTorch initialization rules; this refers only to the CNN encoder. Each sequence model keeps the default initialization of its reference implementation: our S5 and Mamba wrappers instantiate the library modules without overriding their internal parameters. S5 therefore retains its diagonalized HiPPO-based state-matrix initialization with log-uniform step sizes, and Mamba retains the default initialization of the mamba_ssm module.

Thank you for running this. I will raise my score from 1 to 2, and I encourage the authors to continue to develop their study with other biologically-motivated RNN architectures (the RGC of Nayebi et al. 2018 and CORNet-S of Schrimpf et al. 2019, both of whose PyTorch implementation can be found in, e.g. https://github.com/neuroagents-lab/PyTorchTNN and https://github.com/dicarlolab/CORnet/tree/master, respectively) and realistic tasks (e.g. ImageNet) as I have highlighted in my initial review, which weren't addressed.

Reviewer_ELPhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work claims to introduces a novel RNN architecture modelling top-down feedback signals as gating signals that modulate local weights.

Specifically, the mechanism is that the activation in a layer generates gates over the preceding layer, providing a computational abstraction of feedback-dependent local modulation. In contrast with LSTMs etc, in this architecture the gates cross layers, adding explicit pathways through which a downstream representation can selectively modulate individual input or recurrent connections. The authors introduce a task (Clutter-Moving MNIST) designed to test context- and history-dependent stimulus processing, requiring the tracking of location and identity of a MNIST digit moving in a straight line against other random digits moving randomly. They explore how the proposed architecture reshapes the effective input and recurrent connectivity of the lower layer. They compare with Elman RNNs, LSTMs, GRUs, S5 and Mamba architectures.

Critical strengths
  1. The idea seems interesting, but its unclear how novel it is (see below).
  2. Interesting task used. The novel task the authors introduce, CM-MNIST, requires the model to maintain and update both target identity and target location, constituting a setting where state-dependent reconfiguration of local weights should be useful.
  3. Reasonably broad baseline comparisons: GaWF-RNN outperformed Elman RNNs, LSTMs, GRUs, S5 and Mamba in continuous target tracking, especially in maintaining target identity. GaWF had the highest accuracy on both target identity (73%) and location (82.6%). The RNN was the best next architecture with 67.4% and 81.0%, respectively.
  4. The feedback-shuffle analysis is a well-controlled perturbation. Shuffling preserves the marginal distribution of each segment but eliminates its correspondence with the current frame. The network depends heavily on its gate input, almost entirely through the location half: shuffling location costs 38.1 points on location and 36.8 on identity; scrambling identity costs 0.8 and 4.5.
Critical weaknesses
  1. Its unclear how novel this approach is. For example HyperLSTM do something similar, and there are other related approaches. This should be discussed.

  2. Also, there is no proper discussion of the biological motivation for this. Why should we consider weights to be modulated directly by other brain regions? This is perhaps more in light with how people think of neuromodulation, models of neuromodulation that control RNNs have been proposed before, this should be discussed. More importantly what is the evidence for distal/proximal controlling directly the weights?

  3. The biological motivation for GaWF centers on top-down feedback from a higher processing stage. However, it seems that in the actual experiments (Eq. 5) the feedback is generated by linear readouts of the same recurrent hidden state, so it adds no information beyond ht-1. This setup tests whether a task-aligned projection of the network’s own state can multiplicatively modulate its weights but does not directly test feedback from a separate, higher-level representation. To better support the biological motivation, the authors need to evaluate the stacked configuration from Appendix A.1, or preferably a setup in which the upper layer receives contextual information that is not available to the lower layer, compared against the single-layer model and against delivering the same cue to the lower layer additively.

  4. Baseline comparisons. The authors mention multiplicative RNNs, e.g., multiplicative LSTM (Krause et al., 2016), in the Discussion but do not compare with it (see also other relevant models as per point 1 above). A comparison with mLSTMs would test whether GaWF's advantage comes from input-conditioned multiplicative recurrent dynamics more generally, rather than, specifically, from top-down feedback gating. Another useful comparison would be HyperNetworks (Ha et al, 2016) (not cited) and which would test whether GaWF’s advantage arises from dynamically modulating recurrent and input parameters, rather than, again, its feedback-gating mechanism.

  5. Related work should be discussed, for example: modulatory top-down feedback in hierarchical recurrent ANNs is missing and should be cited: Tugsbayar et al. (eLife, 2026) implement apical-dendrite-inspired multiplicative top-down feedback signals between hierarchically organised cortical regions. Also previous proposed mapping between "AI" RNNs and cortical networks (e.g. Costa et al. NeurIPS 2017).

  6. Importantly, its unclear whether this network is better than previous models. Is it more parameter efficient? Or should we expect better performance on harder tasks?

Questions for authors

See points above.

3 comments

Thank you for the thoughtful review and the pointers to related work and biological motivation.

Novelty relative to HyperNetworks and related models. HyperLSTM (Ha et al., 2016) rescales the rows of the main weight matrices, so all connections entering a unit share one scaling factor. In GaWF, each connection receives its own gate, so connections entering the same unit can be modulated differently. GaWF thus combines established ideas: top-down feedback, connection-specific weight modulation, and analysis of which task variable each gate carries.

Biological motivation. The first paragraph of the Introduction sets out the motivation: long-range feedback often targets distal dendritic branches, where it shapes how proximal driving inputs are integrated. Biological modulation takes two forms: neuromodulation broadcasts a largely shared signal, whereas dendritic modulation is compartment-specific. Inputs to distal dendrites, such as feedback, are integrated nonlinearly with proximal drive and thereby determine how that drive shapes neuronal output (Larkum et al., 1999; Larkum, 2013). Such dendritic signals can be vectorized, differing across neurons (Francioni et al., 2025), and contribute to rule-dependent flexible behavior (Zhang et al., 2026). Moreover, dendritic morphology separates drivers from modulators of neuronal activity (Makarov et al., 2023). We therefore conceive of top-down feedback as a modulator that tunes dendritic nonlinearity branch by branch, for example by depolarizing local membranes and by clustering with feedforward synapses to elicit NMDA spikes. GaWF abstracts this function as W_eff,t = G_t ⊙ W: base weights stay fixed during inference, and gates change each connection's instantaneous contribution.

Source of the feedback signal. We agree that, in the single-layer model, the readout feedback is a function of h_{t−1} and adds no information beyond it. What it adds is a constraint: the gates are driven by a low-dimensional, output-relevant representation (10 identity and 9 location logits) instead of the full h_{t−1}, which is the inductive bias built into the single-layer model.

We also examined whether the benefits of the GaWF design extend to a different, harder task and are retained in a multi-layer architecture. In preliminary experiments, we trained a three-layer GaWF with DQN on Atari Breakout from pixels for 3M agent steps, with four-frame action repeat and frame stacking and the minimal four-action set. In this multi-layer GaWF, the first two layers receive feedback from the previous hidden state of the adjacent higher layer, and the third receives the previous step's four action values from the readout. GaWF, RNN, LSTM, and GRU each used three recurrent layers, approximately 7.55M parameters, and 10 seeds. GaWF has the highest mean of the four models in final 100-episode training return (mean ± SEM, ε = 0.05):

GaWFRNNLSTMGRU
128.74 ± 2.17119.02 ± 13.8199.00 ± 2.8249.01 ± 5.13

Parameter efficiency. With feedback of the same width, a GaWF layer with input dimension m and hidden dimension n has 2nm + 3n² + 4n parameters including LayerNorm, versus 4nm + 4n² + 8n for an LSTM and 3nm + 3n² + 6n for a GRU. For readout feedback, the gate generator needs only d(m + 2n) = 31,616 parameters (d = 19, m = 1152, n = 256), and unlike HyperLSTM, no auxiliary recurrent controller. With the factor width set to n, as in the original models, an mLSTM (Krause et al., 2016) has 5nm + 5n² + 8n parameters, whereas an mRNN (Sutskever et al., 2011) needs fewer, 2nm + 2n² + n, because it replaces its hidden-to-hidden matrix with an input-dependent factorized product. In contrast, GaWF keeps the base weights W_in and W_rec and treats the gates as the multiplicative modulation on them, not as the weight matrices themselves.

Reviewer_ELPhOpenReview ↗

Reply

Thank you for your detailed response. If the primary contribution of this work is its biological motivation, the framing needs to be much more explicit — both throughout the main text and visually in Figure 1. While the proposed model is an interesting direction, the current manuscript feels somewhat preliminary. Clarifying the mapping to neuroscience and incorporating the more advanced tasks you mentioned in your rebuttal would significantly strengthen the paper. Conversely, if the focus is on empirical performance, the evaluation requires a more thorough comparison against modern LSTM and GRU variants. I will maintain my current score, but I encourage you to continue developing this work for a future submission.

Thank a lot for your interest and suggestions. We agree that the current manuscript is a first step toward this direction and will extend the model to more demanding tasks. More thorough behavioral comparisons and targeted ablations are also planned to establish the feasibility of this biological inspiration. Thank you again for the encouragement; your feedback will guide our continued development of this work.

Reviewer_Q1GYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GaWF: a connection-level gate Gt=σ(1τ∑kft,k ukvk⊤)G_t=\sigma\big(\tfrac{1}{\tau}\sum_k f_{t,k}\,u_kv_k^{\top}\big), computed from the subsequent layer's activations at the preceding time step, modulates the input and recurrent weights of the layer below, giving an RNN state-dependent, connection-level computation. The architecture pairs a shared convolutional encoder, in-recurrence LN + ReLU + dropout, and two linear readout heads, and is compared on a custom CM-MNIST task (a moving target plus randomly moving distractors, with uncued switches) against five parameter-matched models (RNN/LSTM/GRU/Mamba/S5, 10 seeds each). The authors also report mechanistic analyses (U-shaped gate distributions, a two-path variance decomposition under the balanced condition, sign- and magnitude-dependent gate modulation, and the net change in recurrent current by weight sign). The experiments show that the identity readout lies above every parameter-matched baseline (next best the RNN), and likewise for the location readout; the input gate predominantly carries location and the recurrent gate predominantly identity; shuffling the sector-segment feedback substantially lowers both readouts. The main experiment is a single-layer configuration with unprojected output feedback, and the feedback vector is the 19 raw logits of that layer's two linear readout heads.

Critical strengths
  1. The gate modulates the lower layer's input and recurrent weights connection by connection. The subsequent layer's activation at the preceding time step is expanded through the UU and VV factors, divided by the gate temperature τ\tau and passed through a sigmoid, and is then multiplied elementwise onto WinW_{\mathrm{in}} and WrecW_{\mathrm{rec}} (Eqs. (1)–(3), pp. 2–3; A.1, p. 12); the abstract calls this mechanism top-down connection-specific feedback gating (p. 1), and §2.1 calls it connection-specific feedback (p. 2). This places the granularity of feedback modulation on each individual input and recurrent connection, so that "top-down, connection-by-connection" modulation has a directly instantiable definition.
  2. Using the condition-mean variance decomposition under the balanced condition (Figure 4B), the paper reports the division of labour between the two gates: the input gate is location-dominated and the recurrent gate identity-dominated, and both are more concentrated than the encoder and hidden representations they read (pp. 6–7). The abstract describes this division of labour as a functional division (p. 1). The gates thus concentrate the dominant variables already present in the source representations; the readout does not depend on accuracy, so that whether the two gates specialize can be assessed independently.
Critical weaknesses
  1. The abstract writes the conclusion as superiority over an entire class of models, but the primary comparison differs in two respects, regularization and the breadth of the hyperparameter search. All six models are parameter-matched, but only GaWF and the RNN carry in-recurrence LN + ReLU + dropout (A.2, p. 13); The additive-feedback controls of D.4 likewise have no in-recurrence dropout (p. 17).
  2. In the experiments the "subsequent layer" is the task readout layer feeding back to the recurrent layer: the main experiment is single-layer, and the feedback is the raw logits of the task readout layer (pp. 4, 5, 12); p. 2 states that "we focus here on its application to a single-layer RNN", while A.1 only says that the implementation supports stacked layers (p. 13) and reports no recurrent-layer-to-recurrent-layer stacking result.
Questions for authors
  • Can the authors add the same LN + ReLU + dropout inside the recurrence of LSTM/GRU and an analogous normalization block to Mamba/S5 (or remove them from GaWF and the RNN), reselect the baseline hyperparameters under the same search protocol, and then rerun Figure 2 (under the same LR/WD grid and regularization configuration)? Apart from the GaWF-vs-RNN pair, how much of the remaining gap survives? Is the gap on the location readout outside seed-to-seed variability?
  • The RNN/GRU/LSTM additive-feedback variants in D.4 have no in-recurrence dropout (GaWF and the reported, non-feedback RNN do); after rerunning with the same dropout, does the main-text sentence "feedback availability alone does not account for the advantage of GaWF" (p. 5) still hold?
  • Is there a retrained ablation in which GaWF's gate is reduced to the unit level (units serving the same purpose sharing one gate value)? F.2 notes that the existing unit-level comparison "leaves connection-level granularity confounded with the use of output-derived feedback"; such an ablation would allow the contribution of connection-level granularity to be separated from that of output-derived feedback.
  • The digit-segment shuffle produces only a small drop, so the claim that any feedback perturbation lowers the readout does not account for the observed difference; has a non-feedback placebo (shuffled inputs or an equal-length random vector) and an identity permutation also been run? D.3 already states that the permutation preserves the marginal distribution.
  • Are there results on a second task family or an external dataset besides CM-MNIST? If not, how does the paper delimit the scope of "top-down connection-specific feedback gating as an effective and interpretable mechanism"?
  • How does the paper position itself relative to the line in which a signal generated on one side modulates the weights or gates on the other side: multiplicative RNNs and fast weight programmers (Krause et al., 2016; Irie & Gershman, 2025), and using feedback as an attention-like signal to select modules (Mittal et al., 2020)?
  • τ\tau is fixed at 0.5 in A.1 (p. 12), and the main text contains no ablation on τ\tau; if τ\tau is increased, do the U-shaped gate distribution and the "quasi-binary" reading (p. 5) still hold?
1 comment

Thank you for the careful reading and the precise questions. We address them below.

Regularization and the location readout. We retrained all six architectures with the same dropout configuration (no dropout in the recurrent layer) and evaluated them under the protocol of Figure 2 (Appendix B.2; mean ± SEM, 10 seeds):

ModelTarget identity (%)Target location (%)
GaWF67.86 ± 0.3080.02 ± 0.18
RNN65.37 ± 0.5179.94 ± 0.29
LSTM62.12 ± 0.2877.18 ± 0.24
GRU57.14 ± 0.4474.57 ± 0.41
Mamba61.48 ± 0.2680.54 ± 0.18
S535.72 ± 0.4152.85 ± 0.47

GaWF retains the highest identity accuracy, and its location accuracy is comparable to that of Mamba.

Feedback perturbation. Section 3.1 reports an asymmetric dependence on the two feedback segments: the sector shuffle substantially lowers both readouts, whereas the digit shuffle modestly lowers digit accuracy and largely preserves sector accuracy (Figure 2D). As a placebo, for each of the 10 GaWF checkpoints, we fitted a Gaussian to each feedback segment from validation logits, matching its mean and full covariance; samples from it therefore preserve the segment's statistics but carry no correspondence with the current input. At test time, we replaced one segment at every time step with an independent sample from this Gaussian, while the other segment kept its original value. We repeated the Figure 2 evaluation under each condition:

Feedback conditionTarget identity (%)Target location (%)
Original67.86 ± 0.3080.02 ± 0.18
Random identity segment60.74 ± 0.4079.58 ± 0.14
Random location segment30.69 ± 0.5747.90 ± 0.65

Replacing the identity segment mainly lowers identity accuracy, whereas replacing the location segment lowers both readouts, so the shuffle and this placebo both support an asymmetric dependence on feedback.

Second task family and stacked layers. In preliminary experiments, we trained a three-layer GaWF with DQN on Atari Breakout from pixels for 3M agent steps, with four-frame action repeat and frame stacking and the minimal four-action set. In this multi-layer GaWF, the first two layers receive feedback from the previous hidden state of the adjacent higher layer, and the third receives the previous step's four action values from the readout. GaWF, RNN, LSTM, and GRU each used three recurrent layers, approximately 7.55M parameters, and 10 seeds. GaWF has the highest mean of the four models in final 100-episode training return (mean ± SEM, ε = 0.05):

GaWFRNNLSTMGRU
128.74 ± 2.17119.02 ± 13.8199.00 ± 2.8249.01 ± 5.13

Relation to multiplicative RNNs, fast weights, and module selection. GaWF shares the motivation of this line of work. Multiplicative RNNs were designed to allow flexible input-dependent transitions (Sutskever et al., 2011; Krause et al., 2016), fast weight programmers reconfigure weights at test time from the feedforward input (Irie & Gershman, 2025), and feedback-based module selection uses top-down signals to choose which modules are active (Mittal et al., 2020). GaWF builds on this line, with its design inspired by dendritic computation, in which feedback changes the influence of individual driving inputs. This inspiration shapes three choices. First, the modulating signal is the previous step's task readout, so modulation arrives top-down. Second, GaWF uses the multiplicative factorization U Diag(f_t) V^T, conditioned on the readout feedback f_t, as sigmoid gates on a fixed base weight matrix (Eq. 2). Third, the resulting gates act on individual input and recurrent connections, whereas Mittal et al. use feedback to select modules or modulate hidden states.

Gate temperature. We varied τ at inference time on the 10 checkpoints trained at τ = 0.5, so that the altered gates also shaped the subsequent feedback. Raising τ from 0.5 to 2 lowered the fraction of saturated gates (below 0.05 or above 0.95) from 74.16% to 21.89% in the input pathway and from 69.02% to 15.44% in the recurrent pathway. The quasi-binary gating reported on p. 5 is therefore specific to the low training temperature (τ = 0.5) and is not a structural property of GaWF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.