acceptodds
Under review as a conference paper at ICLR 2027

Learning to Route Recurrent State from Input History

Abstract

Can input history improve recurrent computation by controlling how stored state coordinates interact? We introduce Input-Conditioned Graph Routing (ICGR), a recurrent cell whose compact controller summarizes the input prefix and selects a graph-supported skew-symmetric operator from shared learned factors. The routed state current and encoded input form a bounded drive, combined with a contextual correction under an exponential retention update. Saturation and convex retention guarantee bounded main-state activations for arbitrary sequence length, without guaranteeing gradient stability. We study the mechanism through paired controls and routing interventions. Under identical parameter instantiation—not necessarily equal effective capacity—history conditioning outperforms learned time schedules and current-input-only conditioning on permuted sequential MNIST and character-level Penn Treebank. A symmetric routing control with matched support and prior spectral norm performs worse under the tested pSMNIST protocol. Donor trajectories and training-derived class templates redirect predictions toward the supplied class, while a frozen linear probe decodes class labels from the complete routing trajectory near complete-model accuracy. ICGR is competitive across four benchmarks under our evaluated configurations, although several pSMNIST baseline reruns fall below their published results and Penn Treebank shows mixed parameter efficiency. These findings identify class-bearing, history-conditioned routing signals as a task-relevant information pathway in the trained models, without establishing improved retention of target input history.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
recurrent neural networks, sequence modeling, history-conditioned routing, dynamic recurrent transitions, structured recurrent dynamics, graph-supported operators, long-range dependencies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EBkROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper describes an approach to improving recurrent networks by letting a representation of the input history influence hidden-to-hidden state transitions. Experimental results show that the proposed model shows good performance in comparison to several baselines.

Critical strengths

The paper seems to describe a method that allows state transitions in a recurrent network to be conditioned on a representation of the input history. This is an intriguing idea that to the best of my knowledge has not been widely explored in the literature.

Critical weaknesses

Most of the paper is complete gibberish. Even though I am quite familiar with recurrent networks there are many terms and statements whose meaning I do not understand and that should be introduced before use. A small sample: What does this mean: “The routed state current and encoded input form a bounded drive” What does this mean: “a recurrent cell selecting graph-supported skew operators from a learned K-factor family” What are “donors” and “donor maps”?

There is plenty of unused space that could have been used for providing a readable motivation, introduction of terms and discussion of intermittent results. So this is not a space issue.

In some of the experimental evaluations, results for comparing methods are below their published performance values and are instead based on a re-implementation. For the reported performance values to be meaningful they would need to be derived from detailed sweeps over hyperparameters.

Questions for authors

Line 107: If h_1 is the initial hidden state and h_k occurs before x_k, does that mean the input sequence starts with x_2?? This seems confusing.

Results are shown on seemingly random collections of compared-to models (for example, why are there GRU and Mamba results in the experiment from Table 3 (b) but not in the experiments from Table 3 (a), and vice versa for HyperRNN, NM-RNN and others?)

Section 4: What is the context for these propositions, how are they motivated, why are they useful?

Reviewer_Ka8HOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces 'Input-Conditioned Graph Routing' which uses a separate ‘history controller’ RNN to modify the recurrent weight matrix of another RNN via quite a complicated structure.

They show how this architecture performs on a host of benchmarks, showing good performance on the pSMNIST and noise-padded CIFAR-10 tasks.

They include ablations which help show how parts of the architecture operates.

Critical strengths

The authors include many benchmarks. The performance on pSMNIST is better than the author’s re-runs of the comparisons. The performance and active parameter count of ICGR are good on noise-padded CIFAR-10.

The authors highlight the ablation results which show that the model does better when the controller is given the input history. They also show better performance on pSMNIST with a particular operator structure.

Critical weaknesses

The method is quite complicated and I’ve not grasped all the details. But unfortunately it’s hard interpreting the results as justifying this complexity.

In the results for the benchmark Character PTB it seems to perform a little better than a GRU with a lower parameter count but worse than a GRU with a high parameter count, making it hard to make any claims about ICGR here. In the other benchmarks it is the best performer, but the differences appear to be quite small.

I’m confused as to why for every experiment there is a different selection of comparison models. Parameter counts vary wildly. I don’t think they have to be identical, but it’s only really useful if they’re close.

Authors note reruns are below published baselines. It might not be a problem, but worth understanding why.

The method trains much slower than an LSTM (on Table 7), though it’s really not clear to me if the gains are worth the cost.

I’m not sure what the major take-away from the ablations really is. Should we be surprised that the outputs of the controller contain class information? Perhaps the main take-away could be stated more clearly.

Questions for authors

Why do the selections for models to compare change each time? For example, GRU is the best performer in Character PTB, but it’s not in pSMNIST, IMDb or noise-padded CIFAR-10.

Will you be able to re-run the comparisons with similar parameter counts?

Reviewer_TEHxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new type of recurrent cell called ICGR (input-conditioned graph routing). The authors combine a number of ideas in their proposed cell architecture, with the graph-supported skew-symmetric operator being its central part. The authors test their architecture in a number of benchmarks and show strong performance. They also ablate elements of their cell in particular investigating the effect of a skew-symmetric dynamic in their graph-mix submodule, showing its importance.

Critical strengths

The paper is thorough in its evaluations, supporting its claims with multiple experiments and ablations.

The proposed cell shows strong performance in a number of tasks.

Critical weaknesses
  1. Novelty & significance. While the paper recombines existing ideas in a sufficiently novel way, the key contribution seems to be in a very specific choice of the defining elements in this particular combination.

This, combined with meaningful but not dramatic improvements in performance, less than ideal asymptotic for throughput, as well as testing mostly restricted to a relatively limited number of benchmarks that are not directly relevant in practice, substantially limits potential impact.

  1. Presentation and terminology. While this is not my primary expertise, I am familiar with the literature. With this contribution, I had the impression that the authors routinely used nonstandard terminology, sometimes without carefully defining what exactly they mean. It oftentimes made it difficult to understand the paper, not just when it comes to detail, but also the general intuitions behind the specific architectural choices that were made.

  2. The theory section of the paper, in my view, does not sufficiently aid in developing strong intuitions about model behavior. For example, this is the conclusion after a fairly technical theoretical section: “The bound follows by induction from saturation, the bounded correction, and coordinate-wise convex retention, not skew-symmetry; it implies neither convergence nor gradient stability.” To me this doesn’t clarify the importance of the derivation before that. Neither did Appendix B, although I did not read it with equal scrutiny.

At present, I find this submission very borderline and will be open to adjust my score up or down after familiarizing myself with the other reviewer’s opinions and the authors responses.

Questions for authors
  1. Do you foresee the application of your cell in more modern & challenging settings?

  2. Apart from coming up with implementational optimizations, is there a chance to improve compute-normalized performance of your model?

  3. Your model shows promising performance per trainable parameter. What, do you think, is it in your architectural decisions that allows it. I believe it’s the most interesting part of the paper, but, in my view, there aren’t enough experiments or theoretical results to develop a strong intuition for why this is happening. To clarify: I am not saying that there isn’t enough theory in the paper. In fact, the theory section is already quite substantial. But, in my view, it currently reads more as a potentially interesting side observation than a way to develop clear intuitions about model behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.