acceptodds
Under review as a conference paper at ICLR 2027

HiCA-RNN: Multi-Scale Credit Assignment for Long-Term Dependency Learning

Abstract

Windowing a recurrent computation changes how credit is transported; adding intermediate targets changes where credit enters. Neither is sufficient alone: reinitializing the state at each window boundary does not shorten the backward horizon unless the boundary is also detached, and intermediate targets supply local training signals without preserving long-range credit. In this work we introduce HiCA-RNN that combines both within a single fast–slow recurrence. A shared ungated cell processes each window, a gated slow state connects the windows, and a feedback recurrence supplies detached targets to intermediate fast states. The states remain connected across the full sequence: the task loss is trained by full-graph BPTT, while the target loss introduces credit directly at selected states. We derive the resulting adjoint and identify the contribution of each target to the shared-parameter gradient. The derivation exposes a constraint on the method: local credit is useful only when its magnitude, its transport through the slow dynamics, and its direction relative to the task gradient are favorable. Neither short windows nor orthogonal feedback ensures these properties. HiCA-RNN achieves on sequential MNIST and on permuted MNIST, exceeding tuned LSTMs on both, obtains the lowest Copy loss on all horizons, and matches or improves on burn-in LSTMs in eight of twelve system-identification settings. On sequential CIFAR-10 HiCA-RNN uses approximately less peak training memory than a parameter-matched LSTM, which achieves slightly better accuracy; LSTM also remains stronger on Adding.

Primary area:
learning on time series and dynamical systems
Keywords:
recurrent neural networks, temporal credit assignment, long-range dependencies, backpropagation through time, local representation alignment, target propagation, multiscale recurrent architectures, fast–slow recurrence, feedback alignment, sequence modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zhYNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HiCA-RNN, a recurrent framework that combines multi-scale fast–slow dynamics with detached intermediate representation targets to address long-term credit assignment. An ungated fast recurrent cell processes inputs within fixed-length windows, while a gated slow state transfers information across windows. A feedback recurrence generates auxiliary targets for intermediate fast states, introducing additional learning signals while preserving full-graph backpropagation through time (BPTT).

Critical strengths

The paper provides a useful perspective on long-term credit assignment by distinguishing how gradients are transported through recurrent dynamics from where additional learning signals enter the computation graph. The adjoint decomposition in Theorem 1 makes the contribution of detached intermediate targets explicit while preserving the exact task-gradient paths. I appreciate that the theoretical analysis acknowledges its limitations: neither windowing nor orthogonal feedback alone guarantees gradient preservation or beneficial task descent.

Critical weaknesses

W1. Serious inconsistencies in experimental results and evaluation protocols.

The Copy-task results contain several contradictions. The Abstract claims the best performance at all four horizons, while Section 5 (lines 359–361) claims only three and states that HiCA-RNN performs worse than LSTM at T=100. However, Table 2 shows that HiCA-RNN achieves lower mean loss at all four horizons, including T=100 (0.000433 vs. 0.4950). More seriously, Section 5 and Appendix B.2 describe Copy loss as computed over the full sequence, whereas Appendix G.1 states that blank positions are excluded. These are fundamentally different evaluation protocols. Additionally, Table 7 specifies 500 training epochs, while Figure 6 shows 1000 epochs for T=50 and T=100. These inconsistencies undermine confidence in the reported results.

W2. An implementation flaw and missing critical ablation experiments.

Appendix H.4 (lines 1690–1698) explicitly acknowledges that gradient clipping includes residual feedback gradients from the previous iteration, potentially changing the actual optimization updates. It is unclear whether the reported results remain valid after correcting this issue. More importantly, the paper lacks the essential ($\lambda=0$) ablation, making it impossible to determine whether improvements arise from the proposed auxiliary target mechanism or simply from the fast–slow architecture, gating, normalization, and regularization. Without this control, the paper's central methodological claim is not convincingly supported.

Questions for authors
  1. Copy-task inconsistencies: Why do the Abstract, Section 5, and Table 2 report contradictory conclusions? Is Copy cross-entropy computed over the full sequence or only the recall positions? Please clarify the exact training and evaluation metrics used for all models.

  2. Training budget: Why does Table 7 specify 500 epochs while Figure 6 shows 1000 epochs for T=50 and T=100? Were the results in Table 2 obtained using identical training budgets and checkpoint-selection procedures?

  3. Implementation correctness: Were the reported experiments conducted with the gradient-clipping issue described in Appendix H.4? Can the authors provide corrected results after excluding the retained feedback gradients from the main clipping operation?

  4. Attribution: Can the authors provide a matched ($\lambda=0$) ablation to isolate the contribution of auxiliary targets?

4 comments
AuthorsOpenReview ↗

Rebuttal Response

Response to Reviewer zhYN

We thank the reviewer for identifying the reporting inconsistencies and requesting the auxiliary-target ablation. We have corrected the implementation, updated the results, and added the requested control. All revisions are marked in red.

1. Copy metric and consistency of results

The reported Copy metric is mean cross-entropy over the 10 recalled symbols, excluding blank positions, for every model. The learning curves also show full-sequence cross-entropy, which is labelled separately.

We have corrected the text to match the updated results: HiCA-RNN achieves lower mean recall cross-entropy than LSTM at all four tested horizons, including T=100T=100. The Abstract, Section 5, and Table 2 now state the same conclusion.

2. Training budget and model selection

All models use the same maximum epoch budget and early-stopping procedure for each synthetic task, and hyperparameters are optimized for every model. Training can finish earlier when the stopping criterion is met.

The original Copy experiments allowed 1,000 epochs at T=50,100T=50,100 and 500 epochs at T=150,200T=150,200, consistently across models within each setting. We have corrected the uniform 500-epoch description. The revised Copy comparisons use a 300-epoch maximum for every model at every horizon. We report best and final results separately; image checkpoints are selected using validation data.

For SYSID, we also include published LSTM results from Schiller et al. (ICLR 2026), with their original training settings. Their released RLC implementation uses 1,500 epochs for stateless and stateful TBPTT, and 97,500 epochs for full BPTT. We identify these results as published baselines.

3. Implementation correction and evidence that the gains remain

The earlier implementation retained feedback gradients that could affect clipping. We now clear both optimizers’ gradient buffers before computing the current gradients, then apply gradient-norm clipping with threshold 1. The updated results use this corrected implementation and the training recipe documented in Appendix H.4, including fresh synthetic data each epoch.

The corrected comparison uses approximately 50k parameters per model. Final Copy recall cross-entropy, reported as mean ± standard deviation over three seeds, is:

Horizon TTLSTMHiCA-RNN
500.0444±0.04790.0444 \pm 0.04798.19×10−5±3.22×10−58.19\times10^{-5} \pm 3.22\times10^{-5}
1001.97±0.1031.97 \pm 0.1032.10×10−4±3.06×10−52.10\times10^{-4} \pm 3.06\times10^{-5}
1500.633±0.1520.633 \pm 0.1520.00238±0.001420.00238 \pm 0.00142
2001.97±0.191.97 \pm 0.190.00562±0.005020.00562 \pm 0.00502

HiCA-RNN’s advantage therefore remains after the implementation correction and is present in the final results at every horizon.

4. Contribution of the auxiliary targets

Appendix I adds the requested λLRA=0\lambda_{\mathrm{LRA}}=0 ablation, with the architecture and VJP regularizer (μ=0.1\mu=0.1) held fixed.

In the corrected seed-0 Copy experiment at T=100T=100, enabling the auxiliary targets reduces best recall cross-entropy from 1.17×10−41.17\times10^{-4} to 6.47×10−56.47\times10^{-5}, a 44.7% reduction. Both settings achieve perfect exact-match accuracy, so the improvement here is in cross-entropy.

The corresponding permuted-MNIST ablation improves test accuracy from 96.43% to 97.04%, with checkpoints selected on a 5,000-image validation split.

These controls show that auxiliary targets provide an additional benefit while the architecture, gating, normalization, and regularizer remain unchanged.

Dear Reviewer zhYN,

As the rebuttal discussion period is approaching its conclusion, we wanted to kindly follow up on our responses and the additional experiments provided in the revised manuscript. We would be happy to address any remaining concerns or provide further clarifications if needed. If our responses and additional evidence have satisfactorily addressed your concerns, we would sincerely appreciate your reconsideration of the rating. Thank you again for your time, thoughtful feedback, and consideration.

Thank you for the rebuttal. After re-reading the revised manuscript, I acknowledge that the multiscale T/WT/W fast–slow mechanism and theoretical decomposition meaningfully alleviate vanishing gradients to a certain extent. Accordingly, I will raise my Informativeness sub-score from 1 to 2.

However, I am not convinced by the experimental rigor and consistency of the revised paper, so I will maintain my overall rating of 2 due to three unresolved contradictions on the Copy task alone in the revised PDF:

  1. Conclusions: Abstract (line 26) and Table 2 claim HiCA-RNN wins on all 4 horizons, whereas Introduction (line 68) still says "three of four horizons" and Section 5 (lines 358–361) still states "at T = 100 its mean is above LSTM with a large seed variance."
  2. Metric: Section 5 (lines 356, 364–365) and Appendix B.2 (line 978) still state that Copy loss "includes blank positions" / is "over the full sequence," contradicting Appendix G.1 and the rebuttal.
  3. Configs and budgets: Table 7 (line 1459) and Appendix G.1 (line 1508) list 500 epochs and HiCA parameter counts of 274,322 / 168,626 / 52,786 / 199,762, whereas Figure 6 shows 300 epochs, different parameter counts (274k / 274k / 274k / 70k), and different optimization settings, while the rebuttal introduces yet another ~50k setup. It is unclear which exact recipe produced Table 2.

Finally, why HiCA-RNN's convergence curves exhibit severe instability and sharp spikes?

AuthorsOpenReview ↗

Clarification of Copy-Task Results and Experimental Consistency

We thank the reviewer for carefully identifying these inconsistencies. During the rebuttal, we added several experiments requested by the reviewers, but overlooked corresponding updates in parts of the manuscript. We acknowledge this oversight and have now reconciled the results, tables, and experimental descriptions.

  1. Copy results and metric: HiCA-RNN achieves the lowest mean cross-entropy at all four horizons (T=50,100,150,200T=50,100,150,200). We have corrected the Abstract, Introduction, Section 5, and appendices to consistently report pattern-wise cross-entropy computed only over the 10 recalled symbols, excluding blanks.

  2. Configurations and training budgets: Table 2, Table 7, and Figure 6 now describe the same 300-epoch experiments, using fresh training samples each epoch and three seeds. RNN, LSTM, and HiCA-RNN were independently tuned using Optuna under the same trial budget, including hidden size and model capacity. The differing parameter counts therefore reflect validation-selected configurations. Separate fixed-budget experiments were added to address reviewers' concerns about whether the gains persist without increasing model size.

  3. Convergence spikes: Figure 6 shows unsmoothed test cross-entropy. Parameter updates can temporarily disrupt recurrent memory and produce confident incorrect predictions, leading to sharp loss increases. We use AMSGrad (β2=0.9999\beta_2=0.9999, zero weight decay) to moderate these fluctuations, although they are not entirely eliminated. All three seed trajectories are shown without smoothing.

We have also added an intermediate-target ablation and comparisons with Fast–Slow LSTMs and Transformers. We believe these corrections and additional experiments address the concerns about consistency, reproducibility, and the practical value of HiCA-RNN.

Reviewer_6r4FOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Effective solutions to the vanishing gradient problem include GRU and LSTM; multi-rate RNNs and local learning targets are proposed solutions that have been far less effective. This manuscript proposes to add local learning targets to a multi-rate RNN, and demonstrates that the resulting architecture is competitive with an LSTM on many of the sample tasks that are used to demonstrate the difference between LSTMs and previous non-gated RNNs.

Critical strengths
  • Local learning targets are defined in a remarkably innovative fashion, by projecting memory discontinuities backward through a projection matrix E_0 that is optimized per batch based on forward-propagation outputs.
  • Analysis is quite strong, e.g., Proposition 2 uses the results of Theorem 1 to define bounds on the vanishing/exploding gradient problem that are not yet computable on theoretic grounds, but that could be measured in any given experimental run; Propositions 3 and 4 demonstrate conditions under which the local learning targets improve convergence toward the global task, providing step size limits that could be computed in any given expeirmental run
  • Computational cost and performance results are shown to be comparable to an LSTM and sometimes superior
Critical weaknesses
  • The proposed architecture sometimes outperforms LSTM by a factor of five, sometimes underperforms LSTM by a factor of five. It's not clear whether these differences have meaning; it seems not? The conclusion seems to be only that HiCA is as good as LSTM, which may not yet be sufficient reason for other researchers to use it.
  • The limitations noted in the "Limitations" section are all pretty important, and contribute to the feeling that this is still somewhat a work in progress: experimental tasks are limited, horizons are limited, obvious ablations are not performed (the impact of multi-resolution BPTT and of local learning targets are not separately measured), obvious applications of theory are not yet tested.
Questions for authors
  • Performing some of the studies suggested in "Limitations" of the current draft would be helpful to connect the interesting theoretical analysis with the experiments.
  • Is it possible to be a little more explicit about the reasons one might choose HiCA vs. LSTM or vice versa?
  • There are some typos, e.g., "equation Eq." occurs several times.
2 comments
AuthorsOpenReview ↗

Rebuttal Response

Response to Reviewer 6r4F

We thank the reviewer for recognizing the target-construction mechanism and theoretical analysis. We agree that the practical motivation should be clearer. We have added all our changes red in the updated manuscript.

1. Practical value beyond matching LSTM

At approximately 50k parameters and across three seeds, our additional comparisons show:

Task and metricLSTMHiCA-RNN
Copy, T=100T=100: final recall cross-entropy ↓1.972.10×10−42.10\times10^{-4}
Temporal order, T=150T=150: final accuracy ↑25.22%100%
Temporal order, T=200T=200: final accuracy ↑26.03%100%
Permuted MNIST: validation-selected test accuracy ↑91.69±0.8391.69\pm0.83%95.31±0.7495.31\pm0.74%

These results establish a concrete reason to consider HiCA for delayed retrieval and order retention under a small parameter budget. LSTM’s stronger Adding results indicate that this advantage depends on the task’s memory requirements.

2. Connecting the mechanism to experiments

We added auxiliary-target and regularization controls in Appendices I and H.4. With the architecture and regularizer fixed, the corrected seed-0 ablation improves permuted-MNIST accuracy from 96.43% to 97.04% and reduces Copy cross-entropy at T=100T=100 by approximately 45%.

These controls quantify the auxiliary targets’ incremental contribution, while the three-seed comparisons evaluate the complete method. They provide empirical support for the mechanism; independently isolating the multiscale schedule and measuring the theory’s alignment conditions remain further evaluations.

3. RLC and burn-in

For RLC, HiCA achieves errors comparable in scale to standard LSTM at N=100N=100 and 200200. This should be distinguished from the separately reported LSTM with validation-optimized burn-in m∗m^*. The reference benchmark explicitly treats burn-in as a training hyperparameter that can substantially affect performance (Schiller et al., 2026).

We employed a similar burn-in strategy, but a dedicated search over mm, including its interaction with HiCA’s window length WW, is outside this study. Standard GRUs and vanilla RNNs likewise require selecting mm for their own dynamics under this protocol. An LSTM-selected m∗m^* therefore cannot be presumed optimal for HiCA. The RLC results characterize performance under the chosen procedure and do not establish an intrinsic limitation of HiCA on continuous dynamics.

We have also corrected the equation-reference inconsistencies.

Dear Reviewer 6r4F,

As the rebuttal discussion period is approaching its conclusion, we wanted to kindly follow up on our responses and the additional experiments provided in the revised manuscript. We would be happy to address any remaining concerns or provide further clarifications if needed. If our responses and additional evidence have satisfactorily addressed your concerns, we would sincerely appreciate your reconsideration of the rating. Thank you again for your time, thoughtful feedback, and consideration.

Reviewer_JpiJOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The model, HiCA-RNN, combines the advantages of Fast-Slow RNN backbone (Mujika et al, 2017, NeurIPS), LRA-styled intermediate training loss (Ororbia & Mali, 2019, AAAI; Ororbia et al., 2023, AAAI) and full-graph BPTT. The target mechanism has advantages over ordinary LSTM especially for long-term dependency learning. More specifically, the loss function is L=Ltask+λLa+μRproxyL=L_{\text{task}}+\lambda L_a+\mu R_{\text{proxy}}, where λLa\lambda L_a is one of the key novelty of the model by inserting LRA-like losses to Fast-Slow RNN.

Critical strengths

The paper is well-structured and easy to read.

Critical weaknesses
  • (1) HiCA-RNN combinates the advantages of Fast-Slow RNN and LRA. But comparison with these two models is absent. There is no ablation study, like setting λ=0\lambda=0 and μ=0\mu=0 to support the validity of the combined model. We appreciate the authors listing this as a limitation (Sec 6), but this should not be an immunity, and I believe such ablation study is indispensable because all contributions of this paper count on its performance, which should have shown an improvement against its ingredients.

  • (2) The advantage over LSTM is not satisfactorily stable and consistent. The paper also lacks direct comparison with related modern models, such as transformers (Vaswani et al., 2017, NeurIPS), Mamba (Gu & Dao, 2023) and S4 (Gu et al., 2022, ICLR)

  • (3) All the ingredients of the proposed methods are not novel. Combined with point (2), this makes the paper somewhat insular.

Questions for authors
  • (1) How does HiCA-RNN perform compared to the ablated models like Fast-Slow RNN and recurrent LRA?

  • (2) How does HiCA-RNN perform compared to the existing models like Transformers, Mamba and S4?

  • (3) Would there be a more general rule (or rule-of-thumb) on when this model outperforms LSTM and vice versa? Or can the authors explain just for the dataset used in the paper, in a heuristic way?

9 comments
AuthorsOpenReview ↗

Rebuttal Response Part-1

Response to Reviewer JpiJ

We thank the reviewer for the questions. We first clarify that HiCA-RNN is not FS-RNN combined with recurrent LRA. All revisions are highlighted in red in the updated manuscript.

1. Architectural and methodological distinction

FS-RNN performs multiple fast transitions and one slow update per input timestep (Mujika et al., 2017). HiCA instead processes successive observations through a shared ungated fast cell, updates its gated slow state once per window, and initializes the next fast trajectory through a differentiable bridge. These are different temporal computation graphs. Although FS-RNN permits different cell types, Mujika et al. evaluate an LSTM-based instantiation; our comparison follows this FS-LSTM configuration.

Forward complexity: For sequence length TT, equal fast/slow width hh, input dimension dd, and two fast cells, the dominant dense costs are T(4dh+20h2)T(4dh+20h^2) MACs for FS-LSTM and T(dh+h2)+5⌈T/W⌉h2T(dh+h^2)+5\lceil T/W\rceil h^2 for HiCA, excluding output heads. Both remain linear in TT, but HiCA amortizes its slow-state and bridge computation over windows of length WW.

Distinction from LRA through time. The more directly related approach is RNN-LRA-diff, studied by Manchev and Garcia-Peraza-Herrera (2025). It operates on a temporally unrolled RNN, constructs each intermediate target from the next timestep’s local loss, and updates recurrent weights using local objectives. Their analysis shows that this decomposition alone does not prevent vanishing learning signals; directional gradient regularization improves performance, but the benefit remains task-dependent. Thus LRA wasn't able to beat TBPTT trained RNNs, in our results we have shown that we beat RNNs by an order of mangnitude.

HiCA addresses a different credit-assignment formulation. It retains task-gradient BPTT through the connected fast–slow computation graph and adds detached auxiliary targets derived from cross-window discrepancies through learned feedback. Thus, HiCA does not replace the global task gradient with a cascade of local objectives. Its slow memory and differentiable bridges determine how information and gradients move between windows, while its auxiliary targets introduce additional learning signals within those windows.

Theorem 1 characterizes how these additional signals enter the recurrent adjoint, while Propositions 3–4 establish conditions for beneficial task descent. Thus, the contribution includes a specific cross-window target construction and an analysis of its interaction with global task gradients. Setting λLRA=0\lambda_{\mathrm{LRA}}=0 retains HiCA’s architecture and task-gradient paths; it does not recover FS-RNN or recurrent LRA.

2. Requested comparisons and attribution

We added a parameter-matched FS-LSTM comparison in Appendix J and the λLRA=0\lambda_{\mathrm{LRA}}=0 control in Appendix I. These address complementary questions: the former evaluates the complete method against an alternative recurrent architecture, while the latter isolates the auxiliary learning signal within HiCA.

At approximately 50k parameters, the seed-0 Copy comparison at T=100T=100 gives recall cross-entropy of 1.96×10−41.96\times10^{-4} for HiCA versus 1.768 for FS-LSTM under the shared training recipe. In the separate corrected seed-0 ablation, enabling the auxiliary targets reduces Copy cross-entropy from 1.17×10−41.17\times10^{-4} to 6.47×10−56.47\times10^{-5}, approximately 45% lower, and improves permuted-MNIST accuracy from 96.43% to 97.04%.

These results support both the integrated method and the incremental contribution of its auxiliary targets. The λ=0\lambda=0 experiment is a controlled attribution study, given recurrent-LRA as shown (https://arxiv.org/pdf/2504.13531), struggles beyond T=100, we have not included it in the current results.

3. Modern architectures

Our central question concerns credit assignment in recurrent networks trained through BPTT. The Transformer baseline reproduced in the S5 paper uses approximately 500k parameters (Trinh et al., 2018), while S5 employs dataset-specific depths, widths, and state dimensions (Smith et al., 2023, Table 11). These published results provide useful context but do not establish a controlled ranking against our approximately 50k-parameter models.

Our matched recurrent comparisons and auxiliary-target ablation directly examine the proposed mechanism. Broader matched comparisons with Transformers, Mamba, and S4/S5 remain future work. Separately, transferring the mechanism would require corresponding target constructions and gradient analyses for attention, S4’s convolutional training formulation, or S5’s recurrent states evaluated through parallel scans.

AuthorsOpenReview ↗

Rebuttal Response part-2

4. When should HiCA help?

The strongest observed gains concern delayed recall, temporal order, and permuted inputs, where specific information must survive intervening observations. Adding admits a compact running accumulator, for which LSTM remains strong. This provides a plausible task-based explanation for the results: the structure of the memory requirement, rather than sequence length alone, determines where HiCA’s multiscale credit assignment is most useful.

Reviewer_JpiJOpenReview ↗

Reply to rebuttal

We appreciate the authors clarifying the model's architecture, especially its difference with FS-RNN combined with LRA. However, since it is still not demonstrated why we would prefer the proposed model to the more modern, widely-used, and thoroughly tested models, and neither is it shown that the synthesis of different model components indeed outperforms its individual components, we would like to maintain our rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.