- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HiCA-RNN, a recurrent framework that combines multi-scale fast–slow dynamics with detached intermediate representation targets to address long-term credit assignment. An ungated fast recurrent cell processes inputs within fixed-length windows, while a gated slow state transfers information across windows. A feedback recurrence generates auxiliary targets for intermediate fast states, introducing additional learning signals while preserving full-graph backpropagation through time (BPTT).
Critical strengths
The paper provides a useful perspective on long-term credit assignment by distinguishing how gradients are transported through recurrent dynamics from where additional learning signals enter the computation graph. The adjoint decomposition in Theorem 1 makes the contribution of detached intermediate targets explicit while preserving the exact task-gradient paths. I appreciate that the theoretical analysis acknowledges its limitations: neither windowing nor orthogonal feedback alone guarantees gradient preservation or beneficial task descent.
Critical weaknesses
W1. Serious inconsistencies in experimental results and evaluation protocols.
The Copy-task results contain several contradictions. The Abstract claims the best performance at all four horizons, while Section 5 (lines 359–361) claims only three and states that HiCA-RNN performs worse than LSTM at T=100. However, Table 2 shows that HiCA-RNN achieves lower mean loss at all four horizons, including T=100 (0.000433 vs. 0.4950). More seriously, Section 5 and Appendix B.2 describe Copy loss as computed over the full sequence, whereas Appendix G.1 states that blank positions are excluded. These are fundamentally different evaluation protocols. Additionally, Table 7 specifies 500 training epochs, while Figure 6 shows 1000 epochs for T=50 and T=100. These inconsistencies undermine confidence in the reported results.
W2. An implementation flaw and missing critical ablation experiments.
Appendix H.4 (lines 1690–1698) explicitly acknowledges that gradient clipping includes residual feedback gradients from the previous iteration, potentially changing the actual optimization updates. It is unclear whether the reported results remain valid after correcting this issue. More importantly, the paper lacks the essential ($\lambda=0$) ablation, making it impossible to determine whether improvements arise from the proposed auxiliary target mechanism or simply from the fast–slow architecture, gating, normalization, and regularization. Without this control, the paper's central methodological claim is not convincingly supported.
Questions for authors
-
Copy-task inconsistencies: Why do the Abstract, Section 5, and Table 2 report contradictory conclusions? Is Copy cross-entropy computed over the full sequence or only the recall positions? Please clarify the exact training and evaluation metrics used for all models.
-
Training budget: Why does Table 7 specify 500 epochs while Figure 6 shows 1000 epochs for T=50 and T=100? Were the results in Table 2 obtained using identical training budgets and checkpoint-selection procedures?
-
Implementation correctness: Were the reported experiments conducted with the gradient-clipping issue described in Appendix H.4? Can the authors provide corrected results after excluding the retained feedback gradients from the main clipping operation?
-
Attribution: Can the authors provide a matched ($\lambda=0$) ablation to isolate the contribution of auxiliary targets?
4 comments
Dear Reviewer zhYN,
As the rebuttal discussion period is approaching its conclusion, we wanted to kindly follow up on our responses and the additional experiments provided in the revised manuscript. We would be happy to address any remaining concerns or provide further clarifications if needed. If our responses and additional evidence have satisfactorily addressed your concerns, we would sincerely appreciate your reconsideration of the rating. Thank you again for your time, thoughtful feedback, and consideration.
Thank you for the rebuttal. After re-reading the revised manuscript, I acknowledge that the multiscale fast–slow mechanism and theoretical decomposition meaningfully alleviate vanishing gradients to a certain extent. Accordingly, I will raise my Informativeness sub-score from 1 to 2.
However, I am not convinced by the experimental rigor and consistency of the revised paper, so I will maintain my overall rating of 2 due to three unresolved contradictions on the Copy task alone in the revised PDF:
- Conclusions: Abstract (line 26) and Table 2 claim HiCA-RNN wins on all 4 horizons, whereas Introduction (line 68) still says "three of four horizons" and Section 5 (lines 358–361) still states "at T = 100 its mean is above LSTM with a large seed variance."
- Metric: Section 5 (lines 356, 364–365) and Appendix B.2 (line 978) still state that Copy loss "includes blank positions" / is "over the full sequence," contradicting Appendix G.1 and the rebuttal.
- Configs and budgets: Table 7 (line 1459) and Appendix G.1 (line 1508) list 500 epochs and HiCA parameter counts of
274,322 / 168,626 / 52,786 / 199,762, whereas Figure 6 shows 300 epochs, different parameter counts (274k / 274k / 274k / 70k), and different optimization settings, while the rebuttal introduces yet another ~50k setup. It is unclear which exact recipe produced Table 2.
Finally, why HiCA-RNN's convergence curves exhibit severe instability and sharp spikes?
Clarification of Copy-Task Results and Experimental Consistency
We thank the reviewer for carefully identifying these inconsistencies. During the rebuttal, we added several experiments requested by the reviewers, but overlooked corresponding updates in parts of the manuscript. We acknowledge this oversight and have now reconciled the results, tables, and experimental descriptions.
-
Copy results and metric: HiCA-RNN achieves the lowest mean cross-entropy at all four horizons (). We have corrected the Abstract, Introduction, Section 5, and appendices to consistently report pattern-wise cross-entropy computed only over the 10 recalled symbols, excluding blanks.
-
Configurations and training budgets: Table 2, Table 7, and Figure 6 now describe the same 300-epoch experiments, using fresh training samples each epoch and three seeds. RNN, LSTM, and HiCA-RNN were independently tuned using Optuna under the same trial budget, including hidden size and model capacity. The differing parameter counts therefore reflect validation-selected configurations. Separate fixed-budget experiments were added to address reviewers' concerns about whether the gains persist without increasing model size.
-
Convergence spikes: Figure 6 shows unsmoothed test cross-entropy. Parameter updates can temporarily disrupt recurrent memory and produce confident incorrect predictions, leading to sharp loss increases. We use AMSGrad (, zero weight decay) to moderate these fluctuations, although they are not entirely eliminated. All three seed trajectories are shown without smoothing.
We have also added an intermediate-target ablation and comparisons with Fast–Slow LSTMs and Transformers. We believe these corrections and additional experiments address the concerns about consistency, reproducibility, and the practical value of HiCA-RNN.
Rebuttal Response
Response to Reviewer zhYN
We thank the reviewer for identifying the reporting inconsistencies and requesting the auxiliary-target ablation. We have corrected the implementation, updated the results, and added the requested control. All revisions are marked in red.
1. Copy metric and consistency of results
The reported Copy metric is mean cross-entropy over the 10 recalled symbols, excluding blank positions, for every model. The learning curves also show full-sequence cross-entropy, which is labelled separately.
We have corrected the text to match the updated results: HiCA-RNN achieves lower mean recall cross-entropy than LSTM at all four tested horizons, including T=100. The Abstract, Section 5, and Table 2 now state the same conclusion.
2. Training budget and model selection
All models use the same maximum epoch budget and early-stopping procedure for each synthetic task, and hyperparameters are optimized for every model. Training can finish earlier when the stopping criterion is met.
The original Copy experiments allowed 1,000 epochs at T=50,100 and 500 epochs at T=150,200, consistently across models within each setting. We have corrected the uniform 500-epoch description. The revised Copy comparisons use a 300-epoch maximum for every model at every horizon. We report best and final results separately; image checkpoints are selected using validation data.
For SYSID, we also include published LSTM results from Schiller et al. (ICLR 2026), with their original training settings. Their released RLC implementation uses 1,500 epochs for stateless and stateful TBPTT, and 97,500 epochs for full BPTT. We identify these results as published baselines.
3. Implementation correction and evidence that the gains remain
The earlier implementation retained feedback gradients that could affect clipping. We now clear both optimizers’ gradient buffers before computing the current gradients, then apply gradient-norm clipping with threshold 1. The updated results use this corrected implementation and the training recipe documented in Appendix H.4, including fresh synthetic data each epoch.
The corrected comparison uses approximately 50k parameters per model. Final Copy recall cross-entropy, reported as mean ± standard deviation over three seeds, is:
HiCA-RNN’s advantage therefore remains after the implementation correction and is present in the final results at every horizon.
4. Contribution of the auxiliary targets
Appendix I adds the requested λLRA=0 ablation, with the architecture and VJP regularizer (μ=0.1) held fixed.
In the corrected seed-0 Copy experiment at T=100, enabling the auxiliary targets reduces best recall cross-entropy from 1.17×10−4 to 6.47×10−5, a 44.7% reduction. Both settings achieve perfect exact-match accuracy, so the improvement here is in cross-entropy.
The corresponding permuted-MNIST ablation improves test accuracy from 96.43% to 97.04%, with checkpoints selected on a 5,000-image validation split.
These controls show that auxiliary targets provide an additional benefit while the architecture, gating, normalization, and regularizer remain unchanged.