acceptodds
Under review as a conference paper at ICLR 2027

Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrence

Abstract

As new evidence arrives, a sequence model must update both what it remembers and how that memory influences subsequent predictions. While Transformers incur computation and cache costs that scale with context length, fixed-state recurrent models offer constant-memory inference. However, linear and spectral recurrences traditionally rely on static, time-invariant transitions: fixed dynamics can append new content, but cannot dynamically revise how stored representations decay, rotate, or evolve. Although recent selective architectures introduce input-dependent transitions, they typically assign independent controls to every memory mode, coupling control cost directly to state capacity. We show that high-dimensional spectral memory does not require high-dimensional control, and introduce Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrence (SPARC). SPARC employs just two input-dependent scalar signals to coordinate memory retention and phase rotation across heterogeneous complex modes, while preserving mode-specific baseline timescales and frequencies. Its diagonal, affine recurrence supports parallel associative scans for sequence-level BPTT, as well as exact structured Real-Time Recurrent Learning (RTRL) for online credit assignment. Across partially observable continuous control, POPGym, and sequence classification, SPARC achieves a relative return improvement on Walker-P and a relative accuracy gain on FordA over second-best methods. On an NVIDIA Blackwell GPU, our implementation reduces recurrent-mixer training latency by – in fixed-token workloads and accelerates scans by – over an optimized RG-LRU baseline. These results show that two shared control signals can efficiently govern adaptive spectral memory across both online and full-sequence learning settings.

Primary area:
learning on time series and dynamical systems
Keywords:
Adaptive Spectral Recurrence, Recurrent Neural Networks, State Space Models, Real-Time Recurrent Learning, Parallel Scan

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vzCQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPARC, a diagonal complex recurrence that uses two shared input-dependent scalars (retention ctc_t and phase dtd_t) to modulate HH complex modes, coupled with a (1−ρj,t)(1-\rho_{j,t}) phase clock factor. It supports both parallel prefix scan for BPTT and exact structured RTRL for online RL. The authors also implement fused Triton kernels on Blackwell GPUs showing 3.1-4.7x scan speedup over RG-LRU.

Critical strengths
  1. Neat idea to decouple control dimension (2 scalars) from state dimension (HH modes).

  2. The retention-coupled phase clock (1−ρj,t)(1-\rho_{j,t}) makes physical sense (protects near-unit-circle persistent modes from phase noise) and shows strong empirical value in ablations (Table 11, CIFAR-10 drops ~11% without it).

  3. Solid Triton kernel engineering (recomputing modal transitions inside the scan to save HBM bandwidth by only storing two B×TB \times T control arrays) and thorough numerical verification (Table 21).

Critical weaknesses
  1. Unshared control baseline is missing: The whole paper argues that 2 shared controls are enough compared to 2H2H per-mode controls. Yet Table 3 and Line 1509 admit the unshared variant was never run! With 9 ablations in Appendix E, skipping the most important baseline (Unshared control) makes it impossible to tell how much expressivity is lost by sharing.

  2. Scale mismatch between speed tests and accuracy tests: Section 5 benchmarks speed at LLM widths (D=2048,2560D=2048, 2560), but Section 4 only tests accuracy/RL at D=64D=64. Having only 2 scalar degrees of freedom might work for D=64D=64, but won't it severely bottleneck capacity at D=2048D=2048? You need either wider model evaluations or a multi-head/grouped control study.

  3. Evaluation issues in RL and Classification: 1) In RL, SPARC uses full RTRL traces while Mamba-3 and GRU only use length-4 TBPTT (Table 4). Truncating BPTT to 4 steps on POMDPs obviously hurts Mamba-3/GRU. 2) In classification (Appendix D.5), you split the training set into fit/val, pick the best checkpoint on val, and report that same peak val accuracy instead of evaluating on the official test sets. Also, SLC and UWave val sets only have 100 and 90 samples.

Questions for authors

See "weaknesses" above.

4 comments
AuthorsOpenReview ↗

Direct control-sharing ablations and language modeling at D=2048

Thank you for your specific suggestions on direct ablations of control sharing, model-scale matching, and evaluation protocols. We have added grouped and Unshared control within the same cell, complete language-model training at D=2048D=2048, longer-window RL comparisons, and official TEST evaluations. Changes specific to your review are pale blue; shared reviewer requests are pale yellow.

1. Shared (G=1G=1), grouped (G=4,16G=4,16), and Unshared (G=HG=H) within the same cell

We compare G∈{1,4,16,H}G\in\{1,4,16,H\} using the same SPARC cell, state capacity, and training budget. Each group shares retention and phase controls ct,dtc_t,d_t, giving 2G2G dynamic scalars per layer; H=192H=192 for continuous control and H=64H=64 for the remaining tasks. Results are means ± sample SDs over three seeds; classification reports TEST accuracy (%).

TaskG=1G=1G=4G=4G=16G=16G=HG=H
Ant-P4665.31 ± 457.853565.62 ± 530.004212.76 ± 336.234023.15 ± 767.38
Walker-P1153.70 ± 322.29941.08 ± 232.11995.28 ± 94.39943.07 ± 197.16
Hopper-P1378.90 ± 139.941333.01 ± 81.391323.96 ± 203.641082.24 ± 275.25
Cheetah-P2868.21 ± 24.122641.21 ± 44.562489.62 ± 35.892888.18 ± 29.79
Autoencode-0.421 ± 0.012-0.438 ± 0.017-0.400 ± 0.019-0.408 ± 0.010
CountRecall-0.471 ± 0.045-0.505 ± 0.085-0.537 ± 0.025-0.416 ± 0.032
RepeatFirst0.817 ± 0.0450.777 ± 0.2200.610 ± 0.4340.783 ± 0.155
RepeatPrevious0.906 ± 0.0600.750 ± 0.0580.854 ± 0.1240.727 ± 0.247
Noisy Pendulum0.513 ± 0.0120.461 ± 0.0430.422 ± 0.0090.328 ± 0.108
HigherLower0.499 ± 0.0020.505 ± 0.0020.504 ± 0.0100.513 ± 0.005
FordA94.66 ± 0.1694.55 ± 0.3294.36 ± 0.7094.32 ± 0.64
SLC97.14 ± 0.0396.70 ± 0.6496.76 ± 0.2296.87 ± 0.33
UWave94.29 ± 0.8394.41 ± 0.7894.62 ± 0.6094.92 ± 0.80
pMNIST96.67 ± 0.2996.62 ± 0.1196.61 ± 0.1396.75 ± 0.29
sCIFAR-1063.11 ± 1.2661.08 ± 0.8562.79 ± 1.5762.88 ± 1.12

Increasing control dimensionality does not yield monotonic gains. Shared has 13 wins and 2 losses against G=4G=4, 12 wins and 3 losses against G=16G=16, and 9 wins and 6 losses against Unshared. Shared leads 9 tasks, Unshared 5, and G=16G=16 only Autoencode.

On reinforcement learning and noisy-memory tasks, Shared has higher means and smaller SDs than Unshared on Ant-P, Hopper-P, Noisy Pendulum, and RepeatFirst. Each mode retains distinct timescales and responses through νj,θj,sj,vj\nu_j,\theta_j,s_j,v_j, consistent with a regularizing effect of low-dimensional modulation under noisy policy gradients.

On supervised BPTT classification, Unshared is higher only on pMNIST and UWave, by 0.08 and 0.63 percentage points. Shared exceeds Unshared on FordA, SLC, and sCIFAR-10, including a 0.23-point gain on sCIFAR-10. Sharing reduces control-projection parameters from 2H(D+1)2H(D+1) to 2(D+1)2(D+1) and avoids materializing O(BTH)O(BTH) transition tensors in the scan.

2. Complete language-model performance and downstream evaluation at D=2048D=2048

To address the mismatch between wide-layer speed benchmarks and narrow-model task results, we trained 8-layer, width-20482048 language models for 19,073 updates each, totaling 4,999,872,512 tokens, with a 10M-token held-out TEST corpus. Both use the same decoder backbone and FineWeb-Edu training data; SPARC has 1024 complex modes and RG-LRU has 2048 real state coordinates per layer.

(1) Validation and held-out TEST perplexity

Method and recipeTraining tokensvalidation PPLTEST NLLTEST PPL
SPARC4,999,872,51222.4011293.09719622.135794
RG-LRU4,999,872,51222.8416843.11742822.588206

SPARC's TEST PPL is 22.14, 2.00% lower than RG-LRU, using two controls per layer.

(2) Eight downstream zero-shot tasks

TaskPrimary metricSPARC (%)RG-LRU (%)
piqaacc65.66963.602
hellaswagacc30.59229.974
arc_easyacc57.32356.355
arc_challengeacc22.69622.611
boolqacc61.16255.199
rteacc53.43050.903
winograndeacc50.67151.776

SPARC achieves higher accuracy on 6 of them, with gains of approximately 5.96 percentage points on BoolQ, 2.53 percentage points on RTE, and 2.07 percentage points on PIQA.

(3) Update time on two H800 GPUs

The measured median is 1.92 s/update for SPARC versus 2.29 for RG-LRU over 5,980 post-warmup updates per model, including data preparation, transfer, forward and backward computation.

AuthorsOpenReview ↗

BPTT-16 and RTRL baselines, official TEST evaluation, and LRA

3. TBPTT-16 and Mamba-3 RTRL, official TEST, and LRA

(1) Extending GRU and Mamba-3 to TBPTT-16 and adding Mamba-3 RTRL

We agree that TBPTT-4 is too short. Under multi-epoch PPO, 64–256-step unrolls incur substantial memory and time overhead. We extended GRU and Mamba-3 to TBPTT-16 and added Mamba-3 RTRL, matching the state budget and readout. POPGym uses 1,953 PPO updates. Results are means ± sample SDs over three seeds.

Continuous control

MethodAnt-PWalker-PHopper-PCheetah-P
RTU (RTRL)4757.32 ± 65.00928.70 ± 151.411369.88 ± 185.172855.31 ± 24.50
LRU (RTRL)4593.59 ± 759.21814.23 ± 98.811376.24 ± 84.582929.07 ± 6.58
RG-LRU (RTRL)2836.39 ± 502.32725.97 ± 174.55976.57 ± 161.772806.78 ± 13.65
GateLoop (RTRL)3807.76 ± 157.991033.30 ± 58.621336.70 ± 62.332638.09 ± 23.56
SPARC (RTRL)4665.31 ± 457.851153.70 ± 322.291378.90 ± 139.942868.21 ± 24.12
GRU (TBPTT-16)4230.12 ± 432.69957.05 ± 194.441128.68 ± 581.502615.44 ± 80.67
Mamba-3 (TBPTT-16)2143.40 ± 33.33784.51 ± 110.881117.52 ± 470.562867.58 ± 492.63
Mamba-3 (RTRL)1803.42 ± 142.19564.54 ± 54.32846.86 ± 241.492496.70 ± 252.47

POPGym

MethodAutoencodeCountRecallRepeatFirstRepeatPreviousNoisy PendulumHigherLower
RTU (RTRL)-0.443 ± 0.017-0.540 ± 0.0110.432 ± 0.0700.843 ± 0.0870.441 ± 0.1150.496 ± 0.002
LRU (RTRL)-0.424 ± 0.042-0.537 ± 0.3420.017 ± 0.3010.884 ± 0.0880.428 ± 0.0250.497 ± 0.004
RG-LRU (RTRL)-0.473 ± 0.010-0.582 ± 0.0880.564 ± 0.073-0.079 ± 0.0610.467 ± 0.0140.504 ± 0.001
GateLoop (RTRL)-0.425 ± 0.010-0.642 ± 0.0430.708 ± 0.0360.199 ± 0.0650.543 ± 0.0260.498 ± 0.001
SPARC (RTRL)-0.421 ± 0.012-0.471 ± 0.0450.817 ± 0.0450.906 ± 0.0600.513 ± 0.0120.499 ± 0.002
GRU (TBPTT-16)-0.472 ± 0.011-0.401 ± 0.1010.862 ± 0.0520.881 ± 0.0800.327 ± 0.0760.496 ± 0.000
Mamba-3 (TBPTT-16)-0.497 ± 0.002-0.669 ± 0.0880.605 ± 0.0340.347 ± 0.2480.355 ± 0.1140.491 ± 0.003
Mamba-3 (RTRL)-0.498 ± 0.001-0.816 ± 0.0830.199 ± 0.085-0.293 ± 0.0460.197 ± 0.0150.471 ± 0.020

TBPTT-16 substantially strengthens GRU, but SPARC is higher on all four control tasks, ranking 1st on Walker-P and Hopper-P and 2nd on Ant-P and Cheetah-P. On POPGym, SPARC has two 1st places and four 2nd places. SPARC exceeds Mamba-3 RTRL on all ten tasks, indicating that its gains do not arise solely from RTRL.

(2) Complete official TEST sets for the five classification tasks

All methods use complete official TRAIN sets and full-sequence BPTT for 20,000 updates, then one final evaluation on official held-out TEST sets (means ± sample SDs, three seeds):

MethodFordASLCUWavepMNISTsCIFAR-10
RTU94.70 ± 0.0097.02 ± 0.1594.60 ± 1.0496.47 ± 0.1961.36 ± 1.29
LRU94.66 ± 0.8096.89 ± 0.2694.12 ± 1.1596.44 ± 0.2061.54 ± 1.38
RG-LRU93.03 ± 0.1196.92 ± 0.2396.35 ± 0.0686.54 ± 0.2352.45 ± 1.08
GateLoop92.39 ± 0.0596.93 ± 0.1790.77 ± 0.5864.31 ± 2.4051.85 ± 0.34
Mamba-390.72 ± 1.6695.90 ± 0.2696.57 ± 0.4785.62 ± 4.9347.20 ± 2.36
GRU93.56 ± 0.2197.20 ± 0.0591.65 ± 0.3590.58 ± 0.5662.28 ± 0.64
SPARC94.66 ± 0.1697.14 ± 0.0394.29 ± 0.8396.67 ± 0.2963.11 ± 1.26

On the official TEST sets, SPARC ranks 1st on both pMNIST and sCIFAR-10; sCIFAR-10 reaches 63.11%, exceeding the sequential GRU's 62.28% by 0.83 percentage points; it ranks 2nd on both FordA and SLC, only 0.04 and 0.06 percentage points behind the respective leaders.

(3) Complete TEST sets of four LRA benchmarks (%)

LRA uses the same residual backbone and full-sequence BPTT, validation-selected checkpoints, and one run: 20,000 updates for Image and ListOps and 5,000 for Text and Retrieval.

MethodImageListOpsTextRetrieval
RTU62.63037.20082.76466.904
LRU61.69038.60082.22466.158
RG-LRU56.85039.05076.98865.631
GateLoop56.81039.00078.43663.474
Mamba-331.73037.95062.86056.965
GRU66.25045.30087.18865.590
SPARC68.34039.35083.21666.921

SPARC ranks 1st on Image and Retrieval and 2nd on ListOps and Text; it ranks 1st among all five parallelizable spectral baselines on every task.

We will include these results and protocols in the main text and appendices.

Thank you for the thorough rebuttal and the substantial experimental additions.

Resolved Concerns:

  • Control-Sharing Ablation: The comparison across G∈{1,4,16,H}G \in \{1, 4, 16, H\} directly answers my core question. It is insightful to see that while Shared (G=1G=1) and Unshared (G=HG=H) perform similarly on supervised BPTT classification, low-dimensional shared control provides a clear regularization benefit (higher returns and lower variance) in noisy online RL settings.
  • Fairer Baselines & Official Test Sets: Reporting official held-out TEST accuracies resolves the validation selection bias. Extending GRU/Mamba-3 to TBPTT-16 and adding Mamba-3 RTRL makes the RL evaluation much fairer—confirming that GRU remains competitive under TBPTT-16 while Mamba-3 struggles with online RTRL traces.
  • Scaling to D=2048D=2048: The 5B-token language modeling run on FineWeb-Edu shows that SPARC scales cleanly to D=2048D=2048, outperforming RG-LRU in both perplexity/zero-shot accuracy and training step time (1.92s vs. 2.29s).

Remaining Limitations:

  • The G=1G=1 vs. Unshared/Grouped ablation was only run at narrow widths (H=64/192H=64/192), while the D=2048D=2048 LM experiment only compares against RG-LRU. Testing G=4/16G=4/16 at D=2048D=2048 would better clarify whether 2 scalars eventually bottleneck capacity at large scales.
  • Please make sure all new tables are integrated into the final manuscript and Figure 2(b) is updated to match Eq. (31).

Overall, the new experiments solidify the paper's empirical claims. I am raising my score from 2 to 3.

AuthorsOpenReview ↗

Thank you for your constructive feedback

Thank you for reviewing our new results.

  • Grouped control at D=2048D = 2048: We will run the matched GG at D=2048D = 2048 to directly test whether two layer-wide scalars bottleneck capacity at larger scales.
  • Final manuscript and Figure 2(b): We confirm that all new experimental tables will be fully integrated into the final manuscript and appendices, and Figure 2(b) will be updated.

Thank you again for your time and constructive feedback, which have greatly helped us improve the paper.

Reviewer_sBwFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPARC, a diagonal complex-valued (spectral) linear recurrence in which two scalar, input-dependent control signals are shared across all modes of a layer. One signal rescales every mode learned decay rate (retention). The other adds a phase offset to every mode learned frequency, damped so that persistent modes rotate less. The same two signals also drive a per-mode write gate, combined with a retention-dependent write normalization. The motivation has two parts: (i) fixed (time-invariant) spectral recurrences such as LRU or RTU can add content but cannot change how stored information decays or rotates, and (ii) existing selective models assign independent controls to each memory mode, which couples control cost to state size. The authors ask whether a high-dimensional memory requires equally high-dimensional control, and argue that it does not.

The claimed contributions are (1) the shared retention-phase control mechanism; (2) exact structured RTRL derivations that exploit the diagonal state Jacobian and the shared controllers; (3) GPU kernels for SPARC and RG-LRU; and (4) an empirical evaluation. The experiments cover partially observable continuous control and POPGym tasks, both trained with PPO and stored recurrent sensitivities, and sequence classification (UCR, pMNIST, sCIFAR-10) trained with full BPTT. Baselines are RTU, LRU, RG-LRU, GateLoop, Mamba-3 and GRU at matched state size, over 3 seeds. The authors report the best mean results on several tasks (notably Walker-P and FordA), along with lower training latency and faster scans than their optimized RG-LRU kernel. Two synthetic tasks motivate the design.

Reviewer expertise: my expertise is strong in sequence learning architectures, especially recurrent models (linear/gated recurrences, state-space models), and general (familiar) in reinforcement learning methods. My assessment of the RL experiments should be read with this in mind. I invite the authors to point out anything I may have missed or misunderstood, in particular regarding the motivation of the problem, since this is the main factor in my recommendation.

Critical strengths

Clear presentation of a simple, scan-friendly design. The paper is well written (even if some pedagogical background on rtrl would increase the quality of the paper) and the mechanism is well presented such that it is easy to understand the ideas: two shared scalars modulate mode-specific decays and frequencies, and the recurrence stays diagonal and affine in the previous state. This preserves parallel-scan training and keeps forward-mode sensitivity propagation tractable.

Critical weaknesses
  1. The premise and the stated gap do not reflect the current state of the field. The abstract states that linear and spectral recurrences "traditionally rely on static, time-invariant transitions" and "cannot dynamically revise how stored representations decay, rotate, or evolve". This holds only for LTI models (S4, S5, LRU, RTU). Gated linear recurrences, selective state-space models, memory recurrent units (RG-LRU, Mamba, GateLoop, Mamba-3, MRU, among others) already are LTV, as the paper itself acknowledges in the introduction and related work.

    The fallback claim, that existing selective models tie control dimensionality to state size, is also not accurate in general. In Mamba, the input-dependent step size Delta is shared across the state dimensions of a channel while each state dimension keeps its own A, so the decay exp(Delta_t * A_n) is already a shared scalar rescaling mode-specific rates, which is in the idea the SPARC retention control.
    Mamba-2 and mLSTM share a scalar decay across a whole head; RWKV-6 and GLA produce decays through low-rank projections; RG-LRU uses block-diagonal gates.

    Sharing or compressing the control pathway is therefore already standard practice.

    This is the main reason for my recommendation: I am not convinced that the problem the paper addresses is a real bottleneck, or that the proposed answer is novel relative to existing selective models.

  2. The evidence does not support the central claim or the headline improvements. The question the paper poses ("does a high-dimensional memory require equally high-dimensional control?") is never tested directly: there is no comparison of shared, grouped and per-mode control within the same cell with everything else fixed. The baselines differ from SPARC in many respects at once, so the comparison cannot isolate the effect of sharing. The reported gains are also largely within seed variability. Main-text tables report means over 3 seeds with no dispersion for the baselines. Where dispersion is available (appendix), the gaps are often smaller than the spread across seeds. Classification accuracies are measured on small internal validation splits that are also used to select the checkpoint, with no held-out test set.

Questions for authors

Clarification requests (answers could change my assessment):

  1. Problem reality. Can you provide evidence that per-mode control is an actual bottleneck in existing selective recurrences, in compute, memory or accuracy? Given that Mamba already shares Delta across state dimensions within a channel, Mamba-2/mLSTM share a scalar decay per head, and RWKV-6/GLA use low-rank decay projections, what gap does SPARC fill? Please position SPARC explicitly against these mechanisms, and in particular explain how SPARC retention control differs from Mamba beyond sharing at the layer level instead of per channel.
  2. Direct test of the hypothesis. Can you report SPARC with per-mode (unshared) controls and with grouped controls, with everything else fixed, in terms of both task performance and latency? Without this, the claim that high-dimensional memory does not need high-dimensional control is not supported.
  3. Statistical reliability. Please report confidence intervals for all methods in the main tables, ideally with more seeds for the RL tasks. Which of the headline improvements remain significant?

Suggestions (presentation; not decisive for acceptance, but they would raise my presentation score):
A. Figure 1 does not convey its message clearly. I really think that a better first figure could help the reader to understand the point of mode control.
B. Please add a short background paragraph on RTRL in the main text (what it computes, its cost, why it is used for online RL) and move the exactness qualifications out of the appendix, since they describe the regime of all RL experiments.
C. Please rewrite the abstract and introduction so that the stated gap reflects existing selective recurrences.
If the authors can show convincingly that per-mode control is a real limitation of existing selective recurrences, and that sharing yields a measurable and statistically reliable benefit over per-mode or grouped control, I would be willing to raise my soundness and informativeness scores. With a reworked Figure 1 and a proper introduction of RTRL, I would raise my presentation score.

5 comments
AuthorsOpenReview ↗

Problem motivation, mechanistic positioning, and the phase clock

Thank you for your comments on research motivation, control sharing, and statistical rigor. We respond to your 3 core questions and revision suggestions A, B, and C below. Changes specific to your review are highlighted in pale pink, while shared reviewer requests are highlighted in pale yellow.

1. Validity of the problem and positioning of the method (Question 1)

(1) Mechanistic comparison: control dimensionality, modulus, phase, and writing

SPARC separates modal static parameters, including decays νj\nu_j, frequencies θj\theta_j, gains, gate responses sj,vjs_j,v_j, and content projections Bxt\mathbf B\mathbf x_t, from two-dimensional control (ct,dt)(c_t,d_t) shared across the layer:

\lambda_{j,t}=\rho_{j,t}\exp\left(i[\theta_j+(1-\rho_{j,t})d_t]\right).$$ Under $\lambda_{j,t}=\rho_{j,t}e^{i\phi_{j,t}}$, the differences are: | Method | Dynamic scan control | Retention and phase | Write/state form | | :--- | :--- | :--- | :--- | | Mamba | Per-channel $\Delta_{d,t}$ | $e^{-\nu_j\Delta_{d,t}}$; real, or phase $\theta_j\Delta_{d,t}$ | Discretized writing; selective $B_t,C_t$ | | Mamba-2 / mLSTM | Per-head $a_{h,t}$ | Shared head decay; real | Matrix KV state | | RWKV-6 / GLA | Low-rank projection expanded to $O(D)$ controls | Vector decay; real | Matrix KV state | | RG-LRU / CG-LRU | Per-mode $r_{j,t},q_{j,t}$ | $e^{-\nu_jr_{j,t}}$; real, or phase $\theta_jr_{j,t}$ | Normalized gated writing; diagonal state | | GateLoop | Per-mode $\alpha_{j,t},\phi_{j,t}$ | $\sigma(\alpha_{j,t})e^{i\phi_{j,t}}$ | Input-dependent writing; diagonal complex state | | Mamba-3 | Head/channel controls | Selective decay and rotation | Two-endpoint writing; multiple-state caches | | **SPARC** | **2 layer-wide scalars $(c_t,d_t)$** | **Heterogeneous $\nu_j,\theta_j$; damped phase offsets** | **Normalized modal writing; diagonal complex state** | 1. **Decoupling retention and phase: Mamba and CG-LRU.** Mamba's modulus matches SPARC's when $\Delta_{d,t}=e^{c_t}$, but complex extensions tie rotation and decay to the same control. SPARC separates $c_t$ and $d_t$, damps phase offsets by $(1-\rho_{j,t})$, and uses both controls in $\sqrt{1-\rho_{j,t}^2}\,\tilde\sigma_j(c_t,d_t)\mathbf B_j\mathbf x_t$ for writing. 2. **Heterogeneous fixed spectra and two-dimensional scan control: Mamba-2, mLSTM, RWKV-6, GLA.** Head-shared decay does not retain distinct modal decay rates within a head; low-rank projections still expand to vector controls before the scan. SPARC retains each mode's $\nu_j,\theta_j$ and content mapping, transfers two $B\times T$ control streams, and reconstructs modal transitions inside the kernel. 3. **Long-range phase constraints and diagonal RTRL: GateLoop and Mamba-3.** GateLoop lacks the fixed baseline spectrum and retention-coupled phase damping; Mamba-3 carries multiple recurrent caches. SPARC's diagonal state supports recurrent-layer sensitivity propagation with $O(HD+H)$ running traces. **(2) Experiments: balancing long-range modeling and selectivity** | Method | pMNIST TEST (%) | sCIFAR-10 TEST (%) | RepeatFirst final100 | RepeatPrevious final100 | | :--- | :--- | :--- | :--- | :--- | | RTU | *96.47 ± 0.19* | 61.36 ± 1.29 | 0.432 ± 0.070 | 0.843 ± 0.087 | | LRU | 96.44 ± 0.20 | 61.54 ± 1.38 | 0.017 ± 0.301 | *0.884 ± 0.088* | | RG-LRU | 86.54 ± 0.23 | 52.45 ± 1.08 | 0.564 ± 0.073 | -0.079 ± 0.061 | | GateLoop | 64.31 ± 2.40 | 51.85 ± 0.34 | 0.708 ± 0.036 | 0.199 ± 0.065 | | Mamba-3 | 85.62 ± 4.93 | 47.20 ± 2.36 | 0.605 ± 0.034 | 0.347 ± 0.248 | | GRU | 90.58 ± 0.56 | *62.28 ± 0.64* | **0.862 ± 0.052** | 0.881 ± 0.080 | | SPARC | **96.67 ± 0.29** | **63.11 ± 1.26** | *0.817 ± 0.045* | **0.906 ± 0.060** | Fixed-spectrum RTU and LRU preserve long-range pMNIST representations but reach only 0.432 and 0.017 on RepeatFirst, which requires event-triggered resetting and latching. RG-LRU, GateLoop, and the Mamba-3 adaptation improve RepeatFirst, but reach only 64.31%–86.54% on pMNIST and 47.20%–52.45% on sCIFAR-10. GRU is higher on RepeatFirst but reaches 90.58% on pMNIST and cannot use parallel scans. SPARC maintains both capabilities with 0.68% and 1.02% more total parameters than LRU for Ant-P and classification, respectively. **(3) Ablation and theoretical error bound for $(1-\rho_{j,t})d_t$** With retention rates and baseline phases fixed, perturbing phase control by at most $\epsilon$ per step gives, for cumulative decay $L=-\sum\log\rho$, $$|\widetilde P-P|\leq\epsilon L e^{-L}\leq\epsilon/e.$$ The bound is independent of sequence length and vanishes as $L\to0$. It concerns existing memory; writes and full-network gradients are evaluated separately. | Task | SPARC | No phase clock | | :--- | :--- | :--- | | pMNIST | **96.67 ± 0.29** | 96.31 ± 0.20 | | sCIFAR-10 | **63.11 ± 1.26** | 50.77 ± 3.38 | Removing the phase clock reduces sCIFAR-10 TEST accuracy by 12.34 percentage points, supporting the role of damping phase offsets according to persistence.
AuthorsOpenReview ↗

Direct control-sharing tests and same-kernel cost isolation

2. Hypothesis testing: control grouping, latency isolation, and official TEST (Question 2)

(1) Shared (G=1G=1), grouped (G=4,16G=4,16), and Unshared (G=HG=H) within the same cell

We compare G∈{1,4,16,H}G\in\{1,4,16,H\} using the same SPARC cell, state capacity, and training budget. Each group shares ct,dtc_t,d_t, giving 2G2G scalars per layer; H=192H=192 for continuous control and H=64H=64 otherwise. Results are means ± sample SDs over three seeds; classification reports TEST accuracy (%).

TaskG=1G=1G=4G=4G=16G=16G=HG=H
Ant-P4665.31 ± 457.853565.62 ± 530.004212.76 ± 336.234023.15 ± 767.38
Walker-P1153.70 ± 322.29941.08 ± 232.11995.28 ± 94.39943.07 ± 197.16
Hopper-P1378.90 ± 139.941333.01 ± 81.391323.96 ± 203.641082.24 ± 275.25
Cheetah-P2868.21 ± 24.122641.21 ± 44.562489.62 ± 35.892888.18 ± 29.79
Autoencode-0.421 ± 0.012-0.438 ± 0.017-0.400 ± 0.019-0.408 ± 0.010
CountRecall-0.471 ± 0.045-0.505 ± 0.085-0.537 ± 0.025-0.416 ± 0.032
RepeatFirst0.817 ± 0.0450.777 ± 0.2200.610 ± 0.4340.783 ± 0.155
RepeatPrevious0.906 ± 0.0600.750 ± 0.0580.854 ± 0.1240.727 ± 0.247
Noisy Pendulum0.513 ± 0.0120.461 ± 0.0430.422 ± 0.0090.328 ± 0.108
HigherLower0.499 ± 0.0020.505 ± 0.0020.504 ± 0.0100.513 ± 0.005
FordA94.66 ± 0.1694.55 ± 0.3294.36 ± 0.7094.32 ± 0.64
SLC97.14 ± 0.0396.70 ± 0.6496.76 ± 0.2296.87 ± 0.33
UWave94.29 ± 0.8394.41 ± 0.7894.62 ± 0.6094.92 ± 0.80
pMNIST96.67 ± 0.2996.62 ± 0.1196.61 ± 0.1396.75 ± 0.29
sCIFAR-1063.11 ± 1.2661.08 ± 0.8562.79 ± 1.5762.88 ± 1.12

Shared has 13 wins and 2 losses against G=4G=4, 12 wins and 3 losses against G=16G=16, and 9 wins and 6 losses against Unshared. Increasing control dimensionality yields neither monotonic gains nor a consistent advantage. Shared often has higher means and smaller SDs in noisy online RL, consistent with a regularizing effect. Unshared improves classification only on pMNIST and UWave, and shared is higher on FordA, SLC, and sCIFAR-10.

(2) Control-cost isolation using the same GPU-accelerated scan implementation

All settings use the same chunked scan: packed control projection, fused writing, and modal-transition reconstruction inside the scan. State capacity, spectra, writing, and readout are matched. Measurements used one A800.

A classification layer matched to task width (D=H=64,B=32,T=1024D=H=64,B=32,T=1024, FP32):

Control settingControl parameters per layermixer F+B (ms)Peak allocated GPU memory (MiB)
Shared (G=1G=1)1302.657137.478
Grouped (G=4G=4)5202.664140.106
Grouped (G=16G=16)2,0802.742150.618
Unshared (G=HG=H)8,3202.913192.665

Shared reduces latency by 8.79% and peak GPU memory by 28.64% relative to Unshared. The latency reductions relative to G=4G=4 and G=16G=16 are 0.26% and 3.10%. Shared also has the highest SLC TEST accuracy.

A wide layer matched to large-model width (D=2048,H=1024,B=4,T=2048D=2048,H=1024,B=4,T=2048, BF16):

Control settingControl parameters per layermixer F+B (ms)Peak allocated GPU memory (MiB)
Shared (G=1G=1)4,0982.704336.482
Grouped (G=4G=4)16,3922.783337.044
Grouped (G=16G=16)65,5682.879339.294
Unshared (G=HG=H)4,196,35214.644534.317

At D=2048D=2048, Shared gives a 5.41× mixer F+B speedup and 37.03% lower peak memory than Unshared. At 8K tokens per batch, FP32 control streams shrink from 64 to 0.0625 MiB. Relative to G=4G=4 and G=16G=16, Shared uses 75% and 93.75% fewer control parameters and avoids group-count selection.

(3) Official held-out TEST evaluation

All methods use complete official TRAIN sets, full-sequence BPTT, and a fixed endpoint of 20,000 updates, followed by one evaluation on the official held-out TEST sets:

MethodFordASLCUWavepMNISTsCIFAR-10
RTU94.70 ± 0.0097.02 ± 0.1594.60 ± 1.0496.47 ± 0.1961.36 ± 1.29
LRU94.66 ± 0.8096.89 ± 0.2694.12 ± 1.1596.44 ± 0.2061.54 ± 1.38
RG-LRU93.03 ± 0.1196.92 ± 0.2396.35 ± 0.0686.54 ± 0.2352.45 ± 1.08
GateLoop92.39 ± 0.0596.93 ± 0.1790.77 ± 0.5864.31 ± 2.4051.85 ± 0.34
Mamba-390.72 ± 1.6695.90 ± 0.2696.57 ± 0.4785.62 ± 4.9347.20 ± 2.36
GRU93.56 ± 0.2197.20 ± 0.0591.65 ± 0.3590.58 ± 0.5662.28 ± 0.64
SPARC94.66 ± 0.1697.14 ± 0.0394.29 ± 0.8396.67 ± 0.2963.11 ± 1.26

SPARC ranks 1st on pMNIST and sCIFAR-10, tied 2nd on FordA, and 2nd on SLC, preserving the conclusion.

AuthorsOpenReview ↗

Statistical reliability and revisions to Figure 1, RTRL, and presentation

3. Dispersion and statistical reliability (Question 3)

(1) Means and sample standard deviations in the main tables

Each method uses three independent seeds per task for continuous control, POPGym, and classification, reporting mean±SD\text{mean}\pm\text{SD}. With n=3n=3, the Student-tt 95% confidence-interval half-width is t0.975,2⋅SD/3≈2.484 SDt_{0.975,2}\cdot\text{SD}/\sqrt3\approx2.484\,\text{SD}. The RL results are:

Continuous control

MethodAnt-PWalker-PHopper-PCheetah-P
RTU (RTRL)4757.32 ± 65.00928.70 ± 151.411369.88 ± 185.172855.31 ± 24.50
LRU (RTRL)4593.59 ± 759.21814.23 ± 98.811376.24 ± 84.582929.07 ± 6.58
RG-LRU (RTRL)2836.39 ± 502.32725.97 ± 174.55976.57 ± 161.772806.78 ± 13.65
GateLoop (RTRL)3807.76 ± 157.991033.30 ± 58.621336.70 ± 62.332638.09 ± 23.56
SPARC (RTRL)4665.31 ± 457.851153.70 ± 322.291378.90 ± 139.942868.21 ± 24.12
GRU (TBPTT-16)4230.12 ± 432.69957.05 ± 194.441128.68 ± 581.502615.44 ± 80.67
Mamba-3 (TBPTT-16)2143.40 ± 33.33784.51 ± 110.881117.52 ± 470.562867.58 ± 492.63
Mamba-3 (RTRL)1803.42 ± 142.19564.54 ± 54.32846.86 ± 241.492496.70 ± 252.47

POPGym

MethodAutoencodeCountRecallRepeatFirstRepeatPreviousNoisy PendulumHigherLower
RTU (RTRL)-0.443 ± 0.017-0.540 ± 0.0110.432 ± 0.0700.843 ± 0.0870.441 ± 0.1150.496 ± 0.002
LRU (RTRL)-0.424 ± 0.042-0.537 ± 0.3420.017 ± 0.3010.884 ± 0.0880.428 ± 0.0250.497 ± 0.004
RG-LRU (RTRL)-0.473 ± 0.010-0.582 ± 0.0880.564 ± 0.073-0.079 ± 0.0610.467 ± 0.0140.504 ± 0.001
GateLoop (RTRL)-0.425 ± 0.010-0.642 ± 0.0430.708 ± 0.0360.199 ± 0.0650.543 ± 0.0260.498 ± 0.001
SPARC (RTRL)-0.421 ± 0.012-0.471 ± 0.0450.817 ± 0.0450.906 ± 0.0600.513 ± 0.0120.499 ± 0.002
GRU (TBPTT-16)-0.472 ± 0.011-0.401 ± 0.1010.862 ± 0.0520.881 ± 0.0800.327 ± 0.0760.496 ± 0.000
Mamba-3 (TBPTT-16)-0.497 ± 0.002-0.669 ± 0.0880.605 ± 0.0340.347 ± 0.2480.355 ± 0.1140.491 ± 0.003
Mamba-3 (RTRL)-0.498 ± 0.001-0.816 ± 0.0830.199 ± 0.085-0.293 ± 0.0460.197 ± 0.0150.471 ± 0.020

(2) Baseline comparisons and significance tests

Against RG-LRU, SPARC gains 10.13 percentage points on pMNIST (Holm p≈0.000062p\approx0.000062) and 10.66 on sCIFAR-10 (Holm p≈0.011775p\approx0.011775). Two-sided Welch tests use the rounded means and SDs with n=3n=3, followed by Holm correction for 30 comparisons (five tasks × six baselines); pp-values are approximate. SPARC retains top-two means on ten RL tasks.

4. Presentation suggestions A, B, and C

Suggestion A: redesign Figure 1

MethodEvent-order error rate (%)Contraction and rotation tracking NMSE
RTU16.99 ± 0.910.359290 ± 0.003742
RG-LRU0.00 ± 0.000.344748 ± 0.006710
SPARC0.00 ± 0.000.084534 ± 0.001705

We will clarify the comparison of selective retention and resetting, and contraction and rotation adaptation of existing states, supported by five independent seeds.

Suggestion B: RTRL background, costs, exactness, and BPTT correspondence

We will add RTRL background and costs to Section 3.3. Dense traces need O(H∣ψ∣)O(H|\psi|) storage and O(H2∣ψ∣)O(H^2|\psi|) work; SPARC needs O(HD+H)O(HD+H) for both. On Ant-P, with D=64D=64 and 384 real state coordinates:

Methods/updateGPU (MiB)host trace (MiB)running trace (MiB)
SPARC3.8034612.330.39
GRU198.17853224264758.25
Mamba-36.23452426.5413.33

Host trace is the CPU sensitivity cache; running trace is the current trace. Rollout storage is O(THD)O(THD).

RTRL and BPTT correspondence in Appendix B. Let aj,t=mtλj,ta_{j,t}=m_t\lambda_{j,t} include resets, gj,tψg_{j,t}^{\psi} be the local parameter Jacobian, and uj,tu_{j,t} the readout gradient:

\bar h_{j,k}=\overline a_{j,k+1}\bar h_{j,k+1}+u_{j,k}.$$ With zero initial sensitivity and terminal adjoint, exchanging finite sums gives $$\sum_{t,j}\mathrm{Re}(\overline{u}_{j,t}Z_{j,t}^{\psi})=\sum_{k,j}\mathrm{Re}(\overline{\bar{h}}_{j,k}g_{j,k}^{\psi}).$$ Adding direct readout terms gives identical recurrent gradients at fixed parameters and inputs. PPO uses local encoder gradients and cached rollout-time traces, without recomputing them after parameter changes. **Suggestion C: rewrite the abstract and introduction** We revised the introduction to frame **spectral and LTV control costs**, decouple memory width from shared control via Figure 1, and present **clock-damped SPARC**, **RTRL and GPU kernels**, and **benchmark gains**.
Reviewer_MkWfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SPARC, a diagonal complex RNN that modulates decay and phase across modes using just two shared input-dependent scalars. Because the update stays diagonal and affine, it supports both parallel GPU scans for BPTT and online RTRL, tested on POMDP control, POPGym, and sequence classification.

Critical strengths
  1. Modulating heterogeneous per-mode timescales and frequencies with just two shared scalars is simple and clean. Scaling the phase shift by 1-\rho_{j, t} nicely protects long-memory modes from input jitter, while keeping the cell compatible with both parallel scans and online RTRL.
  2. The custom GPU kernel turns shared control into real wall-clock speedups. Storing just two scalar controls per step instead of full per-mode transitions cuts memory traffic, giving 3-5 times faster scans and ~18-34 faster mixer passes than their RG-LRU baseline (outside small-batch regimes).
Critical weaknesses
  1. The central claim is that spectral memory doesn't need per-mode control, yet the authors define a matched "unshared control" SPARC variant in Table 3 and never run it. The classification setup also skips standard long-range benchmarks like LRA that its LRU backbone was designed for.
  2. In RL, SPARC and the other diagonal models get full-horizon credit assignment via stored RTRL traces, while GRU and Mamba-3 are restricted to length-4 truncated BPTT. Giving baselines only a 4-step gradient window is not a fair comparison on memory-heavy POMDPs.
Questions for authors
  1. Why was the "unshared control" variant in Table 3 left out, and how much accuracy does sharing actually cost?
  2. How do GRU and Mamba-3 perform in RL with a longer TBPTT window (say 64–256 steps) instead of length 4?
  3. Can you report official held-out test accuracies in Table 2 (evaluated once from the best validation checkpoint) instead of max validation scores?
3 comments
AuthorsOpenReview ↗

Shared-control ablations and LRA long-range benchmarks

Thank you for your comments. We have added the ablations of control sharing and grouping, LRA benchmarks, comparisons using longer TBPTT windows and Mamba-3 RTRL, and evaluations on the complete official TEST sets for all five classification tasks, as detailed below.

In the revised paper, changes specific to your review are highlighted in pale green, while changes addressing shared reviewer requests are highlighted in pale yellow.

1. On not testing the unshared-control variant and the lack of LRA long-range benchmarks

(1) Comparisons of shared control (G=1G=1) with grouped or independent control (G=4,16,HG=4,16,H) within the same cell

We compare G∈{1,4,16,H}G\in\{1,4,16,H\} using the same SPARC cell, state capacity, and training budget: each group shares retention and phase controls ct,dtc_t,d_t, yielding 2G2G scalars per layer. G=1G=1 is SPARC's default global sharing; G=HG=H gives each complex mode independent control. H=192H=192 for continuous control and H=64H=64 for the remaining tasks. Results are means ± sample SDs over three seeds; classification uses TEST accuracy (%).

TaskG=1G=1G=4G=4G=16G=16G=HG=H
Ant-P4665.31 ± 457.853565.62 ± 530.004212.76 ± 336.234023.15 ± 767.38
Walker-P1153.70 ± 322.29941.08 ± 232.11995.28 ± 94.39943.07 ± 197.16
Hopper-P1378.90 ± 139.941333.01 ± 81.391323.96 ± 203.641082.24 ± 275.25
Cheetah-P2868.21 ± 24.122641.21 ± 44.562489.62 ± 35.892888.18 ± 29.79
Autoencode-0.421 ± 0.012-0.438 ± 0.017-0.400 ± 0.019-0.408 ± 0.010
CountRecall-0.471 ± 0.045-0.505 ± 0.085-0.537 ± 0.025-0.416 ± 0.032
RepeatFirst0.817 ± 0.0450.777 ± 0.2200.610 ± 0.4340.783 ± 0.155
RepeatPrevious0.906 ± 0.0600.750 ± 0.0580.854 ± 0.1240.727 ± 0.247
Noisy Pendulum0.513 ± 0.0120.461 ± 0.0430.422 ± 0.0090.328 ± 0.108
HigherLower0.499 ± 0.0020.505 ± 0.0020.504 ± 0.0100.513 ± 0.005
FordA94.66 ± 0.1694.55 ± 0.3294.36 ± 0.7094.32 ± 0.64
SLC97.14 ± 0.0396.70 ± 0.6496.76 ± 0.2296.87 ± 0.33
UWave94.29 ± 0.8394.41 ± 0.7894.62 ± 0.6094.92 ± 0.80
pMNIST96.67 ± 0.2996.62 ± 0.1196.61 ± 0.1396.75 ± 0.29
sCIFAR-1063.11 ± 1.2661.08 ± 0.8562.79 ± 1.5762.88 ± 1.12

Reinforcement learning and noisy-memory tasks. G=1G=1 achieves higher means than G=HG=H on 9 of the 15 tasks, with advantages concentrated in online PPO tasks, such as Ant-P, Walker-P, Hopper-P, RepeatPrevious, and Noisy Pendulum at 0.513 versus 0.328, alongside smaller standard deviations across seeds on Noisy Pendulum, RepeatFirst, and Ant-P. Across all 15 tasks, Shared has 13 wins and 2 losses against G=4G=4, and 12 wins and 3 losses against G=16G=16. Increasing control dimensionality therefore yields neither monotonic gains nor a consistent advantage.

Supervised BPTT sequence tasks. Independent control has a slight advantage on pMNIST and UWave, exceeding shared control by 0.08 and 0.63 percentage points, respectively; shared control reaches 63.11% on sCIFAR-10, exceeding independent control's 62.88% by 0.23 percentage points, and maintains higher means on FordA and SLC. Sharing reduces control-projection parameters from 2H(D+1)2H(D+1) to 2(D+1)2(D+1) and avoids materializing O(BTH)O(BTH) transition tensors during the scan. Each mode retains its own baseline decay, frequency, write response, and content projection.

(2) Accuracy on the complete TEST sets of four LRA benchmarks (%)

We have added official TEST results on four LRA tasks under the same residual backbone and full-sequence BPTT protocol. Checkpoints are selected using the validation set, with 20,000 training updates for Image and ListOps and 5,000 for Text and Retrieval. These results use a single training run.

MethodImageListOpsTextRetrieval
RTU62.63037.20082.76466.904
LRU61.69038.60082.22466.158
RG-LRU56.85039.05076.98865.631
GateLoop56.81039.00078.43663.474
Mamba-331.73037.95062.86056.965
GRU66.25045.30087.18865.590
SPARC68.34039.35083.21666.921

On LRA, SPARC ranks 1st on Image and Retrieval, exceeding the runner-up GRU by 2.09 percentage points on Image; it ranks 2nd on ListOps and Text. Among parallelizable spectral recurrent and selective state-space baselines (RTU, LRU, RG-LRU, GateLoop, and Mamba-3), SPARC ranks 1st on all four LRA tasks, demonstrating that two shared control signals can also operate reliably in long-range sequence modeling.

AuthorsOpenReview ↗

Longer-window RL baselines and official held-out TEST results

2. On extending the TBPTT windows of GRU and Mamba-3 in RL

We agree that TBPTT-4 is too short for a thorough comparison. Under PPO with multiple epochs, extending dense or cached-state baselines to 64–256 steps incurs substantial GPU-memory and time overhead. We therefore extended GRU and Mamba-3 to TBPTT-16 under a matched protocol and additionally implemented Mamba-3 RTRL, matching the state budget and readout head. Results report means and sample SDs over three training seeds.

Continuous control

MethodAnt-PWalker-PHopper-PCheetah-P
RTU (RTRL)4757.32 ± 65.00928.70 ± 151.411369.88 ± 185.172855.31 ± 24.50
LRU (RTRL)4593.59 ± 759.21814.23 ± 98.811376.24 ± 84.582929.07 ± 6.58
RG-LRU (RTRL)2836.39 ± 502.32725.97 ± 174.55976.57 ± 161.772806.78 ± 13.65
GateLoop (RTRL)3807.76 ± 157.991033.30 ± 58.621336.70 ± 62.332638.09 ± 23.56
SPARC (RTRL)4665.31 ± 457.851153.70 ± 322.291378.90 ± 139.942868.21 ± 24.12
GRU (TBPTT-16)4230.12 ± 432.69957.05 ± 194.441128.68 ± 581.502615.44 ± 80.67
Mamba-3 (TBPTT-16)2143.40 ± 33.33784.51 ± 110.881117.52 ± 470.562867.58 ± 492.63
Mamba-3 (RTRL)1803.42 ± 142.19564.54 ± 54.32846.86 ± 241.492496.70 ± 252.47

POPGym

MethodAutoencodeCountRecallRepeatFirstRepeatPreviousNoisy PendulumHigherLower
RTU (RTRL)-0.443 ± 0.017-0.540 ± 0.0110.432 ± 0.0700.843 ± 0.0870.441 ± 0.1150.496 ± 0.002
LRU (RTRL)-0.424 ± 0.042-0.537 ± 0.3420.017 ± 0.3010.884 ± 0.0880.428 ± 0.0250.497 ± 0.004
RG-LRU (RTRL)-0.473 ± 0.010-0.582 ± 0.0880.564 ± 0.073-0.079 ± 0.0610.467 ± 0.0140.504 ± 0.001
GateLoop (RTRL)-0.425 ± 0.010-0.642 ± 0.0430.708 ± 0.0360.199 ± 0.0650.543 ± 0.0260.498 ± 0.001
SPARC (RTRL)-0.421 ± 0.012-0.471 ± 0.0450.817 ± 0.0450.906 ± 0.0600.513 ± 0.0120.499 ± 0.002
GRU (TBPTT-16)-0.472 ± 0.011-0.401 ± 0.1010.862 ± 0.0520.881 ± 0.0800.327 ± 0.0760.496 ± 0.000
Mamba-3 (TBPTT-16)-0.497 ± 0.002-0.669 ± 0.0880.605 ± 0.0340.347 ± 0.2480.355 ± 0.1140.491 ± 0.003
Mamba-3 (RTRL)-0.498 ± 0.001-0.816 ± 0.0830.199 ± 0.085-0.293 ± 0.0460.197 ± 0.0150.471 ± 0.020

Extending the window to TBPTT-16 substantially improves GRU's performance, yet SPARC retains higher means on Ant-P, Walker-P, Hopper-P, Cheetah-P, Autoencode, RepeatPrevious, and Noisy Pendulum (0.513 versus 0.327). SPARC ranks 1st on both Walker-P and Hopper-P and 2nd on Ant-P and Cheetah-P. On POPGym, it ranks in the top two on all 6 tasks, taking 1st place on 2 tasks and 2nd place on 4.

Even with RTRL added, Mamba-3 still performs below SPARC on all 10 RL tasks, reaching 1803.42 versus 4665.31 on Ant-P. This indicates that the advantage does not arise solely from using RTRL, but also from the diagonal spectral structure and shared control supporting forward sensitivity propagation. The TBPTT-16 results strengthen the comparison, while 64–256-step windows remain outside the completed evaluation.

3. On reporting official held-out TEST accuracy in Table 2

We trained on the complete official TRAIN sets using full-sequence BPTT to a fixed endpoint of 20,000 updates, followed by a single final evaluation on the official held-out TEST sets:

MethodFordASLCUWavepMNISTsCIFAR-10
RTU94.70 ± 0.0097.02 ± 0.1594.60 ± 1.0496.47 ± 0.1961.36 ± 1.29
LRU94.66 ± 0.8096.89 ± 0.2694.12 ± 1.1596.44 ± 0.2061.54 ± 1.38
RG-LRU93.03 ± 0.1196.92 ± 0.2396.35 ± 0.0686.54 ± 0.2352.45 ± 1.08
GateLoop92.39 ± 0.0596.93 ± 0.1790.77 ± 0.5864.31 ± 2.4051.85 ± 0.34
Mamba-390.72 ± 1.6695.90 ± 0.2696.57 ± 0.4785.62 ± 4.9347.20 ± 2.36
GRU93.56 ± 0.2197.20 ± 0.0591.65 ± 0.3590.58 ± 0.5662.28 ± 0.64
SPARC94.66 ± 0.1697.14 ± 0.0394.29 ± 0.8396.67 ± 0.2963.11 ± 1.26

On the official TEST sets, SPARC ranks 1st on pMNIST and sCIFAR-10; sCIFAR-10 reaches 63.11%, exceeding GRU's 62.28% by 0.83 percentage points. Compared with the pronounced performance drops of GRU on pMNIST and UWave, and of Mamba-3, RG-LRU, and GateLoop on pMNIST and sCIFAR-10, SPARC has the strongest overall ranking across the five tasks.

In the final version, we have updated Table 2 with the above official TEST results and incorporated the control-grouping ablations, LRA benchmarks, and comparisons with TBPTT-16 and Mamba-3 RTRL into the main text and appendices.

AuthorsOpenReview ↗

Following up on our response

Dear Reviewer, as the discussion deadline is approaching, we wanted to gently check in to see if our responses, additional experiments, and revisions have addressed your concerns, or if there is anything else you would like us to clarify or add.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Manuscript Revisions

Thank you to all reviewers for your constructive feedback and time in evaluating our paper. Following your suggestions, we have added new experiments, statistical analyses, and manuscript revisions to the updated paper and appendices:

  • Reviewer MkWf: On control sharing and LRA, we compared G∈{1,4,16,H}G \in \{1, 4, 16, H\} across all 15 tasks, where G=1G=1 outperforms G=HG=H on 9 of 15 tasks, and added 4 LRA test benchmarks, where SPARC ranks first among all parallelizable baselines. For stronger RL baselines, after extending GRU and Mamba-3 to TBPTT-16 and adding Mamba-3 RTRL, SPARC still leads Walker-P and Hopper-P, places in the top two on all 6 POPGym tasks, and outperforms Mamba-3 RTRL on all 10 RL tasks. Evaluating all 5 classification tasks on the official test sets, SPARC ranks first on pMNIST at 96.67% and sCIFAR-10 at 63.11%.

  • Reviewer sBwF: On positioning and the phase clock, we added a unified LTV comparison and verified our uniform clock bound over sequence length, while removing the clock lowers sCIFAR-10 accuracy by 12.34 percentage points. Matched G∈{1,4,16,H}G \in \{1, 4, 16, H\} tests and unified A800 kernel benchmarks show that G=1G=1 yields a 5.41×5.41\times speedup and 37.03% lower memory than G=HG=H at D=2048D=2048, alongside official test evaluations. Main tables now report the mean and SD across 3 seeds, with Welch tests under Holm correction confirming gains of 10.13 percentage points on pMNIST and 10.66 percentage points on sCIFAR-10 over RG-LRU. For presentation, we redesigned Figure 1, moved RTRL costs and Ant-P profiling to Section 3.3, added RTRL and BPTT duality in Appendix B, and rewrote the Abstract and Introduction.

  • Reviewer vzCQ: Across matched G∈{1,4,16,H}G \in \{1, 4, 16, H\} ablations, increasing GG gives no monotonic gain, and G=1G=1 uniquely leads 9 of 15 tasks with lower PPO variance while reducing control parameters by a factor of HH. On 5B FineWeb-Edu tokens with 8 layers at D=2048D=2048, SPARC lowers test perplexity by 2.00% over RG-LRU, improves 7 of 8 zero-shot tasks, and updates 1.19×1.19\times faster on two H800 GPUs. For evaluation protocols, we upgraded RL baselines to TBPTT-16 and Mamba-3 RTRL, moved classification to official test splits, and added 4 LRA benchmarks.

Overall, these additions confirm across narrow and D=2048D=2048 scales that modulating a heterogeneous spectrum through two shared scalars and a phase clock coupled to retention reconciles selective filtering with recall over long horizons while keeping scan traffic and O(HD+H)O(HD+H) RTRL traces compact. All questions are addressed point by point in our individual responses below. Thank you again for helping strengthen this work.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.