- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SPARC, a diagonal complex recurrence that uses two shared input-dependent scalars (retention and phase ) to modulate complex modes, coupled with a phase clock factor. It supports both parallel prefix scan for BPTT and exact structured RTRL for online RL. The authors also implement fused Triton kernels on Blackwell GPUs showing 3.1-4.7x scan speedup over RG-LRU.
Critical strengths
-
Neat idea to decouple control dimension (2 scalars) from state dimension ( modes).
-
The retention-coupled phase clock makes physical sense (protects near-unit-circle persistent modes from phase noise) and shows strong empirical value in ablations (Table 11, CIFAR-10 drops ~11% without it).
-
Solid Triton kernel engineering (recomputing modal transitions inside the scan to save HBM bandwidth by only storing two control arrays) and thorough numerical verification (Table 21).
Critical weaknesses
-
Unshared control baseline is missing: The whole paper argues that 2 shared controls are enough compared to per-mode controls. Yet Table 3 and Line 1509 admit the unshared variant was never run! With 9 ablations in Appendix E, skipping the most important baseline (Unshared control) makes it impossible to tell how much expressivity is lost by sharing.
-
Scale mismatch between speed tests and accuracy tests: Section 5 benchmarks speed at LLM widths (), but Section 4 only tests accuracy/RL at . Having only 2 scalar degrees of freedom might work for , but won't it severely bottleneck capacity at ? You need either wider model evaluations or a multi-head/grouped control study.
-
Evaluation issues in RL and Classification: 1) In RL, SPARC uses full RTRL traces while Mamba-3 and GRU only use length-4 TBPTT (Table 4). Truncating BPTT to 4 steps on POMDPs obviously hurts Mamba-3/GRU. 2) In classification (Appendix D.5), you split the training set into fit/val, pick the best checkpoint on val, and report that same peak val accuracy instead of evaluating on the official test sets. Also, SLC and UWave val sets only have 100 and 90 samples.
Questions for authors
See "weaknesses" above.
4 comments
BPTT-16 and RTRL baselines, official TEST evaluation, and LRA
3. TBPTT-16 and Mamba-3 RTRL, official TEST, and LRA
(1) Extending GRU and Mamba-3 to TBPTT-16 and adding Mamba-3 RTRL
We agree that TBPTT-4 is too short. Under multi-epoch PPO, 64–256-step unrolls incur substantial memory and time overhead. We extended GRU and Mamba-3 to TBPTT-16 and added Mamba-3 RTRL, matching the state budget and readout. POPGym uses 1,953 PPO updates. Results are means ± sample SDs over three seeds.
Continuous control
| Method | Ant-P | Walker-P | Hopper-P | Cheetah-P |
|---|---|---|---|---|
| RTU (RTRL) | 4757.32 ± 65.00 | 928.70 ± 151.41 | 1369.88 ± 185.17 | 2855.31 ± 24.50 |
| LRU (RTRL) | 4593.59 ± 759.21 | 814.23 ± 98.81 | 1376.24 ± 84.58 | 2929.07 ± 6.58 |
| RG-LRU (RTRL) | 2836.39 ± 502.32 | 725.97 ± 174.55 | 976.57 ± 161.77 | 2806.78 ± 13.65 |
| GateLoop (RTRL) | 3807.76 ± 157.99 | 1033.30 ± 58.62 | 1336.70 ± 62.33 | 2638.09 ± 23.56 |
| SPARC (RTRL) | 4665.31 ± 457.85 | 1153.70 ± 322.29 | 1378.90 ± 139.94 | 2868.21 ± 24.12 |
| GRU (TBPTT-16) | 4230.12 ± 432.69 | 957.05 ± 194.44 | 1128.68 ± 581.50 | 2615.44 ± 80.67 |
| Mamba-3 (TBPTT-16) | 2143.40 ± 33.33 | 784.51 ± 110.88 | 1117.52 ± 470.56 | 2867.58 ± 492.63 |
| Mamba-3 (RTRL) | 1803.42 ± 142.19 | 564.54 ± 54.32 | 846.86 ± 241.49 | 2496.70 ± 252.47 |
POPGym
| Method | Autoencode | CountRecall | RepeatFirst | RepeatPrevious | Noisy Pendulum | HigherLower |
|---|---|---|---|---|---|---|
| RTU (RTRL) | -0.443 ± 0.017 | -0.540 ± 0.011 | 0.432 ± 0.070 | 0.843 ± 0.087 | 0.441 ± 0.115 | 0.496 ± 0.002 |
| LRU (RTRL) | -0.424 ± 0.042 | -0.537 ± 0.342 | 0.017 ± 0.301 | 0.884 ± 0.088 | 0.428 ± 0.025 | 0.497 ± 0.004 |
| RG-LRU (RTRL) | -0.473 ± 0.010 | -0.582 ± 0.088 | 0.564 ± 0.073 | -0.079 ± 0.061 | 0.467 ± 0.014 | 0.504 ± 0.001 |
| GateLoop (RTRL) | -0.425 ± 0.010 | -0.642 ± 0.043 | 0.708 ± 0.036 | 0.199 ± 0.065 | 0.543 ± 0.026 | 0.498 ± 0.001 |
| SPARC (RTRL) | -0.421 ± 0.012 | -0.471 ± 0.045 | 0.817 ± 0.045 | 0.906 ± 0.060 | 0.513 ± 0.012 | 0.499 ± 0.002 |
| GRU (TBPTT-16) | -0.472 ± 0.011 | -0.401 ± 0.101 | 0.862 ± 0.052 | 0.881 ± 0.080 | 0.327 ± 0.076 | 0.496 ± 0.000 |
| Mamba-3 (TBPTT-16) | -0.497 ± 0.002 | -0.669 ± 0.088 | 0.605 ± 0.034 | 0.347 ± 0.248 | 0.355 ± 0.114 | 0.491 ± 0.003 |
| Mamba-3 (RTRL) | -0.498 ± 0.001 | -0.816 ± 0.083 | 0.199 ± 0.085 | -0.293 ± 0.046 | 0.197 ± 0.015 | 0.471 ± 0.020 |
TBPTT-16 substantially strengthens GRU, but SPARC is higher on all four control tasks, ranking 1st on Walker-P and Hopper-P and 2nd on Ant-P and Cheetah-P. On POPGym, SPARC has two 1st places and four 2nd places. SPARC exceeds Mamba-3 RTRL on all ten tasks, indicating that its gains do not arise solely from RTRL.
(2) Complete official TEST sets for the five classification tasks
All methods use complete official TRAIN sets and full-sequence BPTT for 20,000 updates, then one final evaluation on official held-out TEST sets (means ± sample SDs, three seeds):
| Method | FordA | SLC | UWave | pMNIST | sCIFAR-10 |
|---|---|---|---|---|---|
| RTU | 94.70 ± 0.00 | 97.02 ± 0.15 | 94.60 ± 1.04 | 96.47 ± 0.19 | 61.36 ± 1.29 |
| LRU | 94.66 ± 0.80 | 96.89 ± 0.26 | 94.12 ± 1.15 | 96.44 ± 0.20 | 61.54 ± 1.38 |
| RG-LRU | 93.03 ± 0.11 | 96.92 ± 0.23 | 96.35 ± 0.06 | 86.54 ± 0.23 | 52.45 ± 1.08 |
| GateLoop | 92.39 ± 0.05 | 96.93 ± 0.17 | 90.77 ± 0.58 | 64.31 ± 2.40 | 51.85 ± 0.34 |
| Mamba-3 | 90.72 ± 1.66 | 95.90 ± 0.26 | 96.57 ± 0.47 | 85.62 ± 4.93 | 47.20 ± 2.36 |
| GRU | 93.56 ± 0.21 | 97.20 ± 0.05 | 91.65 ± 0.35 | 90.58 ± 0.56 | 62.28 ± 0.64 |
| SPARC | 94.66 ± 0.16 | 97.14 ± 0.03 | 94.29 ± 0.83 | 96.67 ± 0.29 | 63.11 ± 1.26 |
On the official TEST sets, SPARC ranks 1st on both pMNIST and sCIFAR-10; sCIFAR-10 reaches 63.11%, exceeding the sequential GRU's 62.28% by 0.83 percentage points; it ranks 2nd on both FordA and SLC, only 0.04 and 0.06 percentage points behind the respective leaders.
(3) Complete TEST sets of four LRA benchmarks (%)
LRA uses the same residual backbone and full-sequence BPTT, validation-selected checkpoints, and one run: 20,000 updates for Image and ListOps and 5,000 for Text and Retrieval.
| Method | Image | ListOps | Text | Retrieval |
|---|---|---|---|---|
| RTU | 62.630 | 37.200 | 82.764 | 66.904 |
| LRU | 61.690 | 38.600 | 82.224 | 66.158 |
| RG-LRU | 56.850 | 39.050 | 76.988 | 65.631 |
| GateLoop | 56.810 | 39.000 | 78.436 | 63.474 |
| Mamba-3 | 31.730 | 37.950 | 62.860 | 56.965 |
| GRU | 66.250 | 45.300 | 87.188 | 65.590 |
| SPARC | 68.340 | 39.350 | 83.216 | 66.921 |
SPARC ranks 1st on Image and Retrieval and 2nd on ListOps and Text; it ranks 1st among all five parallelizable spectral baselines on every task.
We will include these results and protocols in the main text and appendices.
Thank you for the thorough rebuttal and the substantial experimental additions.
Resolved Concerns:
- Control-Sharing Ablation: The comparison across directly answers my core question. It is insightful to see that while Shared () and Unshared () perform similarly on supervised BPTT classification, low-dimensional shared control provides a clear regularization benefit (higher returns and lower variance) in noisy online RL settings.
- Fairer Baselines & Official Test Sets: Reporting official held-out TEST accuracies resolves the validation selection bias. Extending GRU/Mamba-3 to TBPTT-16 and adding Mamba-3 RTRL makes the RL evaluation much fairer—confirming that GRU remains competitive under TBPTT-16 while Mamba-3 struggles with online RTRL traces.
- Scaling to : The 5B-token language modeling run on FineWeb-Edu shows that SPARC scales cleanly to , outperforming RG-LRU in both perplexity/zero-shot accuracy and training step time (1.92s vs. 2.29s).
Remaining Limitations:
- The vs. Unshared/Grouped ablation was only run at narrow widths (), while the LM experiment only compares against RG-LRU. Testing at would better clarify whether 2 scalars eventually bottleneck capacity at large scales.
- Please make sure all new tables are integrated into the final manuscript and Figure 2(b) is updated to match Eq. (31).
Overall, the new experiments solidify the paper's empirical claims. I am raising my score from 2 to 3.
Thank you for your constructive feedback
Thank you for reviewing our new results.
- Grouped control at : We will run the matched at to directly test whether two layer-wide scalars bottleneck capacity at larger scales.
- Final manuscript and Figure 2(b): We confirm that all new experimental tables will be fully integrated into the final manuscript and appendices, and Figure 2(b) will be updated.
Thank you again for your time and constructive feedback, which have greatly helped us improve the paper.
Direct control-sharing ablations and language modeling at D=2048
Thank you for your specific suggestions on direct ablations of control sharing, model-scale matching, and evaluation protocols. We have added grouped and Unshared control within the same cell, complete language-model training at D=2048, longer-window RL comparisons, and official TEST evaluations. Changes specific to your review are pale blue; shared reviewer requests are pale yellow.
1. Shared (G=1), grouped (G=4,16), and Unshared (G=H) within the same cell
We compare G∈{1,4,16,H} using the same SPARC cell, state capacity, and training budget. Each group shares retention and phase controls ct,dt, giving 2G dynamic scalars per layer; H=192 for continuous control and H=64 for the remaining tasks. Results are means ± sample SDs over three seeds; classification reports TEST accuracy (%).
Increasing control dimensionality does not yield monotonic gains. Shared has 13 wins and 2 losses against G=4, 12 wins and 3 losses against G=16, and 9 wins and 6 losses against Unshared. Shared leads 9 tasks, Unshared 5, and G=16 only Autoencode.
On reinforcement learning and noisy-memory tasks, Shared has higher means and smaller SDs than Unshared on Ant-P, Hopper-P, Noisy Pendulum, and RepeatFirst. Each mode retains distinct timescales and responses through νj,θj,sj,vj, consistent with a regularizing effect of low-dimensional modulation under noisy policy gradients.
On supervised BPTT classification, Unshared is higher only on pMNIST and UWave, by 0.08 and 0.63 percentage points. Shared exceeds Unshared on FordA, SLC, and sCIFAR-10, including a 0.23-point gain on sCIFAR-10. Sharing reduces control-projection parameters from 2H(D+1) to 2(D+1) and avoids materializing O(BTH) transition tensors in the scan.
2. Complete language-model performance and downstream evaluation at D=2048
To address the mismatch between wide-layer speed benchmarks and narrow-model task results, we trained 8-layer, width-2048 language models for 19,073 updates each, totaling 4,999,872,512 tokens, with a 10M-token held-out TEST corpus. Both use the same decoder backbone and FineWeb-Edu training data; SPARC has 1024 complex modes and RG-LRU has 2048 real state coordinates per layer.
(1) Validation and held-out TEST perplexity
SPARC's TEST PPL is 22.14, 2.00% lower than RG-LRU, using two controls per layer.
(2) Eight downstream zero-shot tasks
SPARC achieves higher accuracy on 6 of them, with gains of approximately 5.96 percentage points on BoolQ, 2.53 percentage points on RTE, and 2.07 percentage points on PIQA.
(3) Update time on two H800 GPUs
The measured median is 1.92 s/update for SPARC versus 2.29 for RG-LRU over 5,980 post-warmup updates per model, including data preparation, transfer, forward and backward computation.