- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates memory compression for the recurrent matrix states in Gated Delta Networks (GDNs) within hybrid Large Language Models. Motivated by the insight of capacity-geometry decoupling, the paper introduces FactorState, a training-free dynamic factor representation that stores state as low-rank factors. Until the size , the write buffer size, the computation is an exact match to the original. After , it uses compresses it back to the rank to avoid large grow in memory. Therefore, it improves end-to-end throughput and peak GPU memory.
Critical strengths
1. Clear Theoretical & Empirical Motivation
Prior work on linear attention / GDN state compression largely focused on static rank profiles or structural channel pruning. By separating capacity loss from geometry loss and measuring their direct impact on pre-write key reads and post-write query reads , the paper provides a compelling explanation for why static low-rank projections fail during autoregressive decoding.
2. Hardware-Aware Systems Engineering
The authors design a three-stage CUDA maintenance kernel using regularized Gram matrices, Cholesky decomposition, and warp-level Jacobi SVD. This avoids global synchronization spikes and eliminates external solver workspace overhead.
Critical weaknesses
W1. Inflated Memory and Throughput Claims via an FP32 Baseline.
While model weights and attention KV caches are stored in BF16, the evaluations use FP32 for GDN recurrent states (Sec. 4.1). Because FactorState relies on regularized Gram-Cholesky recompression, FP32 precision appears favorable for maintaining its numerical stability, whereas standard Dense GDN states are commonly stored in BF16 in practice. The paper does not evaluate performance or memory savings under a BF16 state setting, where the relative memory reduction of FactorState of FP32 over Dense BF16 would be considerably smaller.
Questions for authors
Q1. Performance and Memory Trade-offs Under BF16 Recurrent States
How does a Dense GDN baseline with BF16 recurrent states compare against Dense FP32 and FactorState in terms of downstream accuracy, peak memory, and decode throughput? Relatedly, can FactorState store its committed factors in BF16 while upcasting to FP32 in shared memory/registers during recompression step, or does low-precision factor storage degrade numerical stability?
6 comments
We thank the authors for the additional BF16 experiments and the clarification regarding the Qwen3.5 configuration. Cross-referencing the new results with the main paper and appendices raises a few specific follow-up questions:
1. Benchmark Protocol Discrepancy and 5.0 pp Accuracy Drops
-
Evaluation Protocol: The rebuttal evaluates the 400-problem subset (100 per task, 2,048-token cap for math) from Appendix E.3 (Table 14) rather than the full 5,683-instance suite in Sec. 4.3 (Figure 4 / Table 8, 8,192-token cap for MATH-500). Why was the reduced subset chosen over the main evaluation setup, and can the authors provide BF16 results under the full Sec. 4.3 protocol?
-
Cause of 5.0 pp Degradation: Comparing the rebuttal's BF16 results against the matching FP32 baseline on the exact same 400-case subset (Table 14) reveals a 5.0 pp drop on IFEval () and a 5.0 pp cumulative drop on Mathematics (). What is the primary cause of this degradation when storing factors in BF16?
3. Changed Systems Protocol and Loss of Throughput Advantage
- Protocol Change: Instead of using the established RTX 3090 setups in Sec. 4.4, where either fixed-batch pure decode throughput (C=192, T=512, Table 2) or continuous batching (C=96, Table 16) is used, the rebuttal introduces a new protocol (C=192 with paged attention on real task inputs, including prefill and cleanup in the throughput calculation). Why was the protocol changed, and can the authors report matched BF16 numbers under the exact Table 2 and Table 16 settings?
BF16 Follow-up: Accuracy and System Performance
Thank you for pointing out the differences between the evaluation settings in the appendix and the main paper. Given the time and compute available during the rebuttal period, we used the fixed 400-example subset from Table 14 in Appendix E.3 and retained its 2,048-token output cap for mathematics. This let us compare results on the same examples as in the appendix. We have not completed the BF16 evaluation on the full 5,683-example suite in Section 4.3.
On the 5-point accuracy drop. Our original Dense baseline followed the official Qwen3.5-4B configuration, which stores the GDN recurrent state in FP32 (mamba_ssm_dtype: float32). The earlier FP32 and BF16 evaluations used the same examples but different versions of the implementation. To isolate the effect of state storage precision, we reran all four configurations on the same 100 IFEval prompts and 100 mathematics problems from the original subset. For each method, we kept the implementation and generation settings fixed and changed only the storage precision of the persistent recurrent state.
| Task (100 examples each) | Dense FP32 | Dense BF16 | FactorState FP32 | FactorState BF16 |
|---|---|---|---|---|
| IFEval (strict prompt-level accuracy) | 77% | 76% | 81% | 80% |
| Mixed mathematics subset | 88% | 85% | 89% | 89% |
In this controlled comparison, switching FactorState to BF16 reduced IFEval accuracy by 1 percentage point (95% paired CI: [−5, +3] pp) and left mathematics accuracy unchanged (95% paired CI: [−4, +4] pp). Neither point estimate reproduces the earlier 5-point drop. The earlier comparison across implementation versions therefore does not isolate the effect of BF16 factor storage.
On the systems evaluation protocol. We used real task inputs in the initial rebuttal to measure end-to-end throughput for variable-length requests, including prefill and cleanup. To address your question about the original settings, we also ran four paired BF16 trials on an RTX 3090 for each workload from Tables 2 and 16.
| Workload | Dense BF16 end-to-end time (median) | FactorState BF16 end-to-end time (median) |
|---|---|---|
| Table 2: C = 192, 512 decode steps | 30.074 s | 30.056 s |
| Table 16: C = 96, continuous batching | 428.70 s | 417.01 s |
End-to-end time was nearly identical in the Table 2 workload. In the Table 16 workload, both methods processed the same 1,024 requests with identical per-request output lengths, generating 983,040 output tokens per method in each trial. FactorState achieved a median paired end-to-end throughput gain of 2.92%. With BF16 storage for both methods, FactorState reduced recurrent-state storage per request from 24 MiB to 9.375 MiB, a 60.94% reduction.
Thank you for suggesting these comparisons under the original workloads. We hope the additional results clarify the trade-offs among accuracy, throughput, and recurrent-state storage for FactorState with BF16 storage.
Rebuttal Response: FP32 Baseline and BF16 Memory–Accuracy Trade-offs
We thank the reviewer for recognizing the motivation behind capacity–geometry decoupling and the CUDA implementation. We have added matched-precision BF16 comparisons to directly address the fairness of the FP32 baseline and the practical benefits under lower-precision state storage.
Response to Major Concerns
Why Dense FP32 is a valid baseline. Dense FP32 was not selected to artificially inflate the baseline's memory usage. The official Qwen3.5-4B configuration uses BF16 model weights but specifies
mamba_ssm_dtype="float32"for the recurrent state. The precision of model weights and attention KV caches does not necessarily determine the precision of the repeatedly updated GDN state. Nevertheless, we agree that BF16 deployment is important and have therefore added a direct Dense BF16 versus FactorState BF16 comparison rather than extrapolating from FP32 results.BF16 factor storage and downstream fidelity. FactorState can persistently store both left/right factors and all buffered columns in BF16, while performing recurrent updates and Gram–Cholesky/SVD recompression in FP32. No additional persistent FP32 factor copies are required. On Qwen3.5-4B, Dense BF16 uses 24 MiB of GDN recurrent state per request, while FactorState BF16 uses 9.375 MiB, including all 25 physical factor columns. This is a 60.94% reduction in recurrent-state bytes at the same storage precision. Across 12 validation trajectories, relative to a common Dense FP32 reference, FactorState BF16 obtains a final-state error of 0.13153 and query/key-read NRMSE of 0.02459/0.02282. We also evaluated four 100-problem downstream subsets using the same BF16 inference backend:
All four paired 95% confidence intervals for the score differences include zero. These results show comparable observed task performance under substantial state-storage reduction; we do not claim statistical equivalence or zero accuracy cost.
Thus, in this matched-workload setting, FactorState reduces full-model peak GPU memory by approximately 10%, with a modest throughput reduction (approximately 1.6%). We will clearly distinguish recurrent-state byte savings from measured full-model GPU memory savings, and specify the measurement protocol in the revision.
We appreciate the reviewer's suggestion. We hope the official state-precision clarification and these matched-BF16 quality and system measurements address the central concern about baseline fairness. If so, we would be grateful if the reviewer could reconsider the overall assessment in light of the additional evidence.