acceptodds
Under review as a conference paper at ICLR 2027

FactorState: Dynamic Low-Rank Tracking for Memory-Efficient Gated Delta Networks

Abstract

Long-context inference incurs growing memory costs from attention key–value (KV) caches. Gated Delta Networks (GDNs) mitigate this growth by summarizing history in fixed-size recurrent matrix states rather than maintaining a growing KV cache. Yet in hybrid large language models, these states are replicated across layers, heads, and concurrent requests, leaving a substantial aggregate memory footprint. Existing work has revealed low-rank structure in recurrent states and used it to motivate compression, but rank alone does not determine whether retained subspaces remain useful during decoding. We identify a temporal phenomenon, capacity–geometry decoupling: at a fixed rank, optimal approximations of the current state maintain nearly unchanged normalized squared reconstruction error, while reusing earlier subspaces leads to growing reconstruction and memory-read errors. Motivated by this observation, we introduce FACTORSTATE, a training-free representation that tracks the evolving recurrent state with compact factors under a fixed memory budget. FACTORSTATE performs recurrent reads and updates directly in factor space and periodically recompresses the factors, leaving pretrained weights and dense prompt processing unchanged. Experiments across Qwen and OLMo models show that FACTORSTATE reduces recurrent-state storage by approximately 61% while achieving task performance close to dense execution. GPU evaluations further demonstrate lower execution time and peak memory under both fixed-active-batch and continuous-batching workloads.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Gated Delta Networks, Linear Attention, Low-Rank State Compression, Recurrent State Tracking, Memory-Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_gQNxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates memory compression for the recurrent matrix states in Gated Delta Networks (GDNs) within hybrid Large Language Models. Motivated by the insight of capacity-geometry decoupling, the paper introduces FactorState, a training-free dynamic factor representation that stores state as low-rank factors. Until the size BB, the write buffer size, the computation is an exact match to the original. After BB, it uses compresses it back to the rank rr to avoid large grow in memory. Therefore, it improves end-to-end throughput and peak GPU memory.

Critical strengths

1. Clear Theoretical & Empirical Motivation

Prior work on linear attention / GDN state compression largely focused on static rank profiles or structural channel pruning. By separating capacity loss Ct,rC_{t,r} from geometry loss Ga,t,rG_{a,t,r} and measuring their direct impact on pre-write key reads mt=Sˉt⊤ktm_t = \bar{S}_t^\top k_t and post-write query reads ot=St⊤(qt/dk)o_t = S_t^\top (q_t/\sqrt{d_k}), the paper provides a compelling explanation for why static low-rank projections fail during autoregressive decoding.

2. Hardware-Aware Systems Engineering

The authors design a three-stage CUDA maintenance kernel using regularized Gram matrices, Cholesky decomposition, and warp-level Jacobi SVD. This avoids global synchronization spikes and eliminates external solver workspace overhead.

Critical weaknesses

W1. Inflated Memory and Throughput Claims via an FP32 Baseline.

While model weights and attention KV caches are stored in BF16, the evaluations use FP32 for GDN recurrent states (Sec. 4.1). Because FactorState relies on regularized Gram-Cholesky recompression, FP32 precision appears favorable for maintaining its numerical stability, whereas standard Dense GDN states are commonly stored in BF16 in practice. The paper does not evaluate performance or memory savings under a BF16 state setting, where the relative memory reduction of FactorState of FP32 over Dense BF16 would be considerably smaller.

Questions for authors

Q1. Performance and Memory Trade-offs Under BF16 Recurrent States

How does a Dense GDN baseline with BF16 recurrent states compare against Dense FP32 and FactorState in terms of downstream accuracy, peak memory, and decode throughput? Relatedly, can FactorState store its committed factors (Ut,Vt)(U_t, V_t) in BF16 while upcasting to FP32 in shared memory/registers during recompression step, or does low-precision factor storage degrade numerical stability?

6 comments
AuthorsOpenReview ↗

Rebuttal Response: FP32 Baseline and BF16 Memory–Accuracy Trade-offs

We thank the reviewer for recognizing the motivation behind capacity–geometry decoupling and the CUDA implementation. We have added matched-precision BF16 comparisons to directly address the fairness of the FP32 baseline and the practical benefits under lower-precision state storage.

Response to Major Concerns

  1. Why Dense FP32 is a valid baseline. Dense FP32 was not selected to artificially inflate the baseline's memory usage. The official Qwen3.5-4B configuration uses BF16 model weights but specifies mamba_ssm_dtype="float32" for the recurrent state. The precision of model weights and attention KV caches does not necessarily determine the precision of the repeatedly updated GDN state. Nevertheless, we agree that BF16 deployment is important and have therefore added a direct Dense BF16 versus FactorState BF16 comparison rather than extrapolating from FP32 results.

  2. BF16 factor storage and downstream fidelity. FactorState can persistently store both left/right factors and all buffered columns in BF16, while performing recurrent updates and Gram–Cholesky/SVD recompression in FP32. No additional persistent FP32 factor copies are required. On Qwen3.5-4B, Dense BF16 uses 24 MiB of GDN recurrent state per request, while FactorState BF16 uses 9.375 MiB, including all 25 physical factor columns. This is a 60.94% reduction in recurrent-state bytes at the same storage precision. Across 12 validation trajectories, relative to a common Dense FP32 reference, FactorState BF16 obtains a final-state error of 0.13153 and query/key-read NRMSE of 0.02459/0.02282. We also evaluated four 100-problem downstream subsets using the same BF16 inference backend:

BenchmarkDense BF16FactorState BF16
IFEval77.0%75.0%
RULER87.7%87.7%
Mathematics89.0%86.0%
EvalPlus70.0%69.0%

All four paired 95% confidence intervals for the score differences include zero. These results show comparable observed task performance under substantial state-storage reduction; we do not claim statistical equivalence or zero accuracy cost.

  1. Full-model GPU memory and throughput. We additionally measured offline concurrency on an RTX 3090 using real task inputs, matched output trajectories, and the same paged-attention backend. The measurement includes prefill, state initialization, decoding, maintenance, and cleanup. At maximum concurrency C192, we obtain:
MetricDense BF16FactorState BF16
Peak allocated memory18.894 GiB16.992 GiB
Peak reserved memory19.871 GiB17.918 GiB
Throughput889.57 tokens/s875.57 tokens/s

Thus, in this matched-workload setting, FactorState reduces full-model peak GPU memory by approximately 10%, with a modest throughput reduction (approximately 1.6%). We will clearly distinguish recurrent-state byte savings from measured full-model GPU memory savings, and specify the measurement protocol in the revision.

We appreciate the reviewer's suggestion. We hope the official state-precision clarification and these matched-BF16 quality and system measurements address the central concern about baseline fairness. If so, we would be grateful if the reviewer could reconsider the overall assessment in light of the additional evidence.

We thank the authors for the additional BF16 experiments and the clarification regarding the Qwen3.5 configuration. Cross-referencing the new results with the main paper and appendices raises a few specific follow-up questions:

1. Benchmark Protocol Discrepancy and 5.0 pp Accuracy Drops

  • Evaluation Protocol: The rebuttal evaluates the 400-problem subset (100 per task, 2,048-token cap for math) from Appendix E.3 (Table 14) rather than the full 5,683-instance suite in Sec. 4.3 (Figure 4 / Table 8, 8,192-token cap for MATH-500). Why was the reduced subset chosen over the main evaluation setup, and can the authors provide BF16 results under the full Sec. 4.3 protocol?

  • Cause of 5.0 pp Degradation: Comparing the rebuttal's BF16 results against the matching FP32 baseline on the exact same 400-case subset (Table 14) reveals a 5.0 pp drop on IFEval (80.0→75.080.0 \to 75.0) and a 5.0 pp cumulative drop on Mathematics (91.0→89.0→86.091.0 \to 89.0 \to 86.0). What is the primary cause of this degradation when storing factors in BF16?

3. Changed Systems Protocol and Loss of Throughput Advantage

  • Protocol Change: Instead of using the established RTX 3090 setups in Sec. 4.4, where either fixed-batch pure decode throughput (C=192, T=512, Table 2) or continuous batching (C=96, Table 16) is used, the rebuttal introduces a new protocol (C=192 with paged attention on real task inputs, including prefill and cleanup in the throughput calculation). Why was the protocol changed, and can the authors report matched BF16 numbers under the exact Table 2 and Table 16 settings?
AuthorsOpenReview ↗

BF16 Follow-up: Accuracy and System Performance

Thank you for pointing out the differences between the evaluation settings in the appendix and the main paper. Given the time and compute available during the rebuttal period, we used the fixed 400-example subset from Table 14 in Appendix E.3 and retained its 2,048-token output cap for mathematics. This let us compare results on the same examples as in the appendix. We have not completed the BF16 evaluation on the full 5,683-example suite in Section 4.3.

On the 5-point accuracy drop. Our original Dense baseline followed the official Qwen3.5-4B configuration, which stores the GDN recurrent state in FP32 (mamba_ssm_dtype: float32). The earlier FP32 and BF16 evaluations used the same examples but different versions of the implementation. To isolate the effect of state storage precision, we reran all four configurations on the same 100 IFEval prompts and 100 mathematics problems from the original subset. For each method, we kept the implementation and generation settings fixed and changed only the storage precision of the persistent recurrent state.

Task (100 examples each)Dense FP32Dense BF16FactorState FP32FactorState BF16
IFEval (strict prompt-level accuracy)77%76%81%80%
Mixed mathematics subset88%85%89%89%

In this controlled comparison, switching FactorState to BF16 reduced IFEval accuracy by 1 percentage point (95% paired CI: [−5, +3] pp) and left mathematics accuracy unchanged (95% paired CI: [−4, +4] pp). Neither point estimate reproduces the earlier 5-point drop. The earlier comparison across implementation versions therefore does not isolate the effect of BF16 factor storage.

On the systems evaluation protocol. We used real task inputs in the initial rebuttal to measure end-to-end throughput for variable-length requests, including prefill and cleanup. To address your question about the original settings, we also ran four paired BF16 trials on an RTX 3090 for each workload from Tables 2 and 16.

WorkloadDense BF16 end-to-end time (median)FactorState BF16 end-to-end time (median)
Table 2: C = 192, 512 decode steps30.074 s30.056 s
Table 16: C = 96, continuous batching428.70 s417.01 s

End-to-end time was nearly identical in the Table 2 workload. In the Table 16 workload, both methods processed the same 1,024 requests with identical per-request output lengths, generating 983,040 output tokens per method in each trial. FactorState achieved a median paired end-to-end throughput gain of 2.92%. With BF16 storage for both methods, FactorState reduced recurrent-state storage per request from 24 MiB to 9.375 MiB, a 60.94% reduction.

Thank you for suggesting these comparisons under the original workloads. We hope the additional results clarify the trade-offs among accuracy, throughput, and recurrent-state storage for FactorState with BF16 storage.

Reviewer_r66AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the paper observes the capacity-geometry decoupling phenomenon, where at fixed rank, optimal approximation is better than extracting and reusing an earlier subspaces across decoding steps. Given this observation, a decode-only state is maintained and updated to ensure less inference memory while maintaining general accuracy across a broad set of tasks.

Critical strengths
  1. Paper provides a new observation upon the existing low-rank observation on decoding state. It states that the prior derived low-rank subspace can be suboptimal if reused directly and thus needs to be updated properly.

  2. The method is designed to be training-free and calibration free, which are intentionally lightweight and less biased.

Critical weaknesses
  1. While the paper identifies capacity–geometry decoupling, it does not sufficiently investigate the underlying mechanisms driving subspace evolution. For instance, it does not provide concrete reason or characterization on why the decoding generation is changing, is it simply because delta-rule based updates or it is because some new mechanism underlying? This mechanism from current paper is missing.

  2. The novelty is thus limited. Author primarily proposed an engineering heuristic based upon the observation, which combines standard factorized updates with periodic low-rank recompression. without explicitly exploiting the observed subspace dynamics to guide its algorithmic design. This weakens the connection between the claimed insight and the actual algorithmic contribution.

  3. The empirical evidence is not convincing. While paper presents some impressive numbers and does provides system-side designs such as continuous batching, the benchmark setting was primarily synthetic, where the author assumes a input prompt size of fixed 64 and sequence length of 2048, that is not a convincing setting for high concurrency, nor a trustworthy workload.

Questions for authors
  1. Could the authors elaborate on underlying mechanism of capacity-geometry decoupling?

  2. The main experiments focused primarily on small models that are 4B ~ 9B size. How does the method perform when scales to some current open-sourced 30B and 70B models? This is important because 30B-class open-source hybrid attention models, such as Qwen3.8-27B, are widely used for complex real-world tasks, so providing an efficient approach for these models is meaningful.

  3. Given that the FactorState is decode focused, does prefill-heavy workloads changes paper's conclusion?

3 comments
AuthorsOpenReview ↗

Rebuttal Response: Subspace Dynamics, Design, and Practical Workloads

We thank the reviewer for recognizing the capacity–geometry decoupling observation and the training-free, calibration-free nature of FactorState. We address the mechanism, its connection to our design, and the representativeness of our system evaluation below.

Response to Major Concerns

  1. Mechanism of subspace evolution. We do not posit a new state-update rule beyond the existing gated delta rule: St=αtSt−1+ktδt⊤S_t=\alpha_tS_{t-1}+k_t\delta_t^\top, where δt=βt(vt−αtSt−1⊤kt)\delta_t=\beta_t(v_t-\alpha_tS_{t-1}^\top k_t). Scalar decay alone does not rotate the singular subspaces, but new writes can introduce directions outside earlier subspaces. Crucially, the rank-one form only makes subspace evolution possible; it does not establish how strongly it occurs in pretrained GDNs, whether earlier subspaces become inadequate, or whether this affects key/query reads. We therefore isolate capacity loss from geometry loss on identical dense trajectories at fixed rank. In Figure 1, at rank 16, the optimal normalized squared reconstruction error remains around 1.00%, while reusing a subspace aged 128 steps yields 5.79% error. Additional analysis of out-of-subspace writes across 16 validation streams finds a positive association with subspace changes after controlling for relative write strength, cumulative decay, average update gating, and time position (stream-level partial Spearman ρ=0.408\rho=0.408, 95% CI [0.374,0.441][0.374,0.441]). This characterizes the temporal-geometric consequences of the existing update rule; we do not interpret the correlation as causal proof.

  2. How the finding motivates FactorState. Capacity–geometry decoupling shows that a state can remain compressible at a low rank even as a fixed basis becomes inadequate. Thus, compression must both bound storage and allow the retained directions to evolve. FactorState directly exploits the GDN rank-one write: new writes are appended to the factor representation, while the pre-write key-memory read, state update, and post-write query read operate in factor space. Periodic recompression reselects directions from the current represented state rather than freezing the prefill basis, and write buffering amortizes this maintenance cost. The empirical finding determines the required dynamic representation; the GDN recurrence determines how to implement it efficiently. We do not claim a new general-purpose SVD primitive.

  3. More representative workloads. The original fixed-length benchmarks control workload size to isolate state-update and maintenance costs. We additionally evaluated 192 real task inputs from IFEval, mathematics, and EvalPlus, with full prompt lengths of 30–914 tokens rather than a fixed 64-token prompt. The RTX 3090 offline-concurrency experiment uses matched output trajectories and a shared paged-attention backend, and includes prefill, initialization, decoding, and maintenance in the measured execution. At maximum concurrency C192, Dense BF16 versus FactorState BF16 yields 18.894 vs. 16.992 GiB peak allocated memory, 19.871 vs. 17.918 GiB peak reserved memory, and 889.57 vs. 875.57 tokens/s throughput. Thus, the structural compression translates into approximately 10% lower full-model peak memory on this measured real-input workload, while throughput remains close. We do not generalize this offline workload to all production-serving conditions.

  4. Larger models and prefill-heavy workloads. For one recurrent matrix, FactorState changes storage complexity from O(dkdv)O(d_kd_v) to O((dk+dv)(r+B+1))O((d_k+d_v)(r+B+1)). This representation does not depend algebraically on total model parameter count, but suitable ranks and end-to-end efficiency must still be validated for each model. FactorState retains dense prefill and primarily reduces persistent state storage during decoding; consequently, the prefill/decode ratio affects the net performance benefit. We limit empirical claims to the models and workloads evaluated rather than extrapolating to untested 30B–70B configurations.

We appreciate these constructive questions and hope the additional mechanistic evidence, design clarification, and real-input measurements help address the concerns about the connection between our finding and method, as well as its practical value.

Thank you for the rebuttal. I remain unconvinced by the mechanistic novelty, as the observed subspace evolution largely follows from the existing rank-one update rule, and its connection to the proposed algorithm is not sufficiently distinctive. Given these limitations, strong empirical evidence becomes particularly important. I was particularly concerning with the fact that the evaluation remains limited to relatively small models (4B–9B) and does not establish the method's effectiveness on larger models or more representative serving workloads. Therefore, my concerns about the paper's overall significance remain, and I will maintain my score.

AuthorsOpenReview ↗

Follow-up: Scope and Limitations

Thank you for reviewing our response again and for clearly communicating your remaining concerns. We understand your reservations regarding methodological novelty, model scale, and the representativeness of the serving workloads.

Following your suggestions, we have further clarified the scope of our evaluations in the revised manuscript and included additional system measurements using real task inputs. Evaluation on larger models and a broader range of serving scenarios still requires further investigation.

We appreciate your feedback throughout the review process, which has helped us more clearly define the contributions and limitations of our paper.

Reviewer_gKr3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a low rank factorization and online evolving method for GDN, motivated by the observation that a low rank structure can retain sufficient memory capacity, while the underlying subspace must evolve with the recurrent state. The authors evaluate their method on several benchmarks, demonstrating improved decoding efficiency while maintaining task performance close to the dense baseline.

Critical strengths

The paper presents a simple but valuable observation that a low rank representation can preserve sufficient memory capacity, while the underlying subspace need to evolve with the recurrent state. This observation provides a clear motivation for the method. The method itself is simple and practical. It performs recurrent update directly in the low rank space without requiring additional training.

Critical weaknesses

Although the authors evaluate the method on some benchmarks and report performance close to dense baseline, I would like to see more experiments investigating whether the model can maintain its performance over longer decoding length. Since the method periodically perform low rank truncation, approximation errors my accumulate over time. The paper can include its performance during much longer generation.

Questions for authors

Since factorState performs low rank compression and truncation, approximation errors may accumulate over time. Could the authors provide additional experiments with longer decoding to demonstrate whether task performance remain stable.

3 comments
AuthorsOpenReview ↗

Rebuttal response: Long-horizon generation quality

Response to the Main Concern

We thank the reviewer for recognizing FactorState's motivation and training-free implementation. Your concern that repeated truncation might lead to accumulated errors during long decoding is important. We have therefore added long-horizon, free-running generation evaluations with the full model to address it directly.

1. Task performance during long generation. The new experiments use Qwen3.5-4B, with four stochastic samples for each of all 500 MATH-500 problems and eight samples for each of 175 LiveCodeBench problems. Both methods generate autoregressively and independently, rather than merely reusing Dense's layer-wise input trajectories. The generation lengths also directly cover the regime of concern: MATH-500 outputs average approximately 2K tokens, with a 95th percentile above 8K; LiveCodeBench outputs average approximately 7K tokens, with 95th-percentile lengths of 18,542 and 20,188 tokens for Dense and FactorState, respectively. Average pass@1 over these long-generation experiments is as follows:

BenchmarkSamples per methodDenseFactorStateDifference and paired 95% CI
MATH-500500 problems × 494.85%95.05%+0.20 pp [−0.70, +1.10]
LiveCodeBench175 problems × 836.21%36.57%+0.36 pp [−2.14, +2.86]

Both experiments use stochastic sampling with thinking disabled. The table reports average pass@1 across repeated samples. The paired statistical analysis treats each problem as the unit of observation and retains all of its samples, rather than treating multiple samples from the same problem as independent problems. Average performance on both tasks remains close to Dense. Because these results come from independent full-model generation, they incorporate cross-layer state propagation, changes in token selection, and subsequent autoregressive feedback. They therefore address long-horizon task quality more directly than the original short-output tasks and conditional state replay.

We thank the reviewer again for this suggestion. We hope this targeted long-generation evidence addresses your primary concern, and we would be grateful if you would reconsider your assessment in light of these results.

Thank you for the rebuttal and the additional experiment. My main concern has been addressed, and I will change my score.

AuthorsOpenReview ↗

Thank You for Your Positive Feedback

We sincerely appreciate your careful review and constructive suggestions, which have been very helpful in improving our work.

Reviewer_RqorOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a post-training method for recurrent-state memory compression during decoding in (hybrid) large language models equipped with Gated Delta Network (GDN) layers. While GDN layers replace a linearly growing KV cache with a fixed-size per-head matrix state, these states still consume substantial GPU memory and HBM bandwidth. The authors first present an empirical diagnosis of subspace aging during decoding to motivate dynamic subspace tracking. At a fixed rank, reusing an earlier (aged) low-rank subspace causes growing reconstruction and read errors, while the optimal rank-r approximation of the current state does not degrade. In the proposed FactorState method, after each GDN state is SVD-factored once into rank-r factors following prefill, the gated delta rule runs exact updates directly in factor space during decoding by appending new columns to the left and right factors. After B buffered writes, the factors are recompressed back to rank r, providing a clean formulation of the left and right state updates under Gated DeltaNet. Under the default configuration (r=16, B=8), FactorState retains 39.06% of the dense FP32 state.

Critical strengths
  1. Clear presentation and motivating empirical diagnosis: The paper is clearly presented and easy to follow. Prior linear-attention compression work has relied primarily on static rank profiles or offline channel and head pruning. By holding dense input trajectories and the rank budget fixed, the authors cleanly show that relying on aged static subspaces during decoding degrades both pre-write key reads and post-write query reads, whereas tracking the evolving low-rank subspaces online limits these errors substantially. This provides a clear empirical justification for why post-training GDN state compression should track evolving subspaces online rather than freezing prefill bases.

  2. Clean formulation of factor-space recurrence with buffered recompression: FactorState provides a clean formulation of the GDN recurrence in factor space for both left and right factors, using buffered writes and periodic recompression to update balanced rank-r factors without materializing the full d_k × d_v recurrent matrix during either updates or recompression. Implemented in CUDA, this buffered factor-space update reduces persistent FP32 GDN state memory by 60.9% while improving high-concurrency decode throughput by 21.8%–29.6%.

Critical weaknesses
  1. Limited algorithmic novelty: The dynamic evolution of the state geometry, which forms the main analytical motivation of the paper, follows directly from the rank-one state updates in the gated delta rule. Moreover, truncating a buffered low-rank update back to rank r via thin QR factorizations of the two factors followed by an SVD of the small core matrix is textbook block-incremental SVD. As a result, the core mathematical and algorithmic contribution is incremental.

  2. Limitations in baselines and evaluation scope:

    • Weak subspace baselines: Static and Oja at rank 23 barely move away from the prefill subspace, while COD with median singular-value shrinkage is not designed for Frobenius state reconstruction.
    • No low-precision / quantized dense state baselines (BF16/FP16 and INT8/FP8): While FactorState (r16/B8) uses 39.06% of FP32 dense state bytes, it extra FLOPs per token for its compression. Meanwhile, in the concurrent work LeapQuant [1], training-free 8-bit GDN/KDA state quantization reports near-lossless accuracy and decoding speedups using a similar window (buffer)-based scheme, with quantization applied at the end of each window. Without comparing accuracy, latency, and peak memory against dense BF16 (50.0% of FP32 bytes) and dense 8-bit (25.0% of FP32 bytes) states, one cannot tell whether FactorState's dynamic SVD complexity buys anything beyond a simpler precision reduction method.
    • Missing adaptive-rank and unbuffered incremental-SVD fidelity analysis: It would be valuable to evaluate per-head adaptive rank allocation as well as the state-replay fidelity of unbuffered per-step truncated incremental SVD (r24/B0), especially given that r24/B0 achieves worse IFEval and math accuracy than r16/B8 (Table 15).
    • Limited coverage of long-decode regimes: Most evaluations avoid the long-decode regime where state compression is most likely to hurt. Because prefill remains dense and unchanged, near-zero score differences (Δ) on short-output benchmarks such as RULER (capped at 32–128 output tokens) and ChartQA (capped at 100 tokens) say little about the actual quality of factorized decoding over long horizons. Moreover, Thinking mode, the main long-decode use case of Qwen3.5, is disabled. In addition, state-representation fidelity is evaluated layer-wise on recorded dense inputs, excluding cross-layer error propagation.
Questions for authors

Please address the points raised under Critical Weaknesses above, and specifically answer the following:

  1. Matched-byte precision baselines: How does FactorState (r16/B8) compare with (a) a dense BF16 state (50.0% of FP32 bytes) and (b) dense 8-bit state quantization (25.0% of FP32 bytes, e.g., simple per-row scaling or LeapQuant-style windowed quantization [1]) on the Figure 3 replay metrics and on MATH-500 / long generations? Can FactorState's factors also be stored in BF16, and how does that affect reconstruction and read errors?
  2. Stronger subspace-tracking baselines: Please report state and read replay fidelity (Figure 3 metrics) for r24/B0 (per-step truncated incremental SVD at equal storage), r16/B0, and COD without singular-value shrinkage.
  3. Long-context and long-decode regime: Please report long-decode generation quality, as well as peak GPU memory and throughput on 16K–64K sequences where the attention layers' KV cache dominates, to clarify what fraction of total memory the method saves in long-context serving.
  4. Why does r24/B0 achieve lower task quality than r16/B8 (Table 15) despite maintaining a higher rank at the same storage budget?

Reference

[1] Pan, Yi, et al. "LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization." arXiv preprint arXiv:2609.38166 (2026).

4 comments
AuthorsOpenReview ↗

Rebuttal response : Novelty and stronger baselines

Response to Major Concerns

We thank the reviewer for the careful evaluation of our paper, particularly for recognizing our empirical observation of capacity–geometry decoupling, the factor-space recurrence, and the CUDA implementation. To address your concerns about novelty, comparison methods, and practical effectiveness, we have conducted additional analyses and experiments, as detailed below.

1. Novelty and methodological motivation. We agree that thin QR and small-matrix SVD are well-established, efficient numerical tools, and we do not claim these factorization steps as new mathematical primitives. FactorState is not simply an off-the-shelf incremental SVD formula applied to GDN. Prior work has observed low-rank structure in GDN recurrent states; we further find that a relatively stable low-rank capacity does not imply that the subspaces carrying the information remain effective over time. To quantify this phenomenon, we distinguish the capacity loss of the optimal low-rank approximation from the geometry loss incurred by reusing historical subspaces, under the same Dense trajectories and a fixed rank. Figure 1 of the original paper shows that, at rank 16 and a subspace age of 128, the normalized squared reconstruction error of the current optimal approximation is approximately 1.00%, whereas reusing the aged subspace incurs 5.79% error. The pre-write key-memory read and post-write query read also exhibit pronounced aging-related errors. This finding demonstrates that retaining sufficient rank alone is not enough: the representation directions must also be updated during decoding. We designed FactorState to meet this requirement. Its factor updates directly reflect the rank-one writes and low-rank state structure of GDN: a new write can be appended as a pair of factor columns, while reads and writes are performed entirely in factor space without repeatedly materializing the full dk×dvd_k\times d_v state matrix. Periodic recompression updates the retained directions under a fixed storage budget. Thus, our contributions are the discovery of temporal state geometry in GDN, an online representation tailored to its recurrence, and a practical systems implementation, rather than merely combining standard factorization formulas.

2. Stronger subspace baselines. We thank the reviewer for suggesting unbuffered updates and a COD variant without singular-value shrinkage. We have added r16/B0, r24/B0, and COD without shrinkage on the same 12 validation trajectories, following the state, query-read, and key-read error metrics of Figure 3 (512 updates, covering all GDN layers and heads):

MethodPhysical columnsState error ↓Query NRMSE ↓Key NRMSE ↓
FactorState r16/B8250.110310.011670.01775
r16/B0170.112340.011870.01840
r24/B0250.077810.008010.01044
COD median shrink250.275420.038430.05443
COD without shrink250.137520.014970.02429

At the same physical capacity of 25 columns, r24/B0 achieves higher conditional-replay fidelity, whereas the default r16/B8 balances fidelity against online maintenance cost. Over 512 updates, r16/B8 performs approximately 56 recompressions per state, compared with 512 for B0. Under the same FP32, RTX 3090, C192 workload reported in Appendix E.3 of the original paper, the full-request times of r16/B8 and r24/B0 are 36.27 s and 73.93 s, respectively. Removing COD's shrinkage also substantially reduces its error, confirming the value of this suggested ablation, although FactorState r16/B8 still outperforms this COD variant. We will include these results in the revised manuscript to distinguish the two design objectives of state fidelity and maintenance cost more clearly.

AuthorsOpenReview ↗

Rebuttal response : BF16, long generation, and scope

Response to Major Concerns (continued)

3. Low-precision states and memory savings. We appreciate the suggestion to include low-precision comparisons. First, the original Dense FP32 baseline was not chosen to artificially inflate baseline memory consumption: the official Qwen3.5-4B configuration uses BF16 weights while specifying mamba_ssm_dtype="float32". We have additionally implemented a matched-precision comparison between Dense BF16 and FactorState BF16. In FactorState BF16, both the left and right factors and the buffered columns are persistently stored in BF16, while recurrent computation and recompression use FP32 arithmetic. For Qwen3.5-4B, Dense BF16 requires 24 MiB of recurrent-state storage per request, compared with 9.375 MiB for FactorState BF16, corresponding to a 60.94% reduction in recurrent-state bytes at the same storage precision. Across four task evaluations containing 100 problems each, the Dense BF16 / FactorState BF16 scores are IFEval 77% / 75%, RULER 87.7% / 87.7%, Math 89% / 86%, and EvalPlus 70% / 69%, respectively. In a matched-workload experiment with real inputs on an RTX 3090 at C192, full-model peak allocated memory decreases from 18.894 GiB to 16.992 GiB, with throughput of 889.57 / 875.57 tokens/s, respectively. These results show that FactorState's compression stems from its low-rank representation and can yield actual GPU memory savings even under BF16 storage. Regarding quantization methods, we view numerical bit-width reduction and structured low-rank compression as two approaches with different cost components; we therefore do not rank their performance without measurements on a common platform.

4. Long-generation quality. To further examine the impact of periodic truncation over full autoregressive trajectories, we have added evaluations on MATH-500 (500 problems, four samples per problem) and LiveCodeBench (175 problems, eight samples per problem) using Qwen3.5-4B. Dense and FactorState generate independently in free-running mode with thinking disabled. We report average pass@1 across repeated stochastic samples, with paired 95% confidence intervals computed at the problem level. On MATH-500, Dense / FactorState achieves 94.85% / 95.05%, a difference of +0.20 percentage points (pp), 95% CI [−0.70, +1.10]. On LiveCodeBench, the corresponding results are 36.21% / 36.57%, a difference of +0.36 pp, 95% CI [−2.14, +2.86]. LiveCodeBench outputs average approximately 7K tokens, with 95th-percentile lengths of 18,542 and 20,188 tokens for Dense and FactorState, respectively. These experiments cover substantially longer generations than the short-output benchmarks and incorporate cross-layer propagation as well as the autoregressive feedback from subsequently generated tokens. They support the conclusion that FactorState maintains average task quality close to Dense on the evaluated long-generation tasks.

5. Adaptive rank and long contexts. We appreciate the suggestion to explore further optimizations. We investigated rank allocation at the layer/head level but have not observed consistent improvements over uniform rank across different capacity budgets. We therefore retain the default uniform configuration, which has a clearly specified memory budget and requires no calibration. For longer inputs, such as 16K–64K tokens, GDN recurrent-state storage remains fixed while the full-attention KV cache grows with context length. It is therefore important to distinguish the compression ratio of recurrent-state bytes from changes in full-model peak GPU memory. Our existing systems experiments report these quantities separately. Adaptive rank and a broader range of long-context workloads remain directions for future investigation.

We again thank the reviewer for the concrete suggestions. We hope that these additional experiments and clarifications better convey FactorState's empirical finding, design motivation, and practical value, and we respectfully invite you to reconsider your overall assessment in light of the new evidence.

Thank you for the detailed response and for running the additional experiments during the rebuttal period. The COD-without-shrinkage and B0 replay ablations, together with the longer free-running MATH-500 and LiveCodeBench evaluations, address my questions on shrinkage bias and longer-output generation in FP32. However, there are some concerns that remain important for assessing practical impact: against Dense BF16 (50% state bytes), FactorState BF16 no longer shows a throughput gain (875.6 vs. 889.6 tok/s) and incurs a 2–3 pp drop on IFEval/Math while 8-bit state quantization remains untested. I also noticed that the manuscript is not updated to include these discussions and new results. Please make sure all of these results are included in the revised manuscript. I remain positive about this work and will take these responses into account in my final assessment during the discussion phase.

AuthorsOpenReview ↗

Follow-up: BF16 Performance and Future Directions

We acknowledge your concerns regarding practical BF16 performance and comparisons with lower-bit quantization. The Dense baseline uses the highly optimized FLA implementation, while there may still be opportunities for further optimization of FactorState's CUDA implementation, which could be one of the factors affecting its BF16 throughput.

Due to the limited time and computational resources available during the rebuttal period, we have not completed the 8-bit quantization comparisons or more extensive performance analysis and optimization. We consider these important directions for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.