- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SenSeed, a sensitivity-aware extension of seed-based LLM weight compression. SenSeed represents each eight-weight block using an LFSR seed and quantized basis coefficients, while assigning different seed lengths and basis counts according to a checkpoint-derived estimate of block importance. A rate–distortion allocation determines the configuration of each block under a target bit budget, and the decoder reconstructs the same allocation without storing per-block metadata.
Critical strengths
SenSeed enables different blocks to use different seed lengths and basis counts without storing a configuration for every block. The decoder reproduces the allocation from existing model parameters and small shared tables, reducing the metadata overhead of sensitivity-aware allocation.
Critical weaknesses
At the same 4-bit rate, SenSeed improves average zero-shot accuracy over SeedLM by only 0.5–1.1 points and is often comparable to or worse than S-Quant and AWQ. Its lower-rate setting saves only 3–6% of the storage and does not consistently improve accuracy.
The method mainly applies standard sensitivity-aware mixed-precision allocation to SeedLM; the metadata-free decoding schedule is useful, but the overall algorithmic contribution and empirical gains are fairly modest.
Questions for authors
Can SenSeed be efficiently executed on existing GPUs? What's the speed advantage over existing 4-bit baselines?
The security claim is unclear to me. Why should making a single fault damage more weights be considered protection rather than making the model more fragile?
5 comments
Response to Reviewer Kiv1 (2/2)
C3. How would SenSeed run on GPUs, and how fast is it compared with 4-bit baselines?
Response. We have not built a GPU kernel and do not claim a GPU speedup. Like SeedLM (evaluated on an FPGA) and S-Quant (on an ASIC), SenSeed targets hardware where reading weights from memory is the bottleneck and generating them with an LFSR is cheap. Variable rates do not force sequential decoding. The size of every block depends only on the RMSNorm gains and the small side tables, so the decoder can compute where each block starts (a prefix sum) before reading any weight data, and then decode all blocks in parallel. On a GPU, the random basis for each seed length could be read from a precomputed table, as codebook-based kernels do. Compared with a 4-bit integer format at the same rate, the memory traffic is the same, so in the memory-bound regime we expect no speed advantage at Mode 2 and 2.5% to 6% less traffic at Mode 1. The decoding work per weight is the same as SeedLM's (on average 3.00 to 3.13 coefficients per block at 4.0 bits, Appendix A.8).
Changes in the paper: This discussion is added to Section 4.2.
C4. A single fault corrupts far more weights in SenSeed than in SeedLM (Section 4.3). Why is this larger fault footprint presented as protection rather than fragility?
Response. The proposed weight representation approach obscures the correspondence between stored bits and reconstructed weights, potentially making targeted bit-flip attacks more difficult. While random bit-flips are possible in SenSeed, it makes it easily detectable by amplifying the fault effect, as established in the paper. We agree that the same amplification may increase fragility under noise-induced bit-flips. However, common practice to achieve robustness against environmental noise is to use ECC, which is often inadequate for attack-induced bit-flips. We note that SenSeed can be combined with ECC to simultaneously provide protection against both noise and attack-induced bit-flips. Without detection and recovery, fault amplification increases the impact of corruption and could be viewed as increased fragility. Our original language did not make this distinction sufficiently clear, and we have revised it accordingly.
Section 4.3 does not claim intrinsic bit-flip resistance, but characterizes a distinct fault-propagation property of generated-weight representations. In SenSeed, corruption of shared schedule-determining state can propagate to many reconstructed weights, producing a larger model-level deviation than a localized perturbation. This amplification can increase the observability of corruption to an external integrity monitor.
As discussed in our response to Reviewer UvpM (A1), SenSeed is complementary to ECC or cryptographic integrity checks rather than a replacement for them. In particular, the high fan-out of a small set of shared decoding quantities suggests that strong protection could be concentrated on this critical state instead of uniformly protecting the entire compressed payload, potentially reducing the redundancy and hardware cost of full-payload protection.
Changes in the paper: We have accordingly revised the manuscript to narrow down the claims of intrinsic protection or robustness.
Follow-up on our response
Dear Reviewer Kiv1,
We have carefully addressed all the constructive points and concerns you raised in your initial review in our responses above, and we have reflected the requested changes in the revised manuscript (marked in red). Since the discussion period ends on November 18, we respectfully ask whether our responses have addressed your concerns. If you still have questions or suggestions, please let us know and we will do our best to address them before the deadline. Your comments are highly valuable to us. Thank you once again for your time and insightful comments.
Regards, Authors
Response to Reviewer Kiv1 (1/2)
We thank the reviewer for the careful review.
C1. The gains are modest (+0.5 to 1.1 points of accuracy at 4 bits) and comparable to or worse than S-Quant and AWQ, and Mode 1 saves only 3% to 6% of the bits.
Response. We thank the reviewer. At 4.0 bits per weight, SeedLM is only 1.2 to 3.8 points of mean zero-shot accuracy below FP16, so any method can gain at most a few points at this rate. What matters is how much of this gap a method closes. SenSeed's Mode 2 raises the mean accuracy by 0.5 to 1.1 points, which closes 27% to 55% of the gap (Table 2). In perplexity, it removes 23% to 42% of SeedLM's degradation from FP16, measured in log-perplexity (5.85 to 5.69, 5.18 to 5.06, 7.17 to 6.92 and 6.32 to 5.96). Both comparisons use the same codec and the same evaluation code, so the difference comes only from what SenSeed adds to SeedLM: an allocation that needs no stored bits, and four FP16 outlier columns chosen by a rule that needs no data (Section 3.5).
The S-Quant and AWQ numbers are published results from other implementations and are listed for context. As Appendix B.5 shows, the 0.2 bits that S-Quant appears to save are spent elsewhere: its 3.8 bits do not count the number of coefficients of each block, which its decoder must read. Stored with a fixed number of bits, this count adds 0.19 bits per weight, so its actual rate is about 3.99 bits, essentially that of our Mode 2. AWQ is an integer format calibrated on data, while SenSeed is a seed-based codec that uses no data, so the two target different settings. Even so, Mode 2 reaches a lower perplexity than AWQ on all three Llama models.
Mode 1 is defined as the lowest rate at which SenSeed matches SeedLM's perplexity, so its accuracy is meant to match SeedLM's, and it does, within about one point. It saves 2.5% to 6% of the bits (0.40 GB on Llama-2-13B) without storing any metadata or using calibration data. While checking these numbers, we found an error in the submitted Table 1: its Mode 1 row repeated SeedLM's perplexities, and for Llama-3-8B and Mistral-7B it listed interpolated rates (3.88 and 3.78) instead of measured ones. The measured Mode 1 configurations use 3.77, 3.75, 3.90 and 3.85 bits per weight and reach perplexities of 5.84, 5.17, 7.11 and 6.17. Table 2 already reported the accuracies of these configurations. The saving is therefore 2.5% to 6%, not 3% to 6%.
Changes in the paper: Table 1 now gives the measured Mode 1 rates and perplexities, and the Mode 2 perplexity of Llama-2-7B is corrected from 5.68 to 5.69 (the 42% in Section 4.1 was already computed from the measured value and does not change). "3% to 6%" is now "2.5% to 6%" throughout the paper.
C2. SenSeed is a standard sensitivity-aware mixed-precision allocation applied to SeedLM.
Response. We agree that the allocation rule itself is classical, and we cite it as such (Shoham and Gersho; Everett). What is new is making such an allocation worthwhile for individual blocks of 8 weights in a seed-based codec. The obstacle is that the decoder must know each block's configuration before it can read the block. Storing this choice with a fixed number of bits costs ⌈log2J⌉ bits per block for J configurations, which is 0.25 to 0.375 bits per weight in our setting (Section 3.1). That is as much as or more than the entire saving of Mode 1 (0.10 to 0.25 bits per weight). This is why SeedLM uses one configuration for the whole model, and why S-Quant's per-block choice of k is not free. Our contribution is to design the allocation so that the decoder can recompute it from information it already holds, without any per-block entry, together with a sensitivity estimate that needs no calibration data and meets this requirement. As a result, the allocation costs no stored bits per block. The only extra storage is a few small tables, one number per row or column of the attention output and MLP down projections plus a short global list, which take at most 0.005 bits per weight (Appendix A.7).
The experiment requested by Reviewer UvpM (Table R1 in our response to that review, Llama-2-13B) measures this directly. When the map is paid for out of the same 4.0 bits, a map with a fixed number of bits per block makes the result worse than uniform SeedLM (0.0561 against 0.0398 in loss increase over FP16), and so does an ideally compressed map once it is sized for the allocation it describes (0.0445). SenSeed needs neither the map nor an entropy decoder.
Changes in the paper: Table R1 is added to the paper as Table 8 (Appendix B.6), with a summary in Section 4.1.