acceptodds
Under review as a conference paper at ICLR 2027

SenSeed-LLM: Sensitivity-Aware Secure Seed-Based Weight Quantization for LLMs

Abstract

Large language models (LLMs) incur substantial storage, memory-bandwidth and energy costs, motivating compact weight representations. Existing seed-based compression methods reconstruct weights from compact pseudo-random representations but do not explicitly account for the non-uniform sensitivity of model weights. We introduce SenSeed, a sensitivity-aware seed-based weight compression framework that uses lightweight Linear Feedback Shift Register (LFSR)-based weight generation with non-uniform bit allocation. SenSeed assigns larger representation budgets to sensitive weights while aggressively compressing less sensitive regions. Importantly, this non-uniform allocation requires no side-information: the decoder deterministically reconstructs the bit-allocation schedule, with no rung depending on the decoded weights, eliminating the need to store per-block metadata or use calibration data while preserving the baseline coding rate. Experiments on Llama-2-7B/13B, Llama-3-8B and Mistral-7B models show that SenSeed matches 4-bit perplexity of SeedLM with 3-6% fewer bits, while at the same 4 bits/weight it reduces perplexity degradation by 23-42% and zero-shot accuracy loss by 27-55% relative to SeedLM. We also show that SenSeed simultaneously achieves high security against bit-flip attacks on model parameters, as bit corruption affects multiple reconstructed weights, greatly amplifying its impact and making it easier to detect. We further implement SenSeed in an ASIC-based accelerator and demonstrate modest hardware overhead compared to prior seed-based approaches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Seed-Based Compression, Sensitivity-Aware Quantization, LFSR-Based Weight Generation, Hardware-Aware Compression, Bit-Flip Attack Detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Kiv1OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SenSeed, a sensitivity-aware extension of seed-based LLM weight compression. SenSeed represents each eight-weight block using an LFSR seed and quantized basis coefficients, while assigning different seed lengths and basis counts according to a checkpoint-derived estimate of block importance. A rate–distortion allocation determines the configuration of each block under a target bit budget, and the decoder reconstructs the same allocation without storing per-block metadata.

Critical strengths

SenSeed enables different blocks to use different seed lengths and basis counts without storing a configuration for every block. The decoder reproduces the allocation from existing model parameters and small shared tables, reducing the metadata overhead of sensitivity-aware allocation.

Critical weaknesses

At the same 4-bit rate, SenSeed improves average zero-shot accuracy over SeedLM by only 0.5–1.1 points and is often comparable to or worse than S-Quant and AWQ. Its lower-rate setting saves only 3–6% of the storage and does not consistently improve accuracy.

The method mainly applies standard sensitivity-aware mixed-precision allocation to SeedLM; the metadata-free decoding schedule is useful, but the overall algorithmic contribution and empirical gains are fairly modest.

Questions for authors

Can SenSeed be efficiently executed on existing GPUs? What's the speed advantage over existing 4-bit baselines?

The security claim is unclear to me. Why should making a single fault damage more weights be considered protection rather than making the model more fragile?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Kiv1 (1/2)

We thank the reviewer for the careful review.

C1. The gains are modest (+0.5 to 1.1 points of accuracy at 4 bits) and comparable to or worse than S-Quant and AWQ, and Mode 1 saves only 3% to 6% of the bits.

Response. We thank the reviewer. At 4.0 bits per weight, SeedLM is only 1.2 to 3.8 points of mean zero-shot accuracy below FP16, so any method can gain at most a few points at this rate. What matters is how much of this gap a method closes. SenSeed's Mode 2 raises the mean accuracy by 0.5 to 1.1 points, which closes 27% to 55% of the gap (Table 2). In perplexity, it removes 23% to 42% of SeedLM's degradation from FP16, measured in log-perplexity (5.85 to 5.69, 5.18 to 5.06, 7.17 to 6.92 and 6.32 to 5.96). Both comparisons use the same codec and the same evaluation code, so the difference comes only from what SenSeed adds to SeedLM: an allocation that needs no stored bits, and four FP16 outlier columns chosen by a rule that needs no data (Section 3.5).

The S-Quant and AWQ numbers are published results from other implementations and are listed for context. As Appendix B.5 shows, the 0.2 bits that S-Quant appears to save are spent elsewhere: its 3.8 bits do not count the number of coefficients of each block, which its decoder must read. Stored with a fixed number of bits, this count adds 0.19 bits per weight, so its actual rate is about 3.99 bits, essentially that of our Mode 2. AWQ is an integer format calibrated on data, while SenSeed is a seed-based codec that uses no data, so the two target different settings. Even so, Mode 2 reaches a lower perplexity than AWQ on all three Llama models.

Mode 1 is defined as the lowest rate at which SenSeed matches SeedLM's perplexity, so its accuracy is meant to match SeedLM's, and it does, within about one point. It saves 2.5% to 6% of the bits (0.40 GB on Llama-2-13B) without storing any metadata or using calibration data. While checking these numbers, we found an error in the submitted Table 1: its Mode 1 row repeated SeedLM's perplexities, and for Llama-3-8B and Mistral-7B it listed interpolated rates (3.88 and 3.78) instead of measured ones. The measured Mode 1 configurations use 3.77, 3.75, 3.90 and 3.85 bits per weight and reach perplexities of 5.84, 5.17, 7.11 and 6.17. Table 2 already reported the accuracies of these configurations. The saving is therefore 2.5% to 6%, not 3% to 6%.

Changes in the paper: Table 1 now gives the measured Mode 1 rates and perplexities, and the Mode 2 perplexity of Llama-2-7B is corrected from 5.68 to 5.69 (the 42% in Section 4.1 was already computed from the measured value and does not change). "3% to 6%" is now "2.5% to 6%" throughout the paper.

C2. SenSeed is a standard sensitivity-aware mixed-precision allocation applied to SeedLM.

Response. We agree that the allocation rule itself is classical, and we cite it as such (Shoham and Gersho; Everett). What is new is making such an allocation worthwhile for individual blocks of 8 weights in a seed-based codec. The obstacle is that the decoder must know each block's configuration before it can read the block. Storing this choice with a fixed number of bits costs ⌈log⁡2J⌉\lceil\log_2 J\rceil bits per block for JJ configurations, which is 0.25 to 0.375 bits per weight in our setting (Section 3.1). That is as much as or more than the entire saving of Mode 1 (0.10 to 0.25 bits per weight). This is why SeedLM uses one configuration for the whole model, and why S-Quant's per-block choice of kk is not free. Our contribution is to design the allocation so that the decoder can recompute it from information it already holds, without any per-block entry, together with a sensitivity estimate that needs no calibration data and meets this requirement. As a result, the allocation costs no stored bits per block. The only extra storage is a few small tables, one number per row or column of the attention output and MLP down projections plus a short global list, which take at most 0.005 bits per weight (Appendix A.7).

The experiment requested by Reviewer UvpM (Table R1 in our response to that review, Llama-2-13B) measures this directly. When the map is paid for out of the same 4.0 bits, a map with a fixed number of bits per block makes the result worse than uniform SeedLM (0.0561 against 0.0398 in loss increase over FP16), and so does an ideally compressed map once it is sized for the allocation it describes (0.0445). SenSeed needs neither the map nor an entropy decoder.

Changes in the paper: Table R1 is added to the paper as Table 8 (Appendix B.6), with a summary in Section 4.1.

AuthorsOpenReview ↗

Response to Reviewer Kiv1 (2/2)

C3. How would SenSeed run on GPUs, and how fast is it compared with 4-bit baselines?

Response. We have not built a GPU kernel and do not claim a GPU speedup. Like SeedLM (evaluated on an FPGA) and S-Quant (on an ASIC), SenSeed targets hardware where reading weights from memory is the bottleneck and generating them with an LFSR is cheap. Variable rates do not force sequential decoding. The size of every block depends only on the RMSNorm gains and the small side tables, so the decoder can compute where each block starts (a prefix sum) before reading any weight data, and then decode all blocks in parallel. On a GPU, the random basis for each seed length could be read from a precomputed table, as codebook-based kernels do. Compared with a 4-bit integer format at the same rate, the memory traffic is the same, so in the memory-bound regime we expect no speed advantage at Mode 2 and 2.5% to 6% less traffic at Mode 1. The decoding work per weight is the same as SeedLM's (on average 3.00 to 3.13 coefficients per block at 4.0 bits, Appendix A.8).

Changes in the paper: This discussion is added to Section 4.2.

C4. A single fault corrupts far more weights in SenSeed than in SeedLM (Section 4.3). Why is this larger fault footprint presented as protection rather than fragility?

Response. The proposed weight representation approach obscures the correspondence between stored bits and reconstructed weights, potentially making targeted bit-flip attacks more difficult. While random bit-flips are possible in SenSeed, it makes it easily detectable by amplifying the fault effect, as established in the paper. We agree that the same amplification may increase fragility under noise-induced bit-flips. However, common practice to achieve robustness against environmental noise is to use ECC, which is often inadequate for attack-induced bit-flips. We note that SenSeed can be combined with ECC to simultaneously provide protection against both noise and attack-induced bit-flips. Without detection and recovery, fault amplification increases the impact of corruption and could be viewed as increased fragility. Our original language did not make this distinction sufficiently clear, and we have revised it accordingly.

Section 4.3 does not claim intrinsic bit-flip resistance, but characterizes a distinct fault-propagation property of generated-weight representations. In SenSeed, corruption of shared schedule-determining state can propagate to many reconstructed weights, producing a larger model-level deviation than a localized perturbation. This amplification can increase the observability of corruption to an external integrity monitor.

As discussed in our response to Reviewer UvpM (A1), SenSeed is complementary to ECC or cryptographic integrity checks rather than a replacement for them. In particular, the high fan-out of a small set of shared decoding quantities suggests that strong protection could be concentrated on this critical state instead of uniformly protecting the entire compressed payload, potentially reducing the redundancy and hardware cost of full-payload protection.

Changes in the paper: We have accordingly revised the manuscript to narrow down the claims of intrinsic protection or robustness.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer Kiv1,

We have carefully addressed all the constructive points and concerns you raised in your initial review in our responses above, and we have reflected the requested changes in the revised manuscript (marked in red). Since the discussion period ends on November 18, we respectfully ask whether our responses have addressed your concerns. If you still have questions or suggestions, please let us know and we will do our best to address them before the deadline. Your comments are highly valuable to us. Thank you once again for your time and insightful comments.

Regards, Authors

Reviewer_BXbPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SenSeed-LLM, a sensitivity-aware seed-based weight quantization framework for large language models. The method introduces data-free sensitivity estimation and non-uniform bit allocation to assign different representation budgets to weight blocks. It further develops a decoder-reproducible allocation strategy to avoid per-block metadata overhead. Experiments demonstrate improvements in perplexity and storage efficiency compared with SeedLM. The paper also presents hardware implementation results and investigates the impact of bit-flip attacks.

Critical strengths
  1. The paper addresses an important problem in efficient LLM inference by improving the storage efficiency of seed-based weight compression.
  2. The sensitivity-aware allocation strategy is reasonable and provides a useful extension to existing seed-based quantization methods.
Critical weaknesses
  1. The experiments mainly focus on relatively small and older models, including Llama-2-7B/13B, Llama-3-8B, and Mistral-7B. The authors should evaluate more recent model families and larger-scale LLMs, such as 32B or 70B models, to demonstrate the scalability and generalizability of the proposed method.
Questions for authors
  1. There is a substantial amount of unnecessary whitespace around Table 3 on page 7, which negatively affects the overall presentation. The authors should improve the table placement and page layout.
  2. Although Table 3 reports area, power, and delay for the weight generators, the paper lacks comprehensive end-to-end inference latency and energy evaluations. Such experiments would better demonstrate the practical hardware benefits of SenSeed.
  3. The bit-flip experiments rely partly on simulated fault propagation rather than actual corrupted-bitstream decoding. Moreover, the paper does not implement an explicit detection or recovery mechanism. Additional experiments are needed to substantiate the claimed security benefits.
2 comments
AuthorsOpenReview ↗

Response to Reviewer BXbP

We thank the reviewer for the review and the concrete suggestions.

B1. The evaluated models are older and at most 13B; a larger model (for example 32B or 70B) would strengthen the evaluation.

Response. We agree that a larger model would strengthen the evaluation. We started Llama-3-70B, but it could not be completed within the discussion period. Most of the cost is in compression: like SeedLM, SenSeed tries all 2S2^S seeds for every block of 8 weights, once for each configuration it uses. On Llama-3-8B, the configuration with 16-bit seeds alone took 6.6 hours on four GPUs (Appendix A.8). Llama-3-70B is roughly an order of magnitude larger, and based on the observed runtime we estimate that the three required configurations would take approximately 100 hours on four GPUs and about 0.7 TB of storage. Based on the runtime of the first 400 of 560 weight matrices, the 16-bit-seed configuration alone is projected to require approximately 68 hours. This one-time offline cost scales linearly with the number of weights, while the allocation and decoding procedures are independent of model size. We will add the Llama-3-70B results to the final version upon acceptance.

Changes in the paper: We have revised the conclusion to better reflect the demonstrated flexibility of our approach.

B2. Whitespace around Table 3 on page 7.

Response. We thank the reviewer. We have fixed the layout around Table 3.

Changes in the paper: Layout around Table 3 has been fixed.

B3. End-to-end latency and energy, beyond the generator comparison in Table 3.

Response. We thank the reviewer for this question. Table 3 compares only the three weight generators, under identical conditions (the same cell library, clock, output precision and design flow). As Appendix C notes, it does not include schedule generation, reading the compressed bitstream or moving data into the PE array, so it is not an end-to-end measurement. In memory-bound inference, such as token-by-token generation, end-to-end throughput and memory energy can be dominated by weight traffic. At 3.77 bits per weight, SenSeed reduces the ideal weight-payload traffic by 76.4% relative to FP16 (Section 4.2). SeedLM's FPGA study reports that its 4-bit design approaches the resulting 4×4\times speedup over FP16. SenSeed uses the same reconstruction step as SeedLM, with 3.00 to 3.13 coefficients per block on average at 4.0 bits. To compute its schedule, it adds one multiplication and at most one comparison per configuration for each block, using a small on-chip table (Appendix A.8). We therefore expect its end-to-end behavior to follow SeedLM's, with 2.5% to 6% less memory traffic at Mode 1.

Changes in the paper: Section 4.2 now states what Table 3 measures and what it leaves out.

B4. The bit-flip experiments are simulated, and no mechanism for detecting a corrupted checkpoint is proposed or evaluated.

Response. We thank the reviewer for the comment. The current experiment is intended to characterize fault propagation, not to demonstrate a deployed detector or physical bit-flip attack. We now make this scope explicit: the evaluation considers selected faults in one tensor, and the S-Quant/SenSeed propagation is simulated rather than generated by a corrupted packed decoder. Accordingly, we no longer claim that SenSeed itself prevents bit-flip attacks. Instead, we report the resulting fault footprint and model-level degradation as a potential signal that could be exploited by an external integrity monitor. We also clarify that validating such a defense requires an implemented detection/recovery mechanism, repeated stored-bit injections across layers and fields, and measurements of detection coverage, false positives, and overhead.

Changes in the paper: Section 4.3 and Appendix D have been revised accordingly.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer BXbP,

We have carefully addressed all the constructive points and concerns you raised in your initial review in our responses above, and we have reflected the requested changes in the revised manuscript (marked in red). Since the discussion period ends on November 18, we respectfully ask whether our responses have addressed your concerns. If you still have questions or suggestions, please let us know and we will do our best to address them before the deadline. Your comments are highly valuable to us. Thank you once again for your time and insightful comments.

Regards, Authors

Reviewer_UvpMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper builds on previous works in randomized compression, particularly SeedLM. It develops a data-free non-uniform weight allocation framework that improves on the results of SeedLM and other methods. In order to support the non-uniformity without additional parameters, they develop an online block-importance metric allowing the decoder to recover the format without additional information. For results they compare against several relevant baselines and show that SenSeed-LLM improves upon them in a variety of settings. Finally they add that SenSeed-LLM has a much higher sensitivity to bit flips, which they claim is a desirable property for detecting model corruption.

Critical strengths

The side-information free formulation of the weight allocation was impressive and felt like a real novelty. A good identification of an opportunity and implementation of a solution.

Critical weaknesses

The section on security and fault detection were built on a fundamentally interesting insight: that SenSeed-LLM is a lot more sensitive to bit corruption than any of the other compression methods. However, it did feel that this section was largely orthogonal to the rest of the work and it is unclear whether it is generally a desirable property as described or whether it should be seen as a potential pitfall.

Similarly, some ablation around side-information using methods would be appreciated to show the necessity of the data-free bit allocation scheme.

Questions for authors

Can you provide more motivation for practical settings in which the increased sensitivity of SenSeed-LLM to bit flips is desirable and where other counter-measures (e.g. redundant coding + hash check) are innapropriate.

3 comments
AuthorsOpenReview ↗

Response to Reviewer UvpM (1/2)

We thank the reviewer for the positive and insightful review.

A1. The security analysis is orthogonal to the compression contribution. Is the larger bit-flip sensitivity a desirable property, and why not use redundant coding with a hash check instead?

Response. We agree that fault amplification is not, by itself, a security mechanism. A larger fault footprint has two consequences: it can make corruption less stealthy at the model level, but in the absence of detection and recovery it can also cause greater service degradation. Our experiments establish the former only as a potential monitoring signal. They do not establish attack resistance. We also agree that redundancy, ECC, or cryptographic hashes are more direct mechanisms for protecting model integrity. SenSeed is not intended to replace them. A hash or authenticated reference can detect changes in the compressed representation independently of their model-level effect, while ECC can correct faults within its correction capability.

These mechanisms, however, incur additional redundancy and often significant hardware overhead. For example, standard BCH(255,239, t=2t=2) and BCH(255,223, t=4t=4) configurations introduce 6.69% and 14.35% redundancy, respectively [1]. Applied to an entire 4-bit 8B checkpoint, these correspond to approximately 0.27 GB and 0.57 GB of additional storage cost, respectively.

SenSeed’s representation automatically eliminates the precise information on weight bit locations, making targeted bit-flip attacks extremely difficult. It reduces the amount of additional protection logic needed by making a corruption observable, because a fault in a sensitive weight bit can naturally amplify into a large model-level deviation. In contrast, a conventional design may rely more heavily on explicit redundancy or full-payload ECC to expose or correct localized corruption. Hence, ECC can still be used selectively on critical shared state for correction, while SenSeed’s intrinsic fault amplification may reduce the need for protecting the entire compressed payload with stronger, higher-overhead ECC.

Changes in the paper: To address the reviewer's comment, we have revised the abstract, introduction, contribution statement, Section 4.3, and conclusion to clarify and narrow down the claims on bit-flip attack resistance. Section 4.3 now explicitly discusses ECC, cryptographic integrity checks, and runtime signatures as complementary defenses requiring separate evaluation.

[1] Y. M. Lin, H. C. Chang and C. Y. Lee, "Improved High Code-Rate Soft BCH Decoder Architectures With One Extra Error Compensation," in IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 21, no. 11, pp. 2160-2164, Nov. 2013, doi: 10.1109/TVLSI.2012.2227847.

AuthorsOpenReview ↗

Response to Reviewer UvpM (2/2)

A2. Ablation against side-information methods: how does the zero-side-information schedule compare with storing the allocation explicitly?

Response. We thank the reviewer for suggesting this test. We have run it on Llama-2-13B and added the results (Table R1).

In SenSeed, each block of 8 weights is coded with one of a few configurations (a seed length and a number of coefficients), and the decoder must know a block's configuration before it can read the block. SenSeed lets the decoder compute every block's configuration from information it already holds (the RMSNorm gains and a few small tables), so nothing is stored per block. The alternative is to store a map that records the configuration of every block. For a fair comparison, we keep the total size fixed at 4.0 bits per weight: the bits used by the map are taken from this budget, and the allocation is rebuilt with the bits that remain for the weights. We compare (a) uniform SeedLM, where all blocks share one configuration and no map is needed; (b) SenSeed, which needs no map; (c) a per-block map stored with a fixed number of bits (⌈log⁡2J⌉\lceil\log_2 J\rceil bits per block for JJ configurations); (d) the same map ideally compressed, whose size is the zero-order entropy of the observed configuration labels, providing an optimistic lower bound for an explicit map encoded without additional structural modeling. All variants keep the same four FP16 outlier columns; the results without them are in parentheses.

Table R1. Llama-2-13B: increase in loss over FP16 at a total of 4.0 bits per weight, that is, the log of the ratio between the WikiText-2 perplexity of the compressed model and that of the FP16 model (lower is better), with the four FP16 outlier columns (without them in parentheses). (16,3) is SeedLM's 4-bit configuration: 16-bit seeds and 3 coefficients per block. The map size is computed from the configuration counts of SenSeed's own allocation; in brackets is the size of the map that the rebuilt allocation actually needs. SenSeed stores no map, only small tables, once per model, that let the decoder recompute the allocation (one number per row or column of the attention output and MLP down projections, plus a short global list); as in the paper, these tables (at most 0.005 bits per weight) are not subtracted from the 4.0 bits.

VariantWeight bitsMap bits [needed]Loss increase
(a) Uniform SeedLM (16,3)4.00000.0398 (0.0583)
(b) SenSeed, no map4.00000.0359 (0.0543)
(c) Per-block map, fixed size3.7500.250 [0.250]0.0561 (0.0716)
(d) Per-block map, ideally compressed3.9680.032 [0.082]0.0372 (0.0532)
(e) Same, sized for its own allocation3.8700.131 [0.131]0.0445 (0.0601)

The size of a map depends on how many blocks use each configuration. We first compute it from SenSeed's own allocation at 4.0 bits, which favors the map: with fewer bits left for the weights, the rebuilt allocation moves more blocks to other configurations, so the map that actually describes it is larger (the numbers in brackets). Even with this favorable size, every map ends worse than SenSeed. With a fixed number of bits per block, the map takes 0.25 bits per weight and the loss increase rises to 0.0561, far above both SenSeed's 0.0359 and uniform SeedLM's 0.0398. SenSeed changes the configuration of only 4% of the blocks, so the ideally compressed map is small and comes close to SenSeed at the favorable size (0.0372 against 0.0359). Sized for the allocation it actually describes (last row), this map takes 0.13 bits per weight, leaves 3.87 bits for the weights and raises the loss increase to 0.0445 (0.0601), which is worse than uniform SeedLM's 0.0398 (0.0583). Storing a map therefore costs more than the allocation gains.

Changes in the paper: The comparison is added to Section 4.1, with Table R1 as Table 8 in Appendix B.6.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer UvpM,

We have carefully addressed all the constructive points and concerns you raised in your initial review in our responses above, and we have reflected the requested changes in the revised manuscript (marked in red). Since the discussion period ends on November 18, we respectfully ask whether our responses have addressed your concerns. If you still have questions or suggestions, please let us know and we will do our best to address them before the deadline. Your comments are highly valuable to us. Thank you once again for your time and insightful comments.

Regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of responses and revisions

Summary for the Area Chair

We thank the reviewers for their valuable and constructive feedback. We have carefully considered each comment. Here, we summarize their main concerns and our responses.

(i) Novelty and side information (Reviewers 1:UvpM and 3:Kiv1). As requested, we compared SenSeed with storing the allocation explicitly, as a map that records the configuration of every block (Table R1). On Llama-2-13B, with the same total of 4.0 bits per weight, every map we tried, whether stored with a fixed number of bits per block or ideally compressed, performs worse than SenSeed. Once the map is sized for the allocation it actually describes, the result is even worse than uniform SeedLM, so storing the map costs more than the allocation gains. The contribution is therefore an allocation that the decoder computes by itself, with no stored map, which is what makes per-block allocation worthwhile in a seed-based codec.

(ii) Size of the gains (Reviewer 3:Kiv1). At 4 bits, SeedLM is only 1.2 to 3.8 points of mean zero-shot accuracy below FP16. SenSeed closes 27% to 55% of this gap and removes 23% to 42% of SeedLM's perplexity degradation, measured in log-perplexity. The 0.2 bits that S-Quant appears to save are effectively consumed by storing the per-block coefficient count, which is not included in its reported rate. While verifying these numbers, we identified and corrected an error in the Mode 1 row of Table 1. Mode 1 saves 2.5% to 6% of the bits.

(iii) Scale and system cost (Reviewers 2:BXbP and 3:Kiv1). We estimate that compressing a 70B model with the three configurations required by SenSeed takes approximately 100 hours on four GPUs, which could not be completed within the discussion period. We revised the conclusion accordingly and will add the Llama-3-70B results to the final version upon acceptance. We now state that Table 3 measures the weight generators alone, and we discuss end-to-end behavior and GPU decoding without claiming a speedup.

(iv) Security (Reviewers 1:UvpM, 2:BXbP, and 3:Kiv1). We agree that our original language was not adequately clear on what the fault experiment establishes. Our experiments characterize fault amplification for the effective detection of bit-flip attacks. The proposed weight representation approach eliminates information on precise weight bit locations and hence makes targeted bit-flip attacks very hard. While random bit-flips are possible in SenSeed, they are easily detectable by amplifying the fault effect, as established in the paper. We agree with the reviewers that the same amplification may increase fragility under noise-induced bit-flips. However, common practice to achieve robustness against environmental noise is to use ECC, which is often inadequate for attack-induced bit-flips. We claim that SenSeed can be combined with ECC to simultaneously provide protection against both noise and attack-induced bit-flips. To account for the reviewers' valuable insight, we have narrowed the security claim throughout the paper, explicitly described the scope and limitations, and positioned cryptographic integrity checks, ECC, and runtime weight-signature mechanisms as complementary defenses that require separate evaluation.

We have appropriately edited the manuscript to address the comments of the reviewers. All changes to the paper are listed under each response (Changes in the paper) and are marked in red in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.