acceptodds
Under review as a conference paper at ICLR 2027

PICO: Lossless Compression Stacked on Pruning and Quantization for Efficient LLM Inference

Abstract

Large Language Models (LLMs) are memory-bound during inference, where memory access dominates latency and energy consumption, making the combination of pruning and quantization essential for efficient deployment. Unstructured pruning stays near dense accuracy up to about 50% sparsity without retraining, whereas structured pruning degrades sharply at such sparsity without fine-tuning, but storing the irregular positions of the remaining weights limits its memory savings. Previous XOR-based compression regularizes such sparse weights into fixed-size blocks, yet at this sparsity it generates excessive correction patches that offset its gains, and shift-register-based variants reduce patches at the cost of substantial hardware and encoding overhead. We propose PICO (Patch-minimized Inverted XOR COmpression), a lossless, training-free scheme that stacks on existing pruning and quantization pipelines. PICO combines a hardware- and encoding-efficient XOR map with grouped inverted-XOR encoding, which selects the best map per group and skips patch metadata for mismatch-free groups. PICO applies regardless of unstructured pruning criterion or quantization format while preserving the regular, fixed-size access patterns required for parallel hardware decoding. Across LLaMA2/3.1, Phi-4, and Qwen2.5/3, PICO delivers 11.9% additional memory reduction at 50% sparsity with the smallest XOR decoder, 31% less area and 30% less power than the strongest prior scheme, while encoding 5.9× faster, and its sequential variant PICO-S doubles the reduction of that scheme.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Compression, Lossless, Unstructured Pruning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XM9uOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PICO is a codec-like layer applied on top of pruned and quantized models, aiming to improve the memory and inference performance while maintaining lossless quality of the pruned and quantized model.

Critical strengths
  • Lossless compression applied on top of already pruned and quantized models is an underexplored area, yet with the growing local AI needs, an exploration here seems reasonable and important.
Critical weaknesses
  • Like any lossless compression work, the main question is "how big is the overhead?" This work does not answer that because there is no end-to-end latency report.
  • While the grouping with an inverter map idea sounds interesting, this means multiple vectors share group-level decoding metadata, which could create significant locality challenges.
  • This work is applied on top of a quantized model, but does not pay much attention to how the overhead would interact with the dequantization step.

Overall, there seem to be many challenges in making PICO capable of inference on GPUs with real-life efficiency gains, yet few are properly discussed, with no end-to-end efficiency evaluation. This alone feels disqualifying.

Further, this work seems to do a rough job on related work. Most of the compared baselines are old, and some are not even LLM-focused. I am not an expert on lossless LLM compression, but with a quick search, we already have EntroLLM https://arxiv.org/abs/2505.02380 and https://arxiv.org/abs/2606.15789 for lossless compression on quantized LLMs (despite the latter one being too close to ICLR), and there is a whole line of work on lossless compression of BF16 LLMs (NeuZip, DFloat11, ZipServ). It seems like none of them are discussed.

Questions for authors

na

6 comments
AuthorsOpenReview ↗

Response to Reviewer XM9u (1/2)

We sincerely thank the reviewer for the careful reading and the constructive comments. We have uploaded the revised paper with all changes marked in blue.

[W1]

We thank the reviewer for raising this central question. We replace the byte estimate in Sec. 5.3 with end-to-end decode latency from ONNXim (Ham et al., 2024), a cycle-level NPU simulator. ONNXim served as the simulation of NeuPIMs (Heo et al., ASPLOS 2024), and was extended into PyTorchSim (Yang et al., MICRO 2025), which is used for evaluating memory interface. Its core timing is validated against Gemmini RTL (0.23% error, as reported by its authors), the platform on which PEC integrated an XOR-S decoder, and it models DRAM cycle by cycle with Ramulator2 (Luo et al., 2023), the part that determines decode latency. Patch reads and bandwidth contention are therefore included. Each weight tile is fetched at its encoded size, counting the mask, map indices, group flags, and patch stream as in Eq. 2, and the decoder latency and patch stalls follow our verified RTL rather than assumptions.

We simulate Qwen3-14B at 50% sparsity with INT4 on a four-core NPU with Jetson Orin NX-class LPDDR5 (102.4 GB/s), where decoding is memory-bound. We measured the end-to-end performance. PICO runs 1.10× faster than INT4, including the LM head. Compared with the same simulation without the decoder, decoding and patch reads change the latency by less than 0.3%, so the gain follows the reduction in bytes read, close to the byte-based estimate in Fig. 7(a) (1.14×), which we will replace with this result in the final version.

Table-C1 also prices the full decoder hardware described in Appendix F. We synthesized complete decoders for all schemes, verified bit by bit, at the same throughput of about 128 INT4 weights per cycle and 1 GHz (Design Compiler, NanGate 45 nm). PICO has the smallest decoder, and when every decoder is given the same area, PICO-S and PICO are the two fastest schemes (1.16× and 1.10×).

A fixed-to-variable bitmap, mentioned in response to W5, stores fewer bytes but needs a 4.69× larger decoder to place a varying number of values per cycle, so at the area of PICO it becomes the bottleneck and falls to 0.68× of INT4.

Table C1. Memory, decoder cost, and speed of fixed-to-fixed and fixed-to-variable formats (Qwen3-14B, 50% sparsity, INT4). Decoder area and energy per weight are for complete decoders at about 128 INT4 weights per cycle and 1 GHz (Design Compiler, NanGate 45 nm), normalized to PICO. Speedup is end-to-end decode latency, including the LM head, on 102.4 GB/s LPDDR5 (Jetson Orin NX class), with every decoder scaled to the area of PICO.

MethodMemory sizeDecoder areaDecoder energyMax frequencySpeedup vs. INT4 (same area)
Dense INT4100%–––1.00×
Bitmap + compact75.00%4.695.340.99 GHz0.68×
XOR95.75%1.801.591.05 GHz1.02×
sXOR109.88%2.692.561.05 GHz0.91×
XOR-S90.14%1.211.231.05 GHz1.07×
PICO87.71%1.001.001.08 GHz1.10×
PICO-S81.55%1.201.201.06 GHz1.16×
AuthorsOpenReview ↗

Response to Reviewer XM9u (2/2)

[W2]

Thank you for raising this. It is now described in Appendix F. We store each group's header (group-level decoding metadata) right next to its encoded vectors in memory, so one read brings in both, and the shared metadata does not break locality. The header is small: 9 bits (an 8-bit map index and a 1-bit group flag) for a group of 8 vectors, about 1.1 bits per vector. Since every group record (header plus 8 encoded vectors) has the same size, its address is computed directly from the group index. The decoder keeps the header in a register and uses it for all 8 vectors of the group. All header and patch reads are included in our end-to-end simulation (response to W1).


[W3]

Thank you for raising this and giving us the chance to improve our paper. Dequantization belongs to the compute step: for example, NVIDIA Blackwell Tensor Cores natively process NVFP4 with its block scaling in hardware (https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference), and for weight-only INT4, it runs inside the GEMM kernel. This is separate from the memory path that PICO targets. PICO is lossless and restores the exact quantized codes before they reach the compute unit, so dequantization, including its scales, runs exactly as in the original pipeline. This is why PICO works across INT4, FP8, NVFP4, and MXFP4 without any change (Table 3). PICO also does not require quantization at all. It uses the don't-care bits left by pruning, so it applies to pruned-only models as well: on FP16 models pruned to 50% by AutoPrune, PICO reduces memory by 33.4% on Qwen3-14B and 32.9% on LLaMA-2-7B (Appendix E in revised form). Since other readers may have the same question, we have added this explanation to Sec. 2 (Lines 180- 183, in blue) of the revised paper.


[W4] End-to-end efficiency on GPUs

Thank you for this comment. We agree that the paper did not explain where PICO runs, and Fig. 1(a) made it look like a GPU method. We clarify this below and now provide an end-to-end evaluation in our response to W1.

Where PICO runs. PICO is a small hardware decoder placed on the memory path, between DRAM and the processing units. It is not a GPU software kernel. Its main target is edge NPUs, where memory bandwidth limits LLM inference the most. The decoder reads compressed weights from DRAM, restores the exact quantized weights, and passes them to the processing units, which work exactly as before.

What about GPUs. On today's GPUs without such a decoder, PICO would have to decode in software, which adds work to every weight load, so we do not claim gains there. However, GPUs can also place a decoder on the memory path: NVIDIA A100 already compresses data on its L2 cache path, and Ecco (ISCA 2025) adds a hardware decompressor for LLM data at the L2 cache of a GPU. PICO fits the same position, and its decoder is simple enough to keep up with DRAM bandwidth.


[W5] More related works

We thank the reviewer for suggesting related works and have added them to Sec. 3 of the revised paper Lines.197- 211 (marked in blue). Most of them (EntroLLM,NeuZip, DFloat11, Tan et al.) apply entropy coding such as Huffman or ANS, a fixed-to-variable format classified in Park et al., 2022 (XOR-S), while ZipServ compresses the exponents of dense BF16 weights rather than the pruned positions that PICO handles. Fixed-to-variable and fixed-to-fixed formats are a design choice for the decoder.

Fixed-to-variable formats reach higher ratios, but their output length varies, so the decoder needs large buffers and alignment logic, and decoding can become the bottleneck (ZipServ reports that DFloat11 runs at only 0.28–0.34× the speed of a dense GEMM). Fixed-to-fixed formats decode at a fixed rate with simple hardware but compress less. A bitmap decoder, which is a fixed-to-variable format for sparse weights that we implemented as a representative, shows this trade-off. It takes 4.69× the area and 5.34× the energy of PICO, and at the same decoder area it falls to 0.68× of INT4 (see response to W1, Table C1).

PICO is an in-depth study of the fixed-to-fixed side for LLM sparsity. The prior XOR-based schemes belong to this class, but none was designed for LLMs, and at 50% sparsity they gain little or even enlarge the model (Table 2). Within this class, PICO and PICO-S give the largest reductions, PICO with the smallest decoder, and the byte reduction carries over to latency almost fully, as PICO runs 1.10× faster than INT4 end to end with decoding stalls below 0.3% (see response to W1, Table C1).

AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer XM9u,

We are grateful for your thoughtful review, which pointed us to important aspects of deployment. We have added an end-to-end evaluation including all patch and metadata reads, compared PICO with variable-length compression formats, and discussed recent lossless LLM compression methods, with all changes marked in blue. We would appreciate hearing whether our responses address your concerns.

Reviewer_UZaKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PICO applies lossless compression after language model pruning and quantization. It uses XOR-based encoding and grouping to reduce the metadata stored with sparse weights. The paper reports additional memory savings across several model families and compares decoder cost and encoding speed with the main baseline.

Critical strengths
  1. PICO can be applied after pruning and quantization without further altering weight values, which supports its use with existing compression pipelines.
  2. The experiments include seven models, several sparsity levels, and different pruning and quantization methods.
Critical weaknesses
  1. The concluding statement in Section 5.1 is not fully consistent with Table 2. For Qwen2.5-7B at 60% sparsity, the reported memory reduction is 20.54% for PICO and 20.80% for XOR-S. Although the paragraph acknowledges this exception, it concludes that PICO matches or exceeds the strongest existing scheme at every operating point.
  2. Tables 3 and 7 report different average accuracies for the same stated configuration. For Qwen3-14B with Wanda, 50% sparsity and INT4 AWQ, the values are 65.65% and 65.88%, respectively.
Questions for authors

Please compare with more methods, i.e., Automatic Pruning Discovery for Large Language Models (ICML 2026)

5 comments
AuthorsOpenReview ↗

Response to Reviewer UZaK (1/2)

We sincerely thank the reviewer for the careful reading and the constructive comments. We respond to each point below. We have uploaded the revised paper with all changes marked in blue.

[W1]

Thank you for pointing this out. You are right that the text and Table 2 did not match. The error was a typo in Table 2: PICO's reduction for Qwen2.5-7B at 60% sparsity is 21.54%, not 20.54%. With the correct value, PICO is higher than XOR-S (20.80%) here as well, so PICO now exceeds XOR-S at every operating point. The typo happened while copying results from our server logs to a spreadsheet and then to LaTeX. We are sorry for this, as we know such errors can affect trust in our results. After your comment, we checked every number in the paper. In the revised paper, we have fixed Table 2, rewritten Sec. 5.1, and corrected the other errors we found.


[W2]

Thank you for catching this. You are right, and the error was in Appendix Table 7. When copying the results to LaTeX, we made typos in two task scores of this row: OBQA is 31.00 (not 31.60) and ARC-e is 80.81 (not 81.81). With the correct scores, the average is 65.65%, which matches Table 3. We have fixed Table 7 in the revised paper and also corrected a similar typo in the Wanda NVFP4 row (65.46%). We appreciate your careful check.

AuthorsOpenReview ↗

Response to Reviewer UZaK (2/2)

[Q1]

Thank you for suggesting AutoPrune. It is a recent training-free unstructured pruning method, and PICO stacks on it without any change. We pruned Qwen3-14B and LLaMA-2-7B with AutoPrune, quantized them with GPTQ or AWQ, and applied PICO. We also added SparseGPT to cover more pruning methods.

Table-B1: Zero-shot accuracy (avg. of 7 tasks) and memory reduction (%)

ModelPruningQuant.Acc.sXORXORXOR-SPICOPICO-S
Qwen3-14BDense–68.40–––––
RIA 50%GPTQ W464.97−9.884.259.8612.2918.45
Wanda 50%AWQ W465.65−9.755.3810.3512.2519.13
AutoPrune 50%– (FP16)66.3616.6329.3529.9033.4439.47
AutoPrune 50%GPTQ W465.54−9.755.4110.3612.7319.13
AutoPrune 50%AWQ W466.16−9.755.3710.3512.7319.13
AutoPrune 60%GPTQ W461.155.6118.3422.3923.0829.19
SparseGPT 50%GPTQ W466.17−9.934.279.8512.2618.42
LLaMA-2-7BDense–59.71–––––
RIA 50%GPTQ W454.85−10.142.329.0811.9218.15
Wanda 50%AWQ W454.88−9.765.4810.3712.7319.19
AutoPrune 50%– (FP16)55.9315.5928.5729.6632.8539.19
AutoPrune 50%GPTQ W455.13−9.785.4310.3512.7219.15
AutoPrune 50%AWQ W455.90−9.765.4210.3512.7219.15
AutoPrune 60%GPTQ W449.394.2518.7021.0022.3128.84
SparseGPT 50%GPTQ W454.58−10.98−2.497.0110.3516.20

On AutoPrune, PICO removes about 12.7% of memory at 50% sparsity, similar to RIA, Wanda, and SparseGPT, while AutoPrune keeps comparable or higher accuracy. This confirms that PICO's gain does not depend on the pruning criterion. In the revised paper, we have added these results as a new appendix (Appendix E) and cited AutoPrune.

Setup. All accuracy experiments run on one NVIDIA RTX PRO 6000 (96 GB) and report acc. Pruning and quantization use 128 C4 samples of 2,048 tokens (seed 0), and pruning is done before quantization. AutoPrune uses uniform sparsity without SDSA. All XOR-based encoding and decoding run on a dual-socket Intel Xeon Gold 6230R.

AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer UZaK,

Thank you again for your review. We have fixed the inconsistencies you pointed out and added AutoPrune results (Appendix E), with all changes marked in blue. We would be grateful to know whether our responses address your concerns.

Reviewer_MJAWOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PICO is a lossless, training-free scheme that compresses LLM weights after unstructured pruning and quantization. It builds on fixed-to-fixed XOR encoding: each NdN_{d}-bit bitplane vector is encoded into NcN_{c} bits; patch metadata corrects mismatched bits. PICO adds a constrained XOR map (E-XOR), inverters on the XOR outputs whose setting is shared by a group of vectors, and a group flag that drops patch flags for mismatch-free groups. PICO-S applies this encoding to a shift-register decoder. Storage is measured. Experiments cover seven LLMs at 40-60% sparsity, several pruners and number formats on Qwen3, 45 nm decoder synthesis, and encoding time.

Critical strengths

S1. A common accounting applied to prior XOR schemes at LLM scale. Under Eq. 2, which counts mask and patches, Table 2 measures three prior schemes on seven LLMs and reports their negative values (sXOR: -10.13% at 50%). Why this matters to my assessment: it shows prior XOR schemes do not transfer to the roughly 50% sparsity reachable without retraining.

S2. Each design step is ablated. Per-vector map indices raise memory size from 97.1% to 131.3%, grouping lowers it to 89.6%, the group flag to 88.0% (l.279-288). Losses are disclosed (l.365, l.474-475). Why this matters to my assessment: each component's effect is traceable.

Critical weaknesses

W1. The storage gains are below an omitted simple baseline, and several numbers cannot be reproduced from the stated configuration.

  • By Eq. 2 and l.258-260 (Nc=8N_{c}=8, Nd=16N_{d}=16), ∣Wenc∣=0.5 mnb|W_{enc}| = 0.5\,mnb at any sparsity, plus mask MM (Eq. 1). At 50% sparsity this equals a bitmap plus compact non-zero values, before PICO adds patches, indices and flags. If MM costs 1 bit per weight (its width is not stated), that baseline gives 25% reduction at b=4b=4, against 11.91% (PICO) and 18.05% (PICO-S) in Table 2. Only CSR is compared (Table 4). Sec. 3 values fixed-size offsets "rather than raw compression ratio" (l.188-189) without pricing them.
  • Under the same configuration no scheme can exceed 25% at b=4b=4, yet PICO-S at 60% reports 26.39-29.58% on all seven models (Table 2).
  • Table 3 is nearly constant across bit widths (W3 11.86/11.97, INT4 GPTQ 12.21/12.33, FP8 11.64/11.72), although the mask share scales with 1/b1/b. This needs unreported format-dependent patch overhead, so l.376-377 lacks support.

Why this matters to my assessment: memory reduction is the main claim, and I can neither verify it nor judge whether the XOR format is worth its storage cost.

W2. Lack of novelty, and the advantages that would justify the XOR format are asserted, not measured.

  • The components adapt mechanisms the paper credits to prior work: XOR fixed-to-fixed encoding (l.191-192), whole-vector flipping in XOR-S (l.196), a per-vector selection bit in sXOR (l.199-200), an end flag replacing patch flags (l.271). The authors note the grouped encoding and flag "work with any base map, not only E-XOR" (l.290-291). The claimed first is a conjunction ("simultaneously hardware-efficient, training-free, and fast to encode", l.104-105). The 50% gain over XOR-S averages 2.54 points (Table 2).
  • "keeping every access fixed-size and regular" (Fig. 1) is not shown: patches are stored only for mismatched groups (l.286-287), and l.136-138 locates vectors sequentially via the patch count. Locating a group's patches is neither described nor counted in Eq. 2.
  • The 4.5×/4.9× speedups over FP16 (l.101) equal 4/(1−r)4/(1-r), so about 4× comes from quantization; over INT4 they are 1.14×/1.22× (l.427-428). Both are byte estimates (l.353-354) that ignore patch reads, which l.81-82 blames for stalls in prior work.
  • Using the Gemmini figures (l.424-426), the decoder saving over XOR-S is about 0.12% of NPU area.

Why this matters to my assessment: without a storage win over a bitmap, the case rests on access regularity and decoder cost, neither demonstrated, for an incremental step within the XOR family.

Questions for authors
  1. What is the width of MM, and how does PICO compare with a bitmap plus compact non-zero values under Eq. 2? Please quantify what PICO buys in exchange.
  2. Which configuration produces each number in Tables 2 and 3? Please give NdN_{d}, NcN_{c}, group size, inverters, bb and per-component bits, and explain the 60% PICO-S values.
  3. How does the decoder locate an arbitrary group's patches? Please give the layout, its cost in Eq. 2, and whether the speedup estimate includes patch reads.
  4. Which design point, map and LoRA recipe are used? l.261 gives only ranges, and run-to-run variation is unreported while the 60% margin over XOR-S spans -0.26 to +1.96 points. Qwen3-14B rows in Table 3 match the ×2 LoRA rows of Table 7, not the recipe in l.414-415; with ×1, W3 falls to 62.41, below SpinQuant W3 (64.93).
10 comments
AuthorsOpenReview ↗

Response to Reviewer MJAW (1/4)

We sincerely thank the reviewer for reading our paper so carefully, down to checking the bounds by hand. These comments helped us look at our work again and make its claims clearer. We have uploaded the revised paper with all changes marked in blue, and added Appendices D–F with the details behind our answers.

[W1-2]

Thank you for checking the bound so carefully. The bound holds if Nd is fixed at 16, but in Lines 258–260 we intended Nd to follow the sparsity, as in prior XOR schemes (Kwon et al., 2020; Park et al., 2021): Nd = ⌊Nc / (1 − S)⌋ with Nc = 8, giving Nd = 13, 16, and 20 at 40%, 50%, and 60%. With a 1-bit mask per weight, the INT4 limits are:

SparsityNdWencMaskLimitPICOPICO-S
40%1361.5%25%13.5%3.45%7.02%
50%1650.0%25%25.0%11.91%18.05%
60%2040.0%25%35.0%22.78%28.33%

All values are below their limits. We have stated Nd for each sparsity in the revised paper so others do not misread it (Line 249-250). We truly appreciate this comment, which helps make the paper clearer.


[W1-3]

Thank you for catching this. You are right. The accuracy in Table 3 was measured on the correct 3-bit and 8-bit models, but our export script for PICO inputs was shared with the 4-bit pipeline, so the W3 and FP8 files were saved as 4-bit indices. These rows were therefore computed as 4-bit weights. The 4-bit rows (INT4, NVFP4, MXFP4) and Table 2 are not affected. The corrected values are shown below. We apologize for the confusion this caused during your review. As you noted, the gain now grows with the bit width as the 1-bit mask takes a smaller share. We have corrected Table 3 and rewritten Sec. 5.2 accordingly in the revised paper.

ModelSparsityPruningQuantizationsXORXORXOR-SPICOPICO-S
Qwen3-14B50%RIAFP8 (E4M3)2.1816.6022.9426.1130.95
50%WandaFP8 (E4M3)2.3117.7823.3526.4631.63
50%ALPSFP8 (E4M3)2.1916.8923.0526.2231.10
50%ALPSSpinQuant W3−18.26−3.741.664.0610.31
Qwen3-32B50%RIAFP8 (E4M3)2.2016.9223.0526.2231.11
50%WandaFP8 (E4M3)2.3217.9223.3526.4631.66
50%ALPSFP8 (E4M3)2.2217.0723.1226.2931.23
50%ALPSSpinQuant W3−18.10−3.541.734.1610.33
AuthorsOpenReview ↗

Response to Reviewer MJAW (2/4)

[W1-1]

We thank the reviewer for the suggestion. The mask M costs 1 bit per weight, and bitmap plus compact non-zero values is indeed the natural reference.

The bitmap, however, is a fixed-to-variable format in the classification of Park et al. (2022). Fixed-to-fixed and fixed-to-variable formats are a design choice for the decoder, each with its own strengths. A fixed-to-fixed decoder expands each block directly with simple hardware. A fixed-to-variable format usually compresses more, but its decoded output is unbalanced, so its decoder needs substantial buffering and alignment logic, such as the prefix popcount and crossbar of the bitmap that place a varying number of values in each cycle. Neither is the single right answer for an accelerator, and PICO is an in-depth study of the fixed-to-fixed side. Its only variable-length part is a small patch stream (5.68% on Qwen3-14B at 50%) that flips a few bits, whereas the bitmap stores every non-zero weight in its variable-length part. To quantify what this difference costs in memory, decoder hardware, and speed, we compare all schemes in Table-A1.

In Table-A1, we measure end-to-end decode latency with a cycle-level simulation (see our response to Reviewer XM9u for why we chose this simulator), which models every patch read and patch correction stall, and PICO still runs 1.10× faster than INT4. Compared with the same simulation without the decoder, decoding and patch reads change the latency by less than 0.3% (1.10× in both cases), so the gain follows the reduction in bytes read. Since Fig. 7 measured only the decoding datapath, we synthesized complete decoders (Appendix F, Fig. 9) in Table A1, adding the input and mask registers, patch control, and the group header register of PICO. We verified each bit by bit on Qwen3-14B weights and matched their throughput at about 128 INT4 weights per cycle at 1 GHz (Design Compiler, NanGate 45 nm). PICO remains the smallest, but against XOR-S the saving becomes 17% (vs. 31% in Fig. 7, which measured only the datapath), and the speedup over INT4 becomes 1.10× (vs. the 1.14× byte estimate). We will update Fig. 7, the abstract, and Sec. 5.3 accordingly in the final version.

We agree that Sec. 3 did not price this advantage, and Table A1 now does, including a bitmap decoder (fixed-to-variable). The bitmap stores fewer bytes (75.00%) but pays for it in the decoder: it takes 4.69× the area and 5.34× the energy of PICO and still falls short of 1 GHz (0.99 GHz) with three extra pipeline stages. Since one decoder is replicated per NPU core to match DRAM bandwidth (four cores in our simulated NPU), this gap grows with the number of cores. At the same decoder area as PICO, the bitmap falls to 0.68× of INT4, while PICO and PICO-S keep 1.10× and 1.16×. Among fixed-to-fixed formats, PICO and PICO-S give the largest reductions (Table 2), and PICO has the smallest decoder, 17% below XOR-S (1 − 1/1.21). PICO therefore targets area- and power-constrained edge accelerators, where a fixed-rate decoder with minimal area is preferred.

Table-A1. Memory, decoder cost, and speed of fixed-to-fixed and fixed-to-variable formats on Qwen3-14B at 50% sparsity with INT4. For the bitmap, it excludes offset tables. Decoder area and energy per weight are for complete core decoders sustaining about 128 INT4 weights per cycle (Design Compiler, NanGate 45 nm, 1 GHz), normalized to PICO. Speedup is end-to-end decode latency including the FP16 LM head, on LPDDR5 at 102.4 GB/s (Jetson Orin NX class).

MethodMemory sizeVariable-length partDecoder areaDecoder energyMax frequencySpeedup vs. INT4 (same area)
Dense INT4100%––––1.00×
Bitmap + compact75.00%values (50.00%)4.695.340.99 GHz0.68×
XOR95.75%patches (20.75%)1.801.591.05 GHz1.02×
sXOR109.88%patches (26.60%)2.692.561.05 GHz0.91×
XOR-S90.14%patches (8.89%)1.211.231.05 GHz1.07×
PICO87.71%patches (5.68%)1.001.001.08 GHz1.10×
PICO-S81.55%patches (5.02%)1.201.201.06 GHz1.16×
AuthorsOpenReview ↗

Response to Reviewer MJAW (3/4)

[W2-1]

We respectfully clarify our contribution.

[XOR-S] PICO's inverters differ from the flipping in XOR-S. XOR-S flips every bit of a weight vector that has more ones than zeros, so the vector has more zeros, which the XOR decoder can match more easily. The decoder then needs an extra step to flip the output back. PICO's inverters are instead part of the map itself: they invert selected decoder outputs to turn one E-XOR map into 2^N different maps, and the encoder picks the map with the fewest mismatches for each group, so the decoder output is already the original data. In this way, one small inverter stage replaces both the shift register and the flip-back step of XOR-S. Both store the same amount of metadata for this choice: XOR-S uses a 1-bit flip flag per vector, and PICO uses an 8-bit map index shared by a group of 8 vectors. This gives PICO two advantages over XOR-S. First, with the same metadata cost and without a shift register, PICO achieves higher memory reduction (12.29% vs. 9.86% on Qwen3-14B at 50%). Second, it does not depend on the ratio of zeros and ones: XOR-S notes that its flipping has no effect on signed INT8 weights (Park et al., 2021, Table 2), while PICO's map selection works for any number format.

[sXOR] sXOR's per-vector selection bit was effective only on small pre-LLM models at 60–90% sparsity, where many don't-care bits absorb its cost. At the 50% sparsity LLMs reach without retraining, this cost makes the output larger than its input: sXOR needs 110% of the original memory on average over seven LLMs (Table 2). Our key finding is that such a choice only pays off when the index is shared by a group: a per-vector index raises the memory size from 97.1% to 131.3%, while sharing one index across a group of 8 vectors lowers it to 88.0% (Fig. 6).

[End flag] XOR-S's patch flag still marks every vector, while PICO's group flag skips whole groups with no mismatch.

[l.290–291] We see this as a strength. Because the grouped encoding works with any map, it improves both decoder types keeping xor count small: PICO on a simple decoder, and PICO-S on a shift-register decoder, which reaches 18.05% at 50%, about twice XOR-S.

[l.104–105] We agree and will state the claim more simply: PICO is the first lossless XOR-based scheme that still reduces memory at 40% sparsity, where every prior XOR scheme makes the model larger (Table 2).


[W2-2]

The fixed-size claim in Fig. 1 refers to the encoded stream, which carries 93.5% of the compressed bits on Qwen3-14B at 50% (encoded weights, mask, map indices, and group flags, 82.03% of 87.71%). The per-vector patch count in Sec. 2 is used by prior schemes, not PICO. We revised the paper as pointing to Appendix F (Fig. 9), which shows the layout of PICO. Its patches are kept in a separate stream, so locating a slice does not depend on them. How the decoder locates a group’s patches and its cost in Eq. 2 are given in our response to Q3.


[W2-3]

We agree that most of the speedup over FP16 comes from quantization. This is exactly the setting PICO targets, since it stacks on top of quantization and gains more after INT4 has already taken its share. PICO also helps where a low-bit format is not supported by the processing unit, because it encodes the bit planes of any format, so pruning followed by PICO still reduces weight traffic, as shown for FP8 in Table 3. We now report the additional gain directly, using end-to-end decode latency from a cycle-level simulation that includes every patch read and the decoder stalls they cause. Please refer to details in our response to W1-1.


[W2-4]

We agree that the saving over XOR-S is small relative to a Gemmini-class NPU, and we no longer present it as a main contribution. With complete decoders (Table A1), the PICO decoder is 17% smaller than XOR-S (1 − 1/1.21) rather than 31% as in Fig. 7 (datapath implemented), and we will revise the abstract and Sec. 5.3 accordingly in the final version. The comparison instead shows that PICO gives the largest reduction among fixed-to-fixed formats with the smallest decoder, while sXOR and the bitmap need 2.69× and 4.69× larger decoders. Since one decoder is replicated per NPU core to match DRAM bandwidth (four cores in our simulated NPU), this gap grows with the number of cores and the bandwidth. We discuss the trade-off against the bitmap in our response to W1-1.

Comments on the paper

4 comments

Dear Reviewer,

I hope this message finds you well. I am writing to kindly remind you that the authors’ rebuttal has been posted and is now available for your review in the system.

We would greatly appreciate it if you could review the rebuttal and provide your assessment at your earliest convenience, as your input is important for us to move forward with a timely decision.

Thank you very much for your time and continued support.

ACs

AuthorsOpenReview ↗

General Response: Summary of Revisions

We sincerely thank all reviewers and the AC for their time and thoughtful feedback, which has helped us improve the paper considerably. We have uploaded a revised paper with all changes marked in blue, keeping the main text within the 10-page limit. Additional experimental results and details needed to support our responses have been added to the appendices (Appendices D–F). The main updates are summarized below.

  1. End-to-end evaluation (XM9u, MJAW): cycle-level simulation (ONNXim) of Qwen3-14B on an edge NPU, including every mask, group metadata, and patch read and the patch handling in the decoder. PICO runs 1.10× faster than INT4, and its decoding and patch reads add less than 0.3% to the latency.
  2. Decoder and baselines (MJAW, XM9u): complete decoders synthesized for all schemes, together with a bitmap baseline that represents the fixed-to-variable formats raised by the reviewers (Table A1/C1). PICO has the smallest decoder, and at equal decoder area, PICO-S and PICO are the fastest.
  3. Corrected numbers (MJAW, UZaK): Table 2 typo (PICO now exceeds XOR-S at every point), W3/FP8 rows of Table 3, and Table 7.
  4. Clarified settings (MJAW): N_d per sparsity (Appendix D) and the decoder and DRAM layout (Appendix F).
  5. More methods and related work (UZaK, XM9u): AutoPrune and SparseGPT (Appendix E), and lossless LLM compression methods (Sec. 3).

As only 24 hours remain in the author rebuttal period, I sincerely invite the reviewers to review the author's response and promptly raise any remaining questions or indicate whether you believe the author has addressed the issues you have raised.

AuthorsOpenReview ↗

General Response: Summary of Revisions #2

We are grateful to all reviewers and the AC for engaging in this discussion despite their busy schedules. Following the latest feedback, we have uploaded a second revision, with changes in this round marked in purple (changes from the first revision remain in blue). The main text stays within the 10-page limit. The main updates are summarized below.

  1. End-to-end results in the main paper (XM9u, MJAW): Fig. 7 and Sec. 5.3 now report end-to-end decoding from cycle-level simulation in place of the earlier byte-based estimate, and the abstract and introduction state these measured results.
  2. Complete decoder costs (MJAW, XM9u): the decoder comparison now uses complete decoders for all schemes, including a bitmap decoder, synthesized at the same throughput, replacing the earlier datapath-only figures.
  3. Workload coverage and settings (XM9u): the new Appendix G covers three models, 40–60% sparsity, INT4 and FP16, batch 1 and 4, prompts of 128–2048 tokens, and the memories of the Jetson Orin NX and AGX Orin, and it states all simulation and workload settings.
  4. Clearer framing (XM9u): Fig. 1(a) has been redrawn to show where PICO runs, from DRAM through the PICO decoder to the processing units.
  5. Related work (XM9u): Sec. 3 has been rewritten to cover variable-length codecs and sparse-weight formats with decompression hardware, and to state where PICO stands among them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.