- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
PICO is a codec-like layer applied on top of pruned and quantized models, aiming to improve the memory and inference performance while maintaining lossless quality of the pruned and quantized model.
Critical strengths
- Lossless compression applied on top of already pruned and quantized models is an underexplored area, yet with the growing local AI needs, an exploration here seems reasonable and important.
Critical weaknesses
- Like any lossless compression work, the main question is "how big is the overhead?" This work does not answer that because there is no end-to-end latency report.
- While the grouping with an inverter map idea sounds interesting, this means multiple vectors share group-level decoding metadata, which could create significant locality challenges.
- This work is applied on top of a quantized model, but does not pay much attention to how the overhead would interact with the dequantization step.
Overall, there seem to be many challenges in making PICO capable of inference on GPUs with real-life efficiency gains, yet few are properly discussed, with no end-to-end efficiency evaluation. This alone feels disqualifying.
Further, this work seems to do a rough job on related work. Most of the compared baselines are old, and some are not even LLM-focused. I am not an expert on lossless LLM compression, but with a quick search, we already have EntroLLM https://arxiv.org/abs/2505.02380 and https://arxiv.org/abs/2606.15789 for lossless compression on quantized LLMs (despite the latter one being too close to ICLR), and there is a whole line of work on lossless compression of BF16 LLMs (NeuZip, DFloat11, ZipServ). It seems like none of them are discussed.
Questions for authors
na
6 comments
Response to Reviewer XM9u (2/2)
[W2]
Thank you for raising this. It is now described in Appendix F. We store each group's header (group-level decoding metadata) right next to its encoded vectors in memory, so one read brings in both, and the shared metadata does not break locality. The header is small: 9 bits (an 8-bit map index and a 1-bit group flag) for a group of 8 vectors, about 1.1 bits per vector. Since every group record (header plus 8 encoded vectors) has the same size, its address is computed directly from the group index. The decoder keeps the header in a register and uses it for all 8 vectors of the group. All header and patch reads are included in our end-to-end simulation (response to W1).
[W3]
Thank you for raising this and giving us the chance to improve our paper. Dequantization belongs to the compute step: for example, NVIDIA Blackwell Tensor Cores natively process NVFP4 with its block scaling in hardware (https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference), and for weight-only INT4, it runs inside the GEMM kernel. This is separate from the memory path that PICO targets. PICO is lossless and restores the exact quantized codes before they reach the compute unit, so dequantization, including its scales, runs exactly as in the original pipeline. This is why PICO works across INT4, FP8, NVFP4, and MXFP4 without any change (Table 3). PICO also does not require quantization at all. It uses the don't-care bits left by pruning, so it applies to pruned-only models as well: on FP16 models pruned to 50% by AutoPrune, PICO reduces memory by 33.4% on Qwen3-14B and 32.9% on LLaMA-2-7B (Appendix E in revised form). Since other readers may have the same question, we have added this explanation to Sec. 2 (Lines 180- 183, in blue) of the revised paper.
[W4] End-to-end efficiency on GPUs
Thank you for this comment. We agree that the paper did not explain where PICO runs, and Fig. 1(a) made it look like a GPU method. We clarify this below and now provide an end-to-end evaluation in our response to W1.
Where PICO runs. PICO is a small hardware decoder placed on the memory path, between DRAM and the processing units. It is not a GPU software kernel. Its main target is edge NPUs, where memory bandwidth limits LLM inference the most. The decoder reads compressed weights from DRAM, restores the exact quantized weights, and passes them to the processing units, which work exactly as before.
What about GPUs. On today's GPUs without such a decoder, PICO would have to decode in software, which adds work to every weight load, so we do not claim gains there. However, GPUs can also place a decoder on the memory path: NVIDIA A100 already compresses data on its L2 cache path, and Ecco (ISCA 2025) adds a hardware decompressor for LLM data at the L2 cache of a GPU. PICO fits the same position, and its decoder is simple enough to keep up with DRAM bandwidth.
[W5] More related works
We thank the reviewer for suggesting related works and have added them to Sec. 3 of the revised paper Lines.197- 211 (marked in blue). Most of them (EntroLLM,NeuZip, DFloat11, Tan et al.) apply entropy coding such as Huffman or ANS, a fixed-to-variable format classified in Park et al., 2022 (XOR-S), while ZipServ compresses the exponents of dense BF16 weights rather than the pruned positions that PICO handles. Fixed-to-variable and fixed-to-fixed formats are a design choice for the decoder.
Fixed-to-variable formats reach higher ratios, but their output length varies, so the decoder needs large buffers and alignment logic, and decoding can become the bottleneck (ZipServ reports that DFloat11 runs at only 0.28–0.34× the speed of a dense GEMM). Fixed-to-fixed formats decode at a fixed rate with simple hardware but compress less. A bitmap decoder, which is a fixed-to-variable format for sparse weights that we implemented as a representative, shows this trade-off. It takes 4.69× the area and 5.34× the energy of PICO, and at the same decoder area it falls to 0.68× of INT4 (see response to W1, Table C1).
PICO is an in-depth study of the fixed-to-fixed side for LLM sparsity. The prior XOR-based schemes belong to this class, but none was designed for LLMs, and at 50% sparsity they gain little or even enlarge the model (Table 2). Within this class, PICO and PICO-S give the largest reductions, PICO with the smallest decoder, and the byte reduction carries over to latency almost fully, as PICO runs 1.10× faster than INT4 end to end with decoding stalls below 0.3% (see response to W1, Table C1).
Follow-Up to Your Review
Dear Reviewer XM9u,
We are grateful for your thoughtful review, which pointed us to important aspects of deployment. We have added an end-to-end evaluation including all patch and metadata reads, compared PICO with variable-length compression formats, and discussed recent lossless LLM compression methods, with all changes marked in blue. We would appreciate hearing whether our responses address your concerns.
Response to Reviewer XM9u (1/2)
We sincerely thank the reviewer for the careful reading and the constructive comments. We have uploaded the revised paper with all changes marked in blue.
[W1]
We thank the reviewer for raising this central question. We replace the byte estimate in Sec. 5.3 with end-to-end decode latency from ONNXim (Ham et al., 2024), a cycle-level NPU simulator. ONNXim served as the simulation of NeuPIMs (Heo et al., ASPLOS 2024), and was extended into PyTorchSim (Yang et al., MICRO 2025), which is used for evaluating memory interface. Its core timing is validated against Gemmini RTL (0.23% error, as reported by its authors), the platform on which PEC integrated an XOR-S decoder, and it models DRAM cycle by cycle with Ramulator2 (Luo et al., 2023), the part that determines decode latency. Patch reads and bandwidth contention are therefore included. Each weight tile is fetched at its encoded size, counting the mask, map indices, group flags, and patch stream as in Eq. 2, and the decoder latency and patch stalls follow our verified RTL rather than assumptions.
We simulate Qwen3-14B at 50% sparsity with INT4 on a four-core NPU with Jetson Orin NX-class LPDDR5 (102.4 GB/s), where decoding is memory-bound. We measured the end-to-end performance. PICO runs 1.10× faster than INT4, including the LM head. Compared with the same simulation without the decoder, decoding and patch reads change the latency by less than 0.3%, so the gain follows the reduction in bytes read, close to the byte-based estimate in Fig. 7(a) (1.14×), which we will replace with this result in the final version.
Table-C1 also prices the full decoder hardware described in Appendix F. We synthesized complete decoders for all schemes, verified bit by bit, at the same throughput of about 128 INT4 weights per cycle and 1 GHz (Design Compiler, NanGate 45 nm). PICO has the smallest decoder, and when every decoder is given the same area, PICO-S and PICO are the two fastest schemes (1.16× and 1.10×).
A fixed-to-variable bitmap, mentioned in response to W5, stores fewer bytes but needs a 4.69× larger decoder to place a varying number of values per cycle, so at the area of PICO it becomes the bottleneck and falls to 0.68× of INT4.
Table C1. Memory, decoder cost, and speed of fixed-to-fixed and fixed-to-variable formats (Qwen3-14B, 50% sparsity, INT4). Decoder area and energy per weight are for complete decoders at about 128 INT4 weights per cycle and 1 GHz (Design Compiler, NanGate 45 nm), normalized to PICO. Speedup is end-to-end decode latency, including the LM head, on 102.4 GB/s LPDDR5 (Jetson Orin NX class), with every decoder scaled to the area of PICO.