- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how to exploit the non-uniqueness of neural network parameterizations to reduce the physical storage size of losslessly compressed model checkpoints while preserving model functionality or maintaining functional distortion within acceptable limits. The authors formulate this problem as minimizing physical compression cost over functional equivalence classes and constrained functional neighborhoods, and propose a Proposal–Audit–Physical Selection framework. The method generates candidate weight representations through exact function-preserving transformations, such as hidden-channel permutations and invertible rescalings, as well as significand rounding and activation-aware sparsification. Candidate representations are then selected through functional admissibility checks and evaluations using the actual compressor, without modifying the compression backend. Experiments are conducted on SmolLM2-135M, SmolLM2-360M, and Qwen3-0.6B, with evaluations using Zstandard, gzip, Brotli, and ZipNN. The results show that exact function-preserving transformations provide relatively limited compression gains, whereas allowing controlled functional distortion leads to substantially larger reductions. On SmolLM2-360M, the method achieves up to a 23.41% reduction in Zstandard-compressed checkpoint size, with gains that also transfer to other compression backends. The paper demonstrates that representational freedom in neural networks can serve as an additional resource for storage optimization, although the achievable gains depend on the specific transformations, model architectures, and permitted functional distortion.
Critical strengths
S1. Rather than directly optimizing parameter count, quantization bitwidth, or matrix rank, this paper takes the actual compressed byte count as the optimization objective and searches over functionally admissible parameter representations. This perspective is valuable because it establishes a direct connection between functional equivalence in neural networks and physical storage cost, offering a new direction for compression optimization beyond conventional quantization, pruning, and low-rank decomposition.
S2. The authors clearly separate candidate generation, functional admissibility checks, and physical compression evaluation. Compared with approaches that rely entirely on differentiable entropy estimates or surrogate compression rates, directly measuring the compressed size of the entire checkpoint avoids discrepancies between surrogate objectives and actual physical compression outcomes.
Critical weaknesses
W1. Although the paper compares different representation classes and distortion constraints, it does not sufficiently establish whether the main compression gains arise from the proposed search framework or primarily from the existing significand rounding operations. Furthermore, the main paper does not adequately isolate the individual contributions of Q1–Q4 rounding and sparsification.
W2. The proposed method requires repeatedly compressing the entire checkpoint and performing functional checks on candidate models. Although the authors employ staged filtering to reduce some computational overhead, the paper does not sufficiently report the end-to-end search time, the number of physical compressor invocations, or the cost of functional evaluations. In addition, the main experiments are limited to models with 135M, 360M, and 0.6B parameters. For larger models, repeated whole-model compression and functional evaluations may introduce substantial computational and I/O overhead, raising concerns about the scalability of the proposed approach.
Questions for authors
Q1. Contribution of the search framework: In the Qwen3-0.6B experiment, all 84 MLP projections ultimately adopt the Q1 candidate. Is the resulting checkpoint identical to the one obtained by directly applying Q1 rounding to all corresponding tensors? If so, how do the authors justify the incremental value of the greedy physical compression search in this experiment?
Q2. Contributions of different candidate strategies: Could the authors report, based on existing experimental logs, the number of accepted updates or the respective contributions of Q1–Q4 and candidates at representative operating points for SmolLM2-135M/360M? Are there any existing ablation results comparing significand rounding alone, sparsification alone, and their combined candidate set?
Q3. Optimization overhead: Could the authors provide the end-to-end search time, the number of whole-model compression calls, and the number of functional evaluations for a representative operating point on SmolLM2-360M? These details would help assess the trade-off between the achieved storage reduction and the offline optimization cost.
Thank you for the focused questions about the Qwen3 endpoint, proposal-family contributions, and computational cost. We have addressed them directly in the revised manuscript and added a matched rounding-baseline study.
Q1. Does Qwen3 reduce to uniform Q1? Yes, at the parameter-proposal level. The saved Qwen3-0.6B trajectory accepts deterministic Q1 edits on all 84 eligible MLP projections, followed by a zero-accept sweep. Because every proposal is constructed from its immutable original tensor, the recorded trajectory reaches the all-MLP-Q1 configuration. It saves 44.289 MB (4.802%) at validation KL 0.001631. We now identify this point as a proposal-family saturation control. A bytewise replay would complete the serialized-checkpoint identity check.
Q2. What do Q1--Q4 and ℓ1 contribute? The original SmolLM2-360M multibit+ℓ1 records show 67 accepted updates at the 3×10−3 search control and 97 at 8×10−3, saving 11.722% and 23.344% of Zstandard storage, respectively. Those saved summaries aggregate the candidate families. We therefore report these totals as combined-family trajectories and identify per-family accepted-edit attribution as the remaining ablation.
We also added a controlled precision-allocation result. On 64 C4-validation documents with an identical 90-MLP-projection scope, uniform BF16 Q2 saves 22.317 MB at mean KL 0.003158, while a frozen Fisher-guided BF16 allocation saves 27.241 MB at 0.002818. A separately C4-calibrated AWQ-inspired packed-INT8 CPU reference saves 46.518 MB at 0.002879, demonstrating the strong storage advantage of activation-aware integer representation at similar measured KL. The comparison directly isolates uniform versus adaptive BF16 proposals.
Q3. How expensive is optimization? A measured SmolLM2-135M CPU local-search pilot performed 28 proposals and 31 full-checkpoint Zstandard-9 evaluations in 226.8 s of warm elapsed time. The two frozen quality audits gave mixed results for the selected swap relative to the static Fisher schedule. In a separate encoding benchmark, 4 MiB independently framed checkpoint storage reduced median complete-archive update time from 5.88 s (monolithic) to 0.36 s (framed) on six matched probes, with 0.929 MB of storage overhead and format-specific byte rankings. These numbers quantify the cost of exact physical feedback and the benefit of local recompression. The original 360M record contains rate--distortion endpoints and aggregated accepted updates; per-proposal call counts, peak memory, and full end-to-end timing are natural targets for a dedicated profiler run.