acceptodds
Under review as a conference paper at ICLR 2027

Neural Compression over Equivalence Classes: Exact Gauges and Functional Rate–Distortion

Abstract

Neural networks admit many parameterizations of the same or nearly the same function, but deployment typically stores whichever parameter vector training happens to produce. We show that this representational freedom can itself be used for compression. Rather than redesigning the codec, we search over exact function-preserving transformations and controlled functional perturbations, using the unchanged physical compressor itself to select cheaper representations. Exact gauge families provide a zero-distortion reference under the strict audit; allowing small functional changes exposes much larger rate–distortion gains. Across SmolLM2 and Qwen3, our method reduces Zstandard-compressed checkpoint size by up to 23.4%, with similar relative reductions under gzip, Brotli, and ZipNN, establishing trained-network representational freedom as a practical storage resource.

Primary area:
optimization
Keywords:
Compression, LLM, Functional equivalence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wnVAOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to exploit the non-uniqueness of neural network parameterizations to reduce the physical storage size of losslessly compressed model checkpoints while preserving model functionality or maintaining functional distortion within acceptable limits. The authors formulate this problem as minimizing physical compression cost over functional equivalence classes and constrained functional neighborhoods, and propose a Proposal–Audit–Physical Selection framework. The method generates candidate weight representations through exact function-preserving transformations, such as hidden-channel permutations and invertible rescalings, as well as significand rounding and activation-aware ℓ1\ell_1 sparsification. Candidate representations are then selected through functional admissibility checks and evaluations using the actual compressor, without modifying the compression backend. Experiments are conducted on SmolLM2-135M, SmolLM2-360M, and Qwen3-0.6B, with evaluations using Zstandard, gzip, Brotli, and ZipNN. The results show that exact function-preserving transformations provide relatively limited compression gains, whereas allowing controlled functional distortion leads to substantially larger reductions. On SmolLM2-360M, the method achieves up to a 23.41% reduction in Zstandard-compressed checkpoint size, with gains that also transfer to other compression backends. The paper demonstrates that representational freedom in neural networks can serve as an additional resource for storage optimization, although the achievable gains depend on the specific transformations, model architectures, and permitted functional distortion.

Critical strengths

S1. Rather than directly optimizing parameter count, quantization bitwidth, or matrix rank, this paper takes the actual compressed byte count as the optimization objective and searches over functionally admissible parameter representations. This perspective is valuable because it establishes a direct connection between functional equivalence in neural networks and physical storage cost, offering a new direction for compression optimization beyond conventional quantization, pruning, and low-rank decomposition.

S2. The authors clearly separate candidate generation, functional admissibility checks, and physical compression evaluation. Compared with approaches that rely entirely on differentiable entropy estimates or surrogate compression rates, directly measuring the compressed size of the entire checkpoint avoids discrepancies between surrogate objectives and actual physical compression outcomes.

Critical weaknesses

W1. Although the paper compares different representation classes and distortion constraints, it does not sufficiently establish whether the main compression gains arise from the proposed search framework or primarily from the existing significand rounding operations. Furthermore, the main paper does not adequately isolate the individual contributions of Q1–Q4 rounding and ℓ1\ell_1 sparsification.

W2. The proposed method requires repeatedly compressing the entire checkpoint and performing functional checks on candidate models. Although the authors employ staged filtering to reduce some computational overhead, the paper does not sufficiently report the end-to-end search time, the number of physical compressor invocations, or the cost of functional evaluations. In addition, the main experiments are limited to models with 135M, 360M, and 0.6B parameters. For larger models, repeated whole-model compression and functional evaluations may introduce substantial computational and I/O overhead, raising concerns about the scalability of the proposed approach.

Questions for authors

Q1. Contribution of the search framework: In the Qwen3-0.6B experiment, all 84 MLP projections ultimately adopt the Q1 candidate. Is the resulting checkpoint identical to the one obtained by directly applying Q1 rounding to all corresponding tensors? If so, how do the authors justify the incremental value of the greedy physical compression search in this experiment?

Q2. Contributions of different candidate strategies: Could the authors report, based on existing experimental logs, the number of accepted updates or the respective contributions of Q1–Q4 and ℓ1\ell_1 candidates at representative operating points for SmolLM2-135M/360M? Are there any existing ablation results comparing significand rounding alone, ℓ1\ell_1 sparsification alone, and their combined candidate set?

Q3. Optimization overhead: Could the authors provide the end-to-end search time, the number of whole-model compression calls, and the number of functional evaluations for a representative operating point on SmolLM2-360M? These details would help assess the trade-off between the achieved storage reduction and the offline optimization cost.

1 comment

Thank you for the focused questions about the Qwen3 endpoint, proposal-family contributions, and computational cost. We have addressed them directly in the revised manuscript and added a matched rounding-baseline study.

Q1. Does Qwen3 reduce to uniform Q1Q_1? Yes, at the parameter-proposal level. The saved Qwen3-0.6B trajectory accepts deterministic Q1Q_1 edits on all 84 eligible MLP projections, followed by a zero-accept sweep. Because every proposal is constructed from its immutable original tensor, the recorded trajectory reaches the all-MLP-Q1Q_1 configuration. It saves 44.289 MB (4.802%) at validation KL 0.001631. We now identify this point as a proposal-family saturation control. A bytewise replay would complete the serialized-checkpoint identity check.

Q2. What do Q1Q_1--Q4Q_4 and ℓ1\ell_1 contribute? The original SmolLM2-360M multibit+ℓ1\ell_1 records show 67 accepted updates at the 3×10−33\times10^{-3} search control and 97 at 8×10−38\times10^{-3}, saving 11.722% and 23.344% of Zstandard storage, respectively. Those saved summaries aggregate the candidate families. We therefore report these totals as combined-family trajectories and identify per-family accepted-edit attribution as the remaining ablation.

We also added a controlled precision-allocation result. On 64 C4-validation documents with an identical 90-MLP-projection scope, uniform BF16 Q2Q_2 saves 22.317 MB at mean KL 0.003158, while a frozen Fisher-guided BF16 allocation saves 27.241 MB at 0.002818. A separately C4-calibrated AWQ-inspired packed-INT8 CPU reference saves 46.518 MB at 0.002879, demonstrating the strong storage advantage of activation-aware integer representation at similar measured KL. The comparison directly isolates uniform versus adaptive BF16 proposals.

Q3. How expensive is optimization? A measured SmolLM2-135M CPU local-search pilot performed 28 proposals and 31 full-checkpoint Zstandard-9 evaluations in 226.8 s of warm elapsed time. The two frozen quality audits gave mixed results for the selected swap relative to the static Fisher schedule. In a separate encoding benchmark, 4 MiB independently framed checkpoint storage reduced median complete-archive update time from 5.88 s (monolithic) to 0.36 s (framed) on six matched probes, with 0.929 MB of storage overhead and format-specific byte rankings. These numbers quantify the cost of exact physical feedback and the benefit of local recompression. The original 360M record contains rate--distortion endpoints and aggregated accepted updates; per-proposal call counts, peak memory, and full end-to-end timing are natural targets for a dedicated profiler run.

Reviewer_bMmkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies an interesting question in model compression: whether the many equivalent parameterizations of a neural network can be used to make its stored checkpoint smaller. Usually, deployment stores the parameter vector that training produces, but other parameterizations of the same function can give different bytes and different compressed sizes. To address this, the authors propose to search over exact function-preserving transformations and small functional changes, and to use the unchanged lossless compressor (Zstandard) to select the representation with the fewest bytes. The exact transformations include channel permutations and power-of-two rescalings in SwiGLU blocks, basis changes of the attention value/output maps, and factor changes in low-rank adapters. The approximate changes round low-order significand bits and apply activation-aware l1 sparsification. A change is accepted only if the real codec gives fewer bytes and the distortion stays below a threshold. The search is greedy over tensors, and each candidate is scored by compressing the whole model. The experiments use SmolLM2-135M, SmolLM2-360M and Qwen3-0.6B, with gzip, Brotli and ZipNN as additional codecs. Exact transformations give very small gains (about 14 KB of 208 MB for SmolLM2-135M). Small functional changes give much larger gains: about 5–7% at low distortion and up to 23.4% on SmolLM2-360M at the largest tested distortion, with similar relative gains for the other codecs.

Critical strengths
  1. The paper studies a practical problem and proposes a simple and general idea. The method does not change the codec, the tensor shapes or the dtype. A change is accepted only if the real codec shows fewer bytes, and the gains of the main endpoints also hold for gzip, Brotli and ZipNN, which are not used in the main search.
  2. The authors performed a careful evaluation. They use separate data for calibration, validation and test, check the models on C4 and five zero-shot tasks, and report negative results, for example almost no gain for natural LoRA trajectories.
  3. The results give useful observations for practice. Exact transformations do change the compressed size; byte gains found for separate modules do not add up when the whole model is compressed; and a differentiable codec surrogate can rank candidates poorly, so the real codec must make the final decision.
Critical weaknesses
  1. Missing simple baselines. The main gains come from approximate edits of MLP weights. By my estimate, the 1-bit endpoints save about 1.3 bits per MLP weight, which is close to removing one low-order bit from every MLP weight. On Qwen3 for some limits the final state is exactly the 1-bit rounding of all 84 MLP projections, so here the search gives the same result as a simple uniform rule. However, the paper does not compare with the same rounding applied to all MLP tensors without search, or with standard quantization (e.g., FP8 or INT8 round-to-nearest, GPTQ, AWQ) at the same KL or perplexity. FP8 alone would halve the raw size of the MLP weights. Without these baselines, it is hard to see the practical value of the search.
  2. Limited novelty of the main gain. Exact transformations, which are a key part of the title, give almost nothing (about 0.007% of the compressed size). The large gains come from rounding low-order significand bits before lossless compression, which is a known technique for scientific data, e.g., Bit Grooming.
  3. Limited evaluation and unclear scalability. The models are small (135M–0.6B). The headline 23.4% is obtained only on SmolLM2-360M; on Qwen3, only the 1-bit search was run, with at most 4.8%. The search cost is not reported, although each candidate requires compressing the whole model.
Questions for authors
  1. How does the method compare with simple baselines at the same KL or perplexity: the same sors without search, and standard quantization such as FP8 or INT8 round-to-nearest, GPTQ orAWQ? Can the method be applied on top of an already quantized checkpoint?
  2. What is the search cost (wall-clock time and number of codec calls) for each model, and how would it scale to 7B+ models? Have you tried the multibit search on Qwen3 or on larger models?
3 comments

Thank you for requesting stronger simple baselines and a clearer connection to rounding, quantization, and prior work. We have revised the paper around a precise contribution: functional constraints define the admissible representations, and the deployed lossless codec measures their actual storage cost. A new matched-tensor-set experiment provides a direct uniform-rounding comparison and a stronger integer-storage reference.

Uniform and activation-aware controls. We evaluate 90 eligible SmolLM2-135M MLP projections under whole-checkpoint Zstandard v1.5.7 level 9. On 64 distinct C4-validation documents, uniform BF16 Q1Q_1 saves 12.852 MB at KL 0.002371, versus 20.881 MB at 0.002185 for a frozen Fisher-guided allocation. Uniform Q2Q_2 saves 22.317 MB at 0.003158, versus 27.241 MB at 0.002818 for a more aggressive allocation. Thus adapting per-tensor BF16 rounding depth improves measured storage and fidelity relative to these uniform rules. The Fisher schedules were frozen before this audit; the submitted codec-guided greedy algorithm is a distinct selection policy.

C4-calibrated AWQ-style storage reference. We additionally evaluate a group-128 AWQ-inspired CPU packed-INT quantizer, calibrated on 32 C4 documents and evaluated alongside the BF16 schedules. Its 8-bit point saves 46.518 MB at KL 0.002879, substantially exceeding the 27.241 MB saved by Fisher BF16 at KL 0.002818. The paired KL difference is +0.000061+0.000061 with a descriptive 95% interval [−0.000081,+0.000206][-0.000081,+0.000206]. The reported procedure and packed format define this reference precisely. Calibration and evaluation documents are disjoint within the same C4-validation shard. This strong result sharpens the paper's focus on physical representation selection and its interaction with established quantization strategies.

Novelty relative to existing methods. Bit Grooming (Zender, 2016) introduced low-order significand trimming; we now cite it explicitly. Our proposal combines exact algebraic gauges, activation-aware neighborhood construction, functional auditing, and direct optimization of complete serialized checkpoint size under a fixed physical codec. Exact-gauge savings are 14,416 bytes from a 208,502,655-byte Zstandard baseline (about 0.0069%), providing a measured zero-audit-error reference. Module-level optimization gives a different byte optimum from whole-checkpoint optimization, and a Qwen3 surrogate study shows why final codec measurements matter for discrete decisions.

Scale and fidelity. The original SmolLM2-360M frontier saves 65.651 MB (11.722%) at validation KL 0.002904 and 130.745 MB (23.344%) at KL 0.007468. At the latter point, WikiText/C4 perplexity changes by +0.668%/+0.293%, and the maximum raw-accuracy deviation across five zero-shot tasks is 0.49 percentage points. The revision presents these as measured rate--distortion operating points and distinguishes BF16 representation shaping from packed-INT storage.

Thank you for adding the comparisons with simple rounding and int8. I still have a question about how much the search itself helps. On Qwen3, it chose the same rounding rule for all 84 MLP matrices. It is also unclear how long the full search took on the 360M model. For these reasons, I will maintain my score.

Thank you for the follow-up. The Qwen3 experiment reaches a saturation point where all 84 eligible MLP matrices select Q1, providing a useful uniform-rounding reference. Adaptive allocation becomes more valuable when tensor sensitivities differ. In our SmolLM2-135M C4 audit, Fisher-guided allocation saved 27.24 MB at KL 0.00282, compared with 22.32 MB at KL 0.00316 for uniform Q2.

We have also developed a fast incremental encoder to reduce the cost of repeated physical compression evaluations during search. With independently compressed Zstandard chunks, median local checkpoint-update time decreased from 2.073 s to 0.319 s (6.50×). In a separate 30-proposal benchmark including durable disk writes and final compaction, total codec-session time decreased from 74.70 s to 23.32 s (3.20×), while disk traffic decreased from 5.73 GB to 1.27 GB. All reconstructed checkpoints passed byte-exact verification.

We further evaluated Zstandard level 9, reducing median complete in-memory archive-update time from 5.884 s to 0.359 s (16.4×), with approximately 0.93 MB of additional checkpoint storage.

These benchmarks quantify the acceleration achieved by our encoding implementation on SmolLM2-135M CPUs. An instrumented local search on this model required 226.8 s for 28 proposals and 31 physical Zstandard-9 evaluations.

These results demonstrate that exact compressed-size feedback can be made substantially more efficient, supporting its use in larger representation-search problems.

Reviewer_8u5nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper searches for alternative representations of BF16 model weights that compress better with existing codecs while keeping model behavior within a specified tolerance. Exact function-preserving transformations give small savings, while approximate changes reduce cross compression methods size by a quarter on SmolLM2-360M. The results suggest a useful approach to preparing trained models for storage and distribution.

Critical strengths

Optimizing actual compressed file size is a useful perspective. The transfer across compression methods strengthens the empirical results.

The paper carefully separates exact transformations from approximate changes and evaluates the latter on held-out data and downstream tasks.

I appreciated the negative results and the discussion of where further compression starts to noticeably affect model behavior.

Critical weaknesses

The computational cost is difficult to assess. The search repeatedly compresses whole checkpoints and evaluates calibration behavior, but runtime and peak-memory measurements are missing for the main LLM experiments. With models only up to 0.6B parameters, practical scalability remains unclear.

The benefit of the search itself is less clearly established than the compression gains. For example, the saturated Qwen3 endpoint applies the same one-bit rounding operation to every eligible MLP tensor. A comparison with simpler rounding schedules would help show when codec-guided selection provides an advantage.

Questions for authors

willing to increase the score when feasible additional compute are in relation to saved storage.

recompressing the entire models is a design choice here and I'd like to better understand why proxies or delta compressions are not feasible, as desired patterns of compression methods should be known or finished matrices should remain invariant?

Could you report wall-clock time and peak memory for the LLM searches, including the time spent on compression and forward passes?

Do useful gains persist on larger models or already quantized checkpoints? Which transformations would remain applicable in those settings?

How does the search compare with simpler rounding schedules using the same eligible tensors, BF16 format, compressor, and held-out distortion metric?

The title "over Equivalence Classes" overemphasizes the neglibile part of exact reformulation that does not seem to contribute much, and more severely not scale?

(Divergent Token Metrics of Deiseroth et al could be discussed as related work for discrepancy of PPL/KL to token generation)

figure 1 should be cleaned - the core goal and concept should be simpler and more intuitive presentable

1 comment

Thank you for highlighting the physical-byte objective, the functional constraints, and the need for stronger rounding controls and computational accounting. The revision now foregrounds the measured contribution: selecting a model representation under a functional constraint using actual compressed-checkpoint bytes. We also report new controlled baselines, the scale of exact-gauge savings, and measured CPU search costs.

Value beyond uniform rounding. On SmolLM2-135M, the supplementary comparison fixes the same 90 eligible MLP projections and uses whole-checkpoint Zstandard v1.5.7 level 9. On 64 separate C4-validation documents, uniform BF16 Q1Q_1 saves 12.852 MB at mean KL 0.002371; a frozen Fisher-guided BF16 allocation saves 20.881 MB at 0.002185. Uniform Q2Q_2 saves 22.317 MB at 0.003158, compared with 27.241 MB at 0.002818 for a more aggressive Fisher allocation. The paired descriptive 95% intervals for adaptive-minus-uniform KL are [−0.000331,−0.000048][-0.000331,-0.000048] and [−0.000541,−0.000150][-0.000541,-0.000150]. These points demonstrate the benefit of adaptive precision allocation within the tested BF16 proposal family. The original paper's greedy search also evaluates candidate storage with the actual codec; the new Fisher allocations are frozen static schedules and isolate the allocation mechanism.

Qwen3 proposal saturation. The Qwen3-0.6B search accepts deterministic Q1Q_1 edits on all 84 eligible MLP projections, then completes a zero-accept sweep. This establishes an all-MLP-Q1Q_1 proposal-family ceiling at 44.289 MB (4.802%) saved and validation KL 0.001631. Accordingly, we present Qwen3 as architecture transfer and a useful direct-rounding saturation control.

Optimization cost and physical scoring. A separate SmolLM2-135M CPU local-search pilot measured 28 two-tensor proposals, 31 full-checkpoint Zstandard-9 evaluations, and 226.8 seconds of warm search time. A 4 MiB independently framed checkpoint reduced median complete-archive update time from 5.88 s for monolithic recompression to 0.36 s over six matched probes, with a 0.929 MB baseline storage overhead. The format can change candidate rankings: four of 28 byte-improvement decisions differed. These results make the search--storage tradeoff explicit. The archived 360M evidence provides accepted-edit totals and rate--distortion measurements; full wall-time and peak-memory profiling is the next instrumentation target.

Exact symmetry and presentation. The strict-gauge study saves 14,416 bytes (0.0069%) on SmolLM2-135M, establishing the exact-functional reference point. Approximate output-functional edits account for the larger MB-scale savings. A Qwen3 module probe further shows that the differentiable codec surrogate has Spearman −0.193-0.193 against eight actual outcomes, motivating exact physical reranking. We simplified Figure 1, sharpened the distinction between exact algebraic preservation and sampled numerical audits, and expanded the discussion of sequence-level behavior using Deiseroth et al. (2024).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.