acceptodds
Withdrawn from ICLR 2027

Decoupling Coordinate Permutations from Low-Dimensional Quantiles in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training weight modeling and quantization schemes predominantly treat neural network tensors as monolithic coordinate arrays, simultaneously optimizing value discretization and positional assignment. In this work, we investigate a fundamental question: How much information truly resides within the scalar values of a Large Language Model (LLM) weight tensor when strictly decoupled from coordinates? We formalize an orthogonal decomposition of any weight matrix W ∈ Rm×n into a sorted value spectrum s = sort(vec(W)) and a spatial permutation π ∈ SN . Stripping away positional entropy reveals an unexpected empirical regularity: the position-agnostic value spectrum of trained LLM tensors is universally governed by an ultra-low-dimensional continuum. Across diverse open-weight architectures (Meta-Llama-3.2, Qwen-3, Gemma-4) and across every internal layer type, an unconstrained 5-term Metalog quantile distribution fits the empirical spectrum with near-zero error (MAE < 0.0002, relative MSE < 0.002). This closed-form parameterization decisively outperforms classical Gaussian, extreme-value, and generalized quantile baselines. Crucially, our findings expose an extreme information-theoretic asymmetry: while the value distribution collapses to an O(1) description length of just 5 scalars (160 bits), spatial permutations retain an irreducible combinatorial entropy of ⌈log2(N !)⌉ ∼ O(N log N ) bits. We demonstrate that learned model capacity resides almost entirely within spatial permutations rather than value variance, establishing that future model compression paradigms must pivot toward structured permutation coding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Lineage Verification, Weight-Space Forensics, Metalog Distributions, Quantile Representations, Data-Free Auditing, Open-Weight LLMs, Parameter-Space Invariants, Model Governance

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_mePfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the potentials of LLM weight compression/quantization by looking at the weight scalar distribution while neglecting all positional information. The experiment shows that Metalog quantile distribution can fit these sorted weight values precisely in a low-dimension characterization. However, the positional information remains the bottleneck for compression/quantization.

Critical strengths
  • The paper demonstrates that the empirical distribution of LLM weights can be precisely approximated using a low-dimensional Metalog parameterization.
Critical weaknesses
  • Metalog distributions are inherently flexible in approximating probability distributions. Therefore, a good fit to sorted LLM weights is not particularly surprising. A critical question that the paper does not address is how this property can be leveraged for practical model quantization or compression, such as reducing storage requirements or memory bandwidth for LLMs.
  • The evaluation is limited to fitting sorted weights using local error metrics (MAE and rel-MSE). More global, task-relevant evaluations are needed to determine whether separating positional information from scalar weight values can provide practical benefits.
Questions for authors
  • Beyond the low fitting errors achieved by Metalog distributions, could you explain how the proposed method could provide practical benefits for LLMs, given that positional information must also be preserved?
  • Could you provide extra evaluations showing end-to-end improvements in LLM quantization/compression, beyond local fitting metrics? For example, could the proposed method compress a model to an average of X bits per weight while limiting the downstream accuracy degradation to Y%?
Reviewer_JU7dOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

By separating weight matrices into sorted values and spatial permutations, this paper examines how compactly the distribution of LLM weight values can be represented. A five-parameter Metalog-based model is fitted to the sorted weights using a chunked least-squares procedure. Experiments on Llama, Qwen, and Gemma models compare approximation errors across layers, basis sizes, and alternative distribution families. The reported results show low fitting errors with five parameters. From these observations, the authors argue that most parameter information resides in spatial permutations and suggest permutation coding as a direction for future model compression.

Critical strengths
  1. By examining weight values separately from their positions, the paper provides a useful perspective on how the sorted weight distribution can be approximated with a small number of parameters.

  2. Across several model families and weight matrix types, the experiments provide an initial comparison of fitting accuracy at different basis sizes.

Critical weaknesses
  1. I found the paper poorly written and difficult to follow. The motivation for the experiments is unclear, and I had trouble understanding what question each experiment is intended to answer and how the results connect to the main claims. Figure 1 sketches the value-position decomposition, but it does not give a clear overview of the full analysis pipeline or the reasoning behind the experimental design. Overall, the presentation feels rushed and underdeveloped.
  2. The paper does not establish that the fitted weights preserve model performance. The experiments mainly measure weight reconstruction error, without reporting perplexity or downstream task performance after replacing the original weights with the fitted values. Small numerical errors alone do not demonstrate that model capabilities are preserved. I therefore find the claim that model capacity resides almost entirely in permutations insufficiently supported.
  3. The information-theoretic interpretation needs more justification. The five coefficients describe an approximate curve, so their 160-bit storage cost does not account for the residuals needed to recover the original weights exactly. Likewise, counting all possible permutations does not establish an irreducible information content for trained weight arrangements. The distinction between approximation error, exact reconstruction, and description length is not sufficiently clear.
  4. The Metalog formulation needs clarification. The fifth basis function in the manuscript differs from the standard definition in the cited reference. It is unclear whether this is a typographical error or an intentional modification. The paper also does not report whether the fitted functions satisfy the monotonicity requirement of a valid quantile function.
  5. The baseline comparison raises questions about the fitting procedure. GLD contains Tukey Lambda as a special case, yet its reported fitting error is substantially higher. Under the same fitting objective and with sufficient optimization, GLD should perform at least as well as this subfamily. The difference needs an explanation before these results can support the claimed advantage of Metalog.
  6. The discussion of related work and the theoretical motivation are incomplete. Prior work has already studied the separation of weight values and positions, so the additional contribution of this paper needs to be explained more clearly. The motivation also appears to conflate the distribution of matrix eigenvalues with the distribution of individual weight entries. Heavy-tailed behavior in the former does not directly establish the same property in the latter.
  7. The claims of universality are stronger than the evidence supports. Results from three models do not establish a universal property, and low average error across layers does not mean that every layer is fitted equally well. Figure 2 already shows some layers with noticeably larger errors. Claims such as a near-perfect fit across all layers need to be qualified.
Questions for authors
  1. What happens to perplexity and downstream task performance when the original permutation is retained but the weight values are replaced by the fitted quantiles? How do the results differ between replacing individual layers and replacing weights throughout the model?
  2. Does the 160-bit estimate cover only the five fitted coefficients? How would the residuals be handled if exact reconstruction were required? What assumptions justify interpreting the number of possible permutations as an irreducible information content of trained weights?
  3. Which five basis functions are actually used in the experimental code? Is the departure from the standard Metalog definition intentional, and have all fitted functions been checked for monotonicity?
  4. Are GLD and Tukey Lambda fitted to the same data using the same objective? Why does the more flexible GLD obtain a worse fit, and have initialization and convergence issues been checked?
  5. What is the specific contribution beyond earlier work on separating weight values from their positions? How does the cited heavy-tail theory support claims about the distribution of individual weight entries?
  6. Could the authors provide the complete layer-wise errors for all three models, including the largest errors and the errors in the distribution tails? This would help clarify where the conclusions hold and where exceptions occur.
Reviewer_EQViOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how LLM weight values can be represented when separated from their coordinate assignments. Each weight matrix in an LLM is decomposed into a sorted value vector and a permutation that restores the original positions. The authors approximate the sorted values using a five-parameter Metalog quantile function. Experiments across three model families (Llama-3.2-3B, Qwen3-8B, and Gemma4-12B) report low reconstruction errors.

Critical strengths
  1. Separating scalar values from their positions provides an intuitive framework for investigating the statistical structure of LLM parameters. The distribution of scalar values in LLMs also provides a meaningful research direction in model compression and quantization.

  2. The procedure is simple: the Metalog formulation is linear in its coefficients, enabling efficient estimation through a small least-squares system.

  3. The experiments report low average reconstruction errors across several model families and projection types.

Critical weaknesses

I notice that the submission ends with a few references and an empty appendix. I am not sure if this submission is a completed paper for publication. Anyway, I will focus on the methodology and experiments and talk about my concerns below:

Methodology:

  1. The method tries to make the reconstructed weights numerically close to the original weights. However, some weights matter more to the model’s predictions than others. A small change to an important weight could cause more harm than a larger change to an unimportant one. As a result, the reconstructed weights may look close to the originals while still changing how the model behaves. So, I am concerned that the fitting process does not account for this difference.

  2. The method uses one curve with only five parameters to approximate all the weight values in a matrix. This curve may fit most values well but approximate a small number of unusual values poorly. These values may be important for the model, even though they represent only a small fraction of its weights. Therefore, a low average fitting error can hide errors in a few important weights. The method does not include a separate mechanism to preserve these values.

Experiments:

  1. The experiments measure weight reconstruction error, but do not evaluate the language model performance. Small average weight errors do not necessarily imply small changes in model outputs, particularly when errors affect sensitive coordinates or accumulate across layers. I suggest the authors should evaluate perplexity and downstream performance of the reconstructed model with general benchmarks. Without these experiments, the conclusion that learned capacity resides almost entirely in permutations is not convincing.

  2. It would be better if the paper reported total storage, decoding cost, inference overhead, or performance at a given bit budget. Preserving an exact permutation can be expensive. Moreover, fitting a continuous weight distribution does not solve the problem of selecting a small discrete codebook while preserving model behavior. Therefore, the claim that value discretization is “effectively solved,” and that future compression must focus on structured permutation coding, needs more experimental evidence.

Questions for authors

Q1: After replacing the original weights with the fitted values and restoring their positions, how do perplexity and downstream task accuracy change, for example, in GSM8K or Math 500?

Q2: Some weights may strongly affect predictions even when their values change only slightly. Could fitting all weights equally damage these sensitive weights?

Q3: I am still not convinced by the claim that learned information mainly resides in permutations. Do you think the small differences between the original and fitted values still contain information essential to the model’s capabilities? If some important weights are changed a little, do you think it will change the model's capabilities a lot?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.