acceptodds
Under review as a conference paper at ICLR 2027

When Does Quantization Form Reusable Concepts?

Abstract

Quantization encodes learned representations as discrete tokens. When do joint encoder and codebook updates organize them into reusable concepts? A preserved property such as shape may still require complex token combinations to read across colors and viewpoints. We study reuse through one rule shared across contexts. Our principle is that parameter updates move assignment boundaries, and inputs crossing them alter subsequent updates and shared-rule errors. Reference models show how boundary geometry constrains factor selection and relative update rates change local stability and relaxation. In a frozen residual model, we bound the time until one token test reads the target without error across contexts. Learned quantizers preserve the target perfectly at the same token budget, yet have certified minimum error-free rule lengths of 15 and 34 bits under one grammar. These results connect training conditions to factor organization and distinguish target recovery from concise reuse.

Primary area:
learning theory
Keywords:
Vector Quantization, Dynamics Analysis, Reusable Concepts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_aFHROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates how quantization training shapes the organization of target information in discrete representations and whether this information can be read out using a concise rule shared across contexts. The authors distinguish preserving target information from making it accessible through a concise rule. They study the feedback between parameter updates and assignment changes, examining how the data distribution near assignment boundaries and the relative update rates of the encoder and codebook affect factor organization, and how assignment changes affect fixed-rule error. They derive a rule-formation time bound in a frozen residual setting and evaluate their analysis using controlled experiments, Shapes3D representations, and ECG measurements.

Critical strengths
  1. The paper makes a useful distinction between preserving target information and making it accessible through concise rules shared across contexts. The authors show that two quantizers trained from the same initialization both preserve the target perfectly at the same token budget but have different minimum rule lengths under the specified grammar.

  2. The paper provides an analytical connection between assignment changes during training and fixed-rule errors. By distinguishing changes that correct, introduce, or leave errors unchanged, it explains why token churn alone is insufficient to characterize changes in rule performance.

Critical weaknesses
  1. The experiments do not clearly isolate the added predictive value of modeling assignment exchanges. On the controlled task, Full does not significantly outperform Branch in MAE or ranking. On Shapes3D, Full also includes additional boundary-mass and churn features. Although this combined-feature comparison is acknowledged in the paper, it does not isolate the contribution of the exchange-aware response analysis. An ablation separating these additional features from the exchange-aware response features would clarify the empirical contribution of the proposed analysis.

  2. The presentation makes it difficult to follow the connections between the different theoretical settings. A clearer overview of what is fixed or trained in each setting, what each result establishes, and how the results relate would make the scope and contribution of the analysis easier to assess.

Questions for authors

Based on the findings, what practical guidance can the authors offer for training quantized representations that support concise rules shared across contexts?

Reviewer_oGmyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors study the setting of quantizers and ask about the relationship between encoding certain parts of information (like, for instance, the type of flower in images that vary in color) and the difficulty of reading it out. They take a kind of theoretical computer science approach to measuring the difficulty of readout in terms of the bit-length of the shortest decision tree. They focus their study on an example of two networks that have very similar loss, but have quite different complexities in terms of the number of bits needed, 15 versus 34. After that they ran a number of experiments and analyses, mostly in toy models, to try to explain why training might land in the more complex readout case, and to understand the relationship between training and readout more generally.

Critical strengths

I really liked their approach to how they measured complexity in terms of bit length of decision trees. The normal way that at least some parts of ML would go about it would be training probes for certain properties. I think their approach with decision trees is much more rigorous and substantial.

Critical weaknesses

From my reading, the main weakness seemed to be that the title of the paper was not directly answered. Their main empirical result is on two particular networks, and their theory is in kind of toy models: two-dimensional linear settings and frozen encoders. In the end, I think what is shown is more of an existence proof of these kinds of two solutions, rather than a more thorough understanding of what the title purports to be asking and answering.

Questions for authors

My understanding is that the length of the program for the decision tree depends on the grammatical rules that you use to express decision trees. I'm wondering how much of the gap (15 versus 34) is due to the choice of grammar. Are there other reasonable choices of grammar you can test to see if that gap holds? If it’s possible, it would be nice to even be able to sample from grammars and put error bars on this.

For clarification, I am wondering your response to my critical weakness. Do you think I am misunderstanding either the title or the work?

Reviewer_KWhdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when quantized representations support concise rules that recover a target factor across contexts. It distinguishes target preservation from the complexity of accessing that target, analyzes assignment-boundary feedback in reference models, and derives a rule-formation time bound for a frozen residual quantization setting. Experiments examine learned rule complexity, training interventions, frozen Shapes3D representations, and ECG morphology measurements.

Critical strengths
  1. The distinction between information preservation and concise cross-context access is meaningful.

  2. The enumeration in the small discrete setting provides a concrete demonstration that representations with target recovery can require different minimum rule lengths under a fixed grammar.

Critical weaknesses
  1. The connection between the operational metric and reusable concepts needs stronger justification.

  2. Description-length comparisons are sensitive to the chosen encoding scheme. Using alternative encodings, shared preprocessing costs, and threshold precision would help establish which conclusions are robust.

Questions for authors

Does Full retain its Shapes3D advantage when Branch receives the same boundary-mass and churn features?

Reviewer_P7MDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As per my rough understanding, the paper looks at quantization of tokens into codebooks, and explores how the data and training schemes affect how the codebooks are formed, what rules are created, and how reusable and complex they are.

Critical strengths

None.

Critical weaknesses

The writing is very hard to understand. For example:

  • The introduction starts with very little context, especially broader context that is important in any paper. For example, why is quantization important? Where is it applicable? These may be evident for researchers working exactly in the field but at least having some broader context is highly important in a paper.
  • L031: what is a reusable interpretation?
  • L038-040: what layers are being referred to here? It becomes somewhat clearer after carefully looking at Figure 1, but is very hard to understand otherwise. The writing should guide the reader.
  • L040: what shared rule is being referred to? Also in L014
  • L050: what does "Tracking exchanges by target value and context connects organization formation to its use." mean?
  • L056: what does "Assignment exchanges connect the update mechanism to fixed-rule risk, while endpoint rule selection measures the access each organization supports. A frozen residual reference then links the distribution created by a preceding layer to the time an error-free token test becomes available.". A lot of terms are used without being defined or without proper context. What is fixed-rule risk? What is endpoint rule selection? And so on.
  • L060: very specific result without context. Which grammar is being referred to here?
  • L063-065: no citations provided for any of the datasets and setups being referred to here.

Overall, independent of the paper's content and its significant, in my view the writing alone makes it currently unsuitable for publication, and a major revision is necessary. Despite having read and to an extent worked with quantization, I could not understand the paper.

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.