acceptodds
Withdrawn from ICLR 2027

Linger and Lose: Knowledge Collapse in Low-Bit Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training language models with ternary weights is commonly judged by loss and downstream accuracy, which record only a modest cost relative to full precision. We show that these metrics can conceal a much larger failure. We instead measure knowledge capacity, the factual bits stored per parameter, on synthetic biographies with known information content. We train GPT-2-style models from scratch with 2.5M to 50M parameters at five precisions. Under the standard cosine schedule, ternary models retain as little as 6% of an identically trained fp16 model's capacity. The deficit widens with model size while perplexity rises by only 1.4 to 1.6 times. Measured throughout training, these models first acquire capacity and then lose most of it. We identify this knowledge collapse as a learning-rate dwell instability. Held near – with no decay, a pre-collapse model collapses within a few hundred exposures, and returning to a safe rate does not restore capacity. We then locate the collapse in the output head. It happens at a value the model can never predict. The weights there grow unchecked, while every other prediction the model makes is unchanged. We find that making the value predictable removes the collapse, regardless of which attribute carries it. A warmup-stable-decay schedule with a 10% cooldown increases ternary capacity by 2.7 times at 25M and 4.3 times at 50M. Gains are larger at lower precision. Cutting the output head's learning rate prevents failure when training from scratch. The post-training quantization methods we tested recover no measurable capacity below 4 bits. Our findings suggest judging low-precision training by retained capacity during training rather than final loss.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
low-bit quantization, Quantization Aware Training, knowledge capacity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4cW3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates knowledge capacity degradation in low-bit language models, showing that ternary models can exhibit severe capacity collapse despite relatively modest increases in training loss. Through experiments on GPT-2-style models ranging from 2.5M to 50M parameters using synthetic biographies, the authors attribute this phenomenon to learning-rate dwell instability and abnormal output-head behavior on difficult-to-learn tokens. They further demonstrate that short WSD cooldowns and reduced output-head learning rates can mitigate the observed collapse.

Critical strengths
  1. Interesting empirical observations. The paper highlights a discrepancy between conventional loss-based evaluation and measured knowledge capacity in low-bit training, offering a potentially useful perspective on quantization quality.
  2. Detailed training dynamics analysis. The authors conduct extensive controlled experiments involving learning-rate schedules, fixed-LR interventions, corpus variants, and output-head optimization. These analyses provide useful insights into the observed instability.
Critical weaknesses
  1. Severely limited scale and questionable generalizability. All experiments are conducted on extremely small models (at most 50M parameters) and synthetic data, far removed from practical LLMs. More importantly, the central findings concern learning-rate dynamics and training stability, which are strongly dependent on model scale and training configuration. The paper itself observes substantially different collapse-onset learning rates between 25M and 50M models. Without validation at more representative scales (e.g., 1B–8B), it is difficult to assess whether the proposed mechanism and remedies remain relevant to modern LLMs. This substantially limits the significance of the conclusions.
  2. Unclear validity of the claimed knowledge collapse. The observed capacity degradation appears largely driven by abnormal predictions for a particular token (birth month), while other factual attributes remain mostly unaffected. Moreover, the reported collapse is sensitive to the capacity estimator, and the alternative compression-based estimator does not detect the same collapse at 25M. These findings raise concerns about whether the paper demonstrates genuine knowledge loss or primarily an output-head instability amplified by the evaluation metric.
Questions for authors
  1. Generalization to practical LLMs. Can the authors provide evidence that the observed collapse and learning-rate dwell mechanism persist at substantially larger model scales and under more realistic training conditions? Given the scale dependence already observed between 25M and 50M, even targeted experiments at intermediate scales (e.g., 800M–1B) would materially strengthen the conclusions.
  2. Distinguishing knowledge loss from metric artifacts. Can the authors provide more direct evidence that factual knowledge is genuinely forgotten, rather than the capacity reduction being dominated by a few abnormal token predictions? In particular, how should the apparent absence of collapse under the alternative compression-based estimator be reconciled with the paper's central claim?
Reviewer_gwYpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates knowledge capacity collapse in low-bit language models, revealing limitations of conventional loss-based evaluation. Through synthetic biography experiments, the authors identify learning-rate dwell and output-head instability as key factors and show that short WSD cooldowns and reduced output-head learning rates can mitigate the collapse.

Critical strengths
  1. The paper highlights a substantial discrepancy between conventional loss-based metrics and measured factual knowledge capacity in low-bit language models, revealing a potentially overlooked aspect of quantized training.
  2. The extensive learning-rate interventions, dataset ablations, and output-head analyses provide valuable insights into training instability. The identified mitigation strategies, particularly short WSD cooldowns and reduced output-head learning rates, are simple and empirically effective in the studied setting.
Critical weaknesses
  1. The primary capacity metric is sensitive to the clamping convention, and the Morris-style estimator does not detect the same collapse at 25M. More importantly, the degradation is largely concentrated on an initially unlearned birth-month token, while previously learned attributes remain stable. Thus, the results do not convincingly establish actual factual knowledge loss rather than output-head instability or a measurement artifact.

  2. All core experiments are conducted on synthetic biographies with models of at most 50M parameters. The observed collapse strongly depends on specific unlearnable tokens, raising concerns about whether this phenomenon occurs in realistic pretraining. Validation on natural-language corpora, larger models, and independent factual recall benchmarks is necessary.

  3. Although the authors identify learning-rate dwell and output-head divergence as important factors, the experiments do not sufficiently disentangle ternary quantization, output-head precision, and optimization dynamics. In particular, the collapse also occurs in FP16 models, while alternative gradient estimators can avoid it. More systematic component-wise precision ablations and cross-configuration analyses are needed to establish the proposed mechanism.

Questions for authors
  1. Can the authors provide independent factual recall accuracy and attribute-level retention before and after collapse to demonstrate actual forgetting of previously learned facts rather than artifacts of the capacity estimator?

  2. Does the observed collapse also occur on realistic pretraining corpora and larger models? Can the authors validate it through factual recall, validation loss, and downstream performance?

  3. Is knowledge collapse specific to ternary training, or does it primarily arise from output-head optimization instability? Can component-wise precision ablations disentangle these factors?

Reviewer_sTu3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigate the knowledge capacity of low bit LLM using QAT. The method follow the setting of Allen-Zhu and Li (2024), they train GPT-2 model from scracth at different bit-width and model size. After that, they measure knowledge capacity and compare to model loss. The observation is that ternary model may lose capacity under cosine learning rate schedule. They hypothesize this observation due to the usage of specific learning rate for a long time. Then they also observe what happen when the model collapse (lose capacity), and find out that output head has very high big logits due to tokens representing birth-month info. Then they propose a WSD learning rate schedule and reduce output head learning rate to improve instability

Critical strengths

1.Interesting insight: The paper provide new insight that link the degradation in model capacity with training instability due to learning rate schedule, instead of just lower bit representation

  1. Many ablation studies: The paper conduct a wide range of controlled experiment across different setting (learning rate schedule, optimizers,...), providing useful evidences for the insights.
Critical weaknesses
  1. Limited technical novelty: The paper uses existing metric, QAT method, optimizer,... there is little technical contribution

  2. The interpretation of observation is not clear: The paper provide another metric measure the knowledge capacity. The first metric show model collapse, the second metric does not show any collapse at 25M, meaning the collapse depend on how we define the metric. I have doubted whether the model collapse actually denote the loss of knowledge or not, depend on the choice of dataset and metric.

  3. The insight may not generalize to other settings, it use a specific unconventional dataset with specific informations (bith month, age, name,...) and small models (up to 50M parameters). It may not generalize to other datasets or architecture

  4. Lack of intuitive explanation why specific learning rate (1-2 x 10^{-4}) make the model collapse beyond empirical finding.

Questions for authors

Please answer the concern in Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.