acceptodds
Desk rejected at ICLR 2027

Capability Scaling-Down Laws for LLM Compression

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM compression reduces inference costs and memory requirements, but selecting a method and configuration remains largely empirical because comparable resource reductions can produce different capability losses. We present a systematic investigation of capability scaling-down laws for LLM compression across pruning, quantization, and distillation. Our framework measures capability loss in mathematics, code generation, and question answering, and relates these measurements to model size, training stage, compression settings, data availability, and training exposure. We develop simple predictive relations and evaluate their accuracy, measurement efficiency, and generalization to unseen configurations and model states. The results identify response patterns that can be shared across capabilities and differences that require separate treatment. Sharing the density response across pruning levels halves the configuration measurements needed to fit a pruning predictor: on new Pythia states, on pre-registered OLMo-2 test states and under Wanda pruning, the compact relation matches a regression fitted with all measurements on math and code to within 0.020 nats per token, with coefficients refitted for each setting. Controlled distillation experiments show that the cost of heavy data reuse recurs across question-answering distributions, while the net benefit depends on the evaluation distribution. We further evaluate the decision value of these predictions by comparing numerical selection with configuration medians and fixed method priorities. Independent evaluations across two model families show that selection captures most of the available cross-method benefit for question answering within the tested candidate sets, where a fixed method priority attains the same regret, with smaller opportunities for mathematics and code. These results clarify the predictive scope of capability scaling-down laws and their use in compression method selection. Our code is publicly available at: https://anonymous.4open.science/r/scaling_down_law .

Primary area:
foundation or frontier models, including LLMs
Keywords:
Scaling-down law, LLM compression, LLM capability

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_HxWGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper primarily predicts changes in reference-answer likelihood under pruning, quantization, and distillation across mathematics, coding, and QA, with supplementary downstream performance checks. It develops predictive relations and evaluates unseen configurations, model states, and compression strategies. Its strongest result is a compact pruning predictor using half the configuration measurements while approaching full-grid accuracy, replicated on Pythia and OLMo-2 with refitted coefficients.

Critical strengths
  1. Measurement-efficient pruning prediction replicated across two families.
  2. Strong simple baselines and careful distinction between interpolation and transfer.
  3. Transparent reporting of failed predictions and limited selection benefits.
Critical weaknesses
  1. Insufficient evaluations: emphasizes likelihood rather than downstream task accuracy
  2. Insufficient model family coverage: pruning validation covers two families; controlled distillation laws primarily cover Gemma-3
  3. Misaligned coverage: Descriptive coverage reaches 32B, but predictive validation covers only smaller models
  4. Work explore base/pretrained models only, leaving behavior of post-trained models untested
  5. Numerical selection provides limited improvement over simpler policies
Questions for authors
  1. Do measurement savings persist when predicting task accuracy or execution success directly?
  2. Can the predictors generalize to larger and post-trained models without refitting?
  3. When does numerical selection materially outperform configuration medians or fixed method priorities?
Reviewer_w6q9OpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores how models accuracies change with quantization, pruning, and distillation. It tries to quantify the degradation in accuracy using plots and various equations.

Critical strengths
  • Evaluated a wide range of model sizes, families, as well as training budgets
  • Evaluated multiple compression techniques: pruning, quantization, distillation
  • Figure 3 shows an interesting insight: larger models suffer more than smaller models from higher reuse (although it is not clear from the text if authors mentioned his insight. If the text was generated by an LLM it is unfortunate that it rambled about irrelevant details and didn't mention this important insight).
Critical weaknesses
  • The paper's writing is heavily condensed, heavily incoherent, and repeatedly uses custom terminology without defining them. I personally did not understand most of the paper and most of its Figures. Even if the paper has decent scientific contributions, it cannot be accepted in this current state. It will not be understood by readers. Authors need to re-write the paper thoroughly. Examples of problematic pharses or sentences:
    • Lines 111 to 114: "Pythia development states span three sizes at three training stages; held-out Pythia states up to 6.9B and OLMo-2 checkpoints (Team OLMo et al., 2024) test transfer, and a heterogeneous panel of twelve public models from 0.6B to 32B establishes the phenomena of Section 3" An example of an incoherent sentence that is difficult to understand.
    • Line 108: "The implementations are fixed by protocol (Appendix A): global magnitude pruning to a retained density d, with Wanda (Sun et al., 2024) as the second criterion". It is not clear here what was the 1st criterion when listing the 2nd criterion.
    • Equation 2: terms cc, and LobsL^{\text{obs}} are not defined
    • Lines 124 - 125: "each prediction keeps the status it was given, whatever the outcome." Awkward statements like this. How can a prediction "keeps the status it was given"?
    • "and the strongest baseline chosen inside the development folds is the comparator." Sounds like AI slop
    • "Model state sets the scale of the pruning response for mathematics and code". What do authors mean by "scale"?
    • Line 192: "the structure the predictors of Section 4 build", grammatical typo
    • Lines 243 to 245: "The observed structure differs by method: pruning shares one density shape across states, quantization acts through the full configuration, and distillation separates independent data, training exposure and evaluation distribution." Awkward sentences.
  • It seems that the QA benchmark or its evaluation approach is problematic: accuracy seems to improve significantly with quantization and pruning on multiple models. I am not sure if this is because the model performs better on such tasks or is it because the evaluation harness and the way answers are parsed are problematic. On the other hand, it seems that Math and Code are similar, and having plots for them are redundant. I would have preferred to measure average loss or perplexity on a diverse dataset as the sole criterion.
  • The equations to fit accuracy degradation were awkward, and used terms that were not defined. The authors did not provide a motivation behind their equation formats. Why didn't the authors use equations similar to "Scaling Laws for Precision" ( https://arxiv.org/abs/2411.04330 ).
Questions for authors
  • Questions about terminology:
    • Line 070: "Sharing the response structure across pruning": what is the "response"
    • Line 098: what is the "endpoint"
    • Line 193: what are "development states"
  • Equation 3: Please define each term in the equation
Reviewer_D8L2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies how pruning, quantization, and distillation affect mathematics, code, and question-answering tasks, using changes in reference-completion cross-entropy as the prediction target. Its main result is that sharing the pruning density-response shape across model states allows a predictor to use half the configuration measurements, with mathematics and code prediction errors within 0.020 nats/token of a regression fitted on the full grid. The authors evaluate held-out checkpoints and reproduce this result on OLMo-2 and under Wanda pruning, refitting the coefficients in each setting. They also compare numerical predictions with configuration medians and fixed method priorities for compression selection, finding that more detailed prediction does not always improve the choice.

Critical strengths
  1. The pruning measurement-efficiency result is concrete and supported by held-out evaluations. The compact predictor uses half the configuration measurements while keeping mathematics and code prediction errors within 0.020 nats/token of the full-grid regression. The result is reproduced on OLMo-2 and under Wanda pruning, with coefficients refitted in each setting. These experiments provide useful evidence that the shared response structure can reduce the cost of building a predictor across the tested settings.

  2. The comparisons with simple baselines provide useful insight into when numerical prediction helps. The authors compare their predictors with configuration medians and fixed method priorities, and report cases where the simpler approaches perform equally well. I found this analysis valuable because it distinguishes the benefit of considering multiple compression methods from the additional benefit of predicting their losses precisely.

Critical weaknesses
  1. The benefit for compression choices based on task accuracy remains limited. In Table 24, the Pythia 410M student associated with the 1.4B reference has lower reference-completion loss than the quantization-only choice, but also lower 96-token exact match. The authors acknowledge this mismatch. However, it leaves the practical value of loss-based selection less clear when the goal is to preserve answer accuracy.

  2. The generation results in Tables 24 and 28 need clarification. The distilled-student results in Table 24 differ from the “Original” column in Table 28 for 96-token EM and F1, although loss and 32-token EM agree. The captions suggest that these are the same models evaluated on the same 384 examples. I could not determine whether the difference comes from answer extraction, scoring, or another evaluation setting. This makes it difficult to reconcile the EOS control with the preceding generation evaluation.

Questions for authors

Could you explain the differences between the distilled-student results in Table 24 and the “Original” column in Table 28? The 96-token EM values are 0.091 vs. 0.076 for the Pythia 410M student, and 0.115 vs. 0.083 for the Gemma 270M student. The corresponding F1 scores also differ, while loss and 32-token EM agree. Were different answer-extraction or scoring rules used? Please also clarify whether the original and EOS-supervised students, and the quantized and initial-student baselines used in the comparisons in Appendix H.2, were evaluated under the same protocol.

Reviewer_SpoLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies, within a unified framework, the impact of pruning, quantization, and distillation on the loss of three capabilities — mathematics, code, and question answering — and proposes simple relations that predict the post-compression loss: a shared pruning-density shape can halve the measurements needed for fitting, replicated on a second family and a second pruning criterion; and in distillation, the cost of data reuse emerges across distributions. Method-selection experiments show that a fixed method priority achieves the same effect as numeric prediction.

Critical strengths

The paper handles pruning, quantization, and distillation under a single unified metric and protocol, whereas prior compression-law work typically treats only one of these categories. The experimental coverage is also fairly complete: a controlled panel in which scale and training stage can be varied independently, a preregistered replication on a second model family, a replication on a second pruning criterion, a heterogeneous panel of over ten public models, and method-selection validation on independent reference models plus generation readouts on fresh questions. This scale gives the paper's several negative conclusions — that coefficients do not transfer across families, that source information on new states is no better than the configuration median, and that loss ordering and generation ordering can diverge — sufficient support rather than an artifact of a single panel. It should be noted that this breadth is at the category level: the implementation used for each class of method is the simplest representative.

Critical weaknesses

W1. The capability division is an untested modeling assumption, yet the core findings" of depend on it

The paper treats Math, Code, and QA as given natural kinds, and on the basis of this division states response structure, laws, and selection rules, yet it never tests whether the division actually holds in the sense of compressed responses. The paper's own results do not support it: under pruning, the math and code responses differ only by a scale factor, which by the paper's own standard means they are not two distinct capabilities; and the claim that "the order is not fixed" is driven almost entirely by the single endpoint QA — precisely the endpoint the paper itself concedes shows a dissociation between loss and behavior and is sensitive to the reference format, so the alleged order variation is more plausibly a difference in each family's likelihood over a reference format. The paper contains no per-sample between-group versus within-group variance decomposition of responses. If the within-group variance is of the same order of magnitude as the between-group gaps, then "compression affects capabilities unevenly and the order is not fixed" is a trivial fact that any division would exhibit, and the descriptive level chosen for per-capability laws and per-capability selection — and indeed the very name "capability scaling-down laws" — all lose their justification.

I should note that the view of "model capabilities" expressed above is my own position: whether the internal capability structure of large models aligns with human task taxonomies is an open question. I believe the existing evidence leans toward misalignment, and I hold that this judgment is correct. It is not difficult to test, and the cost is low: perform unsupervised co-clustering on the response profiles of the paper's existing 192 probes across all states and configurations, and see whether it spontaneously recovers the human trichotomy. My expectation is that it will not. If it indeed does not, this negative result is by itself more informative than any single "law" in the main text, because it directly answers the paper's premise-level question of whether "capability" is the correct descriptive level for compressed responses; if it does, only then does the division acquire the empirical support it currently completely lacks. Either way, this experiment should appear in the paper.

W2. Section 3 takes the low-information "difference" conclusions as its main thread, while the genuinely informative shared-structure results are relegated to the appendix

The difference conclusions are almost unfalsifiable, and no one had predicted the opposite in advance; the sharedness and invariance conclusions are the risky predictions, yet the paper's narrative inverts exactly their relative positions. That "pruning and quantization responses differ in shape" is expected from the outset in each of their native coordinates; the informative question is whether they coincide once moved to a common coordinate system, and this is not tested, even though the appendix in fact already contains a method-independent second-order account giving an affirmative answer. That "model state determines the pruning scale of math/code" is largely a normalization effect of damage scaling with baseline loss, with the independent contributions of model scale and training data being small and non-extrapolable; its nontrivial version is effectively a null result. And "ineffective for QA" is mechanical: responses that change sign cannot be fit by a shared shape multiplied by a scale. The result is that the "findings" a reader takes away from the main text have almost no predictive power, the significance of the contributions is underestimated, while the over-packaged "laws" do not stand.

Questions for authors

see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.