acceptodds
Under review as a conference paper at ICLR 2027

Behavioral Capacity Certificates for Quantized Language Models

Abstract

Activation and key–value cache precision change what a quantized language model computes without altering its stored weights. Direct weight-code bounds, however, assign identical complexity to deployments that behave differently and charge separately for weight codes that behave identically. Behavioral Capacity Certificates (BCC) charge for behavior using the aggregate prior mass of complete implementations—weights, scales, activation and cache rules—that induce the same bounded loss. When quantization merges implementations, this shared mass lowers the complexity penalty, and a break-even law determines when the saving survives the cost of validating it. BCC supports a three-step deployment workflow, and our experiments verify each step. First, a forward-only screen shortlists per-layer bit-widths by how often candidate perturbations preserve the reference predictions, with quality comparable to Hessian-guided selection at lower preprocessing cost. Second, margin-certified cells identify weights that can be pruned or sign-flipped without changing the deployed behavior: every permitted combination preserves all declared predictions, and on OLMoE-1B-7B and SmolLM2-1.7B, independent probes bound the probability that any permitted combination changes a prediction on new text. Third, BCC bounds the population loss of the deployed model, nonvacuously for complete decoders and more tightly than the compressed-code route. At equal cache memory, giving keys higher precision than values yields lower NLL and higher prediction agreement on GPT-2, Qwen2.5, and SmolLM2, together with a tighter complexity bound in the GPT-2 audit.

Primary area:
learning theory
Keywords:
Weight-Activation Quantization, Generalization Certificates, PAC-Bayes Bounds, Mixed-Precision Allocation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_K5mPOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

BCC aggregates prior mass over implementations sharing the same bounded loss function, aiming to tighten risk certificates. It also constructs edit families and accounts for their transfer beyond construction data.

Critical strengths
  1. Certifying shared behavior rather than individual parameter codes is a well-motivated perspective for quantized models.

  2. The framework explicitly accounts for independent validation and certifies simultaneous output-head edits, providing guarantees beyond testing individual perturbations.

Critical weaknesses
  1. Practical gains at modern-model scale remain insufficiently established. Useful complete-code bounds are demonstrated on small decoders, whereas large-model experiments mainly certify restricted output-head edits rather than useful full-model complexity reductions.

  2. The certificates depend on the declared prior, loss, and input population. Although the appendix acknowledges these distinctions, the additional deployment value of the resulting complexity reductions remains unclear, particularly beyond finite-corpus prediction preservation.

Also, the presentation needs improvement: figures such as Figure 2 contain small text and visual elements that are difficult to read at normal viewing size, making the conceptual explanation harder to follow.

Questions for authors
  1. Can the authors demonstrate a deployment decision or useful modern-model risk bound that improves specifically because of behavioral aggregation?

  2. How sensitive are the practical conclusions to reasonable alternative priors and to the choice between prediction loss and bounded NLL?

I would be willing to raise my rating if the authors provide convincing answers to these two questions, supported by experimental results clarifying the practical value and sensitivity of behavioral aggregation, and improve the presentation, particularly the readability of Figure 2 and the explanation of the certificates’ scope.

8 comments
AuthorsOpenReview ↗

Response to Reviewer K5mP (1/5): modern-scale gains and the role of aggregation (Comments 1–2, Question 1)

Response to Comments 1–2 and Question 1. We understand the concern about practical gains at modern-model scale and deployment value beyond finite-corpus preservation. The new results address compression beyond output-head edits and a deployment decision that improves specifically through aggregation. The clearest gain is on Llama-3.1-8B: BCC certifies a 4.0 GB deployment at an illustrative 2% harmful-change requirement, compared with 4.4 GB for the strongest control—a 9.1% storage reduction.

Model and taskStorageBCC risk boundStrongest control
Llama-3.1-8B / TriviaQA4.0 GB1.730%2.371%, validation-loss allocation
Phi-3.5-mini / ARC-Challenge2.332 GB9.5988%10.0705%, fixed champions
Phi-3.5-mini / MathQA2.332 GB8.5035%8.6061%, pooled calibration

A harmful change occurs when the reference answers correctly and the compressed deployment answers incorrectly. These percentages bound that event over all questions from the declared task population. For Phi-3.5-mini, BCC supports deployment selection on science question answering (ARC-Challenge) and mathematical reasoning (MathQA) using certified harmful-change risk. The illustrative 10% requirement means certifying, with the stated confidence, that compression turns a correct reference answer into an incorrect answer on at most 10% of all questions from the declared task population. At the same 2.332 GB budget, only BCC meets this requirement on ARC, while it provides the tightest MathQA guarantee, where all routes already meet the requirement (Table 36, p. 46).

Modern-model gains beyond head edits—Comment 1. Both Llama-3.1-8B and Phi-3.5-mini experiments quantize attention and feed-forward layers throughout the transformer. On Llama, all methods share candidate banks and an independent audit of 10,000 draws, with 95% simultaneous coverage over 20 frozen finalists. On Phi, the BCC envelopes have 66/1,172 harmful changes on ARC and 180/2,985 on MathQA; the reported bounds use 98.75% simultaneous confidence accounting. At an illustrative 10% requirement, only BCC passes on ARC; all routes pass on MathQA. Settings and comparisons appear in Table 35, p. 45, and Table 36, p. 46 (of the revised manuscript).

Deployment benefit specifically from aggregation—Comment 2 and Question 1. At 4.0 GB, Llama’s aggregation-guided search yields 137 harmful changes, versus 200 for the same search using individual prior mass. Their bounds are 1.730% versus 2.426%, under the same audit. Aggregation therefore changes the selected deployment and its certified risk. Validation-loss allocation gives 195 harmful changes and a 2.371% bound. The smallest passing sizes are 4.0 GB for BCC versus 4.4–4.8 GB for the controls (Table 6b, p. 10).

Llama-3.1-8B / TriviaQA search settingHarmful changes at 4.0 GBRisk upper bound at 4.0 GBSmallest certified size at illustrative 2%
Aggregation off: individual prior mass200/10,000 (2.00%)2.426%4.4 GB
Aggregation on: aggregated prior mass137/10,000 (1.37%)1.730%4.0 GB

This ablation concerns quantization throughout the transformer. “Aggregation off” uses individual implementation prior mass in the selection score; “aggregation on” uses the summed prior mass of the declared behavioral cell. Both searches use the same candidate banks, and their selected deployments receive the same independent binomial audit. Aggregation changes the selection score and consequently which deployment is chosen: at 4.0 GB, only the aggregation-guided selection meets the illustrative 2% requirement.

Why this matters: the practical choice is between deploying at 4.0 GB or 4.4 GB. Each method selects from the same 64 candidates at 4.0 GB; among these selections, only the aggregation-guided pick meets the illustrative 2% requirement. The strongest controls first pass at 4.4 GB, requiring 10% more storage. Aggregation balances measured harm with how many candidates share the same harmful-change pattern on the 512 selection questions. On 10,000 independent audit draws, its selected deployment produces 137 harmful changes versus 200 for the same search without aggregation—31.5% fewer.

A second example is SmolLM2-1.7B. Charging each implementation separately, 8,192 probes certify at most 105 editable head coordinates at 1% risk; treating the family as one behavior certifies 131,072 on mathematical text and 262,144 on code (Appendix B.23, p. 49). This is an output-head result, cited here only for the role of aggregation.

AuthorsOpenReview ↗

Response to Reviewer K5mP (2/5): sensitivity to priors and losses (Question 2)

Response to Question 2: Thank you for asking us to quantify this. Across two tasks, three priors, and three endpoints on Phi-3.5-mini, BCC gives a tighter bound than direct coding in all 18 settings. Against the strongest of seven controls it wins 10, ties 5, and loses 3. Under uniform and precision-weighted priors it beats every control for harmful change and added NLL (8 of 8) and ties random matching for answer error (4 of 4). Under the champion-heavy prior it wins 2, ties 1, and loses 3 (Appendix B.25 (Tables 45–46, pp. 53–54)).

We use the same 2.332 GB cap, 1,172 ARC-Challenge questions, and 2,985 MathQA questions; pooled routes include 256 additional development questions. Harmful change counts reference-correct/compressed-incorrect answers, while answer error measures absolute error. Added NLL is the positive difference between compressed and FP16 NLL after clipping each at 10 nats. All entries are upper bounds; lower is better.

ARC-Challenge

PriorEndpointDirectBCCBest control (of seven)Outcome
UniformHarmful change (%)10.14389.5988FC: 10.0705Win
UniformAnswer error (%)25.279624.5613RM: 24.5613Tie
UniformAdded NLL (nats)0.27490.2516PD: 0.2544Win
Precision-weightedHarmful change (%)10.07499.5226PD: 10.0313Win
Precision-weightedAnswer error (%)25.184824.4545RM: 24.4545Tie
Precision-weightedAdded NLL (nats)0.27070.2472PD: 0.2508Win
Champion-heavyHarmful change (%)10.32389.8924FC: 10.0705Win
Champion-heavyAnswer error (%)25.645824.9707RM: 24.9707Tie
Champion-heavyAdded NLL (nats)0.28690.2692PD: 0.2544Loss

MathQA

PriorEndpointDirectBCCBest control (of seven)Outcome
UniformHarmful change (%)8.77108.5035PC: 8.6061Win
UniformAnswer error (%)76.505876.2001RM: 76.2001Tie
UniformAdded NLL (nats)0.24710.2299PC: 0.2396Win
Precision-weightedHarmful change (%)8.73168.4605PD: 8.5950Win
Precision-weightedAnswer error (%)76.450976.1386RM: 76.1386Tie
Precision-weightedAdded NLL (nats)0.24480.2275PC: 0.2396Win
Champion-heavyHarmful change (%)8.92358.6691PC: 8.6061Loss
Champion-heavyAnswer error (%)76.716576.4346PD: 76.2971Loss
Champion-heavyAdded NLL (nats)0.25580.2394PC: 0.2396Marginal win

FC: fixed champions; RM: random matched envelope; PD: pooled direct; PC: pooled calibration. The seven controls are these four, direct coding, canonical coding, and the format envelope. Bounds are rounded upward.

Uniform coding assigns each of 768 implementations mass 1/7681/768. Precision weighting gives weight 2 to 256 predefined lower-precision implementations and weight 1 to the remaining 512, giving the relevant candidates mass 1/5121/512. The champion-heavy prior is pH(e)=0.2768+0.824 1{e∈H}p_H(e)=\frac{0.2}{768}+\frac{0.8}{24}\ \mathbf{1}\lbrace e\in H\rbrace where the 24 champions are selected on development questions and frozen before confirmation. BCC’s cells contain 40 implementations on ARC and 21 on MathQA, all outside HH. Their masses fall fivefold relative to uniform coding, adding ln⁡5\ln 5 to the complexity charge; all three losses occur under this prior. Pooled direct retains a uniform prior in these rows because its sample includes the champion-selection questions. The answer-error ties involve different BCC and random cells with equal size, envelope error counts, prior mass, and confidence allocation. Pooling improves their bounds over direct coding, while behavioral grouping adds no further benefit on this endpoint. MathQA’s approximately 72% observed compressed-model error limits the practical strength of its absolute-error result. Its champion-heavy NLL win is marginal.

Coding priors over weights show additional aggregation gains on the exact tasks: 1.344–1.954 bits beyond the cheapest individual realization under histogram-Huffman coding (Table 8, p. 23) and 1.613–2.436 bits under the sparse/uniform ternary-mixture prior (Table 34, p. 43). An implemented interval-mass encoder also recovers 98.29–99.55% of the BCC saving on six complete decoders; an interval-index code achieves shorter records under its alternative coding distribution (§B.24, Table 41, p. 51).

AuthorsOpenReview ↗

Response to Reviewer K5mP (3/5): Question 2, continued (bounded NLL on SmolLM2)

On SmolLM2-1.7B, BCC certifies 131,072 math and 262,144 code coordinates, versus 4,096 for the activation- and magnitude-based controls—32× and 64× gains. Using 8,192 independent confirmation prefixes per domain, the BCC continuation-risk bounds are 0.664115% and 0.776436%, and added clipped-NLL bounds are 0.023617 and 0.024253 nats/token. Both satisfy the 1% continuation-risk and 0.10-nat added-NLL requirements. The capacities are set by the continuation requirement; the NLL requirement is not binding at these sizes (§B.23, Table 39, p. 49).

Reviewer_7HNQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Behavioral Capacity Certificates (BCC), which group quantized implementations with the same loss function and use their combined prior probability to tighten generalization bounds. It gives ways to certify these groups and accounts for validation when they are inferred from samples. Experiments show tighter full-model bounds on small decoders and bound the probability of prediction changes from output-head edits on OLMoE and SmolLM2, alongside exact finite-domain results on GPT-2.

Critical strengths
  1. Careful treatment of behavioral equivalence and generalization. The paper distinguishes exact equivalence from agreement observed on a finite sample, and explicitly accounts for the independent validation needed in the latter case.

  2. Concrete constructions with supporting experiments. The margin-based certificates cover simultaneous weight edits without enumerating every combination, with results on GPT-2, OLMoE, and SmolLM2 under the declared evaluation settings. The complete small-decoder experiments also improve on the tested lossless-compression bounds.

Critical weaknesses
  1. The complete-model gains are measured against codes that reconstruct the original weights exactly. A practical behavior-preserving encoder might recover some of the same savings, especially for the embedding intervals used here. Without this comparison, it is hard to judge how much additional benefit the proposed framework provides.

  2. The practical benefit at larger model scales remains limited in the evidence presented. Useful full-model loss bounds are demonstrated on small decoders, while the modern-model results mainly certify output-head edits.

Questions for authors
  1. Could the gains in Table 2 also be obtained with a behavior-preserving compression scheme? Your baseline still has to reconstruct the original weights exactly.
  2. How much do the large-model edit families actually tighten a full-model loss bound? The results seem to establish safe head edits more clearly than useful bounds for the whole model.
  3. In Table 3, when would the certificate change the deployment choice compared with looking at memory and held-out quality alone?
  4. What is the total cost of the proposed workflow? Table 5 reports screening time, but it's hard to tell how much cell construction and validation add in practice.
Reviewer_P7PpOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a framework that certifies quantized language models based on their deployed behavior. The proposed certificate aggregates prior probability across complete implementations (including weights, activation precision, and cache rules) that induce the same bounded loss function, reducing the complexity cost when different implementations behave the same way. The paper derives generalization and perturbation stability guarantees and identifies when complexity savings outweigh the costs of independent validation. The authors apply this framework to precision screening, certification of prediction-preserving weight edits, and KV-cache allocation, with experiments demonstrating tighter bounds than those of the evaluated compression baselines and providing guidance for quantized deployment.

Critical strengths
  • The authors propose a margin-based method for certifying large families of simultaneous output-head weight edits, including sign flips and pruning. The guarantees cover every permitted combination, with independent probes bounding prediction-disagreement risk on independently sampled evaluation prefixes.

  • The paper presents interesting findings: The authors show that the basic mechanism behind BCC works exactly on an exhaustively checked example. Multiple numerical implementations can share one behavior and therefore one smaller complexity charge. The authors also show empirically that allocating more precision to keys yields higher quality within the same memory budget. They further demonstrate that the output head contains substantial redundancy: many carefully selected weights can change together while preserving the predicted tokens.

Critical weaknesses
  • The OLMoE and SmolLM2 models' certification experiment is evaluated only on WikiText, which makes it unclear whether comparable certified edit capacity holds for other domains and tasks, such as code, reasoning, and instruction following.

  • While the experiments include modern language models (OLMoE and SmolLM2), their guarantees address selected output heads while the backbone is kept fixed, and they provide informative guarantees that account for the entire model only for small decoders.

  • The precision/allocation evaluation focuses on weight-bit selection with FP32 activations, leaving the effectiveness and computational cost of joint weight-activation allocation insufficiently assessed.

Questions for authors

Q: Could the authors extend the edit-certification evaluation beyond WikiText to additional domains or tasks, such as code, mathematical reasoning, or instruction following?

3 comments
AuthorsOpenReview ↗

Response to Reviewer P7Pp (1/3): evaluation beyond WikiText and results beyond output-head edits

Thank you for identifying these concerns. We understand the need to establish broader task coverage, practical value beyond output-head edits, and the effectiveness and cost of joint weight–activation allocation. The additional results address these points directly: 32× and 64× larger certified edit families on mathematical text and code, a smaller certified Llama deployment, and improved Qwen accuracy over every control sharing the same weight and activation budgets. accordingly, we submitted the revised manuscript.

1. Evaluation beyond WikiText: code, mathematical reasoning, and question answering

Yes. We extend edit certification to mathematical text, Python code, and complete question-answering responses.

On SmolLM2-1.7B, the largest tested families satisfying the same 1% continuation-disagreement requirement are:

DomainBCC editable coordinatesActivation-based selectionMagnitude selectionBCC risk upper bound
Mathematical text131,0724,0964,0960.6641%
Python code262,1444,0964,0960.7764%

These are 32× and 64× capacity gains on the tested grids, using 8,192 independent confirmation prefixes per domain and at least 95% simultaneous confidence across both domains. Each certificate covers every permitted combination of pruning, sign changes, and magnitude reductions within the selected family. Both families also satisfy the added clipped-NLL requirement, with upper bounds below 0.0243 nats/token. These results appear in Appendix B.23, Table 39, p. 49.

We also evaluate Qwen2.5-1.5B-Instruct on GSM8K and BoolQ, certifying preservation of the complete capped greedy response. On GSM8K, BCC permits 1,024 editable coordinates versus 256 for magnitude and random selection—a 4× gain. On BoolQ, all three selectors reach the tested ceiling of 32,768 coordinates. The BCC response-disagreement bound is 0.7179% on each task, based on 1,024 independent held-out draws per task and simultaneous 95% confidence. Response caps are 512 tokens for GSM8K and 128 for BoolQ (Table 6a, p. 10; Appendix B.22, Table 38, p. 48).

These results establish certified behavioral preservation across additional domains and instruction-tuned tasks. Their practical value is permission to modify many weights jointly while bounding the probability that any permitted modification changes the reference continuation or response.

2. Practical value beyond output-head edits and complete-model accounting

The additional modern-model experiments quantize attention and feed-forward layers throughout the transformer. For pretrained checkpoints, we certify a practitioner’s deployment question: when the checkpoint is given, how often does compression turn a correct reference answer into an incorrect answer?

On Llama-3.1-8B-Instruct/TriviaQA, aggregation-guided selection at 4.0 GB gives 137 harmful changes in 10,000 audit questions, with a 1.730% risk upper bound. The same search using individual prior mass gives 200 harmful changes and a 2.426% bound; validation-loss selection gives 195 and 2.371%. All methods share the candidate banks and independent audit, with 95% simultaneous coverage over 20 frozen finalists.

At an illustrative 2% risk requirement (certifying, with the stated confidence, that compression turns a correct reference answer into an incorrect answer on at most 2% of all questions from the declared task population), BCC selects 4.0 GB, whereas the strongest control first passes at 4.4 GB—9.1% less stored-weight memory. Here, aggregation changes the deployment selected under a common risk requirement (Table 6b, p. 10; Appendix B.19, Table 35, p. 45).

We separately evaluate complete reconstruction-code bounds on a TinyStories decoder with 25,482,496 dense parameters in a 32,768-dimensional seeded affine subspace. Every route charges the coordinates, scales, seeds, and decoder specification needed to reconstruct the entire model. BCC reduces the expected clipped byte-NLL upper bound from 3.8220 nats for the strongest direct-code control to 3.5104 nats, an 8.15% improvement, with joint confidence of at least 95%. The measured held-out byte NLL is 2.193 nats; uniform byte prediction gives approximately 5.5452 nats (Appendix B.21, Table 37, p. 48).

The edit-family certificates provide a further operational benefit: they identify which weights may change together. For the SmolLM2 mathematical-text and code experiments, the specified uniform per-implementation Occam–KL calculation could certify at most 105 independently editable sign coordinates at the same 1% requirement, even with zero observed failures. The family-wide audit certifies the much larger sets above without enumerating their members (Appendix B.23, p. 49).

AuthorsOpenReview ↗

Response to Reviewer P7Pp (2/3): joint weight–activation allocation, cost, and presentation

3. Effectiveness and computational cost of joint weight–activation allocation

We now evaluate joint allocation on Qwen2.5-1.5B-Instruct across all 28 transformer blocks. Weight precision and linear-input activation precision are selected jointly from 4 and 8 bits, covering attention and feed-forward projections.

At the budget allowing 21 W8 blocks and 21 A8 blocks, joint behavioral screening gives:

Allocation methodBoolQ accuracy ↑ARC-Challenge accuracy ↑BoolQ harmful-change bound ↓ARC harmful-change bound ↓
Joint BFMS71.68%54.39%18.12%32.29%
Joint MSE50.29%22.75%43.89%62.96%
Weight-only screening, fixed activation placement55.86%29.88%38.39%57.98%
Best of three random allocations, separately per task61.04%32.91%34.54%54.21%

All listed methods share the separate weight and activation budgets and have approximately 1,362.48 MiB encoded checkpoints. Evaluation uses 1,024 independent draws per task from the declared held-out pools, with at least 95% simultaneous confidence. Joint BFMS therefore improves accuracy and harmful-change bounds over every matched-budget control at this setting.

BFMS makes the screening stage practical by using forward-pass prediction retention to rank precision choices. Independent evaluation then certifies the selected complete deployment, including interactions among its quantized layers.

Screening is a one-time shared cost of 145.49 minutes, with scores reused across budgets and selectors. On one NVIDIA T4, the screen evaluates 168 local block/format settings using forward passes only. Assigning weight and activation precision across all 28 transformer blocks then takes just 0.053 seconds. At the 21 W8 / 21 A8 budget, the primary joint-BFMS deployment takes 16.95 seconds to quantize, 348.02 seconds to validate independently, and 22.27 seconds to serialize—6.45 minutes in total after screening. The full 19-deployment experiment, including shared screening, completed in 267.29 minutes.

The screen replaces a search that cannot be run. At the 21/21 budget alone there are about 1.4×10121.4\times10^{12} joint allocations. Evaluating a single one costs about six minutes, so the entire screening time would pay for roughly two dozen direct evaluations.

The return is large. At equal memory, the screened allocation is 10.6–21.4 points more accurate on BoolQ and 21.5–31.6 points on ARC-Challenge than the three matched-budget controls.

These times are conservative. They come from an FP32 quantize/dequantize simulation on a single T4, with no backpropagation and no fine-tuning. The per-block screens are independent, so they parallelize directly across GPUs. These are measured costs for an offline compression-and-certification workflow using FP32 quantize/dequantize simulation. The benefit is a substantially better allocation under the declared resource constraints, accompanied by an explicit risk assessment.

4. Presentation and interpretation of the guarantees

Table 1, p. 6, identifies each experiment’s model, certified quantity, input population, and code charge. It makes clear which results charge complete-model reconstruction and which condition on a supplied checkpoint or fixed edit family.

Figure 2, p. 8, explicitly labels joint edits and cache-specific prediction-preserving sets. Its caption explains that the shared cell preserves each cache setting’s own reference predictions. The main discussion in Section 4, pp. 9–10, connects the certificates to concrete decisions: selecting precision allocations, identifying jointly editable weights, and choosing a compressed deployment under a risk requirement.

AuthorsOpenReview ↗

Response to Reviewer P7Pp (3/3): new Phi-3.5 results on science and mathematical questions

New results: BCC improves Phi certification on science and mathematical questions

On Phi-3.5-mini, BCC gives the tightest harmful-change bounds among the eight evaluated certification routes at a common 2.332 GB stored-weight cap. The experiments cover science question answering on ARC-Challenge and mathematical reasoning on MathQA, with quantization across transformer layers.

TaskBCC risk upper boundStrongest of seven controls
ARC-Challenge9.5988%10.0705%, fixed champions
MathQA8.5035%8.6061%, pooled calibration

The BCC envelopes contain 66 harmful changes in 1,172 confirmation questions on ARC and 180 in 2,985 on MathQA. The bounds use 98.75% simultaneous confidence accounting.

The practical win is a better certification decision at the same storage budget. An illustrative 10% requirement demands a certified upper bound of at most 10% on the probability that compression turns a correct reference answer into an incorrect answer, counting all questions in the declared population. Only BCC meets this requirement on ARC; every control exceeds it. On MathQA, all routes meet 10%, and BCC provides the tightest guarantee.

Reference: revised manuscript, Appendix B.20, Table 36, p. 46.

New results: aggregation selects a smaller certified Llama deployment

On Llama-3.1-8B / TriviaQA, aggregation enables a 4.0 GB deployment to meet an illustrative 2% harmful-change requirement, while the strongest controls first pass at 4.4 GB—a 9.1% reduction in stored-weight memory. These experiments quantize attention and feed-forward layers throughout the transformer.

A harmful change occurs when the reference answers correctly and the compressed model answers incorrectly. The 2% requirement means certifying that this happens on at most 2% of all questions from the declared population, with the stated confidence.

Selection methodHarmful changes at 4.0 GBRisk upper bound at 4.0 GBSmallest certified size at 2%
Reconstruction-error allocation240/10,0002.862%4.8 GB
Hessian-based allocation215/10,0002.590%4.6 GB
Validation-loss allocation195/10,0002.371%4.4 GB
Same search, aggregation off200/10,0002.426%4.4 GB
Same search, aggregation on137/10,0001.730%4.0 GB

All methods select from the same bank of 64 candidates at each storage budget. The 20 finalists are frozen before an independent audit of 10,000 draws, with 95% simultaneous coverage.

The aggregation ablation directly shows its contribution to selection: replacing individual prior mass with aggregated behavioral-cell mass changes the chosen deployment and reduces observed harmful changes from 200 to 137—31.5% fewer. Both selections receive the same independent audit. Among the evaluated selections at 4.0 GB, only aggregation meets the illustrative 2% requirement.

Reference: revised manuscript, Table 6b, p. 10; Appendix B.19, Table 35, p. 45.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Response to Reviewer 7HNQ (1/3): behavior-preserving encoders and complete-model bounds (Questions 1–2)

Thank you for your comments. We understand these concerns. The new results establish large families of jointly permitted edits, realizable compression, a smaller certified Llama deployment, and measured workflow costs. We address each question below.

References below use the revised manuscript. Submitted Tables 2 and 3 are now Tables 3 and 4, respectively, both on p. 8.

1. Could behavior-preserving compression recover the reported savings?—Weakness 1 and Question 1.

Yes. We implemented complete behavior-preserving encoders, and the interval-mass encoder recovers 98.29–99.55% of the BCC complexity saving, including metadata and padding. Its records lie only 14.36–24.59 bits above the BCC complexity.

For example, on seed 191 at W4/A4, exact reconstruction requires 53,632 bits, the interval-mass encoder uses 50,440 bits, and BCC assigns 50,425.6442 bits. Every decoded model preserves all logits, probabilities, and losses over the entire declared prefix population.

The interval-index encoder achieves 50,152 bits in this example and provides shorter records and tighter bounds in all six cases under its alternative coding distribution. We believe this directly addresses the requested baseline: the comparison shows how much of the behavioral saving an actual encoder realizes and where a different coding distribution improves it further (Appendix B.24, Tables 41–43, p. 51).

The two are complementary: BCC identifies and certifies the intervals of behavior-preserving codes, and the encoder is what turns that certified mass into an actual record. Proposition A.1 states that an ideal behavior-preserving code attains the BCC complexity, and the experiment confirms this to within 14.36–24.59 bits.

2. How much do the results improve a complete-model bound?—Weakness 2 and Question 2.

The edit families are not full-model loss bounds; they certify which weights may change together. Our main deployment aim is to certify a compressed implementation: whether it meets a specified behavioral-risk requirement and which further modifications can be permitted under that guarantee.

For a practitioner compressing a released model, the checkpoint is given. The practical questions are whether compression harms its answers and whether additional pruning or sign changes remain acceptable. Our certificates support these decisions by bounding harmful changes on independent questions, identifying jointly editable weights, and helping select a smaller deployment that satisfies a risk requirement.

On SmolLM2-1.7B, the certified families cover 131,072 coordinates on mathematical text and 262,144 on code, with continuation-risk bounds of 0.6641% and 0.7764%, using 8,192 independent prefixes per domain. Under the specified uniform per-implementation Occam–KL rule, the same sample size and confidence allocation permit at most 105 coordinates at the 1% requirement, even with zero observed failures (Appendix B.23, p. 49).

Certifying the frozen family’s maximum-disagreement event jointly avoids this per-implementation charge. The activation and magnitude controls also use joint family certification and reach 4,096 coordinates; BCC’s coordinate selection yields the separate 32× and 64× capacity gains.

For complete reconstruction, the TinyStories experiment improves the strongest direct bound from 3.8220 to 3.5104 nats, a reduction of 0.3116 nats, or 8.15%. The decoder has 25,482,496 reconstructed dense parameters in a 32,768-dimensional affine subspace. Its code charges the coordinates, scales, seeds, and decoder specification. The bound concerns expected byte NLL clipped at eight nats, with at least 95% joint confidence (Table 37, p. 48).

The two analyses are compatible. A cell’s prior mass is at least that of its highest-mass member, so aggregation cannot increase the complexity term; the certificate accounts for additional loss and validation costs. Aggregation can therefore be applied on top of a complete-model code, with additional savings after Huffman and mixture coding shown in Tables 8 and 34, pp. 23 and 43.

Our new BCC results extend beyond output-head edits. The Llama-3.1-8B and Phi-3.5-mini experiments quantize attention and feed-forward layers throughout the transformer. Using independent questions, they bound the risk of harmful answer changes relative to the supplied pretrained checkpoints (Table 6b, p. 10; Tables 35–36, pp. 45–46).

AuthorsOpenReview ↗

Response to Reviewer 7HNQ (2/3): when certification changes the deployment choice (Question 3)

3. When does certification change the deployment choice?—Question 3.

It changes the choice when the deployment must satisfy a population-risk requirement. On Llama-3.1-8B/TriviaQA, validation-loss allocation has 1.95% observed harmful changes at 4.0 GB, but its 2.371% upper bound does not certify an illustrative 2% requirement. BCC’s 1.730% bound does.

Selection methodObserved damage at 4.0 GBRisk upper boundSmallest certified size at illustrative 2%
Validation-loss allocation1.95%2.371%4.4 GB
Same search, individual prior mass2.00%2.426%4.4 GB
BCC, aggregated prior mass1.37%1.730%4.0 GB

A harmful change means a correct reference answer becomes incorrect; all questions enter the denominator. Methods share candidate banks and an independent audit of 10,000 draws, with 95% simultaneous coverage over 20 frozen finalists. The same-search comparison isolates aggregation’s contribution: 200 versus 137 harmful changes. BCC permits 9.1% less storage than the strongest control at this requirement (Table 6b, p. 10; Table 35, p. 45).

Held-out damage provides an empirical estimate. The certificate additionally accounts for finite-sample uncertainty and simultaneous coverage over the 20 frozen finalists. Consequently, an observed damage rate below the required threshold can still have an upper bound above it. This explains why validation-loss selection’s observed 1.95% does not certify the illustrative 2% requirement, whereas BCC’s 1.730% upper bound does.

Phi supplies a second decision example at 2.332 GB: BCC’s ARC bound is 9.5988%, versus 10.0705% for the strongest control, so only BCC passes an illustrative 10% requirement. On MathQA, the bounds are 8.5035% versus 8.6061%, with all routes passing 10% (Table 36, p. 46). Both the Llama and Phi experiments quantize transformer layers throughout the model. We believe these comparisons directly answer when certification changes the deployment decision.

AuthorsOpenReview ↗

General response to all reviewers: summary of revisions

We thank the reviewers for their careful and constructive comments. The revision adds experiments and clarifications that address each main concern. We summarize them here; details are in the individual responses.

1. Evaluation beyond WikiText (P7Pp). Edit certification now covers mathematical text, Python code, and question answering. On SmolLM2-1.7B the certified families hold 131,072 and 262,144 coordinates, versus 4,096 for both controls (32× and 64×), with risk bounds below 1% (Appendix B.23). On Qwen2.5-1.5B we certify complete GSM8K and BoolQ responses (Appendix B.22).

2. Modern-model results beyond output-head edits (all reviewers). We quantize transformer layers throughout Llama-3.1-8B and Phi-3.5-mini and bound how often compression turns a correct answer into an incorrect one. On Llama, the aggregation-selected deployment meets an illustrative 2% requirement at 4.0 GB, versus 4.4 GB for the strongest control (Appendix B.19, pp. 44–45, Table 35 and Appendix B.20, pp. 45–47, Table 36).

3. Gains specifically from aggregation (K5mP, 7HNQ). With the search otherwise unchanged, aggregated prior mass gives 137 harmful changes per 10,000 on Llama, versus 200 for individual prior mass (Appendix B.19, pp. 44–45, Table 35). On SmolLM2, charging each implementation separately certifies at most 105 coordinates from the same sample; treating the family as one behavior certifies 131,072 or more.

4. Complete-model bounds and realizable compression (all reviewers). For a TinyStories decoder with 25.5M reconstructed parameters, BCC improves the best direct-code bound from 3.8220 to 3.5104 nats (Appendix B.21). An implemented behavior-preserving encoder recovers 98.29–99.55% of the BCC saving (Appendix B.24). We also added the complete-model bound for GPT2-medium and Pythia-1.4B to further address this concern (Appendices E and F (Tables 52–53, pp. 61–62)).

5. Sensitivity to priors and losses (K5mP). Across three priors, three endpoints, and two tasks on Phi-3.5, BCC is tighter than direct coding in all 18 settings and tighter than every control in 10. We also report the 5 ties and 3 losses (Appendix B.25, Tables 45–46, pp. 53–54).

6. Joint weight–activation allocation and cost (P7Pp, 7HNQ). On Qwen2.5-1.5B, joint screening reaches 71.68% on BoolQ and 54.39% on ARC-Challenge, versus 61.04% and 32.91% for the best matched-budget control.

7. Presentation (P7Pp, K5mP). Table 1 states the model, certified quantity, population, and code charged for each main guarantee. Figure 2 has explicit labels and a caption defining each set. A dedicated Related Work subsection (p. 2) and Appendix D (pp. 58–60) clarify BCC’s contributions: Table 50 positions them against prior work, and Table 51 connects them to the principal experimental gains and their guarantee scopes. Appendix C states what remains open.

8. Addendum: complete-model bounds for 355M- and 1.4B-parameter models. Several reviewers noted that complete-model bounds were shown only for small decoders. We therefore computed complete-code bounds for two larger models, each charging the entire checkpoint: embeddings, all 24 transformer blocks, normalization parameters, scales, and reconstruction metadata. The models are a dense GPT-2-medium (355M parameters), trained separately on 9 billion OpenWebText tokens, and Pythia-1.4B (1.41B parameters, 300 billion training tokens). The bounds are token-level guarantees on the smoothed next-token loss, with 95% simultaneous confidence (Appendices E and F, Tables 52–53, pp. 61–62).

ModelRouteComplete-model charge (billion bits)Upper bound (bits/token) ↓
GPT-2-mediumBest direct code (4-bit)0.98688.8322
GPT-2-mediumBest individual canonical code0.98318.8242
GPT-2-mediumPractical interval-mass encoder0.90338.6630
GPT-2-mediumBCC0.90298.6621
Pythia-1.4BBest direct code (4-bit)5.85305.1525
Pythia-1.4BCanonical individual code5.84105.1506
Pythia-1.4BBCC3.42215.0668

In both models, the BCC cell is verified by propagating the permitted weight changes jointly through all 24 transformer blocks, and the bound charges the resulting loss envelope. BCC improves the best individual-code bound by 0.1621 bits/token (1.84%) on GPT-2-medium, where the practical encoder realizes 99.48% of the saving, and by 0.0838 bits/token (1.63%) on Pythia-1.4B after paying 0.345 bits/token of loss inflation. This extends complete-model evidence from a 25.5M-parameter subspace model to fully dense models with 355M and 1.4B parameters. The complete protocols are in Appendices E and F (Tables 52–53, pp. 61–62).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.