- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
BCC aggregates prior mass over implementations sharing the same bounded loss function, aiming to tighten risk certificates. It also constructs edit families and accounts for their transfer beyond construction data.
Critical strengths
-
Certifying shared behavior rather than individual parameter codes is a well-motivated perspective for quantized models.
-
The framework explicitly accounts for independent validation and certifies simultaneous output-head edits, providing guarantees beyond testing individual perturbations.
Critical weaknesses
-
Practical gains at modern-model scale remain insufficiently established. Useful complete-code bounds are demonstrated on small decoders, whereas large-model experiments mainly certify restricted output-head edits rather than useful full-model complexity reductions.
-
The certificates depend on the declared prior, loss, and input population. Although the appendix acknowledges these distinctions, the additional deployment value of the resulting complexity reductions remains unclear, particularly beyond finite-corpus prediction preservation.
Also, the presentation needs improvement: figures such as Figure 2 contain small text and visual elements that are difficult to read at normal viewing size, making the conceptual explanation harder to follow.
Questions for authors
-
Can the authors demonstrate a deployment decision or useful modern-model risk bound that improves specifically because of behavioral aggregation?
-
How sensitive are the practical conclusions to reasonable alternative priors and to the choice between prediction loss and bounded NLL?
I would be willing to raise my rating if the authors provide convincing answers to these two questions, supported by experimental results clarifying the practical value and sensitivity of behavioral aggregation, and improve the presentation, particularly the readability of Figure 2 and the explanation of the certificates’ scope.
8 comments
Response to Reviewer K5mP (2/5): sensitivity to priors and losses (Question 2)
Response to Question 2: Thank you for asking us to quantify this. Across two tasks, three priors, and three endpoints on Phi-3.5-mini, BCC gives a tighter bound than direct coding in all 18 settings. Against the strongest of seven controls it wins 10, ties 5, and loses 3. Under uniform and precision-weighted priors it beats every control for harmful change and added NLL (8 of 8) and ties random matching for answer error (4 of 4). Under the champion-heavy prior it wins 2, ties 1, and loses 3 (Appendix B.25 (Tables 45–46, pp. 53–54)).
We use the same 2.332 GB cap, 1,172 ARC-Challenge questions, and 2,985 MathQA questions; pooled routes include 256 additional development questions. Harmful change counts reference-correct/compressed-incorrect answers, while answer error measures absolute error. Added NLL is the positive difference between compressed and FP16 NLL after clipping each at 10 nats. All entries are upper bounds; lower is better.
ARC-Challenge
| Prior | Endpoint | Direct | BCC | Best control (of seven) | Outcome |
|---|---|---|---|---|---|
| Uniform | Harmful change (%) | 10.1438 | 9.5988 | FC: 10.0705 | Win |
| Uniform | Answer error (%) | 25.2796 | 24.5613 | RM: 24.5613 | Tie |
| Uniform | Added NLL (nats) | 0.2749 | 0.2516 | PD: 0.2544 | Win |
| Precision-weighted | Harmful change (%) | 10.0749 | 9.5226 | PD: 10.0313 | Win |
| Precision-weighted | Answer error (%) | 25.1848 | 24.4545 | RM: 24.4545 | Tie |
| Precision-weighted | Added NLL (nats) | 0.2707 | 0.2472 | PD: 0.2508 | Win |
| Champion-heavy | Harmful change (%) | 10.3238 | 9.8924 | FC: 10.0705 | Win |
| Champion-heavy | Answer error (%) | 25.6458 | 24.9707 | RM: 24.9707 | Tie |
| Champion-heavy | Added NLL (nats) | 0.2869 | 0.2692 | PD: 0.2544 | Loss |
MathQA
| Prior | Endpoint | Direct | BCC | Best control (of seven) | Outcome |
|---|---|---|---|---|---|
| Uniform | Harmful change (%) | 8.7710 | 8.5035 | PC: 8.6061 | Win |
| Uniform | Answer error (%) | 76.5058 | 76.2001 | RM: 76.2001 | Tie |
| Uniform | Added NLL (nats) | 0.2471 | 0.2299 | PC: 0.2396 | Win |
| Precision-weighted | Harmful change (%) | 8.7316 | 8.4605 | PD: 8.5950 | Win |
| Precision-weighted | Answer error (%) | 76.4509 | 76.1386 | RM: 76.1386 | Tie |
| Precision-weighted | Added NLL (nats) | 0.2448 | 0.2275 | PC: 0.2396 | Win |
| Champion-heavy | Harmful change (%) | 8.9235 | 8.6691 | PC: 8.6061 | Loss |
| Champion-heavy | Answer error (%) | 76.7165 | 76.4346 | PD: 76.2971 | Loss |
| Champion-heavy | Added NLL (nats) | 0.2558 | 0.2394 | PC: 0.2396 | Marginal win |
FC: fixed champions; RM: random matched envelope; PD: pooled direct; PC: pooled calibration. The seven controls are these four, direct coding, canonical coding, and the format envelope. Bounds are rounded upward.
Uniform coding assigns each of 768 implementations mass . Precision weighting gives weight 2 to 256 predefined lower-precision implementations and weight 1 to the remaining 512, giving the relevant candidates mass . The champion-heavy prior is where the 24 champions are selected on development questions and frozen before confirmation. BCC’s cells contain 40 implementations on ARC and 21 on MathQA, all outside . Their masses fall fivefold relative to uniform coding, adding to the complexity charge; all three losses occur under this prior. Pooled direct retains a uniform prior in these rows because its sample includes the champion-selection questions. The answer-error ties involve different BCC and random cells with equal size, envelope error counts, prior mass, and confidence allocation. Pooling improves their bounds over direct coding, while behavioral grouping adds no further benefit on this endpoint. MathQA’s approximately 72% observed compressed-model error limits the practical strength of its absolute-error result. Its champion-heavy NLL win is marginal.
Coding priors over weights show additional aggregation gains on the exact tasks: 1.344–1.954 bits beyond the cheapest individual realization under histogram-Huffman coding (Table 8, p. 23) and 1.613–2.436 bits under the sparse/uniform ternary-mixture prior (Table 34, p. 43). An implemented interval-mass encoder also recovers 98.29–99.55% of the BCC saving on six complete decoders; an interval-index code achieves shorter records under its alternative coding distribution (§B.24, Table 41, p. 51).
Response to Reviewer K5mP (3/5): Question 2, continued (bounded NLL on SmolLM2)
On SmolLM2-1.7B, BCC certifies 131,072 math and 262,144 code coordinates, versus 4,096 for the activation- and magnitude-based controls—32× and 64× gains. Using 8,192 independent confirmation prefixes per domain, the BCC continuation-risk bounds are 0.664115% and 0.776436%, and added clipped-NLL bounds are 0.023617 and 0.024253 nats/token. Both satisfy the 1% continuation-risk and 0.10-nat added-NLL requirements. The capacities are set by the continuation requirement; the NLL requirement is not binding at these sizes (§B.23, Table 39, p. 49).
Response to Reviewer K5mP (1/5): modern-scale gains and the role of aggregation (Comments 1–2, Question 1)
Response to Comments 1–2 and Question 1. We understand the concern about practical gains at modern-model scale and deployment value beyond finite-corpus preservation. The new results address compression beyond output-head edits and a deployment decision that improves specifically through aggregation. The clearest gain is on Llama-3.1-8B: BCC certifies a 4.0 GB deployment at an illustrative 2% harmful-change requirement, compared with 4.4 GB for the strongest control—a 9.1% storage reduction.
A harmful change occurs when the reference answers correctly and the compressed deployment answers incorrectly. These percentages bound that event over all questions from the declared task population. For Phi-3.5-mini, BCC supports deployment selection on science question answering (ARC-Challenge) and mathematical reasoning (MathQA) using certified harmful-change risk. The illustrative 10% requirement means certifying, with the stated confidence, that compression turns a correct reference answer into an incorrect answer on at most 10% of all questions from the declared task population. At the same 2.332 GB budget, only BCC meets this requirement on ARC, while it provides the tightest MathQA guarantee, where all routes already meet the requirement (Table 36, p. 46).
Modern-model gains beyond head edits—Comment 1. Both Llama-3.1-8B and Phi-3.5-mini experiments quantize attention and feed-forward layers throughout the transformer. On Llama, all methods share candidate banks and an independent audit of 10,000 draws, with 95% simultaneous coverage over 20 frozen finalists. On Phi, the BCC envelopes have 66/1,172 harmful changes on ARC and 180/2,985 on MathQA; the reported bounds use 98.75% simultaneous confidence accounting. At an illustrative 10% requirement, only BCC passes on ARC; all routes pass on MathQA. Settings and comparisons appear in Table 35, p. 45, and Table 36, p. 46 (of the revised manuscript).
Deployment benefit specifically from aggregation—Comment 2 and Question 1. At 4.0 GB, Llama’s aggregation-guided search yields 137 harmful changes, versus 200 for the same search using individual prior mass. Their bounds are 1.730% versus 2.426%, under the same audit. Aggregation therefore changes the selected deployment and its certified risk. Validation-loss allocation gives 195 harmful changes and a 2.371% bound. The smallest passing sizes are 4.0 GB for BCC versus 4.4–4.8 GB for the controls (Table 6b, p. 10).
This ablation concerns quantization throughout the transformer. “Aggregation off” uses individual implementation prior mass in the selection score; “aggregation on” uses the summed prior mass of the declared behavioral cell. Both searches use the same candidate banks, and their selected deployments receive the same independent binomial audit. Aggregation changes the selection score and consequently which deployment is chosen: at 4.0 GB, only the aggregation-guided selection meets the illustrative 2% requirement.
Why this matters: the practical choice is between deploying at 4.0 GB or 4.4 GB. Each method selects from the same 64 candidates at 4.0 GB; among these selections, only the aggregation-guided pick meets the illustrative 2% requirement. The strongest controls first pass at 4.4 GB, requiring 10% more storage. Aggregation balances measured harm with how many candidates share the same harmful-change pattern on the 512 selection questions. On 10,000 independent audit draws, its selected deployment produces 137 harmful changes versus 200 for the same search without aggregation—31.5% fewer.
A second example is SmolLM2-1.7B. Charging each implementation separately, 8,192 probes certify at most 105 editable head coordinates at 1% risk; treating the family as one behavior certifies 131,072 on mathematical text and 262,144 on code (Appendix B.23, p. 49). This is an output-head result, cited here only for the role of aggregation.