- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper composes single-block margin statistics to predict compressed-model accuracy on a labeled reference set. Experiments cover several models, compression methods, and allocation tasks.
Critical strengths
- Margin statistics retain information that scalar block accuracies lose, making the approach well motivated and reusable.
- The broad evaluation, exhaustive smaller spaces, and explicit fit/selection/test separation support the estimator. I lean toward weak acceptance.
Critical weaknesses
- Cross-block interactions need better validation. An effect measured with all other blocks dense need not remain valid after other blocks are compressed. Appendix D’s answer-score covariance analysis does not directly address this.
- Low average prediction error does not guarantee good allocation choices. The 9.1-billion-configuration experiment evaluates only 24 presampled allocations, not an optimized selection. The cost comparison should also let direct-search baselines reuse compressed-option caches.
Questions for authors
- How large are the errors when composing two or more block effects, particularly under heavier compression?
- What selection regret and total cost does a prospectively optimized allocation achieve against search baselines with the same caching and evaluation budget?
We appreciate the reviewer's insightful comments. Here, we answer each of the questions individually. We will update our manuscript accordingly.
Q1 (and W1). How large are the errors when composing two or more block effects, particularly under heavier compression?
The errors stay small. Across 48 jointly compressed configurations, prediction MAE is 0.93 pp on K2-Horizon-7B and 1.14 pp on Gemma-4-12B PT, and it remains below 3 pp at 70% block sparsity.
We compressed two, three, or all blocks jointly at 30%, 50%, and 70% block sparsity and compared measured accuracy with the prediction composed from single-block measurements. Values are prediction MAE (pp).
We also measured the cross-block interaction directly. On 20 two-block configurations, the residual standard deviation of the jointly compressed model is 0.88–0.95 of the value given by the composition rule (Eq. 5). Effects measured with the other blocks dense therefore remain valid after those blocks are compressed. The complete-space results in Table 1 support the same conclusion, since every allocation there composes six block effects.
We will add these results to the appendix.
Q2 (and W2). What selection regret and total cost does a prospectively optimized allocation achieve against search baselines with the same caching and evaluation budget?
In the 9.1-billion-configuration space of Section 4.7, the optimized allocation reaches 69.12% test accuracy (the dense model reaches 69.98% and an evenly spaced allocation at the same budget reaches 64.30%) with zero selection regret, 3.3–3.4 pp above greedy and evolutionary search under the same compressed-option cache and evaluation budget.
Our predictor scores 35,392 candidate allocations without compressed-model execution and selects one before any compressed model is evaluated. All methods share the same compressed-option cache and receive 72 compressed-model evaluations; we also run search with 100 evaluations to match our total time. Selection regret is relative to the best measured allocation.
Our allocation is better at both equal evaluation budget and equal time.
The parameter library also reduces cost when several budgets are requested. For three K2-Horizon-7B GPTQ budgets (3.33, 3.50, and 3.67 bits), one library of 12 single-block measurements serves all three, while search needs 12 evaluations per budget even with the shared cache, warm starts, and reuse of earlier evaluations. Total time is 382 s versus 632 s and 646 s, about 40% less, with selection regret of 0, 0, and 0.70 pp.
We will add the optimized-selection result to Section 4.7 and the cost comparison to Appendix F.