- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SparseDiag refines an existing quantized language model by testing candidate updates to individual layers. At each round, every candidate starts from the same accepted checkpoint and updates only the backend’s existing trainable parameters. Candidates are ranked by held-out full-model negative log-likelihood, with a separate calibration split used to check eligibility and accept repairs. After repairing the selected layer, the method repeats the diagnosis. Experiments compare reconstruction and cross-entropy objectives across several quantized Llama and Qwen models, reporting perplexity improvements alongside cases where calibration gains fail to transfer to full-test evaluation.
Critical strengths
Evaluating a candidate update through its effect on full-model NLL is a sensible design choice. Restoring the same checkpoint before each candidate also makes the within-round comparison interpretable. The paper carefully defines repairability as conditional on the current checkpoint, update operator, and evaluation data. This provides a useful framework for studying what remains worth optimizing after PTQ.
The experiments demonstrate meaningful remaining room for refinement in some strong checkpoints. For example, Qwen3-8B after SliderQuant improves from 10.4845/14.1249 to 8.2824/12.8742 on WikiText-2/C4 under CE→CE. Improvements are reported across four models and a second backend using its native parameters. The explicit reporting of unsuccessful repairs and stronger full-model results also helps establish the method’s limits.
Critical weaknesses
The benefit of repeated diagnosis is not established. Figure 2 shows that eligible layers change across checkpoints, which motivates reconsidering layer selection. It does not establish that an exhaustive rescan after every repair improves the final result over a ranking computed once or a simple cyclic selection strategy. Sections 2.3 and 4.2 explicitly acknowledge the missing matched-budget comparison. This is the experiment I most need to assess the contribution. Each diagnosis round trains candidates for every layer and evaluates them in the complete model, so counting only committed repairs substantially understates the work. Although the authors do not claim a compute speedup, the budget-allocation motivation still requires a quality-versus-total-cost comparison. The existing full-model comparison is unmatched, and full-model CE already achieves lower perplexity on both test corpora for Qwen3-8B.
The reliability of calibration-based selection remains poorly characterized. The method repeatedly consults 16 ranking sequences and 16 confirmation sequences across layers and rounds. Because the confirmation split determines candidate eligibility, it participates in adaptive selection. Section 4.3 shows that accepted repairs can worsen both full-test corpora, but the paper does not measure how often this happens across calibration samples or random seeds. This uncertainty also affects the interpretation of small improvements: OmniQuant MSE→MSE changes WikiText-2 perplexity from 5.7439 to 5.7389, while the Llama-2-13B SliderQuant result changes C4 from 6.6716 to 6.6707. Without repeated trials, it is difficult to distinguish reproducible improvements from sensitivity to the particular calibration sample.
Questions for authors
Contribution of re-diagnosis: With the repair objective, starting checkpoint, and repair budget fixed, how does repeated diagnosis compare with computing the ranking once and with random or cyclic layer selection? Does its advantage survive when the full diagnostic cost is included in the budget?
Cost and implementation: What are the candidate and repair update budgets, learning rates, and schedules for the main Table 1 experiments? The 32-update setting is specified for Figure 2. Please also report end-to-end runtime and peak memory, including diagnosis, against tuned full-model continuation from the same checkpoint.
Selection reliability: Across independent calibration samples and seeds, how frequently do accepted repairs worsen untouched evaluation data? Do larger confirmation sets or a minimum-gain threshold materially reduce that frequency, and are the smaller reported improvements reproducible?