acceptodds
Under review as a conference paper at ICLR 2027

SparseDiag: State-Dependent Diagnosis and Repair for Low-Bit Large Language Models

Abstract

Post-training quantization (PTQ) methods for large language models (LLMs) have largely advanced by changing how quantized layers are optimized, through improved objectives, learnable parameterizations, reconstruction granularities, and layer- or window-wise schedules. However, existing methods generally follow optimization schedules that are specified independently of the residual state that emerges as quantization progresses, leaving a complementary question underexplored: once a model reaches a particular quantized state, is every scheduled layer still worth optimizing? Through controlled layer-wise probing, we find that the marginal utility of further optimization is highly non-uniform and strongly state-dependent. In relatively weak quantized states, repairable errors can be broadly distributed across the network. After stronger PTQ optimization, however, many layers provide little additional benefit, while the remaining headroom often concentrates in a small set of non-contiguous residual bottlenecks. Moreover, repairing one bottleneck reshapes the subsequent repair landscape, suggesting that the layers worth optimizing cannot, in general, be determined reliably from the initial state alone. Motivated by these observations, we propose Sparse Diagnosis (SparseDiag), an iterative framework that explicitly separates diagnosis from correction. At each round, SparseDiag constructs lightweight layer-wise repair candidates using calibration data, ranks them on a disjoint calibration split, verifies the selected candidate on held-out calibration samples, and updates only a validated residual bottleneck. It then re-diagnoses the updated model and automatically stops when no reproducible repair opportunity remains. Consequently, the optimization schedule is discovered from the evolving quantized state rather than prescribed in advance. Experiments on Llama and Qwen models under W4A16 and W2A16 show that SparseDiag consistently exposes additional repairable headroom across different quantized starting states, while stronger PTQ states exhibit substantially sparser residual repair landscapes than weaker ones. Repeated diagnosis further reveals an evolving progression from broadly distributed repairability to sparse residual bottlenecks and eventually to headroom exhaustion. These results recast low-bit PTQ refinement from merely asking how to optimize each layer to deciding whether, where, and when further optimization is still beneficial.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, post-training quantization, low-bit quantization, efficient inference, adaptive optimization

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_9sS4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SparseDiag refines an existing quantized language model by testing candidate updates to individual layers. At each round, every candidate starts from the same accepted checkpoint and updates only the backend’s existing trainable parameters. Candidates are ranked by held-out full-model negative log-likelihood, with a separate calibration split used to check eligibility and accept repairs. After repairing the selected layer, the method repeats the diagnosis. Experiments compare reconstruction and cross-entropy objectives across several quantized Llama and Qwen models, reporting perplexity improvements alongside cases where calibration gains fail to transfer to full-test evaluation.

Critical strengths

Evaluating a candidate update through its effect on full-model NLL is a sensible design choice. Restoring the same checkpoint before each candidate also makes the within-round comparison interpretable. The paper carefully defines repairability as conditional on the current checkpoint, update operator, and evaluation data. This provides a useful framework for studying what remains worth optimizing after PTQ.

The experiments demonstrate meaningful remaining room for refinement in some strong checkpoints. For example, Qwen3-8B after SliderQuant improves from 10.4845/14.1249 to 8.2824/12.8742 on WikiText-2/C4 under CE→CE. Improvements are reported across four models and a second backend using its native parameters. The explicit reporting of unsuccessful repairs and stronger full-model results also helps establish the method’s limits.

Critical weaknesses

The benefit of repeated diagnosis is not established. Figure 2 shows that eligible layers change across checkpoints, which motivates reconsidering layer selection. It does not establish that an exhaustive rescan after every repair improves the final result over a ranking computed once or a simple cyclic selection strategy. Sections 2.3 and 4.2 explicitly acknowledge the missing matched-budget comparison. This is the experiment I most need to assess the contribution. Each diagnosis round trains candidates for every layer and evaluates them in the complete model, so counting only committed repairs substantially understates the work. Although the authors do not claim a compute speedup, the budget-allocation motivation still requires a quality-versus-total-cost comparison. The existing full-model comparison is unmatched, and full-model CE already achieves lower perplexity on both test corpora for Qwen3-8B.

The reliability of calibration-based selection remains poorly characterized. The method repeatedly consults 16 ranking sequences and 16 confirmation sequences across layers and rounds. Because the confirmation split determines candidate eligibility, it participates in adaptive selection. Section 4.3 shows that accepted repairs can worsen both full-test corpora, but the paper does not measure how often this happens across calibration samples or random seeds. This uncertainty also affects the interpretation of small improvements: OmniQuant MSE→MSE changes WikiText-2 perplexity from 5.7439 to 5.7389, while the Llama-2-13B SliderQuant result changes C4 from 6.6716 to 6.6707. Without repeated trials, it is difficult to distinguish reproducible improvements from sensitivity to the particular calibration sample.

Questions for authors

Contribution of re-diagnosis: With the repair objective, starting checkpoint, and repair budget fixed, how does repeated diagnosis compare with computing the ranking once and with random or cyclic layer selection? Does its advantage survive when the full diagnostic cost is included in the budget?

Cost and implementation: What are the candidate and repair update budgets, learning rates, and schedules for the main Table 1 experiments? The 32-update setting is specified for Figure 2. Please also report end-to-end runtime and peak memory, including diagnosis, against tuned full-model continuation from the same checkpoint.

Selection reliability: Across independent calibration samples and seeds, how frequently do accepted repairs worsen untouched evaluation data? Do larger confirmation sets or a minimum-gain threshold materially reduce that frequency, and are the smaller reported improvements reproducible?

Reviewer_Kx2EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes SparseDiag, a post-PTQ refinement procedure. In each round, it trains an independent candidate update for every layer using the PTQ backend’s native parameters, ranks candidates by full-model NLL gain on a small held-out split, confirms the sign on a second split, repairs the selected layer, and re-diagnoses. Experiments on W4A16 Llama and Qwen models from several starting checkpoints report WikiText-2/C4 perplexity gains, along with several candidly reported failure cases.

Critical strengths
  1. The question of where to spend additional optimization after PTQ is practical and well posed. Separating a candidate’s training objective from its selection criterion is a clean design choice.

  2. Reusing native PTQ parameters without new adapters makes the method easy to plug into existing backends, as the OmniQuant experiment shows.

Critical weaknesses
  1. The motivation for state-dependent diagnosis is not directly tested. The paper argues that an initial sensitivity ranking is a poor substitute for diagnosing the current checkpoint, and this is the main reason for SparseDiag’s per-round scan over all layers. However, the experiments do not compare SparseDiag against static sensitivity ranking, random layer selection, or round-robin repair with the same number of committed repairs.

  2. Selective refinement does not consistently outperform full-model CE. From identical SliderQuant checkpoints (Sec. 4.3), full-model CE achieves lower perplexity on both datasets for Qwen3-8B (7.566/12.624 vs. 8.282/12.874) and on WikiText-2 for Llama-3-8B (6.658 vs. 6.746). SparseDiag’s only clear advantage is Llama-3-8B C4, where full-model CE degrades below the baseline (10.897 vs. 10.451) while SparseDiag improves it (10.348). This hints at better cross-domain robustness, but the evidence is limited to one model and one dataset. Given the cost of a full per-layer scan each round, the paper should either show this advantage more systematically or clarify when selective repair is preferable.

Questions for authors
  1. How does SparseDiag compare with static, random, and round-robin layer selection at a matched number of repairs?

  2. What is the total compute cost of a SparseDiag run relative to SliderQuant and full-model CE?

  3. How stable are the eligible layer sets in Fig. 2 across repeated probes at the same checkpoint?

  4. Do larger ranking/acceptance splits fix the transfer failure in Section 4.3?

Reviewer_F1EaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes a PTQ post optimization framework SparseDiag for low to large language models. This method optimizes candidate layers separately for different Transformer layers and selects the layers that need further optimization based on the complete model NLL on the independent validation set. After each round of repair is completed, perform layer diagnosis again to adapt to optimization requirements under different quantization states. The author conducted experiments on multiple Llama and Qwen models and demonstrated the improvement in perplexity of some W4A16 quantization models.

Critical strengths
  1. This article focuses on how to further optimize the quantization model after the completion of PTQ, which has certain practical application value.
  2. The author studied the changes in optimizable layers under different PTQ states and compared MSE and CE optimization objectives, which helps to understand the further optimization behavior of quantitative models.
  3. The experiment covered multiple models and different PTQ initialization states, and objectively reported the failure of some optimization strategies.
Critical weaknesses
  1. The main design of SparseDiag is to train candidate models layer by layer, improve and select the optimal layer based on the validation set NLL, and then iteratively update it. This is closer to the existing greedy layer by layer optimization and validation set based model selection ideas, and the paper lacks more explicit methodological innovation.
  2. This article emphasizes that the layers worth optimizing will change under different PTQ states, so dynamic re diagnosis is needed. However, the paper did not directly compare with strategies such as fixed layer sorting, random layer selection, or continuous optimization after a single diagnosis under the same computational budget, making it difficult to prove that repeated diagnosis is indeed superior to simpler alternative methods.
  3. SparseDiag requires candidate training and complete model evaluation for all Transformer layers in each round, but only selects one layer for updating in the end. This may introduce significant additional computational overhead, but the paper does not provide a complete analysis of runtime and computational costs, nor does it demonstrate its superiority over conventional optimization methods under the same computational budget.
  4. The author only used 16 sequences for candidate sorting and another 16 sequences for confirmation, and repeated these data in multiple optimization rounds. Section 4.3 of the paper has demonstrated the situation where the validation set is updated, but the performance of the complete test set actually decreases, which raises questions about the robustness and reliability of the current selection mechanism.
  5. On Qwen3-8B, regular full model CE fine-tuning achieved lower WikiText-2 and C4 perplexity than SparseDiag. In addition, the main experiments only report perplexity, lacking downstream task performance and sufficient validation of low bit settings, so the actual benefits and generalization of the method are still not clear enough.
Questions for authors
  1. Can you provide a comparison of the same computational budget with fixed layer sorting, random layer selection, and non repetitive diagnostic strategies to demonstrate the necessity of dynamic diagnosis?
  2. What is the total computational cost for each round of layer by layer candidate training and evaluation? Is SparseDiag still superior to full model fine-tuning under the same GPU time or training FLOPs?
  3. How to improve the reliability of the current acceptance criteria when NLL improvements on the validation set cannot be migrated to the complete test set?
  4. Can the effectiveness of the method be further validated on more quantization bit widths and downstream tasks?
Reviewer_65tKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates which layers remain repairable after a strong post-training quantization (PTQ) checkpoint has been obtained. In each round, it independently trains a candidate for every layer from the same checkpoint, ranks and confirms candidates using held-out full-model negative log-likelihood (NLL), restores the checkpoint to train the selected repair, and repeats diagnosis after an accepted update. The framework distinguishes the candidate-training objective, selection metric, and final repair objective, and compares MSE→MSE, MSE→CE, and CE→CE.

Critical strengths
  1. The research question is practically meaningful. On pp. 1–4, the paper distinguishes initial quantization sensitivity from whether a specific additional update improves the current checkpoint. It explicitly states that repairability depends on the checkpoint, update operator, and data, rather than presenting it as an intrinsic property of a layer.
  2. The results are not uniformly small. In Table 1 on p. 7, CE→CE refinement of the SliderQuant Qwen3-8B checkpoint reduces WikiText-2/C4 PPL from 10.4845/14.1249 to 8.2824/12.8742, a reduction of approximately 21% on the former. Results from multiple strong W4A16 starting points and the supplementary OmniQuant experiment show that further improvement is possible.
  3. The failure analysis is candid and useful. Page 8 reports cases where small held-out splits accept repairs but full-test performance deteriorates, where full-model CE outperforms selective repair, and where longer training is confounded with the learning-rate schedule. Page 5 also explicitly states that the ranking and acceptance splits are reused across candidates and rounds, without claiming an independent generalization guarantee.
Critical weaknesses

W1 [Major] The distinct value of repeated diagnosis has not been experimentally established. Locations: p. 3, §2.3; pp. 7–8, §§4.2–4.3. The change from 22 to 11 to 6 repairable layers and the different selected trajectories establish state dependence. They do not establish that rescanning every layer in each round is better than scanning once and retaining a fixed ranking, or than random/cyclic selection. The main results show that further optimization improves the starting checkpoint, but do not separate the benefit of the updates themselves from the benefit of dynamic diagnosis. The paper explicitly acknowledges the absence of a matched-budget static-ranking comparison. This reduces the risk of misleading readers, but does not supply the missing evidence for the central algorithmic contribution. Suggested remedy: On one or two existing representative models, compare per-round rescanning, a fixed ranking from the first scan, and random/cyclic selection. Control the starting point and candidate/repair objectives, and report both a fixed repair-update budget and the total computational budget including diagnosis. A quality-versus-GPU-time comparison would be more useful than establishing an absolute speed advantage in isolation.

W2 [Moderate; not an independent basis for rejection] The reliability of small-sample screening needs to be quantified. Locations: p. 5, Eqs. 4–7; p. 6, calibration setup; p. 8, failure cases. The 16 ranking sequences and 16 acceptance sequences are reused across layers and rounds, and the acceptance set helps determine the eligibility of every candidate. The positive-gain threshold imposes no requirement on effect size or uncertainty. Small improvements may therefore include candidate-selection bias. This is not an allegation of test-set leakage: the paper explicitly states that full-test evaluation occurs only after the trajectory has been fixed. The question is whether the screening rule reliably selects updates that remain beneficial on new samples. Suggested remedy: Sample a small number of independent calibration pools and report the gains of accepted repairs on additional data not used for selection, the rate of accepted repairs that fail this evaluation, and variability in final PPL. A minimum-gain or uncertainty threshold could also be examined. The small changes for Llama-2-13B particularly need a variability context: WikiText-2 5.0304→5.0256 and C4 6.6716→6.6707 cannot be considered stable improvements merely because several decimal places are reported. If the paper explicitly positions the method only as an exploratory diagnostic framework, this concern primarily limits its applicability and the stability of its observations. The lack of a generalization guarantee should not then be an additional reason for rejection; W1 and W3 remain the main grounds for my current recommendation.

W3 [Major] Reproduction details and the total cost of the main algorithm are incompletely reported. Locations: p. 6, §4.1; p. 9, Reproducibility Statement. The manuscript specifies 32 MSE updates per candidate for Figure 2, the data split, and the maximum number of rounds. However, it does not fully list the update counts, learning rates, optimizers, batch sizes, native LoRA/LWC configurations, and training costs for all diagnosis/repair arms in the main table. The Figure 2 probe settings cannot be assumed to define candidate training and repair in every main experiment. Each round trains candidates for all layers and performs repeated full-model evaluations; sparse committed updates do not imply low training overhead. Suggested remedy: Provide a unified configuration table, distinguish candidate training from final repair, precisely define how each starting checkpoint is constructed, and report the GPU time spent on diagnosis, repair, and evaluation separately. The issue is whether the protocol is reproducible, rather than a requirement to release code during anonymous review.

Questions for authors

Q1. At the same total budget including all diagnosis costs, how much better is repeated diagnosis than retaining the ranking from the first scan? Q2. Are layer selection and final improvements stable when the calibration pool changes? How often do repairs accepted by the positive-gain rule fail on data not used for selection? Q3. For each objective pair in Table 1, how many updates are used for candidate training and final repair, and what learning rates and parameter configurations are used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.