acceptodds
Withdrawn from ICLR 2027

SALTQ: Saliency-Allocated Low-Bit Trainability for Quantized LLM Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-tuning for deployment below 4 bits should directly produce a uniform low-bit weight-only checkpoint of integer codes and per-group quantization parameters, without an adapter branch or FP16 columns. Existing approaches fall short in different ways: adapter-based methods require post-merge re-quantization, mergeable variants freeze integer codes, weight-updating QAT computes the full weight gradient, and selective methods keep salient columns in FP16. To address this gap, we propose SALTQ, which allocates trainability by saliency within the deployed representation. A second-order analysis of the task loss shows that the columns where rounding costs the most loss are also those where a fine-tuning update moves the output the most, namely the few input columns with the largest activation energy, so that repairing the quantization error and adapting to the task can share one set of trainable columns. SALTQ makes these salient columns fully trainable at the deployment bit-width. In every other quantization group it trains only the zero-point, whose gradient pools within the group and is therefore almost free, whereas training the scales or the integer codes would require the full weight gradient. A channel permutation gathers the salient columns into complete groups; it is shared within contiguous segments of layers rather than across the whole network, which leaves one to three residual-stream reorderings at inference. Controlled comparisons show that the salient columns need to be trained but not in FP16: training them at the deployment bit-width retains nearly all of the gain of training them in FP16, with a gap of at most 0.8 points. At INT2 on Llama-2-7B and Qwen2.5-32B, SALTQ outperforms the compared mergeable and mixed-precision baselines by at least 0.7 points on the Commonsense-170k average and 3.8 points on GSM8K. At INT3, it reaches the level of the FP16 QLoRA reference on commonsense reasoning and GSM8K. It trains in about the time of QLoRA and in less than a third of the time of LR-QAT, which updates all weights and reaches a similar accuracy. The code will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization-aware training, low-bit quantization, parameter-efficient fine-tuning, large language models, weight-only quantization, saliency, channel permutation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JwczOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes SALTQ, an efficient fine-tuning method for weight-only quantized LLMs. It partitions weight columns into salient and non-salient groups based on activation statistics. Salient weights are updated through quantization-aware training, while only the zero-points are updated for non-salient weights. The method is evaluated against several fine-tuning baselines on Llama-2-7B and Qwen2.5-32B.

Critical strengths
  1. Efficient low-bit fine-tuning is an important problem for practical LLM deployment.
  2. SALTQ achieves competitive accuracy at INT2 and INT3, outperforming the compared parameter-efficient baselines, particularly at INT2.
  3. The derivation of efficient zero-point gradients is sound.
Critical weaknesses
  1. The design criteria in Figure 1 and Table 1 appear somewhat subjective, and their practical importance is not sufficiently justified.
  2. The paper does not demonstrate the practical inference benefits of SALTQ, such as latency or throughput improvements. It mainly emphasizes training efficiency, whereas inference costs can dominate in long-term deployment.
  3. The baseline comparisons do not control for computational cost. Since cheaper methods may naturally achieve lower accuracy, the paper should provide clearer accuracy-cost trade-offs to support its efficiency claims.
  4. The evaluation is limited to two relatively old models, Llama-2-7B and Qwen2.5-32B. Experiments on more recent models would help establish generalizability.
  5. The presentation is difficult to follow. The introduction is overly long, the method section is dense, Figure 2 provides limited help in understanding the method, and Figures 4 and 5 are hard to understand.
Questions for authors
  1. The paper shows that activation-based saliency outperforms random column selection. How does it compare with task-gradient-based or quantization-error-aware criteria for selecting salient columns for downstream fine-tuning?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Jwcz

Thank you for the review.

Design conditions (Figure 1, Table 1)

Each condition corresponds to a consequence measured in the paper, as the caption of Table 1 will state.

  • Uniform low-bit deployment: the checkpoint runs on a standard packed low-bit kernel, with no mixed-precision kernel and no adapter branch (measured below).
  • Weights trainable: methods that freeze all integer codes are five to nine points below the FP16 reference on GSM8K at INT2, SALTQ about three (Table 2).
  • No full weight gradient: 1 h 44 min of training, against 3 h 37 min to 8 h 17 min for the methods that compute it (Tables 5, 6).
  • No re-quantization: re-quantizing the merged QLoRA model to INT2 lowers GSM8K from 58.07 to 22.06 (Table 2).

These conditions conflict: trainable weights normally cost the full weight gradient or FP16 columns, and avoiding both means frozen codes or a second quantization. Every existing method gives up at least one, and SALTQ meets all four.

Inference benefits

SALTQ delivers a uniform low-bit checkpoint and so inherits the inference benefits of this format, adding only one to three reorderings (gathers) of the residual stream. We implemented the packed formats in vLLM on its low-bit GPTQ kernel and measured them at INT2 (Llama-2-7B architecture, RTX 6000 Ada, random weights, since latency does not depend on their values).

ConfigurationWeightsDecoding step, batch 1Batch 16
FP1612.55 GiB17.78 ms24.90 ms
Standard GPTQ format2.42 GiB4.37 ms15.53 ms
QA-LoRA export format2.75 GiB4.84 ms16.34 ms
QEFT (mixed precision)2.97 GiB5.51 ms19.80 ms
SALTQ, 1 gather2.75 GiB4.82 ms16.34 ms

Against FP16, the SALTQ checkpoint is 4.6 times smaller and decodes 3.7 times faster at batch 1 (1.5 times at batch 16). The gather has no measurable cost: latency is the same without it, and each gather takes 3 to 4 microseconds. SALTQ matches the uniform format of QA-LoRA, while the mixed-precision QEFT model is 8% larger and decodes 14% to 21% slower. Against the standard GPTQ format, SALTQ decodes 5% to 10% slower, because its zero-points are floating-point values.

Controlling computational cost

All methods are trained on the same data with the same epochs, steps and batch size. Under this protocol SALTQ takes the training time of QLoRA (1 h 44 min against 1 h 37 min at INT2), and the more expensive methods are not more accurate. On the commonsense average at INT2, SALTQ scores 74.85. PEQA takes 2.1 times as long and scores 72.18, LR-QAT 3.5 times and 75.19, EfficientQAT 4.8 times and 73.33 (Tables 5, 6). Accuracy does not grow with cost, which is the thesis of the paper: the extra computation goes to weights that do not need the full gradient. The cheaper QEFT and QA-LoRA have the same arithmetic cost per token as SALTQ. Their limit lies in what they can update (a static dense part, frozen integer codes). We will move the accuracy-versus-time plot (Figure 7) to the main text.

Newer models

We chose Llama-2-7B because the compared methods report results on it in their own papers. We have now fine-tuned Qwen3.5-9B (March 2026, linear attention in 24 of 32 layers) on MetaMathQA at INT2, with the same recipe and no re-tuning.

MethodBitsGSM8K
FP16 reference1686.58
GPTQ (reference re-quantized)251.63
QA-LoRA272.25
QEFT, k = 2562+1673.92
SALTQ, k = 256279.53

SALTQ is 5.6 points above the best baseline. The premise of the method also holds on this model: the top 1% of channels carry 61% of the activation energy at the attention input (40% on Llama-2-7B).

Presentation

We agree. In the revision we will shorten the introduction and open it with the deliverable and the question of where to spend the full weight gradient. We will lighten the method section by moving implementation details to the appendix. Figure 2 will show which fields are updated in training and what is stored at export. Figure 4 will separate the storage of the salient columns from their training, and Figure 5 will label the default budget and the two controls.

Other selection criteria

We compared the columns selected by other scores with those selected by E[x²] on Llama-2-7B (Jaccard overlap averaged over layers, with a chance level of 0.02 to 0.03).

  • Activation energy × quantization error, as in OWQ and QEFT: 0.95 to 0.96. The rounding error differs little between columns, so E[x²] decides the ranking.
  • Quantization error alone: 0.02 to 0.04, at chance level.
  • Task gradient (empirical Fisher): 0.32 to 0.34, a partial overlap. We therefore trained SALTQ at INT2 with gradient-selected columns. It scores 74.31 on the commonsense average, against 74.85 with E[x²]. The difference is of the size of the seed-to-seed standard deviation (0.44), so the task gradient is not a better criterion, and E[x²] needs only a forward pass.

Thank you for the reply.

Weakness 1: The rebuttal clarifies the motivation behind the four design conditions, but their practical importance as a combination remains insufficiently justified.

Weakness 3: Figure 7 does not align all methods to the same wall-clock budget, so the cost is not controlled.

Weakness 4: The evaluation on the newer model covers only one downstream task, limiting the evidence for generalizability.

Weakness 5: The proposed presentation improvements are not yet reflected in the manuscript.

I therefore maintain my original rating.

Reviewer_Nso2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SALTQ, a weight-only quantization-aware fine-tuning framework. To enhance both training- and inference-time efficiency, SALTQ combines multiple parameter-efficient fine-tuning (PEFT) method. Specifically, it determines a salient channels and applies full-weight updates (like quantization-aware training; QAT) on those channels, and for the remaining channels, it fine-tunes only the quantization parameters to reduce the ovall training complexity. The proposed method achieves higher accuracy on language modeling, commonsense QA, math, and MMLU tasks than quantization-aware PEFT baselines across Llama2-7B and Qwen2.5-32B, while enhancing both training efficiency and deployment efficiency.

Critical strengths
  • There are various types of quantization-aware fine-tuning methods including both PEFT and full QAT. They are all differ in terms of both training efficiency and deployment efficiency. This paper is worth to integrate the existing method and propose the unified solution. I think this paper can provide a good guideline for anyone who want to fine-tune their model in resource-constraint environment.
  • This paper provides comprehensive backgrounds, which is helpful to read. Also, the paper treats many QAT/PEFT baselines, enabling understanding the position of this work easier.
  • The reported training cost and accuracy, and ablation studies are informative.
Critical weaknesses
  • The major concern of this work is lacks of novelty. The proposed method is close to combination of the existing methods, rather than innovation. Reordering and GPTQ were already supported in the official repository in GPTQ, and training only salient channels are already proposed method as the paper mentioned.
  • The model used in this paper and the training/evaluation protocols are somewhat outdated. Recent models now adapt a new architecture far from tranditional Transformer layers (e.g., MoE, linear attention, MLA, ...). There can be different behavior of salient weights or activaiton statistics compared to well-known channel-wise activation outliers studied in Llama2-7B.
  • Also, fine-tuning with Alpaca dataset then measuring MMLU score is not only oudated, but also no useful, and this is well-known issue. As shown in Table 3, there is no MMLU score gain in FP16 reference, compared to FP16 w/o fine-tuning. The paper should update the models and training/evaluation datasets to highlight the practical needs of the proposed method.
  • The proposed method relies on calibration set in multiple stages. Is there any potential risk of overfitting compared to existing PEFT methods?
Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Nso2

Thank you for the careful review, and for recognizing the value of a unified solution. We did not state the purpose of SALTQ clearly enough in the paper, so we clarify it first.

What SALTQ is for

The goal is to obtain a uniform low-bit weight-only checkpoint directly from fine-tuning, at the training cost of the widely used QLoRA setting, without paying for quantization-aware training of all weights. Each existing method misses one part of this goal. LR-QAT and EfficientQAT take 3.3 to 5 times the training time. QA-LoRA and PEQA freeze every integer code and recover less accuracy, five to nine points below the FP16 reference on GSM8K at INT2. QEFT delivers a mixed-precision model, not a uniform low-bit checkpoint. In our vLLM measurement at INT2 it is 8% larger and decodes 14% to 21% slower than the uniform format.

What is new

Two findings make this goal reachable, and neither follows from combining existing methods.

  • The salient columns need to be trained, but not in FP16. As you note, OWQ-WCT and QEFT already train these columns, but only as FP16 columns. Training them at the deployment bit-width loses at most 0.8 points against training them in FP16 (Table 4).
  • The full weight gradient is needed only on these columns. Accuracy saturates at 128 to 256 columns per layer (Table 14), and LR-QAT and EfficientQAT, which pay this gradient on every column, are not more accurate (Table 5).

We also tested the combination directly. Quantizing the FP16 columns of the INT2 QEFT model with GPTQ lowers its commonsense average from 73.71 to 68.75, against 74.85 for SALTQ. The columns have to be trained inside the low-bit representation.

On reordering

You are right that act-order in the GPTQ repository sorts the columns by the Hessian diagonal, which is proportional to E[x²]. We will state this in the paper, and we do not claim reordering as a contribution. The two serve different purposes. act-order is a step of post-training quantization that decides the order in which the columns of one layer are quantized. Our permutation serves training: it gathers the trained columns into whole quantization groups, and it is shared across layers so that the full model needs only one to three reorderings at inference. We measured these in vLLM on a packed low-bit kernel: each reordering takes 3 to 4 microseconds, and together they take less than 0.2% of a decoding step.

On models and new architectures

We chose Llama-2-7B because the compared methods report results on it in their own papers, so our reproductions can be checked. We agree that newer architectures should be covered, and added two experiments.

The premise of the method holds on them. At the attention input, the top 1% of channels carry 61% of the activation energy E[x²] on Qwen3.5-9B (linear attention in 24 of 32 layers) and 79% on DeepSeek-V2-Lite (MoE with MLA), against 40% and 42% on Llama-2-7B under the same calibration data. Adjacent layers also share more of their salient channels (Jaccard 0.74 and 0.67, against 0.61 and 0.59).

We then fine-tuned Qwen3.5-9B (released in March 2026) on MetaMathQA at INT2, with the recipe and learning rates of Llama-2-7B and no re-tuning.

MethodBitsGSM8K
FP16 reference (QLoRA merged)1686.58
GPTQ (reference re-quantized)251.63
QA-LoRA272.25
QEFT, k = 2562+1673.92
SALTQ, k = 256279.53

SALTQ is 5.6 points above QEFT and 7.3 above QA-LoRA.

On Alpaca fine-tuning and MMLU

We agree that fine-tuning on Alpaca does not improve MMLU, and we do not use it to show adaptation. We report it because QLoRA, QA-LoRA and IR-QLoRA do, and because the unchanged FP16 score (45.97 before, 45.84 after) means that differences between quantized models on it reflect how much quantization damage is repaired, not how much of a task is learned. We claim no large advantage there. Adaptation is shown on Commonsense-170k and MetaMathQA, the benchmarks of recent PEFT work, where SALTQ leads on GSM8K at INT2 by 3.8 and 5.1 points on the two models of the paper and by 5.6 points on Qwen3.5-9B. We will state this role in the caption of Table 3.

On calibration and overfitting

The calibration set never enters a gradient step, is a subset of the training data, is shared by all baselines, and does not overlap with any test set. Beyond what the baselines use it for, SALTQ uses it only to choose the salient columns and the segment boundaries. With two re-drawn disjoint calibration sets, the segmentation is identical and the columns nearly so (Jaccard 0.95 to 0.97, covering over 99% of the activation energy of the default selection), and the INT2 GPTQ starting point scores 66.09 against 66.22.

Reviewer_Nso2OpenReview ↗

Thank you for the rebuttal

Thank you for the rebuttal and the additional experiments. I understand that Llama-2-7B and Alpaca fine-tuning have been widely used in prior work, and the authors’ motivation for choosing them to facilitate comparisons. I also appreciate the clearer articulation of SALTQ’s contributions. In particular, the substantial recovery of GSM8K performance through math fine-tuning on a newer architecture at INT2 is encouraging.

I view the potential impact of this work on the research community positively. However, since I see the algorithmic novelty as relatively modest, I believe the empirical contribution is particularly important. The paper would be stronger if it more broadly demonstrated that SALTQ provides competitive or better accuracy at lower training and deployment costs across recent models, architectures, and hardware.

The new end-to-end fine-tuning result is a valuable addition, but it covers only one additional model and one benchmark. I therefore maintain my original rating, while taking a more positive view of the work.

Thank you for the reply and for the more positive view of the work. We agree that the empirical evidence matters most here. With the new result, SALTQ is evaluated end to end on three models (Llama-2-7B, Qwen2.5-32B, and Qwen3.5-9B with linear attention), and on GSM8K at INT2 it leads the best baseline on each of them, by 3.8, 5.1 and 5.6 points. On cost, the table orders the methods on Llama-2-7B by the training time of one epoch and gives the commonsense average.

MethodINT2 timeINT2 avg.INT3 timeINT3 avg.
QEFT (mixed precision)1 h 06 min73.711 h 05 min77.68
QA-LoRA1 h 13 min72.461 h 11 min77.71
SALTQ1 h 44 min74.851 h 39 min78.84
PEQA3 h 37 min72.183 h 12 min77.81
LR-QAT6 h 06 min75.195 h 24 min78.31
EfficientQAT8 h 17 min73.337 h 28 min77.54

SALTQ lies on the Pareto front at both bit-widths. Beyond it, 3.5 times the training time buys 0.34 points at INT2, and at INT3 no slower method is more accurate. At deployment its checkpoint matches the uniform format of QA-LoRA, while the mixed-precision QEFT model is 8% larger and decodes 14% to 21% slower.

We accept that the newest architecture is covered by one model and one benchmark so far, which is what the rebuttal period allowed. We will add the commonsense benchmark on Qwen3.5-9B to the final version.

Reviewer_zf55OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a quantization aware training methods that allocate training resources based on the saliency of the weights. Specifically, the full weight gradient goes only to the salient columns that contribute the most to the output, while the remaining weights only train the zer-point at almost no cost. Experimental results show that the proposed method achieve on par or better results than other more costly quantization aware training baseline method.

Critical strengths
  1. The analysis that only a few column of weights contribute the most to output is sound, the idea proposed by the paper to conduct full weight gradient only on this few columns makes intuitive sense, as previous method either implement full weight gradient training which is costly, or only train zero points and scale parameters, which has degraded performance. The proposed method strikes balance between training cost and performance
  2. The experimental results are comprehensive and convincing
Critical weaknesses
  1. No obvious weaknesses
Questions for authors

what is the criterion to construct calibration set to make sure that the corresponding results derived from it is reliable?

1 comment
AuthorsOpenReview ↗

Response to Reviewer zf55

Thank you for the review and for the question on calibration.

How the calibration set is built and whether the results depend on it

The calibration set follows four rules.

  1. It is drawn only from the training split of the fine-tuning data and has no overlap with any test set.
  2. It is balanced over tasks: 438 examples from each of the eight commonsense tasks (3,500 in total after shuffling), and 125 from each of the eight problem types for mathematics (1,000 in total).
  3. The examples use the same prompt format as training, and the statistics are collected on the pretrained model.
  4. All baselines share the same calibration set, and it is never used in a gradient step.

To test reliability we drew two further calibration sets by the same rules, disjoint from the default set and from each other, and repeated the calibration steps on Llama-2-7B.

Compared with the default calibration setINT3INT2
Layer segmentationidenticalidentical
Jaccard overlap of the selected salient columns (mean over layers)0.95 to 0.970.96 to 0.97
Activation energy of the default selection covered by the new selectionabove 99%above 99%

The INT2 starting point produced by GPTQ with a re-drawn set scores 66.09 on the commonsense average, against 66.22 with the default set. The selected columns, the segmentation and the starting point are therefore stable under re-sampling. What does matter is that the set is in-domain and task-balanced: Table 17 shows that a single-task or out-of-domain set gives a much worse INT2 starting point, and we state this as a requirement of the pipeline. We will add the full protocol and this check to Appendix D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.