- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes SALTQ, an efficient fine-tuning method for weight-only quantized LLMs. It partitions weight columns into salient and non-salient groups based on activation statistics. Salient weights are updated through quantization-aware training, while only the zero-points are updated for non-salient weights. The method is evaluated against several fine-tuning baselines on Llama-2-7B and Qwen2.5-32B.
Critical strengths
- Efficient low-bit fine-tuning is an important problem for practical LLM deployment.
- SALTQ achieves competitive accuracy at INT2 and INT3, outperforming the compared parameter-efficient baselines, particularly at INT2.
- The derivation of efficient zero-point gradients is sound.
Critical weaknesses
- The design criteria in Figure 1 and Table 1 appear somewhat subjective, and their practical importance is not sufficiently justified.
- The paper does not demonstrate the practical inference benefits of SALTQ, such as latency or throughput improvements. It mainly emphasizes training efficiency, whereas inference costs can dominate in long-term deployment.
- The baseline comparisons do not control for computational cost. Since cheaper methods may naturally achieve lower accuracy, the paper should provide clearer accuracy-cost trade-offs to support its efficiency claims.
- The evaluation is limited to two relatively old models, Llama-2-7B and Qwen2.5-32B. Experiments on more recent models would help establish generalizability.
- The presentation is difficult to follow. The introduction is overly long, the method section is dense, Figure 2 provides limited help in understanding the method, and Figures 4 and 5 are hard to understand.
Questions for authors
- The paper shows that activation-based saliency outperforms random column selection. How does it compare with task-gradient-based or quantization-error-aware criteria for selecting salient columns for downstream fine-tuning?
2 comments
Thank you for the reply.
Weakness 1: The rebuttal clarifies the motivation behind the four design conditions, but their practical importance as a combination remains insufficiently justified.
Weakness 3: Figure 7 does not align all methods to the same wall-clock budget, so the cost is not controlled.
Weakness 4: The evaluation on the newer model covers only one downstream task, limiting the evidence for generalizability.
Weakness 5: The proposed presentation improvements are not yet reflected in the manuscript.
I therefore maintain my original rating.
Response to Reviewer Jwcz
Thank you for the review.
Design conditions (Figure 1, Table 1)
Each condition corresponds to a consequence measured in the paper, as the caption of Table 1 will state.
These conditions conflict: trainable weights normally cost the full weight gradient or FP16 columns, and avoiding both means frozen codes or a second quantization. Every existing method gives up at least one, and SALTQ meets all four.
Inference benefits
SALTQ delivers a uniform low-bit checkpoint and so inherits the inference benefits of this format, adding only one to three reorderings (gathers) of the residual stream. We implemented the packed formats in vLLM on its low-bit GPTQ kernel and measured them at INT2 (Llama-2-7B architecture, RTX 6000 Ada, random weights, since latency does not depend on their values).
Against FP16, the SALTQ checkpoint is 4.6 times smaller and decodes 3.7 times faster at batch 1 (1.5 times at batch 16). The gather has no measurable cost: latency is the same without it, and each gather takes 3 to 4 microseconds. SALTQ matches the uniform format of QA-LoRA, while the mixed-precision QEFT model is 8% larger and decodes 14% to 21% slower. Against the standard GPTQ format, SALTQ decodes 5% to 10% slower, because its zero-points are floating-point values.
Controlling computational cost
All methods are trained on the same data with the same epochs, steps and batch size. Under this protocol SALTQ takes the training time of QLoRA (1 h 44 min against 1 h 37 min at INT2), and the more expensive methods are not more accurate. On the commonsense average at INT2, SALTQ scores 74.85. PEQA takes 2.1 times as long and scores 72.18, LR-QAT 3.5 times and 75.19, EfficientQAT 4.8 times and 73.33 (Tables 5, 6). Accuracy does not grow with cost, which is the thesis of the paper: the extra computation goes to weights that do not need the full gradient. The cheaper QEFT and QA-LoRA have the same arithmetic cost per token as SALTQ. Their limit lies in what they can update (a static dense part, frozen integer codes). We will move the accuracy-versus-time plot (Figure 7) to the main text.
Newer models
We chose Llama-2-7B because the compared methods report results on it in their own papers. We have now fine-tuned Qwen3.5-9B (March 2026, linear attention in 24 of 32 layers) on MetaMathQA at INT2, with the same recipe and no re-tuning.
SALTQ is 5.6 points above the best baseline. The premise of the method also holds on this model: the top 1% of channels carry 61% of the activation energy at the attention input (40% on Llama-2-7B).
Presentation
We agree. In the revision we will shorten the introduction and open it with the deliverable and the question of where to spend the full weight gradient. We will lighten the method section by moving implementation details to the appendix. Figure 2 will show which fields are updated in training and what is stored at export. Figure 4 will separate the storage of the salient columns from their training, and Figure 5 will label the default budget and the two controls.
Other selection criteria
We compared the columns selected by other scores with those selected by E[x²] on Llama-2-7B (Jaccard overlap averaged over layers, with a chance level of 0.02 to 0.03).