- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper uses the task vectors (the difference between pre-trained and fine-tuned weights) to improve post-training data-free quantization to 3 and 4 bits.
Critical strengths
- The method seems effective, cheap, and novel.
- The paper is well-written and well-motivated.
Critical weaknesses
-
It is not clear whether we pay some hidden cost by using the task vectors in the loss. Specifically, how does performance change on other fine-tuning tasks (for which we don't have task vectors), or on the pre-training loss? i.e., does the task vector cause the model to over-align to the specific fine-tuning tasks it was trained on, so we forget the pre-trained model more or degrade out-of-distribution? Do we need to tune alpha to make sure this doesn't happen too much?
-
It would be good to explain more why this is a practically interesting setting (the setting in which we have access to the task vectors, but not the fine-tuning data).
Questions for authors
I would appreciate some experimental results to clarify the above weakness.
Minor comments: • From figure 2a, why should we choose \alpha>0 at all?
• In figure 3b, I wonder what happens for larger \beta ? Does the red line (L/14) start decreasing, or not? If not, then for L/14, we should just use the initial scale and not change it? For B/32 we \beta=0 seems optimal. Any idea what causes this difference?
• Why use SVD (eq. 9) and not the original objective (eq. 9)? Is it for mathematical convenience only, or does it help accuracy? show ablation if there is some impact.
• Table 7: which gray background? I don't see any gray background.
• Figure 1 was not very clear to me when I first read it. e.g., why is the quantization error on the diagonal of x_i and x_j? And why does minimizing this error fail to reach the green region? How does the rotation make the region reachable? Maybe better to put this figure after the method is defined mathematically.
• In table 3, the difference from Hadamard+RTN is negligible in 3-bit L/14 and 4-bit B/32. Also, in Table 10 4bit and table 11, comparison with other methods doesn't show a clear win. Any idea why, in these specific cases, the gain is so small or nonexistent?
5 comments
Response to Reviewer piyo (Part 2/2)
Question 1: Explanation and analysis of and
- About : It is the most basic optimization objective at the beginning of our exploration, which is to minimize the weight quantization error. We retain this term as we step-by-step develop our method. Based on empirical experience, we set .
- About : We further evaluate two larger (100 and 1000) on 3-bit ViT-L/14 in the table below. The results show ViT-L/14 accuracy stays nearly flat at the two larger values. Nevertheless, we should still adjust the scales for ViT-L/14, since table 4 in our submission (i.e., the result of in the table below) shows that fixing the scales lowers accuracy from 90.94% to 90.41%. ViT-B/32 performs better with a smaller . These results suggest that the best value of can vary by model. For any finite , we still refine the scales, and controls how far they move from their initial values. Scale refinement helps both models, but the amount of adjustment that works best may differ.
| ViT-L/14 | Average (%) |
|---|---|
| 86.16 | |
| 90.94 | |
| 91.14 | |
| 91.14 | |
| (Using initial scales) | 90.41 |
Question 2: Explanation of SVD and the original objective
Eq. (9) gives the idea behind our method. It uses the Gram matrix of the task vector to give more weight to quantization errors along directions that matter for the task. We use the SVD form in Eq. (11) because it shows each direction and its weight separately, making the objective function more intuitive and easier to understand. When all singular components are kept, the Gram form and the SVD form are mathematically equivalent. With the same coefficients and regularizers, both give the same solution in theory.
Question 3: The gray background
The gray background refers to the results for "baseline w. TIGER-Q". We have explicitly clarified this in the revised version (caption of Table 7).
Question 4: Clarity of Figure 1
Figure 1 is a simple sketch of the idea. We draw the quantization error on a diagonal to show an example with a component along both axes. The green region represents small errors along directions that matter for the task. Among the available quantized weights, an update can reduce the total error while leaving a large error along one of these directions. The mentioned diagonal path illustrates one such case. The task basis lets us give different weights to the error components. This changes which updates the objective favors and can guide the search toward the green region.
Question 5: Clarification about some modest cases
At these higher bit width settings, quantized models often approach full precision performance. This leaves less room for refinement and narrows the gaps between methods. It is worth noting that our TIGER-Q can also improve accuracy by refining the weights produced by Hadamard+RTN. For example, on ViT-L/14 at 3 bits, this raises average accuracy from 90.92% to 92.41%, a gain of 1.49 percentage points (Table 5 of our submission).
Dear Reviewer piyo,
Hope this message finds you well.
We truly appreciate the time and effort you’ve already dedicated to reviewing our submission. As the discussion period is nearing its end, we would like to check if our rebuttal and additional experiments have satisfactorily addressed your concerns. If you have the opportunity to take a look at our response, we would be grateful for any further thoughts or clarifications you might wish to share.
Thank you again for your contribution to the review process.
Best regards,
The Authors
Response to Reviewer piyo (Part 1/2)
Thank you for your time with our work. All your concerns are carefully addressed as follows. We sincerely hope the revised paper by taken into these valuable comments would meet the expectations of the reviewer, and we are glad to provide further responses if needed.
Weakness 1: Quantization performance beyond the fine-tuning task
As suggested, we evaluate WizardMath-7B at 3 bits with groups of 128 weights, and report GSM8K accuracy for the target task and perplexity (PPL) on WikiText-2 and C4 for language modeling tasks (out-of-distribution and non-target tasks) in the table below.
TIGER-Q improves GSM8K accuracy significantly over SINQ. On both general text corpora, its PPL remains close to SINQ and lower than RTN. This shows that TIGER-Q retains the general text performance of a strong baseline while improving accuracy on the target task. At fixed β=0.02, changing α has little effect on PPL on either corpus. At fixed α=1/(MN), increasing β from 0.02 to 0.2 lowers PPL on both corpora, while GSM8K accuracy also rises from 38.59% to 39.42%. We have added this analysis in our revised version (Appendix B.5)
Weakness 2: Explanation of the calibration-free setting
We would like to clarify that our design can address a practically relevant deployment scenario. With the growing push toward open-source foundation models, organizations often fine-tune public foundation models to their business tasks rather than train models from scratch. For example, organization A may fine-tune a public base model on proprietary data and release the resulting model weights while keeping its data confidential. Organization B or an individual deploying this model can obtain both the base and fine-tuned weights, but cannot access organization A’s private fine-tuning data. In this scenario, the deployer can compute the task vector directly from the available weights and use it to guide quantization without accessing the confidential dataset. Therefore, we believe it offers the open-source community a novel insight for calibration-free quantization. We have added this explanation in the revised version (Line53-69).