- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose RAG-QAF, a new technique to improve robustness of quantization-aware fine-tuning for both in-distribution and out-of-distribution scenarios. They examine the robustness challenges faced by quantized model and design a method of 1) local adversarial optimization guided by quantization-error and 2) global curriculum schedule for adaptation. The result shows promising improvement on overheads comparing other baseline approaches as well as performance on the benchmarks.
Critical strengths
- The paper is well-written with clear figure presentations to explain the current limitations and the proposed methodology (figure 1 and 2).
- It is interesting to see a similar idea as adversarial training but on the normalized channel space guided by quantization error, and then integrated with curriculum learning for smoother control. The authors carefully compare the proposed method with adversarial optimization alone and quantization error guidance in figure 3.
- It is good the see that the method improves the overhead over other approaches as the computation can be done in the same forward/backward pass.
Critical weaknesses
- A core design choice of the proposed method is the local adversarial optimization, the authors state that the adversarial noise is added to the frozen RMSNorm scaling parameter without detailed explanation and empirical testing on how and why this is determined.
- None of the reported results show standard deviation from independent runs. Some of the improvements from RAG-QAF over other baselines (table 1) and ablation results (figure 3) are not consistent and many of the numbers are very close. It would make the claims more grounded with reproducibility and std.
Questions for authors
Thanks for submitting this work to ICLR 2027! The questions below are mainly based on the comments in the previous section. I think the proposed method is promising with good presentation and a combination of two main well-motivated techniques, I am open to further discussion if the questions below can be addressed.
- What if applying the proposed local adversarial optimization to different locations other than the RMSNorm scaling parameters (e.g., different layers or other parameters) during fine tuning? How much does the location matter comparing to techniques from related works focusing on other parameters? Does RAG-QAF still outperform others in those cases? It would be interesting to see an ablation on the proposed technique and the applied location.
- Given that some improvements are small comparing to the baselines or ablation components, what are the standard deviations from multiple independent runs? Are the resulting numbers overlapping with each other a lot?