- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether applying the same weight-only quantization scheme to an identical FP32 checkpoint across PyTorch, JAX, and TensorFlow produces consistent quantized weight states and model decisions. The authors evaluate INT8 and INT4 quantization on 10 pretrained models and compare packed states, numerical outputs, task-level decisions, and execution overhead. In addition, the authors conduct packed-state transplantation experiments on ViT and CLIP to further separate differences introduced during quantization from numerical differences introduced during backend execution.
Critical strengths
1、The paper clearly distinguishes inconsistencies in quantized weight states, numerical outputs, and final decisions. This is important because a tensor hash mismatch does not necessarily imply a change in model predictions. 2、The packed-state transplantation experiment is a valuable part of the paper. By forcing different backends to use exactly the same quantized weight state, it can cleanly separate differences introduced during quantization from numerical differences introduced during backend execution.
Critical weaknesses
1、The packed-state transplantation experiment is only conducted on ViT and CLIP, while cross-backend decisions for these two models are already very stable. In contrast, more obvious INT4 decision differences appear on Qwen3-0.6B, Qwen3-VL-2B, and Gemma3-1B. Therefore, for these more important failure cases, the paper still cannot answer whether the final disagreement mainly comes from quantized weight construction or numerical differences during backend execution. 2、The current analysis of state differences relies heavily on tensor-level hash mismatches. This metric only indicates whether a tensor contains any difference and cannot reflect the actual magnitude of the difference. The paper itself also shows that a tensor with a hash mismatch may have only a few elements differing by one quantization level among hundreds of thousands of quantized values.
Questions for authors
1、Could the authors extend the packed-state transplantation experiment to at least one model with clear INT4 disagreement, such as Qwen3-0.6B, Qwen3-VL-2B, or Gemma3-1B? After forcing different backends to use exactly the same packed state, does the observed continuation disagreement still remain? 2、Besides tensor-level hash mismatches, could the authors further report element-level differences, such as the fraction of different quantized codes? 3、Does the current INT4 generation disagreement remain stable over a larger set of prompts or images? This would help determine whether these observations are isolated cases or a more systematic backend effect.