acceptodds
Under review as a conference paper at ICLR 2027

Same Recipe, Different Weights: Quantization and Decision Stability Across Backends

Abstract

Can one weight-only recipe be reapplied across tensor backends without changing its checkpoint, decisions, or cost? We evaluate symmetric INT8 and grouped INT4 on ten pretrained checkpoints spanning text, vision, audio, dual-encoder, and generative vision-language architectures in one multi-backend implementation on an A100. All 30 cross-backend FP32 state pairs match exactly; none of the 60 independently quantized pairs does. Exact PyTorch packed-state transplantation succeeds for eight ViT/CLIP target interventions and preserves all 800 held-out image decisions per intervention, yet a prespecified 99% calibration-agreement selector misses its holdout target on ViT and CLIP. Across three language-model families, all FP32 and INT8 16-token continuations agree across backends on eight fixed prompts, whereas minimum pairwise INT4 agreement is 5/8 on Qwen3 and Gemma3. Image-conditioned Qwen3-VL continuations agree across all backends for FP32 and INT8, but the worst INT4 pair agrees on only 15/20 images. Median stored-variable bytes fall to 0.254 and 0.184 of FP32 under INT8 and INT4, but portable direct-eager INT4 execution is slower across all three backends. These findings separate recipe reproducibility, exact state transfer, observed decision stability, and deployment cost without treating a changed tensor hash as a changed prediction.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
weight-only quantization, cross-backend reproducibility, numerical equivalence, decision stability, Keras 3, PyTorch, JAX, TensorFlow, inference benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tqk4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether applying the same weight-only quantization scheme to an identical FP32 checkpoint across PyTorch, JAX, and TensorFlow produces consistent quantized weight states and model decisions. The authors evaluate INT8 and INT4 quantization on 10 pretrained models and compare packed states, numerical outputs, task-level decisions, and execution overhead. In addition, the authors conduct packed-state transplantation experiments on ViT and CLIP to further separate differences introduced during quantization from numerical differences introduced during backend execution.

Critical strengths

1、The paper clearly distinguishes inconsistencies in quantized weight states, numerical outputs, and final decisions. This is important because a tensor hash mismatch does not necessarily imply a change in model predictions. 2、The packed-state transplantation experiment is a valuable part of the paper. By forcing different backends to use exactly the same quantized weight state, it can cleanly separate differences introduced during quantization from numerical differences introduced during backend execution.

Critical weaknesses

1、The packed-state transplantation experiment is only conducted on ViT and CLIP, while cross-backend decisions for these two models are already very stable. In contrast, more obvious INT4 decision differences appear on Qwen3-0.6B, Qwen3-VL-2B, and Gemma3-1B. Therefore, for these more important failure cases, the paper still cannot answer whether the final disagreement mainly comes from quantized weight construction or numerical differences during backend execution. 2、The current analysis of state differences relies heavily on tensor-level hash mismatches. This metric only indicates whether a tensor contains any difference and cannot reflect the actual magnitude of the difference. The paper itself also shows that a tensor with a hash mismatch may have only a few elements differing by one quantization level among hundreds of thousands of quantized values.

Questions for authors

1、Could the authors extend the packed-state transplantation experiment to at least one model with clear INT4 disagreement, such as Qwen3-0.6B, Qwen3-VL-2B, or Gemma3-1B? After forcing different backends to use exactly the same packed state, does the observed continuation disagreement still remain? 2、Besides tensor-level hash mismatches, could the authors further report element-level differences, such as the fraction of different quantized codes? 3、Does the current INT4 generation disagreement remain stable over a larger set of prompts or images? This would help determine whether these observations are isolated cases or a more systematic backend effect.

Reviewer_2a6UOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates cross-backend reproducibility of weight-only quantization using a shared Keras 3 implementation across PyTorch, JAX, and TensorFlow. It evaluates ten checkpoints under FP32, INT8, and INT4, distinguishing quantized-state equality, numerical differences, decision agreement, and storage–latency tradeoffs. Complete-state transplantation on ViT and CLIP separates state-construction effects from residual execution differences. The experiments find that independently quantized states differ across backends, while tested image decisions remain consistent; some INT4 text and image-conditioned continuations diverge.

Critical strengths
  1. Controlled separation of state construction and execution. The authors verify identical FP32 sources and, for ViT/CLIP, exactly transplant the quantized state across backends. This cleanly separates tensor-level discrepancies from behavioral changes.

  2. Detailed specification and candid scoping. The appendices specify the quantization arithmetic, inputs, and execution flags in unusual detail. The limitations section is explicit about what the results do not support.

Critical weaknesses
  1. Limited new insight and no mechanism. Most findings are qualitatively expected (floating-point rounding differences, argmax robustness, no speedup without native kernels), and the paper does not explain when discrepancies become consequential. The Sec. 5.4 probe (identical scales, codes off by one level) points to the w/s rounding step, yet this is not pursued, and INT4 generation divergence is not tested with the transplant.

  2. Latency results are affected by uncontrolled overhead. Gradient tracking was not uniformly disabled during timing (Sec. 6, A.1), and each cell has only five same-process samples. The specific slowdown ratios in Table 2 are therefore unreliable.

  3. Small samples and limited practical relevance. The selector “failure” largely follows from a 200-image calibration set, and generation results rest on 8 prompts and 20 upsampled CIFAR images. INT4 disagreements arise where the naive recipe already departs heavily from FP32 (e.g., Qwen2.5 INT4 retains 0/8), so relevance to GPTQ/AWQ or native kernels is unclear.

Questions for authors
  1. Does INT4 generation divergence persist when the quantized state is transplanted, i.e., held identical across backends?

  2. Are the discrepant codes in the CLIP probe at near-boundary values of w/s, and can a specific operation be identified?

  3. Can latency be re-measured with gradients disabled and multiple independent processes?

Reviewer_6EdSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a measurement study to determine whether a single weight-only quantization recipe (symmetric INT8 or grouped INT4) yields the same model across PyTorch, JAX, and TensorFlow. The authors argue that a shared API, such as Keras 3, suggests portability, but quantization still relies on backend-specific arithmetic. They also say prior work reports tensor error, compression, or speed alone, but do not provide details on whether decisions change. They separate the questions of recipe reproducibility, exact state transfer, decision stability, and cost. They quantize ten pretrained checkpoints (text, vision, audio, and a VLM) on each backend in one Keras 3 implementation. They compare tensor hashes, copy PyTorch's quantized state into the other backends, and check decisions on CIFAR-10 images, text prompts, and VLM images. Inferences are timed on A100. All FP32 states match, but the quantized pairs (60 pairs) do not match. At INT4, Qwen3 and Gemma3 agree on only 5 of 8 generations in the worst backend pair. Although INT4 significantly reduces model size, it incurs a runtime penalty of 1.8× to 4.6×.

Critical strengths
  1. The paper explores an interesting study and provides a clean decomposition of backend-level differences into source match, packed state, execution arithmetic, and decision. The transplant test is interesting since it separates quantization-time and execution-time differences.
  2. The evaluation covers a wide range of models across different tasks and key backends.
  3. The study and comparison are thorough and well-designed.
Critical weaknesses
  1. While the paper focuses on an interesting study, the insights are somewhat underwhelming, with most observations being intuitive or known. For example, CRADLE already makes the point that model outputs across different backends are never bit-identical and decision-based checks are needed.
  2. The latency results are on the Keras backend. However, low-precision models use different deployment stacks in practice (TensorRT, ONNX, llama.cpp) that are not covered here. Hence, the general claim that INT4 is 1.8x-4.6x slower should be rephrased
  3. The sample set of image and text generation is relatively small for this type of statistical study. The transplat study is not performed for Qwen3 and Gemma3 for INT4, where the generation actually diverges.
  4. The paper can be made stronger by including a section on impact on real deployment scenarios, recommendations, and possible future work. A clear boundary between what observations are well-known and what are the new observations made in this paper, how well do they generalize is required.
Questions for authors
  1. Can this study be used to find out the root cause of decision level differences observed across different backends? I would expect so. Can you provide a case study where the decomposition used in this work and the proposed methodology can find the root cause?
  2. The latency results are on the Keras backend. However, low-precision models use different deployment stacks in practice (TensorRT, ONNX, llama.cpp) that are not covered here. What is the main reason of the slowdown you observed? How much is it hardware-specific. Can you provide latency numbers with more practical backends that are used in real-world for faster inference?
  3. The sample set of image and text generation is relatively small for this type of statistical study. Can you provide more information/explanation on this sample size?
  4. The transplat study is not performed for Qwen3 and Gemma3 for INT4, where the generation actually diverges. Can you include a study on that?
  5. I found it difficult to separate well-known observations vs. new observations that the proposed decomposition and methodology provided. Can you expand upon this?
  6. With 200 calibration images, is 97.6% vs. 99% beyond sampling error? Please report binomial confidence intervals for the calibration and held-out agreement.
  7. Real deployments use BF16 or FP16. Does cross-backend agreement change there, as Yuan et al. (2025) found for LLMs?
  8. Why did you use upscaled 32×32 CIFAR images for ImageNet classifiers instead of ImageNet validation images? How does margin-based certification change with in-distribution inputs?
  9. Beyond "quantize once and ship integer state," what should a practitioner do differently because of these results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.