- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies W4A4 quantization on VAEs for the World Action Models (WAMs). It first demonstrates that although joint QAT keeps reconstruction quality close to FP32 (VBench-7 0.7403 vs. 0.7409 on Wan2.1), the action success rate significantly degrades, from 95.5% to 10.5% on LIBERO (Sec. 6.2). The paper attributes the failure to decoder activation quantize-dequantize, which rotates the latent gradient by 67.45 degrees and causes persistent encoder drift. Thus, the authors propose LatentQuant with two stages: First aligning the quantized encoder to full-precision encoder sample-wise, then freezing the encoder and adapts only the decoder. This keeps the success rate (+0.25% on LIBERO, -1.0% on RoboTwin) while giving 1.17-1.26x speedup for the VAE module. The authors also conduct experiments on real robots, showing competitive success rate with the original policy (Sec. 6.4).
Critical strengths
- The "policy-facing latent contract" is clear and useful. And it’s very intuitive based on the analysis on the joint-QAT, whose collapse demonstrates that reconstruction metrics can hide policy incompatibility.
- The mechanism analysis is careful. It includes decoder-only QDQ, a one-site causal chain, a telescoping layerwise sweep, and evaluation of the master encoder with all QDQ removed, and it separates learned drift from instantaneous quantization error. The authors also conducted experiments on real robots, which has practical value to some extent.
- The intuition is natural, and the writing is clear and quite easy to understanding.
Critical weaknesses
- Limited practical usage. The computation overhead of VAE takes very little portion within the whole backbone, while the denoising process for the DiT should be the focus of acceleration. Furthermore, even considering the VAE only, the 1.17-1.26x speedup is limited. Other orthogonal methods about distillation, adaptive action chunk lengths, or CUDA graphs can easily surpass this acceleration ratio end-to-end with similar success rate on LIBERO / RoboTwin. Besides, direct NVFP4 already keeps 94.38% / 68.00% success. Although it decreases the reconstruction quality, the major focus of this paper (as well as the main contribution) is to maintain the success rate of robotic tasks, rather than generating or reconstructing videos.
- Limited demonstration of real robots. To enhance the practical applicability of the paper, the paper should report the success rate on these 15 tasks (Sec. 6.4, Fig. 3), rather than only demonstrate the progressive score. Also it’s recommended that the authors provide a demo / video for the actual manipulation process to contrast the operation and latency for the baseline and the method.
- Limited novelty of the analysis. It is a long-established conclusion in the field of visual generation that the capability of VAE reconstruction is not equivalent to its efficiency when applied to downstream tasks. To solve this, the pipeline is also too simple to be a substantial methodological contribution. The solution amounts to (i) feature distillation of the quantized encoder toward its full-precision teacher with a channel-normalized MSE, and (ii) freezing the encoder while the decoder is fine-tuned for reconstruction. Both are standard practices in quantized feature distillation and staged QAT.
Questions for authors
- What share of end-to-end WAM latency does the VAE take, and what is the resulting end-to-end speedup? The authors are recommended to provide a latency breakup table to better illustrate this.
- In the field of quantization for WAMs, how does LatentQuant compare with stronger PTQ methods applied to the VAE? Moreover, if we apply quantization to the DiT backbone, can we achieve better acceleration ratio?
- What's the edge device for real robot experiments? Do you compare the acceleration ratio with B300 simulator?
7 comments
Thank you for the review and clarification. We provide additional system-level results and address the methodological and practical contributions of LatentQuant.
Q1 & W1. VAE latency breakdown and end-to-end acceleration.
After submission, we further optimized our WAM inference pipeline and native NVFP4 implementation. In the optimized B300 profile, VAE encoding accounts for 29.6% of GPU execution time (38.6/130.6 ms), while prefill, eight-step denoising, and other stages account for the remaining 92.0 ms.
LatentQuant reduces VAE encoding to 15.9 ms (2.43× speedup), bringing total GPU execution time from 130.6 ms to 107.9 ms, corresponding to 1.21× pipeline acceleration. Video decoding is outside the action-return critical path.
These results demonstrate the practical significance of VAE acceleration in WAM inference. We will incorporate the detailed latency breakdown and updated system-level results into the revised paper.
Q2. Comparison with stronger PTQ methods and DiT optimization.
To our knowledge, LatentQuant is the first systematic study of NVFP4 W4A4 video-VAE quantization for frozen WAM policies. Existing VAE PTQ methods, such as PassionSR (CVPR 2025), focus on W8A8/W6A6 image super-resolution, whereas our setting requires preserving latent representations consumed by a frozen action policy.
Under identical NVFP4 arithmetic and layer coverage, our controlled baselines isolate the contribution of different adaptation strategies. Compared with Direct NVFP4, LatentQuant improves VBench-7 from 0.7266 to 0.7340 on Wan2.1 VAE and from 0.7325 to 0.7414 on Wan2.2 VAE. Action deviation decreases from 0.033 to 0.018 and from 0.202 to 0.003, respectively, while maintaining near-reference control performance (Table 1).
Regarding the suggested alternatives, few-step distillation, adaptive action chunking, and CUDA Graph/compilation optimize denoising steps, action-generation scheduling, and execution overhead, respectively. LatentQuant targets a complementary component: low-precision visual encoding while preserving compatibility with the frozen action policy.
As backbone inference becomes more efficient, visual encoding accounts for a larger fraction of the remaining computation. In our optimized WAM, VAE encoding already contributes 29.6% of GPU execution time, demonstrating the practical value of VAE quantization alongside these optimizations.
Q3. Real-robot deployment hardware.
Our real-robot inference uses cloud-based serving on NVIDIA B300 GPUs, with robots connecting remotely to the inference service rather than executing the policy onboard.
The native NVFP4 performance measurements are conducted on physical B300 hardware, not a simulator, including online activation scaling and packing. The optimized VAE is therefore evaluated on the same GPU platform used for cloud-based robot inference.
W2. Real-robot evaluation protocol.
Our real-robot evaluation covers 15 manipulation tasks, with three independent trials per method, totaling 45 physical executions per method.
We adopt a predefined milestone-based progress scoring protocol rather than binary task success. Each task is decomposed into intermediate manipulation stages, and the score reflects the progress achieved throughout the task. This provides a more fine-grained assessment of multi-stage manipulation, capturing partial completion that binary success would not reflect.
Appendix D.5 (Table 9) reports the complete task-level and trial-level results under this evaluation protocol.
W3. Mechanistic and methodological novelty.
Our contribution goes beyond the established observation that reconstruction quality does not guarantee downstream performance. We identify a specific optimization-induced failure mechanism under low-bit VAE QAT.
Our controlled decoder-only quantization experiments show that activation QDQ redirects reconstruction gradients flowing into the encoder. A one-site intervention produces a 67.45° latent-gradient rotation, and the resulting representation drift persists even after removing all QDQ operations (§4; Appendix E). This establishes how decoder quantization can induce persistent changes in the representation consumed by a frozen policy.
The consequences are substantial: Joint QAT preserves near-reference reconstruction quality on Wan2.1 (Robot VBench-7: 0.7403 vs. 0.7409), yet LIBERO success decreases from 95.50% to 10.50%.
LatentQuant directly addresses this mechanism through a two-stage optimization procedure. Stage I aligns the quantized encoder with the reference representation; Stage II freezes this interface while adapting the decoder. This converts latent preservation from a soft regularization objective into an explicit optimization constraint, preventing reconstruction-driven encoder drift while recovering reconstruction quality.
Feedbacks for the Rebuttal of LatentQuant
Q1 & W1
a) The VAE takes 29.6% of the pipeline now, is it calculated on the optimized WAM inference (Your response to Q2, last line)? I think you should integrate the method of the optimization for WAM into the revised paper.
b) What different techniques do you apply to significantly improving the ratio from 1.17x-1.26x to 2.43x? I think it's an important contribution which should be added.
c) Since the decoding is off the action path (I have just realized it), why do we still need Stage II and the decoder-side analysis?
d) Your technique of quantizing VAE is indeed orthogonal to the methods of distillation, CUDA graph, adaptive action chunking, etc. But can it be effectively combined with these methods? (i.e. Maintaining success rate, while achiving higher acceleration ratios over simply adopting these methods) Otherwise, it's a wiser choice to directly adopt these tested-over-time methods.
Q2: VBench and action deviation are only indicators, while end-to-end SR is important. Direct NVFP4 already reaches 94.38% on LIBERO and 68.00% on RoboTwin. If you achieve higher SR or latency gains over it in your new 2.43x speedup, I think it's more important to report it.
Q3: Resolved. But as I said in Q1, since inference runs on the cloud, please also report end-to-end per-step latency including the network round trip.
W2: I've noticed your Table 9, but it reports milestone scores, what I want it success rate (i.e. Only fully and stably completing a task is seen as 100%, otherwise the score is 0%). Also, I think 3 tests is too few, 10-12 trials would be better.
W3: The persistent drift identify a concrete failure mechanism indeed; I accept this as a deep and interesting insight. However, considering that the Stage II is no longer needed for the robotic task (Q1), I think this analysis is not related to the main contribution of LatentQuant, and it's a common practice for Stage I which lacks novelty.
Clarification of Weakness #1
In weaknesses 1, I mention "Other orthogonal methods about distillation, adaptive action chunk lengths, or CUDA graphs". Here distillation means few-step distillation (e.g. 10 steps -> 3 steps), adaptive action chunk lengths means we can flexible select a shorter or longer action chunk length based on current visual inputs, previous actions, action entropies (Reference: [1]) etc. CUDA graph / Compilation is also applied to the whole backbone, not only the VAE module. Sorry about any potential ambiguity!
[1] Y. Liang, X. Wang, K. Wang, S. Wang, X. Peng, H. Chen, D. K. H. Chua, and P. Vadakkepat. Adaptive Action Chunking at Inference-time for Vision-Language-Action Models. CVPR 2026.