acceptodds
Under review as a conference paper at ICLR 2027

LatentQuant: Preserving the Policy-Facing Latent Contract under NVFP4 VAE Quantization

Abstract

Recent world action models (WAMs) reuse pretrained video VAEs whose encoder latents directly condition downstream action policies. Quantization must therefore preserve not only reconstruction fidelity but also the policy-facing latent contract expected by the frozen policy. Direct NVFP4 leaves W4A4 quantization error uncompensated, whereas joint quantization-aware training (QAT) can recover reconstruction by moving this representation. On Wan2.1, joint QAT nearly matches FP32 VBench-7 (0.7403 versus 0.7409), yet LIBERO success collapses from 95.5% to 10.5%. Controlled decoder-only experiments show that activation quantize–dequantize operations alter the reconstruction signal and decoder Jacobian, redirecting the gradient returned to the encoder and inducing persistent latent drift. Based on this mechanism, we introduce LatentQuant, a two-stage NVFP4 QAT framework that first aligns the quantized encoder with its high-precision counterpart, then freezes it while adapting the decoder. Across Wan2.1 and Wan2.2, LatentQuant preserves near-baseline control and high reconstruction quality, achieving 95.75% success on LIBERO and 68.8% on RoboTwin. On NVIDIA B300 GPUs, NVFP4 execution achieves 1.17×–1.26× end-to-end VAE speedups over BF16 cuDNN.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
quantization-aware training, NVFP4, video VAE, world action models, latent representation alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BKJZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies W4A4 quantization on VAEs for the World Action Models (WAMs). It first demonstrates that although joint QAT keeps reconstruction quality close to FP32 (VBench-7 0.7403 vs. 0.7409 on Wan2.1), the action success rate significantly degrades, from 95.5% to 10.5% on LIBERO (Sec. 6.2). The paper attributes the failure to decoder activation quantize-dequantize, which rotates the latent gradient by 67.45 degrees and causes persistent encoder drift. Thus, the authors propose LatentQuant with two stages: First aligning the quantized encoder to full-precision encoder sample-wise, then freezing the encoder and adapts only the decoder. This keeps the success rate (+0.25% on LIBERO, -1.0% on RoboTwin) while giving 1.17-1.26x speedup for the VAE module. The authors also conduct experiments on real robots, showing competitive success rate with the original policy (Sec. 6.4).

Critical strengths
  1. The "policy-facing latent contract" is clear and useful. And it’s very intuitive based on the analysis on the joint-QAT, whose collapse demonstrates that reconstruction metrics can hide policy incompatibility.
  2. The mechanism analysis is careful. It includes decoder-only QDQ, a one-site causal chain, a telescoping layerwise sweep, and evaluation of the master encoder with all QDQ removed, and it separates learned drift from instantaneous quantization error. The authors also conducted experiments on real robots, which has practical value to some extent.
  3. The intuition is natural, and the writing is clear and quite easy to understanding.
Critical weaknesses
  1. Limited practical usage. The computation overhead of VAE takes very little portion within the whole backbone, while the denoising process for the DiT should be the focus of acceleration. Furthermore, even considering the VAE only, the 1.17-1.26x speedup is limited. Other orthogonal methods about distillation, adaptive action chunk lengths, or CUDA graphs can easily surpass this acceleration ratio end-to-end with similar success rate on LIBERO / RoboTwin. Besides, direct NVFP4 already keeps 94.38% / 68.00% success. Although it decreases the reconstruction quality, the major focus of this paper (as well as the main contribution) is to maintain the success rate of robotic tasks, rather than generating or reconstructing videos.
  2. Limited demonstration of real robots. To enhance the practical applicability of the paper, the paper should report the success rate on these 15 tasks (Sec. 6.4, Fig. 3), rather than only demonstrate the progressive score. Also it’s recommended that the authors provide a demo / video for the actual manipulation process to contrast the operation and latency for the baseline and the method.
  3. Limited novelty of the analysis. It is a long-established conclusion in the field of visual generation that the capability of VAE reconstruction is not equivalent to its efficiency when applied to downstream tasks. To solve this, the pipeline is also too simple to be a substantial methodological contribution. The solution amounts to (i) feature distillation of the quantized encoder toward its full-precision teacher with a channel-normalized MSE, and (ii) freezing the encoder while the decoder is fine-tuned for reconstruction. Both are standard practices in quantized feature distillation and staged QAT.
Questions for authors
  1. What share of end-to-end WAM latency does the VAE take, and what is the resulting end-to-end speedup? The authors are recommended to provide a latency breakup table to better illustrate this.
  2. In the field of quantization for WAMs, how does LatentQuant compare with stronger PTQ methods applied to the VAE? Moreover, if we apply quantization to the DiT backbone, can we achieve better acceleration ratio?
  3. What's the edge device for real robot experiments? Do you compare the acceleration ratio with B300 simulator?
7 comments
Reviewer_BKJZOpenReview ↗

Clarification of Weakness #1

In weaknesses 1, I mention "Other orthogonal methods about distillation, adaptive action chunk lengths, or CUDA graphs". Here distillation means few-step distillation (e.g. 10 steps -> 3 steps), adaptive action chunk lengths means we can flexible select a shorter or longer action chunk length based on current visual inputs, previous actions, action entropies (Reference: [1]) etc. CUDA graph / Compilation is also applied to the whole backbone, not only the VAE module. Sorry about any potential ambiguity!

[1] Y. Liang, X. Wang, K. Wang, S. Wang, X. Peng, H. Chen, D. K. H. Chua, and P. Vadakkepat. Adaptive Action Chunking at Inference-time for Vision-Language-Action Models. CVPR 2026.

Thank you for the review and clarification. We provide additional system-level results and address the methodological and practical contributions of LatentQuant.

Q1 & W1. VAE latency breakdown and end-to-end acceleration.

After submission, we further optimized our WAM inference pipeline and native NVFP4 implementation. In the optimized B300 profile, VAE encoding accounts for 29.6% of GPU execution time (38.6/130.6 ms), while prefill, eight-step denoising, and other stages account for the remaining 92.0 ms.

LatentQuant reduces VAE encoding to 15.9 ms (2.43× speedup), bringing total GPU execution time from 130.6 ms to 107.9 ms, corresponding to 1.21× pipeline acceleration. Video decoding is outside the action-return critical path.

These results demonstrate the practical significance of VAE acceleration in WAM inference. We will incorporate the detailed latency breakdown and updated system-level results into the revised paper.

Q2. Comparison with stronger PTQ methods and DiT optimization.

To our knowledge, LatentQuant is the first systematic study of NVFP4 W4A4 video-VAE quantization for frozen WAM policies. Existing VAE PTQ methods, such as PassionSR (CVPR 2025), focus on W8A8/W6A6 image super-resolution, whereas our setting requires preserving latent representations consumed by a frozen action policy.

Under identical NVFP4 arithmetic and layer coverage, our controlled baselines isolate the contribution of different adaptation strategies. Compared with Direct NVFP4, LatentQuant improves VBench-7 from 0.7266 to 0.7340 on Wan2.1 VAE and from 0.7325 to 0.7414 on Wan2.2 VAE. Action deviation decreases from 0.033 to 0.018 and from 0.202 to 0.003, respectively, while maintaining near-reference control performance (Table 1).

Regarding the suggested alternatives, few-step distillation, adaptive action chunking, and CUDA Graph/compilation optimize denoising steps, action-generation scheduling, and execution overhead, respectively. LatentQuant targets a complementary component: low-precision visual encoding while preserving compatibility with the frozen action policy.

As backbone inference becomes more efficient, visual encoding accounts for a larger fraction of the remaining computation. In our optimized WAM, VAE encoding already contributes 29.6% of GPU execution time, demonstrating the practical value of VAE quantization alongside these optimizations.

Q3. Real-robot deployment hardware.

Our real-robot inference uses cloud-based serving on NVIDIA B300 GPUs, with robots connecting remotely to the inference service rather than executing the policy onboard.

The native NVFP4 performance measurements are conducted on physical B300 hardware, not a simulator, including online activation scaling and packing. The optimized VAE is therefore evaluated on the same GPU platform used for cloud-based robot inference.

W2. Real-robot evaluation protocol.

Our real-robot evaluation covers 15 manipulation tasks, with three independent trials per method, totaling 45 physical executions per method.

We adopt a predefined milestone-based progress scoring protocol rather than binary task success. Each task is decomposed into intermediate manipulation stages, and the score reflects the progress achieved throughout the task. This provides a more fine-grained assessment of multi-stage manipulation, capturing partial completion that binary success would not reflect.

Appendix D.5 (Table 9) reports the complete task-level and trial-level results under this evaluation protocol.

W3. Mechanistic and methodological novelty.

Our contribution goes beyond the established observation that reconstruction quality does not guarantee downstream performance. We identify a specific optimization-induced failure mechanism under low-bit VAE QAT.

Our controlled decoder-only quantization experiments show that activation QDQ redirects reconstruction gradients flowing into the encoder. A one-site intervention produces a 67.45° latent-gradient rotation, and the resulting representation drift persists even after removing all QDQ operations (§4; Appendix E). This establishes how decoder quantization can induce persistent changes in the representation consumed by a frozen policy.

The consequences are substantial: Joint QAT preserves near-reference reconstruction quality on Wan2.1 (Robot VBench-7: 0.7403 vs. 0.7409), yet LIBERO success decreases from 95.50% to 10.50%.

LatentQuant directly addresses this mechanism through a two-stage optimization procedure. Stage I aligns the quantized encoder with the reference representation; Stage II freezes this interface while adapting the decoder. This converts latent preservation from a soft regularization objective into an explicit optimization constraint, preventing reconstruction-driven encoder drift while recovering reconstruction quality.

Reviewer_BKJZOpenReview ↗

Feedbacks for the Rebuttal of LatentQuant

Q1 & W1

a) The VAE takes 29.6% of the pipeline now, is it calculated on the optimized WAM inference (Your response to Q2, last line)? I think you should integrate the method of the optimization for WAM into the revised paper.

b) What different techniques do you apply to significantly improving the ratio from 1.17x-1.26x to 2.43x? I think it's an important contribution which should be added.

c) Since the decoding is off the action path (I have just realized it), why do we still need Stage II and the decoder-side analysis?

d) Your technique of quantizing VAE is indeed orthogonal to the methods of distillation, CUDA graph, adaptive action chunking, etc. But can it be effectively combined with these methods? (i.e. Maintaining success rate, while achiving higher acceleration ratios over simply adopting these methods) Otherwise, it's a wiser choice to directly adopt these tested-over-time methods.

Q2: VBench and action deviation are only indicators, while end-to-end SR is important. Direct NVFP4 already reaches 94.38% on LIBERO and 68.00% on RoboTwin. If you achieve higher SR or latency gains over it in your new 2.43x speedup, I think it's more important to report it.

Q3: Resolved. But as I said in Q1, since inference runs on the cloud, please also report end-to-end per-step latency including the network round trip.

W2: I've noticed your Table 9, but it reports milestone scores, what I want it success rate (i.e. Only fully and stably completing a task is seen as 100%, otherwise the score is 0%). Also, I think 3 tests is too few, 10-12 trials would be better.

W3: The persistent drift identify a concrete failure mechanism indeed; I accept this as a deep and interesting insight. However, considering that the Stage II is no longer needed for the robotic task (Q1), I think this analysis is not related to the main contribution of LatentQuant, and it's a common practice for Stage I which lacks novelty.

Reviewer_VtN8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies NVFP4 quantization of video VAEs used by world action models (WAMs). Joint quantization-aware training (QAT) can keep reconstruction quality high while changing the latent representation used by a frozen policy. This can harm control. LatentQuant uses two stages: sample-wise encoder alignment, followed by decoder training with the encoder frozen. The paper tests the cause of latent drift, control performance, and VAE speed on an NVFP4 backend.

Critical strengths
  1. The paper tests a possible cause of latent drift. Quantizing only decoder activations changes the gradient sent to a high-precision encoder. After training updates, the latent change remains even when all quantize–dequantize operations (QDQ) are switched off (Section 4; Appendix E). This supports a path from decoder quantization to a lasting change in the encoder.
  2. The control tests address the main problem. Table 1 shows that joint QAT can retain good reconstruction scores while control performance drops sharply. LatentQuant brings control performance close to the high-precision reference in both tested settings. The speed tests also include activation packing during execution, which is part of the real NVFP4 cost.
Critical weaknesses
  1. The paper has not fully shown why freezing is needed. Joint+KL in the main table matches channel statistics. Appendix B.2 tests the closer alternative: joint training with sample-wise alignment. That test reduces latent and action error; an alignment weight of 0.03 also improves PSNR over the version without alignment. However, it is a short test without closed-loop control. Freeze-Only in Table 2 also lacks control results. These two control comparisons would help show which part of LatentQuant is needed.
Questions for authors
  1. Clarification: were control runs paired across methods using the same seeds or starting conditions? Please report the paired outcomes or confidence intervals that account for differences between tasks.
  2. Clarification: please list the data and training budget for each baseline. Were the main control results obtained with QDQ or the native NVFP4 backend? How was agreement between these two paths checked?
1 comment

Thank you for the constructive review.

1. Joint Alignment, Freeze-Only, and the Role of Freezing

We extended Joint+Align to full-scale training with λ=0.03, achieving 94.25% LIBERO success and 0.7489 VBench-7, compared with 95.75% and 0.7513 for LatentQuant.

Our training-dynamics analysis explains the distinction: joint optimization allows reconstruction gradients to continuously influence the encoder. Decoder-only QDQ experiments demonstrate a 67.45° latent-gradient rotation and persistent encoder drift after QDQ removal (§4; Appendix E). LatentQuant instead freezes the aligned encoder during decoder adaptation, turning latent preservation from a soft regularization objective into an explicit optimization constraint.

Table 2 further isolates the role of Stage I. Freeze-Only improves Robot VBench-7 from 0.7266 to 0.7326, while retaining Direct NVFP4's latent/action deviations (0.288/0.033). LatentQuant further reduces these deviations to 0.202/0.018, achieving 0.7340 VBench-7. Since the decoder is not used for action generation, Freeze-Only retains the same policy-facing encoder and action computation as Direct NVFP4, which achieves 94.38% LIBERO success.

Together, these controls demonstrate the complementary roles of the two stages: Stage I corrects encoder quantization error at the policy interface, while Stage II recovers reconstruction without altering that interface.

2. Paired Control Evaluation and Confidence Intervals

Our original LIBERO and RoboTwin evaluations use matched random seeds and initial environment states across methods, with identical task lists, reset procedures, frozen policies, and success criteria (Appendix B.4).

LatentQuant achieves 95.75% (766/800) on LIBERO versus 95.50% (764/800) for FP32, and 68.80% (344/500) on RoboTwin versus 69.80% (349/500) for BF16.

To account for task heterogeneity, we performed 10,000 matched-task bootstrap resamples using the original RoboTwin task-level results (Table 8). LatentQuant differs from the high-precision reference by −1.0 percentage point (95% CI [−5.2, +2.8]) and from Direct NVFP4 by +0.8 percentage points (95% CI [−2.8, +4.2]). These results support near-reference control performance across diverse manipulation tasks.

3. Training Protocols and QDQ/Native Execution

All methods initialize from pretrained Wan VAEs and follow the same NVFP4 W4A4 quantization configuration, with frozen downstream policies. LatentQuant and Joint+Align use 17-frame, 208×224 video clips. LatentQuant performs one epoch of encoder alignment followed by five epochs of decoder adaptation. Joint+Align trains both components for six epochs with λ=0.03, matching the full-scale training schedule, while Freeze-Only matches the five-epoch decoder adaptation stage. Direct NVFP4 requires no additional training, and the original Joint QAT and Joint+KL baselines jointly optimize the encoder and decoder. LatentQuant checkpoints are selected using validation alignment loss and held-out reconstruction, while Joint+Align uses held-out PSNR.

The control experiments use strict NVFP4 W4A4 QDQ, while deployment acceleration is measured using native NVFP4 kernels. Appendix F.4 documents numerical correctness checks against the reference quantization implementation across tensor layouts, data types, quantization boundaries, and sequence/chunk execution.

We additionally performed a direct QDQ/native comparison of the complete Wan2.1 VAE using the same checkpoint and matched 60-convolution quantization coverage, across 16 LIBERO and robot-video clips. Latent relative-L2 differences between native NVFP4 kernel and QDQ kernel were 4.16% and 4.00%, respectively, while reconstruction PSNR differed by only +0.053 dB and +0.002 dB. Separately, we verified that further native kernel optimizations preserved encoder latents and reconstructions bitwise identically across all 16 clips.

Reviewer_6RHBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies NVFP4 quantization of video VAEs whose encoder representations are consumed by frozen world-action models. It shows that joint reconstruction-based QAT can preserve reconstruction quality while substantially degrading downstream control. The proposed LatentQuant first aligns the quantized encoder outputs with a high-precision reference on corresponding samples, then freezes the encoder and adapts the quantized decoder. Validation includes two VAE configurations within WALL-WM, downstream simulation and physical-robot evaluations, controlled gradient diagnostics, and native VAE execution measurements on NVIDIA B300 GPUs.

Critical strengths

The paper identifies a practically relevant failure mode: reconstruction quality alone can conceal substantial degradation in a frozen downstream policy. The decoder-only quantization interventions and evaluation after removing QDQ provide useful evidence that quantization-induced training updates can cause persistent encoder representation drift. The native NVFP4 backend also strengthens the practical evaluation by including online activation scaling and packing in the reported VAE timing measurements.

Critical weaknesses
  1. Limited methodological advance and demonstrated benefit. The method combines sample-wise latent alignment with encoder freezing and decoder adaptation. Its advantage over joint QAT with the same alignment loss is not established under the full training protocol. Reported VAE speedups are modest (1.17×–1.26×), while Direct NVFP4 already retains near-reference control performance.

  2. Missing comparisons with existing VAE quantization methods. The baselines primarily compare training variants rather than published VAE quantization approaches. Consequently, the advantage over existing methods remains unclear.

  3. Limited cross-WAM validation. Both evaluated configurations use WALL-WM. Results on two VAE configurations do not establish applicability across independent WAM architectures.

  4. Unclear system-level significance and scope. The paper does not establish that the VAE is a major bottleneck in WAM inference or report complete-policy acceleration. It also insufficiently distinguishes WAM-specific challenges from the broader problem of preserving representations consumed by frozen downstream models.

  5. Deployment terminology needs clarification. The reported “end-to-end” speedups cover VAE encode–decode execution, rather than the complete WAM pipeline. This distinction should be consistently explicit.

Questions for authors
  1. Under matched full-scale training and checkpoint selection, how does LatentQuant compare with joint QAT using a tuned sample-wise alignment loss?
  2. How does LatentQuant compare with published VAE quantization methods adapted to comparable model and precision settings?
  3. Is there evidence that the method preserves control performance on an independent WAM architecture, such as DreamZero or LingBot-VA?
  4. What fractions of action-generation latency are attributable to VAE encoding and decoding? Is decoding on the control-critical path, and what complete-policy speedup is achieved?
  5. Which aspects of the problem require a WAM-specific treatment, and under what conditions would the proposed method apply to other frozen downstream models?
1 comment

Thank you for the questions.

Q1. Comparison under matched training and checkpoint selection.

We extended Joint+Align to full-scale training, using λ=0.03 selected from the alignment sweep. Joint+Align achieves a VBench-7 of 0.7489 and a LIBERO success rate of 94.25%, compared with 0.7513 and 95.75% for LatentQuant. While the success-rate gap is modest on LIBERO, preserving the policy-facing representation may be even more consequential under the greater visual and interaction variability of real-world manipulation.

The key distinction is how the policy-facing latent is preserved. Joint QAT combines reconstruction and alignment gradients, leaving the encoder continuously affected by decoder optimization. Our training-dynamics analysis demonstrates that decoder quantization perturbs encoder gradients and causes persistent latent drift (§4; Appendix E). LatentQuant instead establishes the aligned encoder interface first, then freezes it during decoder adaptation. This turns latent preservation from a soft regularization objective into an explicit optimization constraint, preventing reconstruction-driven encoder drift.

Q2. Comparison with published VAE quantization methods.

To our knowledge, this is the first systematic study of low-bit video-VAE quantization for frozen WAM policies, linking quantization-induced representation drift to closed-loop control. Existing VAE quantization work, such as PassionSR (CVPR 2025), focuses on W8A8/W6A6 post-training quantization for image super-resolution, optimizing quantization and transformation parameters while keeping the original model weights fixed.

Our work addresses a different setting: NVFP4 W4A4 quantization-aware training of video VAEs whose latent representations are consumed by frozen action policies. Our comparisons include Direct NVFP4, Joint QAT, and joint training with latent regularization under identical quantization formats and layer coverage. These controlled baselines isolate the effects of training strategies on reconstruction quality, latent compatibility, and downstream control, directly validating the effectiveness of LatentQuant.

Q3. Evidence on an independent WAM.

We further validated LatentQuant on LingBot-VA across all LIBERO suites, totaling 800 closed-loop episodes per method.

The original BF16 VAE achieved 448/800 (56.00%) success, while LatentQuant with NVFP4 achieved 450/800 (56.25%), preserving overall control performance. The per-suite results are also consistent: Long (78% vs. 80%), Object (57% vs. 57%), Spatial (81% vs. 78%), and Goal (8% vs. 10%), with BF16 listed first.

These results extend our validation to other WAMs and demonstrate that LatentQuant preserves downstream control performance on an independent WAM architecture without modifying the action policies.

Q4. Encoding and decoding latency, the critical path, and policy acceleration.

In our optimized B300 WAM pipeline, VAE encoding takes 38.6 ms out of 130.6 ms (29.6%), while prefill, eight-step denoising, and other stages account for the remaining 92.0 ms. Video decoding is outside the action-return critical path and does not contribute to action-generation latency.

After submission, we further optimized our native NVFP4 VAE implementation, achieving a 2.43× encoder speedup (38.6 → 15.9 ms). With the updated encoder, the GPU pipeline improved from 130.6 ms to 107.9 ms, corresponding to 1.21× overall acceleration. We will incorporate our latest kernel optimizations and updated system-level performance results into the revised paper.

Q5. WAM-specific aspects and transfer conditions.

The core challenge of preserving representations consumed by frozen downstream models is general. What makes WAMs particularly sensitive is that VAE latents directly condition action generation, where representation drift can alter control decisions and propagate through closed-loop interactions. Therefore, WAM-specific treatment primarily lies in evaluating action fidelity and closed-loop success, beyond conventional reconstruction metrics.

LatentQuant itself does not depend on WAM-specific architectures or objectives. It applies when a pretrained encoder provides representations to a frozen downstream model, a compatible high-precision reference encoder is available for sample-wise alignment, and reconstruction adaptation can be performed with the encoder frozen. The two-stage procedure preserves the downstream representation interface without requiring task labels, downstream gradients, or modifications to the frozen model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.