- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SteerQuant, a post-training quantization framework for World-Action Models (WAMs). The main observation is that quantization errors at different layers, denoising steps, and heterogeneous token streams have very different impacts on robot action outputs and closed-loop task succes, while conventional PTQ objectives based on local reconstruction loss fail to capture. SteerQuant builds an Action-Impact Stream Map via VJP and Rademacher projection, and adopts a two-stage calibration scheme consisting of shared-weight error routing and stream-specific activation modulation. A fused Rudder kernel is introduced to avoid extra kernel overhead. Experiments cover multiple WAMs on simulated robotic benchmarks and real-arm hardware, showing strong performance under W4A4/W4A8 quantization with notable inference acceleration.
Critical strengths
-
The empirical observation is meaningful: local reconstruction error is a poor objective for WAM quantization, and the harm caused by quantization error depends jointly on layer, denoising step, and stream.
-
The design respects WAM's hard constraint of shared transformer weights across heterogeneous streams, which is well aligned with the problem setting.
-
The system-level optimization, particularly the fused Rudder kernel, helps preserve the speedup that quantization is supposed to deliver.
-
The evaluation is fairly comprehensive, covering both simulated closed-loop tasks and real robot deployment, with ablation studies for the core modules.
Critical weaknesses
-
A simple heuristic baseline is missing: quantizing only the dominant video stream while keeping proprioceptive/action streams in full precision. The authors should report its success rate, latency, and memory usage, and justify why the full pipeline is necessary if this heuristic performs similarly.
-
QuantWAMs, a dedicated quantization method for WAMs, is not compared in either the main paper or the appendix. Even if it originally supports mixed-bit allocation, it should be adapted under a fixed W4 constraint for a fair comparison, with an analysis of the performance gap.
-
It is unclear whether SteerQuant applies to WAM variants that predict future frames without action outputs. If so, the authors should describe how to modify the Action-Impact Stream Map and what additional calibration compute/memory overhead this entails. If not, this limitation should be stated clearly and the paper's claims revised accordingly.
-
The paper only reports end-to-end wall-clock speedup. The authors should separately quantify the gains from bandwidth reduction and arithmetic reduction, and also quantify the extra memory/compute overhead introduced by the Rudder kernel, along with its impact on the theoretical speedup.
Questions for authors
-
In typical WAM architectures, video tokens dominate the overall token volume. A natural heuristic baseline is to quantize only the video stream while keeping proprioceptive and action streams in full precision. Could the authors evaluate this baseline? Please report its task success rate, GPU latency, and memory footprint, and justify whether the full SteerQuant pipeline is necessary if this simple heuristic achieves comparable performance.
-
Is SteerQuant applicable to WAM variants that only predict future observation frames without outputting action sequences? If yes, how should the Action-Impact Stream Map be modified, and what is the induced overhead in terms of calibration computation and GPU memory?
-
The paper only reports end-to-end wall-clock speedup. Could the authors separately quantify the speedup gains contributed by memory bandwidth reduction and arithmetic computation reduction? Further, please quantify the extra memory and computation overhead introduced by the Rudder fused kernel and analyze how it erodes the theoretical quantization speedup.
6 comments
Response to Reviewer skbH (Part 2)
Task performance and efficiency. SteerQuant achieves 98.30 ± 0.26% SR, compared with 95.60 ± 0.56% for QuantWAMs, while reducing DiT latency by 30.71% and peak memory by 40.16%. These results show that the success-rate improvement comes alongside lower inference cost.
R3-Table 2: Task success and inference efficiency of the fixed-W4 QuantWAMs adaptation and SteerQuant on Cosmos-Policy / LIBERO.
| Precision | Method | SR (%) ↑ | Latency (ms) ↓ | Speed up ↑ | Peak memory (GB) ↓ | Storage (GiB) ↓ |
|---|---|---|---|---|---|---|
| FP | Original | 98.52 ± 0.41 | 261.460 | 1.000× | 4.241 | 3.644 |
| W4A4* | QuantWAMs-FixedW4 | 95.60 ± 0.56 | 169.320 | 1.544× | 2.749 | 1.400 |
| W4A4 | SteerQuant | 98.30 ± 0.26 | 117.322 | 2.229× | 1.645 | 1.194 |
*W4A4 indicates the base precision. QuantWAMs-FixedW4 retains A8 layer and denoising-step protection and approximately 2% BF16 exception activations, yielding a nominal layer–step-averaged activation precision of 5.65 bits. All target weights, including those in the exception branch, remain INT4.
Analysis of the performance gap. At fixed W4, QuantWAMs has less flexibility to allocate weight precision, while SteerQuant redistributes errors through shared channel scaling and stream-specific activation thresholds. These transformations are fused into low-bit kernels, providing a hardware-friendly execution path that improves the success–efficiency trade-off in this setting.
[1] Jiacheng Zhou et al. QuantWAMs: Calibrating at the Right Granularity for World Action Models. arXiv preprint arXiv:2607.28405, 2026.
Q2&W3: Applicability to Models without Action Outputs
Applicability. As presented in this paper, SteerQuant does not directly apply to WAM variants that only predict future frames without action outputs. Our Action-Impact Stream Map measures how quantization errors affect predicted action sequences (Eq. (5)), so it requires action outputs.
Why we focus on joint prediction. Joint prediction is a mainstream WAM approach [1], used by Cosmos Policy [2] and DreamZero [3]. For example, Cosmos Policy represents actions, future states, and values as latent frames within a pretrained video diffusion model [2]. This lets the model use its learned video priors to generate actions, while predicted states and values support planning. The shared computation also creates a quantization challenge: similar errors in different streams can have different effects on the final action sequence (Section 3.2). SteerQuant addresses this by using action impact to guide quantization while retaining shared weights and fixed bit-widths.
Scope and limitation. Extending SteerQuant to variants without action outputs requires a different sensitivity target; we have not measured the calibration computation or GPU memory overhead of this extension. We will state this limitation clearly and revise the paper's claims accordingly.
[1] Moritz Reuss. Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models. NVIDIA Technical Blog, June 2026.
[2] Moo Jin Kim et al. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. ICLR, 2026.
[3] Seonghyeon Ye et al. World Action Models are Zero-shot Policies. arXiv preprint arXiv:2602.15922, 2026.
Response to Reviewer skbH (Part 3)
Q3&W4: Inference Speedup Breakdown and Rudder Overhead
1. Bandwidth and compute gains. We combine Nsight Systems profiling with an empirical roofline model on matched five-step LIBERO inputs. Across the 280 quantized linear layers, the GEMM workload remains 29.60 trillion operations at all three precisions. The gains come from reduced data movement and higher low-bit arithmetic throughput, with the operation count unchanged. R3-Table 3 separates modeled bandwidth- and compute-limited times from measured GEMM times. We will include the profiling protocol and analysis in the revised appendix.
R3-Table 3: Memory and compute analysis of the quantized GEMM workload.
| GEMM analysis | BF16 | W4A8 | W4A4 |
|---|---|---|---|
| Operand payload, GB (model) | 40.75 | 21.83 | 18.97 |
| Bandwidth-limited time, ms (model) | 44.15 | 23.65 | 20.55 |
| Compute-limited time, ms (model) | 183.12 | 53.39 | 27.89 |
| GEMM time, ms (nsys) | 199.29 | 67.80 | 38.69 |
| Roofline-estimated GEMM-stage speedup | 1.00× | 3.43× | 6.57× |
| Measured GEMM-stage speedup | 1.00× | 2.94× | 5.15× |
The model assumes one read of each matrix operand and one BF16 output write, using measured streaming bandwidth and large-matrix throughput as references. It gives 1.87×/2.15× memory-side potential and 3.43×/6.57× compute-side potential for W4A8/W4A4. These describe separate limiting cases: using aggregate workload totals, the roofline estimate is set by the larger of data-transfer time and compute time, . The two potentials therefore cannot be added as independent speedup contributions. The measured GEMM-stage speedups are 2.94× and 5.15×, respectively.
2. Rudder's additional memory and computation. R3-Table 4 reports the allocated reshaping state, additional temporary allocation, and channel-scaling operation count. The allocation measurements describe storage requirements; the operation count covers channel scaling only.
R3-Table 4: Additional memory and arithmetic costs of Rudder.
| Metric | Result |
|---|---|
| Allocated reshaping state, including retained copies | 5.80 MiB, 0.69% of packed weights |
| Additional per-linear temporary allocation peak | 0 across all four observed shapes |
| Added channel-scaling multiplications | 0.0193% of GEMM operation count, excluding per-row work |
3. Theoretical and measured speedup. For W4A8/W4A4, the roofline model estimates 3.43×/6.57× GEMM-stage speedups, compared with 2.94×/5.15× measured speedups. The measured speedups reach approximately 86%/78% of the model estimates. This overall gap includes hardware utilization and execution effects and cannot be attributed entirely to Rudder.
Response to Reviewer skbH (Part 1)
Thank you for your suggestions on baselines, applicability, and execution costs. We have added baseline comparisons, clarified our scope, and profiled Rudder’s overhead, as detailed below.
Q1&W1: Video-Only Quantization Baseline
Video-only baseline. Following your suggestion, we evaluated a video-only activation-quantization baseline on Cosmos-Policy / LIBERO. The shared target Linear weights remain packed INT4, while video activations use A8 or A4 and non-video activations, including proprioceptive and action streams, remain BF16. This tests selective activation quantization while retaining one shared low-bit weight matrix per layer.
Task performance and efficiency. At W4A8, the video-only baseline achieves 97.80 ± 0.53% SR, close to the FP result, but its DiT latency is 316.597 ms, more than twice SteerQuant’s 133.455 ms, while SteerQuant achieves 98.57 ± 0.25% SR. At W4A4, the distinction is larger: video-only quantization only achieves 12.07 ± 0.60% SR, whereas SteerQuant retains 98.30 ± 0.26%, with latency reduced from 284.523 to 117.322 ms. The video-only baseline combines low-bit video processing with BF16 non-video computation, introducing conversion and separate execution overhead that can outweigh the savings from quantization.
*For the video-only baseline, A8/A4 applies only to video activations; non-video activations remain BF16. All target Linear weights are shared and stored in INT4.
Why action-guided quantization matters. These results show that retaining BF16 non-video activations alone is insufficient to preserve task success at W4A4. Video-stream errors can still propagate to predicted actions through the shared model, and token volume alone does not capture their impact across layers and denoising steps. SteerQuant uses this action impact to guide channel-level error balancing and stream-specific activation resolution, then executes the calibrated transformations through fused low-bit kernels. The comparison supports the practical value of this design: high task success and lower latency at a similar memory footprint.
W2: Comparison with QuantWAMs under Fixed-W4 Weights
Fixed-W4 adaptation. Following your suggestion, we adapted QuantWAMs [1] to Cosmos-Policy / LIBERO by fixing all 280 target Linear layers to packed INT4 weights. We retained its rotation and smoothing, shared-mask calibration, joint video–action saliency, GPTQ compensation, and rollout-based denoising-step protection, using the unified 320-window calibration protocol.
Retained activation protections. The adaptation preserves A8 activations in 56/280 layers, A8 protection at the fifth of five denoising steps, and approximately 2% BF16 exception channels. The exception branch also uses W4 weights. These retained paths yield a nominal average activation precision of 5.65 bits, with equal weighting over target layers and denoising steps, compared with 4 bits for SteerQuant’s target Linear activations. Thus, the comparison fixes weight precision while preserving QuantWAMs’ activation protections.