acceptodds
Under review as a conference paper at ICLR 2027

SteerQuant: Steering Quantization Error with Action-Guided Scaling in World–Action Models

Abstract

World–action models (WAMs) jointly generate future world states and actions through iterative denoising, using shared weights to process heterogeneous semantic streams of video, proprioceptive, and action tokens. Quantization reduces inference cost, but comparable numerical errors in different streams can have markedly different effects on final actions, making numerical accuracy alone insufficient for reliable control. We introduce **SteerQuant**, a 4-bit quantization framework for WAMs that steers errors toward computations with less influence on final actions. It maps how each stream's quantization errors affect final actions and uses this map to guide shared channel scaling. Activation scaling is further calibrated for each stream and denoising step to accommodate changes in activation ranges and action impact. This adapts quantization to different stream requirements without duplicating weights or increasing bit-widths for selected streams. To reduce the extra kernel launches and memory traffic introduced by scaling, we develop **Rudder**, a 4-bit inference engine for WAMs that fuses scaling and output compensation into low-bit kernels. Under W4A8 and W4A4, SteerQuant maintains mean LIBERO success within 0.8 percentage points of full precision, while delivering up to denoising speedup over BF16 across three WAMs with reduced peak GPU memory usage. On a real dual-arm robot, W4A8 deployment achieves a end-to-end inference speedup while maintaining average task success relative to BF16.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world-action models, post-training quantization, low-bit inference, action impact

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_skbHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SteerQuant, a post-training quantization framework for World-Action Models (WAMs). The main observation is that quantization errors at different layers, denoising steps, and heterogeneous token streams have very different impacts on robot action outputs and closed-loop task succes, while conventional PTQ objectives based on local reconstruction loss fail to capture. SteerQuant builds an Action-Impact Stream Map via VJP and Rademacher projection, and adopts a two-stage calibration scheme consisting of shared-weight error routing and stream-specific activation modulation. A fused Rudder kernel is introduced to avoid extra kernel overhead. Experiments cover multiple WAMs on simulated robotic benchmarks and real-arm hardware, showing strong performance under W4A4/W4A8 quantization with notable inference acceleration.

Critical strengths
  1. The empirical observation is meaningful: local reconstruction error is a poor objective for WAM quantization, and the harm caused by quantization error depends jointly on layer, denoising step, and stream.

  2. The design respects WAM's hard constraint of shared transformer weights across heterogeneous streams, which is well aligned with the problem setting.

  3. The system-level optimization, particularly the fused Rudder kernel, helps preserve the speedup that quantization is supposed to deliver.

  4. The evaluation is fairly comprehensive, covering both simulated closed-loop tasks and real robot deployment, with ablation studies for the core modules.

Critical weaknesses
  1. A simple heuristic baseline is missing: quantizing only the dominant video stream while keeping proprioceptive/action streams in full precision. The authors should report its success rate, latency, and memory usage, and justify why the full pipeline is necessary if this heuristic performs similarly.

  2. QuantWAMs, a dedicated quantization method for WAMs, is not compared in either the main paper or the appendix. Even if it originally supports mixed-bit allocation, it should be adapted under a fixed W4 constraint for a fair comparison, with an analysis of the performance gap.

  3. It is unclear whether SteerQuant applies to WAM variants that predict future frames without action outputs. If so, the authors should describe how to modify the Action-Impact Stream Map and what additional calibration compute/memory overhead this entails. If not, this limitation should be stated clearly and the paper's claims revised accordingly.

  4. The paper only reports end-to-end wall-clock speedup. The authors should separately quantify the gains from bandwidth reduction and arithmetic reduction, and also quantify the extra memory/compute overhead introduced by the Rudder kernel, along with its impact on the theoretical speedup.

Questions for authors
  1. In typical WAM architectures, video tokens dominate the overall token volume. A natural heuristic baseline is to quantize only the video stream while keeping proprioceptive and action streams in full precision. Could the authors evaluate this baseline? Please report its task success rate, GPU latency, and memory footprint, and justify whether the full SteerQuant pipeline is necessary if this simple heuristic achieves comparable performance.

  2. Is SteerQuant applicable to WAM variants that only predict future observation frames without outputting action sequences? If yes, how should the Action-Impact Stream Map be modified, and what is the induced overhead in terms of calibration computation and GPU memory?

  3. The paper only reports end-to-end wall-clock speedup. Could the authors separately quantify the speedup gains contributed by memory bandwidth reduction and arithmetic computation reduction? Further, please quantify the extra memory and computation overhead introduced by the Rudder fused kernel and analyze how it erodes the theoretical quantization speedup.

6 comments
AuthorsOpenReview ↗

Response to Reviewer skbH (Part 1)

Thank you for your suggestions on baselines, applicability, and execution costs. We have added baseline comparisons, clarified our scope, and profiled Rudder’s overhead, as detailed below.


Q1&W1: Video-Only Quantization Baseline

Video-only baseline. Following your suggestion, we evaluated a video-only activation-quantization baseline on Cosmos-Policy / LIBERO. The shared target Linear weights remain packed INT4, while video activations use A8 or A4 and non-video activations, including proprioceptive and action streams, remain BF16. This tests selective activation quantization while retaining one shared low-bit weight matrix per layer.

Task performance and efficiency. At W4A8, the video-only baseline achieves 97.80 ± 0.53% SR, close to the FP result, but its DiT latency is 316.597 ms, more than twice SteerQuant’s 133.455 ms, while SteerQuant achieves 98.57 ± 0.25% SR. At W4A4, the distinction is larger: video-only quantization only achieves 12.07 ± 0.60% SR, whereas SteerQuant retains 98.30 ± 0.26%, with latency reduced from 284.523 to 117.322 ms. The video-only baseline combines low-bit video processing with BF16 non-video computation, introducing conversion and separate execution overhead that can outweigh the savings from quantization.

R3-Table 1: Task success and inference efficiency of video-only quantization and SteerQuant on Cosmos-Policy / LIBERO.

PrecisionMethodSR (%) ↑Latency (ms) ↓Speed up ↑Peak memory (GB) ↓Storage (GiB) ↓
FPOriginal98.52 ± 0.41261.4601.000×4.2413.644
W4A8*Video-only97.80 ± 0.53316.5970.826×1.6391.186
W4A8SteerQuant98.57 ± 0.25133.4551.959×1.6421.191
W4A4*Video-only12.07 ± 0.60284.5230.919×1.6391.186
W4A4SteerQuant98.30 ± 0.26117.3222.229×1.6451.194

*For the video-only baseline, A8/A4 applies only to video activations; non-video activations remain BF16. All target Linear weights are shared and stored in INT4.

Why action-guided quantization matters. These results show that retaining BF16 non-video activations alone is insufficient to preserve task success at W4A4. Video-stream errors can still propagate to predicted actions through the shared model, and token volume alone does not capture their impact across layers and denoising steps. SteerQuant uses this action impact to guide channel-level error balancing and stream-specific activation resolution, then executes the calibrated transformations through fused low-bit kernels. The comparison supports the practical value of this design: high task success and lower latency at a similar memory footprint.


W2: Comparison with QuantWAMs under Fixed-W4 Weights

Fixed-W4 adaptation. Following your suggestion, we adapted QuantWAMs [1] to Cosmos-Policy / LIBERO by fixing all 280 target Linear layers to packed INT4 weights. We retained its rotation and smoothing, shared-mask calibration, joint video–action saliency, GPTQ compensation, and rollout-based denoising-step protection, using the unified 320-window calibration protocol.

Retained activation protections. The adaptation preserves A8 activations in 56/280 layers, A8 protection at the fifth of five denoising steps, and approximately 2% BF16 exception channels. The exception branch also uses W4 weights. These retained paths yield a nominal average activation precision of 5.65 bits, with equal weighting over target layers and denoising steps, compared with 4 bits for SteerQuant’s target Linear activations. Thus, the comparison fixes weight precision while preserving QuantWAMs’ activation protections.

AuthorsOpenReview ↗

Response to Reviewer skbH (Part 2)

Task performance and efficiency. SteerQuant achieves 98.30 ± 0.26% SR, compared with 95.60 ± 0.56% for QuantWAMs, while reducing DiT latency by 30.71% and peak memory by 40.16%. These results show that the success-rate improvement comes alongside lower inference cost.

R3-Table 2: Task success and inference efficiency of the fixed-W4 QuantWAMs adaptation and SteerQuant on Cosmos-Policy / LIBERO.

PrecisionMethodSR (%) ↑Latency (ms) ↓Speed up ↑Peak memory (GB) ↓Storage (GiB) ↓
FPOriginal98.52 ± 0.41261.4601.000×4.2413.644
W4A4*QuantWAMs-FixedW495.60 ± 0.56169.3201.544×2.7491.400
W4A4SteerQuant98.30 ± 0.26117.3222.229×1.6451.194

*W4A4 indicates the base precision. QuantWAMs-FixedW4 retains A8 layer and denoising-step protection and approximately 2% BF16 exception activations, yielding a nominal layer–step-averaged activation precision of 5.65 bits. All target weights, including those in the exception branch, remain INT4.

Analysis of the performance gap. At fixed W4, QuantWAMs has less flexibility to allocate weight precision, while SteerQuant redistributes errors through shared channel scaling and stream-specific activation thresholds. These transformations are fused into low-bit kernels, providing a hardware-friendly execution path that improves the success–efficiency trade-off in this setting.

[1] Jiacheng Zhou et al. QuantWAMs: Calibrating at the Right Granularity for World Action Models. arXiv preprint arXiv:2607.28405, 2026.


Q2&W3: Applicability to Models without Action Outputs

Applicability. As presented in this paper, SteerQuant does not directly apply to WAM variants that only predict future frames without action outputs. Our Action-Impact Stream Map measures how quantization errors affect predicted action sequences (Eq. (5)), so it requires action outputs.

Why we focus on joint prediction. Joint prediction is a mainstream WAM approach [1], used by Cosmos Policy [2] and DreamZero [3]. For example, Cosmos Policy represents actions, future states, and values as latent frames within a pretrained video diffusion model [2]. This lets the model use its learned video priors to generate actions, while predicted states and values support planning. The shared computation also creates a quantization challenge: similar errors in different streams can have different effects on the final action sequence (Section 3.2). SteerQuant addresses this by using action impact to guide quantization while retaining shared weights and fixed bit-widths.

Scope and limitation. Extending SteerQuant to variants without action outputs requires a different sensitivity target; we have not measured the calibration computation or GPU memory overhead of this extension. We will state this limitation clearly and revise the paper's claims accordingly.

[1] Moritz Reuss. Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models. NVIDIA Technical Blog, June 2026.

[2] Moo Jin Kim et al. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. ICLR, 2026.

[3] Seonghyeon Ye et al. World Action Models are Zero-shot Policies. arXiv preprint arXiv:2602.15922, 2026.

AuthorsOpenReview ↗

Response to Reviewer skbH (Part 3)

Q3&W4: Inference Speedup Breakdown and Rudder Overhead

1. Bandwidth and compute gains. We combine Nsight Systems profiling with an empirical roofline model on matched five-step LIBERO inputs. Across the 280 quantized linear layers, the GEMM workload remains 29.60 trillion operations at all three precisions. The gains come from reduced data movement and higher low-bit arithmetic throughput, with the operation count unchanged. R3-Table 3 separates modeled bandwidth- and compute-limited times from measured GEMM times. We will include the profiling protocol and analysis in the revised appendix.

R3-Table 3: Memory and compute analysis of the quantized GEMM workload.

GEMM analysisBF16W4A8W4A4
Operand payload, GB (model)40.7521.8318.97
Bandwidth-limited time, ms (model)44.1523.6520.55
Compute-limited time, ms (model)183.1253.3927.89
GEMM time, ms (nsys)199.2967.8038.69
Roofline-estimated GEMM-stage speedup1.00×3.43×6.57×
Measured GEMM-stage speedup1.00×2.94×5.15×

The model assumes one read of each matrix operand and one BF16 output write, using measured streaming bandwidth and large-matrix throughput as references. It gives 1.87×/2.15× memory-side potential and 3.43×/6.57× compute-side potential for W4A8/W4A4. These describe separate limiting cases: using aggregate workload totals, the roofline estimate is set by the larger of data-transfer time and compute time, t≈max⁡(D/B,F/P)t \approx \max(D/B, F/P). The two potentials therefore cannot be added as independent speedup contributions. The measured GEMM-stage speedups are 2.94× and 5.15×, respectively.

2. Rudder's additional memory and computation. R3-Table 4 reports the allocated reshaping state, additional temporary allocation, and channel-scaling operation count. The allocation measurements describe storage requirements; the operation count covers channel scaling only.

R3-Table 4: Additional memory and arithmetic costs of Rudder.

MetricResult
Allocated reshaping state, including retained copies5.80 MiB, 0.69% of packed weights
Additional per-linear temporary allocation peak0 across all four observed shapes
Added channel-scaling multiplications0.0193% of GEMM operation count, excluding per-row work

3. Theoretical and measured speedup. For W4A8/W4A4, the roofline model estimates 3.43×/6.57× GEMM-stage speedups, compared with 2.94×/5.15× measured speedups. The measured speedups reach approximately 86%/78% of the model estimates. This overall gap includes hardware utilization and execution effects and cannot be attributed entirely to Rudder.

Reviewer_1WVxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to quantatize pretrained world-action-models. The method first measures where of the model, (i.e., the ), affects the predicted action most and then use this map to weigh the quantization. Also, it follows previous work to introduce the scaling matrix to redistribute weight–activation quantization errors across channels. Lastly, it further introduces one more weight to control activation quantization. And a threshold hyperparameter is used to decide the quantization range. In short, the method includes three different weights to quantities the model adaptively, according to their effect on the action prediction error.

Critical strengths
  1. It is important to speed up the inference of world action models. And the proposed method aims to quantize the model adaptively based on their effect on action prediction. The motivation and method is sensible.

  2. The experiments are across three different WAM, and two benchmarks. It also includes real robot experiments.

  3. The writing is generally easy to follow.

Critical weaknesses
  1. The method is complicated as it needs to design three weights to adaptively quantities the model.

  2. Also, the optimization process seems challenging. It needs to accumulate a lot of data including different denoising steps, different input tokens to optimize the weights. The optimization might be affected a lot by the data collection.

  3. The results in both Table 2 and 4 show that the speedup is not significant although with huge efforts on optimizing the quantization.

Questions for authors

How is the optimization in Equation (7) and (9) implemented?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 1WVx (Part 1)

Thank you for recognizing the motivation of our method and the breadth of our evaluation.

To address your concern, we clarify the roles of the calibration components and their optimization, present evidence of sensitivity-map stability across calibration subsets, and provide additional RoboCasa results and profiling evidence to substantiate the practical efficiency gains.


W1: Component Contributions and Calibration Cost

Roles and rationale. All three components are needed to attain the full method's action fidelity in our ablations (Table 3). They address differing action sensitivities across streams and denoising steps under shared model weights. The Action-Impact Stream Map identifies regions where quantization errors most affect actions. Routing uses these priorities to optimize shared channel scales, balancing weight and activation errors while retaining a single quantized weight matrix per layer. Since shared scales cannot fully capture step- and stream-dependent variations, Modulation adjusts stream gains and clipping thresholds at each step without changing the quantized weights. Map weights are estimated once from calibration data, followed by sequential optimization of Routing and Modulation parameters.

Ablation evidence. To assess whether this additional complexity is justified, we extend the ablations in Table 3 with closed-loop success rates (SR), using Cosmos-Policy on 40 tasks with 3 seeds and 5 rollouts per task per seed (600 rollouts in total).

R2-Table 1: Ablation of the three components with action RMSE and closed-loop success rate. ΔRMSE = RMSE(SteerQuant) − RMSE(variant).

VariantRoutingModulationWeightingRMSE ↓95% CI of ΔRMSESR (%) ↑
Base××Uniform0.1581[−0.0430, −0.0357]2.17
Routing only✓×Map-derived0.1224[−0.0044, −0.0026]78.16
Modulation only×✓Map-derived0.1529[−0.0368, −0.0312]6.50
Both, uniform✓✓Uniform0.1416[−0.0262, −0.0052]17.83
SteerQuant✓✓Map-derived0.1189—97.33

The full method achieves the lowest RMSE (0.1189) and highest SR (97.33%). Although Routing-only approaches the full method in RMSE (0.1224), its SR is 19.17 percentage points lower, demonstrating the practical benefit of Modulation. Replacing map-derived weights with uniform weights further reduces SR to 17.83%, confirming the importance of action-guided weighting. These results justify the complementary components in terms of closed-loop task success, beyond offline action fidelity.

Calibration and inference costs. All calibration is performed offline. Map estimation uses 16 projections per observation, reusing projected reverse passes across regions without constructing dense Jacobians (Section 4.1, Appendix B.2). At inference, Rudder fuses scaling into activation quantization and output compensation into the GEMM epilogue, requiring no online optimization or separate scaling kernels (Section 4.4). For Cosmos-Policy W4A8, using 320 LIBERO observations on a single NVIDIA A100-SXM4-80GB, complete calibration takes 4.967 hours with 35.56 GiB peak allocated GPU memory, including the resident model.

R2-Table 2: Offline calibration time and memory requirements.

StageTime (min)Peak GPU memory (GiB)
Activation statistics6.215.06
Map estimation218.0535.56
Activation caching22.715.06
Routing17.234.64
Modulation24.134.29
Loading, saving and other overhead9.70—
Complete calibration298.0335.56

Times include loading, data collection, optimization and saving; evaluation is excluded. Routing and Modulation times are accumulated across all 280 layers.

R2-Table 3: Calibration time under matched experimental conditions.

MethodCalibration time (h) ↓
SmoothQuant0.617
QuaRot5.317
Atom3.750
SVDQuant7.340
PTQ4DiT15.140
Q-DiT14.330
SteerQuant4.967
AuthorsOpenReview ↗

Response to Reviewer 1WVx (Part 2)

SteerQuant calibrates faster than QuaRot, SVDQuant, PTQ4DiT, and Q-DiT. Although map estimation accounts for 73.2% of calibration time, Routing and Modulation together take only 41.36 minutes. Thus, the additional action-guided optimization is confined to offline calibration, while inference retains a single quantized weight matrix per layer and uses fused scaling operations.


W2: Calibration Data Requirements and Robustness

Calibration data and optimization. The same calibration observations cover different denoising steps and semantic streams through model execution, so separate datasets are not needed for individual regions. As reported in Appendix D.2 of the original submission, we use 320 and 100 LIBERO training observations for Cosmos-Policy and FastWAMJoint, respectively, and 512 DROID training observations for Cosmos-Edge. Within each model, all methods share the same calibration set, separate from evaluation data. Using cached activations, we optimize channel scales first, followed by activation gains and thresholds; Q1 below details this sequential procedure.

Stability across calibration samples. Appendix B.4 of the original submission evaluates Cosmos-Policy maps estimated from two disjoint halves of the 320 observations, with 160 observations in each half. Across 200 random partitions, the maps have a median Spearman correlation of 0.967 over all 10,360 regions and a median top-10% overlap of 97.9%. Thus, different calibration subsets identify largely the same high-impact regions within the evaluated data pool.

Convergence with calibration-set size. Table 7 of the original submission compares maps from 200 subsets at each sample size with the 320-observation reference map. With 40 observations, the median Spearman correlation is 0.960 and the median top-10% overlap is 98.0%; with 160 observations, these reach 0.993 and 98.9%. These results show that smaller subsets recover similar regional priorities, with agreement improving as the sample size increases.

Full recalibration across subsets. We rerun Map estimation, Routing and Modulation on 3 calibration subsets each of 40 and 160 observations. All models are evaluated on the same 900 observations with matched FP references for action RMSE and 60 closed-loop episodes across 20 tasks from LIBERO-Spatial and LIBERO-10. All recalibration experiments use W4A8 quantization. Each task uses three matched initial states, with one fixed policy seed across all models.

R2-Table 4: Sensitivity of action fidelity and task success to calibration-set size and subset selection. Mean ± sample SD is computed across three subsets; the 320-observation row is the original calibrated reference.

Calibration observationsStandardized action RMSE ↓Closed-loop SR (%) ↑
400.12493 ± 0.0052494.44 ± 0.96
1600.12197 ± 0.0010397.22 ± 0.96
320 (reference)0.1231498.33

Performance varies modestly across the tested subsets. Increasing calibration coverage from 40 to 160 observations lowers mean RMSE and improves mean SR from 94.44% to 97.22%, approaching the 98.33% reference SR.

AuthorsOpenReview ↗

Response to Reviewer 1WVx (Part 3)

Q1: Implementation of the Two-Stage Optimization

For each linear layer, we optimize Eq. (7) first and Eq. (9) second. Both stages use Adam in log space to keep parameters positive, with a straight-through estimator (STE) for gradients through rounding.

Step 1: Prepare calibration data. Cache layer inputs across steps and streams and compute the corresponding reference outputs during optimization. Compute the map-derived importance weights once and keep them fixed throughout calibration (Section 4.1 and Appendix C.1).

Step 2: Optimize channel scales with Eq. (7). Update only DℓD_\ell, shared across steps and streams, while keeping the original model weights fixed. Recompute QW(DℓWℓ)Q_W(D_\ell W_\ell) as the scales change and minimize the map-weighted reconstruction error against the full-precision outputs. Perform 20 Adam updates per layer at learning rate 0.02, accumulating gradients across denoising steps and streams before each update. Fix the selected scales Dℓ∗D_\ell^* and quantized weights W^ℓ=QW(Dℓ∗Wℓ)\widehat W_\ell=Q_W(D_\ell^*W_\ell).

Step 3: Optimize activation parameters with Eq. (9). Keep Dℓ∗D_\ell^\ast and W^ℓ\widehat W_\ell fixed. At each denoising step, jointly optimize the stream gains γ\gamma and one shared clipping threshold cc, using 30 Adam updates per step, with learning rates 0.02 for the gain parameters and 0.01 for the log threshold. The target X~W^ℓ\widetilde X\widehat W_\ell isolates activation-quantization error. To enforce the gain constraints, center the raw log gains using token fractions ps=ns/∑jnjp_s=n_s/\sum_j n_j, then apply a common shrink factor before exponentiation. This preserves ∑snslog⁡γs=0\sum_s n_s\log\gamma_s=0 while keeping γs∈[0.25,4]\gamma_s\in[0.25,4]. After each update, clamp log⁡c\log c to [log⁡(0.01c0),log⁡(4c0)][\log(0.01c_0),\log(4c_0)], where c0c_0 is the initial clipping threshold.

Step 4: Select parameters by hard-quantization loss. During each stage, evaluate the hard-quantization loss on all cached rows every five updates and at the final update. Select the lowest-loss candidate, including the initialization, before proceeding to the next stage or inference.

Reviewer_dRvoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a novel method for 4-bit quantization of World Action Models which distributes quantization error to minimize downstream errors in action generation. The framework is motivated by the observed phenomenon that the impact of quantization on downstream action generation is highly dependent on layer, modality stream, and denoising timestep, which is demonstrated via action RMSE evaluations on Cosmos-Policy under local quantization. Leveraging a linearized approximation of this action-level error propagation, the proposed method modulates the weights and activations to minimize this downstream error while fixing the quantization. This is efficiently implemented via a custom inference engine which folds the added computations into fused kernels. Across WAM architectures and simulation environments, as well as deployed on real-world manipulation tasks, the proposed method preserves near full precision success while meaningfully reducing memory footprint and inference time.

Critical strengths

• The presented method is novel and conceptually appealing. Leveraging an action-centric framing of quantization error is intuitive for the robotics setting. Determining where and how quantization affects downstream action generation to inform how to distribute errors is elegant and well-motivated by the observational study.
• Evaluation across simulation and hardware demonstrates practical improvements in inference time and memory over the base model, and motivates further development in this “adaptive” approach to quantization in multimodal model architectures.

Critical weaknesses

• Results do not convincingly motivate the proposed method over baselines. For the LIBERO evaluations (Table 1), all methods including the base model achieve nearly 100 percent SR, with all baselines within ~1 percent of each other, while the proposed method fails to consistently outperform baselines. On RoboLab, SteerQuant achieves higher SR than baseline, but improvements remain within 1.3%. As there are no confidence intervals or variances reported, it is unclear whether either of these results are statistically meaningful, and thus whether there is a substantive improvement over baselines. Hardware evaluations, while appreciated, do not alleviate this concern as there are no reported baselines. Furthermore, gains in inference time over baselines are minor enough such that they may also be subject to noise, and it is unclear whether the improvements come from the method itself, or simply a more optimized implementation of the kernel. Including statistics on results, as well as a clarification what motivates using the proposed method (which introduces added machinery and complexity) over baselines, would strengthen the paper.

Questions for authors
  1. Where does the inference time improvement come from over other quantization baselines? Since there are added operations in SteerQuant through the error reshaping, it is not clear why this would be faster than a direct quantization method.
  2. Why does W4A8 underperform W4A4 on RoboLab for SteerQuant? Does this indicate substantial variance in the SR results?
  3. How do the RMSE values in Table 3 translate to success rate? Is Routing-only sufficient for high success rate? If so, what is the value of the modulation step?
9 comments
AuthorsOpenReview ↗

Response to Reviewer dRvo (Part 1)

We really appreciate your detailed and constructive feedback, and your recognition of the novelty and conceptual appeal of our action-guided approach to quantization.

Following your comments, we have added statistical reporting across evaluation seeds, experiments on a new benchmark, and controlled latency measurements. Below, we explain the sources of the inference gains, examine the W4A8/W4A4 success-rate ordering, and present evidence of the method’s practical benefits.


W1: Statistical Reliability and Benefits over Baselines

Statistical evidence. We added sample SDs across three evaluation seeds to the original success-rate comparisons (R1-Table 1). SteerQuant's SD ranges from 0.22 to 0.48 percentage points. We also added repeated W4A8 latency measurements against SmoothQuant on matched LIBERO inputs (10 processes × 30 randomized, interleaved repetitions). SteerQuant is faster in every process, with a mean saving of 1.838 ms per five-step sequence (95% CI [1.794, 1.882] ms; 1.20%).

R1-Table 1: Closed-loop success rates (%), reported as mean ± sample SD across 3 evaluation seeds. LIBERO results are averaged across the four suites.

PrecisionMethodCosmos-Policy / LIBEROFastWAMJoint / LIBEROCosmos-Edge / RoboLab
FPOriginal98.52 ± 0.4198.9 ± 0.3921.83 ± 0.12
W4A8SmoothQuant97.93 ± 0.3297.2 ± 0.2613.67 ± 0.25
PTQ4DiT98.07 ± 0.4297.6 ± 0.2513.67 ± 0.21
Q-DiT97.66 ± 0.0997.6 ± 0.1612.17 ± 0.36
SteerQuant98.57 ± 0.2598.1 ± 0.2214.67 ± 0.30
W4A4QuaRot97.93 ± 0.2398.7 ± 0.2112.25 ± 0.31
Atom97.20 ± 0.2098.2 ± 0.2713.50 ± 0.46
SVDQuant98.13 ± 0.2398.2 ± 0.3016.00 ± 0.27
SteerQuant98.30 ± 0.2698.4 ± 0.2417.33 ± 0.48

Larger improvements on RoboCasa. We added Cosmos-Policy experiments on 24 RoboCasa tasks, with 50 rollouts per task per seed across three seeds (3,600 rollouts per method) to evaluate improvements beyond the near-saturated LIBERO results. SteerQuant exceeds the baseline with the highest SR by 4.59 percentage points at W4A8 (63.20% vs. 58.61%, PTQ4DiT) and 5.55 percentage points at W4A4 (60.14% vs. 54.59%, QuaRot). It also achieves the lowest measured latency at each precision among methods with available latency results (R1-Table 2).

R1-Table 2: Cosmos-Policy results on RoboCasa. SR is reported as mean ± sample SD across 3 evaluation seeds. Latency measures DiT inference, and speedup is relative to BF16. Efficiency metrics for PTQ4DiT and Q-DiT are unavailable because their official implementations do not provide low-bit inference kernels for the evaluated settings.

PrecisionMethodSR (%) ↑Latency (ms) ↓Speed up ↑Peak memory (GB) ↓Storage (GiB) ↓
FPOriginal63.33 ± 0.72366.1001.000×4.7423.644
W4A8SmoothQuant54.30 ± 0.87242.5701.509×2.1091.189
PTQ4DiT58.61 ± 1.05N/AN/AN/AN/A
Q-DiT57.92 ± 0.84N/AN/AN/AN/A
SteerQuant63.20 ± 2.55239.8861.526×2.2331.189
W4A4QuaRot54.59 ± 2.60198.5221.844×2.1061.189
Atom53.05 ± 0.87312.3271.172×2.1841.258
SVDQuant50.83 ± 1.50231.2041.583×2.2831.320
SteerQuant60.14 ± 1.97188.8171.939×2.2361.191
AuthorsOpenReview ↗

Response to Reviewer dRvo (Part 2)

Sources of the inference-time improvement. All methods included in the latency comparison use optimized kernel implementations. In the controlled W4A8 comparison, SmoothQuant and SteerQuant share the CUTLASS backend. SteerQuant removes runtime threshold estimation through offline calibration, while Rudder fuses scaling and compensation into existing kernels. Our matched execution ablation reduces five-step LIBERO DiT latency from 205.87 to 140.05 ms, a 31.97% mean paired reduction, and removes 5,040 kernel executions with unchanged integer GEMM work. Q1 provides the implementation details and controlled comparisons.


Q1: Sources of Inference Speedup

The gains come from two sources: offline calibration and fused execution.

First, offline calibration reduces runtime work. SteerQuant calibrates channel scales, stream gains and clipping thresholds offline. Compared with the evaluated SmoothQuant [1] implementation, this removes dynamic per-token threshold estimation, with profiling confirming lower activation-quantization cost. Its execution path also avoids SVDQuant’s additional low-rank projections [2] and Atom’s INT8 keeper-channel computation [3].

Second, kernel fusion reduces execution overhead. Rudder folds channel scaling into activation quantization and output compensation into the GEMM epilogue. It further integrates LayerNorm/AdaLN and gate/residual operations, reducing standalone kernel launches and intermediate memory traffic.

Comparison with optimized QuaRot. To examine the execution advantage, we evaluate two QuaRot [4] DiT adaptations under the same timing protocol: one retaining applicable official optimizations, and another with additional Hadamard–quantization fusion and output dequantization fused into the GEMM epilogue. SteerQuant achieves lower latency than both adaptations, including the additionally optimized baseline, as shown below.

R1-Table 3: Cosmos-Policy W4A4 on RoboCasa, RTX 4090, batch size 1. Latency covers five DiT denoising steps.

MethodLatency (ms) ↓Speedup over QuaRot, first row ↑
QuaRot (adaptation retaining applicable official optimizations)287.4371.000×
QuaRot (our additionally optimized baseline)188.8771.522×
SteerQuant171.4191.677×

All three methods use identical captured inputs, CUDA Graphs and CUDA-event timing, with deterministic allocation filling disabled. Each implementation is measured in 3 independent processes × 30 timed repetitions, with cyclically balanced process order. We report the median of the three process medians. SteerQuant reduces latency by 40.36% and 9.24%, respectively. Rudder extends fusion to LayerNorm/AdaLN and gate/residual operations, further reducing standalone execution overhead. The 171.419 ms SteerQuant latency in R1-Table 3 comes from this controlled comparison protocol, whereas the 188.817 ms result in R1-Table 2 comes from the separate RoboCasa main-benchmark protocol.

Matched ablation of Rudder's execution optimizations. We evaluate three execution configurations of Cosmos-Policy W4A8 on an NVIDIA RTX 4090, with batch size 1, using matched BF16-captured LIBERO inputs for five denoising steps. All configurations use identical calibrated parameters and packed weights, with the same GEMM workload. A materializes channel scaling; B fuses it into activation quantization; C additionally enables Rudder's block-level normalization/AdaLN and gate/residual fusion. Stream gains and output compensation remain fused in all three configurations. We measure five-step DiT replay latency using CUDA Graphs and CUDA events after warm-up, over 10 independent processes × 30 randomized, interleaved repetitions, reporting the median of process medians. Timing excludes model loading, compilation, graph construction and policy stages outside the DiT. Kernel counts are obtained from separate profiling of a representative input.

AuthorsOpenReview ↗

Response to Reviewer dRvo (Part 3)

R1-Table 4: Matched ablation of Rudder's execution optimizations. Kernel counts cover one five-step DiT replay.

ConfigurationChannel-scaling fusionBlock-level fusionDiT latency (ms) ↓Kernel executions ↓
A: Materialized channel scaling××205.8712,045
B: Channel-scaling fusion✓×151.139,245
C: Rudder✓✓140.057,005

Full Rudder reduces latency by 31.97% relative to A, with block-level fusion providing a further 7.32% reduction relative to B (mean paired reductions). It eliminates 5,040 kernel executions while retaining the same 1,400 integer GEMM calls. C is faster than both A and B in every process. These latencies come from this matched ablation protocol and are reported separately from the 133.455 ms main-benchmark result in Table 2 of the original submission.

[1] Guangxuan Xiao et al. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML, 2023.

[2] Muyang Li et al. SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models. ICLR, 2025.

[3] Yilong Zhao et al. Atom: Low-Bit Quantization for Efficient and Accurate LLM Serving. MLSys, 2024.

[4] Saleh Ashkboos et al. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. NeurIPS, 2024.


Q2: W4A8/W4A4 Success Rates and Evaluation Variability

Variation across evaluation seeds. As shown in the added R1-Table 1, across three evaluation seeds, SteerQuant achieves 14.67 ± 0.30% with W4A8 and 17.33 ± 0.48% with W4A4 on RoboLab. Both configurations exhibit limited cross-seed variation, and the 2.66-percentage-point gap is considerably larger than either observed SD, suggesting that large evaluation fluctuations alone are unlikely to explain the ordering.

Activation precision and closed-loop success. Action fidelity matters for task success, but activation bit-width alone does not determine it. Higher bit-width provides finer numerical resolution, while closed-loop SR depends on how quantization errors affect final actions, particularly at critical steps, and subsequent trajectories [2]. This explains why finer resolution need not translate into higher SR, consistent with the non-monotonic results reported in VLAQuantBench [1]. SteerQuant therefore prioritizes errors by their impact on final actions, rather than by numerical precision alone.

[1] Xu et al. VLAQuantBench: Closed-Loop Evaluation of Post-Training Quantization for Vision-Language-Action Models. arXiv preprint arXiv:2609.25376, 2026.

[2] Ross, Gordon, and Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS, 2011.


Q3: Action RMSE, Task Success, and Modulation Benefits

RMSE and success rate. We added closed-loop SR measurements for the ablation variants under the same experimental settings as Table 3 (R1-Table 5 below), using Cosmos-Policy on 40 tasks with 3 seeds and 5 rollouts per task per seed (600 rollouts in total). Given the rebuttal timeline, we retain this ablation protocol rather than the main evaluation protocol of 3,000 rollouts. Across these variants, lower action RMSE accompanies higher SR, but the changes are not proportional. RMSE averages action deviations, while SR measures final task outcomes. Prior work also reports a mismatch between offline validation loss and task performance [1], and action deviations affect subsequent observations and trajectories [2].

R1-Table 5: Action RMSE and closed-loop success rate for the ablation variants. ΔRMSE denotes RMSE(SteerQuant) − RMSE(variant).

VariantRoutingModulationWeightingRMSE ↓95% CI of ΔRMSESR (%) ↑
Base××Uniform0.1581[−0.0430, −0.0357]2.17
Routing only✓×Map-derived0.1224[−0.0044, −0.0026]78.16
Modulation only×✓Map-derived0.1529[−0.0368, −0.0312]6.50
Both, uniform✓✓Uniform0.1416[−0.0262, −0.0052]17.83
SteerQuant✓✓Map-derived0.1189–97.33

Routing-only success rate. Routing-only achieves 78.16% SR, compared with 97.33% for full SteerQuant. It therefore falls short of the full method's task success.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of feedback and improvements

Dear Area Chair,

Thank you for your time and consideration. We summarize our contributions, the reviewers’ assessments, and the progress made during the discussion to support your final evaluation.

Core Contributions and Recognized Strengths

  • An action-centric view of quantization error. All three reviewers recognized the rationale for guiding quantization by its effect on the robot’s predicted actions. dRvo called the approach “novel and conceptually appealing.” skbH highlighted how local errors of similar magnitude affect actions differently across layers, streams, and denoising steps. 1WVx emphasized the importance of faster WAM inference and found the motivation and method sensible.

  • Practical low-bit inference with Rudder. Rudder folds SteerQuant’s calibrated scaling and output compensation into low-bit kernels. skbH singled out the fused kernel for helping preserve the speedup from quantization. dRvo highlighted the practical reductions in inference time and memory relative to the full-precision model.

  • Evaluation across models, simulation, and real hardware. All three reviewers highlighted the scope of the evaluation. 1WVx noted that the original submission evaluated three WAMs across two simulation benchmarks and included real-robot experiments. skbH also valued the closed-loop evaluations and component ablations, while dRvo highlighted the evidence from testing in both simulation and on hardware.

On Cosmos-Policy, SteerQuant achieved 1.96× and 2.23× DiT denoising speedups at W4A8 and W4A4, respectively, while maintaining mean LIBERO success rates of 98.57% and 98.30%, close to the BF16 reference of 98.52%. On the real robot, W4A8 FastWAMJoint achieved a 1.35× end-to-end inference speedup with an average task score of 82.41%, compared with 75.93% for BF16.

Key Issues and Responses

  • Component contributions to task success (dRvo, 1WVx). We extended the component ablation to closed-loop LIBERO tasks. Adding Modulation to Routing improved task success, while replacing action-derived calibration weights with uniform weights reduced it. These results show how both design choices contribute to control performance.

  • Comparisons with quantization baselines (dRvo, skbH). The new RoboCasa evaluation showed higher mean success than the strongest tested baseline at both precisions. On LIBERO, W4A4 SteerQuant outperformed both baselines requested by skbH—video-only quantization and fixed-W4 QuantWAMs—in closed-loop task success and DiT denoising speed, demonstrating the practical benefit of action-guided quantization.

  • Sources of inference speedup (dRvo, skbH). We added a roofline analysis to quantify the benefits of low-bit arithmetic and reduced memory traffic in GEMMs. Controlled ablations showed how Rudder further reduces denoising latency by fusing SteerQuant’s offline-calibrated scaling into low-bit inference kernels, avoiding extra kernel launches and memory transfers.

We also reported offline calibration time and memory, tested sensitivity to calibration-set size and sample selection, and clarified the method’s scope for models with action outputs.

Discussion Progress

  • dRvo. After our initial response, dRvo asked for clearer attribution of the latency gains, narrower benchmark claims, and more context for the RoboLab results. We revised the manuscript on all three points. Following this exchange, dRvo raised their rating.

  • 1WVx. We responded to all questions raised in the review and provided closed-loop component ablations, measurements of calibration time and memory usage, and recalibration results across data subsets. We have not yet received a reply from the reviewer.

  • skbH. We responded to all questions raised in the review, provided the requested baseline comparisons and profiling results for Rudder, and clarified the method’s scope for models with action outputs. We have not yet received a reply from the reviewer.

We believe the additional experiments and analyses strengthen the contributions of SteerQuant and Rudder to low-bit inference for world action models. Thank you again for your time and care in evaluating our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.