- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a training-free W4A8 PTQ framework with nested loop format that can be applied for VLA models, named as LoopQuant. LoopQuant adapted an activation calibration across multiple observations with denoising steps and coodbooks. LoopQuant was validated in LIBERO suites, with improvement over other methods on GR00T-N1.5, although a gap to FP16 remains. The authors also reported a 1.88x speedup with a GPU memory reduction to 59% on an RTX 5090 setting.
Critical strengths
- This paper's proposed method was evaluated in a closed-loop task environment, where numerical reconstruction error itself doesn't show whether a quantized policy preserves robot behavior. The authors tested their methods in two models environments and the results backed their hypothesis and system design.
- The calibration experiments are good. Especially in the Table 3, RMAEC outperforms other calibration settings with large gaps.
- The authors also showed the real-world implementation of GPU memory use case, where the ablations and detailed appendix experiments helped readers to understand the reproducibility and limits of this evaluation.
Critical weaknesses
- Although Tables 2 and 3 provide useful controlled experiments, Table 1 does not establish whether LoopQuant’s advantage over competing methods remains under matched calibration data and denoising-step coverage.
- The benefit of exact codebook optimization over a strong approximate solver is not established. The comparison with uniform grids demonstrates the value of fitted non-uniform levels, but does not isolate the value of finding the exact optimum.
- Although the authors provided numerous ablation studies and comparison experiments, still, the main evaluation supports a narrower scope than broad generalizability across VLAs or deployment settings. It includes two models, one simulation benchmark, and one evaluation seed per configuration. Hardware measurements are limited to a desktop GPU.
- Several findings would benefit from deeper interpretation. For example, single-frame calibration substantially reduces clipping without improving task success, and the codebook contributes differently across the two models.
- In case of the writing, the conclusion section mainly focuses the components and results. It would be good if the author can restate/introduce the problem, and higher conclusion (e.g. what these findings teach us about quantizing iterative robot policies)
Questions for authors
- How does LoopQuant compare with a BF16 model using equivalent dispatch optimization? Could you separate the latency effects of packed quantization and single-launch execution?
- Would the accuracy advantage remain if compatible baselines used the same 32 calibration observations and denoising-step coverage? I acknowledge the controls in Tables 2 and 3, but a matched comparison with competing methods would strengthen Table 1.
- Why does the codebook provide little additional benefit after RMAEC on π0.5 but a larger improvement on GR00T? Similarly, what explains the weak relationship between the clipping reduction from single-frame calibration and task success?
- Could you provide evidence from another architecture, such as SmolVLA, or a second benchmark? Either would help clarify how well the findings transfer beyond the current setting.
2 comments
Response to Reviewer phKY (2/2)
Q3 (first part). "Why does the codebook provide little additional benefit after RMAEC on π0.5 but a larger improvement on GR00T?"
Answer. The difference lies in how much headroom RMAEC leaves, rather than in the codebook itself. On π0.5, uniform W4 + RMAEC already matches BF16, so there is little left to recover; on GR00T-N1.5 a 6 pp gap remains, and the codebook recovers about half of it (Table R5). Consistent with this, the codebook's reconstruction gain is similar on both models (median per-projection reduction of 39.3% vs. 38.5%, Sec. 4.3), and on π0.5 it still beats uniform W4 + RMAEC on all three seeds (Table R5 in our response to Reviewer gt7p).
Table R5. Headroom after RMAEC.
| Model | Uniform W4 + RMAEC | + exact codebook | Unquantized | Headroom |
|---|---|---|---|---|
| π0.5 | 97.1% | 97.3% (+0.2 pp) | BF16 97.1% | 0 pp |
| GR00T-N1.5 | 81.0% | 84.3% (+3.3 pp) | FP16 87.0% | 6 pp |
W4 / Q3 (second part). "What explains the weak relationship between the clipping reduction from single-frame calibration and task success?"
Answer. Single-frame calibration removes most clipping on average, but not the clipping that matters. It covers the denoising steps of one request, while activations from other observations still exceed its thresholds, and success only rises once the thresholds also cover those observations, as in RMAEC (Table R6).
Table R6. Clipped mass (32 unseen episodes, 960 requests) and success (800 episodes) on π0.5.
| Calibration | Action-expert clipped mass | Success |
|---|---|---|
| First-forward baseline | 29.78% | 82.4% |
| Single-frame envelope | 6.98% | 82.5% |
| RMAEC | 2.14% | 97.1% |
Two further mechanisms plausibly add to this mismatch: errors concentrated in particular channels or moments, and closed-loop feedback, in which one poor action changes later observations.
W3 / Q4. "Could you provide evidence from another architecture, such as SmolVLA, or a second benchmark?"
Answer. Thank you for suggesting SmolVLA. We evaluated it on the four LIBERO suites against ZSQ, the strongest baseline on π0.5 in Table 1. LoopQuant reaches 50.25%, against 67.63% for FP16 and 22.13% for ZSQ, and is above ZSQ on every suite (Table R7). In addition, the held-out-task test in the appendix shows transfer to tasks unseen in calibration: 94.4% on LIBERO tasks 8–9 vs. 98.0% on tasks 0–7.
Table R7. SmolVLA success on LIBERO (W4A8 for ZSQ and LoopQuant).
| SmolVLA | Spatial | Object | Goal | 10 | Avg. |
|---|---|---|---|---|---|
| FP16 | 63.00% | 73.50% | 79.00% | 55.00% | 67.63% |
| ZSQ | 28.00% | 13.00% | 20.00% | 27.50% | 22.13% |
| LoopQuant | 52.50% | 62.50% | 37.50% | 48.50% | 50.25% |
The gap to FP16 is concentrated on LIBERO-Goal, while on the other suites it is 6.5–11.0 pp. LoopQuant applies to SmolVLA because it shares the dual-loop flow: a cached vision-language prefix and a flow-matching action expert called repeatedly per request (SmolVLA, Sec. 3.1). We will add these results to Sec. 4.
W5. "... the conclusion section mainly focuses the components and results ... restate/introduce the problem, and higher conclusion ..."
Answer. Thank you for this suggestion. We rewrote Sec. 5 so that it opens with the problem and ends with what the findings teach about quantizing iterative robot policies:
In this work, we traced W4A8 quantization errors through the two nested loops of VLA inference, where the same quantized weights and activation thresholds serve every denoising step and every request. We showed that channel activation peaks drift across successive requests, and that an extreme-value model captures the sub-linear threshold growth with calibration frames. Grounded in this analysis, we introduced LoopQuant, a W4A8 PTQ framework tailored to these loops. For activations, RMAEC aggregates diverse observations while preserving per-request extrema, with ablations confirming that both broad coverage and extremal aggregation are critical to closed-loop success. For weights, an exact non-uniform codebook determines globally optimal four-bit levels under fixed row scales. Evaluations on π0.5 and GR00T-N1.5 show that LoopQuant nearly matches the closed-loop success of unquantized models. At runtime, packed storage reduces resident weight memory and weight traffic, so that under the same single-launch execution LoopQuant serves requests 1.87× faster than the unquantized BF16 model. Our work stresses the need to calibrate quantized robot policies across both loops and to judge them by closed-loop task success.
Response to Reviewer phKY (1/2)
We thank the reviewer for the constructive comments and for the positive remarks on the closed-loop evaluation and the calibration experiments.
Q1. "How does LoopQuant compare with a BF16 model using equivalent dispatch optimization? Could you separate the latency effects of packed quantization and single-launch execution?"
Answer. Thank you for raising this; another reviewer asked the same question, so we measured BF16 with equivalent dispatch. With the same single-launch core and graph capture on both sides, LoopQuant is 1.87× faster than BF16 (86.67 vs. 162.01 ms per request; Table R1). Since dispatch is identical in this row, the 1.87× comes from packed quantization. In the Table 4 row, by contrast, only LoopQuant used the single-launch core, so its 1.88× mixes both effects. We did not measure the single-launch core in isolation.
Table R1. Request latency under the two dispatch settings (new entries: mean ± std over 30 requests after 5 warm-up requests).
Where the quantization speedup comes from. In the new setting we also varied only the decoding path. Decoding into a buffer before the GEMM already gives 93.61 ms (1.73×), and fusing the decoding steps brings it to 86.67 ms (−6.94 ms); both paths compute the same decoded weights and match the reference bit for bit (App. C.1). Apart from latency, 4-bit storage lowers resident memory to 4.100 GiB (0.59× of BF16). We will report both settings in Table 4 and make the abstract's speedup refer to the matched setting.
W1 / Q2. "Would the accuracy advantage remain if compatible baselines used the same 32 calibration observations and denoising-step coverage?"
Answer. Thank you for this question. Table 2 of the paper provides this comparison: its base row is the QuantVLA baseline of Table 1 (82.4% / 72.8%), and all other W4A8 settings are fixed while only the calibration changes. Given the same 32 observations and full step coverage, QuantVLA rises to 97.1% on π0.5 and 81.0% on GR00T-N1.5, yet LoopQuant still leads it by 0.2 pp and 3.3 pp, respectively (Table R2). The advantage thus remains under matched calibration, although much of it comes from the calibration itself, which is the component we propose.
Table R2. Effect of RMAEC with all other W4A8 settings fixed (from Table 2).
For DuQuant, however, a matched version cannot be built without changing the method. Its calibration data only locate outlier channels for its rotations (DuQuant, App. E.4), and its activation quantizer is per-token, asymmetric, and recomputed from each input, so plugging in RMAEC's per-channel static thresholds would change granularity, symmetry, and dynamic vs. static scales at once. Table 2 avoids this, as both arms share the same static per-channel quantizer.
W2. "The benefit of exact codebook optimization over a strong approximate solver is not established."
Answer. Thank you for this suggestion. We added weighted Lloyd, a strong approximate solver for the same objective. Exact DP lowers the total reconstruction objective by 0.374% relative to Lloyd and is no worse on any of the 252 projections (Table R3). In closed loop it is 0.3 pp higher without RMAEC and 0.2 pp higher with RMAEC (Table R4). Exact DP thus stays ahead of a strong approximate solver at both levels, and exactness further gives a global-optimality guarantee and deterministic solving without initialization or restarts.
Table R3. Weighted reconstruction objective of Sec. 3.2 over the 252 projections of π0.5 (same row scales, activation second moments, 16 levels; lower is better).
Table R4. Closed-loop W4A8 success on π0.5 (Lloyd is new; the rest is from Table 2).