- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CRAFT, a training-free method for compressing diffusion/flow-matching VLAs, with a focus on the action head. Because the action head is evaluated to generate each action chunk, it can account for a large share of the weight traffic under single-sample closed-loop inference. CRAFT therefore reduces both the precision and the evaluation count of the action head. It combines W2A4 quantization using an 8-dimensional E8-lattice codebook, and static caching that evaluates the head only at fixed compute nodes and reuses the last predicted field while still updating the solver state at every step.
Critical strengths
The paper offers useful analytical insights into compressing diffusion-head VLAs.
- The weight traffic and latency of the diffusion head, which grow with the number of denoising steps
- The concentration of W2 output error in a few high-energy directions
- The model-dependent weight tails that determine whether rotation helps
Critical weaknesses
The novelty of this paper is limited. Apart from Tail-guided rotation, the components of the method appear to be drawn from prior work: block rotation from DuQuant, step-wise activation scales from HoloQ-VLA, and fixed-interval caching all existed before this work.
The "matched W2A4" comparison is not matched in bits. The action head in CRAFT uses 3.14 bpw. This is a measured value that includes rotation matrices (0.997 bpw) and activation tables (0.126 bpw). The scalar baselines (QuantVLA-style, PTQ4DiT, TFMQ-DM) are reported at 2.25 bpw, a nominal value that excludes such overhead. CRAFT therefore uses roughly 0.9 more bits per weight than the baselines it is compared against.
Questions for authors
Q1. With caching, the action head is evaluated only twice per action chunk (NFE = 2), so the backbone dominates the weight traffic. In this setting, lowering the head precision from W4 to W2 reduces nominal weight traffic by only 0.21 GiB on GR00T N1.5 (2.64 -> 2.43 GiB, Table 1). What is the measured end-to-end latency difference between CRAFT and CRAFT (W4 head)?
Q2. Under the full CRAFT configuration, the success rate drops sharply relative to FP16 on π0 (98.4 ->24.0) and CogACT (72.1 -> 37.5), while π0.5 is barely affected. What causes these failures?
Q3. The static schedule appears to be motivated by the observation that existing adaptive methods (TeaCache, MagCache) mostly select a single step set (Table 11). I may have misunderstood this, so if any part of the schedule was newly designed for the characteristics of diffusion or flow-matching heads in VLAs, could the authors clarify it?
Q4. The compute-node sets differ across models ({0}, {0, 2}, {0, 4}, {0, 5}, {0, 3, 7}). What criterion and data were used to choose them? In particular, was performance on the LIBERO-Object task suite, which is also used for evaluation, used in the selection?
Q5. Adding A4 quantization raises the average success rate from 92.20 to 95.90 (Table 2), exceeding FP16 (94.00, Table 10) on all four suites. CRAFT also exceeds FP16 on GR00T N1.5 (98.4 vs. 96.4, Table 1). Could the authors report results over multiple seeds with confidence intervals?
Q6. According to Appendix D.6, rotation matrices account for about 1 bpw (0.997) of the 3.14 bpw used by the head. Why did the authors choose a stored block rotation over a Hadamard rotation, which requires no storage?
5 comments
What does the W2 head contribute to latency, and how were the schedules chosen?
Response to concern 3 (Q1)
What is the measured latency of the W2 head versus a W4 head? A component-wise timing
The reviewer observes that at NFE = 2 the backbone dominates weight traffic, so lowering the head from W4 to W2 saves only 0.21 GiB nominally. Our measurement agrees: at NFE 2 head precision barely changes end-to-end latency.
- Latency: on GR00T N1.5 (same FP16 backbone, caching {0, 4}, RTX 4090), CRAFT runs at 1.46× with the W2 head and 1.44× with the W4 lattice head, and on CogACT at 1.117× and 1.115×.
- Success: the same manuscript Table 1 success on GR00T N1.5 (98.4 for both).
- What the W2 head buys: storage (3.14 vs. 5.14 bpw); its lower traffic is nominal, since the kernel expands W2 codes to INT4 operands at load time.
To show where the speedup does come from, Table R13 adds one component at a time:
policy-call time, one component at a time (ms; RTX 4090, exclusive; instrumented capture).
| Model (backbone, head) | FP16 | + caching | + integer backbone, FP16 head | + CRAFT head |
|---|---|---|---|---|
| GR00T N1.5 (W4A8, W2A4) | 51.79 | 35.67 | 29.24 | 29.77 |
| π0.5 (W4A8, W2A4) | 71.59 | 47.24 | 31.86 | 32.35 |
- Caching: saves 16.12 / 24.35 ms.
- Integer backbone: saves a further 6.43 / 15.38 ms.
- CRAFT head: adds 0.53 / 0.49 ms.
(GR00T N1.5 / π0.5; end to end 1.74× / 2.21×, within 1 % of Table G1.) , and the W2 head contributes storage while preserving success.
A natural follow-up is whether a simpler integer W4 head would be faster; Table R14 compares CRAFT with an RTN W4A4 head on the same INT4 GEMM, in one process and capture per backbone column (caching {0, 4}): CRAFT's W2A4 lattice head vs. an RTN W4A4 head — GR00T N1.5, RTX 4090 (speedup vs. uncached FP16).
| Head | Speedup, W4A4 backbone | Speedup, W4A8 backbone | Success (W4A4 backbone) |
|---|---|---|---|
| RTN W4A4 | 1.98× | 1.92× | 59.2 |
| CRAFT W2A4 lattice | 1.84× | 1.79× | 95.64 (5 runs; FP16 96.07) |
RTN: per-column round-to-nearest W4 weights and calibrated A4 scales, without rotation or error compensation — the simplest integer W4A4 head, not a tuned W4 method.
- Result: the RTN W4 head is 7–8 % faster but loses 36 pp.
- Caveat: this compares two implementations with method-specific pre- and post-processing and does not imply that lowering precision from W4 to W2 improves accuracy; with a stronger W4 quantizer the gap closes (manuscript Table 1, GR00T N1.5: HoloQ-VLA W4A4 and CRAFT W4 stay near FP16).
Response to concern 4 (Q2)
Why do π0 and CogACT fail while π0.5 does not?
The causes differ (General Response 1/2; responses to mBz2):
- π0: W2 weight precision is a major factor: with A8 activations fixed, W3 weights restore success from 26.4 to 96.8 on all ten tasks.
- CogACT: it is the interaction with reuse: its DDIM ε-prediction solver has a high gain on a reused-field error (3.00), and a W4 head reduces the quantization–reuse interaction from −22.1 to −7.5 pp.
- π0.5: it shows neither: its head-only W2 equals FP16 (98.0).
Response to concern 5 (Q3)
Is anything in the schedule specific to VLA heads?
Yes. The predicted field drifts less than the solver state on six of eight models (manuscript Figure 5), so CRAFT reuses the field while the solver state continues to update at every original step using the most recently evaluated field; the schedule fixes only the steps at which the field is evaluated.
Response to concern 6 (Q4)
What criterion and data chose the node sets?
- Data: LIBERO-Object (initial states 40–49 for GR00T N1.5).
- Criterion: the candidates' closed-loop success in this short screen. The sets were fixed before the reported evaluations but not independently of closed-loop results.
- Later changes: π0.5's set was changed from {0, 6, 9} to {0, 5} after closed-loop results, and the common rule was written afterwards and serves only as a sensitivity check.
- CogACT: its three-node schedule was chosen on its SimplerEnv window and checked on held-out tasks.
On the checks we ran, the choice did not change the conclusions:
- π0.5, unused suites: {0, 5} is +0.8 / −1.2 / +0.6 pp from uncached FP16 on LIBERO-Spatial / Goal / Long and within 1.0 pp on five LIBERO-Plus axes.
- Elsewhere: Table R7 (response to mBz2) shows limited sensitivity (SmolVLA: 4–5 pp lower under the common rule).
Tables R13–R14 and this procedure will be added to the revised manuscript.
Do the accuracy claims hold across runs, and why a stored rotation?
Response to concern 7 (Q5)
Do the results over FP16 hold across repeated runs? A repeated-run test with confidence intervals
The reviewer notes that CRAFT exceeds FP16 on GR00T N1.5 (98.4 vs. 96.4) and that A4 activations raise success above FP16 (manuscript Table 2), and asks for repeated runs with confidence intervals.
- In the manuscript: each entry of manuscript Tables 1–2 is a single run of 500 episodes (240 for CogACT), treated as independent Bernoulli trials (manuscript Assumption 7).
- Test: if these differences were run-to-run noise, they should not persist across repetitions.
- Protocol: we repeated FP16 and CRAFT as independent rollouts (n = 500 each, unseeded noise, one RTX 4090 machine) and report pooled intervals and run- and task-level bootstrap intervals.
GR00T N1.5, LIBERO-Object — independent runs.
| Setting | Runs | Pooled | vs. FP16 [95 % CI] | Run- and task-level bootstrap |
|---|---|---|---|---|
| FP16 | 95.6 / 94.8 / 97.8 | 96.07 | — | — |
| CRAFT (fake quantization) | 98.6 / 98.4 / 98.2 | 98.40 | +2.33 pp [+1.17, +3.56] | [+0.20, +4.60] |
| CRAFT, deployment kernels | 98.0 / 97.6 / 97.4 | 97.67 | +1.60 pp [+0.35, +2.88] | [−0.73, +3.93] |
| W4A8-backbone configuration, deployment kernels | 98.0 / 98.2 / 98.0 | 98.07 | +2.00 pp [+0.80, +3.25] | [−0.27, +4.33] |
- Pooled: across the three runs CRAFT's observed range is 0.4 pp against 3.0 pp for FP16 (a descriptive range, not a variance test), and every pooled difference is positive.
- Run and task level: the intervals of the two deployment configurations include zero, so we state and do not read it as equivalence or superiority.
- Original single-run H100 entries: McNemar tests with Holm correction likewise detect no significant difference.
Repeating manuscript Table 2 in the same way tests the A4 effect:
W2 vs. W2A4 head — three runs per arm (n = 1,500 per arm, RTX 4090).
| Suite | W2A4 | W2 (A16 activations) | Difference [95 % CI] | Run- and task-level bootstrap |
|---|---|---|---|---|
| Long | 91.27 | 86.47 | +4.80 [+2.55, +7.06] | [−0.87, +10.87] |
| Spatial | 96.47 | 89.20 | +7.27 [+5.45, +9.13] | [+3.67, +11.00] |
| Goal | 96.93 | 95.80 | +1.13 [−0.21, +2.50] | [−1.27, +3.67] |
| Object | 98.20 | 98.00 | +0.20 [−0.80, +1.21] | [−1.40, +1.80] |
- LIBERO-Spatial: step-wise A4 activations yield under both kinds of interval.
- Other three suites: no significant difference; on LIBERO-Long, FP16 in these runs reaches 91.27 (W2A4 − FP16 +0.00 pp [−2.03, +2.03]).
- Revision: it replaces "at or above FP16" with "no statistically significant difference from FP16" and reports manuscript Table 2 with these runs and both intervals.
These repeated runs support the A4 improvement on LIBERO-Spatial; the other comparisons carry no claim of significant superiority.
Response to concern 8 (Q6)
Why a stored rotation rather than a storage-free Hadamard? A direct replacement test
- Already in CRAFT: a storage-free randomized Hadamard is applied to every head (incoherence processing, 0.0155 bpw, manuscript Appendix D.6).
- Why a stored rotation on top: the stored block rotation is added only where , because rotation shrinks heavy weight tails but enlarges light ones (manuscript Section 3.1).
We tested the reviewer's alternative in its place:
storage-free Hadamard vs. stored block rotation — W2A4 head (success %, n = 500 per run).
| Model | Storage-free Hadamard | Stored block rotation |
|---|---|---|
| GR00T N1.5, head-only (n = 1,500) | 98.47 (2.14 bpw) | 98.27 (3.14 bpw) |
| GR00T N1.5, full configuration | 97.6 | 98.4 (n = 1,500) |
| GR00T N1.7 | 98.4 | 98.2 |
| X-VLA | 99.0 | 99.2 |
| π0.5 | 98.0 | 99.6 |
| VLA-JEPA (SimplerEnv, n = 828) | 64.4 (2.17 bpw) | 68.2 (4.10 bpw) |
- Matches: the storage-free Hadamard matches the stored rotation on GR00T N1.5, GR00T N1.7 and X-VLA at about 1 bpw less (full GR00T N1.5 configuration: −0.8 pp [−2.6, +0.5]).
- Falls behind: on π0.5 and VLA-JEPA.
- Decision: we retain the tail-guided stored rotation as the default, since it is never more than 0.2 pp worse in the head-only comparisons and gains more on π0.5 and VLA-JEPA, and we report the storage-free variant as an alternative.
Again, we sincerely thank the reviewer for the constructive feedback. If you feel your concerns have been addressed, we would be grateful if you could reconsider the rating.
Response to Reviewer aeLW (1/3): What is new, and are the bits compared fairly?
We thank the reviewer for the careful reading and valuable feedback. Our responses follow; shared points are in the General Responses.
Response to concern 1 (W1)
Are CRAFT's components only drawn from prior work? A test against a head assembled from them
The reviewer attributes block rotation to DuQuant, step-wise activation scales to HoloQ-VLA and fixed-interval caching to prior work. We agree that these primitives are established, and that quantization and caching have been combined before (CacheQuant); we claim novelty neither in them nor in the identities of manuscript Propositions 1 and 4.
The question specific to CRAFT is whether head precision and action-field reuse can be designed independently in a closed-loop VLA policy — the subject of manuscript Section 3, which guides CRAFT's design (manuscript Section 4). Two measurements show what the analysis adds.
(1) Quantization and caching are not additive. If they were, the extra loss caching causes relative to FP16 would not depend on head precision.
(2) The error and execution analyses lead to measurable choices.
CRAFT is therefore an empirically grounded quantization–execution co-design whose decisions follow from measured interactions, not a new quantizer or caching primitive.
Response to concern 2 (W2)
Is the W2A4 comparison matched in bits? A measurement under one accounting
The reviewer notes that CRAFT's 3.14 bpw is measured, including rotation and activation tables, whereas the baselines' 2.25 bpw is nominal.
Table R11: measured head storage (bpw), eight heads.
CRAFT's extra storage is the rotation, present only on the four heads with Drot>1.
Table R11 will be added per model to the revised manuscript.