- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper aims to make 4-bit (W4A4, NVFP4) inference of Cosmos 3 world models more accurate without sparse kernels or token reordering. The core idea is to quantize the leftover quantization error into extra FP4 matrices and add a few FP4 products that correct the base result. Applying these corrections everywhere is costly. The method therefore repairs only selected tiles under a fixed compute budget, guided by attention patterns in Cosmos 3 and by offline calibration. The same selection also decides which residuals are stored and packed, which reduces memory traffic. The evaluation uses Cosmos 3 Edge (4B) and Nano (16B) on RBench and PAI-Bench-G. At the 2× budget, ThunderWorld gives 3.3 to 7.8% lower LPIPS than ARCQuant (2.0) and 1.17 to 1.88× higher throughput on RTX PRO 6000 and Jetson AGX Thor, with task scores close to BF16.
Critical strengths
S1. Careful and honest evaluation. The evaluation uses about 1,700 scenes per model with paired seeds, bootstrap intervals, and multiple-comparison correction. The controlled study in Appendix A.2 shows that a few extra FP4 products can match native FP8 quality, which is a useful finding on its own. The paper also reports its negative results openly, which makes its claims trustworthy.
S2. Real end-to-end speed measurements. The paper reports full-request latency, kernel-time breakdowns, and weight memory on both a server GPU and an edge device, rather than GEMM speedups alone. It shows that the data preparation around GEMMs, rather than the GEMMs themselves, dominates the cost of existing FP4 correction methods. This is useful for deploying world models on edge hardware.
Critical weaknesses
W1. The method gives back the speed advantage of FP4. Each residual plane recovers part of the precision that FP4 drops, so bilateral repair essentially emulates a higher-precision product using extra FP4 work. At the 2× budget, the matrix work is roughly equal to native FP8, since FP4 has nominally about twice the peak throughput of FP8. Yet FP8 stays more accurate in every setting. The paper never explains why “W4A4 plus repair” is a better design point than simply using FP8, or than mixed designs such as FP4 weights with FP8 activations, which keep most of the memory savings. Without this justification, the core premise of the paper is unclear.
W2. FP8 latency is missing. Native FP8 latency is not reported in Table D.1 or Figure 6, even though FP8 is more accurate than every FP4 method in Table 2. This measurement is needed to judge whether ThunderWorld offers a better accuracy-speed trade-off than FP8.
W3. The baselines may be unoptimized. The speedup over ARCQuant and SharQ comes mostly from data preparation overhead in the authors’ own adaptations of these LLM methods. ARCQuant (1.5) adds only 0.5× extra linear work, yet it runs 1.46 to 2.10× slower than plain W4A4. This points to an implementation issue rather than an inherent cost of the baseline methods.
W4. The speedup is shown only in a narrow setting. All tests use a low 320×192 resolution with batch size one, a setting where the overheads around GEMMs are large. The gain is already smallest when GEMMs dominate (1.17× for Nano on RTX PRO 6000), so it may shrink at realistic resolutions.
W5. The attention analysis is not tested in full generation. Section 3 is the main motivation for tile selection, but it is evaluated only with a local attention-error diagnostic. That diagnostic also uses a different selector from the one deployed in the final method.
W6. Ablations are missing. No full-generation experiment separates the effects of tile selection, attention repair, and offline weight calibration. It is therefore unclear which part of the method produces the gains.
W7. The accuracy gain over ARCQuant is small. At the 2× budget, the LPIPS gain over ARCQuant (2.0) is 3.3 to 7.8%, and it is not statistically significant on Edge RBench after correction. At the 1.5× budget, ThunderWorld is significantly worse than ARCQuant (1.5) in all four settings.
W8. Task scores are missing for the main baselines. Task scores are reported only for BF16, FP8, W4A4, and ThunderWorld. Without scores for ARCQuant and SharQ, it is unknown whether the LPIPS gain matters in practice.
Questions for authors
Q1. What are the latencies of native FP8, BF16, and NVFP4-AWQ under the Table D.1 protocol on both devices?
Q2. Do the W4A4, ARCQuant, and SharQ runs use the same CUDA Graphs, fused kernels, and tuning as ThunderWorld? Why is ARCQuant (1.5) 1.46 to 2.10× slower than W4A4?
Q3. At the 2× budget, how does full-generation quality change if you (a) replace your selector with random or current-mass-only selection, (b) remove attention repair, or (c) remove offline weight calibration?
Q4. Can you report task scores for ARCQuant (2.0), SharQ, and NVFP4-AWQ? Is the 8B judge sensitive enough, given that it cannot separate your 1.5× and 2× settings?
Q5. Do the results hold when development and calibration scenes are excluded from the test set?
Q6. Why is SVDQuant, a W4A4 method for diffusion transformers, not included as a baseline?
Q7. Does the speedup hold at higher resolution or larger batch sizes?
12 comments
Response to Reviewer hKJk (2/5)
W1 (continued).
Table D.10. Exclusive B200 at 600 W, n = 18 per arm (Super 640: n = 12). Latency is a ratio to FP8 in the same campaign; best FP8 is the faster of batched and unbatched FP8.
| Workload | Hybrid W4A4 / FP8 | Hybrid 1.5× before fixes | Hybrid 1.5× fixed / unbatched FP8 | Hybrid 1.5× fixed / best FP8 | Hybrid 2× fixed / best FP8 |
|---|---|---|---|---|---|
| Edge 320×192 | 0.915 | 1.178 | 0.966 | 1.056 | 1.099 |
| Nano 320×192 | 0.872 | 1.226 | 1.008 | 1.089 | 1.190 |
| Super 320×192 | 0.776 | 1.262 | 1.017 | 1.041 | 1.185 |
| Super 640×384 | 0.818 | — | 1.061 | 1.061 | — |
| Edge 1280×704 | 0.945 | 1.052 | 1.019 | 1.024 | 1.047 |
Takeaway: after the fixes, repaired FP4 is 2–9% slower than the best FP8. The unbatched column gives CFG batching to ThunderWorld only, so it does not show ThunderWorld to be faster.
Table C.10. LPIPS to BF16 (lower is better) on 64 scenes × 2 seeds per benchmark, paired bootstrap with family-wise 95% intervals against FP8.
| Config | Edge RBench | Edge PAI | Nano RBench | Nano PAI |
|---|---|---|---|---|
| Native FP8 | .127 | .088 | .108 | .081 |
| ThunderWorld 1.5× | .203 (worse) | .167 (worse) | .151 (worse) | .126 (worse) |
| ThunderWorld 2× | .178 (worse) | .141 (worse) | .127 (worse) | .102 (worse) |
| ThunderWorld 2.5× | .140 (n.s.) | .104 (worse) | .114 (n.s.) | .086 (n.s.) |
| Full three-term | .092 (better) | .059 (better) | .092 (better) | .069 (better) |
Takeaway: using FP4 MMAs only, three-term repair beats FP8 in all four settings and 2.5× matches it in three. Both are slower than FP8.
What the design offers is memory (static weights 31–34% below FP8; peak device memory 5–9% below FP8 with matched fusion) and the fidelity range above, built from FP4 products only.
Table D.12. Device memory on an exclusive B200 at 320×192 (GiB). Quantized linears are the 168 generation-tower linears, including codes, scales, residual banks and metadata. Request peak is peak allocated memory per request, as in Table 4.
| Component (GiB) | BF16 Edge | FP8 Edge | Hybrid 1.5× Edge | FP8 Nano | Hybrid 1.5× Nano |
|---|---|---|---|---|---|
| Quantized linears | 2.625 | 1.313 | 0.864 | 6.469 | 4.255 |
| Whole transformer | 6.285 | 4.973 | 4.524 | 21.828 | 19.614 |
| VAE | 1.334 | 1.334 | 1.334 | 1.334 | 1.334 |
| Request peak | 7.856 | 6.543 | 6.142 | 23.535 | 21.410 |
| Peak reserved | 9.051 | 8.035 | 7.566 | 24.961 | 22.973 |
Takeaway: the quantized linears are 34% smaller than with FP8 (67% smaller than with BF16). The request peak falls only 6–9% vs FP8, diluted by the unquantized understanding tower, output head and embeddings (3.7 GiB Edge, 15.4 GiB Nano) and the 1.33 GiB VAE, which no method quantizes, FP8 included. On Super the linears shrink 34.3% and the request peak 10.7% (Super table), so the saving grows with model size.
W4A8. Our runtime has no native packed W4A8 kernel, and the available stand-ins either give up the weight-storage saving or use BF16 GEMMs, so we report no W4A8 number (Appendix D.4).
Response to Reviewer hKJk (3/5)
W2. FP8 latency is missing.
Now reported, and FP8 is faster. We give FP8 and BF16 the peripheral fusions that our Edge recipes carry, which the submission omitted. Tables D.2 and D.5 add unfused FP8 and BF16, W1 gives the post-fix ratios, and the submission's Figure 6 is now Figure D.3. We use no timing recorded on a shared GPU.
Table 4. Median latency per complete request at 320×192, batch 1, n = 30 per method. B200 is an exclusive GPU at the site's 600 W cap; Thor is measured on the whole board. FP8 and BF16 get the same peripheral fusions as our Edge recipes.
| Method | B200 Edge (s) | B200 Nano (s) | Thor Edge (s) | Thor Nano (s) |
|---|---|---|---|---|
| BF16 + fusion | 0.844 | 2.630 | 5.974 | 20.615 |
| FP8 + fusion | 0.849 | 2.159 | 5.496 | 13.849 |
| ThunderWorld 1.5× | 1.153 | 2.902 | 7.204 | 21.195 |
| ThunderWorld 2× | 1.224 | 3.336 | 7.876 | not run |
| Hybrid 1.5× | 1.000 | 2.647 | 6.315 | 19.004 |
| Hybrid 2× | 1.076 | 3.077 | 6.896 | 22.022 |
Takeaway: FP8 is fastest everywhere. Hybrid 1.5× (FP4 repaired projections, BF16 attention) is 15–37% slower; all-FP4 1.5× is 31–53% slower.
W3. The baselines may be unoptimized.
Correct; they were not execution-matched. Our ARCQuant and SharQ adapters ran eagerly, because the official ARCQuant path needs a host-side scalar that our adapter supports only eagerly. They also ran without peripheral fusion, while ThunderWorld replayed CUDA Graphs (Table D.7).
- We withdraw the 1.17–1.88× throughput advantage and its attribution to repair organization.
- We did not run an execution-matched comparison: our server GPUs were shared, and porting the baselines to B200 was outside the scope of the discussion period.
- The paper makes no latency claim against ARCQuant or SharQ and instead compares against FP8 and BF16 under matched execution.
W4. Speedup shown only at 320×192 and batch 1.
Fidelity claims are now limited to 320×192 and batch 1. Against FP8, the gap narrows as GEMMs grow but does not close. We did not measure batch sizes above 1.
Table D.3. Cosmos3 Edge on an exclusive B200, before the fixes; latency as a ratio to fused FP8. The 320×192 plans are reused at the larger sizes.
| Configuration | 320×192 | 640×384 | 1280×704 |
|---|---|---|---|
| FP8 + fusion (s) | 0.849 | 3.585 | 20.188 |
| Hybrid W4A4 | 0.91 | 0.92 | 0.95 |
| Hybrid 1.5× | 1.18 | 1.10 | 1.05 |
| Hybrid 2× | 1.27 | 1.19 | 1.10 |
Takeaway: the gap narrows with resolution. After the fixes, Hybrid 1.5× is 1.024× the best FP8 at 1280×704 (Table D.10).
Cosmos3-Super at 320×192 on an exclusive B200 at 600 W. Latency ratios are to FP8 of the same campaign (no fusion recipe). Before the fixes: FP8 6.625 s, n = 30 (Table D.4). After the fixes: n = 18, or n = 12 at 640×384 (Table D.10). Memory is request-time (Table D.12). LPIPS is against B200 BF16 on 6 videos.
| Arm | Before fixes / FP8 | Fixed / best FP8, 320 / 640 | Quant. linears (GiB) | Request peak (GiB) | LPIPS |
|---|---|---|---|---|---|
| FP8 | 1.000 | 1.000 / 1.000 | 29.06 | 92.7 | 0.074 |
| BF16 | 1.318 | — | 58.13 | 121.8 | — |
| Hybrid W4A4 | 0.786 | — | 18.19 | 82.0 | 0.131 |
| Hybrid 1.5× | 1.262 | 1.041 / 1.061 | 19.10 | 82.8 | 0.099 |
| Hybrid 2× | 1.524 | 1.185 / — | 20.92 | 84.7 | 0.077 |
Takeaway: on the largest model, unrepaired FP4 is faster than FP8. The fixed Hybrid 1.5× is 4–6% slower than the best FP8 (1.017× / 1.061× unbatched FP8). Its quantized linears are 34.3% smaller and its request peak 10.7% smaller than FP8 (32.0% below BF16). Our on-device preparation stages the BF16 checkpoint, so every arm briefly reaches about 121 GiB at load time; this is excluded above. The earlier PRO 6000 smoke figures did not include the full model and are withdrawn.
Response to Reviewer hKJk (1/5)
Thank you for the detailed review. W1, W2 and W3 were correct: at this scale FP8 is the better design point for latency, the submission did not report FP8 latency, and our baselines did not run under the same execution conditions as ThunderWorld. All three change what the revised paper claims. We answer in your order; Appendix F of the revised PDF repeats these answers next to your comments.
W1. Repair gives back FP4's speed advantage; why W4A4 plus repair rather than FP8 or W4A8?
At this scale, FP8 is faster than every repaired configuration and more faithful than our 1.5× and 2× configurations, so for latency FP8 is the better choice, and the paper now says so. We measured why the repaired path is slower and fixed most of it (Section 5.4, Appendix D.6).
Table D.9 and Appendix D.6. Per-GEMM traces of one request on an exclusive B200, before the fixes.
Takeaway: unrepaired FP4 with BF16 attention is 0.76–0.95× FP8 latency. The gap comes from the repair kernel, whose 1-SM tile waited on residual scale loads, not from the FP4 tensor path. It is largest where requests are GEMM-bound: repair adds 61% to unrepaired FP4 on Cosmos3-Super, against 30% on Edge at 320×192.
Appendix D.6. Kernel and runtime changes, end-to-end latency on an exclusive B200 at 600 W.
Takeaway: the first three changes are kept. The K256 tile does not help because activation-repair masks are scattered, so most K256 pairs carry a zero-scaled half. A 2-SM repair collective was not attempted.