- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SoloQ, a calibration-free quantization method for diffusion language models. The main motivation is that activation distributions in diffusion LLMs change across denoising steps and mask ratios, making it difficult for a small calibration set to cover all inference states. SoloQ addresses this issue through a structured random rotation, K-RPBH, which aims to make the activation distribution more suitable for low-bit quantization. The method also introduces a scale-based bias correction and a commit-time KV cache quantization strategy.
Authors evaluates SoloQ on several diffusion LLMs, including LLaDA, Dream, and Fast-dLLM v2, covering weight, activation, and KV cache quantization. The results show promising performance, with improvements in quantization accuracy and reductions in memory usage and inference latency.
Critical strengths
-
The proposed K-RPBH rotation combines within-block mixing, random permutation, and cross-block mixing to improve the distribution of activation energy across channels. The ablation results show that adding permutation and cross-block mixing reduces KL divergence from 0.11396 to 0.00413 and energy imbalance from 67.89% to 22.12%. These results provide useful evidence that the proposed rotation makes activations more suitable for quantization.
-
Activation distributions in diffusion LLMs vary across denoising steps and mask ratios, which makes conventional calibration-based quantization less reliable. SoloQ avoids the need for calibration data and is evaluated across several model families and quantization settings. This makes the method relevant to practical deployment, especially when representative calibration data are difficult to obtain.
Critical weaknesses
-
The paper mainly shows that the proposed rotation improves the distribution of activation energy across coordinates. For a normalized vector and a rotation , the transformed vector has a more balanced coordinate-wise energy distribution in expectation. However, this does not directly establish that quantization error after rotation will be smaller in terms of model output or downstream task accuracy. The current analysis does not provide a clear bound on the resulting output error or task loss. The paper would be stronger if it connected the rotation properties to weight-dependent sensitivity, error propagation, or an end-to-end quantization error bound.
-
Although the paper explicitly identifies the limitations of OrbitQuant's RPBH rotation and introduces cross-block mixing to address them, the proposed K-RPBH remains a relatively incremental extension of an existing rotation scheme. The main modification is to add a Kronecker-structured cross-block mixing stage to improve block-wise energy balancing. While the paper provides theoretical analysis and empirical evidence for this design, it is less clear whether the modification constitutes a sufficiently substantial methodological advance beyond OrbitQuant, rather than an adaptation of an existing technique to diffusion language models. The authors should better clarify the fundamental novelty of K-RPBH and distinguish the contributions of the rotation itself from those arising from its application to diffusion LLMs and KV-cache quantization.
Questions for authors
-
The paper claims that K-RPBH improves upon original RPBH by introducing cross-block mixing via Kronecker product , and proves the expected coordinate variance equals . However, this guarantee holds only in expectation over random . For a fixed pre-sampled K-RPBH matrix used during inference, there will be residual block-level energy imbalance. Could you quantify how the choice of fixed affects the KL divergence between the empirical coordinate distribution and the ideal spherical marginal distribution? And how sensitive is the final downstream generation accuracy to different random seeds of ?
-
In the ablation studies of K-RPBH, the paper evaluates and separately, but does not conduct a clean controlled experiment to disentangle the gain from cross-block mixing versus the codebook / bias correction. Could you provide an experiment that fixes the same quantization pipeline (same codebook, scale-only bias correction) and only swaps the rotation module between vanilla RPBH and K-RPBH on your dLLM benchmarks? This would help isolate how much performance gain purely comes from the cross-block mixing component of K-RPBH.
-
OrbitQuant is the most closely related prior work that adopts RPBH rotation + Lloyd-Max codebook for diffusion models. However, the paper does not include a direct head-to-head comparison with OrbitQuant under identical dLLM setting (same model, same W4A4 bit-width, same evaluation tasks). Why is this baseline missing? What are the key differences in activation statistics between image DiT and dLLM that make OrbitQuant fail or underperform on diffusion language models?
2 comments
W2 / Q2 / Q3
K-RPBH builds on OrbitQuant's RPBH. Our contribution is diagnosing why RPBH fails on dLLMs and fixing it with a cross-block mixer. Below we separate the contributions (W2), swap only the rotation (Q2), and compare with the OrbitQuant recipe (Q3).
(1) Contributions (W2).
Rotation itself (model-agnostic):
- Diagnosis: a channel with energy fraction forces some RPBH block energy under every permutation, so RPBH's floor grows with the block count ( in Dream/Fast-dLLM v2 FFNs).
- Fix that brings the marginal to the Haar target, shown in theory and in measurement: a Haar cross-block mixer for any (no Hadamard of order needed) at extra cost. In theory, every coordinate has variance exactly in expectation (Prop. 1), and a single draw shrinks the block-energy variance to of RPBH's (Prop. 3, Q1). In measurement, one fixed K-RPBH matches the near-Gaussian target marginal as closely as dense Haar at every mask ratio (Dream, : KL 1.7–4.7e-5 vs. 1.6–2.6e-5 for Haar and 0.011–0.018 for RPBH, Q1).
Application to dLLMs and KV cache (not rotation novelty):
- Scale-only bias correction (BC), commit-time KV-cache quantization for block diffusion, the fold/unfold analysis, and calibration-free NVFP4 / E8P (QuIP#'s codebook) / INT4 paths.
(2) Experiments (Q2, Q3). "C/N, RPBH" is SoloQ-C/N with only the rotation swapped. "OrbitQuant recipe" is our implementation inside the SoloQ pipeline (RPBH, Lloyd–Max weights and activations, no BC). All rows use one harness and are means over the same 3 fixed random seeds. Fast-dLLM v2 IFEval is prompt-level strict. The Fast-dLLM v2 rows are re-runs with new seeds, so they are compared only with each other, not with Table 2.
Table R3. Fast-dLLM v2.
| Fast-dLLM v2 | HE | HE+ | IFEval |
|---|---|---|---|
| OrbitQuant recipe | 58.53 | 54.70 | 53.54 |
| C, RPBH | 56.70 | 52.60 | 54.77 |
| SoloQ-C | 60.17 | 55.10 | 57.18 |
| N, RPBH | 60.26 | 55.38 | 56.67 |
| SoloQ-N | 64.16 | 59.14 | 57.19 |
Table R4. Dream-7B.
| Dream-7B | ARC-C | Wino | PIQA |
|---|---|---|---|
| OrbitQuant recipe | 53.27 | 69.69 | 72.25 |
| C, RPBH | 55.55 | 70.22 | 73.56 |
| SoloQ-C | 56.46 | 71.77 | 74.48 |
| N, RPBH | 56.80 | 71.59 | 74.32 |
| SoloQ-N | 56.34 | 72.09 | 74.61 |
| FP16 | 59.98 | 74.43 | 75.46 |
(3) Rotation only (Q2). The "C/N, RPBH" rows above are this experiment: same codebook and BC, with only the rotation swapped. K-RPBH improves SoloQ-C on all six tasks (+0.9 to +3.5). SoloQ-N gains less or not at all (Dream ARC-C −0.5), likely because its per-16 scales remove the codebook-mismatch term of Prop. 2 (W1). At (LLaDA-1.5, SoloQ-N HellaSwag, 3 seeds, not in the tables above), the gain is small: 72.99 → 73.54.
(4) OrbitQuant head-to-head (Q3). OrbitQuant targets image and video DiTs, and we did not port the recipe to dLLMs. Given the shared RPBH rotation, this was an omission, and the tables above add it. SoloQ-C beats the OrbitQuant recipe on all Dream tasks (+2.1 to +3.2) and on Fast-dLLM v2 IFEval (+3.6), and HE (+1.6) and HE+ (+0.4) are close. Codebook + BC alone (C, RPBH vs. the recipe) helps on Dream but not on Fast-dLLM v2 HE/HE+ (−1.8 / −2.1), where adding does (+3.5 / +2.5).
To compare activation statistics directly, we ran the same probe on FLUX.1-schnell, one of the DiTs evaluated in OrbitQuant, over all 4 sampling steps. As in both methods, is the largest power of two dividing , so .
Table R5. Activation statistics, image DiT vs. dLLM (5 rotation draws, ranges over sampling steps or mask ratios). is the energy share of the largest channel per token.
| Model | Layer | RPBH worst block | KL RPBH | KL K-RPBH | KL Haar | |||
|---|---|---|---|---|---|---|---|---|
| FLUX.1-schnell | FFN down | 12288 | 3 | 0.018–0.021 | 1.08× | 2.4–3.2e-5 | 2.2–3.5e-5 | 2.3–3.1e-5 |
| FLUX.1-schnell | attn input | 3072 | 3 | 0.050–0.079 | 1.14–1.19× | 1.0–1.3e-4 | 0.95–1.4e-4 | 0.98–1.6e-4 |
| Dream | FFN down | 18944 | 37 | 0.093–0.111 | 4.41–5.04× | 0.011–0.018 | 1.7–4.7e-5 | 1.6–2.6e-5 |
| Fast-dLLM v2 | FFN down | 18944 | 37 | 0.089–0.110 | 4.28–5.04× | 0.009–0.018 | 2.0–9.4e-5 | 1.7–5.2e-5 |
The two differ in both structure and activations. FLUX has few large blocks () and a weak top channel (), so RPBH already reaches the Haar KL and adds nothing. The dLLMs have many small blocks () and a top channel carrying about 10% of the energy, so RPBH leaves a worst block of 4.3–5.0×, consistent with the floor – of contribution 1, and a KL more than 400× that of Haar. removes both.
We will add these results to the revised version.
We greatly appreciate your efforts in reviewing our work and providing valuable feedback, which encourages us to improve our work.
W1
Thank you for the technical review. We agree that the paper argues only about the rotated activation distribution and gives no task-level bound. We connect the rotation to output error in three steps: a weight-dependent error model, measured error propagation, and task accuracy.
(1) Weight-dependent error model (new Prop. 2). For a rotated linear layer, we derive the expected output error under standard assumptions (uncorrelated coordinate errors, Gaussian coordinates within each block). It has three terms:
RPBH applies the same permutation to x and to the columns of W, so an outlier channel and its weight column stay in one block and the covariance term does not shrink. K-RPBH's Qk shrinks both the block-energy variance and this covariance by 2/(k+2) in expectation (Prop. 3, stated in our answer to Q1 below), so both extra terms shrink. This is an error model, not a bound. The full statement and derivation will be in App. A.
(2) Measured error propagation. SoloQ-C at W4A4, mask ratios 0.1–0.9, 3 rotation seeds. "Layer error" is ∥Wx−y^∥/∥Wx∥ of the FFN down-projection, averaged over layers and ratios. Logit KL and top-1 agreement are measured against the FP model.
Table R1. Error propagation.
At k = 37, K-RPBH lowers the layer error and the logit KL of RPBH to the level of dense Haar. At k = 3 (LLaDA-1.5) the blocks are already nearly balanced, so the two rotations nearly coincide: the layer error is the same, and RPBH's logit KL is marginally lower (0.0667 vs. 0.0683). The gain of K-RPBH therefore grows with the block count k.
(3) Task accuracy. With only the rotation swapped, K-RPBH improves SoloQ-C over RPBH by +0.9 to +3.5 points on Fast-dLLM v2 and Dream (Table R3 of Q2).
Q1
Your point is right that Prop. 1 holds only in expectation over Qk. We quantify (a) the distribution of one fixed draw and (b) the accuracy across draws. Each rotation seed redraws P, D and Qk, so the seed variation includes the variation due to Qk alone and does not understate it.
(a) One fixed draw. Five fixed draws, FFN down-projection, mask ratios 0.1–0.9 (k=37). "Worst block" is the per-token maximum block energy divided by the average, averaged over tokens and layers.
Table R2. One fixed draw (k=37).
Every fixed draw of K-RPBH stays close to dense Haar in KL. A residual block imbalance remains (1.5–1.7× vs. 1.14× for Haar), but it is about 3× smaller than RPBH's. Our new Prop. 3 quantifies this: for any orthogonal Qk the block energies still average to 1, and for a Haar Qk their expected variance is k+22 of RPBH's, i.e., 19.5× smaller at k=37 (a single realized draw adds a small O(1/h) term). At k=3 (LLaDA-1.5) the worst block drops from 1.22–1.28× to 1.08–1.20×, but the KL does not drop, so KL is only a proxy.
(b) Accuracy across seeds. Tables R3 and R4 (in our answer to W2/Q2/Q3) report means over 3 fixed random rotation seeds. Across these seeds, SoloQ accuracy varies by at most 2.7 points (std) on every task.