acceptodds
Under review as a conference paper at ICLR 2027

BTT: Quantizing Diffusion Language Models via Block-wise Token Transformations

Abstract

Low-bit activation quantization remains challenging for diffusion language models (dLLMs), whose activations vary across token positions and denoising states. Existing channel transformations reshape feature coordinates within tokens, leaving the token dimension available for complementary transformations. We propose Block-wise Token Transformation (BTT) for linear-input activation quantization in dLLMs. BTT applies randomized Hadamard transforms within blocks of simultaneously processed tokens before activation quantization and restores token coordinates after the linear projection. This construction preserves the linear function in exact arithmetic without quantization, while redistributing activations presented to the low-bit quantizer. The block-wise design enables efficient computation and composes naturally with fixed channel rotations. Because token transformations do not benefit all operators equally, we select them independently for each linear operator by minimizing output reconstruction loss on shared calibration inputs with fixed quantized weights, retaining the identity when no candidate improves reconstruction. Experiments show that adding BTT to QuaRot and DuQuant improves downstream accuracy under W4A4 quantization. These results demonstrate the potential of token-dimension transformations as a complement to channel transformations for low-bit dLLM quantization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Language Models, Post-Training Quantization, Low-Bit Quantization, Token Mixing, Orthogonal Transformations

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_KopbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Block-wise Token Transformation (BTT), a post-training quantization technique for diffusion language models (dLLMs). In contrast to existing transformation-based quantization approaches that mainly operate along the channel dimension, BTT exploits the token dimension by applying an orthogonal transformation to groups of simultaneously processed tokens before activation quantization and restoring the original token coordinates after the linear projection. Because linear projections share weights across tokens, the transformation and its inverse preserve the original linear function in the absence of quantization.

Critical strengths
  1. The token-dimension perspective is simple but conceptually interesting.
  2. The formulation is mathematically well motivated.
  3. The W4A4 improvements are nontrivial on several tasks.
Critical weaknesses
  1. The paper discusses QDLM, DLLMQuant, Quant-dLLM, STaR-Quant, and FAIR-Calib in Related Work, but the main experiments primarily build on generic LLM PTQ methods such as DuQuant, QuaRot, and SmoothQuant. Since the paper is specifically positioned as a method for diffusion language models, comparisons or combinations with recent dLLM-specific PTQ methods would considerably strengthen the empirical claim. Otherwise, it remains unclear whether BTT provides complementary gains over state-of-the-art techniques that already explicitly model masking states and denoising dynamics.
  2. The current evaluation is relatively narrow. Only two dLLM families are evaluated, both around the 7–8B scale, and downstream evaluation is restricted to four code/math benchmarks. It would be important to demonstrate whether the improvements generalize to additional dLLMs, model scales, and broader language-understanding or instruction-following benchmarks.
  3. The connection to diffusion-specific behavior is not yet convincingly established. The mathematical construction itself does not rely on diffusion modeling; it applies whenever multiple tokens are processed together by a shared linear projection.
  4. The cost of per-operator transformation search is not sufficiently characterized. Searching multiple seeds independently for a large number of linear operators may result in non-negligible calibration cost. The paper reports inference overhead but does not provide a detailed analysis of calibration/search time and memory consumption. This is particularly relevant for a PTQ method.
Questions for authors

Why are the QuaRot experiments performed without the identity fallback proposed in Section 3.4? How would QuaRot+BTT perform under exactly the same selection protocol as DuQuant+BTT?

3 comments

Thank you for the review. We address the QuaRot question first, then clarify the evaluation scope and offline costs.

QuaRot and identity fallback. The initial fallback-enabled QuaRot run stopped at the evaluation wrapper, so the FFN-only QuaRot rows in Table 1 report the completed forced-on runs. The fallback rule enables the best of 16 candidates only if it reduces local reconstruction error relative to identity. All scores are percentages.

Model and quantizationFFN policyFFN outputs enabledHumanEvalMBPPGSM8KMATH
LLaDA, QuaRot W4A4No BTT0/3232.9333.8075.6629.96
LLaDA, QuaRot W4A4Best of 16, forced on32/3239.0234.4075.2829.66
LLaDA, QuaRot W4A4Best of 16 + identity fallback0/3232.9333.8075.6629.96
Dream, QuaRot W4A4K16V16No BTT0/2831.1041.6056.2522.40
Dream, QuaRot W4A4K16V16Best of 16, forced on28/2837.8041.8058.4522.42
Dream, QuaRot W4A4K16V16Best of 16 + identity fallback1/2831.3241.8156.2722.40

In the DuQuant W4A4 FFN-only ablation, the selected transform beat identity at every projection (32/32 on LLaDA and 28/28 on Dream). The fallback-enabled policy therefore applied BTT at every FFN output projection, yielding the same per-layer transforms and downstream scores as the forced-on policy. With QuaRot, fallback enabled 0/32 and 1/28 projections, respectively. LLaDA reproduced baseline scores, and Dream stayed within 0.22 percentage points of baseline on all four tasks. Forced-on QuaRot had higher four-task means on both models and raised LLaDA HumanEval pass@1 from 32.93% to 39.02%. On Dream, the forced-on variant gave higher HumanEval and GSM8K point estimates than fallback. These results motivate making fallback optional for FFN-only BTT in the evaluated settings. The two quantizers use the same identity decision rule. A comparison under identical calibration protocols remains open.

1. Comparison with dLLM-specific PTQ. We built BTT on QDLM’s open-source dLLM implementations of DuQuant and QuaRot. BTT transforms token activations around a shared linear projection and reuses the baseline’s quantized weights. This interface can also be used in other dLLM pipelines that jointly process multiple token positions. Within each group, we held decoding and evaluation settings fixed. In Table 1, FFN-only BTT improved mean W4A4 scores by 1.50–3.13 percentage points across the four groups. On Dream, MBPP pass@1 rose from 43.80% to 49.40% with DuQuant, and HumanEval pass@1 rose from 31.10% to 37.80% with QuaRot.

We could not locate public author-provided implementations of DLLMQuant, STaR-Quant, or FAIR-Calib for a matched evaluation. Quant-dLLM provides public code for approximately 2-bit weight-only PTQ. BTT could also be integrated with dLLM-specific weight–activation quantization methods. Whether these combinations yield additional gains remains an open empirical question.

2. Evaluation coverage. We evaluated LLaDA-8B-Instruct and Dream-v0-Base-7B, two families in the 7–8B range also studied by DLLMQuant and STaR-Quant. HumanEval and MBPP cover Python code generation. GSM8K and MATH cover mathematical reasoning. Further evaluation on larger dLLMs, including MoE models, and on language-understanding and instruction-following tasks would test how broadly the observed gains transfer.

3. Connection to diffusion decoding. We agree that the function-preserving identity is general. BTT exploits token activations made available during diffusion decoding. In standard KV-cached autoregressive decoding, a projection receives one new activation row per sequence per step. A nontrivial within-sequence transform would therefore require additional rows. Masked diffusion decoding supplies multiple positions from the same sequence in a single denoising pass, potentially including both masked and already-unmasked positions. BTT exploits this property by mixing their activations before quantization and restoring their token coordinates after the shared projection within the same pass.

4. Offline calibration and search cost. With block size 64 and 16 random seed candidates per projection, the logged W4A4+BTT offline runs took 61 min 21 s for LLaDA-8B-Instruct and 58 min for Dream-v0-Base-7B on a single NVIDIA A100 GPU per model. The projection scope of these runs is pending verification. Inference reuses the selected transforms. On an A100 80GB, peak allocated GPU memory during BTT search was 16.23 GiB for LLaDA and 15.46 GiB for Dream. Base DuQuant W4A4 calibration used 128 WikiText-2 sequences of 2,048 tokens each. On an A100 80GB, peak allocated memory was 20.50 GiB for LLaDA and 17.87 GiB for Dream. Peak reserved memory was 20.82 GiB and 18.87 GiB, respectively.

Thank you for the authors' explanations and additional experiments. However, I still believe that, as a work on dLLM quantization, the proposed method should be thoroughly compared with existing dLLM quantization methods, rather than demonstrating performance improvements over a single baseline. Furthermore, the rebuttal indicates that the actual inference speed of BTT-quantized models remains relatively slow after deployment, e.g., 61 min 21 s for LLaDA-8B-Instruct and 58 min for Dream-v0-Base-7B on a single NVIDIA A100 GPU. Based on these concerns, I maintain my original recommendation of Weak Reject.

Thank you for the follow-up. The 61 min 21 s for LLaDA and 58 min for Dream are the elapsed times of the one-time offline calibration jobs that searched all linear projections. These jobs include model loading, base quantization, calibration forward passes, and candidate scoring. Inference reuses the resulting seed configuration.

The actual inference measurements are in Table 6. On the same H800 fixed-work decoding benchmark, adding FFN-only BTT to DuQuant W4A4 changed latency from 1.548 to 1.617 s/prompt on LLaDA and from 0.950 to 1.008 s/prompt on Dream. The paired overheads were 4.44% and 6.12%, respectively. The current quantized backend is slower than the corresponding floating-point reference under this workload. These measurements quantify the incremental inference cost of BTT on that backend.

At W4A4, the paper evaluates BTT with both DuQuant and QuaRot. At W8A8, it evaluates BTT with DuQuant and SmoothQuant. These are within-quantizer comparisons that measure the effect of adding BTT under each evaluated configuration. We agree that a matched comparison with dLLM-specific PTQ methods would answer a separate empirical question that the current evaluation leaves open.

Reviewer_6rwNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BTT, which applies randomized Hadamard transforms within contiguous blocks of tokens before token-shared linear projections in diffusion language models and restores the original token coordinates after projection. This construction preserves the unquantized linear function and can be combined with fixed channel rotations. The authors further use a local output reconstruction loss evaluated on shared inputs to select a seed or the identity transformation for each operator. Experiments evaluate LLaDA and Dream on code generation and mathematical reasoning tasks under W4A4 and W8A8 quantization, reporting improvements in average accuracy when BTT is applied to FFN output projections, along with analyses of seed generalization, application scope, block size, and runtime overhead.

Critical strengths
  1. Simple construction that is compatible with existing channel-wise transformations. BTT introduces an invertible token transformation and its inverse around token-shared linear layers without requiring changes to the quantized weights, and explicitly addresses bias, nonlinear boundaries, and padding. The error decomposition correctly distinguishes the unchanged weight-error contribution from the activation-error contribution affected by the transformation.
  2. Relatively consistent within-group results for FFN-only BTT, with useful local validation and cost measurements. All four W4A4 groups report average gains, complemented by seed validation on disjoint sequences and repeated timing measurements. The authors also report declines on individual tasks, degradation when BTT is applied jointly to FFN and attention projections, and additional runtime costs, making the method's limitations relatively clear.
Critical weaknesses
  1. The core adaptive selection mechanism has not yet been consistently or sufficiently validated at the full-model level. The main QuaRot results do not use the identity fallback emphasized in Section 3.4, and transformations for all modules are selected independently using baseline inputs collected with BTT disabled. In Table 3, Dream achieves 41.46 and 43.90 on HumanEval with FFN-only and attention-only BTT, respectively, but combining them yields only 32.93, below the baseline of 37.80. The local held-out NMSE gains in Table 4 are insufficient to explain this degradation. Full-model evaluation of the complete selection policy, comparisons with random-seed and no-search variants, and fallback ablations are needed, together with an analysis of the relationship between the selection objective and downstream performance.
  2. The method's comparative value and accuracy–cost trade-off remain insufficiently supported by matched comparisons. The main results largely evaluate BTT as an add-on to general-purpose quantization methods, without comparisons against the dLLM-specific approaches discussed in the paper under the same protocol; some experimental groups also differ in caching configurations and KV precision. Meanwhile, the latency experiments use an INT8 GEMM backend to compute with W4A4 codes without caching, so they do not directly represent either the cached configurations used in the main results or native INT4 deployment. The modest gains in the main tables are also reported without uncertainty estimates from repeated downstream evaluations, limiting the assessment of how substantial the practical benefits are.
Questions for authors
  1. Please evaluate the same "16-seed selection + identity fallback" policy on both DuQuant and QuaRot, and report per-layer BTT enablement, results for a fixed-random-seed control, and an ablation with fallback disabled. Why do the main QuaRot results use a different variant?
  2. How do you explain why FFN-only and attention-only BTT each improve performance on Dream, whereas combining them leads to substantial degradation? Does the ranking of candidate transformations by local reconstruction loss change when inputs are recollected with BTT enabled? Could this affect the reliability of the proposed selection mechanism?
  3. Please provide full details of the calibration mask ratios, decoding step counts, generation lengths, sampling/caching settings, and evaluation configurations used for the main tables, together with accuracy–latency comparisons under matched settings and paired uncertainty estimates for representative downstream evaluations. Do the gains persist when compared against at least one dLLM-specific quantization baseline under the same protocol?
1 comment

Thank you for the careful review. We address the three questions using full-model ablations, per-example outputs, and runtime logs.

Q1. Seed search and identity fallback. We evaluated FFN-only BTT with DuQuant W4A4 and block size 64. The table reports full-dataset pass@1 (%). The random control reports mean ± sample SD across five fixed base seeds (0–4). The LLaDA ablation is a separate set of matched runs from those in Table 1.

BTT policyDream HumanEvalDream MBPPLLaDA HumanEvalLLaDA MBPP
No BTT37.8043.8034.1534.40
Fixed random seeds (mean ± SD, n = 5)39.51 ± 1.5246.32 ± 1.3638.73 ± 2.0035.24 ± 0.88
Best of 16, forced on41.4649.4039.0237.00
Best of 16, with identity fallback41.4649.4039.0237.00

Relative to the five-seed random mean, search raised the HumanEval and MBPP point estimates by 1.95 and 3.08 percentage points (pp) on Dream and by 0.29 and 1.76 pp on LLaDA. The best searched transform beat identity at every eligible FFN output projection (Dream 28/28, LLaDA 32/32). Both policies therefore used identical transforms and produced identical scores.

An initial fallback-enabled QuaRot run stopped before evaluation, so the submitted results use the completed forced-on runs. Applying the same identity rule to LLaDA QuaRot disabled all 32 FFN output transforms and reproduced baseline HumanEval and MBPP scores of 32.93% and 33.80%. Forced-on BTT scored 39.02% and 34.40%. A subsequent Dream QuaRot W4A4K16V16 fallback evaluation enabled 1/28 FFN output projections and scored 31.32% on HumanEval and 41.81% on MBPP. These runs use the same identity decision rule across quantizers. Evaluation under an identical calibration protocol and fixed-random QuaRot controls remain open.

Q2. Joint FFN and attention BTT on Dream. HumanEval pass@1 was 37.80% with no BTT, 41.46% with FFN-only BTT, 43.90% with attention-only BTT, and 32.93% with both. The joint configuration used the same transforms and enabled projections as the union of the standalone configurations. Relative to the baseline, joint BTT changed HumanEval by −4.88 pp (paired 95% CI [−13.41, 3.66]) and MBPP by +4.00 pp.

Upstream BTT may change the activations seen by later projections, whose seeds were selected on inputs collected with BTT disabled. Table 4 measures held-out local reconstruction on inputs from that setting. Whether candidate rankings change on BTT-enabled inputs and whether this explains the lower HumanEval point estimate remain open.

Q3. Evaluation protocol, uncertainty, and latency. We correct the cache setting in Section 4.1 for the Dream DuQuant W4A4 baseline and FFN-only rows in Table 1. The launcher requested prefix caching. Runtime logs show that these runs used Fast-dLLM parallel decoding with caching disabled.

For these Dream DuQuant runs, HumanEval used 16 nominal denoising steps and a maximum generation length of 512 tokens. MBPP, GSM8K, and MATH used 8 nominal steps and 256 tokens. Temperature was 0 and the confidence threshold was 0.9. Few-shot counts were 0, 3, 5, and 4, respectively. Dream DuQuant seed selection used all 128 WikiText-2 examples at mask ratios of 0.25, 0.50, 0.75, and 1.00. Table 4 uses a separate 64/64 selection/validation split. QuaRot seed selection used a 64/64 split, and Dream QuaRot used 16-bit keys and values.

In the full-dataset Dream DuQuant evaluations, the FFN-only gains over no BTT were +3.66 pp on HumanEval (paired 95% CI [−4.27, 12.20]) and +5.60 pp on MBPP (paired 95% CI [1.20, 9.80]). These intervals use paired resampling of prompts at fixed evaluation settings.

On the same 32 HumanEval prompts, the baseline solved 16 and FFN-only BTT solved 22. Across five same-GPU paired repetitions, latency was 33.44 ± 1.04 and 33.64 ± 1.04 s/prompt, respectively (mean ± sample SD). This measurement uses adaptive Fast-dLLM decoding with simulated W4A4. Table 6 reports fixed-work timing using INT8 GEMM on W4A4 codes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.