acceptodds
Under review as a conference paper at ICLR 2027

TempoQuant: Timestep-Aware Calibration for Post-Training Quantization of Masked Diffusion Language Models

Abstract

Post-training quantization (PTQ) of masked diffusion language models (MDLMs) is challenging because their activation states evolve throughout the denoising trajectory. We find that temporal activation variation alone does not reliably indicate quantization difficulty; instead, target-bit quantization vulnerability is localized across denoising states, channels, and operations. Based on this observation, we propose TempoQuant, a trajectory-aware PTQ framework that directly measures channel-wise computation distortion under joint weight–activation quantization. TempoQuant aggregates distortion across denoising states into trajectory-level vulnerability and uses it to guide quantizer adaptation, allocating greater adaptation freedom to vulnerable channels. To account for accumulated quantization errors, Transformer blocks are progressively calibrated on quantized-prefix inputs. All trajectory-dependent information is consolidated into static quantization parameters, requiring no timestep- or state-dependent quantizers at inference. Experiments on LLaDA-8B-Instruct demonstrate strong performance under both W8A8 and W4A4 quantization, with particularly substantial gains in the challenging W4A4 setting. The code is available at https://anonymous.4open.science/r/TempoQuant-1DC7/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization, Diffusion Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ore8OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper observes that temporal variation of activation magnitudes alone cannot reflect quantization vulnerability, and the vulnerability is concentrated on a few layers and channels. It also finds that errors are propagated across transformer blocks, but conventional PTQ methods perform calibration on full-precision inputs. To this end, this paper proposes a trajectory-aware PTQ framework for MDLMs, dubbed TempoQuant. It directly estimates discrepancies of block-wise outputs before and after quantization to estimate the quantization vulnerability of each channel, and uses the vulnerability to learn quantization parameters. It also progressively calibrates blocks on quantized-prefix inputs. Experimental results demonstrate the empirical effectiveness of TempoQuant.

Critical strengths

(+) The authors show the quantization vulnerability of MDLMs through several empirical observations.

(+) Experimental results demonstrate the empirical effectiveness of TempoQuant.

Critical weaknesses

[Novelty]
(-) The progressive quantized-prefix calibration in Eq. (9) appears closely related to existing block-wise reconstruction PTQ methods, such as QDrop, which already calibrate blocks using quantized intermediate representations. It is also conceptually related to prior diffusion PTQ methods that explicitly account for accumulated quantization errors, e.g., [A, B]. The paper should more clearly explain what is specifically new in this calibration strategy beyond adapting sequential reconstruction to MDLM trajectories.

(-) The rotation matrix introduced in Sec. 3.1. is already introduced in prior works (e.g., [C, D]), so I think a related reference should be cited in Sec. 3.1. Since this component does not appear to be specific to TempoQuant, the paper should cite and clearly distinguish it from these prior works.

[Validity]
(-) The connection between the proposed vulnerability measure and the adaptation rule is not sufficiently justified. TempoQuant assigns weaker regularization on δl,j\delta_{l,j} to channels with large Rl,jR_{l,j}, allowing their equivalent scales to deviate more from the SmoothQuant-style initialization. However, high target-bit distortion does not necessarily imply that a larger deviation from the initial equivalent scale is desirable. I think an analysis about the relationship between Rl,jR_{l,j} and the learned ∣δl,j∣|\delta_{l,j}| would strengthen this design choice.

[Related works]
(-) The paper does not provide a discussion of prior works related to the main components of TempoQuant.

[Experiments]
(-) The experiments are conducted only on LLaDA-8B-Instruct. Evaluation on additional masked diffusion language models would be helpful to demonstrate the generality of TempoQuant.

(-) The paper does not provide an ablation study for the individual loss terms in Eq. (10).

[A] AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models, NeurIPS 2025
[B] Error Propagation Mechanisms and Compensation Strategies for Quantized Diffusion Models, ICML 2026
[C] SpinQuant: LLM quantization with learned rotations, ICLR 2025
[D] Block Rotation is All You Need for MXFP4 Quantization, ICML 2026

Questions for authors

Please refer to the weaknesses. I will adjust my rating if the authors adequately resolve my concerns.

4 comments
AuthorsOpenReview ↗

Clarification of Novelty, Vulnerability-Guided Adaptation, and Related Work

W1–W2. Novelty and Relationship to Prior PTQ Methods

TempoQuant's contribution lies in target-bit vulnerability-guided calibration across MDLM masking states, rather than rotation or progressive reconstruction individually. The fixed block-Hadamard rotation and quantized-prefix reconstruction build on established PTQ techniques, including SpinQuant, Block Rotation, QDrop, AccuQuant, and diffusion error-compensation methods.

The key distinction is how MDLM-specific quantization vulnerability is measured and used. TempoQuant directly measures channel-wise computation distortion under joint target-bit weight–activation quantization across masking ratios and masked/unmasked token groups (Eqs. 4–5). Unlike activation-statistics-based indicators, this measure captures the actual target-bit computation discrepancy. On held-out trajectories, its Spearman correlation with deployment distortion reaches 0.7774 (W8A8) and 0.7506 (W4A4), versus 0.6335 and 0.6152 for temporal variation.

This vulnerability guides equivalent-scale and clipping regularization (Eqs. 6–8), while progressive calibration reconstructs full-precision block outputs from quantized-prefix inputs (Eq. 9). The latter addresses upstream error propagation; Figure 5 reports representation-gap reductions of 26.1% and 24.0% at masking ratios 0.3 and 0.5.

Together, these mechanisms incorporate heterogeneous masking-state information into a single static quantization configuration, without timestep-dependent parameter selection at inference. We will clarify that the novelty lies in the MDLM-specific vulnerability formulation and its integration into calibration, not in rotation or sequential reconstruction alone.

W3. Validity of Vulnerability-Guided Adaptation

Vulnerability guidance relaxes the constraint on scale adaptation without prescribing the direction or magnitude of the update. In Eqs. (6–7), higher vulnerability reduces the regularization weight on the scale residual, while reconstruction determines the learned adjustment. Thus, high vulnerability does not imply that a larger deviation from initialization is necessarily optimal.

We examined vulnerability, learned scale deviation, and relative distortion reduction on LLaDA-8B-Instruct under W4A4.

ConfigurationSpearman: R vs. abs(delta)Spearman: R vs. distortion reduction
TempoQuant (Full)−0.0070.128
Uniform scale reg.0.0430.134
Uniform scale reg. (matched)0.0270.157
Uniform reg. (scale + clip)0.0010.160
Uniform reg. (scale + clip, matched)0.0160.142

The results distinguish vulnerability estimation from the adaptation it permits. Learned scale deviations show negligible monotonic correlation with vulnerability, consistent with reconstruction-driven rather than prescribed adjustments. Relative distortion reduction generally increases from low- to high-vulnerability deciles, but similar trends persist under uniform regularization. This supports an association between vulnerability and channel-level distortion response without attributing the trend solely to guided weighting.

To assess its downstream contribution, we compared TempoQuant with a strength-matched uniform-regularization control, which preserves aggregate regularization strength while removing vulnerability-dependent weights. TempoQuant achieves 50.88% versus 50.49% average accuracy, providing a modest empirical advantage for guided weighting within the complete framework.

We will incorporate these analyses and clarify that vulnerability determines adaptation freedom, while reconstruction determines the learned scale adjustments.

W4. Related Work

We will expand the Related Work discussion to explicitly position TempoQuant against rotation-based PTQ, block-wise reconstruction, diffusion error compensation, and existing MDLM quantization methods, distinguishing established techniques from our target-bit vulnerability formulation and masking-state-aware calibration.

AuthorsOpenReview ↗

Generalization Across MDLMs and Component-wise Ablation Analysis

W5. Generalization to Additional MDLMs

TempoQuant achieves the highest average accuracy among seven W4A4 quantization methods on Dream-v0-Instruct-7B, extending its evaluation beyond LLaDA-8B-Instruct. We evaluated seven benchmarks against six PTQ baselines under the same evaluation protocol.

MethodBitsTruthfulQAARC-CHellaSwagWinoGrandePIQAMMLUC-EVALAverage
RTNW4A453.1225.2624.2849.4949.2926.2423.8535.93
AWQW4A454.8323.3825.3951.2250.8225.7223.7036.44
QuaRotW4A446.3924.4024.5748.6249.8925.0625.8534.97
STaR-QuantW4A448.5923.2924.6449.7251.1426.3125.2635.56
DLLMQuant+W4A454.1023.8923.8750.4350.0526.4325.8536.37
DLLMQuant++W4A450.1822.1825.0249.0149.1825.5724.7435.13
TempoQuantW4A442.8436.7729.6750.5153.8631.5226.9738.88

TempoQuant outperforms all six baselines in seven-task average accuracy, improving by 3.75 percentage points over DLLMQuant++ and 2.44 points over AWQ, the strongest competing method. It achieves the best performance on five benchmarks (ARC-C, HellaSwag, PIQA, MMLU, and C-EVAL), although improvements are not uniform across tasks.

These results provide evidence that TempoQuant's masking-aware calibration framework is applicable beyond LLaDA under aggressive W4A4 quantization. The framework estimates target-bit vulnerability for each model and consolidates calibrated equivalent scales and clipping ratios into fixed inference parameters, without requiring timestep-dependent quantizers.

W6. Component-wise and Loss-Term Ablations

Additional W4A4 ablations identify rotation as the dominant contributor to average accuracy, while quantifying the smaller effects of vulnerability-guided adaptation and the CVaR objective. All experiments use LLaDA-8B-Instruct.

ConfigurationAverage Accuracy (%)
w/o Rotation36.52
w/o V-guided Clipping50.56
Uniform Regularization (matched)50.49
w/o CVaR51.15
TempoQuant (Full)50.88

Removing rotation reduces average accuracy by 14.36 percentage points, confirming the importance of the quantization-friendly representation under W4A4. Removing vulnerability-guided clipping reduces accuracy by 0.32 points, while replacing vulnerability-dependent regularization with strength-matched uniform weights reduces it by 0.39 points.

CVaR targets a different aspect of calibration: high-error reconstruction states rather than average downstream benchmark accuracy. Removing CVaR yields 51.15% versus 50.88% for the full model. Thus, the current results do not establish an independent average-accuracy benefit from CVaR; its intended role is to complement mean reconstruction by explicitly penalizing the upper tail of calibration errors.

Eq. (10) combines mean reconstruction, CVaR, scale regularization, and clipping regularization. The additional experiments assess the CVaR term and vulnerability-dependent regularization, alongside the rotation backbone, rather than independently removing every loss term. We will include these results and clarify the distinct optimization roles and observed accuracy contributions of the components.

Revisions. We will clarify TempoQuant's methodological novelty relative to prior PTQ methods (W1–W2), expand the analysis of vulnerability-guided adaptation and its comparison with uniform regularization (W3), and strengthen the Related Work discussion (W4).

We will also incorporate the additional Dream results (W5) and component-wise ablations (W6), clarifying the distinct roles and empirical contributions of rotation, vulnerability-guided regularization, and CVaR. Throughout the manuscript, we will refine the exposition to improve methodological clarity and readability.

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns, which adequately address my concerns regarding W2, W4, and W5.

However, my main concerns remain unresolved. In particular, the additional ablation results show that Rotation-only (52.06%) outperforms the full TempoQuant framework (50.88%), while removing CVaR further improves accuracy to 51.15%. Moreover, vulnerability-guided regularization provides only a modest improvement (+0.39%p) over uniform regularization.

As the authors acknowledge that rotation and progressive quantized-prefix calibration build upon existing PTQ techniques, the novelty of TempoQuant primarily lies in its vulnerability-guided adaptation. However, the additional experiments do not sufficiently demonstrate the effectiveness of this component, suggesting that the overall methodological contribution is rather limited.

Although the authors have clarified the role of vulnerability in controlling adaptation freedom, the additional analyses do not sufficiently demonstrate its efficiency. Therefore, I will maintain my original rating.

We thank the reviewer for their careful reassessment and for acknowledging that our additional experiments address W2, W4, and W5.

We recognize the remaining concern regarding the contribution of vulnerability-guided adaptation. While its accuracy gains are modest and not consistent across all configurations, our intended contribution is a masking-state-aware calibration framework that explicitly incorporates quantization vulnerability across denoising states. We will clarify its distinction from existing PTQ techniques, transparently report the limitations, and moderate our claims accordingly. We appreciate the reviewer's constructive feedback throughout the discussion.

Reviewer_avU5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TempoQuant proposes post-training weight–activation quantization for masked diffusion language models.

It measures per-channel computation distortion under joint target-bit quantization across masking ratios and masked/unmasked token groups, aggregates these measurements using an upper quantile, and uses the resulting vulnerability to relax regularization on learned equivalent scales and activation clipping.

Blocks are calibrated sequentially on quantized-prefix inputs against full-precision targets, with mean and tail reconstruction losses. The learned transformations and clipping ratios are state-independent at inference.

Critical strengths
  1. A concrete diagnostic tied to the target quantization setting. Measuring the joint weight–activation contribution error is more directly relevant than activation variation alone. Paper reports W4A4 Spearman correlation of 0.7506 versus 0.6152 for temporal variation, supporting the usefulness of this diagnostic.
  2. A substantial W4A4 improvement with state-independent learned parameters. The 5.75 point gain over DLLMQuant++ is meaningful within the shared evaluation protocol. Consolidating multi-state calibration into fixed transformations and clipping ratios avoids selecting state-specific parameter sets during inference.
Critical weaknesses

The contribution to mitigating temporal quantization errors remains unclear. The fixed parameters provide a shared compromise across states, but the paper does not sufficiently establish that vulnerability-guided adaptation mitigates temporal quantization errors better than ordinary multi-state calibration. A matched comparison with uniform regularization, together with held-out errors across masking ratios, is needed to isolate this contribution.

Questions for authors

Handling timestep-dependent activation shifts: Activation distributions change across masking ratios, while TempoQuant uses one fixed set of scaling and clipping parameters. Different states may favor conflicting parameter choices. How does the method handle this conflict? Specifically, how does aggregating state-wise distortion into a single channel vulnerability score and relaxing its regularization mitigate temporal-shift-induced errors? Does the method improve difficult states, or trade improvements at some masking ratios for degradation at others?

3 comments
AuthorsOpenReview ↗

Vulnerability-Guided Calibration and Masking-State Trade-offs

We thank the reviewer for recognizing the value of the target-bit vulnerability diagnostic and the substantial W4A4 improvement with state-independent parameters. We agree that the key question is whether vulnerability-guided regularization provides an additional benefit beyond shared multi-state calibration. We conducted additional comparisons to examine this contribution and clarify how the method handles competing masking-state requirements.

W1. Contribution of Vulnerability-Guided Calibration

Our additional W4A4 evaluation on LLaDA-8B-Instruct shows improvements over the strength-matched uniform-regularization control in both held-out reconstruction statistics and downstream performance. On the same 1,408 held-out cases, TempoQuant reduces mean final hidden-state reconstruction error by 0.86%, p95 by 2.08%, and p99 by 1.14%. Its average score across seven downstream benchmarks also increases from 50.49% to 50.88%, a gain of 0.39 percentage points.

ConfigurationMean E_last ↓p95 ↓p99 ↓
TempoQuant (Full)0.27800.42820.5395
Uniform reg. (matched)0.28040.43730.5457
Uniform reg.0.27810.42790.5391
Uniform scale reg.0.27690.42750.5400
Uniform clip reg.0.27420.42020.5204

Here, E_last denotes final hidden-state reconstruction error. The strength-matched control isolates vulnerability-dependent regularization. Improvements extend to upper-tail errors, not just the mean. Other uniform variants yield mixed results, including lower errors under uniform clip regularization. We therefore interpret the findings as a modest benefit over the matched control, supported by downstream accuracy, not superiority over all uniform settings.

We distinguish this adaptation benefit from the diagnostic's predictive validity. Table 3 shows that target-bit vulnerability predicts held-out distortion better than temporal activation variation. The additional comparisons test whether using this information improves calibration.

W2. Handling Conflicting Masking-State Requirements

TempoQuant learns a shared compromise through reconstruction optimization, with vulnerability controlling how strongly parameters are constrained near initialization.

Specifically, state-wise joint weight–activation distortion is aggregated using an upper quantile (Eqs. 4–5). This allows substantial distortion in the upper tail of the sampled states to influence a channel’s vulnerability score, even when its distortion is smaller in other states. The resulting scale-regularization coefficient is

1−gl,j=κRl,j+κ.1-g_{l,j}=\frac{\kappa}{R_{l,j}+\kappa}.

Thus, a channel with higher measured vulnerability receives a weaker constraint on its equivalent-scale update. Activation clipping is regularized analogously using layer-average vulnerability. Crucially, vulnerability determines the strength of the constraint; it does not prescribe a state-specific update direction.

The shared reconstruction objective determines how scales and clipping ratios change across sampled states. Its mean term captures overall reconstruction quality, while CVaR emphasizes high-error cases. Thus, vulnerability determines where adaptation is less constrained, while reconstruction determines the compromise among states. All learned transformations and clipping ratios remain fixed at inference.

To examine generalization beyond calibration masking ratios, we evaluated held-out reconstruction separately on seen and unseen ratios:

ConfigurationSeen E_last ↓Unseen E_last ↓
Rotation-only0.23320.2292
TempoQuant (Full)0.27810.2779
Uniform reg. (matched)0.28330.2779

Compared with matched uniform regularization, TempoQuant achieves lower average error on seen ratios and the same reported average on unseen ratios. Thus, the seen-ratio improvement does not increase the unseen-group average. However, these averages do not imply improvement at every masking ratio; per-ratio comparisons reveal state-dependent trade-offs. Rotation-only also achieves lower absolute errors, so we do not claim reconstruction superiority over that reference.

TempoQuant therefore accommodates conflicting state requirements through shared optimization, with vulnerability-dependent regularization providing sensitivity-informed constraints. The matched comparison shows improvements in mean and upper-tail held-out errors alongside a downstream gain. We do not claim that every masking ratio benefits and will clarify this limitation.

Revisions. We will add the exact definition of the strength-matched control and all uniform variants, the held-out evaluation protocol, and masking-ratio-resolved results. We will also clarify the distinct roles of vulnerability estimation, vulnerability-guided regularization, and shared mean–CVaR reconstruction in Secs. 3–4.

Thank you for the detailed rebuttal and additional experiments. I appreciate the clarification regarding vulnerability-guided calibration and masking-state trade-offs. The new results help address my concerns, and I appreciate the authors' efforts in addressing my comments.

We sincerely thank the reviewer for their thoughtful reassessment and positive feedback. We appreciate their recognition of our additional experiments and clarifications, as well as their constructive comments throughout the discussion.

Reviewer_gtWaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose TempoQuant, a post-training quantization method for masked diffusion language models. Specifically, the method measures channel-wise computation distortion across masked calibration states, uses it to guide scale and clipping adaptation, and calibrates each Transformer block on quantized-prefix inputs. On LLaDA-8B-Instruct, the W4A4 seven-task average is 50.88%, compared with 45.13% for DLLMQuant++ [3]; the W8A8 gain is only 0.14 percentage points.The W4A4 result is promising. Nevertheless, the lack of component ablations and the limited evaluation make the contributions of the individual designs and the practical inference benefit unclear.

Critical strengths
  1. The authors identify three concrete quantization difficulties in MDLMs and support them with empirical analyses.
  2. The proposed vulnerability measure is more predictive than temporal variation, and quantized-prefix calibration reduces representation drift.
  3. The method obtains a clear W4A4 improvement over the listed baselines [3,4] while using static quantization parameters at inference.
Critical weaknesses
  1. More ablation studies are needed to clarify what the method adds beyond the scale and rotation backbone [1,2]. What are the separate effects of the rotation, vulnerability-guided scale and clipping, CVaR, and progressive calibration on final scores? Table 3 and Fig. 5 do not answer this question.
  2. The method is tested only on LLaDA-8B-Instruct. What would happen on another masked diffusion language model? The generality of the findings is unclear.
  3. It is hard to assess the claimed efficiency benefit from Table 2. Quantization is simulated and matrix multiplications still use BF16 kernels. Please report low-bit latency, throughput, memory, and model size, including transformation overhead.
  4. How does the method perform in actual iterative generation? All seven benchmarks use conditional scores for masked answer candidates. A generation benchmark would better test error propagation during denoising.
  5. The calibration states are 20 independently masked examples rather than generation rollouts. Would the vulnerability estimate remain effective on states produced by real denoising? Please provide a matched comparison.
  6. The W8A8 improvement over DLLMQuant++ is only 0.14 percentage points, whereas the W4A4 gain is 5.75 points. Please report variability across calibration seeds and qualify the W8A8 conclusion.
  7. How much calibration time does the method require? Although it uses four source sequences, it optimizes every block for 80 steps. Please report GPU time and forward-pass or token counts alongside the baselines. References: [1] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, ICML, 2023. [2] QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs, NeurIPS, 2024. [3] DLLMQuant: Quantizing Diffusion-Based Large Language Models, arXiv, 2025. [4] STaR-Quant: State-Time Consistent Post-Training Quantization for Diffusion Large Language Models, arXiv, 2026.
Questions for authors

Please refer to the weakness part.

2 comments
AuthorsOpenReview ↗

Component Contributions and Cross-Model Generalization

We thank the reviewer for the constructive feedback. We conducted additional component ablations, cross-model evaluations on Dream-v0-Instruct-7B, inference and calibration efficiency measurements, iterative generation experiments, and analyses of masking representativeness and calibration stability to comprehensively address the reviewer's concerns.

W1. Component Ablation and Methodological Contribution

TempoQuant's key contribution is a masking-state-aware calibration framework that explicitly incorporates quantization vulnerability across MDLM denoising stages. To examine the contributions of its individual components beyond the rotation backbone, we conducted component-wise ablations on LLaDA-8B-Instruct under W4A4 across seven benchmarks (calibration seed 42). TempoQuant measures target-bit distortion across masking ratios and masked/unmasked token states to guide scale and clipping adaptation, while progressive reconstruction accounts for quantized-prefix errors. All quantization parameters remain fixed at inference.

The ablation results distinguish the contribution of rotation from the effects of vulnerability-guided calibration.

ConfigurationAvg. (%)Δ vs. Full (pp)
w/o Rotation36.52−14.36
Rotation-only52.06+1.18
w/o V-guided Clipping50.56−0.32
Uniform Regularization (matched)50.49−0.39
w/o CVaR51.15+0.27
TempoQuant (Full)50.88—

Removing rotation reduces accuracy by 14.36 pp, confirming its importance as the backbone. Within the full framework, vulnerability-guided regularization improves accuracy by 0.39 pp over strength-matched uniform weighting, while guided clipping contributes 0.32 pp. Removing CVaR improves average accuracy by 0.27 pp.

While rotation-only achieves a higher average accuracy than the full framework, the ablations indicate that the additional calibration components have varying effects on downstream performance. These findings distinguish the contribution of the rotation backbone from that of masking-state-aware calibration.

Importantly, the proposed vulnerability measure correlates more strongly with held-out target-bit distortion than temporal variation (Spearman 0.7506 vs. 0.6152 under W4A4; 0.7774 vs. 0.6335 under W8A8). Quantized-prefix reconstruction further reduces representation gaps by 26.1% and 24.0% at masking ratios 0.3 and 0.5, respectively. These findings provide complementary evidence for the proposed vulnerability estimation and calibration mechanisms.

W2. Generalization to Dream-v0-Instruct-7B

TempoQuant achieves the highest seven-task average of 38.88% on Dream-v0-Instruct-7B under W4A4, outperforming all six PTQ baselines and extending evaluation beyond LLaDA.

MethodBitsTruthfulQAARC-CHellaSwagWinoGrandePIQAMMLUC-EVALAvg.
RTNW4A453.1225.2624.2849.4949.2926.2423.8535.93
AWQW4A454.8323.3825.3951.2250.8225.7223.7036.44
QuaRotW4A446.3924.4024.5748.6249.8925.0625.8534.97
STaR-QuantW4A448.5923.2924.6449.7251.1426.3125.2635.56
DLLMQuant+W4A454.1023.8923.8750.4350.0526.4325.8536.37
DLLMQuant++W4A450.1822.1825.0249.0149.1825.5724.7435.13
TempoQuantW4A442.8436.7729.6750.5153.8631.5226.9738.88

TempoQuant outperforms DLLMQuant++ by 3.75 pp and the strongest competing baseline, AWQ, by 2.44 pp in average accuracy. It achieves the best results on five of seven tasks, with particularly clear gains on ARC-C, HellaSwag, and MMLU. Although performance varies across individual benchmarks, these results demonstrate that TempoQuant's W4A4 improvements extend to a second MDLM backbone and remain competitive against both conventional and diffusion-specific PTQ methods.

Revision. We will incorporate the additional ablations and Dream results, and clarify the distinct methodological roles and empirical contributions of TempoQuant's calibration components.

AuthorsOpenReview ↗

Inference Efficiency, Masking Representativeness, and Robustness

W3 & W7. Inference and Calibration Efficiency

TempoQuant performs vulnerability estimation and progressive calibration offline, while keeping quantization parameters fixed throughout denoising. No timestep-dependent parameter selection is required during inference.

We measured inference performance on an NVIDIA A100 80GB PCIe (batch size 8, sequence length 256), using fake quantization with BF16 matrix multiplications.

ConfigurationLatency (ms)Tokens/sPeak Memory (GB)
FP188.9410,839.316.58
Rotation-only492.324,159.917.04
TempoQuant508.034,031.217.04
DLLMQuant++631.403,243.617.57

TempoQuant incurs 3.2% additional latency over rotation-only, with identical peak memory. Compared with DLLMQuant++, it achieves 19.5% lower latency and 24.3% higher throughput. Estimated weight storage decreases from 16.03 GB (W16) to 8.54 GB (W8) and 4.79 GB (W4), retaining non-linear parameters in BF16.

Offline calibration uses four source sequences and 80 optimization steps per Transformer block. Two runs required 792.6 and 901.4 seconds (14.1 minutes on average), with 38.3 GB peak GPU memory.

W4. Iterative Generation Evaluation

We additionally evaluated LLaDA-8B-Instruct on the full GSM8K evaluation set using iterative answer generation, complementing the conditional candidate scoring in the original benchmarks.

MethodPrecisionGSM8K Accuracy (%)
FP16FP1638.21
DLLMQuant++W4A44.70
TempoQuantW4A424.79

TempoQuant improves generation accuracy by 20.09 pp over DLLMQuant++, retaining 64.9% of FP16 accuracy compared with 12.3% for DLLMQuant++. This demonstrates that the improvement extends to iterative mathematical reasoning, where quantization errors can propagate across denoising steps.

W5. Representativeness of Independent Masking

To assess whether independently sampled masks capture vulnerability patterns in actual denoising states, we compared target-bit vulnerability against FP LLaDA rollouts with matched source sequences, masking ratios, sample counts, and quantization settings.

ComparisonPooled SpearmanWithin-layer Spearman
Independent re-draw0.7620.424
Prefix-conditioned rollout0.7220.370
All-mask rollout0.5000.198

Independent masking captures much of the vulnerability structure observed in prefix-conditioned rollouts. Pooled correlation reaches 0.722, close to the independent re-draw reference of 0.762. Within-layer correlation is lower (0.370 vs. 0.424), indicating greater variation in channel rankings. All-mask rollouts show weaker agreement, highlighting differences across generation-state distributions.

These results support independent masking as a practical approximation of vulnerability structure under prefix-conditioned denoising.

W6. Precision-Dependent Performance and Calibration Stability

We extended the precision comparison to two MDLM backbones.

ModelBitsFPDLLMQuant++TempoQuant
LLaDA-8BW8A853.2953.2853.42
LLaDA-8BW4A453.2945.1350.88
Dream-7BW8A856.7851.8856.88
Dream-7BW4A456.7835.1338.88

Under W4A4, TempoQuant improves average accuracy by 5.75 pp on LLaDA and 3.75 pp on Dream over DLLMQuant++. Under W8A8, LLaDA performance is already close to FP accuracy, while Dream shows a 5.00 pp improvement.

We also evaluated the submitted SmoothQuant-initialized LLaDA W4A4 configuration across three calibration seeds on 1,408 held-out cases.

MetricMean ± StdCV (%)
E_last ↓0.2829 ± 0.00431.5
E_avg ↓0.1190 ± 0.00090.8
Logit KL ↓0.4303 ± 0.00220.5
Argmax agreement ↑0.6141 ± 0.00611.0
E_last p99 ↓0.5920 ± 0.04547.7

Average reconstruction metrics show low calibration-seed variability (CV 0.5–1.5%), while tail reconstruction error varies more (CV 7.7%). These results demonstrate consistent average reconstruction behavior across calibration seeds under W4A4.

Revision. We will incorporate the inference and calibration measurements, GSM8K generation results, matched rollout comparisons, and calibration-seed analysis, clarifying the corresponding evaluation settings and precision-dependent findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.