acceptodds
Under review as a conference paper at ICLR 2027

Mask-On: Compact Activation-Calibrated Merging for Autoregressive and Diffusion Language Models

Abstract

Autoregressive (AR) and diffusion language models offer complementary generation behaviors, but retaining both pretrained endpoints requires storing two full parameter sets. We propose Mask-On, a gradient-free method that represents both modes using an unchanged AR backbone and a compact AR-to-diffusion update. Building on support/sign encoding, Mask-On compresses AR-to-diffusion weight changes into ternary masks and output-channel scales, calibrated by matching the diffusion endpoint's linear-layer outputs on collected activations. Closed-form scale fitting reduces the search to a single pruning parameter without gradient-based optimization or downstream task-specific tuning. Across four AR–diffusion model pairs and four benchmarks spanning diverse domains, we demonstrate that direct parameter merging does not consistently preserve both generation modes, whereas compact mode-specific updates support more consistent diffusion retention alongside an unchanged AR checkpoint. Mask-On achieves the highest average diffusion-mode accuracy among the compared merging methods on all four models, with an update substantially smaller than a full diffusion checkpoint. Experiments on Fast dLLM v2 show similar quality–throughput trend to the diffusion endpoint across six decoding configurations. A simple training-free experiment using the original AR checkpoint further improves diffusion-draft accuracy by up to 3.26 percentage points on GSM8K, illustrating how access to both modes can support complementary generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Model, Model Merging

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xSkZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MASK-ON, a compact model merging method that enables both AR and diffusion generation while keeping the AR model's parameters unchanged. Experiments on four AR-diffusion model pairs and four benchmarks show that MASK-ON achieves the highest average accuracy among the compared merging methods, supporting both generation modes with modest additional storage.

Critical strengths
  • Extensive experimental evaluation.

    • The method is evaluated against various model-merging baselines across multiple models and benchmarks.
  • Analysis of the proposed method.

    • Ablation studies examine the impact of individual components on performance.
Critical weaknesses
  • Limited novelty.

    • Although MASK-ON introduces modifications to scale fitting and reconstruction-error-based pruning selection, its core design builds on existing methods. The extent to which these extensions constitute novel methodological contributions remains unclear.
  • Insufficient analysis of the relationship between the problem and method design.

    • The paper does not sufficiently analyze which AR-to-diffusion-specific challenges motivate each proposed modification. Moreover, comparisons with T-Switch are not conducted under equal storage budgets, leaving the performance--storage trade-off insufficiently examined.
Questions for authors

Please clarify MASK-ON's novelty over existing methods and how its design addresses the challenges of the target problem.

  1. What specific limitations does T-Switch have in this setting, and how does each proposed modification address them? What are the distinct methodological contributions of MASK-ON over existing methods?

  2. What challenges are specific to the AR-to-diffusion setting, and why are the proposed design choices necessary to address them?

Reviewer_wUDXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes a method for compressing AR-to-diffusion parameter updates. By using diffusion activations for per-output-channel calibration, it preserves the original AR model and recovers diffusion capabilities with relatively low additional storage overhead.

Critical strengths

The proposed method achieves good recovery of diffusion capabilities on some models.

Critical weaknesses

There is a clear gap between the paper's motivation and what it actually achieves. The motivation is to retain both AR and diffusion generation modes at low cost, allowing them to be selected according to task requirements or even used together. However, MASK-ON only reduces the persistent storage required to maintain two checkpoints. In practice, the current implementation still requires loading two sets of dense weights for inference. Therefore, compared with directly loading the original AR and diffusion models, the paper does not demonstrate any resource savings during inference, while introducing potential performance degradation due to compression. For example, the average accuracy of LLaDA2.0-mini drops by 14.06 percentage points compared with the original diffusion model.

Questions for authors

see Weaknesses.

Reviewer_UywZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TL;DR: This is a model fusion work considering the special case of adapting an autoregressive (AR) language model to become a diffusion model by adding a highly-compressible "task vector" to its parameters.


Consider an AR language model with weights θA\theta_A. Starting from those weights, there exist techniques to transform this AR model into a diffusion model parameterized by the weights θD\theta_D. This diffusion model has the same transformer architecture, but uses different attention mask and inference strategy. This adaptation process amounts to learning τ=θD−θA\tau = \theta_D - \theta_A.

The present manuscript starts with θA\theta_A and τ\tau already in hands; its goal is to obtain a τ^\hat{\tau} that is much more compressible than τ\tau, while maintaining good performances for a diffusion model parametrized by θ^D=θA+τ^\hat{\theta}_D = \theta_A + \hat{\tau}.

To this end, the manuscript introduces Mask-On, a variation on the pre-existing T-Switch method. No gradient-based optimization is involved, but some "training data" is required to get diffusion model activations necessary to identify parameters α\alpha and λℓ\lambda_\ell.

Empirical evaluations are favorable to Mask-On when compared to other model fusion techniques. For some benchmarks and metrics, θ^D\hat{\theta}_D even performs better than θD\theta_D However, (though it is worse on others).

Critical strengths

The Mask-On approach does well at its stated objective of finding a compressible τ^\hat{\tau} (14.71–25.48% of the original θA\theta_A, not counting uncompressed token embeddings) such that the resulting θ^D=θA+τ^\hat{\theta}_D = \theta_A + \hat{\tau} has good performances.

Critical weaknesses

The stated objective is not well motivated. I can think of three different directions (see Q1-Q3 below) which, if properly investigated and reporting promising results, could have motivated the present work. But in its current state, this is a solution without a problem.

Easier to fix is a clarity issue in Section 2: where do Mask-On departs from T-Switch? For a reader that isn't intimately familiar with T-Switch, this information is absent from the prose of Section 2. Yes, the information is eventually presented in Table 4, but it appears on page 8 of the manuscript. A forward reference to Table 4 in Section 2 would greatly improve the clarity, but I would welcome a solution more integrated in the prose of Section 2.

Questions for authors

Q1. Does Mask-On plays well with quantization?

Despite its core topic being compression, the manuscript's only mention of quantization concerns the AR vLLM pipeline not using it. This opens the door to a possibly naive argument against Mask-On.

That argument goes as follows. Common wisdom is that 8-bits quantization is basically free; going to 4-bits has small cost on simple tasks but can significantly hurt on complex/long ones. So, taking a safe stance, one could hope to use 8-bits quantization to simultaneously hold both the AR and diffusion models in GPU memory, using the same footprint as the original 16-bits AR model. This excludes embeddings and lm_head, which should be kept in BF16, but so does Mask-On: "The diffusion endpoint’s token embeddings and output head are retained separately without compression." Therefore, 8-bits quantization requires 0% extra where Mask-On needs 14.71–25.48%. Plus the quantized solution holds both model in memory simultaneously, whereas Mask-On needs to swap.

Now I did say that this argument could be naive: I'm no expert in quantizing diffusion models, and perhaps the authors could construct a convincing argument against it. My point is that there is absolutely no discussions of such concerns in the manuscript.

(The only quantized version I found for the author's investigated models is https://huggingface.co/zimengxiong/WeDLM-8B-Instruct-MLX-8bit , which comes with the comment "It currently does not work too well or provide meaningfull speedup due to lack of pre compilation." Accuracy benchmarks on a single model, without TPS, would already be much better than nothing.)

Another possible rebuttal is that one could apply Mask-On on top of a quantized AR model, for further compression. True! How well does Mask-On perform on quantized AR models?

Q2. How long does it takes to swap between Mask-On and AR?

The draft-and-review narrative reports TPS using "total generated tokens divided by the sum of separately measured draft and review generation times, excluding mode switching and loading." How long does it takes to switch mode?

Another possible rebuttal is that perhaps no swap is needed, one could keep Mask-On weights to do AR inference. How good is the merged model on AR generation and/or verification?

Q3. Are Mask-On KV caches compatible with AR's?

(The answer may simply be "no".)

Another potential way out for the authors is that perhaps their fusion method naturally maintains KV cache compatibility between the two models, which could save forward passes on long context. And perhaps it does not naturally does it, but minor changes could be made to promote that behavior (future work territory).

Q4. Are there any interesting observations on the structure of SℓS_\ell?

(Again, the answer may simply be "no".)

For example, do similar-roles ℓ\ell from different layer have look-alike SℓS_\ell?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks

We sincerely thank the Area Chair and all reviewers for their careful and constructive feedback on our submission, our method for retaining autoregressive (AR) and diffusion generation with an unchanged AR checkpoint and a compact AR-to-diffusion update.

We are grateful that the reviews recognized strengths of our work, including:

  • Good recovery of diffusion capability from a compact update (Reviewers UywZ, wUDX),
  • An extensive evaluation against model-merging baselines across multiple models and benchmarks (Reviewer xSkZ),
  • Ablation studies that examine the effect of each component (Reviewer xSkZ).

We addressed the reviewers’ main concerns with new experiments and revisions to the manuscript:

  • Motivation and inference cost (Reviewer UywZ W1; Reviewer wUDX W1, W2): We clarified that the work asks whether the AR-to-diffusion update can be represented compactly without losing diffusion performance or altering the AR checkpoint. The compact update also lowers inference cost. It is small enough to stay on the GPU next to a single backbone, so the two modes no longer need two resident models and the freed memory can serve more requests (32 per batch instead of 8 under a 32 GiB cap; Section 5.3). During the discussion, Reviewer UywZ wrote that they see value and novelty in the work as an exploration of the design space for diffusion model inference and are considering raising their score. We agree that it is not yet a production-ready system and leaves room for further optimization.
  • Relationship to quantization (Reviewer UywZ Q1): We show that Mask-On complements quantization rather than competing with it. INT8 quantization lowers diffusion accuracy much more than AR accuracy, as the follow-up results for each INT8 endpoint show, whereas Mask-On keeps both modes in BF16. Applied to an INT8-stored anchor, Mask-On yields a bundle smaller than two INT8 endpoints that is still more accurate in diffusion (Section 5.2 and Appendix F.2).
  • Switching cost, review without switching, and KV reuse (Reviewer UywZ Q2, Q3): Switching modes is cheap. Mask-On restores the exact AR weights in about 22 ms, more than ten times faster than copying full weights from CPU, so one backbone can draft in diffusion mode and review with the original AR model. Review without a switch also works but is less reliable, and KV-cache reuse across modes does not work yet and is left to future work (Section 5.3 and Appendix G.5).
  • Contribution and comparison with T-Switch (Reviewer UywZ W2; Reviewer xSkZ W1, W2, Q1, Q2): We clarified that Mask-On inherits T-Switch’s support-and-sign encoding and does not claim a new compression primitive. As we discussed further with Reviewer xSkZ, its contribution is to show that an AR model and its diffusion counterpart can be kept as one unchanged AR checkpoint plus a compact update, and how to fit and select that update from calibration data alone. To examine the size of the gains more closely, we evaluated four additional benchmarks during the discussion with nothing tuned. The results show that T-Switch is sensitive to its pruning level (up to 24.00 points below the diffusion endpoint on SDAR at its default level), whereas Mask-On stays close to the endpoint on the dense models with one level selected from calibration data. Although the gain is modest once T-Switch is given a suitable level, Mask-On still scores higher in most comparisons (Sections 2, 3.2, and 5.1, and Appendices E.3 and E.5).
  • Accuracy drop on LLaDA2.0-mini (Reviewer wUDX W3): We agree that the 14.06-point drop is substantial and now state it as a limitation, with calibration of the MoE experts as a possible cause (Appendix I). The drop does not appear on the three dense models, where Mask-On recovers the diffusion endpoint’s four-benchmark average without loss.

We also analyzed the structure of the ternary mask (Reviewer UywZ Q4). With fitted scales, even random positions score within about one point of Mask-On, which indicates that the fitted scales, the part that Mask-On adds, carry much of the recovery (Appendix E.4).

In summary, the new results show that one unchanged AR checkpoint and a compact update, fitted from calibration data alone, recover diffusion performance on the dense models and let both modes share one backbone at inference. The reviewers’ follow-up responses indicate that the discussion resolved many of their questions and concerns, and we have responded to the points that remain open. We believe the revised submission now states this contribution more clearly and supports it with direct evidence. We are deeply grateful to the Area Chair and reviewers for the feedback that led to these improvements.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.