acceptodds
Under review as a conference paper at ICLR 2027

Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs

Abstract

Converting a pretrained autoregressive (AR) model to a diffusion language model (dLLM) enables parallel generation without pretraining a new model. Published conversion methods differ by roughly three orders of magnitude in training data and have not been compared under a common protocol. We compare two conversions of the same 30B Mixture-of-Experts (MoE) parent, holding the corpus, supervised-token budget, trainable parameter set and evaluation harness fixed, each under its own training recipe. The in-place model updates a subset of the parent's weights using denoising and representation-alignment losses; the frozen-tower model instead conditions through cross-attention on a frozen causal copy of the parent. With 1 B training tokens, the frozen-tower model scores 71.60 on HumanEval pass@10 against 6.19 for the in-place model, an 11.6× improvement. At the same budget it also keeps 95% of the parent's GSM8K score and 99% of its MMLU-Pro score. A dense-parent experiment reproduces the HumanEval separation. Within the two-tower design at about 500 M tokens, freezing the context tower retains substantially more MMLU-Pro performance than training it, while both give similar observed HumanEval scores. Our theoretical analysis establishes that both conversion classes contain an exact sampler for the AR parent under a hard attention mask and left-to-right commitment of one position per round. Under a shared loss, freezing removes the gradient contribution through the context states. Furthermore, evaluation protocol substantially affects a published 500 B-token conversion's scores in both directions across tasks, while its AR parent's scores vary by less than three points, so comparing dLLMs needs a common protocol. These results show that, in the tested low-budget regime, the frozen-tower configuration retains substantially more of the parent's generation performance than in-place conversion. The construction applies to any causal transformer with softmax attention.

Primary area:
generative models
Keywords:
diffusion language models, autoregressive-to-diffusion conversion, frozen context tower, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8DjfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the low-budget conversion of pretrained autoregressive (AR) language models into diffusion language models (dLLMs), focusing on the trade-off between generation quality and knowledge retention. Under matched training budgets, the authors compare in-place conversion with a frozen context-tower architecture across both MoE and dense backbones. Experiments show that in-place conversion substantially degrades long-form generation, whereas the frozen-tower configuration preserves considerably more of the parent model’s performance.

Critical strengths

The paper addresses an important and practical problem of converting pretrained AR models into diffusion language models under limited training budgets, offering valuable insights into efficient diffusion language modeling.

Critical weaknesses
  1. The frozen context-tower architecture is largely adopted from Nemotron-Labs-TwoTower (Reda et al., 2026). The main contribution lies in empirical comparison and analysis rather than a new conversion method.
  2. Although the main experiments match the training budget and the trainable parameter count, the two configurations differ in several aspects, including architecture, alignment loss, and diffusion corruption procedure. Therefore, it is unclear whether the performance improvements can be attributed solely to freezing the context representations.
  3. The practical inference efficiency of the proposed method appears to be rather limited. Despite supporting parallel decoding, the model achieves only 0.134–0.168× the throughput of the autoregressive baseline at batch size 1.
Questions for authors
  1. Could the authors provide a more controlled ablation that changes only the context architecture while keeping the alignment loss, corruption procedure, and block-size schedule fixed? This would help identify the primary source of the performance gap.
  2. Given that the frozen and trainable context towers achieve similar HumanEval scores, is the main generation improvement attributable to separating context computation from denoising rather than freezing the context representations?
  3. Since the core two-tower architecture follows Reda et al. (2026), could the authors clarify the methodological contributions beyond applying the existing architecture to a lower-budget setting and conducting additional empirical analyses?
  4. The frozen-tower model is substantially slower than the AR baseline despite its parallel decoding capability. What architectural or decoding changes could make the approach competitive in end-to-end inference throughput?
Reviewer_DHmDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies methods for converting pretrained autoregressive (AR) models into diffusion language models (dLLMs) under a limited training budget. In-place conversion updates a subset of the parent model’s parameters. Frozen-tower conversion instead uses a frozen copy of the parent as a context tower and trains selected parameters in a second copy to predict masked tokens through cross-attention to the context tower. The authors report that the frozen-tower configuration retains more of the parent’s generation performance than the tested in-place configuration.

Critical strengths

None.

Critical weaknesses
  • The paper’s central contribution is unclear. The abstract emphasizes a low-budget comparison between frozen-tower and in-place conversion, whereas the introduction lists several additional contributions concerning output length, evaluation protocols, and gradient analysis. The relationship between these analyses and the central research question is not clearly established, making the paper’s main contribution difficult to assess. Moreover, the cited Nemotron-Labs-TwoTower study (Reda et al., 2026) already compares frozen, separate towers against tied-weight towers trained with a joint AR–diffusion objective. Although this differs from the present in-place baseline, the paper should clarify what additional insight its comparison provides.

  • The experimental descriptions are difficult to follow. For example, the first paragraph of Section 2 introduces four conversion recipes using compressed labels such as “frozen experts with layer-wise representation alignment,” without clearly defining the trainable components, alignment targets, or shared and differing settings, or providing precise references to those details. Readers must reconstruct the configurations from scattered descriptions and cited work before interpreting the results. A self-contained overview of the experimental design, with clear references to implementation details, would make the comparisons easier to assess.

Questions for authors
  1. What is the central new finding beyond the tower-decoupling results in Reda et al. (2026), and how do the output-length, evaluation-protocol, and gradient analyses support it?

  2. Could you provide a consolidated description of the four conversion recipes in Section 2, specifying their trainable parameters, objectives, attention patterns, and corruption procedures, with clear references to the implementation details?

  3. How would you revise the paper to clarify its central contribution and improve the organization and accessibility of the methodological and experimental descriptions?

Reviewer_kvpEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies low-budget AR-to-diffusion conversion, comparing in-place fine-tuning against a two-tower architecture where a frozen AR model provides contextual representations to a trainable diffusion denoiser. At 1B training tokens, the two-tower model substantially outperforms in-place conversion on Qwen3-30B-A3B (71.60 vs. 6.19 HumanEval pass@10), while retaining most of the parent's knowledge performance. The authors further investigate context freezing, decoding protocols, and architectural choices.

Critical strengths
  1. Efficient AR-to-dLLM conversion is relevant, and the performance gap is large enough to be meaningful. The authors also show that they could reproduce the main finding on a smaller dense model, which strengthens the observation.

  2. The paper evaluates multiple conversion recipes, model architectures, and downstream tasks. The analysis of decoding-protocol sensitivity is particularly useful, given how strongly dLLM performance can depend on sampling.

  3. The frozen-versus-trainable tower experiment suggests that architectural separation and knowledge retention may play different roles, which is also interesting to know.

Critical weaknesses
  1. The central two-tower architecture, including the frozen causal context model and layer-aligned conditioning, was already introduced by Reda et al. (nemotron-labs-twotower). The contribution is therefore primarily an empirical comparison under limited training budgets. This could still be valuable, but the paper needs to establish clearer insights beyond the benefits demonstrated by the existing architecture.

  2. The main comparison does not isolate the source of improvement. Despite matching the parent model, data, supervised-token budget, and trainable parameter count, the two configurations differ in several important respects: corruption procedures, block-size schedules, alignment losses, etc. They are not compute-matched either. The results demonstrate that one configuration works substantially better, but not why. This is especially concerning given the paper's own finding that decoding protocols can dramatically change dLLM performance. A matched training-recipe ablation would be much more convincing.

  3. Table 2 shows nearly identical HumanEval performance between frozen and trainable towers (64.04 vs. 63.83), suggesting that freezing itself does not explain the generation advantage. The MMLU-Pro gap is interesting (54.2 vs. 40.8), but training the tower also doubles the trainable parameter count without retuning the optimization settings. The theoretical analysis establishes AR realizability under sequential decoding and a gradient decomposition, neither of which explains the observed parallel-generation improvements.

  4. Despite the motivation of enabling parallel generation, the converted model achieves only 0.134× and 0.168× AR serving throughput at output lengths 64 and 256, respectively, in the reported batch-size-one setting. Thus, no end-to-end inference advantage has been demonstrated. The attachment-density and denoiser-size ablations are also difficult to interpret due to single-seed evaluations and additional confounders.

  5. The manuscript is unnecessarily dense, with important experimental details scattered throughout the appendices. The extensive documentation of protocols and acceptance criteria sometimes obscures the main scientific argument. A more focused presentation would help distinguish established findings from proposed explanations.

The paper reports a substantial and potentially useful empirical finding, and I appreciate the effort put into evaluating different conversion configurations. However, the architecture is largely inherited from prior work, while the main comparison changes too many variables to identify what actually drives the improvement. The theoretical analysis provides limited additional explanation, and the converted model does not demonstrate an inference-speed advantage. I lean towards reject in the current form, but a better-controlled comparison isolating architectural separation from the training recipe would substantially strengthen the contribution and could change my assessment.

Questions for authors
  1. Can the authors provide an in-place baseline with matched corruption, block-size schedule, and comparable decoding settings? How much of the performance gap survives when the training recipe is controlled?

  2. Given the nearly identical HumanEval results for frozen and trainable towers, what evidence supports the proposed role of preserving pretrained contextual representations in generation? Can the knowledge-retention comparison be repeated with better-matched optimization?

  3. Is there a speed-quality operating point where the converted model actually outperforms its AR parent in inference efficiency? Otherwise, what is the practical advantage of conversion over simply retaining the AR model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.