acceptodds
Under review as a conference paper at ICLR 2027

Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement

Abstract

Continuous diffusion language models require choosing a representation space in which denoising operates, yet it remains unclear which representations are most compatible with diffusion. We study a basic design question: where inside a pretrained language model should continuous diffusion operate? We formulate this as a hidden-state interface-selection problem and hypothesize that diffusion-friendly interfaces can be identified from representation geometry. We operationalize this hypothesis using three training-free geometric proxies: local compactness, global stiffness, and effective rank. Across two 8B-scale backbones, the resulting geometry score predicts fixed-budget diffusion bridgeability beyond the dominant effect of layer depth. We then instantiate DiHAL, a Locate-and-Replace framework that replaces the transformer prefix below a selected interface with conditional diffusion while retaining the pretrained suffix and LM head. Under full training, geometry-selected interfaces remain close to validation-loss oracles and outperform worst-layer controls, while the diffusion bridge outperforms a parameter-matched deterministic replacement under matched diagnostic conditions. These results suggest that the representation space in which diffusion operates should itself be treated as a first-class design variable for continuous diffusion in language models.

Primary area:
generative models
Keywords:
Diffusion Language Models, Continuous Diffusion Models, Representation Learning, Representation Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2wMYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies where continuous diffusion should operate inside a pretrained language model. It proposes DiHAL, which scores Transformer layers using three geometry-based proxies (local compactness, global stiffness, and effective rank), selects an interface, and replaces the lower Transformer layers with a conditional diffusion bridge while keeping the pretrained suffix and LM head. Experiments on Llama-3.1-8B and Qwen3-8B show strong correlations between the geometry score and fixed-budget bridge reconstruction loss, followed by full-training and bridge-family comparisons.

Critical strengths
  1. The paper asks a useful and relatively underexplored design question: the representation space for continuous diffusion should itself be treated as a design choice. The layer sweep is more informative than simply reporting one chosen interface, and the depth-controlled analysis is a useful control.

  2. The experimental analysis is fairly transparent. In particular, the paper reports comparisons with deterministic and generative bridges and also reports the poor current quality/throughput trade-off rather than presenting DiHAL as an acceleration method.

Critical weaknesses
  1. It is not yet clear how much the geometry score adds beyond the strong preference for shallow layers. The fixed-budget optimum is layer 1 for both backbones, while geometry selects layers 3 and 2. After controlling for depth, the gain is small on Qwen, CKA performs slightly better on some metrics, and restricting the analysis to the first eight Qwen layers makes the geometry predictor worse. With only two closely related 8B decoder-only backbones, I do not think the current evidence establishes that the proposed geometry criterion generalizes as a useful layer-selection rule.

  2. The evidence that diffusion itself is better than generic hidden-state replacement is not sufficiently controlled. Conditioned on the input embeddings, the target hidden state is a deterministic output of the original Transformer prefix. The parameter-matched deterministic baseline uses a different MLP architecture, while the DiHAL bridge uses a Stable-Diffusion-style U-Net, and the comparison is conducted with only 512 activation samples and 300 optimization steps. Similarly, the comparison with existing continuous diffusion methods is not matched in total active model capacity or inference FLOPs because DiHAL retains the large pretrained suffix. These experiments show that the proposed system works better under the chosen setup, but do not cleanly isolate an advantage from diffusion or from the hidden-state interface itself.

Questions for authors
  1. Can the authors provide a same-architecture non-diffusion control, e.g., the same U-Net/projection/conditioning stack trained to directly predict the target hidden state without diffusion, preferably under the full training budget? This would substantially affect my assessment of whether the gain in Table 3 is specifically due to diffusion.

  2. Can the authors test the geometry-based selection on at least one additional held-out backbone/model family? In particular, how does it compare with the simple rule of selecting the shallowest layer, given the weak or negative added value of geometry within the shallow Qwen layers?

  3. Can the authors provide a comparison to the strongest continuous-diffusion baseline that better matches total active capacity or inference compute, or otherwise narrow the interpretation of Table 5? The current module-level budget matching does not isolate the effect of the interface.

4 comments
AuthorsOpenReview ↗

Official Rebuttal [1/4]

We thank the reviewer for their insightful feedback. We appreciate their recognition of our underexplored design question—treating the diffusion representation space as a design choice—along with our layer sweeps, depth-controlled analysis, and transparent evaluation.

W1: It is not yet clear how much the geometry score adds beyond the strong preference for shallow layers. The fixed-budget optimum is layer 1 for both backbones, while geometry selects layers 3 and 2. After controlling for depth, the gain is small on Qwen, CKA performs slightly better on some metrics, and restricting the analysis to the first eight Qwen layers makes the geometry predictor worse. With only two closely related 8B decoder-only backbones, I do not think the current evidence establishes that the proposed geometry criterion generalizes as a useful layer-selection rule.

A1: We thank the reviewer for this rigorous analysis. To directly test cross-family generalization, shallow-regime ranking, and depth confounding without retuning our criterion, we evaluated the fixed geometry score across all 16 layers of a held-out 1B backbone from a distinct model family (allenai/OLMo-2-0425-1B).

Non-trivial interior optimum and downstream gains over Layer 1 on a held-out family. Unlike Llama-3.1-8B and Qwen3-8B—where reconstruction error increases almost monotonically from Layer 1—OLMo-2-0425-1B exhibits a pronounced non-monotonic U-shaped bridgeability curve across depth. Empirical diffusion normalized MSE is worst at the boundaries (L0: 0.9705; L15: 0.9977) and reaches its global minimum at an interior layer, Layer 6 (norm. MSE = 0.9042, R2R^2 = 9.58%), clearly outperforming Layer 1 (norm. MSE = 0.9476, R2R^2 = 5.24%). Without any hyperparameter adjustment, our geometry score peaks at Layer 6 (+0.934 vs. −0.815 at Layer 1), selecting the exact fixed-budget sweep optimum. Crucially, under full-budget bridge training with a frozen suffix on OLMo-2-0425-1B (comparing Layer 6 against Layer 1), Geometry-selected Layer 6 decisively outperforms Layer 1 across every evaluation metric: NLL improves from 5.10 to 4.96, PPL drops from 163.75 to 142.62 (−12.9%), and teacher-student KL divergence drops from 0.54 to 0.42 (−22.2%).

Strong correlation in both full-depth and shallow-layer (L0–L7) regimes. Across all 16 layers of OLMo-2-0425-1B, the geometry score achieves Spearman ρ\rho = +0.965 and Kendall τ\tau = +0.883 against negative normalized MSE. Crucially, restricting the evaluation to the first eight layers (L0–L7)—where bridgeability improves from L0 to L6 against simple shallow-layer preference—the geometry score still maintains a strong Spearman ρ\rho = +0.905, whereas a naive "shallower is better" heuristic is negatively aligned in this regime.

Explicit depth-confounding controls (linear and quadratic depth). Because OLMo-2-0425-1B has an interior optimum, we evaluated leave-one-out cross-validation (LOOCV) RMSE against both linear and quadratic depth baselines. Adding our geometry score consistently reduces prediction error over depth-only models:

  • Full depth (L0–L15, vs. linear depth): LOOCV RMSE drops from 0.03018 to 0.01152 (+61.8% relative improvement).
  • Full depth (L0–L15, vs. quadratic depth): LOOCV RMSE drops from 0.01490 to 0.01303 (+12.5% improvement) and from 0.01529 to 0.00955 (+37.5% improvement) excluding the extreme final-layer outlier (L0–L14).
  • Shallow regime (L0–L7, vs. quadratic depth): LOOCV RMSE drops from 0.01715 to 0.01093 (+36.3% improvement).
AuthorsOpenReview ↗

Official Rebuttal [2/4]

W2: The evidence that diffusion itself is better than generic hidden-state replacement is not sufficiently controlled. Conditioned on the input embeddings, the target hidden state is a deterministic output of the original Transformer prefix. The parameter-matched deterministic baseline uses a different MLP architecture, while the DiHAL bridge uses a Stable-Diffusion-style U-Net, and the comparison is conducted with only 512 activation samples and 300 optimization steps. Similarly, the comparison with existing continuous diffusion methods is not matched in total active model capacity or inference FLOPs because DiHAL retains the large pretrained suffix. These experiments show that the proposed system works better under the chosen setup, but do not cleanly isolate an advantage from diffusion or from the hidden-state interface itself.

A2: We thank the reviewer for identifying both confounds: (1) isolating the diffusion objective from bridge architecture, and (2) controlling retained backbone capacity in baseline comparisons. We conducted two new experiments on allenai/OLMo-2-0425-1B to address each point directly.

Isolating diffusion via a same-architecture non-diffusion control (OLMo-2-0425-1B 16-layer sweep).

To eliminate the MLP-vs.-UNet architectural confound, we conducted a controlled 16-layer sweep comparing the diffusion bridge against a same-architecture non-diffusion deterministic control (the identical U-Net/projection/conditioning stack trained to directly predict target hidden states without diffusion noise). Across 13 of 16 layers (L0 and L3–L14), the diffusion bridge consistently achieves lower normalized MSE and higher explained variance (R2R^2) than direct deterministic regression (with direct regression marginally lower only at L1–L2 and L15). At the optimal Layer 6, diffusion improves normalized MSE from 0.9380 to 0.9042 and raises hidden-state R2R^2 from 6.20% to 9.58% (a 1.55× relative gain in explained variance), while also aligning more closely with representation geometry (16-layer Spearman ρ\rho = +0.965 vs. +0.862; L0–L7 ρ\rho = +0.905 vs. +0.814; Kendall τ\tau = +0.883 vs. +0.745). While this sweep cleanly isolates an advantage from the diffusion objective under fixed-budget bridge training, we will also explicitly clarify that causal superiority under full-budget bridge training with a frozen suffix over all deterministic predictors remains an open question.

Substantially reducing system capacity with DiHAL-small on OLMo-2-0425-1B and narrowing Table 5's scope.

To test whether DiHAL's generation quality relies on a massive 8B suffix or pretrained diffusion weights, we trained DiHAL-small from scratch at Geometry-selected Layer 6 on OLMo-2-0425-1B (1.48B total parameters; 16 layers, H=2048H=2048). Replacing L0–L6 skips 7 layers (469.8M parameters) and retains only 9 upper layers (L7–L15: 604.1M) plus the LM head (205.5M)—809.6M in the frozen suffix+head (1,015.1M including the 205.5M input embedding table), a ~7.5× reduction in retained pretrained parameters compared to Qwen3-8B (~7.61B). Paired with the randomly initialized DiHAL-small bridge (~255.9M trainable / 339.5M active bridge at H=2048H=2048, slightly smaller than CoDAR's 360.9M), total active system size drops by ~6.3× compared to full Qwen3-8B DiHAL (~8.56B →\rightarrow 1.35B including embeddings and frozen VAE, or 1.15B bridge+suffix+head). Despite this drastic reduction, DiHAL-small (OLMo-2-1B) achieves Gen.PPL = 138.32 and Diversity = 0.5812, outperforming both CoDAR (144.83 / 0.4777) and Qwen3-8B DiHAL-small (140.51 / 0.5230). Because total active system parameters (1.35B) and inference FLOPs still exceed standalone CoDAR (360.9M), we also explicitly narrow the interpretation of Table 5 to a diagnostic comparison under matched diffusion/recovery training budgets rather than a total-capacity-matched benchmark.

AuthorsOpenReview ↗

Official Rebuttal [3/4]

Q1: Can the authors provide a same-architecture non-diffusion control, e.g., the same U-Net/projection/conditioning stack trained to directly predict the target hidden state without diffusion, preferably under the full training budget? This would substantially affect my assessment of whether the gain in Table 3 is specifically due to diffusion.

A1: Yes. Following the reviewer's suggestion, we evaluated a same-architecture non-diffusion direct-prediction control (using the identical U-Net/projection/conditioning stack without diffusion noising) alongside our diffusion bridge across all 16 layers of OLMo-2-0425-1B:

Metric (OLMo-2-0425-1B 16-layer sweep)Diffusion BridgeSame-Arch Non-Diffusion Control
Minimum normalized MSE layerL6L6
Layer 6 normalized MSE (↓)0.90420.9380
Layer 6 hidden R2R^2 (↑)9.58%6.20%
Spearman ρ\rho: geometry vs. −norm. MSE (16 layers)+0.965+0.862
Spearman ρ\rho: geometry vs. −norm. MSE (L0–L7)+0.905+0.814
Kendall τ\tau: geometry vs. −norm. MSE (16 layers)+0.883+0.745

Two clear findings emerge: (1) Both methods identify Layer 6 as the optimal reconstruction layer, confirming that layer-wise bridgeability is an intrinsic property of the Transformer's representations. (2) Diffusion outperforms same-architecture direct regression across 13 of 16 layers—specifically at L0 (norm. MSE 0.9705 vs. 0.9735; R2R^2 2.95% vs. 2.65%) and throughout the entire interior regime L3–L14—increasing explained variance at Layer 6 by 54.5% relatively (R2R^2: 9.58% vs. 6.20%), whereas direct prediction is marginally lower only at L1 (0.9420 vs. 0.9476), L2 (0.9635 vs. 0.9704), and L15 (0.9970 vs. 0.9977). Due to rebuttal compute constraints, this same-architecture control was conducted as a 16-layer fixed-budget sweep rather than full-budget bridge training with a frozen suffix; we will report this controlled sweep in the revised paper while softening our wording around the original MLP baseline so as not to overclaim full-budget causal superiority.

Q2: Can the authors test the geometry-based selection on at least one additional held-out backbone/model family? In particular, how does it compare with the simple rule of selecting the shallowest layer, given the weak or negative added value of geometry within the shallow Qwen layers?

A2: Yes. We evaluated our unmodified geometry criterion on allenai/OLMo-2-0425-1B (16 layers) across both the 16-layer fixed-budget sweep and full-budget bridge training with a frozen suffix comparing Layer 1 vs. Layer 6:

Metric on OLMo-2-0425-1BLayer 0 (Shallowest)Layer 1 (Shallow rule)Layer 6 (Geometry-selected)Best Observed Layer
Geometry score (↑, training-free)−1.193−0.815+0.934 (Rank 1)Layer 6 (16-layer)
Sweep Diffusion norm. MSE (↓)0.97050.94760.9042 (Rank 1)Layer 6 (16-layer sweep)
Sweep Diffusion R2R^2 (↑)2.95%5.24%9.58% (Rank 1)Layer 6 (16-layer sweep)
Full-budget NLL (↓, frozen suffix)—5.104.96Layer 6 (L1 vs. L6)
Full-budget PPL (↓, frozen suffix)—163.75142.62Layer 6 (L1 vs. L6)
Full-budget Teacher KL (↓, frozen suffix)—0.540.42Layer 6 (L1 vs. L6)

Decisive win over the shallowest-layer heuristic in both the 16-layer sweep and full-budget L1-vs.-L6 evaluation: Unlike Qwen3-8B and Llama-3.1-8B, OLMo-2-0425-1B does not favor the shallowest layer in the fixed-budget sweep. Geometry guidance accurately captures this non-monotonic profile and selects the 16-layer sweep optimum (Layer 6) before training; under full-budget bridge training with a frozen suffix (evaluated on L1 vs. L6), Layer 6 reduces PPL by 21.13 points (−12.9%) and teacher KL divergence by 22.2% compared to Layer 1.

Strong positive value within shallow layers (L0–L7): Within the first eight layers (L0–L7), our geometry score achieves Spearman ρ\rho = +0.905 with bridgeability and reduces LOOCV RMSE by 36.3% over a quadratic depth baseline (0.01715 →\rightarrow 0.01093), demonstrating substantial predictive value precisely when shallow-layer heuristics fail.

AuthorsOpenReview ↗

Official Rebuttal [4/4]

Q3: Can the authors provide a comparison to the strongest continuous-diffusion baseline that better matches total active capacity or inference compute, or otherwise narrow the interpretation of Table 5? The current module-level budget matching does not isolate the effect of the interface.

A3: We address this in two ways: (1) training DiHAL-small from scratch on the smaller OLMo-2-0425-1B backbone to substantially shrink both the bridge and the retained pretrained suffix, and (2) explicitly narrowing the interpretation of Table 5 with transparent total-system parameter accounting:

MethodBackboneTrainable ParamsActive Bridge / Module ParamsRetained Frozen Backbone (Suffix + LM Head / + Embed)Total Active System ParamsGen.PPL (↓)Diversity (↑)
CoDARStandalone360.9M360.9M—360.9M144.830.4777
DiHAL (full)Qwen3-8B (L2)862.7M946.3M6.99B / 7.61B~8.56B136.020.5913
DiHAL-smallQwen3-8B (L2)~257.4M341.1M6.99B / 7.61B~7.95B140.510.5230
DiHAL-small (OLMo-2-1B)OLMo-2-0425-1B (L6)~255.9M339.5M809.6M / 1,015.1M1.35B (1.15B w/o embed)138.320.5812

Stronger generation quality with a smaller scratch-trained bridge and ~7.5× smaller suffix: Using the from-scratch DiHAL-small bridge (339.5M active bridge parameters at H=2048H=2048, smaller than CoDAR's 360.9M) at L6 on OLMo-2-0425-1B skips 7 decoder layers (469.8M) and retains only 9 upper layers (604.1M) plus the LM head (205.5M), reducing retained pretrained parameters by ~7.5× (~7.61B →\rightarrow 1.02B with embeddings) and total system size by ~6.3× (~8.56B →\rightarrow 1.35B) relative to full Qwen3-8B DiHAL. Notably, DiHAL-small (OLMo-2-1B) achieves Gen.PPL = 138.32 and Diversity = 0.5812—not only outperforming CoDAR (144.83 / 0.4777), but also surpassing DiHAL-small on the much larger Qwen3-8B (140.51 / 0.5230), indicating that performance is driven by interface compatibility rather than sheer frozen-suffix size.

Narrowed interpretation of Table 5: Even with DiHAL-small on OLMo-2-0425-1B, total active system capacity (1.35B including frozen suffix, embeddings, and VAE; 1.15B excluding input embeddings) and inference FLOPs remain larger than standalone CoDAR (360.9M). We therefore agree with the reviewer and will revise Section 4 / Table 5 to explicitly frame this comparison as a diagnostic evaluation under matched diffusion/recovery training budgets (300K examples, 40 H100 GPU-hours) rather than a total-capacity- or FLOPs-equalized benchmark, reporting full system-level parameter counts alongside module-level counts.

Reviewer_xtAFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates where continuous diffusion should operate within a pretrained language model. It shows that geometric metrics can be used as complementary to accurately and stably identify layers that can be replaced. Experiments on Llama-3.1-8B indicate that a diffusion model is more suitable as the replacement module—referred to as a “bridge”—than alternatives such as an MLP or VAE. The paper also shows that the layer selected using the geometric metrics maintains performance close to that of the oracle-selected layer. However, introducing the proposed “diffusion bridge” increases perplexity by approximately 21$\times$ and 11$\times$ relative to the original Llama-3.1-8B and Qwen3-8B models, respectively.

Critical strengths

None

Critical weaknesses
  • The motivation for the hybrid architecture is unclear. The introduction asks, “Where should diffusion enter the pretrained transformer?” but does not adequately address the more fundamental question: “Why should diffusion enter the pretrained transformer?” Because the latter part of the model remains autoregressive, the proposed architecture cannot take advantage of parallel generation. Moreover, given the substantial degradation in perplexity, the hybrid architecture does not appear to offer a clear benefit.

  • The evaluation results and the fairness of the comparisons remain questionable. The original models achieve PPL values of 6.47 and 13.55, whereas the proposed method yields 135.64 and 144.03, respectively. Given this substantial degradation, it is difficult to assess the practical effectiveness of the proposed method. In addition, Table 5 does not clearly explain how each baseline is integrated with or adapted to Qwen3-8B. Consequently, it is unclear what conclusion the authors intend to support with the reported Gen.PPL and Div. values.

Questions for authors
  • Please clarify why introducing the proposed hybrid architecture is important. In particular, what benefit justifies adopting the hybrid model despite the approximately 21$\times$ and 11$\times$ increases in perplexity relative to the original models?
  • Please clarify the evaluation setup for the comparisons in Table 5. In particular, it is unclear what the “Recovery interface” column means, how each method is configured, and whether the methods are compared under genuinely equivalent conditions.
5 comments
AuthorsOpenReview ↗

Official Rebuttal [1/3]

We thank the reviewer for their constructive feedback. We appreciate that they found our study informative and noted how geometric metrics stably identify replaceable layers with near-oracle performance.

W1: The motivation for the hybrid architecture is unclear. The introduction asks, “Where should diffusion enter the pretrained transformer?” but does not adequately address the more fundamental question: “Why should diffusion enter the pretrained transformer?” Because the latter part of the model remains autoregressive, the proposed architecture cannot take advantage of parallel generation. Moreover, given the substantial degradation in perplexity, the hybrid architecture does not appear to offer a clear benefit.

A1: We thank the reviewer for raising this fundamental question. DiHAL does not aim to establish diffusion as inherently superior to autoregressive modeling, but to investigate its potential as an iterative mechanism for reconstructing continuous representations within pretrained Transformers. We clarify our motivation through three points.

1. Why diffusion? Empirical advantages over deterministic reconstruction.

Although Transformer hidden states are deterministic given their inputs, reconstructing them with limited model capacity and training budgets remains challenging. We therefore investigate whether diffusion offers advantages over direct prediction.

To isolate diffusion's contribution, we compared our bridge against a deterministic control using the identical U-Net architecture, projections, and conditioning on OLMo-2-0425-1B. Diffusion achieves lower reconstruction error in 13 of 16 layers. At Layer 6, normalized MSE decreases from 0.9380 to 0.9042, while explained variance increases from 6.20% to 9.58%. These results demonstrate a reconstruction advantage beyond architectural differences under fixed-budget training, without claiming universal superiority over deterministic predictors.

2. Why a hybrid architecture? Leveraging pretrained language decoding.

Continuous diffusion language modeling faces challenges in recovering discrete tokens from continuous representations. DiHAL addresses this by reconstructing intermediate hidden states and passing them through the frozen Transformer suffix and original LM head, avoiding a separately trained token decoder.

While this retains autoregressive decoding rather than enabling parallel generation, it establishes an interface between continuous diffusion and pretrained language modeling. Under matched training data (300K examples) and compute budgets (40 H100 GPU-hours), DiHAL achieves favorable generation quality against evaluated continuous-diffusion baselines (Table 5). Even DiHAL-small, trained from scratch, outperforms CoDAR in Gen.PPL (140.51 vs. 144.83) and diversity (0.5230 vs. 0.4777). We acknowledge that total pretrained capacity differs and interpret these comparisons as evidence of interface feasibility rather than capacity-controlled superiority.

3. Why geometry-guided integration? Identifying suitable diffusion interfaces.

Introducing diffusion into a pretrained Transformer raises a key question: which hidden representations are most suitable for diffusion reconstruction?

Our training-free geometry criterion, combining local compactness, global stiffness, and effective rank, strongly predicts reconstruction difficulty across Llama-3.1-8B and Qwen3-8B (Spearman ρ = 0.9143 and 0.9267). On held-out OLMo-2-0425-1B, the unchanged criterion identifies Layer 6, the empirical optimum in our 16-layer sweep (ρ = 0.965). Under full-budget training, this selection also improves PPL from 163.75 to 142.62 over the shallow-layer heuristic.

In summary, DiHAL establishes a continuous diffusion interface within pretrained Transformers, demonstrates empirical reconstruction advantages over same-architecture deterministic prediction, and provides a training-free principle for selecting insertion layers. We position DiHAL as a proof of concept for geometry-guided diffusion integration, while acknowledging its current perplexity degradation and inference overhead relative to autoregressive models. We will clarify this scope in the revised introduction.

AuthorsOpenReview ↗

Official Rebuttal [2/3]

W2: The evaluation results and the fairness of the comparisons remain questionable. The original models achieve PPL values of 6.47 and 13.55, whereas the proposed method yields 135.64 and 144.03, respectively. Given this substantial degradation, it is difficult to assess the practical effectiveness of the proposed method. In addition, Table 5 does not clearly explain how each baseline is integrated with or adapted to Qwen3-8B. Consequently, it is unclear what conclusion the authors intend to support with the reported Gen.PPL and Div. values.

A2: We thank the reviewer for raising concerns about the evaluation setup and comparison fairness. We clarify the baseline configurations, parameter accounting, and intended interpretation of Table 5.

Quality degradation relative to the original backbones. We acknowledge the substantial perplexity increases from 6.47 to 135.64 for Llama-3.1-8B and from 13.55 to 144.03 for Qwen3-8B. As discussed in our response to W1, DiHAL is not intended as a competitive replacement for the original autoregressive models. Instead, it investigates whether continuous diffusion can reconstruct intermediate Transformer representations and whether the choice of reconstruction space can be predicted before training.

Baseline configurations and evaluation setup. Table 5 does not evaluate diffusion baselines integrated into Qwen3-8B. Rather, each baseline is implemented and trained independently following its original architectural approach. Qwen3-8B is the pretrained backbone used by DiHAL, not a shared backbone for all methods.

Diffusion-LM and SED operate in token-embedding spaces, LD4LG uses learned text latents with pretrained BART-based decoding components, and CoDAR uses a contextual autoregressive recovery decoder. In contrast, DiHAL reconstructs intermediate Qwen3-8B hidden states and passes them through the frozen Transformer suffix and LM head.

The comparison uses the same 300K-example training corpus, a matched diffusion/recovery training budget of 40 H100 GPU-hours, and common generation-quality metrics. Table 9 additionally reports the trainable and active module parameter counts for each method.

Additional module-capacity control. To address differences in model capacity and initialization, we also report a smaller DiHAL bridge trained from scratch (Table 10).

DiHAL-small uses 341.1M active bridge parameters on Qwen3-8B, compared with 360.9M for CoDAR. Despite the substantially reduced bridge size and removal of pretrained diffusion initialization, DiHAL-small achieves lower Gen.PPL (140.51 vs. 144.83) and higher diversity (0.5230 vs. 0.4777).

To further narrow the retained-backbone gap, we additionally trained DiHAL-small from scratch on a smaller 1B backbone (allenai/OLMo-2-0425-1B, Layer 6; 339.5M active bridge, 1.35B total active system parameters—a ~6× reduction from Qwen3-8B DiHAL-small), which achieves 138.32 Gen.PPL and 0.5812 diversity, outperforming both CoDAR and Qwen3-8B DiHAL-small. This provides a more closely controlled comparison across both bridge and retained-suffix scales. However, we acknowledge that the comparison still does not strictly match total system capacity (e.g., 1.35B vs. 360.9M for CoDAR), since DiHAL retains a frozen Transformer suffix and LM head.

Interpretation of Table 5. Table 5 is intended as a diagnostic comparison of continuous-diffusion interfaces under a common training budget, not as evidence that DiHAL outperforms its original autoregressive backbone.

Together, Tables 5, 9, and 10 show that the hidden-state interface achieves favorable generation quality relative to the evaluated continuous-diffusion approaches, including under a more closely matched bridge-module capacity. We will clarify the independent baseline training setup and the distinction between module-level and total-system capacity in the revised experimental description.

AuthorsOpenReview ↗

Official Rebuttal [3/3]

Q1: Please clarify why introducing the proposed hybrid architecture is important. In particular, what benefit justifies adopting the hybrid model despite the approximately 21× and 11× increases in perplexity relative to the original models?

A1: We thank the reviewer for raising this important question. As discussed in our response to W1, we agree that the current hybrid architecture does not offer a practical quality–efficiency advantage over the original autoregressive models. The approximately 21× and 11× increases in perplexity are substantial, and we do not claim that DiHAL should replace the original models in practical deployment.

The value of DiHAL lies in a different objective. Rather than proposing a better language model, we investigate whether continuous diffusion can replace intermediate Transformer computations while preserving the pretrained upper layers and LM head, and whether the reconstruction difficulty of different hidden representations can be predicted before training.

This architecture allows us to systematically study a design choice that existing continuous diffusion approaches do not explicitly investigate: where diffusion should interface with a pretrained Transformer. Our experiments show that hidden-state geometry strongly predicts layer-wise diffusion bridgeability (Spearman ρ = 0.9143 and 0.9267, Table 1), with predictive value extending to full training (Table 4).

Thus, the current results justify DiHAL as a framework for investigating and selecting hidden-state diffusion interfaces, rather than as a practical replacement for autoregressive language modeling. The substantial perplexity gap remains an important limitation and a challenge for future work.

Q2: Please clarify the evaluation setup for the comparisons in Table 5. In particular, it is unclear what the “Recovery interface” column means, how each method is configured, and whether the methods are compared under genuinely equivalent conditions.

A2: A2: We thank the reviewer for raising concerns about the evaluation setup and comparison fairness. We clarify the recovery interfaces, baseline configurations, and scope of the comparisons below.

Recovery interface and baseline configurations. The "Recovery interface" refers to how diffusion-generated continuous representations are converted into discrete tokens. The baselines are independently trained following their original architectures, not adapted to Qwen3-8B. Diffusion-LM and SED use embedding-to-token mappings; LD4LG uses pretrained BART-based decoding; CoDAR employs a learned contextual autoregressive decoder. In contrast, DiHAL reconstructs intermediate Qwen3-8B hidden states using its frozen Transformer suffix and LM head.

Comparison fairness. All methods use the same 300K-example training corpus and a matched diffusion/recovery training budget of 40 H100 GPU-hours, with common Gen.PPL and diversity metrics. Table 9 reports both trainable and active module parameters. Table 10 further controls for module capacity and initialization: DiHAL-small on Qwen3-8B, trained from scratch with 341.1M active parameters, achieves better Gen.PPL (140.51 vs. 144.83) and diversity (0.5230 vs. 0.4777) than CoDAR (360.9M). To further narrow the retained-backbone gap, we additionally trained DiHAL-small from scratch on a smaller 1B backbone (allenai/OLMo-2-0425-1B, Layer 6; 339.5M active bridge, 1.35B total active system parameters—a ~6× reduction from Qwen3-8B DiHAL-small), which still achieves 138.32 Gen.PPL and 0.5812 diversity, outperforming both CoDAR and Qwen3-8B

DiHAL-small. However, total system capacity is still not strictly matched with standalone baselines (e.g., 360.9M for CoDAR) because DiHAL retains a pretrained Transformer suffix. Interpretation. Table 5 is a diagnostic comparison of continuous-diffusion interfaces, not a fully capacity-controlled benchmark against autoregressive backbones. Together, Tables 5, 9, and 10 demonstrate favorable performance of the hidden-state interface under matched training budgets and approximately matched module capacities, while acknowledging differences in pretrained components. We will clarify these distinctions in the revised paper.

Reviewer_SK7KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies where diffusion process should be used in a language model. Instead of doing diffusion on token embeddings or in the latent space, the paper suggests to diffuse in an intermediate hidden-state space in a pretrained transformer. Particularly, it proposes DiHAL, a framework that chooses hidden layers guided by the representation geometry. It shows that geometry has the potential to be a proxy to predict diffusion difficulty across layer depth.

Critical strengths

Motivation

The paper shows an interesting motivation. Rather than directly searching on token embeddings or in a latent space, it proposes to first figure out where to diffuse. Later on authors introduce the idea of representation geometry.

Method

The paper presents a Locate-and-Replace framework by using geometric proxies to analyze and choose which layers should be used for diffusion. Moreover, the method is training-free.

Critical weaknesses
  • In Table 4, it looks like the model suffers significant regression after replacing layers with diffusion. Especially, PPL increases from 6.47 to 135.64. There is also inference overhead as shown in Table 11.
  • Although the geometry score achieves high correlations, the optimal layer is always Layer 1 in Table 1, while geometry guidance selects Layer 3 (or Layer 2 for Qwen). It is unclear if the proposed representation geometry actually offers benefits over simple shallow-layer (Layer 1) selection.
Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Official Rebuttal

We thank the reviewer for their thoughtful evaluation and constructive feedback. We appreciate their recognition of the motivation behind investigating where diffusion should be introduced within a pretrained language model, as well as the value of our Locate-and-Replace framework and its training-free, geometry-guided layer selection. We are also encouraged that the reviewer found the relationship between representation geometry and diffusion difficulty interesting and informative.

W1: In Table 4, it looks like the model suffers significant regression after replacing layers with diffusion. Especially, PPL increases from 6.47 to 135.64. There is also inference overhead as shown in Table 11.

A1: We thank the reviewer for highlighting the substantial quality degradation and inference overhead. We agree that these are important limitations of the current architecture, and we have clarified the intended scope of DiHAL.

What DiHAL focuses on. Our goal is not to improve upon the original autoregressive backbone, but to investigate whether continuous diffusion can reconstruct intermediate Transformer representations that remain usable by the frozen upper layers and LM head, and whether the difficulty of this reconstruction can be predicted before training. Specifically, DiHAL (1) establishes a hidden-state diffusion interface without training a separate token decoder, and (2) introduces a training-free geometry score that closely predicts layer-wise diffusion bridgeability (Spearman ρ = 0.9143 / 0.9267, Table 1), with predictive value extending to full training (Table 4).

What DiHAL does not claim. We fully acknowledge the substantial increase in PPL from 6.47 to 135.64 (20.96×) and the additional inference overhead reported in Table 11. DiHAL is neither a competitive replacement for the original autoregressive model nor a method for accelerating token generation. Its current implementation retains autoregressive decoding, and we do not claim that the present architecture offers a practical quality–efficiency advantage.

Why the results remain meaningful. The quality gap limits the current architecture’s practical applicability, but it does not negate the central finding: hidden representations differ substantially in their diffusion reconstruction difficulty, and this difference can be predicted from their geometry before bridge training. We therefore position DiHAL as a framework for understanding and selecting continuous-diffusion interfaces within pretrained Transformers, rather than as a deployable alternative to autoregressive language modeling.

W2: Although the geometry score achieves high correlations, the optimal layer is always Layer 1 in Table 1, while geometry guidance selects Layer 3 (or Layer 2 for Qwen). It is unclear if the proposed representation geometry actually offers benefits over simple shallow-layer (Layer 1) selection.

A2: To directly test this on an unseen model family and scale, we conducted a 16-layer sweep on a held-out 1B backbone (allenai/OLMo-2-0425-1B), keeping our geometry scoring rule fixed without tuning.

Clear benefit over shallow-layer (Layer-1) selection on a held-out model family. Unlike the two 8B backbones where bridgeability declines nearly monotonically with depth, OLMo-2-0425-1B exhibits a clear non-monotonic interior optimum: empirical diffusion reconstruction error is minimized at Layer 6 (normalized MSE = 0.9042, R2R^2 = 9.58%), substantially outperforming both Layer 0 (0.9705, R2R^2 = 2.95%) and Layer 1 (0.9476, R2R^2 = 5.24%). Crucially, our training-free geometry score peaks at Layer 6 (+0.934 vs. −0.815 at Layer 1), pinpointing the exact empirical sweep optimum where a naive Layer-1 rule fails. Across all 16 layers, the geometry score achieves Spearman ρ\rho = 0.965 (Kendall τ\tau = 0.883) and maintains ρ\rho = 0.905 even within the shallow half (L0–L7). Most importantly, this advantage carries over directly to full-budget bridge training with a frozen suffix on OLMo-2-0425-1B: Geometry-selected Layer 6 substantially outperforms Layer 1 across all downstream metrics—reducing NLL from 5.10 to 4.96, PPL from 163.75 to 142.62 (−12.9%), and teacher-student KL divergence from 0.54 to 0.42 (−22.2%).

Computational benefit of training-free selection. Because optimal diffusion interfaces are not universally located at Layer 1 across model families, identifying the best interface empirically requires training a separate diffusion bridge per candidate layer (O(L)O(L) training runs: 16 runs on OLMo-2-0425-1B and 32–36 runs on 8B backbones). In contrast, our geometry score evaluates pre-extracted hidden states from small subsamples without any bridge training, eliminating O(L)O(L) expensive sweep runs while accurately locating non-trivial interior optima such as Layer 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.