- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies where continuous diffusion should operate inside a pretrained language model. It proposes DiHAL, which scores Transformer layers using three geometry-based proxies (local compactness, global stiffness, and effective rank), selects an interface, and replaces the lower Transformer layers with a conditional diffusion bridge while keeping the pretrained suffix and LM head. Experiments on Llama-3.1-8B and Qwen3-8B show strong correlations between the geometry score and fixed-budget bridge reconstruction loss, followed by full-training and bridge-family comparisons.
Critical strengths
-
The paper asks a useful and relatively underexplored design question: the representation space for continuous diffusion should itself be treated as a design choice. The layer sweep is more informative than simply reporting one chosen interface, and the depth-controlled analysis is a useful control.
-
The experimental analysis is fairly transparent. In particular, the paper reports comparisons with deterministic and generative bridges and also reports the poor current quality/throughput trade-off rather than presenting DiHAL as an acceleration method.
Critical weaknesses
-
It is not yet clear how much the geometry score adds beyond the strong preference for shallow layers. The fixed-budget optimum is layer 1 for both backbones, while geometry selects layers 3 and 2. After controlling for depth, the gain is small on Qwen, CKA performs slightly better on some metrics, and restricting the analysis to the first eight Qwen layers makes the geometry predictor worse. With only two closely related 8B decoder-only backbones, I do not think the current evidence establishes that the proposed geometry criterion generalizes as a useful layer-selection rule.
-
The evidence that diffusion itself is better than generic hidden-state replacement is not sufficiently controlled. Conditioned on the input embeddings, the target hidden state is a deterministic output of the original Transformer prefix. The parameter-matched deterministic baseline uses a different MLP architecture, while the DiHAL bridge uses a Stable-Diffusion-style U-Net, and the comparison is conducted with only 512 activation samples and 300 optimization steps. Similarly, the comparison with existing continuous diffusion methods is not matched in total active model capacity or inference FLOPs because DiHAL retains the large pretrained suffix. These experiments show that the proposed system works better under the chosen setup, but do not cleanly isolate an advantage from diffusion or from the hidden-state interface itself.
Questions for authors
-
Can the authors provide a same-architecture non-diffusion control, e.g., the same U-Net/projection/conditioning stack trained to directly predict the target hidden state without diffusion, preferably under the full training budget? This would substantially affect my assessment of whether the gain in Table 3 is specifically due to diffusion.
-
Can the authors test the geometry-based selection on at least one additional held-out backbone/model family? In particular, how does it compare with the simple rule of selecting the shallowest layer, given the weak or negative added value of geometry within the shallow Qwen layers?
-
Can the authors provide a comparison to the strongest continuous-diffusion baseline that better matches total active capacity or inference compute, or otherwise narrow the interpretation of Table 5? The current module-level budget matching does not isolate the effect of the interface.
4 comments
Official Rebuttal [2/4]
W2: The evidence that diffusion itself is better than generic hidden-state replacement is not sufficiently controlled. Conditioned on the input embeddings, the target hidden state is a deterministic output of the original Transformer prefix. The parameter-matched deterministic baseline uses a different MLP architecture, while the DiHAL bridge uses a Stable-Diffusion-style U-Net, and the comparison is conducted with only 512 activation samples and 300 optimization steps. Similarly, the comparison with existing continuous diffusion methods is not matched in total active model capacity or inference FLOPs because DiHAL retains the large pretrained suffix. These experiments show that the proposed system works better under the chosen setup, but do not cleanly isolate an advantage from diffusion or from the hidden-state interface itself.
A2: We thank the reviewer for identifying both confounds: (1) isolating the diffusion objective from bridge architecture, and (2) controlling retained backbone capacity in baseline comparisons. We conducted two new experiments on allenai/OLMo-2-0425-1B to address each point directly.
Isolating diffusion via a same-architecture non-diffusion control (OLMo-2-0425-1B 16-layer sweep).
To eliminate the MLP-vs.-UNet architectural confound, we conducted a controlled 16-layer sweep comparing the diffusion bridge against a same-architecture non-diffusion deterministic control (the identical U-Net/projection/conditioning stack trained to directly predict target hidden states without diffusion noise). Across 13 of 16 layers (L0 and L3–L14), the diffusion bridge consistently achieves lower normalized MSE and higher explained variance () than direct deterministic regression (with direct regression marginally lower only at L1–L2 and L15). At the optimal Layer 6, diffusion improves normalized MSE from 0.9380 to 0.9042 and raises hidden-state from 6.20% to 9.58% (a 1.55× relative gain in explained variance), while also aligning more closely with representation geometry (16-layer Spearman = +0.965 vs. +0.862; L0–L7 = +0.905 vs. +0.814; Kendall = +0.883 vs. +0.745). While this sweep cleanly isolates an advantage from the diffusion objective under fixed-budget bridge training, we will also explicitly clarify that causal superiority under full-budget bridge training with a frozen suffix over all deterministic predictors remains an open question.
Substantially reducing system capacity with DiHAL-small on OLMo-2-0425-1B and narrowing Table 5's scope.
To test whether DiHAL's generation quality relies on a massive 8B suffix or pretrained diffusion weights, we trained DiHAL-small from scratch at Geometry-selected Layer 6 on OLMo-2-0425-1B (1.48B total parameters; 16 layers, ). Replacing L0–L6 skips 7 layers (469.8M parameters) and retains only 9 upper layers (L7–L15: 604.1M) plus the LM head (205.5M)—809.6M in the frozen suffix+head (1,015.1M including the 205.5M input embedding table), a ~7.5× reduction in retained pretrained parameters compared to Qwen3-8B (~7.61B). Paired with the randomly initialized DiHAL-small bridge (~255.9M trainable / 339.5M active bridge at , slightly smaller than CoDAR's 360.9M), total active system size drops by ~6.3× compared to full Qwen3-8B DiHAL (~8.56B 1.35B including embeddings and frozen VAE, or 1.15B bridge+suffix+head). Despite this drastic reduction, DiHAL-small (OLMo-2-1B) achieves Gen.PPL = 138.32 and Diversity = 0.5812, outperforming both CoDAR (144.83 / 0.4777) and Qwen3-8B DiHAL-small (140.51 / 0.5230). Because total active system parameters (1.35B) and inference FLOPs still exceed standalone CoDAR (360.9M), we also explicitly narrow the interpretation of Table 5 to a diagnostic comparison under matched diffusion/recovery training budgets rather than a total-capacity-matched benchmark.
Official Rebuttal [3/4]
Q1: Can the authors provide a same-architecture non-diffusion control, e.g., the same U-Net/projection/conditioning stack trained to directly predict the target hidden state without diffusion, preferably under the full training budget? This would substantially affect my assessment of whether the gain in Table 3 is specifically due to diffusion.
A1: Yes. Following the reviewer's suggestion, we evaluated a same-architecture non-diffusion direct-prediction control (using the identical U-Net/projection/conditioning stack without diffusion noising) alongside our diffusion bridge across all 16 layers of OLMo-2-0425-1B:
Metric (OLMo-2-0425-1B 16-layer sweep) | Diffusion Bridge | Same-Arch Non-Diffusion Control |
|---|---|---|
| Minimum normalized MSE layer | L6 | L6 |
| Layer 6 normalized MSE (↓) | 0.9042 | 0.9380 |
| Layer 6 hidden (↑) | 9.58% | 6.20% |
| Spearman : geometry vs. −norm. MSE (16 layers) | +0.965 | +0.862 |
Spearman : geometry vs. −norm. MSE (L0–L7) | +0.905 | +0.814 |
| Kendall : geometry vs. −norm. MSE (16 layers) | +0.883 | +0.745 |
Two clear findings emerge: (1) Both methods identify Layer 6 as the optimal reconstruction layer, confirming that layer-wise bridgeability is an intrinsic property of the Transformer's representations. (2) Diffusion outperforms same-architecture direct regression across 13 of 16 layers—specifically at L0 (norm. MSE 0.9705 vs. 0.9735; 2.95% vs. 2.65%) and throughout the entire interior regime L3–L14—increasing explained variance at Layer 6 by 54.5% relatively (: 9.58% vs. 6.20%), whereas direct prediction is marginally lower only at L1 (0.9420 vs. 0.9476), L2 (0.9635 vs. 0.9704), and L15 (0.9970 vs. 0.9977). Due to rebuttal compute constraints, this same-architecture control was conducted as a 16-layer fixed-budget sweep rather than full-budget bridge training with a frozen suffix; we will report this controlled sweep in the revised paper while softening our wording around the original MLP baseline so as not to overclaim full-budget causal superiority.
Q2: Can the authors test the geometry-based selection on at least one additional held-out backbone/model family? In particular, how does it compare with the simple rule of selecting the shallowest layer, given the weak or negative added value of geometry within the shallow Qwen layers?
A2: Yes. We evaluated our unmodified geometry criterion on allenai/OLMo-2-0425-1B (16 layers) across both the 16-layer fixed-budget sweep and full-budget bridge training with a frozen suffix comparing Layer 1 vs. Layer 6:
Metric on OLMo-2-0425-1B | Layer 0 (Shallowest) | Layer 1 (Shallow rule) | Layer 6 (Geometry-selected) | Best Observed Layer |
|---|---|---|---|---|
| Geometry score (↑, training-free) | −1.193 | −0.815 | +0.934 (Rank 1) | Layer 6 (16-layer) |
| Sweep Diffusion norm. MSE (↓) | 0.9705 | 0.9476 | 0.9042 (Rank 1) | Layer 6 (16-layer sweep) |
| Sweep Diffusion (↑) | 2.95% | 5.24% | 9.58% (Rank 1) | Layer 6 (16-layer sweep) |
| Full-budget NLL (↓, frozen suffix) | — | 5.10 | 4.96 | Layer 6 (L1 vs. L6) |
| Full-budget PPL (↓, frozen suffix) | — | 163.75 | 142.62 | Layer 6 (L1 vs. L6) |
| Full-budget Teacher KL (↓, frozen suffix) | — | 0.54 | 0.42 | Layer 6 (L1 vs. L6) |
Decisive win over the shallowest-layer heuristic in both the 16-layer sweep and full-budget L1-vs.-L6 evaluation: Unlike Qwen3-8B and Llama-3.1-8B, OLMo-2-0425-1B does not favor the shallowest layer in the fixed-budget sweep. Geometry guidance accurately captures this non-monotonic profile and selects the 16-layer sweep optimum (Layer 6) before training; under full-budget bridge training with a frozen suffix (evaluated on L1 vs. L6), Layer 6 reduces PPL by 21.13 points (−12.9%) and teacher KL divergence by 22.2% compared to Layer 1.
Strong positive value within shallow layers (L0–L7): Within the first eight layers (L0–L7), our geometry score achieves Spearman = +0.905 with bridgeability and reduces LOOCV RMSE by 36.3% over a quadratic depth baseline (0.01715 0.01093), demonstrating substantial predictive value precisely when shallow-layer heuristics fail.
Official Rebuttal [4/4]
Q3: Can the authors provide a comparison to the strongest continuous-diffusion baseline that better matches total active capacity or inference compute, or otherwise narrow the interpretation of Table 5? The current module-level budget matching does not isolate the effect of the interface.
A3: We address this in two ways: (1) training DiHAL-small from scratch on the smaller OLMo-2-0425-1B backbone to substantially shrink both the bridge and the retained pretrained suffix, and (2) explicitly narrowing the interpretation of Table 5 with transparent total-system parameter accounting:
| Method | Backbone | Trainable Params | Active Bridge / Module Params | Retained Frozen Backbone (Suffix + LM Head / + Embed) | Total Active System Params | Gen.PPL (↓) | Diversity (↑) |
|---|---|---|---|---|---|---|---|
| CoDAR | Standalone | 360.9M | 360.9M | — | 360.9M | 144.83 | 0.4777 |
| DiHAL (full) | Qwen3-8B (L2) | 862.7M | 946.3M | 6.99B / 7.61B | ~8.56B | 136.02 | 0.5913 |
| DiHAL-small | Qwen3-8B (L2) | ~257.4M | 341.1M | 6.99B / 7.61B | ~7.95B | 140.51 | 0.5230 |
DiHAL-small (OLMo-2-1B) | OLMo-2-0425-1B (L6) | ~255.9M | 339.5M | 809.6M / 1,015.1M | 1.35B (1.15B w/o embed) | 138.32 | 0.5812 |
Stronger generation quality with a smaller scratch-trained bridge and ~7.5× smaller suffix: Using the from-scratch DiHAL-small bridge (339.5M active bridge parameters at , smaller than CoDAR's 360.9M) at L6 on OLMo-2-0425-1B skips 7 decoder layers (469.8M) and retains only 9 upper layers (604.1M) plus the LM head (205.5M), reducing retained pretrained parameters by ~7.5× (~7.61B 1.02B with embeddings) and total system size by ~6.3× (~8.56B 1.35B) relative to full Qwen3-8B DiHAL. Notably, DiHAL-small (OLMo-2-1B) achieves Gen.PPL = 138.32 and Diversity = 0.5812—not only outperforming CoDAR (144.83 / 0.4777), but also surpassing DiHAL-small on the much larger Qwen3-8B (140.51 / 0.5230), indicating that performance is driven by interface compatibility rather than sheer frozen-suffix size.
Narrowed interpretation of Table 5: Even with DiHAL-small on OLMo-2-0425-1B, total active system capacity (1.35B including frozen suffix, embeddings, and VAE; 1.15B excluding input embeddings) and inference FLOPs remain larger than standalone CoDAR (360.9M). We therefore agree with the reviewer and will revise Section 4 / Table 5 to explicitly frame this comparison as a diagnostic evaluation under matched diffusion/recovery training budgets (300K examples, 40 H100 GPU-hours) rather than a total-capacity- or FLOPs-equalized benchmark, reporting full system-level parameter counts alongside module-level counts.
Official Rebuttal [1/4]
We thank the reviewer for their insightful feedback. We appreciate their recognition of our underexplored design question—treating the diffusion representation space as a design choice—along with our layer sweeps, depth-controlled analysis, and transparent evaluation.
A1: We thank the reviewer for this rigorous analysis. To directly test cross-family generalization, shallow-regime ranking, and depth confounding without retuning our criterion, we evaluated the fixed geometry score across all 16 layers of a held-out 1B backbone from a distinct model family (
allenai/OLMo-2-0425-1B).Non-trivial interior optimum and downstream gains over Layer 1 on a held-out family. Unlike Llama-3.1-8B and Qwen3-8B—where reconstruction error increases almost monotonically from Layer 1—
OLMo-2-0425-1Bexhibits a pronounced non-monotonic U-shaped bridgeability curve across depth. Empirical diffusion normalized MSE is worst at the boundaries (L0: 0.9705;L15: 0.9977) and reaches its global minimum at an interior layer, Layer 6 (norm. MSE = 0.9042, R2 = 9.58%), clearly outperforming Layer 1 (norm. MSE = 0.9476, R2 = 5.24%). Without any hyperparameter adjustment, our geometry score peaks at Layer 6 (+0.934 vs. −0.815 at Layer 1), selecting the exact fixed-budget sweep optimum. Crucially, under full-budget bridge training with a frozen suffix onOLMo-2-0425-1B(comparing Layer 6 against Layer 1), Geometry-selected Layer 6 decisively outperforms Layer 1 across every evaluation metric: NLL improves from 5.10 to 4.96, PPL drops from 163.75 to 142.62 (−12.9%), and teacher-student KL divergence drops from 0.54 to 0.42 (−22.2%).Strong correlation in both full-depth and shallow-layer (
L0–L7) regimes. Across all 16 layers ofOLMo-2-0425-1B, the geometry score achieves Spearman ρ = +0.965 and Kendall τ = +0.883 against negative normalized MSE. Crucially, restricting the evaluation to the first eight layers (L0–L7)—where bridgeability improves fromL0toL6against simple shallow-layer preference—the geometry score still maintains a strong Spearman ρ = +0.905, whereas a naive "shallower is better" heuristic is negatively aligned in this regime.Explicit depth-confounding controls (linear and quadratic depth). Because
OLMo-2-0425-1Bhas an interior optimum, we evaluated leave-one-out cross-validation (LOOCV) RMSE against both linear and quadratic depth baselines. Adding our geometry score consistently reduces prediction error over depth-only models:L0–L15, vs. linear depth): LOOCV RMSE drops from 0.03018 to 0.01152 (+61.8% relative improvement).L0–L15, vs. quadratic depth): LOOCV RMSE drops from 0.01490 to 0.01303 (+12.5% improvement) and from 0.01529 to 0.00955 (+37.5% improvement) excluding the extreme final-layer outlier (L0–L14).L0–L7, vs. quadratic depth): LOOCV RMSE drops from 0.01715 to 0.01093 (+36.3% improvement).