acceptodds
Under review as a conference paper at ICLR 2027

WHAT DOES FFN COMPRESSION CHANGE DOWNSTREAM? SAME-STATE CAUSAL RESTORATION IN DIFFUSION LANGUAGE MODELS

Abstract

Diffusion language models (DLMs) enable flexible, parallel generation, but their iterative denoising remains computationally expensive, motivating increasingly aggressive compression. Existing compression objectives largely measure how well compressed computation approximates the original locally, but local error does not reveal which removed computations actually matter to the downstream denoising trajectory. We introduce Same-State Causal Restoration (SSR), which restores the original FFN on the exact current input reached by the compressed model and measures how the resulting trajectory changes. To our knowledge, this is the first direct measurement of the same-current-input closed-loop effect of removed FFN computation in DLM compression. Across LLaDA-8B-Instruct and Dream-v0-Instruct-7B, compressed-side state ranks this downstream effect substantially better than local NMSE at fixed denoising phase, while controlled interventions show that correction structure matters beyond magnitude. Using task-label-free calibration, SSR freezes a single restoration window for held-out inference. Under aggressive LLaDA compression, restoring only four transitions recovers 89.9% of the lost accuracy while retaining an estimated 36.8% whole-model MAC saving and outperforming an equal-budget local-error baseline. Dream further shows that restoring dense behavior and repairing the final task are distinct outcomes.

Primary area:
generative models
Keywords:
diffusion language models, LLM compression, FFN compression, model compression, causal intervention, mechanistic analysis, adaptive computation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Fq1jOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SSR to selectively restore full FFN computation at important denoising steps in compressed diffusion language models, aiming to recover model performance with limited additional computation. Overall, the motivation is interesting, but the experimental evaluation is too limited to convincingly demonstrate the generality and robustness of SSR.

Critical strengths
  1. Interesting motivation. The motivation of this paper is interesting. It investigates whether selectively disabling compression at a small number of denoising steps and restoring the original full forward computation can recover most of the performance of the uncompressed model while using substantially less computation overall. This provides an interesting perspective on how computation should be allocated across different denoising steps in diffusion language models.
Critical weaknesses

The main issue is that the experiments are not sufficiently comprehensive to demonstrate the effectiveness and robustness of the proposed method.

  1. Limited practicality of the compression method. Existing DLM compression methods such as Quant-dLLM have already been published, while the experiments in this paper are mainly based on a self-designed FFN compression method that is not compared with existing methods. Since SSR relies on an underlying compression method, the paper lacks evidence that SSR can generalize to mainstream DLM compression methods.

  2. The experimental setting is relatively limited and simple. Mainstream DLM works often evaluate on reasoning tasks such as mathematics and coding with 256 or more denoising steps, while this paper only uses 32 steps on relatively simple commonsense tasks.

There are also some presentation issues, such as the confusing first column of Table 1 and overlapping numbers in Figure 3.

Overall, the motivation is interesting, but the current experiments are insufficient to convincingly demonstrate the generality and robustness of SSR.

Questions for authors

as in weaknesses

Reviewer_ESyKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Same-State Causal Restoration (SSR) to measure how FFN computation removed by compression affects subsequent denoising trajectories. Starting from a state reached by the compressed model, SSR restores the original FFN on the same current FFN input and compares the resulting closed-loop continuation. The authors select and freeze a global restoration window using source data without task-correctness labels. On 297 evaluation examples, the dense, compressed, SSR, and NMSE accuracies for LLaDA c24 are 58.6%, 21.9%, 54.9%, and 49.8%, respectively. SSR and NMSE are close on c14; on Dream, task-repair evidence is inconclusive, although trajectory fidelity improves.

Critical strengths
  1. Comparing the compressed and original FFNs on the same current input isolates operator substitution from changes in the preceding trajectory, while the continuation measures its closed-loop effect.
  2. Window selection does not use task-correctness labels and is compared with an NMSE-selected window. The paper also reports paired statistical tests and several diagnostic experiments.
Critical weaknesses
  1. The compression testbed primarily uses structured FFN-width slicing. Evaluation is concentrated on two model families, deterministic short outputs, and mostly multiple-choice tasks. Generalization to other compression methods, longer generations, and open-ended tasks remains unclear.

  2. The original FFN weights must remain resident during inference, and the MAC savings are analytical estimates rather than measured latency or memory reductions. The results therefore support SSR as a causal diagnostic, but do not establish practical deployment acceleration.

  3. Both evaluated model families, LLaDA and Dream, are masked diffusion language models. Block diffusion models such as LLaDA2.X and SDAR are an increasingly prominent and potentially more application-relevant paradigm, but SSR is not evaluated on them. Since blockwise generation may alter the denoising trajectory and the effect of selectively restoring FFNs, the current experiments do not establish that the findings transfer to this setting.

Questions for authors

See Weakness

Reviewer_uspTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

There is broad interest in speeding up diffusion language models. One approach is to run a smaller compressed feed-forward network (FFN) in some of the diffusion denoising passes. This has been studied before, but not looking carefully at the downstream effect of choosing the compressed or uncompressed model for that pass. Just measuring local error doesn’t tell you what the missing computation does to the rest of the denoising process, because errors can be absorbed or amplified in later passes. This paper's approach, Same-State Causal Restoration (SSR), studies this phenomenon by running both the compressed and uncompressed versions on a particular branch state, and measuring the compression-specific downstream-effect. They find that negative release margin ranks downstream effect more strongly than local NMSE. They argue for distinguishing the effects of local mismatch, downstream consequences, and future repair.

Critical strengths

Clear motivation, good insight separating the three factors. Clear careful experiments.

Nice experimental result: compression with SSR recovers 89.9% of the lost uncompressed accuracy while retaining an estimated 36.8% of whole-model MAC saving relative to dense.

Critical weaknesses

SSR doesn't reduce model-weight memory footprint, since the full model stays in memory. It doesn't seem that the paper measures actual wall-clock savings, only MAC counts. Especially with memory the same, and others finding that reduced FLOPs translated into speedup less than one might expect, it leaves me thirsting for wall clock measurements.

There are many ways to make DLMs more efficient, and a large landscape of current work, including ones that change the way the model itself is trained. While I understand that the landscape is too large to expect direct comparisons, I'm left wondering about the expected longer-term impact of SSR in the evolving landscape.

I found the paper a bit more difficult to read than necessary, for example hunting for the meaning of "c14" at the first mention, essentials of compression and compute accounting in the Appendix, many non-general technical terms used in the abstract or introduction not defined or not defined until significantly later.

Questions for authors

What does SSR add beyond choosing which steps to repair? How much of its gain comes specifically from SSR-based selection rather than simply restoring a reasonably-chosen 4-transition window? For example, uniform random sampling, or various fixed positions.

Can you show measured wall-clock latency and memory, given that original weights stay resident?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.