acceptodds
Under review as a conference paper at ICLR 2027

DFR: Direct Feature Refit for Restoring Depth-Pruned Language Models

Abstract

Depth pruning reduces the inference cost of large language models, but removing Transformer blocks can substantially degrade model quality. Existing training-free methods often address this loss by fitting corrections on the low-dimensional FFN output. This makes fitting substantially cheaper, but can limit recovery quality when useful feature directions fall outside the subspace exposed by the pretrained down projection. We introduce **Direct Feature Refit (DFR)**, which instead fits the pruning-induced correction directly from the surviving intermediate FFN features and merges it into the existing down projection, requiring no fine-tuning or additional inference module. We further derive the globally optimal rank- solution to characterize how the useful correction is distributed across recovery modes. At rank 512, DFR retains 86.39–86.79% of the regularized recovery gain across the three main models, while recovering 94.06–96.47% of the full DFR NLL improvement, showing that most of the language-model improvement is captured by a moderate number of recovery modes. Across two pruning strategies and pruning ratios from 25% to 50%, DFR consistently improves post-pruning model quality on these models, reducing perplexity by up to 53.6% under aggressive pruning relative to the strongest competing baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, model compression, depth pruning, post-pruning recovery, feature accessibility, low-rank recovery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TkzkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Feature Refit (DFR) for recovering depth-pruned language models. Rather than fitting a transformation on the existing FFN branch output, DFR directly regresses the pruning-induced representation gap on the surviving intermediate FFN features and merges the resulting correction into the down projection. The paper derives the optimal rank-constrained solution under a ridge reconstruction objective and decomposes the gap between full-feature and readout-constrained recovery using a data-dependent projection geometry.

Critical strengths
  1. A simple recovery mechanism with no additional deployment module.

  2. Consistent perplexity improvements with useful controlled comparisons.

Critical weaknesses
  1. The motivation for permanent depth pruning is insufficiently established.

What deployment scenarios justify the capability loss incurred by permanently removing layers? Even after DFR recovery, Llama-3-8B’s WikiText-2 perplexity increases from 6.14 to 13.51, while its nine-task average accuracy drops from 67.59% to 56.23%. The paper does not establish that the deployment benefits justify this degradation, nor explain why this approach is preferable to directly using an available open-source smaller model under a comparable resource budget.

  1. Applicability to hybrid attention architectures remains unestablished.

The experiments cover conventional Transformer models but do not establish whether the pruning-and-recovery procedure transfers to hybrid attention architectures, such as Qwen3.5, which interleaves Gated DeltaNet and gated attention blocks. Since these block types may differ in both function and computational cost, the paper should clarify whether layer selection requires architecture-specific constraints and whether refitting a surviving FFN can adequately compensate for removing either type of block.

  1. The evidence supporting the central recovery-structure interpretation is incomplete.

The reported 94–96% NLL recovery is measured relative to the severely degraded pruned model, which can obscure how much of DFR’s additional benefit over a strong baseline is retained. Moreover, the analysis mainly covers 25% pruning, leaving its generality across pruning levels and calibration settings unestablished.

Questions for authors

see Weaknesses 1 2

2 comments

We thank the reviewer for the constructive feedback. We address the comments below.

W1. Motivation for depth pruning and comparison with native smaller models

Depth pruning reduces the number of sequential blocks in an existing pretrained model, providing a direct way to reduce inference latency without retraining. Since inference latency depends on both model size and sequential depth, depth pruning offers a distinct efficiency trade-off explored by prior work such as ShortGPT and LLM-Streamline.

We compare depth-pruned Qwen3-8B with native Qwen3-4B under the same inference measurement protocol.

As shown in Table R4, the native 4B model achieves higher quality and lower memory usage, while depth-pruned models achieve lower inference latency. At 50% pruning, DFR improves WikiText-2 NLL from 4.388 to 3.776 and nine-task accuracy from 36.98% to 41.76% over ReplaceMe-LS, with comparable inference cost.

These results show that DFR improves the quality–latency trade-off of depth pruning without gradient-based training or additional inference computation. The remaining quality gap motivates further research on recovery under aggressive depth reduction.

Table R4. Comparison of native smaller models and depth-pruned models.

ModelParams (B)Wiki NLL ↓QA ↑Latency (ms) ↓GPU (GiB) ↓
Qwen3-8B Dense8.1911.94668.40693715.78
Qwen3-4B Native4.0222.06766.2068718.01
8B, 25% + DFR6.4543.03053.50522912.47
8B, 38.89% + DFR5.4893.06244.41455210.63
8B, 50% + ReplaceMe-LS4.7184.38836.9834949.17
8B, 50% + DFR4.7183.77641.7635399.17

W2. Applicability to hybrid-attention architectures

DFR directly uses intermediate FFN features and does not depend on the attention mechanism. We have added experiments on Qwen3.5-9B-Base, which combines Gated DeltaNet and full-attention layers, under 25% pruning with two layer-selection strategies.

As shown in Table R5, DFR-Full achieves the lowest perplexity on WikiText-2, PTB, and C4, and the highest nine-task accuracy under both pruning strategies. We have added representative cross-architecture results in Section 3.2 and the complete Qwen3.5 comparisons in Appendix E.4.

Table R5. Recovery performance on Qwen3.5-9B-Base under two pruning strategies.

SelectorMethodWiki ↓PTB ↓C4 ↓QA ↑
LLM-StreamlinePruned108.9095.06114.1951.59
LinearPatch-R41.5454.0945.7357.64
ReplaceMe-LS22.3769.2145.7157.11
ReplaceMe-Cos27.0875.6839.4557.40
DFR-Full17.7350.0638.1858.25
ShortGPT (balanced)Pruned20.5036.7228.3153.98
LinearPatch-R18.8034.2425.9052.49
ReplaceMe-LS10.5322.9618.8454.14
ReplaceMe-Cos10.8023.6119.5054.94
DFR-Full9.6420.4317.2155.45

W3. Interpretation and generality of recovery structure

In the paper, we ask whether a small number of recovery components can deliver most of the NLL improvement achieved by full-rank DFR. So we choose the metric FPruned(r)=(LPruned−Lr)/(LPruned−LFull)F_{\mathrm{Pruned}}(r)=(L_{\mathrm{Pruned}}-L_r)/(L_{\mathrm{Pruned}}-L_{\mathrm{Full}}), which measures how much of DFR-Full's NLL improvement is retained as the recovery rank increases. The reported 94–96% indicates that rank 512 retains most of DFR-Full's NLL improvement over the unrecovered pruned model.

To complement this analysis, we report absolute NLL and introduce a strong-baseline reference, FB(r)=(LB−Lr)/(LB−LFull)F_B(r)=(L_B-L_r)/(L_B-L_{\mathrm{Full}}), where BB denotes a competing recovery method.

As shown in Table R6, at 25% pruning with WikiText-2 calibration, rank 512 retains 96.10% of DFR-Full's improvement over Pruned and 17.00% of its additional improvement over ReplaceMe-LS. These metrics provide complementary views of rank-constrained recovery.

Table R6. Rank-512 recovery under different NLL reference points. LS denotes ReplaceMe-LS.

PruningCalibrationLS NLL ↓DFR-512 NLL ↓DFR-Full NLL ↓F_Pruned (%)F_LS (%)
25%WikiText-23.528593.490643.3054296.1017.00
37.5%WikiText-24.383514.300433.8539293.1015.69
50%WikiText-25.951995.633075.1896388.6341.83
25%C44.309614.242074.0737895.7728.64
25%Mixed3.586453.542603.3656396.2219.86

We further examine whether the recovery structure observed in our main experiments holds under different pruning ratios and calibration sources. We focus on two properties: how much fitting improvement is captured by leading recovery components, and how much remains inaccessible to output-based corrections.

As shown in Table R7, across these settings, the leading 512 modes capture 84.97–91.17% of the fitting improvement, while 13.99–19.33% remains inaccessible to output-based corrections. Most of this inaccessible gain comes from later modes. The theoretical decomposition holds to numerical precision, while the relationship between fitting improvement and NLL recovery varies with pruning depth.

We have clarified the original recovery metric in Section 3.3 and added the complete baseline-relative and cross-setting analyses in Appendix D.5.

Table R7. Recovery structure across pruning ratios and calibration sources.

PruningCalibrationTop-512 gain (%)Inaccessible gain (%)After 128 (%)
37.5%WikiText-286.4717.2271.95
50%WikiText-291.1713.9963.61
25%C484.9719.3371.60
25%Mixed85.2219.0570.78
Reviewer_MSNeOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes DFR, which recovers depth-pruned language models by using ridge regression to fit the pruning-induced representation gap directly from surviving intermediate FFN features. The correction is merged into the existing down projection, adding no inference overhead. The paper also derives an optimal rank-constrained solution and analyzes which recovery directions are inaccessible to output-space fitting.

Critical strengths

S1. Simple yet strong feature refitting idea: The idea of directly refitting the surviving FFN down projection from its intermediate features is simple and conceptually appealing, delivering strong performance.

Critical weaknesses

W1. Clarity and organization. The core feature-refitting idea is reasonable, but the connections between the method’s components could be explained more clearly. In particular, the motivation for introducing the globally optimal rank-rr solution, the interpretation of “recovery modes,” and the relationship between DFR and iterative pruning are difficult to follow. Clearer explanations and transitions would help readers understand how these analyses and extensions support the central contribution.

Questions for authors

Q1. Could authors please elaborate the connection between section 2.2 and 2.3? Looks like there is a gap between DFR and global optimal rank solution; For me it is very hard to bridge this gap within the manuscript, and is not easy to follow.

Q2. Would DFR generalize to other arcthiectures (e.g., Qwen 3.5 with hybrid attention), how would one extend DFR to hybrid-attention architectures? Also, I suggest authors to include other architecture results on the main text, not in appendix (it would strengthen the manuscript, validating the generalizability of DFR)

Q3. How would the change in calibration data (Wikitext) affect the performance of DFR? Since it is directly re-fitting the features which heavily depends on the calibration set, I wonder how this could change the performance on donwstream tasks. Does it overfit to Wikitext? How is C4 ppl measured after calibrating with Wikitext + DFR?

Q4. Could authors give short lay-summary of proposed method, highlighting the strength compared to existing apporaches, and the core problem they tackled to achieve such strong performance? I would like to see the revised, clarified manuscript afterwards.

6 comments

We thank the reviewer for the helpful comments. We address the comments below.

Q1 / W1. Connection between DFR, rank-constrained recovery, and iterative pruning

We have revised Section 2 and Appendix A to clarify the connections between DFR, rank-constrained recovery, and iterative pruning. The revised sections are organized as follows:

Sections 2.1–2.2 introduce DFR. Output-based methods such as ReplaceMe correct the FFN output after the original down projection, so their corrections are limited to the projected features. DFR directly refits the down projection using intermediate FFN features.

Section 2.3 derives the optimal rank-rr correction and quantifies the fitting loss from rank reduction. Section 2.4 compares Full DFR with the best output-constrained correction under the same ridge objective, quantifying the additional fitting improvement enabled by intermediate FFN features.

Section 2.5 presents iterative pruning as an optional extension. It applies DFR after each layer removal, so that the next pruning decision uses the recovered model.

The following excerpts illustrate the revisions to Section 2:

Section 2 overview:

We first define the post-pruning recovery problem and introduce DFR, which directly refits the retained FFN down projection using intermediate FFN features (Sections 2.1–2.2). We then derive the optimal rank-rr correction and measure how much of the Full DFR fitting improvement it retains (Section 2.3). We further compare Full DFR with the best output-constrained correction under the same ridge objective to quantify the additional improvement enabled by intermediate FFN features (Section 2.4). Finally, we present iterative pruning as an optional extension, applying DFR after each layer removal before the next pruning decision (Section 2.5).

Section 2.3:

After obtaining the Full DFR solution, we derive the optimal rank-rr correction to measure how much of the Full fitting improvement can be retained using fewer correction components (Izenman, 1975).

Section 2.4:

The rank-constrained analysis measures the fitting improvement contributed by each recovery mode. We next examine how much of this improvement can be achieved through the original FFN output projection, as in output-based methods such as ReplaceMe.

Q2. Generalization to hybrid-attention architectures and additional model results

We have added experiments on Qwen3.5-9B-Base, which combines Gated DeltaNet and full-attention layers. DFR directly refits the retained FFN features without changing the attention mechanisms. We evaluate two layer-selection strategies under 25% pruning.

As shown in Table R2, DFR-Full achieves the lowest perplexity on all three datasets and the highest nine-task average under both pruning strategies.

We have added representative results on Qwen3-8B, Mistral-7B, and Qwen3.5-9B to Section 3.2, with complete hybrid-architecture results in Appendix E.4.

Table R2. Recovery performance on Qwen3.5-9B-Base under two pruning strategies.

SelectorMethodWiki ↓PTB ↓C4 ↓QA ↑
ReferenceDense6.9811.3810.7770.30
LLM-StreamlinePruned108.9095.06114.1951.59
LinearPatch-D67.3885.3585.7756.43
LinearPatch-R41.5454.0945.7357.64
Prune&Comp55.3873.6961.4157.43
ReplaceMe-LS22.3769.2145.7157.11
ReplaceMe-Cos27.0875.6839.4557.40
DFR-51222.6760.7647.1954.82
DFR-Full17.7350.0638.1858.25
ShortGPT (balanced)Pruned20.5036.7228.3153.98
LinearPatch-D15.0328.7621.2554.70
LinearPatch-R18.8034.2425.9052.49
Prune&Comp26.6044.0132.8550.08
ReplaceMe-LS10.5322.9618.8454.14
ReplaceMe-Cos10.8023.6119.5054.94
DFR-51210.5923.0719.1153.77
DFR-Full9.6420.4317.2155.45

Q3. Calibration-data sensitivity and C4 evaluation protocol

We have evaluated calibration sensitivity on DeepSeek-R1-Distill-Llama-8B under 25% LLM-Streamline pruning, using WikiText-2, C4, and mixed calibration with the same budget of 1,024 sequences.

As shown in Table R3, calibration data affects absolute recovery quality. DFR-Full outperforms ReplaceMe-LS across all four metrics under all three calibration settings.

We have added the complete seven-method comparison in Appendix B.6. We have also clarified the C4 evaluation protocol in Appendix B.4: perplexity is evaluated on 512 non-overlapping validation windows of 2,048 tokens, with calibration data drawn separately from the training split.

Table R3. Recovery performance under different calibration data sources.

CalibrationMethodWiki ↓PTB ↓C4 ↓QA ↑
WikiText-2ReplaceMe-LS34.0874.9866.2753.11
DFR-Full27.2668.7052.8753.82
C4ReplaceMe-LS74.4186.7351.1052.16
DFR-Full58.7867.3340.9153.78
MixedReplaceMe-LS36.1172.0852.5452.56
DFR-Full28.9558.8741.8953.77

Q4. Intuitive explanation of DFR

Output-based methods such as ReplaceMe correct the FFN output after the original down projection. Their corrections are therefore limited to the feature directions available through WdW_d. DFR instead directly refits the down projection using intermediate FFN features ZZ, allowing it to use additional feature directions.

The fitted correction is merged into the original FFN weights, requiring no gradient-based fine-tuning or additional inference computation.

We have clarified this distinction in Sections 1 and 2. The revised Section 2.2 states:

Direct Feature Refit (DFR) instead fits a correction directly from the intermediate FFN features ZZ, without constraining it to the original output-projection directions.

The post-output correction satisfies col⁡(ΔT)⊆col⁡(Wd)\operatorname{col}(\Delta_T)\subseteq\operatorname{col}(W_d), whereas DFR imposes no such restriction on Δ\Delta.

Reviewer_MSNeOpenReview ↗

Response to rebuttal

Thank you for the detailed responses and additional experiments, particularly the Qwen3.5 results, which help address my question about applicability to hybrid-attention architectures.

I have two remaining points:

Revised manuscript. Could you please point me to the revised manuscript, or upload it if it has not yet been made available? I could not locate the version incorporating the changes described in your responses. Since presentation and clarity were central concerns in my review, I would appreciate reading the revisions in context.

Calibration sensitivity and generalization. My original question aimed to understand whether directly refitting the features overfits to the calibration distribution. Thank you for including the cross-dataset results in Table R3, including PTB evaluation after WikiText-2 calibration. Could you elaborate on their implications? For example, mixed calibration gives the best PTB perplexity, while WikiText-2 and C4 calibration favor their respective evaluation datasets. How should we interpret this pattern in terms of distribution-specific fitting versus transferable recovery?

For downstream QA, the reported DFR-Full averages are nearly identical across calibration sources (53.82, 53.78, and 53.77). Would it be fair to conclude that QA performance is relatively insensitive to calibration source in this setting, despite the larger perplexity differences? Is there evidence that any calibration source consistently gives better downstream performance, or are these differences within sampling variability?

Reviewer_KWfeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

This paper proposes a training-free method called Direct Feature Refit (DFR) for recovering depth-pruned LLMs. By fitting corrections directly from intermediate FFN features, DFR overcomes the feature-space limitations of output-based recovery. The authors further derive an optimal rank-constrained solution and analyze recovery-mode accessibility. Experiments demonstrate substantial improvements in perplexity across multiple models and pruning settings.

Critical strengths
  1. DFR achieves strong recovery performance without fine-tuning or additional inference overhead.

  2. The recovery-spectrum and feature-accessibility analyses provide useful insights into the limitations of constrained linear recovery.

Critical weaknesses
  1. Ridge-regression compensation and low-rank recovery have been explored in prior work, such as Olica (He & Lin, 2025) and ReplaceMe. The optimal rank-constrained solution also follows classical reduced-rank regression. A more detailed comparison in the main text with existing reconstruction-based approaches, including Olica, ReplaceMe, would help clarify the paper's contributions.

  2. Several key variables, such as (X), (Y), and (Z), are introduced primarily through textual descriptions. More rigorous mathematical definitions, explicitly specifying their relationships to the corresponding Transformer layers and pruning boundaries, would improve the clarity and readability of the paper.

Questions for authors

Full-feature ridge regression can be computationally expensive. How does DFR compare with existing methods in calibration time, memory requirements, and recovery quality under matched computational budgets?

5 comments

We thank the reviewer for the constructive feedback. We address the comments below.

W1. Novelty and comparison with prior work

DFR identifies and overcomes a feature-accessibility limitation in post-pruning recovery, building on established ridge and reduced-rank regression techniques. We also provide theoretical analyses to further distinguish DFR from prior work. Section 2.3 derives the optimal rank-rr correction and quantifies the fitting loss from rank reduction. Section 2.4 derives an exact decomposition of the fitting gap between Full DFR and the best output-constrained correction. Figures 1 and 3, together with downstream evaluations, demonstrate the practical impact of this feature-space restriction.

We have expanded Section 4 to clarify the differences from ReplaceMe and Olica.

Revised Section 4 (excerpt):

ReplaceMe (Shopkhoev et al., 2025) fits linear transformations after the retained FFN down projection and merges them into the model. Its corrections are restricted to feature directions in col⁡(Wd)\operatorname{col}(W_d). DFR instead fits the correction directly from intermediate FFN features, removing this restriction. Olica (He & Lin, 2025) uses ridge regression and low-rank approximation for structured width pruning. Our work focuses on depth-pruning recovery and quantifies the additional fitting improvement enabled by intermediate FFN features under the same ridge objective.

W2. Mathematical definitions and clarity

We have revised Section 2.1 to clarify the pruning boundary, FFN features, and recovery target for a removed block interval [ℓ,u)[\ell,u). We have also updated Fig. 2 to show the boundary representations, fitted correction, and weight merging.

Revised Section 2.1 (excerpts):

Let R∈RN×dR\in\mathbb{R}^{N\times d} denote the residual stream entering the FFN sublayer of block ℓ−1\ell-1, and let H∈RN×dH\in\mathbb{R}^{N\times d} be its normalized FFN input. The corresponding intermediate nonlinear features are denoted by Z∈RN×mZ\in\mathbb{R}^{N\times m}.

With the pretrained down projection Wd∈Rm×dW_d\in\mathbb{R}^{m\times d}, the output of the retained block is X=R+ZWdX=R+ZW_d.

Let Y∈RN×dY\in\mathbb{R}^{N\times d} denote the corresponding hidden representation produced by the dense model after block u−1u-1, evaluated on the same calibration inputs and token positions. Removing the interval causes the subsequent retained computation to receive XX instead of YY. We therefore define the pruning-induced representation gap as D=Y−XD=Y-X.

Q1. Calibration efficiency and matched computational budgets

We have added experiments comparing calibration time, peak GPU memory, and recovery quality on DeepSeek-R1-Distill-Llama-8B under 25% LLM-Streamline pruning.

As shown in Table R1, with 256 sequences, DFR-Full outperforms both ReplaceMe variants using 1,024 sequences across all quality metrics, while requiring less fitting time. At the same calibration size, DFR-Full requires more fitting time than ReplaceMe-LS and higher peak GPU memory. These results demonstrate that DFR can achieve stronger recovery with fewer calibration examples and lower total fitting time.

We have expanded Section 3.5 and added the complete seven-method, five-budget comparison and measurement protocol in Appendix H.

Table R1. Calibration time, GPU memory, and recovery quality under different calibration budgets.

MethodSeqs.Time (s) ↓GPU (GiB) ↓Wiki ↓PTB ↓C4 ↓QA ↑
ReplaceMe-LS512311.216.2434.1575.2666.0453.05
ReplaceMe-LS1024626.416.2434.0874.9866.2753.11
ReplaceMe-Cos5121035.216.0858.92121.76315.3153.33
ReplaceMe-Cos10242068.416.0851.47109.44235.4353.14
DFR-Full256410.620.9027.9470.2953.6253.59
DFR-Full10241564.820.9027.2668.7052.8753.82

I appreciate the authors' detailed responses and additional experiments, which partially address my concerns. The fact that DFR targets depth pruning while Olica focuses on width pruning does not, by itself, constitute a methodological distinction, since DFR and existing works [1–3] both use ridge regression and low-rank compensation. Could you position your contributions relative to these prior works in Section 2.1?

References

[1] Olica: Efficient Structured Pruning of Large Language Models without Retraining.

[2] CORP: Closed-Form One-shot Representation-Preserving Structured Pruning.

[3] GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networks.

Thank you for the follow-up and for highlighting the connections to Olica, CORP, and GRAIL. Your comments helped us improve the positioning of our work.

We have further revised Sections 2.1 and 4 to clarify DFR's methodological contributions relative to these works. The main distinctions are as follows:

  1. Recovery target. DFR directly fits the representation gap across removed Transformer blocks. Olica, CORP, and GRAIL primarily recover information lost through local structured compression. This targets the combined effect of the removed attention and FFN computations while preserving the inference-latency benefit of reduced sequential depth.

  2. Recovery parameterization. DFR uses the intermediate features of the surviving FFN to fit the correction, which is merged into the original down projection. This avoids reconstructing removed intermediate activations and allows corrections beyond the feature directions exposed by the original down projection.

  3. Recovery analysis. We analyze how rank constraints and feature-space restrictions affect recovery. Section 2.3 derives the optimal rank-rr correction and quantifies the fitting loss from rank reduction. Section 2.4 derives the exact fitting gap between Full DFR and the best output-constrained correction under the same ridge objective.

We have uploaded the latest revised manuscript, with the main changes highlighted in blue. The following excerpts show the revisions to Sections 2.1 and 4.

Revised Section 2.1:

Related regression-based formulations have also been explored for post-pruning compensation. Olica fits the residual errors of width-pruned FFNs from their inputs and uses low-rank compensation. CORP reconstructs removed MLP activations from retained ones and compensates pruned attention contributions. GRAIL reconstructs original activations from compressed features and merges the correction into downstream weights.

Our formulation directly fits the representation gap across removed Transformer blocks, which reflects the combined effect of their attention and FFN computations. It uses the intermediate features of the surviving FFN to fit the correction. The resulting correction is merged into its original down projection, without reconstructing removed intermediate activations or adding inference computation. This provides a broader correction space than post-output linear recovery.

We have also revised Section 4 to clarify the differences from prior work and the contributions of our recovery analysis.

Revised Section 4:

Regression-based compensation has also been explored for structured compression. Olica (He & Lin, 2025) fits the residual errors of width-pruned FFNs from their inputs and uses low-rank compensation. CORP reconstructs removed MLP activations from retained ones and compensates pruned attention contributions. GRAIL (Tang et al., 2026) reconstructs original activations from compressed features and merges the correction into downstream weights. LaCo (Liu et al., 2026) sequentially reconstructs hidden states through layer-wise compensation.

ReplaceMe (Shopkhoev et al., 2025) fits linear transformations after the retained FFN down projection and merges them into the model. Its corrections are restricted to feature directions in col⁡(Wd)\operatorname{col}(W_d). DFR directly fits the representation gap across removed Transformer blocks using the intermediate features of the surviving FFN, allowing corrections beyond this restriction. We further quantify the fitting loss from rank reduction and the exact gap between Full DFR and the best output-constrained correction under the same ridge objective.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. Below we summarize the main revisions and additional experiments.

1. Methodological clarification and contributions.

We revised Section 2 to clarify the pruning boundary, mathematical definitions, and connections between Full DFR, rank-constrained recovery, and feature accessibility. We also expanded Sections 2.1 and 4 to explain the methodological differences from ReplaceMe, Olica, CORP, and GRAIL, focusing on the recovery target, parameterization, and analysis of rank and feature-space restrictions.

2. Additional experiments and analyses.

We conducted additional experiments to examine:

  • Calibration efficiency: Fitting time, GPU memory, and recovery quality under matched calibration budgets.
  • Cross-architecture generalization: Recovery on Qwen3.5-9B-Base with hybrid attention under two pruning strategies.
  • Calibration sensitivity: WikiText-2, C4, and mixed calibration, together with repeated experiments examining downstream QA variability.
  • Recovery structure: Rank-constrained recovery and feature accessibility across different pruning ratios and calibration sources.
  • Native smaller models: Quality and inference latency comparisons between depth-pruned models and native smaller models.

3. Revised manuscript.

We have uploaded the revised manuscript with the main changes highlighted in blue to facilitate review. The additional experiments and analyses are included in the revised paper and our discussion responses.

We sincerely appreciate the reviewers' feedback, which has helped us improve the clarity and completeness of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.