- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Direct Feature Refit (DFR) for recovering depth-pruned language models. Rather than fitting a transformation on the existing FFN branch output, DFR directly regresses the pruning-induced representation gap on the surviving intermediate FFN features and merges the resulting correction into the down projection. The paper derives the optimal rank-constrained solution under a ridge reconstruction objective and decomposes the gap between full-feature and readout-constrained recovery using a data-dependent projection geometry.
Critical strengths
-
A simple recovery mechanism with no additional deployment module.
-
Consistent perplexity improvements with useful controlled comparisons.
Critical weaknesses
- The motivation for permanent depth pruning is insufficiently established.
What deployment scenarios justify the capability loss incurred by permanently removing layers? Even after DFR recovery, Llama-3-8B’s WikiText-2 perplexity increases from 6.14 to 13.51, while its nine-task average accuracy drops from 67.59% to 56.23%. The paper does not establish that the deployment benefits justify this degradation, nor explain why this approach is preferable to directly using an available open-source smaller model under a comparable resource budget.
- Applicability to hybrid attention architectures remains unestablished.
The experiments cover conventional Transformer models but do not establish whether the pruning-and-recovery procedure transfers to hybrid attention architectures, such as Qwen3.5, which interleaves Gated DeltaNet and gated attention blocks. Since these block types may differ in both function and computational cost, the paper should clarify whether layer selection requires architecture-specific constraints and whether refitting a surviving FFN can adequately compensate for removing either type of block.
- The evidence supporting the central recovery-structure interpretation is incomplete.
The reported 94–96% NLL recovery is measured relative to the severely degraded pruned model, which can obscure how much of DFR’s additional benefit over a strong baseline is retained. Moreover, the analysis mainly covers 25% pruning, leaving its generality across pruning levels and calibration settings unestablished.
Questions for authors
see Weaknesses 1 2
2 comments
W3. Interpretation and generality of recovery structure
In the paper, we ask whether a small number of recovery components can deliver most of the NLL improvement achieved by full-rank DFR. So we choose the metric , which measures how much of DFR-Full's NLL improvement is retained as the recovery rank increases. The reported 94–96% indicates that rank 512 retains most of DFR-Full's NLL improvement over the unrecovered pruned model.
To complement this analysis, we report absolute NLL and introduce a strong-baseline reference, , where denotes a competing recovery method.
As shown in Table R6, at 25% pruning with WikiText-2 calibration, rank 512 retains 96.10% of DFR-Full's improvement over Pruned and 17.00% of its additional improvement over ReplaceMe-LS. These metrics provide complementary views of rank-constrained recovery.
Table R6. Rank-512 recovery under different NLL reference points. LS denotes ReplaceMe-LS.
| Pruning | Calibration | LS NLL ↓ | DFR-512 NLL ↓ | DFR-Full NLL ↓ | F_Pruned (%) | F_LS (%) |
|---|---|---|---|---|---|---|
| 25% | WikiText-2 | 3.52859 | 3.49064 | 3.30542 | 96.10 | 17.00 |
| 37.5% | WikiText-2 | 4.38351 | 4.30043 | 3.85392 | 93.10 | 15.69 |
| 50% | WikiText-2 | 5.95199 | 5.63307 | 5.18963 | 88.63 | 41.83 |
| 25% | C4 | 4.30961 | 4.24207 | 4.07378 | 95.77 | 28.64 |
| 25% | Mixed | 3.58645 | 3.54260 | 3.36563 | 96.22 | 19.86 |
We further examine whether the recovery structure observed in our main experiments holds under different pruning ratios and calibration sources. We focus on two properties: how much fitting improvement is captured by leading recovery components, and how much remains inaccessible to output-based corrections.
As shown in Table R7, across these settings, the leading 512 modes capture 84.97–91.17% of the fitting improvement, while 13.99–19.33% remains inaccessible to output-based corrections. Most of this inaccessible gain comes from later modes. The theoretical decomposition holds to numerical precision, while the relationship between fitting improvement and NLL recovery varies with pruning depth.
We have clarified the original recovery metric in Section 3.3 and added the complete baseline-relative and cross-setting analyses in Appendix D.5.
Table R7. Recovery structure across pruning ratios and calibration sources.
| Pruning | Calibration | Top-512 gain (%) | Inaccessible gain (%) | After 128 (%) |
|---|---|---|---|---|
| 37.5% | WikiText-2 | 86.47 | 17.22 | 71.95 |
| 50% | WikiText-2 | 91.17 | 13.99 | 63.61 |
| 25% | C4 | 84.97 | 19.33 | 71.60 |
| 25% | Mixed | 85.22 | 19.05 | 70.78 |
We thank the reviewer for the constructive feedback. We address the comments below.
W1. Motivation for depth pruning and comparison with native smaller models
Depth pruning reduces the number of sequential blocks in an existing pretrained model, providing a direct way to reduce inference latency without retraining. Since inference latency depends on both model size and sequential depth, depth pruning offers a distinct efficiency trade-off explored by prior work such as ShortGPT and LLM-Streamline.
We compare depth-pruned Qwen3-8B with native Qwen3-4B under the same inference measurement protocol.
As shown in Table R4, the native 4B model achieves higher quality and lower memory usage, while depth-pruned models achieve lower inference latency. At 50% pruning, DFR improves WikiText-2 NLL from 4.388 to 3.776 and nine-task accuracy from 36.98% to 41.76% over ReplaceMe-LS, with comparable inference cost.
These results show that DFR improves the quality–latency trade-off of depth pruning without gradient-based training or additional inference computation. The remaining quality gap motivates further research on recovery under aggressive depth reduction.
Table R4. Comparison of native smaller models and depth-pruned models.
W2. Applicability to hybrid-attention architectures
DFR directly uses intermediate FFN features and does not depend on the attention mechanism. We have added experiments on Qwen3.5-9B-Base, which combines Gated DeltaNet and full-attention layers, under 25% pruning with two layer-selection strategies.
As shown in Table R5, DFR-Full achieves the lowest perplexity on WikiText-2, PTB, and C4, and the highest nine-task accuracy under both pruning strategies. We have added representative cross-architecture results in Section 3.2 and the complete Qwen3.5 comparisons in Appendix E.4.
Table R5. Recovery performance on Qwen3.5-9B-Base under two pruning strategies.