- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Nested FFN, in which prefill uses only a prefix subset of FFN channels while decode uses the full FFN, with both phases sharing the same parameters. The FFN is fine-tuned with a simulated prompt boundary. Two variants are studied: a uniform width across layers, and per-layer widths learned under an average-width budget via a straight-through mask. Evaluation shows that layer-adaptive widths preserve quality at 50-62.5% average width, while uniform reduction degrade more.
Critical strengths
- Clearly written and easy to follow, with well-posed questions (Q1, Q2) that the experiments address to a reasonable extent.
- The learned profile (more width in early layers, almost none in late layers) is an interesting finding.
- The inference form needs only dense prefix slices, with no input-dependent routing.
Critical weaknesses
- The nested, shared-parameter FFN comes from MatFormer, which is credited only in Related Work, while the contributions say "we develop." Width pruning is also well studied (in LLMs and in CNNs). The contribution reduces to applying it to prefill and learning per-layer widths, and the results are not very surprising.
- Reduced models exceed Vanilla in several settings (e.g., 102.6% on Qwen3-0.6B at 62.5%, 103.0% on Llama-3B at 50%). The paper reports these numbers without explanation. Vanilla might be not trained with the same procedure, so it might be not a perfectly matched control.
- There is no analysis of why later layers are more redundant.
- Savings are argued from FFN FLOPs only. There is no latency or throughput measurement, and attention and the KV cache are unchanged. Slicing weights often does not translate into wall-clock gains, even with channels permuted into a contiguous prefix. The savings apply only to the first P- prompt tokens, so they shrink for short prompts, and uneven per-layer shapes and batched serving with mixed prompt lengths are untested. Training cost relative to Vanilla is also not reported.
Questions for authors
- Why is width reduction applied only to prefill and not decode? Would reduced-width decode also work, and how would it compare?
- The method is applied to pretrained checkpoints. What happens when training from scratch, as MatFormer does?
- Unlike MatFormer, the prefill width is static and only the per-layer widths are optimized. Can channels be ordered by importance, as in prior CNN pruning work, to give a stronger Uniform baseline?
- What are the measured latency and throughput on a real batch serving scenario with mixed prompt lengths?
5 comments
W4 + Q4 — Latency, throughput, and training cost
Efficiency measurement. We measured prefill latency and throughput of Layer-adaptive Width Reduction on one NVIDIA GH200 with batches of mixed prompt lengths. We left-pad the prompts so that their final ω positions are aligned (ω = 512). Throughput counts only non-padding tokens. Vanilla and Nested FFN both use FlashAttention-3, which skips padding in attention, and are timed on the same batch.
| Model | Width | Batch | Latency (Vanilla / Nested FFN) | Throughput (Vanilla / Nested FFN) | Speedup |
|---|---|---|---|---|---|
| Llama-3.2-3B | 62.5% | 8 prompts, 4K to 32K tokens | 3.82 s / 3.12 s (−18.4%) | 38.6K / 47.2K tok/s | 1.23× |
| Qwen3-4B | 62.5% | 4 prompts, 8K to 32K tokens | 2.92 s / 2.47 s (−15.5%) | 28.0K / 33.2K tok/s | 1.18× |
Regarding uneven per-layer shapes, we agree with the reviewer that they may limit achievable speedups, as computation is unevenly distributed across layers and may require further system-level optimization. Our primary focus is to establish that FFN width can be substantially reduced during prefill without degrading model performance. Several prior studies, such as YOCO, have demonstrated this possibility, albeit with some performance degradation. Translating this finding into practical system-level speedups warrants a comprehensive investigation in future work. For example, in MoE settings, this could reduce the number of active experts, thereby lowering the memory footprint and improving system utilization.
Training cost. On Qwen3-0.6B, fine-tuning for 2000 steps at 32K context on 8 A100 GPUs takes about 5.5 hours for Vanilla, and about 7.0 hours for Layer-adaptive Width Reduction. We will add a discussion of training cost in the revision.
Q1 — Why not reduce decode width
Compared with prefill, decoding generally requires greater per-token computation to generate new tokens. Reducing FFN computation during decoding therefore involves tuning hyperparameters in Transformers, such as FFN width and hidden-state dimensionality, which is a more general topic on model optimization.
Q2 — Training from scratch
Training width-elastic models from scratch, as in MatFormer, might yield even better results by allowing the selected channels to specialize in state computation rather than decoding. This is a promising but computationally expensive direction. We will mention this opportunity in the revision. We will also look for GPU resources that are able to enable training models larger than 1B parameters from scratch and validate this hypothesis.
Q3 — Importance-ordered uniform baseline
We trained an importance-ordered Uniform baseline, where channel importance is learned in the same way as Layer-adaptive Width Reduction under a per-layer width budget. The table below shows results on Qwen3-0.6B at 62.5% average width.
| Method | RULER 8K | RULER 16K | RULER 32K | LongBench | Rel.↑ |
|---|---|---|---|---|---|
| Vanilla | 68.75 | 64.06 | 58.03 | 32.09 | 100.0% |
| Uniform | 65.61 | 62.07 | 55.91 | 30.36 | 95.8% |
| Uniform, importance-ordered | 65.37 | 63.27 | 58.96 | 30.21 | 97.4% |
| Layer-adaptive | 70.02 | 66.82 | 61.84 | 31.40 | 102.6% |
Importance ordering improves the Uniform baseline from 95.8% to 97.4%, but it remains below Layer-adaptive Width Reduction. This gap indicates the effectiveness of the layer-wise width allocation.
We thank the reviewer for the thoughtful feedback and suggestions. We address each point below.
W1 — Relation to MatFormer and width pruning
While MatFormer focuses on training width-elastic models in general, our work investigates the computational asymmetry between prefill and decoding, particularly how FFN capacity should be allocated across layers during prefill while retaining full width during decoding. Our primary contribution lies in understanding and exploiting this asymmetry rather than in the Nested FFN architecture itself. We will more explicitly acknowledge MatFormer's contributions in the revised manuscript.
W2 — Results above Vanilla
How Vanilla is obtained. Vanilla is not the original checkpoint taken as-is. It is fine-tuned from the same official checkpoint with the same training recipe as Nested FFN. The only difference is that Vanilla uses the full FFN width during prefill. Vanilla is therefore a matched control for the effect of the width constraint. For reference, we also report the official Qwen3-0.6B checkpoint without any fine-tuning.
Why some settings exceed Vanilla. We do not claim that width reduction improves quality in general, and we read our results as preserving Vanilla-level performance. Similar observations have been reported in prior work. POP [1], which skips deep layers only during prefill, exceeds the full model in several settings (Tables 1 and 4 of their paper). Its authors hypothesize that mild pruning may filter out noise in the deep layers. Gains over the full model have also been reported for methods that remove input content or weight components, including GemFilter [2] (Table 1), LongLLMLingua [3] (Table 1), ChunkKV [4] (Tables 3 and 6), and LASER [5] (Table 1).
W3 — Why later layers are more reducible
One possible reason is that late layers are more responsible for decoding. This is also revealed by many prior studies: late layers mainly refine the next-token prediction [6–10], and deeper layers can be removed, or skipped during prefill, with limited quality loss [11, 12, 1].
References
[1] Junhui He, Zhihui Fu, Jun Wang, and Qingan Li. POP: Prefill-Only Pruning for Efficient Large Model Inference. Findings of ACL 2026.
[2] Zhenmei Shi et al. Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction. Findings of ACL 2026.
[3] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ACL 2024.
[4] Xiang Liu et al. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference. NeurIPS 2025.
[5] Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction. ICLR 2024.
[6] Xuan Luo, Weizhi Wang, and Xifeng Yan. Direct Multi-Token Decoding. COLM 2026.
[7] Vedang Lad, Wes Gurnee, and Max Tegmark. The Remarkable Robustness of LLMs: Stages of Inference? ICML 2024 Workshop on Mechanistic Interpretability.
[8] Nora Belrose et al. Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112, 2023.
[9] Róbert Csordás, Christopher D. Manning, and Christopher Potts. Do Language Models Use Their Depth Efficiently? NeurIPS 2025.
[10] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. EMNLP 2022.
[11] Andrey Gromov et al. The Unreasonable Ineffectiveness of the Deeper Layers. ICLR 2025.
[12] Xin Men et al. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL 2025.