acceptodds
Under review as a conference paper at ICLR 2027

PREFILL DOES NOT NEED THE FULL FFN

Abstract

Recent studies have shown that computation during prefill can be reduced, with several approaches focusing on skipping entire Transformer layers. In this work, we investigate prefill computation at a finer granularity by reducing feed-forward network (FFN) capacity. We ask two questions: **(Q1)** To what extent can FFN capacity be reduced during prefill while preserving model quality? **(Q2)** How should the retained FFN capacity be distributed across layers? To answer these questions, we introduce a __Nested FFN__ architecture, in which prefill activates only a fixed subset of intermediate FFN channels while decode uses the full FFN, with both phases sharing the same parameters. Our experiments show that substantial FFN capacity (37.5% in multiple settings) can be removed during prefill, including in early Transformer layers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Transformer Architecture, Adaptive Computation, Feed-Forward Networks, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ffJKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Nested FFN, in which prefill uses only a prefix subset of FFN channels while decode uses the full FFN, with both phases sharing the same parameters. The FFN is fine-tuned with a simulated prompt boundary. Two variants are studied: a uniform width across layers, and per-layer widths learned under an average-width budget via a straight-through mask. Evaluation shows that layer-adaptive widths preserve quality at 50-62.5% average width, while uniform reduction degrade more.

Critical strengths
  1. Clearly written and easy to follow, with well-posed questions (Q1, Q2) that the experiments address to a reasonable extent.
  2. The learned profile (more width in early layers, almost none in late layers) is an interesting finding.
  3. The inference form needs only dense prefix slices, with no input-dependent routing.
Critical weaknesses
  1. The nested, shared-parameter FFN comes from MatFormer, which is credited only in Related Work, while the contributions say "we develop." Width pruning is also well studied (in LLMs and in CNNs). The contribution reduces to applying it to prefill and learning per-layer widths, and the results are not very surprising.
  2. Reduced models exceed Vanilla in several settings (e.g., 102.6% on Qwen3-0.6B at 62.5%, 103.0% on Llama-3B at 50%). The paper reports these numbers without explanation. Vanilla might be not trained with the same procedure, so it might be not a perfectly matched control.
  3. There is no analysis of why later layers are more redundant.
  4. Savings are argued from FFN FLOPs only. There is no latency or throughput measurement, and attention and the KV cache are unchanged. Slicing weights often does not translate into wall-clock gains, even with channels permuted into a contiguous prefix. The savings apply only to the first P-ω\omega prompt tokens, so they shrink for short prompts, and uneven per-layer shapes and batched serving with mixed prompt lengths are untested. Training cost relative to Vanilla is also not reported.
Questions for authors
  1. Why is width reduction applied only to prefill and not decode? Would reduced-width decode also work, and how would it compare?
  2. The method is applied to pretrained checkpoints. What happens when training from scratch, as MatFormer does?
  3. Unlike MatFormer, the prefill width is static and only the per-layer widths are optimized. Can channels be ordered by importance, as in prior CNN pruning work, to give a stronger Uniform baseline?
  4. What are the measured latency and throughput on a real batch serving scenario with mixed prompt lengths?
5 comments

We thank the reviewer for the thoughtful feedback and suggestions. We address each point below.

W1 — Relation to MatFormer and width pruning

While MatFormer focuses on training width-elastic models in general, our work investigates the computational asymmetry between prefill and decoding, particularly how FFN capacity should be allocated across layers during prefill while retaining full width during decoding. Our primary contribution lies in understanding and exploiting this asymmetry rather than in the Nested FFN architecture itself. We will more explicitly acknowledge MatFormer's contributions in the revised manuscript.

W2 — Results above Vanilla

How Vanilla is obtained. Vanilla is not the original checkpoint taken as-is. It is fine-tuned from the same official checkpoint with the same training recipe as Nested FFN. The only difference is that Vanilla uses the full FFN width during prefill. Vanilla is therefore a matched control for the effect of the width constraint. For reference, we also report the official Qwen3-0.6B checkpoint without any fine-tuning.

SettingAvg. prefill FFN widthRULER 8KRULER 16KRULER 32KLongBenchRel. vs. Vanilla ↑
Official checkpoint100%71.0165.6556.0030.9099.6%
Vanilla100%68.7564.0658.0332.09100.0%
Layer-adaptive~62.5%70.0266.8261.8431.40102.6%
Layer-adaptive~50%68.2163.6157.7930.5598.3%

Why some settings exceed Vanilla. We do not claim that width reduction improves quality in general, and we read our results as preserving Vanilla-level performance. Similar observations have been reported in prior work. POP [1], which skips deep layers only during prefill, exceeds the full model in several settings (Tables 1 and 4 of their paper). Its authors hypothesize that mild pruning may filter out noise in the deep layers. Gains over the full model have also been reported for methods that remove input content or weight components, including GemFilter [2] (Table 1), LongLLMLingua [3] (Table 1), ChunkKV [4] (Tables 3 and 6), and LASER [5] (Table 1).

W3 — Why later layers are more reducible

One possible reason is that late layers are more responsible for decoding. This is also revealed by many prior studies: late layers mainly refine the next-token prediction [6–10], and deeper layers can be removed, or skipped during prefill, with limited quality loss [11, 12, 1].

References

[1] Junhui He, Zhihui Fu, Jun Wang, and Qingan Li. POP: Prefill-Only Pruning for Efficient Large Model Inference. Findings of ACL 2026.

[2] Zhenmei Shi et al. Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction. Findings of ACL 2026.

[3] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ACL 2024.

[4] Xiang Liu et al. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference. NeurIPS 2025.

[5] Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction. ICLR 2024.

[6] Xuan Luo, Weizhi Wang, and Xifeng Yan. Direct Multi-Token Decoding. COLM 2026.

[7] Vedang Lad, Wes Gurnee, and Max Tegmark. The Remarkable Robustness of LLMs: Stages of Inference? ICML 2024 Workshop on Mechanistic Interpretability.

[8] Nora Belrose et al. Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112, 2023.

[9] Róbert Csordás, Christopher D. Manning, and Christopher Potts. Do Language Models Use Their Depth Efficiently? NeurIPS 2025.

[10] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. EMNLP 2022.

[11] Andrey Gromov et al. The Unreasonable Ineffectiveness of the Deeper Layers. ICLR 2025.

[12] Xin Men et al. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL 2025.

W4 + Q4 — Latency, throughput, and training cost

Efficiency measurement. We measured prefill latency and throughput of Layer-adaptive Width Reduction on one NVIDIA GH200 with batches of mixed prompt lengths. We left-pad the prompts so that their final ω positions are aligned (ω = 512). Throughput counts only non-padding tokens. Vanilla and Nested FFN both use FlashAttention-3, which skips padding in attention, and are timed on the same batch.

ModelWidthBatchLatency (Vanilla / Nested FFN)Throughput (Vanilla / Nested FFN)Speedup
Llama-3.2-3B62.5%8 prompts, 4K to 32K tokens3.82 s / 3.12 s (−18.4%)38.6K / 47.2K tok/s1.23×
Qwen3-4B62.5%4 prompts, 8K to 32K tokens2.92 s / 2.47 s (−15.5%)28.0K / 33.2K tok/s1.18×

Regarding uneven per-layer shapes, we agree with the reviewer that they may limit achievable speedups, as computation is unevenly distributed across layers and may require further system-level optimization. Our primary focus is to establish that FFN width can be substantially reduced during prefill without degrading model performance. Several prior studies, such as YOCO, have demonstrated this possibility, albeit with some performance degradation. Translating this finding into practical system-level speedups warrants a comprehensive investigation in future work. For example, in MoE settings, this could reduce the number of active experts, thereby lowering the memory footprint and improving system utilization.

Training cost. On Qwen3-0.6B, fine-tuning for 2000 steps at 32K context on 8 A100 GPUs takes about 5.5 hours for Vanilla, and about 7.0 hours for Layer-adaptive Width Reduction. We will add a discussion of training cost in the revision.

Q1 — Why not reduce decode width

Compared with prefill, decoding generally requires greater per-token computation to generate new tokens. Reducing FFN computation during decoding therefore involves tuning hyperparameters in Transformers, such as FFN width and hidden-state dimensionality, which is a more general topic on model optimization.

Q2 — Training from scratch

Training width-elastic models from scratch, as in MatFormer, might yield even better results by allowing the selected channels to specialize in state computation rather than decoding. This is a promising but computationally expensive direction. We will mention this opportunity in the revision. We will also look for GPU resources that are able to enable training models larger than 1B parameters from scratch and validate this hypothesis.

Q3 — Importance-ordered uniform baseline

We trained an importance-ordered Uniform baseline, where channel importance is learned in the same way as Layer-adaptive Width Reduction under a per-layer width budget. The table below shows results on Qwen3-0.6B at 62.5% average width.

MethodRULER 8KRULER 16KRULER 32KLongBenchRel.↑
Vanilla68.7564.0658.0332.09100.0%
Uniform65.6162.0755.9130.3695.8%
Uniform, importance-ordered65.3763.2758.9630.2197.4%
Layer-adaptive70.0266.8261.8431.40102.6%

Importance ordering improves the Uniform baseline from 95.8% to 97.4%, but it remains below Layer-adaptive Width Reduction. This gap indicates the effectiveness of the layer-wise width allocation.

Reviewer_DDo2OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper has two key questions. How much feed-forward network (FFN) capacity can be reduced during prefill while preserving model quality, and how the retained capacity should be distributed across Transformer layers. The authors introduce Nested FFN, a channel-pruning architecture that activates a fixed subset of intermediate FFN channels for most prompt tokens while retaining the full FFN for the final ω = 512 prompt tokens during prefill phase. The Nested FFN is part of full-width FFN, and the method requires additional parameter training. Experiments include evaluations on RULER at context lengths of 8K, 16K, and 32K tokens. The results show that substantial FFN capacity—37.5% in multiple settings—can be removed during prefill while maintaining model quality. The study also demonstrates the feasibility of layer-wise capacity allocation, allowing different pruning ratios across layers.

Critical strengths
  1. The proposed method achieves higher accuracy on RULER 8K, 16K, and 32K, as well as LongBench, even with a smaller prefill width compared to uniform reduction.
  2. Furthermore, the paper demonstrates that, at the same prefill width, the layer-adaptive approach consistently achieves higher scores than existing methods, such as POP and uniform reduction, on both long-sequence benchmarks, such as RULER, and general-purpose benchmarks.
Critical weaknesses
  1. Lack of evaluation on longer contexts and actual inference efficiency

    One of the main motivations for reducing the prefill computation of LLMs is that, as the input sequence length increases, the prefill stage accounts for a substantial portion of the total computation and significantly affects inference latency.

    To better demonstrate the effectiveness of the proposed method in terms of accuracy, the authors should extend their evaluation to longer context lengths, such as 64K and 128K, on RULER. Furthermore, it is important to quantitatively report the reduction in FLOPs and the actual inference latency on GPUs for these longer contexts.

    Similarly, for the general capability evaluations presented in Appendix B, an analysis of FLOPs reduction and actual GPU latency improvement is necessary, especially considering that full FFN computation is still performed on a subset of input tokens (ω = 512).

  2. Lack of justification for the choice of ω = 512

    The paper does not provide sufficient analysis or justification for selecting ω = 512. An ablation study investigating how different values of w affect accuracy and computational efficiency would help clarify the trade-offs and support this design choice.

  3. Additional concerns with baselines

    • To further strengthen the effectiveness of the proposed layer-adaptive approach, I would like to see a comparison against a random channel selection baseline. Specifically, this baseline should randomly select weight channels while maintaining the same per-layer prefill FFN width and overall computational budget determined by the proposed method. Such a comparison would help clarify whether the performance improvements come from the proposed strategy rather than simply from the allocated per-layer widths.
    • Since the proposed method requires additional training, a direct comparison with methods such as POP, without pruning-specific training, may need further justification in terms of training cost.
  4. Lack of Model Size Reduction.

    Although the proposed Nested FFN approach reduces FFN computation for a subset of input tokens, it still requires full FFN computation for the remaining ω = 512 tokens. Consequently, the entire set of model weight parameters must remain available during inference, meaning that the proposed method does not reduce the model's parameter footprint or the memory required to store its weights.

Questions for authors
  1. Accuracy and practical prefill efficiency (TTFT) across context lengths

    Could you report prefill FLOPs, measured GPU prefill latency accounting ω = 512 prompt tokens, and the corresponding RULER scores at input lengths of 8K, 16K, 32K, 64K, and 128K, where supported by the base models? Specifying the GPU hardware would help assess whether the reduction in FFN capacity translates into practical inference speedups.

  2. Ablation for the full-width suffix length

    Could you provide a quantitative ablation of ω, including values smaller and larger than 512, and report the resulting RULER scores, prefill FLOPs, and GPU latency? What evidence supports choosing ω = 512, and how does this trade-off vary with prompt length?

  3. Random channel selection under matched layer-wise widths

    Could you compare Nested FFN with a random channel-selection baseline that preserves the same layer-specific FFN retention ratios and overall prefill computation budget as the proposed method? For a controlled comparison, the random channel subsets should be selected before fine-tuning and trained under the same conditions, with results reported across multiple random seeds. This experiment would help determine the contribution of proposed method while holding the layer-wise capacity allocation fixed.

5 comments

We thank the reviewer for the constructive feedback and detailed suggestions. We address each point below.

Q1 + W1 — Longer contexts, FLOPs, and latency

Prefill FLOPs. For a prompt of length PP, Nested FFN uses the reduced FFN for the first P−ωP - \omega prompt positions and the full FFN for the final ω\omega positions. The prefill FFN cost of layer ll is

CVanilla(l)∝PddffC_{\mathrm{Vanilla}}^{(l)} \propto P d d_{\mathrm{ff}}

CNested(l)∝[ω+(P−ω)ρ(l)]ddffC_{\mathrm{Nested}}^{(l)} \propto [\omega + (P-\omega)\rho^{(l)}] d d_{\mathrm{ff}}

where dd is the hidden dimension, dffd_{\mathrm{ff}} is the FFN intermediate width, and ρ(l)\rho^{(l)} is the retained width ratio of layer ll. Summing over layers, the fraction of prefill FFN FLOPs saved is

(1−ωP)(1−ρˉ)(1 - \frac{\omega}{P})(1 - \bar{\rho})

where

ρˉ=1L∑l=1Lρ(l)\bar{\rho} = \frac{1}{L}\sum_{l=1}^{L}\rho^{(l)}

The total prefill cost also depends on how attention is implemented. The table below shows the FFN FLOPs saving and the resulting total prefill FLOPs saving under standard, top-k sparse, and linear attention, for Llama-3.2-3B at 62.5% average width and ω=512\omega = 512.

Prompt lengthFFN FLOPs savedTotal saved, standard attentionTotal saved, top-k sparse attention (k = 2048)Total saved, linear attention
8K35.2%21.1%23.4%26.2%
16K36.3%18.2%24.2%27.0%
32K36.9%13.8%24.6%27.5%
64K37.2%9.3%24.8%27.7%
128K37.4%5.6%24.9%27.8%

Under standard attention, the total saving decreases at long context because attention cost grows quadratically. With top-k sparse or linear attention, it instead increases with context length.

Prefill latency. We measured time to first token (TTFT) for Llama-3.2-3B at 62.5% average width and ω=512\omega = 512 on one NVIDIA GH200 under standard attention. The measured TTFT reduction closely follows the total FLOPs saving.

Prompt lengthTotal FLOPs saved, standard attentionMeasured TTFT reduction
8K21.1%18.1%
16K18.2%15.2%
32K13.8%12.1%
64K9.3%8.7%
128K5.6%5.0%

Due to the limited time of the rebuttal period, we have not implemented top-k sparse or linear attention. Under these attention mechanisms, the FFN accounts for a much larger share of prefill computation. The total FLOPs saving will increase. We therefore expect end-to-end TTFT reductions close to these theoretical savings.

Accuracy beyond 32K. Our models are fine-tuned on 32K-token sequences, so we report accuracy up to 32K. Evaluating accuracy at 64K and 128K would require further long-context fine-tuning, which is not feasible within the rebuttal period given our compute resources. We will cover longer context accuracy in the revision.

Speedup on general-capability benchmarks. We measured TTFT with Qwen3-4B and Llama-3.2-3B on MMLU, GSM8K, ARC-Challenge, and HellaSwag. The prompts in these benchmarks are short, typically 700 to 1,200 tokens. Since the final ω = 512 positions always use the full FFN, only about 27% to 57% of prompt tokens go through the reduced FFN, so the gain is limited. The measured speedup ranges from 1.01× to 1.09×, for short prompt length. We will clarify the efficiency of our method with respect to different prompt lengths and attention mechanisms in the revision.

Q2 + W2 — Choice of ω

Accuracy across ω. We retrained Layer-adaptive Width Reduction on Qwen3-0.6B with ω from 4 to 512 at the same 62.5% budget. Rel. is the average retention relative to Vanilla over RULER 8K, 16K, 32K, and LongBench.

ωRULER 8KRULER 16KRULER 32KLongBenchRel. ↑
Vanilla68.7564.0658.0332.09100.0%
463.5760.2855.8628.5292.9%
1668.7763.7054.2030.1496.7%
6467.8864.4558.4932.00100.0%
25666.4564.1759.7532.36100.2%
51270.0266.8261.8431.40102.6%

As ω ∈ {64, 256, 512} already effectively preserve Vanilla-level performance, we did not additionally train larger windows such as ω = 1024.

Computational efficiency across ω. The table below shows the FFN FLOPs saving, the total prefill FLOPs saving under standard attention, and the measured TTFT reduction at 32K on one NVIDIA GH200, at 62.5% average width.

ModelωFFN FLOPs savedTotal FLOPs saved, standard attentionMeasured TTFT reduction
Llama-3.2-3B437.5%14.1%12.6%
Llama-3.2-3B6437.4%14.0%12.4%
Llama-3.2-3B51236.9%13.8%12.1%
Qwen3-4B437.5%11.9%10.6%
Qwen3-4B6437.4%11.9%10.4%
Qwen3-4B51236.9%11.7%10.3%

Trade-off and choice of ω = 512. A larger ω improves performance while slightly reducing the speedup, and this effect diminishes as the prompt grows. In the paper, we use ω = 512 as a conservative choice to preserve model quality, and a smaller ω can be chosen in practice for specific models and settings.

Q3 + W3 — Random channel baseline and training cost

Random channel baseline. On Qwen3-0.6B at 62.5% average width, we trained a random channel-selection baseline with the same per-layer widths as Layer-adaptive Width Reduction. The channel subsets are fixed before fine-tuning and trained under identical conditions. Both methods are reported over 2 seeds. Rel. is the average retention relative to Vanilla over RULER 8K, 16K, 32K, and LongBench.

MethodRULER 8KRULER 16KRULER 32KLongBenchRel. ↑
Vanilla68.7564.0658.0332.09100.0%
Layer-adaptive widths, random channels (mean ± std)65.52 ± 2.1860.95 ± 1.0754.56 ± 2.7629.77 ± 0.3494.3% ± 2.7%
Layer-adaptive widths, learned channels (ours, mean ± std)68.57 ± 2.0665.65 ± 1.6560.47 ± 1.9431.87 ± 0.66101.4% ± 1.7%

With the same per-layer widths but random channels, performance drops to 94.3%, well below Layer-adaptive Width Reduction at 101.4%. This indicates that the per-layer widths alone do not account for the gain, and the learned channel selection is the key.

Training cost. We agree with the reviewer that POP is a training-free method, while Nested FFN requires fine-tuning. We will add a discussion of the training cost in the revision.

W4 — Model size

Nested FFN targets prefill computation rather than model size. The full weights are kept for the last ω tokens in prefill and for decoding, which allows the same model to serve both prefill and decode without any additional parameters. We will clarify this in the paper.

Thank you for the response and clarifications. However, I find that my main concerns have not been sufficiently addressed. The reported improvements appear limited, and I remain unconvinced that the proposed approach provides a sufficiently substantial benefit.

In particular, the efficiency claims rely primarily on theoretical FLOP counts. It remains unclear whether these reductions translate into practical gains on GPU hardware. Measurements of actual runtime or throughput, with the relevant hardware and implementation details, would help establish the practical value of the approach.

The fine-tuning procedure also needs to be explained more clearly, including which parameters are updated, the training setup, and the associated computational cost. These details are important for assessing both reproducibility and the overall benefit of the method.

For these reasons, I will maintain my current score. I appreciate the authors’ efforts on this work and the time they have taken to address the reviewers’ comments.

Reviewer_zLN4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes nested FFN, which skip a subset of channels in FFN layer to reduce the width of FFN layer. The experiments on Qwen and Llama demonstrate the effectiveness of nested FFN.

Critical strengths
  1. The performance is very good compared to existing methods on various LLM backbones.
Critical weaknesses
  1. The writing of this paper needs to be improved in general. The Introduction section is hard to follow and interpret, without a concrete writing structure, focused research question(s), and strong motivation for the methods.

    In particular, there is a research question in paragraph 2, lines 033-034, that "How much computation is needed within each Transformer layer during prefill to preserve model quality?". This research question is immediately followed by two research questions in paragraph 3. The follow-up research questions are not fully discussed and explained. The motivation is weak. It reads like developing the methods first and then asking research questions w.r.t. the methods.

  2. The core design in the proposed method is to simply skip a subset of channels in FFN, with no further information recovery or processings. Similar idea has been proposed in MoC [1], FlashMGLU/MGLU, and ReLUfication.

  3. As nested FFN is designed to reduce the computational complexity, the experiments on the real-world inference speed should be provided, and the theoretical analysis of the inference complexity should also be provided.

[1] Mixture-of-Channels: Exploiting Sparse FFNs for Efficient LLMs Pre-Training and Inference

Questions for authors
  1. Please justify the architectural difference from MoC, and explain why nested FFN can even improve performance with a more significant reduction in the activated parameters and computational complexity.

  2. Please provide results on the inference efficiency.

2 comments

We thank the reviewer for the careful reading and constructive feedback. We address each point below.

W1 — Introduction and motivation

Q1 and Q2 address two complementary aspects of the same overarching question: Q1 examines how much FFN capacity can be removed during prefill, while Q2 investigates how the retained (or removed) capacity should be distributed across layers. In the revision, we will organize the research questions around Q1 and Q2 to streamline the presentation.

Q1 + W2 — Relation to MoC, MGLU, ReLUfication, and results above Vanilla

MoC, MGLU, and ReLUfication all operate on the FFN, but they exploit different sources of redundancy and differ in structure.

MoC and ReLUfication. MoC and ReLUfication dynamically select channels for each token, but the union of selected channels across prefill tokens may still cover nearly all channels. In contrast, our approach permanently disables a large subset of channels during prefill for all prefill tokens. The approaches are therefore complementary: MoC could dynamically select channels from the subset retained by our approach.

MGLU. MGLU shares one weight matrix between the gate and value projections. It keeps the FFN width unchanged and mainly reduces weight memory traffic during decoding, so it is complementary to our method.

Results above Vanilla. We do not claim that reducing prefill FFN width improves quality in general, and we read our results as preserving Vanilla-level performance. We explain the results above Vanilla in detail in our response to the Ethics concern.

In the revision we will add this discussion.

Q2 + W3 — Inference efficiency

Theoretical analysis. For a prompt of length PP, Nested FFN uses the reduced FFN for the first P−ωP - \omega prompt positions and the full FFN for the final ω\omega positions. The prefill FFN cost of layer ll is

CVanilla(l)∝PddffC_{\mathrm{Vanilla}}^{(l)} \propto P d d_{\mathrm{ff}}

[\omega + (P-\omega)\rho^{(l)}] d d_{\mathrm{ff}}$$ where $d$ is the hidden dimension, $d_{\text{ff}}$ is the FFN intermediate width, and $\rho^{(l)}$ is the retained width ratio of layer $l$. Summing over layers, the fraction of prefill FFN FLOPs saved is $$\Big(1 - \frac{\omega}{P}\Big)\big(1 - \bar{\rho}\big), \qquad \bar{\rho} = \frac{1}{L}\sum_{l=1}^{L} \rho^{(l)}.$$ The total prefill cost also depends on how attention is implemented. The table below shows the FFN FLOPs saving and the resulting total prefill FLOPs saving under standard, top-k sparse, and linear attention, for $\omega = 512$. | Model | Avg. Prefill Width | FFN FLOPs saved (8K / 16K / 32K) | Total saved, standard attention | Total saved, top-k sparse attention (k = 2048) | Total saved, linear attention | |---|---|---|---|---|---| | Llama-3.2-3B | 62.5% | 35.2% / 36.3% / 36.9% | 21.1% / 18.2% / 13.8% | 23.4% / 24.2% / 24.6% | 26.2% / 27.0% / 27.5% | | Llama-3.2-3B | 50% | 46.9% / 48.4% / 49.2% | 28.1% / 24.2% / 18.5% | 31.2% / 32.3% / 32.8% | 34.9% / 36.0% / 36.6% | | Qwen3-4B | 62.5% | 35.2% / 36.3% / 36.9% | 19.5% / 16.2% / 11.7% | 22.3% / 23.1% / 23.4% | 25.8% / 26.6% / 27.0% | | Qwen3-4B | 50% | 46.9% / 48.4% / 49.2% | 26.0% / 21.5% / 15.6% | 29.8% / 30.7% / 31.2% | 34.3% / 35.5% / 36.1% | **Inference speedup.** We measured time to first token (TTFT) for Layer-adaptive Width Reduction on one NVIDIA GH200. The table below reports the speedup and TTFT reduction relative to Vanilla. Standard attention is used here. | Model | Avg. Prefill Width | 8K | 16K | 32K | |---|---|---|---|---| | Llama-3.2-3B | 62.5% |   1.22× (−18.1%)   |   1.18× (−15.2%)   |   1.14× (−12.1%)   | | Llama-3.2-3B | 50% |   1.32× (−24.1%)   |   1.26× (−20.7%)   |   1.19× (−16.1%)   | | Qwen3-4B | 62.5% |   1.18× (−15.5%)   |   1.16× (−13.7%)   |   1.11× (−10.3%)   | | Qwen3-4B | 50% |   1.27× (−21.3%)   |   1.23× (−18.7%)   |   1.16× (−14.0%)   | The measured TTFT reduction closely follows the total FLOPs saving under standard attention, whose quadratic cost increasingly dominates prefill as the prompt grows. Due to the limited time of the rebuttal period, we have not implemented Nested FFN with top-k sparse or linear attention. Under these attention mechanisms, the FFN accounts for a larger share of prefill computation, and the total FLOPs saving will increase. We therefore expect end-to-end TTFT reductions of 20% or more with such attention mechanisms. We will add the complexity analysis and these measurements to the paper.
AuthorsOpenReview ↗

Ethics concerns response

Ethics concern

We thank the reviewer for raising this concern. We would like to address it in three parts.

  1. The reduction targets prefill FFN computation only. Nested FFN reduces FFN computation only for prompt tokens preceding the final ω positions, while the final ω positions and all decoding tokens use the full FFN. This design preserves model performance.

  2. Similar observations have been reported in prior work. Regarding the 102.6% result noted by the reviewer, similar improvements over the full model have been reported in prior work. POP [1], which skips deep layers only during prefill, exceeds the full model in several settings (Tables 1 and 4 of their paper). Its authors hypothesize that mild pruning may filter out noise in the deep layers. Gains over the full model have also been reported for methods that remove input content or weight components, including GemFilter [2] (Table 1), LongLLMLingua [3] (Table 1), ChunkKV [4] (Tables 3 and 6), and LASER [5] (Table 1).

  3. Code and checkpoints. To ensure reproducibility, we provide anonymized training and evaluation code, the exact checkpoints for all reported numbers, configurations, and evaluation scripts, so that the results can be verified directly. Training Qwen3-0.6B with the provided code takes about 7 hours on 8 A100 GPUs, and evaluation takes about 2 hours.

Code: [anonymous link]

Checkpoints and data: [anonymous link]

References

[1] Junhui He, Zhihui Fu, Jun Wang, and Qingan Li. POP: Prefill-Only Pruning for Efficient Large Model Inference. Findings of ACL 2026.

[2] Zhenmei Shi et al. Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction. Findings of ACL 2026.

[3] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ACL 2024.

[4] Xiang Liu et al. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference. NeurIPS 2025.

[5] Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction. ICLR 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.