acceptodds
Under review as a conference paper at ICLR 2027

DepthRouter: Privacy-Preserving LLM Inference via Joint Depth-Axis Pruning

Abstract

Secure multi-party computation (MPC) protects both the user's prompt and the provider's weights during large language model (LLM) inference, but makes every executed layer expensive. Reducing model depth could lower the cost of private LLM inference, but removing blocks can degrade answer quality, while input-dependent selection introduces additional privacy and computation requirements. DepthRouter addresses this design problem by constructing pruned model variants with LoRA-based quality recovery and studying query-level selection among them. Variants skip FFN and attention sub-blocks under public, static schedules chosen with an MPC cost model, and adapters placed next to the skipped sub-blocks are distilled from the unpruned model. On Llama-3.1-8B with MPCache, a static variant that skips FFN sub-blocks retains of average F1 on LongBench QA at a % estimated reduction in MPC cost. Selection among variants shows clear headroom: an oracle gains F1 on QA at % lower estimated cost and accuracy points on TREC, while our learned quality estimator matches, but does not yet exceed, the best fixed variants.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Secure Multi-Party Computation, Private Inference, Large Language Models, Depth Pruning, Mixture-of-Depths, Dynamic Routing, Privacy-Preserving Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_6vW7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DepthRouter, which constructs depth-pruned variants of an LLM backbone by skipping FFN or attention sub-blocks and training LoRA adapters in neighboring retained layers through distillation. Some variants additionally incorporate fixed-capacity MoD, while a BERT-mini router selects one variant per query, with the selected index revealed to the servers. Experiments evaluate Llama-3.1-8B-Instruct with MPCache on four LongBench QA datasets and the TREC classification task.

Critical strengths
  1. The paper is well motivated by the high computational cost of MPC-based LLM inference.
Critical weaknesses
  1. Limited novelty. As acknowledged by the authors, DepthRouter combines established techniques. The technical contribution of their adaptation to MPC is limited. Furthermore, the learned quality estimator does not improve the quality–cost frontier of fixed configurations, limiting the benefit of query-level selection.

  2. Insufficient experimental validation. The main pruning baselines are evaluated without recovery, and comparisons against existing whole-layer pruning methods with LoRA healing are missing. Moreover, QA F1 is reported on the same 800 prompts used for recovery training and validation, leaving generalization to unseen queries unclear. MPC savings are extrapolated from per-operator profiles rather than validated through end-to-end secure execution, and the costs of the quality estimator and MoD selection operations are excluded.

  3. Limited validation across backbones and model scales. All model-quality experiments use Llama-3.1-8B-Instruct. The absence of experiments on other model families or larger models leaves the generality of the proposed design and its scalability to larger LLMs unverified.

  4. Imbalanced presentation and insufficient methodological detail. The Introduction and Background occupy substantial space, while important methodological details and controlled evaluations remain underdeveloped.

Questions for authors

See weakness

Reviewer_tX3YOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

MPC-based LLM inference is typically quite slow due to the high cost of MPC protocols on transformer layers. Thus, the authors develop DepthRouter, which prunes portions of FFN and attention blocks, to reduce the MPC cost. They utilize LoRA distillation for each pruning variant to recover model quality, and study query-based pruner selection.

Critical strengths
  1. The authors' combination of FFN pruning and LoRA distillation seems to measurably improve the quality-cost Pareto in MPC-based inference in Table 2. Also, the authors do a nice job of investigating different pruned variants and showing how they make complementary errors, which motivates the query-based router.
Critical weaknesses
  1. The LongBench QA prompts are used for distillation with a 90/10 train/val split, but all of these prompts are used for the reported eval F1 score in Table 2. This is a major form of train-test contamination, and undermines the most significant results.

  2. Perhaps I am misunderstanding, but there seems to be some massive inconsistencies in actual MPC costs and analytical estimates. Figure 1 shows under 2 minutes of MPC execution per layer at context length 10k, but Table 1 estimates over 1500 seconds per layer at context length 4k, for the same model.

Questions for authors
  1. Can the authors demonstrate the strong results in Table 2 on a true held-out set of LongBench QA?

  2. Can the authors explain the Figure 1/Table 1 discrepancy I stated above?

Reviewer_vCthOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DepthRouter, a framework that reduces the cost of privacy-preserving LLM inference under secure multi-party computation (MPC) by executing fewer model sub-blocks. The paper aims to speed up private LLM inference under secure multi-party computation (MPC), where each executed layer is very expensive, by skipping some FFN and attention sub-blocks. It combines existing techniques (static layer skipping, LoRA-based distillation to recover quality, and a small router that picks a pruned variant per query) into a framework called DepthRouter. On Llama-3.1-8B, the best variant keeps most of the full model’s quality (51.70 vs. 54.71 F1 on LongBench QA) at an estimated 22.8% lower MPC cost, but the learned router does not outperform simply using a single fixed variant.

Critical strengths

• The paper tackles the high cost of MPC-based private LLM inference and reduces it with limited quality loss. Skipping 12 FFN sub-blocks with LoRA repair retains 51.70 of 54.71 F1 at an estimated 22.8% reduction in MPC cost.

Critical weaknesses

• The adapters are distilled on 90% of the same 800 LongBench QA prompts used to report F1, so the main result is not a held-out evaluation. The learned router never beats the best fixed variant. On QA it matches always choosing M2, and on TREC its best point is dominated by M1. Savings are never measured in MPC, and the router's own cost is excluded. The results cover one model, about 1K short-answer examples, and no variance or confidence intervals. The oracle's +7 points on TREC is only 14 of 200 questions.

• Layer pruning, LoRA healing, distillation, MoD and quality-based routing are existing techniques. The MPC-specific contribution is mainly a cost model used to choose skip schedules, and quality is only evaluated in plain text.

• FFN-dominance analysis is not an MPC effect. Table 1 charges every matrix multiplication the same per-element cost, so its FFN vs. attention split is just plaintext FLOP counting. Per decode token, the FFN does about 176M multiply-adds, while attention does about 42M + 8192*n. So the FFN dominates whenever the context length is close to the hidden size, with or without MPC. Figure 1's MPC shares for the FFN closely match what FLOP counting alone predicts.

Questions for authors
  1. Could you number tables and figures int the order they are first cited? Currently Table 4 is cited in Section 1 and Table 2 in the contributions, both before Table 1. Figure 5 is placed in Section 4 but only cited in Appendix C.

  2. The variant labels M0–M4 mean different things in different places. M2 is "FFN skip + LoRA" in Figure 1, "M1 + MoD on 4 layers at 50%" in Table 3, and "attention + 5 FFN skipped" in the QA pool (Section 4.3). M4 is "FFN + attention skip" in Figure 1 but "M1 + MoD on 8 layers at 25%" in Figure 3 and Table 3. M1 is the 12-FFN skip in Table 3 but a 2-attention skip in Section 4.3. Can you clarify each one?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.