acceptodds
Under review as a conference paper at ICLR 2027

ScopePEFT: Private Autoregressive Inference with Scoped Joint Computation

Abstract

Secure language-model inference protects users' prompts and proprietary model parameters, but secure multi-party computation (MPC) imposes substantial overhead. Parameter-efficient adaptation of a public backbone reduces secret weights but does not bound the computation that depends on them. In autoregressive inference, constructing persistent adapted key–value (KV) histories can require joint feed-forward network (FFN) work at every earlier position. ScopePEFT fixes joint-head sets and FFN sources before training. The client computes the full native trajectory, while designated attention heads retain complete joint KV histories. Each adapted tail block takes its FFN response from the native trajectory or MPC, determining the upstream work for later KV. Numerical adaptation and separate caches support this graph under semi-honest MPC with trusted preprocessing. We evaluate GPT-2 small and medium on WebNLG. At the single-seed medium Q16 endpoint with all tail heads and FFNs evaluated jointly, BLEU is 27.85 versus 26.19 for parameter-matched three-layer LoRA, while NLL worsens from 1.291 to 1.356. On three separate fixed replay prompts, the same weights reduce mean rank-0 framework bytes by 41.32%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
private inference, secure multi-party computation, autoregressive language models, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_knigOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In my reading, the paper addresses secure two-party inference for a GPT-style language model whose backbone θ\theta is public while the provider's parameter-efficient adapters ϕ\phi are private. The client (Alice) holds the prompt x1:Px_{1:P}; the provider (Bob) holds ϕ\phi; and the computation runs under CrypTen-based semi-honest MPC with a trusted preprocessing provider (TFP) supplying Beaver triples. The intended functionality releases all V=50,257V=50{,}257 logits zt=Fnum(θ,ϕ,x1:nt;C)z_t=F_{\mathrm{num}}(\theta,\phi,x_{1:n_t};\mathcal C) to Alice, who performs the argmax locally.

ScopePEFT splits inference into two trajectories over the same consumed prefix:

  • a native trajectory, which Alice computes in the clear with the public backbone on every token, giving native block outputs, head responses and FFN increments;
  • a joint trajectory, starting from three bias-free rank-rr bottleneck adapters Al(v)=Ul GELU(Dlv)A_l(v)=U_l\,\mathrm{GELU}(D_l v) placed after blocks s=L−3s=L-3, s+1s+1 and s+2s+2. Here, for each tail block, a public head set JlJ_l computes adapted QKV from joint states and keeps a persistent secret-shared KV cache, while the remaining heads RlR_l receive Alice's native attention responses as protected inputs. A public bit flf_l decides whether the FFN increment is computed on the joint residual (fl=1f_l=1) or copied from the native trajectory (fl=0f_l=0).

The analytical contribution is a dependency count. Switching block 11's FFN source from native to joint adds ΔNFFN=P+(G−1)\Delta N_{\mathrm{FFN}}=P+(G-1) joint FFN rows, because the next block's persistent joint KV needs earlier-block states at every retained position. A fixed-point Q16 graph (Q=216Q=2^{16}, truncation toward zero, ODE-style softmax, Newton-iteration LayerNorm, guarded GELU) and 256 straight-through "numerical adaptation" steps support execution.

Experiments use GPT-2 small and medium on WebNLG 2017. In the headline medium comparison (single seed, three fixed replay prompts with P∈{31,54,84}P\in\lbrace31,54,84\rbrace and G=16G=16, loopback TFP simulation on one host):

  • H5-F01 reduces rank-0 framework bytes by 66.97% and session time by 13.51% relative to parameter-matched Tail LoRA, with BLEU 23.810 vs. 23.537 but worse chrF++ and NLL;
  • H16-F11 reaches BLEU 27.848 against 26.188 for three-layer Deep LoRA at 41.32% fewer bytes, with NLL worsening from 1.291 to 1.356.

On GPT-2 small, F01 saves 76.86% of post-initialisation application bytes at P=169P=169, G=3G=3. The authors note that the timing advantage disappears as generation length grows.

Critical strengths

Strength 1. In my opinion, the paper raises a correct and under-appreciated point: reducing the number of private parameters does not by itself reduce the secure computation that depends on them in autoregressive inference. I find the dependency argument behind Eq. (8) clear and useful. Once a later block keeps a persistent, secret-shared KV cache for adapted heads, every earlier-block state that feeds those keys and values must be produced jointly at every retained position. Choosing a joint rather than native FFN source in block 11 therefore costs ΔNFFN=P+(G−1)\Delta N_{\mathrm{FFN}}=P+(G-1) extra joint FFN rows (171 rows at P=169P=169, G=3G=3), even though the private parameter count is unchanged. The conditional ledger in Appendix B.2 matches this exactly: the measured F11-minus-F01 difference is 9,097,955,136=53,204,416⋅1719{,}097{,}955{,}136=53{,}204{,}416\cdot171 bytes per party. Separating "complete access to the prefix" from "complete adaptivity to private corrections", by mixing native head responses (j∈Rlj\in R_l) with jointly adapted heads (j∈Jlj\in J_l) inside one attention block, is a sensible design idea, and I think the observation that KV caching avoids recomputation but not the initial construction of adapted history deserves to be stated explicitly in the private-inference literature.

Strength 2. I would like to acknowledge the authors' transparency about scope and provenance. From my perspective, the paper is unusually explicit about what it does not claim:

  • the byte counts are rank-0 framework counters rather than wire traffic;
  • the TFP simulation runs in one container with the provider inside a participating role;
  • the final medium Q16 results come from one training seed;
  • the development set was explored;
  • the timing advantage over Tail LoRA erodes and partly reverses at G=64G=64. The numerical graph is specified in detail (Eqs. 9–17), all final-checkpoint outputs are checked coordinatewise against saved integer references, and the supplementary CSVs (S1–S8) let a reader recompute every reported cost figure. I was able to confirm the medium-replay means of Table 10 from S7-medium-g16-replay.csv. This candour made a fair assessment possible.
Critical weaknesses

Weakness 1. In my view, the evidence does not show that ScopePEFT improves on the obvious baseline. Neither evaluated configuration dominates parameter-matched Tail LoRA on the quality–cost frontier, and the one apparent quality advantage lies well within single-seed noise.

The comparison that matters is quality and cost against Tail LoRA, the parameter-matched rank-24 Q/V LoRA in the final two blocks, at equal private parameters (196,608 on medium). The paper's own numbers give the following picture:

  • H5-F01 (the pre-specified primary configuration) is cheaper than Tail (3.193 vs. 9.669 GB; 83.87 vs. 96.97 s), but it is worse on chrF++ (42.439 vs. 43.273) and NLL (1.441 vs. 1.374). Its only advantage, BLEU 23.810 vs. 23.537, a gap of 0.273, comes from a single seed. The authors' own three-seed FP32 study of the same configurations (Table 6) shows H5-F01 behind Tail by 3.892 BLEU on the initial 128 groups and 2.771 on the additional 256, worse on every metric in every seed ("H5-F01 is worse than Tail on every metric in every seed"), with seed standard deviations of 0.4–2.8 BLEU. I therefore cannot accept the 0.273-BLEU Q16 difference as evidence of parity.
  • H16-F11 (the configuration with better quality) costs more than Tail: +3.77% bytes and +18.80% session time (115.21 vs. 96.97 s). It is cheaper only than Deep LoRA, and there it worsens NLL from 1.291 to 1.356.

So the cheap configuration loses quality, and the good configuration loses on cost. Is this not the trade-off one would expect from simply evaluating fewer heads jointly? Making 11 of 16 heads native removes their QKV, softmax and AV products from MPC, which reduces cost, and it equally removes their capacity to respond to the private correction, which reduces quality. I could not find evidence that ScopePEFT moves the Pareto frontier rather than sliding along it. A frontier plot sweeping the size of the joint-head set ∣Jl∣|J_l|, the LoRA rank rr and the number of adapted blocks for both ScopePEFT and Tail/Deep LoRA, at several seeds, is in my opinion the minimum needed to support the abstract's claim.

The small-model evidence points the same way. In FP32, F01 obtains BLEU/chrF++ of 22.534/43.492 against Tail's 29.946/48.251 on the 128 test groups (Table 5), and LoRA3 all12 reaches 53.630 BLEU on development data against F01's 40.125 (Table 3), with its MPC cost reported as "unavailable". The 76.86% byte saving on small is thus bought with a large quality deficit that the main text does not foreground.

The conceptual novelty is also thin relative to prior work. Letting the client run a public backbone in the clear and confining MPC to privately adapted layers is already the core idea of MARILL (Rathee et al., 2024: layer freezing, LoRA and head merging for secure LLM inference) and of CryptPEFT (Xia et al., 2026: one-way public-to-private data flow). Yet neither is compared experimentally; the related-work section states only that "we did not test whether prior frameworks can express the same graph". The counting result ΔNFFN=P+(G−1)\Delta N_{\mathrm{FFN}}=P+(G-1) is, as the authors themselves say, "neither a protocol lower bound" nor a cost model, just a direct consequence of causal attention over a persistent cache.

Weakness 2. I believe the experimental methodology is too narrow and too weakly controlled to support any general conclusion about secure autoregressive inference.

(a) Scale and task. The study uses GPT-2 small (124M) and GPT-2 medium (355M) on one data-to-text task, WebNLG 2017, with the final medium MPC comparison on three fixed prompts, two technical repeats each and one training seed. Contemporary secure-LLM-inference work evaluates LLaMA-class models at 7B scale (PUMA, BumbleBee, MPCFormer, BOLT, MARILL) across several tasks. I see no evidence that the head-set and FFN-source effects, which the authors themselves say "vary with the head set and evaluation sample", transfer to larger models, to grouped-query attention, or to other tasks.

(b) Cost measurement. All costs come from a same-host, same-container loopback TFP simulation. The byte figures are rank-0 framework counters ("not wire traffic"); the session times include startup and offline verification but exclude Alice's native preparation, i.e., the full plaintext forward pass of the public backbone over every token. That cost grows with model size and is borne by the client on every decision. No LAN/WAN latency, round counts, preprocessing (Beaver-triple) cost or end-to-end service latency is reported. Since MPC latency in realistic networks is often dominated by rounds rather than bytes, and H5-F01 actually issues 15.79% more backend communication calls than Tail (80,165 vs. 69,233, Table 10), I cannot tell whether the byte savings would turn into wall-clock savings in any deployment.

(c) Timing advantage does not persist. For small F01 versus Tail, the timing gain disappears with generation length: at G=64G=64, F01 is 3.32% and 1.22% slower on two of three prompts, and the third reverses direction between technical repeats (Table 8). Real generation lengths are typically longer than 64 tokens, so the measured benefit reduces to bytes counted in a simulation.

(d) Confounded quality pipeline. The Q16 numerical adaptation (256 straight-through Adam steps) changes quality in metric-specific and configuration-specific ways: F01's BLEU rises while chrF++ falls and NLL worsens (Figure 5). The authors acknowledge that missing unadapted-Q16 and matched FP32-continuation controls "prevent separating arithmetic from additional updates", and that the final Q16 comparison lacks H5-F11 and H16-F01, so head and FFN effects cannot be separated. The development set was also used "in broader exploration".

(e) Reproducibility. The supplementary README states that "the reorganized GPU and MPC entry points have not been rerun end to end", and that raw execution archives, logits and KV tensors are excluded. Combined with the AI-use statement (generative assistants supported "research framing, … implementation and test development, experiment orchestration and collection, result interpretation, and manuscript drafting"), I am not confident that the released code reproduces the reported measurements.

(f) Privacy framing. The guarantee is semi-honest MPC with a trusted dealer, with no new proof and no malicious security, and the simulation "does not verify physical isolation between Alice and Bob". The output functionality releases all logits to Alice, which, as the authors note, enables model extraction (Tramèr et al., 2016). Releasing the full 50,257-dimensional logit vector at every step makes this especially easy for low-rank adapters, whose parameter count is tiny. I think a discussion of how many queries suffice to recover ϕ\phi would strengthen the privacy claim considerably.

Questions for authors
  1. Pareto evidence. Can you show a quality–cost frontier, for example BLEU/chrF++/NLL against rank-0 bytes and session time, obtained by sweeping ∣Jl∣|J_l|, the adapter rank rr and the number of adapted tail blocks for ScopePEFT, and the LoRA rank and number of adapted blocks for Tail/Deep LoRA, over at least three seeds? Without this I cannot tell whether ScopePEFT improves the frontier or merely occupies a different point on it.

  2. Seed noise in the headline claim. The three-seed FP32 study shows H5-F01 behind Tail by 3.892 BLEU (initial groups) and worse on every metric in every seed, while the single-seed Q16 comparison shows H5-F01 ahead by 0.273 BLEU. Could you repeat the final Q16 comparison over seeds 43 and 47 and report means with confidence intervals? Which of the two results do you consider representative?

  3. Comparison with MARILL and CryptPEFT. MARILL (Rathee et al., 2024) also restricts secure computation to privately fine-tuned layers of a public backbone, using layer freezing, LoRA and head merging. Could you implement MARILL's configuration on the same GPT-2/WebNLG setup and compare quality and cost directly? In what precise sense does ScopePEFT's native/joint dependency graph go beyond MARILL's layer-freezing design?

  4. Client-side cost. Your session times exclude Alice's native preparation, i.e., the plaintext forward pass of the full public backbone for every consumed token. Could you report end-to-end per-token latency including this cost, and discuss how it scales from GPT-2 medium to a 7B model on a typical client device?

  5. Network-realistic latency. Could you report measured round counts and end-to-end latency under LAN and WAN settings (e.g., 1 Gbps / 0.5 ms and 100 Mbps / 40 ms RTT), including the offline Beaver-triple preprocessing cost, rather than loopback framework counters? Given that H5-F01 issues 15.79% more backend calls than Tail, does its byte saving survive in a high-latency network?

  6. Longer generation. The small-model timing advantage disappears at G=64G=64. What happens to bytes and time at G∈{128,256}G\in\lbrace128,256\rbrace and for prompts of several hundred tokens, which are typical in practice? Is there a crossover length beyond which ScopePEFT becomes strictly slower than Tail LoRA?

  7. Scaling to modern LLMs. Can you evaluate at least one LLaMA-class model (e.g., LLaMA-3-8B or Qwen2.5-7B) with grouped-query attention, RMSNorm and SwiGLU FFNs? How do the head-set choice JlJ_l and the FFN-source bit flf_l interact with GQA, where several query heads share one KV head, and does ΔNFFN=P+(G−1)\Delta N_{\mathrm{FFN}}=P+(G-1) still dominate the cost?

  8. Disentangling numerical adaptation. Could you add the missing controls, namely unadapted Q16 endpoints and FP32 checkpoints trained for the same additional 256 steps, together with the H5-F11 and H16-F01 Q16 endpoints, so that the effects of fixed-point arithmetic, extra updates, head sets and FFN sources can be separated?

  9. Model extraction. Since all 50,257 logits are released to Alice at every decision, how many queries would a semi-honest client need to recover the private low-rank adapters ϕ\phi (110,592 or 196,608 parameters) by regression on logit differences? Would releasing only the top-kk logits or the sampled token inside MPC change the cost profile?

  10. Reproducibility. The README states that the reorganised GPU and MPC entry points have not been rerun end to end. Could you confirm that the released code reproduces Table 10 and Figure 2 from the provided checkpoints, and clarify which parts of the experimental pipeline, given the AI-use statement, were implemented and verified directly by the authors?

Reviewer_VbQDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper focus on an MPC LLM inference setting where backbone is public, and only some last layers are secret. The two parties use secure computation only for pre-chosen attention heads, and for some feed-forward steps they paste in the user’s public result instead of recomputing it.

Critical strengths

Separating the public trajectory from a smaller joint trajectory increase the performance.

Critical weaknesses

The evaluated setting is narrow. The secret is confined to the last three blocks, and the joint heads are fixed before training. Ordinary LoRA is not limited to the last layers.

The quality tables do not include public GPT-2 with no adapter. It is therefore unclear how much the private piece improves on the public model.

The measured runtime saving is well under 2x, and it can disappear for longer outputs. That is too small to justify the quality loss in the cheaper configurations. Quality is reported only on WebNLG.

There is no evidence about scale. The study is one task and two GPT-2 sizes. Nothing shows whether the same split still saves time, or still holds quality, on a model large enough that anyone would pay for secure inference.

Questions for authors

On the same test set, how does generation quality compare with public GPT-2 and no adapter?

Reviewer_ugJxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

ScopePEFT combines a client-side public backbone with private adapters evaluated through MPC. Fixed attention-head sets and FFN-source choices determine which computations depend on adapted states, with separate native and joint KV caches. Experiments on GPT-2 small and medium using WebNLG report communication savings against parameter-matched LoRA configurations, alongside mixed quality results and diminishing latency benefits for longer generation.

Critical strengths

The paper identifies a relevant dependency issue: reducing private parameter counts does not necessarily reduce the computation needed to construct adapted KV histories. It also explicitly reports quality regressions and cases where communication savings fail to improve latency.

Critical weaknesses
  1. The contribution is insufficiently distinguished from existing architectural approaches. MARILL already restructures fine-tuning to reduce MPC computation, while CryptPEFT separates public-backbone computation from private adaptation. ScopePEFT introduces fixed head and FFN-source choices, but provides neither a principled selection method nor a controlled comparison establishing their advantage over these approaches. The (P+(G-1)) FFN-row count follows directly from constructing each cached position once; its broader methodological significance is limited. Parameter-matched Tail/Deep LoRA controls do not resolve this novelty and baseline gap.

  2. The experiments do not establish a convincing practical quality–efficiency advantage. The final medium Q16 results use one training seed, and MPC costs come from only three fixed-replay prompts with 16 output decisions in a same-host loopback simulation. Medium timings exclude native-input preparation while including startup and offline verification, preventing interpretation as complete inference latency. For small models, incremental decoding is slower, and the latency advantage disappears or reverses at 64 decisions. Quality evidence is similarly mixed: the primary H5-F01 configuration underperforms Tail LoRA on every reported metric across the FP32 seeds, while the stronger H16-F11 Q16 endpoint improves generation metrics over Deep LoRA but worsens NLL and costs more than Tail LoRA. These findings demonstrate configuration-specific trade-offs, but do not establish a robust advance in private autoregressive inference.

Questions for authors
  1. What substantive capability or quality–cost advantage does ScopePEFT provide beyond MARILL and an autoregressive adaptation of CryptPEFT, under matched workloads and privacy assumptions?

  2. Do the reported latency benefits survive when native computation, input preparation, output handling, and preprocessing are consistently included under realistic network conditions?

  3. Does the final Q16 quality–cost advantage hold across training seeds and the missing H5-F11/H16-F01 configurations, allowing the head-selection and FFN-source contributions to be separated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.