- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In my reading, the paper addresses secure two-party inference for a GPT-style language model whose backbone is public while the provider's parameter-efficient adapters are private. The client (Alice) holds the prompt ; the provider (Bob) holds ; and the computation runs under CrypTen-based semi-honest MPC with a trusted preprocessing provider (TFP) supplying Beaver triples. The intended functionality releases all logits to Alice, who performs the argmax locally.
ScopePEFT splits inference into two trajectories over the same consumed prefix:
- a native trajectory, which Alice computes in the clear with the public backbone on every token, giving native block outputs, head responses and FFN increments;
- a joint trajectory, starting from three bias-free rank- bottleneck adapters placed after blocks , and . Here, for each tail block, a public head set computes adapted QKV from joint states and keeps a persistent secret-shared KV cache, while the remaining heads receive Alice's native attention responses as protected inputs. A public bit decides whether the FFN increment is computed on the joint residual () or copied from the native trajectory ().
The analytical contribution is a dependency count. Switching block 11's FFN source from native to joint adds joint FFN rows, because the next block's persistent joint KV needs earlier-block states at every retained position. A fixed-point Q16 graph (, truncation toward zero, ODE-style softmax, Newton-iteration LayerNorm, guarded GELU) and 256 straight-through "numerical adaptation" steps support execution.
Experiments use GPT-2 small and medium on WebNLG 2017. In the headline medium comparison (single seed, three fixed replay prompts with and , loopback TFP simulation on one host):
- H5-F01 reduces rank-0 framework bytes by 66.97% and session time by 13.51% relative to parameter-matched Tail LoRA, with BLEU 23.810 vs. 23.537 but worse chrF++ and NLL;
- H16-F11 reaches BLEU 27.848 against 26.188 for three-layer Deep LoRA at 41.32% fewer bytes, with NLL worsening from 1.291 to 1.356.
On GPT-2 small, F01 saves 76.86% of post-initialisation application bytes at , . The authors note that the timing advantage disappears as generation length grows.
Critical strengths
Strength 1. In my opinion, the paper raises a correct and under-appreciated point: reducing the number of private parameters does not by itself reduce the secure computation that depends on them in autoregressive inference. I find the dependency argument behind Eq. (8) clear and useful. Once a later block keeps a persistent, secret-shared KV cache for adapted heads, every earlier-block state that feeds those keys and values must be produced jointly at every retained position. Choosing a joint rather than native FFN source in block 11 therefore costs extra joint FFN rows (171 rows at , ), even though the private parameter count is unchanged. The conditional ledger in Appendix B.2 matches this exactly: the measured F11-minus-F01 difference is bytes per party. Separating "complete access to the prefix" from "complete adaptivity to private corrections", by mixing native head responses () with jointly adapted heads () inside one attention block, is a sensible design idea, and I think the observation that KV caching avoids recomputation but not the initial construction of adapted history deserves to be stated explicitly in the private-inference literature.
Strength 2. I would like to acknowledge the authors' transparency about scope and provenance. From my perspective, the paper is unusually explicit about what it does not claim:
- the byte counts are rank-0 framework counters rather than wire traffic;
- the TFP simulation runs in one container with the provider inside a participating role;
- the final medium Q16 results come from one training seed;
- the development set was explored;
- the timing advantage over Tail LoRA erodes and partly reverses at .
The numerical graph is specified in detail (Eqs. 9–17), all final-checkpoint outputs are checked coordinatewise against saved integer references, and the supplementary CSVs (S1–S8) let a reader recompute every reported cost figure. I was able to confirm the medium-replay means of Table 10 from
S7-medium-g16-replay.csv. This candour made a fair assessment possible.
Critical weaknesses
Weakness 1. In my view, the evidence does not show that ScopePEFT improves on the obvious baseline. Neither evaluated configuration dominates parameter-matched Tail LoRA on the quality–cost frontier, and the one apparent quality advantage lies well within single-seed noise.
The comparison that matters is quality and cost against Tail LoRA, the parameter-matched rank-24 Q/V LoRA in the final two blocks, at equal private parameters (196,608 on medium). The paper's own numbers give the following picture:
- H5-F01 (the pre-specified primary configuration) is cheaper than Tail (3.193 vs. 9.669 GB; 83.87 vs. 96.97 s), but it is worse on chrF++ (42.439 vs. 43.273) and NLL (1.441 vs. 1.374). Its only advantage, BLEU 23.810 vs. 23.537, a gap of 0.273, comes from a single seed. The authors' own three-seed FP32 study of the same configurations (Table 6) shows H5-F01 behind Tail by 3.892 BLEU on the initial 128 groups and 2.771 on the additional 256, worse on every metric in every seed ("H5-F01 is worse than Tail on every metric in every seed"), with seed standard deviations of 0.4–2.8 BLEU. I therefore cannot accept the 0.273-BLEU Q16 difference as evidence of parity.
- H16-F11 (the configuration with better quality) costs more than Tail: +3.77% bytes and +18.80% session time (115.21 vs. 96.97 s). It is cheaper only than Deep LoRA, and there it worsens NLL from 1.291 to 1.356.
So the cheap configuration loses quality, and the good configuration loses on cost. Is this not the trade-off one would expect from simply evaluating fewer heads jointly? Making 11 of 16 heads native removes their QKV, softmax and AV products from MPC, which reduces cost, and it equally removes their capacity to respond to the private correction, which reduces quality. I could not find evidence that ScopePEFT moves the Pareto frontier rather than sliding along it. A frontier plot sweeping the size of the joint-head set , the LoRA rank and the number of adapted blocks for both ScopePEFT and Tail/Deep LoRA, at several seeds, is in my opinion the minimum needed to support the abstract's claim.
The small-model evidence points the same way. In FP32, F01 obtains BLEU/chrF++ of 22.534/43.492 against Tail's 29.946/48.251 on the 128 test groups (Table 5), and LoRA3 all12 reaches 53.630 BLEU on development data against F01's 40.125 (Table 3), with its MPC cost reported as "unavailable". The 76.86% byte saving on small is thus bought with a large quality deficit that the main text does not foreground.
The conceptual novelty is also thin relative to prior work. Letting the client run a public backbone in the clear and confining MPC to privately adapted layers is already the core idea of MARILL (Rathee et al., 2024: layer freezing, LoRA and head merging for secure LLM inference) and of CryptPEFT (Xia et al., 2026: one-way public-to-private data flow). Yet neither is compared experimentally; the related-work section states only that "we did not test whether prior frameworks can express the same graph". The counting result is, as the authors themselves say, "neither a protocol lower bound" nor a cost model, just a direct consequence of causal attention over a persistent cache.
Weakness 2. I believe the experimental methodology is too narrow and too weakly controlled to support any general conclusion about secure autoregressive inference.
(a) Scale and task. The study uses GPT-2 small (124M) and GPT-2 medium (355M) on one data-to-text task, WebNLG 2017, with the final medium MPC comparison on three fixed prompts, two technical repeats each and one training seed. Contemporary secure-LLM-inference work evaluates LLaMA-class models at 7B scale (PUMA, BumbleBee, MPCFormer, BOLT, MARILL) across several tasks. I see no evidence that the head-set and FFN-source effects, which the authors themselves say "vary with the head set and evaluation sample", transfer to larger models, to grouped-query attention, or to other tasks.
(b) Cost measurement. All costs come from a same-host, same-container loopback TFP simulation. The byte figures are rank-0 framework counters ("not wire traffic"); the session times include startup and offline verification but exclude Alice's native preparation, i.e., the full plaintext forward pass of the public backbone over every token. That cost grows with model size and is borne by the client on every decision. No LAN/WAN latency, round counts, preprocessing (Beaver-triple) cost or end-to-end service latency is reported. Since MPC latency in realistic networks is often dominated by rounds rather than bytes, and H5-F01 actually issues 15.79% more backend communication calls than Tail (80,165 vs. 69,233, Table 10), I cannot tell whether the byte savings would turn into wall-clock savings in any deployment.
(c) Timing advantage does not persist. For small F01 versus Tail, the timing gain disappears with generation length: at , F01 is 3.32% and 1.22% slower on two of three prompts, and the third reverses direction between technical repeats (Table 8). Real generation lengths are typically longer than 64 tokens, so the measured benefit reduces to bytes counted in a simulation.
(d) Confounded quality pipeline. The Q16 numerical adaptation (256 straight-through Adam steps) changes quality in metric-specific and configuration-specific ways: F01's BLEU rises while chrF++ falls and NLL worsens (Figure 5). The authors acknowledge that missing unadapted-Q16 and matched FP32-continuation controls "prevent separating arithmetic from additional updates", and that the final Q16 comparison lacks H5-F11 and H16-F01, so head and FFN effects cannot be separated. The development set was also used "in broader exploration".
(e) Reproducibility. The supplementary README states that "the reorganized GPU and MPC entry points have not been rerun end to end", and that raw execution archives, logits and KV tensors are excluded. Combined with the AI-use statement (generative assistants supported "research framing, … implementation and test development, experiment orchestration and collection, result interpretation, and manuscript drafting"), I am not confident that the released code reproduces the reported measurements.
(f) Privacy framing. The guarantee is semi-honest MPC with a trusted dealer, with no new proof and no malicious security, and the simulation "does not verify physical isolation between Alice and Bob". The output functionality releases all logits to Alice, which, as the authors note, enables model extraction (Tramèr et al., 2016). Releasing the full 50,257-dimensional logit vector at every step makes this especially easy for low-rank adapters, whose parameter count is tiny. I think a discussion of how many queries suffice to recover would strengthen the privacy claim considerably.
Questions for authors
-
Pareto evidence. Can you show a quality–cost frontier, for example BLEU/chrF++/NLL against rank-0 bytes and session time, obtained by sweeping , the adapter rank and the number of adapted tail blocks for ScopePEFT, and the LoRA rank and number of adapted blocks for Tail/Deep LoRA, over at least three seeds? Without this I cannot tell whether ScopePEFT improves the frontier or merely occupies a different point on it.
-
Seed noise in the headline claim. The three-seed FP32 study shows H5-F01 behind Tail by 3.892 BLEU (initial groups) and worse on every metric in every seed, while the single-seed Q16 comparison shows H5-F01 ahead by 0.273 BLEU. Could you repeat the final Q16 comparison over seeds 43 and 47 and report means with confidence intervals? Which of the two results do you consider representative?
-
Comparison with MARILL and CryptPEFT. MARILL (Rathee et al., 2024) also restricts secure computation to privately fine-tuned layers of a public backbone, using layer freezing, LoRA and head merging. Could you implement MARILL's configuration on the same GPT-2/WebNLG setup and compare quality and cost directly? In what precise sense does ScopePEFT's native/joint dependency graph go beyond MARILL's layer-freezing design?
-
Client-side cost. Your session times exclude Alice's native preparation, i.e., the plaintext forward pass of the full public backbone for every consumed token. Could you report end-to-end per-token latency including this cost, and discuss how it scales from GPT-2 medium to a 7B model on a typical client device?
-
Network-realistic latency. Could you report measured round counts and end-to-end latency under LAN and WAN settings (e.g., 1 Gbps / 0.5 ms and 100 Mbps / 40 ms RTT), including the offline Beaver-triple preprocessing cost, rather than loopback framework counters? Given that H5-F01 issues 15.79% more backend calls than Tail, does its byte saving survive in a high-latency network?
-
Longer generation. The small-model timing advantage disappears at . What happens to bytes and time at and for prompts of several hundred tokens, which are typical in practice? Is there a crossover length beyond which ScopePEFT becomes strictly slower than Tail LoRA?
-
Scaling to modern LLMs. Can you evaluate at least one LLaMA-class model (e.g., LLaMA-3-8B or Qwen2.5-7B) with grouped-query attention, RMSNorm and SwiGLU FFNs? How do the head-set choice and the FFN-source bit interact with GQA, where several query heads share one KV head, and does still dominate the cost?
-
Disentangling numerical adaptation. Could you add the missing controls, namely unadapted Q16 endpoints and FP32 checkpoints trained for the same additional 256 steps, together with the H5-F11 and H16-F01 Q16 endpoints, so that the effects of fixed-point arithmetic, extra updates, head sets and FFN sources can be separated?
-
Model extraction. Since all 50,257 logits are released to Alice at every decision, how many queries would a semi-honest client need to recover the private low-rank adapters (110,592 or 196,608 parameters) by regression on logit differences? Would releasing only the top- logits or the sampled token inside MPC change the cost profile?
-
Reproducibility. The README states that the reorganised GPU and MPC entry points have not been rerun end to end. Could you confirm that the released code reproduces Table 10 and Figure 2 from the provided checkpoints, and clarify which parts of the experimental pipeline, given the AI-use statement, were implemented and verified directly by the authors?