acceptodds
Under review as a conference paper at ICLR 2027

PrivWeave: Benchmarking Adaptive Secure Inference for Modern Language Models

Abstract

When language-model services run over private prompts and proprietary weights, secure inference, whether MPC-based, HE-based or hybrid, is the de-facto route. However, our systematic review of 128 records finds 21 of 64 cryptographic LLM-inference systems fix their plan, one is environment-aware, 29 report no network setting, and 40 no offline cost. The key building block is the execution plan: typed operators joined by directional, possibly invalid conversions; no silver-bullet plan exists, since the best tracks installed state and link. We present PRIVWEAVE, a benchmark framework testing when adaptation is admissible and beneficial, with three designs: numerically admitted typed contracts, conversion-aware planning, and stateful session accounting. On 181 held-out configurations, fitted costs cut MAPE 13.05 (298.5% to 22.9%). Against the best executed fixed plan, shaped-loopback sessions show 14.6% lower accounted online cost on an improving link, 1.2% higher on a stable constrained one. Across six models and three contexts, numerical emulation meets the perplexity budget in 15 of 18 cells; only 9 show no observed arithmetic-range violation. Even so, no silver-bullet solution covers multi-host confidentiality, malicious security, universal backend coverage, and cross-model gains simultaneously.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, benchmark, multi-party computation (MPC), homomorphic encryption (HE)

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L1GnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents PrivWeave, a benchmark framework for adaptive secure inference in modern language models. Through a systematic review of existing research, the authors identify limitations in execution-plan selection, network settings, and offline cost evaluation across MPC-based, HE-based, and Hybrid secure inference approaches. PrivWeave introduces typed admission contracts, conversion-aware planning, and stateful session accounting to support execution-plan selection and dynamic adaptation under security constraints. Experimental results show that cost-model calibration significantly reduces prediction errors, while the adaptive policy achieves lower accounted online cost under certain improving network conditions but provides limited gains or incurs performance regressions in other scenarios.

Critical strengths

The paper systematically reviews the limitations of existing secure inference approaches in terms of security constraints, network settings, and offline cost evaluation, and proposes a unified benchmark framework. This is practically meaningful for improving the comparability of different secure inference schemes.

PrivWeave incorporates representation compatibility, directional conversions, numerical constraints, and stateful execution costs into execution-plan evaluation. In particular, through a conversion implementation audit and numerical experiments, the paper highlights the limitations of selecting execution plans based solely on operator performance or task accuracy, demonstrating its technical value.

Critical weaknesses
  1. Although the paper presents a relatively comprehensive evaluation, its presentation combines too many concepts without sufficient intuitive explanations, making it difficult to follow and fully understand the proposed framework.

  2. Although PrivWeave aims to provide a unified benchmark for MPC-based, HE-based, and Hybrid secure inference, the current executable harness supports only a restricted set of protocol combinations and does not sufficiently validate adaptive switching across different cryptographic backends. Furthermore, the end-to-end experiments primarily rely on shaped-loopback and in-process dealer emulation, without comprehensive validation in real-world multi-party deployment environments. These limitations weaken the practical relevance of the proposed framework.

  3. Table 3 shows that the approximately 14.6% reduction in accounted online cost is mainly achieved in the WAN-constrained-to-LAN scenario, while the improvements in other scenarios are relatively small, with some results even reversing in the replicate experiments. Moreover, the native session experiments primarily focus on GPT-2 with limited layer configurations and short generation lengths, making it difficult to establish the stability and generalizability of the reported performance gains.

  4. The evaluation lacks cross-hardware generalization experiments, leaving it unclear whether the calibrated cost model and the resulting execution-plan selection can maintain their effectiveness across different hardware environments.

Questions for authors

Could the authors provide a more intuitive explanation or a concrete end-to-end example illustrating how the typed admission contracts, conversion-aware planning, and stateful switching work together in PrivWeave?

Table 3 shows that the performance gains of adaptive switching vary considerably across network conditions, with some improvements reversing in the replicate experiments. Could the authors further clarify under which conditions the adaptive policy consistently outperforms fixed plans, and whether these gains hold for larger models and longer decoding sessions?

How well does the calibrated cost model generalize across different hardware environments? Could the authors provide cross-hardware evaluation results or further evidence that the execution-plan selection remains effective on unseen hardware?

2 comments
AuthorsOpenReview ↗

Response to Reviewer L1Gn (1/2)

We accept the Presentation-1 criticism: the concept density was too high. We add two on-ramps (a reading map and a worked end-to-end example) and close the cross-hardware and scale gaps with server measurements.

Q1: A more intuitive explanation / end-to-end example?

Now in the paper (blue, end of Section 3, "Worked example"): a recorded four-layer WANc-to-LAN session is walked through with real numbers — typed admission decides what may run (HE admitted only on client-owned input; HE on shared intermediates is rejected), conversion-aware pricing decides which boundary is cheap (P1 prefill 275 s vs P2 523 s), and stateful switching decides whether the switch pays (P2 is rejected every token while constrained: benefit below cost; after the shift, an 8.22 s install buys ≈ 23 s/token, ending at 516.30 s vs P1's 604.43 s, −14.58%). The same example shows the counter-case (stable WANc: +1.22%), so adaptation is not presented as a free lunch.

Q2: When does adaptive consistently beat fixed plans? Do the gains hold for larger models and longer decoding?

New server evidence gives a conditional answer:

  • R4 (new WAN regime, 100 Mbit/s / 40 ms): stable-WAN margin −8.09% (vs P2); shift-WAN-to-LAN margin −0.83% (vs P2).
  • R5 (12-layer anchors × 2 seeds): WANc-to-LAN 12-layer margin −4.56% (mean over 2 seeds, vs P1; the frozen single run is 5.58%), reported per seed.
  • R6 (T=32 long decoding): WANc-to-LAN margin −24.92% (vs P2; P2 1738.1 s vs adaptive 1304.9 s).
  • R8 (prompt length 64): shift-WANc-to-LAN (p64) −12.01% (vs P2); stable-LAN (p64) −6.39% (vs P2).
  • R3a/R3b (full-depth 32-token decode; 256-token prompt): wall 136.5 min; 12 layers, 32 tokens, LAN, peak RSS ≈ 14 GB per process; wall 113.0 min; match float 1.000, emulated 0.750 (6 of 8).
  • R3c (gpt2-medium 355M / large 774M / xl 1.5B full-depth decodes): the scale–cost curve is as follows. gpt2-medium (24 layers): online 1757.1 s, 5.20 GB, 18,232 rounds, 50.9 GB of offline material, peak RSS 23.0 GB per process; gpt2-large (36 layers): wall 55.9 min, peak RSS 48.8 GB per process; gpt2-xl (48 layers): wall 51.7 min, peak RSS 93.5 GB per process. For all three, the float and emulated references agree with each other (match 1.0), but the native execution path diverges on every token (match 0.0), consistent with the probabilistic truncation failure hazard recorded in the artifact's GADGETS records (per-element failure probability ≈ 2^-13, growing with the number of LayerNorm rows and with depth). Per the benchmark's numerical admission rules, these runs are marked as not rankable: only cost data are released, with no quality claims.

Conditional summary (now in the paper): adaptive consistently beats fixed plans iff (1) the link improves so the challenger's per-epoch saving is predictable; (2) the remaining epochs repay the installation (K_install / per-epoch saving < remaining epochs); and (3) the session is in its cold-start window before installation is amortised. Stable-constrained and worsening links do not guarantee gains, and small margins flip across replicates — consistent with Finding 4, and the reason the benchmark reports sign and margin separately.

Q3: How well does the calibrated cost model generalise across hardware?

New Finding 5 (R9a/R9b): the full 292-configuration grid × 3 repeats was measured on the dual-socket AMD EPYC 7542 server. The frozen local fit transfers with MAPE 41.4% (robust 40.7%; handwritten 180.4%; local held-out 22.9%), versus 22.9% local held-out; per-lane error 59.2/20.2/52.4%; lane-decision top-1 84/84 (100%); realized regret 0 s (the handwritten model scores 0/84 correct lane decisions and incurs 285.3 s of regret). An in-host refit on the server reaches 41.3% MAPE. Conclusion: transfer works at the level of plan selection, but recalibration is required for low error — which positively validates the benchmark's rule that calibration must be tested on unseen hardware.

AuthorsOpenReview ↗

Response to Reviewer L1Gn (2/2)

Weakness 1 (presentation)

Reading map + worked example + one-sentence intuitions at each paragraph head (blue); the executed-configuration list is tabulated in Section 4.

Weakness 2 (restricted protocols / no real deployment)

Physical two-host link plus EXTERNAL frameworks plus larger models (R3c):

  • Physical two-host link: a 4-layer GPT-2 session, T=8, privweave policy, all-LOCAL trajectory with no shaper, seed 20260821; the server side ran on this workstation and the client side on the cloud server, directly connected through an SSH tunnel over the campus network. Session wall time 362.7 s (client view; 482.3 s on the server side including model loading); online time 239.6 s; 3,345 rounds; 647 MB sent; sensed RTT 30.8 ms (SSH encapsulation, higher than the bare link).
  • EXTERNAL frameworks: CrypTen 0.4.0 (commit 775868a0), using its built-in two-process multiprocess launcher to run its linear SVM example for 50 epochs to 100% training accuracy, wall time 25 s (no shaping, localhost); two one-line import compatibility fixes are recorded in the manifest; these are EXTERNAL stratum records and are never merged with the local harness records.
  • Larger models: R3c.

The in-process-dealer and separate-party boundaries remain honestly declared.

Weakness 3 (gains concentrated / small sessions)

R4–R8 plus the R3a/b/c evidence and the conditional conclusion (Q2); the paper no longer presents 14.6% as universal but states its conditions and counter-examples.

Weakness 4 (cross-hardware)

New Finding 5 (see Q3).

Reviewer_SdnqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PrivWeave proposes a benchmark framework combining security and numerical contracts, conversion-aware planning, and stateful accounting for adaptive secure Transformer inference. Its evaluation combines a 128-record literature review, operator cost calibration, plaintext numerical emulation, and shaped loopback GPT 2 sessions. The results demonstrate improved cost prediction, numerical admissibility failures, and conditional benefits from adaptive switching.

Critical strengths
  1. The framework usefully integrates security compatibility, directional conversion costs, and preprocessing reuse into a common evaluation methodology.
  2. Cross-model experiments demonstrate that acceptable perplexity can coexist with arithmetic-range violations, exposing a limitation of task-score-only evaluation.
  3. The signed-wrap counterexample provides concrete evidence that superficially compatible HE/MPC conversions can produce incorrect results.
Critical weaknesses
  1. Despite its cross-paradigm scope, executed inference is restricted to one custom GPT-2 harness with MPC Transformer layers and HE embedding, without matched comparisons across independent backends.
  2. Security-conditioned admission remains incompletely validated because the executed H2A conversion lacks a quantified leakage bound and shared-seed dealer emulation does not establish deployment confidentiality.
  3. Excluding dealer provisioning and discarded-material regeneration leaves the headline switching gains unvalidated under the complete lifecycle accounting that motivates the framework.
  4. Table 4 incorrectly labels SAL-ViT, a vision-transformer study, as LLM inference, weakening confidence in the corpus-derived statistics.
Questions for authors
  1. What evidence demonstrates that the custom harness captures the backend trade-offs of independently implemented MPC, HE, and hybrid inference systems?
  2. Which configurations have task quality and latency validated together under actual secure execution rather than separate plaintext emulation?
  3. Does correcting SAL-ViT’s classification change the reported corpus counts, and how were the remaining classifications checked?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Sdnq

We thank the reviewer for catching the SAL-ViT misclassification — verified line by line, it was an error, and the paper is fully corrected (blue) with exact before/after counts. The other three weaknesses accurately restate boundaries the paper already disclosed; this rebuttal addresses them with new measurements.

Q3: Does correcting SAL-ViT change the corpus counts, and how were the remaining classifications checked?

Yes. After flipping is_llm_transformer_inference for SAL-ViT (1→0), every derived count was recomputed:

StatisticBeforeAfter
Transformer LM-inference records8584
Cryptographic LLM systems6463
MPC-based2019 (HE 22, Hybrid 22 unchanged)
semi-honest only6059
2PC + dealer1514
network none-reported2928
offline cost not reported4039
adaptivity model-codesign2120
nonlinear co-design2120
largest model ≤ 110M1817
2023 network-reporting share80%100% (the 2023 sample drops 5→4, below the ≥5-systems panel threshold; Fig. 5(c) now starts at 2024's 64.3%)

How the rest were checked: we re-audited the LLM column of all 128 rows against each paper's evaluated object. SAL-ViT (an ImageNet ViT) was the only error; the other 127 rows match their sources — vision/generic/framework records such as MPCViT, PriViT, Orca, CryptoNets, HE-LoRA, Slalom, 2PC-LTH and DarKnight were already correctly marked "--". CSV, JSON, Tables 1/2/4, Figures 1/2/5 and the text are now mutually consistent, and the generator assertions were updated so future regeneration cannot drift. The qualitative findings survive: 21 systems fix their plan, only one is environment-aware, 20+ HE systems report no network setting, and roughly 3/5 report no offline cost.

Q1: What evidence shows the custom harness captures the trade-offs of independent MPC/HE/hybrid backends?

A stratified answer:

  1. the harness's operators implement the published protocols parameter-for-parameter (Beaver/FSS gates, CKKS linear lane, H2A);
  2. cross-hardware transfer (new Finding 5, 41.4% MAPE; robust 40.7%; handwritten 180.4%; local held-out 22.9%) measures how well the calibrated cost model and plan selection persist on unseen hardware;
  3. the physical two-host run and the EXTERNAL framework run provide independent-implementation evidence. Executed — CrypTen 0.4.0 (commit 775868a0), using its built-in two-process multiprocess launcher to run its linear SVM example for 50 epochs to 100% training accuracy, wall time 25 s (no shaping, localhost); two one-line import compatibility fixes are recorded in the manifest; these are EXTERNAL stratum records and are never merged with the local harness records.

Any inference beyond these layers is explicitly marked unvalidated.

Q2: Which configurations have task quality and latency validated together under actual secure execution?

Honest answer, now written into the paper in blue: no configuration yet validates task quality and latency together under actual secure execution. The closest joint validation is R3a: at full depth (12 layers, 32 tokens, LAN), the executed tokens equal the float32 and emulated references (1.000/1.000, float/emulated, 32 tokens; the four-layer matrix had already agreed in all 40 replicate runs). Quality budgets remain strictly SF-emulation evidence.

Weakness 2 (H2A leakage bound / dealer emulation)

The paper continues to state "no quantified leakage bound" and now separates checked mechanisms from unvalidated guarantees (Section 4 and Appendix C, blue):

  • Checked: the slack ≤ 20-bit cap (source-enforced and audited), the CKKS parameters (N=8192, scale 2^40, moduli (60,40,40,60)), and the fact that the session costs measure exactly this capped-mask construction;
  • Unvalidated: per-secret/mask-range/CKKS-error bounds, accumulated statistical leakage across slots and calls, and circuit privacy.

On the dealer: the physical two-host run shows the protocol path operates in a real two-machine topology over a real network; separate-party dealer provisioning remains an explicitly listed unvalidated extension and is never used in headline claims.

Weakness 3 (full lifecycle accounting)

We add a complete-lifecycle accounting view: session records carry per-epoch dealer timings (4.3–5.8% of wall time; Adaptive 5.8%, P1 4.4%, P2 4.3%), and the regeneration cost of discarded material is measured by the R9c install calibration (wmask_install 768×768 (LAN) ≈ 0.125 s / 9.44 MB (0.013 s/MB); he_embedding 50257×768 (LAN) ≈ 43.1 s / 4.59 MB (9.4 s/MB), each with 3 repeats). With dealer and regeneration included, adaptive's advantage over the best fixed plan on WANc-to-LAN is 13.37% (regeneration of discarded material costs only 0.005 s at the wmask rate; even at the HE embedding rate of 3.70 s, the advantage remains 12.78%), versus 14.58% previously when excluding them.

Reviewer_u8sHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

PrivWeave studies how to compare and adapt secure Transformer inference plans when representations, numerical constraints, network conditions, and setup costs differ. It combines a literature review with typed admission rules, conversion-aware planning, and a controller that accounts for installed state before switching plans. The experiments cover cost prediction, plaintext numerical emulation, and a local two-process inference harness. They show that acceptable task scores can coexist with numerical-range violations, and that accounting for installation costs can make adaptation worthwhile.

Critical strengths
  1. The evaluation protocol makes security assumptions, numerical validity, and setup costs explicit. These are consequential choices in secure inference, and recording them together provides a useful basis for deciding which results can reasonably be compared.

  2. The conversion counterexample and the cases where perplexity remains acceptable despite numerical-range violations provide concrete evidence that task scores alone are insufficient. These findings give substance to the proposed evaluation criteria and identify checks that future implementations can use.

Critical weaknesses
  1. The benchmark’s value for comparing independent systems is only partly demonstrated. The experiments compare plans within the authors’ own harness, without matched runs of independent secure-inference implementations under a common contract. Quality is evaluated through plaintext emulation, while timing uses a local dealer simulation, so security conditions, numerical quality, and cost have not been validated together in one executable configuration. This limits the evidence for the comparative framework that motivates the paper.

  2. The additional contribution of the planning framework remains unclear. Conversion-aware cost models and dynamic programming have precedents in HyCC (Büscher et al., 2018), while the current ablations mainly establish the importance of installation costs. Most sessions generate eight tokens with a four-layer GPT-2 prefix, and the clearest gain in Table 3 comes from postponing mask installation until a WAN-to-LAN transition, with KV-cache migration fixed at zero. This is a useful scheduling result, but provides limited support for the broader benefits of adaptive planning.

Questions for authors
  1. Are matched results from independent secure-inference implementations available under the same security and numerical contract?

  2. Which security conditions and numerical guarantees have been validated for the executable configuration used in the timing experiments, and how does this configuration relate to the plaintext quality evaluation?

  3. What do the existing session traces or cost analysis imply about adaptation gains across repeated requests once reusable installation costs have been amortized?

2 comments
AuthorsOpenReview ↗

Response to Reviewer u8sH (1/2)

We thank the reviewer for the Soundness-3 assessment and for recognising the evaluation protocol. The three evidence gaps identified (matched independent implementations, joint validation, post-amortisation gains) are real and were already disclosed in the paper; in this rebuttal we close them with new measurements on the cloud server and clarify the delta over HyCC-style precedents. All new evidence is in the revised paper (blue).

Q1: Are matched results from independent secure-inference implementations available under the same security and numerical contract?

No off-the-shelf third-party run matches our declared contract field by field — which is exactly why PrivWeave makes origin (author artifact / local reimplementation / reported-only) and scope mandatory record fields. We add three layers of new evidence, kept strictly stratified (EXTERNAL records are never merged with our harness rows):

  • Cross-hardware calibration transfer (new Finding 5): the full 292-configuration operator grid was re-measured (3 repeats) on unseen hardware (dual-socket AMD EPYC 7542); the frozen local fit transfers with MAPE = 41.4% (robust 40.7%; handwritten 180.4%; local held-out 22.9%), versus 22.9% held-out locally; per-lane error 59.2/20.2/52.4%; lane-decision top-1 = 84/84 (100%); realized regret 0 s (the handwritten model scores 0/84 correct lane decisions and incurs 285.3 s of regret). An in-host refit on the server reaches 41.3% MAPE.

  • Physical two-host link: run_two_host_program.py executed a 4-layer session over the real campus-network link between the cloud server and our local workstation with no shaper (4-layer GPT-2 session, T=8, privweave policy, all-LOCAL trajectory with no shaper, seed 20260821; the server side ran on this workstation and the client side on the cloud server, directly connected through an SSH tunnel over the campus network). Session wall time 362.7 s (client view; 482.3 s on the server side including model loading); online time 239.6 s; 3,345 rounds; 647 MB sent; sensed RTT 30.8 ms (SSH encapsulation, higher than the bare link).

  • Independent frameworks (EXTERNAL stratum): Executed — CrypTen 0.4.0 (commit 775868a0), using its built-in two-process multiprocess launcher to run its linear SVM example for 50 epochs to 100% training accuracy, wall time 25 s (no shaping, localhost). Two one-line import compatibility fixes are recorded in the manifest. These are EXTERNAL stratum records and are never merged with the local harness records.

The artifact's external_repositories.csv lists the official repositories of every reviewed system, so any third party can re-run under our declared contract and workload — the benchmark's value is precisely to fix that comparable contract.

Q2: Which security conditions and numerical guarantees are validated for the timed configuration, and how does it relate to the plaintext quality evaluation?

We added an explicit "Validated conditions of the executed configuration" list to Section 4 (blue):

Validated: (i) the protocol path really executes (SH-2PC, CPU-only), and its output tokens match the float32 and emulated references — across all 40 four-layer replicate runs, and now the full-depth 12-layer 32-token decode (R3a): 1.000/1.000 (float/emulated, 32 tokens); (ii) the H2A mask slack ≤ 20 bits is source-enforced and audited; (iii) the admission gate rejects HE linear layers on shared intermediates and the A2H edge; (iv) the audit reproduces the signed-wrap defect under real BFV encryption.

Not validated (stated as such): (1) no quantified leakage bound for the executed H2A construction; (2) the shared-seed dealer is not separate-party provisioning; (3) both processes hold the checkpoint, so no process-level confidentiality follows; (4) malicious security is not instantiated.

Relation to SF quality: SF emulates the same gadgets with the same fixed-point parameters; the two evidence classes are linked only by token-level agreement, so perplexity/accuracy budgets remain SF evidence and are never presented as secure-accuracy claims. R3a extends that link from 4 layers to full depth.

AuthorsOpenReview ↗

Response to Reviewer u8sH (2/2)

Q3: What do the session traces / cost analysis imply about adaptation gains once reusable installation costs are amortised?

We ran the R7 amortisation chain on the server (P2, LAN, N=100 independent requests, 4 layers, T=2, one cold-start install): installation is 25.8% of the first request's cost; per-request amortised cost is 83.9 s at N=10 and 51.0 s at N=100 (warm requests 50.7 s each). Combined with the constant-regime break-even proposition and the measured K, c_p, c_q per trace, the first profitable horizon is computed per trace: in the WANc-to-LAN session the switch pays at token 5 (8.22 s install vs ≈ 23 s/token saved), whereas on stable LAN the install is 35.5 s ≈ 35% of the first request's online time; the installation share falls to 4.2% at N=10 and 0.7% at N=100. As the reviewer anticipated: once reusable installation is amortised, fixed plans regain competitiveness — adaptive gains concentrate in the cold-start / link-transition window. The paper now states this condition explicitly rather than implying a universal speedup.

Weakness 1 (value for comparing independent systems)

See Q1; the External frameworks / Network / Hardware rows of Table 2 change from "equivalent emulated" to measured (blue).

Weakness 2 (no joint validation)

See Q2; the full-depth three-way token agreement is the closest joint validation of quality and protocol execution in one configuration, and the paper states in blue that no configuration yet validates all three conditions simultaneously.

Weakness 3 (planning contribution vs HyCC)

Section 2 now states the three testable deltas in blue — (i) typed admission is a contract predicate, not a cost heuristic; (ii) switching is stateful (installed masks, spent preprocessing and KV state persist across token boundaries; a switch is charged before any gain is credited); (iii) the benchmark scores a causal controller against executed fixed plans plus five ablations. The new R4–R8 session matrices and the amortisation chain widen the evidence for adaptive planning beyond a single trace.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.