- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents PrivWeave, a benchmark framework for adaptive secure inference in modern language models. Through a systematic review of existing research, the authors identify limitations in execution-plan selection, network settings, and offline cost evaluation across MPC-based, HE-based, and Hybrid secure inference approaches. PrivWeave introduces typed admission contracts, conversion-aware planning, and stateful session accounting to support execution-plan selection and dynamic adaptation under security constraints. Experimental results show that cost-model calibration significantly reduces prediction errors, while the adaptive policy achieves lower accounted online cost under certain improving network conditions but provides limited gains or incurs performance regressions in other scenarios.
Critical strengths
The paper systematically reviews the limitations of existing secure inference approaches in terms of security constraints, network settings, and offline cost evaluation, and proposes a unified benchmark framework. This is practically meaningful for improving the comparability of different secure inference schemes.
PrivWeave incorporates representation compatibility, directional conversions, numerical constraints, and stateful execution costs into execution-plan evaluation. In particular, through a conversion implementation audit and numerical experiments, the paper highlights the limitations of selecting execution plans based solely on operator performance or task accuracy, demonstrating its technical value.
Critical weaknesses
-
Although the paper presents a relatively comprehensive evaluation, its presentation combines too many concepts without sufficient intuitive explanations, making it difficult to follow and fully understand the proposed framework.
-
Although PrivWeave aims to provide a unified benchmark for MPC-based, HE-based, and Hybrid secure inference, the current executable harness supports only a restricted set of protocol combinations and does not sufficiently validate adaptive switching across different cryptographic backends. Furthermore, the end-to-end experiments primarily rely on shaped-loopback and in-process dealer emulation, without comprehensive validation in real-world multi-party deployment environments. These limitations weaken the practical relevance of the proposed framework.
-
Table 3 shows that the approximately 14.6% reduction in accounted online cost is mainly achieved in the WAN-constrained-to-LAN scenario, while the improvements in other scenarios are relatively small, with some results even reversing in the replicate experiments. Moreover, the native session experiments primarily focus on GPT-2 with limited layer configurations and short generation lengths, making it difficult to establish the stability and generalizability of the reported performance gains.
-
The evaluation lacks cross-hardware generalization experiments, leaving it unclear whether the calibrated cost model and the resulting execution-plan selection can maintain their effectiveness across different hardware environments.
Questions for authors
Could the authors provide a more intuitive explanation or a concrete end-to-end example illustrating how the typed admission contracts, conversion-aware planning, and stateful switching work together in PrivWeave?
Table 3 shows that the performance gains of adaptive switching vary considerably across network conditions, with some improvements reversing in the replicate experiments. Could the authors further clarify under which conditions the adaptive policy consistently outperforms fixed plans, and whether these gains hold for larger models and longer decoding sessions?
How well does the calibrated cost model generalize across different hardware environments? Could the authors provide cross-hardware evaluation results or further evidence that the execution-plan selection remains effective on unseen hardware?
2 comments
Response to Reviewer L1Gn (2/2)
Weakness 1 (presentation)
Reading map + worked example + one-sentence intuitions at each paragraph head (blue); the executed-configuration list is tabulated in Section 4.
Weakness 2 (restricted protocols / no real deployment)
Physical two-host link plus EXTERNAL frameworks plus larger models (R3c):
- Physical two-host link: a 4-layer GPT-2 session, T=8, privweave policy, all-LOCAL trajectory with no shaper, seed 20260821; the server side ran on this workstation and the client side on the cloud server, directly connected through an SSH tunnel over the campus network. Session wall time 362.7 s (client view; 482.3 s on the server side including model loading); online time 239.6 s; 3,345 rounds; 647 MB sent; sensed RTT 30.8 ms (SSH encapsulation, higher than the bare link).
- EXTERNAL frameworks: CrypTen 0.4.0 (commit 775868a0), using its built-in two-process multiprocess launcher to run its linear SVM example for 50 epochs to 100% training accuracy, wall time 25 s (no shaping, localhost); two one-line import compatibility fixes are recorded in the manifest; these are EXTERNAL stratum records and are never merged with the local harness records.
- Larger models: R3c.
The in-process-dealer and separate-party boundaries remain honestly declared.
Weakness 3 (gains concentrated / small sessions)
R4–R8 plus the R3a/b/c evidence and the conditional conclusion (Q2); the paper no longer presents 14.6% as universal but states its conditions and counter-examples.
Weakness 4 (cross-hardware)
New Finding 5 (see Q3).
Response to Reviewer L1Gn (1/2)
We accept the Presentation-1 criticism: the concept density was too high. We add two on-ramps (a reading map and a worked end-to-end example) and close the cross-hardware and scale gaps with server measurements.
Q1: A more intuitive explanation / end-to-end example?
Now in the paper (blue, end of Section 3, "Worked example"): a recorded four-layer WANc-to-LAN session is walked through with real numbers — typed admission decides what may run (HE admitted only on client-owned input; HE on shared intermediates is rejected), conversion-aware pricing decides which boundary is cheap (P1 prefill 275 s vs P2 523 s), and stateful switching decides whether the switch pays (P2 is rejected every token while constrained: benefit below cost; after the shift, an 8.22 s install buys ≈ 23 s/token, ending at 516.30 s vs P1's 604.43 s, −14.58%). The same example shows the counter-case (stable WANc: +1.22%), so adaptation is not presented as a free lunch.
Q2: When does adaptive consistently beat fixed plans? Do the gains hold for larger models and longer decoding?
New server evidence gives a conditional answer:
Conditional summary (now in the paper): adaptive consistently beats fixed plans iff (1) the link improves so the challenger's per-epoch saving is predictable; (2) the remaining epochs repay the installation (K_install / per-epoch saving < remaining epochs); and (3) the session is in its cold-start window before installation is amortised. Stable-constrained and worsening links do not guarantee gains, and small margins flip across replicates — consistent with Finding 4, and the reason the benchmark reports sign and margin separately.
Q3: How well does the calibrated cost model generalise across hardware?
New Finding 5 (R9a/R9b): the full 292-configuration grid × 3 repeats was measured on the dual-socket AMD EPYC 7542 server. The frozen local fit transfers with MAPE 41.4% (robust 40.7%; handwritten 180.4%; local held-out 22.9%), versus 22.9% local held-out; per-lane error 59.2/20.2/52.4%; lane-decision top-1 84/84 (100%); realized regret 0 s (the handwritten model scores 0/84 correct lane decisions and incurs 285.3 s of regret). An in-host refit on the server reaches 41.3% MAPE. Conclusion: transfer works at the level of plan selection, but recalibration is required for low error — which positively validates the benchmark's rule that calibration must be tested on unseen hardware.