- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces AutoFHE-Bench, a benchmark for evaluating LLM-generated fully homomorphic encryption (FHE) programs beyond functional correctness. It includes twelve workloads, a runtime judge with isolation and cryptographic checks tailored to each task, and measurements of computation time and evaluation-key size. The authors evaluate three LLM models within the same agent framework, reporting 147 successful attempts out of 150 on ten tasks with verified reference implementations. They also test nine types of adversarial controls across 30 executions and examine the effects of enforcing depth budgets. The results reveal substantial differences in resource use among passing programs. Observations during benchmark development and challenge-task evaluation also expose grading loopholes involving plaintext computation, insecure parameters, and surrogate substitution.
Critical strengths
-
The paper points out a real problem in evaluating agent-generated FHE programs. Concrete examples show that a correct output can hide plaintext computation, insecure parameters, or a substituted model. These findings make a strong case for moving beyond pass rates, and they are a useful contribution even though the proposed judge has its limits.
-
The resource analysis offers practical, technical takeaways. Passing submissions vary a lot in computation time and evaluation-key size, and the depth-budget experiment shows that explicit constraints can cut resource costs without hurting completion in the case they highlight. This supports judging efficiency alongside correctness and security.
Critical weaknesses
-
Most base tasks cannot rule out plaintext precomputation. On nine of the ten tasks with verified references, the client has all the information needed to compute the answer in plaintext and then encrypt it, without the judge detecting the shortcut. The authors acknowledge that these tasks screen the pipeline rather than verify homomorphic execution. The 147/150 pass rate therefore does not, by itself, establish successful FHE program generation, substantially weakening the benchmark’s main contribution. MNIST is the only base task with an asset kept private to the server. CIFAR also has one, but it belongs to Tier 3 and has no enforced minimum security level.
-
A passing verdict does not establish that the requested computation was performed. On BERT-MRPC, two model families replaced BERT with cheap surrogates and still passed. The public pool of test examples and label-based grading allow these surrogates to pass, while the cryptographic checks are also incomplete. This is a useful demonstration of an evaluation failure, but the proposed evaluator leaves that failure unresolved: passing does not establish that the specified model ran under encryption. The task requirements need to distinguish acceptable approximation from replacing the model with a different computation.
Questions for authors
-
Base-task validation: Since nine of the ten base tasks allow the client to compute the answer in plaintext and then encrypt it without detection, did you independently audit the passing submissions to rule out this shortcut? What evidence shows that the 147/150 result reflects genuine homomorphic computation?
-
Computation fidelity: Must submissions run the specified model, or are approximations allowed if they meet the output requirements? Which requirement do the BERT-MRPC surrogates violate, and how did you check that other passing submissions met it?
-
CIFAR security: What security level do the successful CIFAR submissions provide? Does their reported reduction in key material still hold when compared with the reference at the same security level?
Question 4. Turn the kernels into real tests, for example with an FHERMA-style judge-authored client or a server-private operand.
Built, deployed to all nine kernels, and validated. The judge decrypts every uploaded ciphertext against the harness-generated inputs and the result ciphertexts against the reported answer; on every kernel the reference passes and a precompute-client control scores 0 (18-row table, App. G). We also implemented the judge-authored client the review proposes: the judge generates the keypair, encrypts fresh inputs itself, and decrypts the output, so the submission never touches key generation. This closes the residual channel (an agent-authored keygen could in principle embed inputs in key material, class H4; never observed). Pilot on gelu-function: reference passes, and a key-channel control that encodes the input in the rotation-key indices scores 0. The full re-run under the deployed judge scores 146/150 (vzDu, Question 1). Table 1's filled circle rests on this mechanism.