acceptodds
Under review as a conference paper at ICLR 2027

AutoFHE-Bench: Beyond Pass Rates in Agent-generated Fully Homomorphic Encryption Programs

Abstract

Fully Homomorphic Encryption (FHE) enables computation over encrypted data without decryption, serving as a foundation for privacy-preserving machine learning. Writing an FHE program remains expert-only work: developers must manually pick encryption parameters, layout ciphertexts, and approximate unsupported operations, where any mistake fails silently under encryption. While LLM agents excel at general software synthesis, applying them to FHE exposes a fundamental vulnerability: a program can produce the correct answer while providing zero confidentiality. Prior evaluations infer security merely from static source code, a check easily satisfied by programs that invoke homomorphic APIs while computing in the clear. We show that their remaining axis, functional correctness, is nearly saturated: on ten end-to-end tasks from established FHE benchmarking suites, each with a verified reference, three frontier models succeed in 147 of 150 attempts. The open question is therefore no longer whether an agent can write a working FHE program, but whether that program is secure and deployable. We introduce AutoFHE-Bench, which treats security as an observable runtime property of an execution. Across multi-stage pipelines from key generation to decryption, an independent judge holds the secret keys, draws fresh inputs, enforces OS-level client–server isolation, and decrypts results itself, and it bounds multiplicative depth and security level on the tasks that admit such a bound. Nine adversarial control types score zero across 30 executions. The benchmark shows that submissions receiving the same verdict differ, at the extremes, by in encrypted-computation time on matrix inversion and in evaluation-key size on encrypted retrieval, and that stating and enforcing a depth budget reduces one model's median key material by at an unchanged pass rate. Two exploratory challenge tasks, reported separately, bound what the judge establishes: on one a model synthesizes a bootstrapped ResNet-20 for CIFAR-10 at one quarter of the reference key material, and on the other a passing submission shows that verifying an encrypted execution does not verify which computation it performed. We open-source the tasks, the judge, and the controls.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
FHE, code generation, privacy, ppml, security

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vzDuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AutoFHE-Bench, a benchmark for evaluating LLM-generated fully homomorphic encryption (FHE) programs beyond functional correctness. It includes twelve workloads, a runtime judge with isolation and cryptographic checks tailored to each task, and measurements of computation time and evaluation-key size. The authors evaluate three LLM models within the same agent framework, reporting 147 successful attempts out of 150 on ten tasks with verified reference implementations. They also test nine types of adversarial controls across 30 executions and examine the effects of enforcing depth budgets. The results reveal substantial differences in resource use among passing programs. Observations during benchmark development and challenge-task evaluation also expose grading loopholes involving plaintext computation, insecure parameters, and surrogate substitution.

Critical strengths
  1. The paper points out a real problem in evaluating agent-generated FHE programs. Concrete examples show that a correct output can hide plaintext computation, insecure parameters, or a substituted model. These findings make a strong case for moving beyond pass rates, and they are a useful contribution even though the proposed judge has its limits.

  2. The resource analysis offers practical, technical takeaways. Passing submissions vary a lot in computation time and evaluation-key size, and the depth-budget experiment shows that explicit constraints can cut resource costs without hurting completion in the case they highlight. This supports judging efficiency alongside correctness and security.

Critical weaknesses
  1. Most base tasks cannot rule out plaintext precomputation. On nine of the ten tasks with verified references, the client has all the information needed to compute the answer in plaintext and then encrypt it, without the judge detecting the shortcut. The authors acknowledge that these tasks screen the pipeline rather than verify homomorphic execution. The 147/150 pass rate therefore does not, by itself, establish successful FHE program generation, substantially weakening the benchmark’s main contribution. MNIST is the only base task with an asset kept private to the server. CIFAR also has one, but it belongs to Tier 3 and has no enforced minimum security level.

  2. A passing verdict does not establish that the requested computation was performed. On BERT-MRPC, two model families replaced BERT with cheap surrogates and still passed. The public pool of test examples and label-based grading allow these surrogates to pass, while the cryptographic checks are also incomplete. This is a useful demonstration of an evaluation failure, but the proposed evaluator leaves that failure unresolved: passing does not establish that the specified model ran under encryption. The task requirements need to distinguish acceptable approximation from replacing the model with a different computation.

Questions for authors
  1. Base-task validation: Since nine of the ten base tasks allow the client to compute the answer in plaintext and then encrypt it without detection, did you independently audit the passing submissions to rule out this shortcut? What evidence shows that the 147/150 result reflects genuine homomorphic computation?

  2. Computation fidelity: Must submissions run the specified model, or are approximations allowed if they meet the output requirements? Which requirement do the BERT-MRPC surrogates violate, and how did you check that other passing submissions met it?

  3. CIFAR security: What security level do the successful CIFAR submissions provide? Does their reported reduction in key material still hold when compared with the reference at the same security level?

Reviewer_H8hROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AutoFHE-Bench has twelve privacy-preserving-ML workloads from FHERMA and the HomomorphicEncryption.org suite, in which an LLM agent writes the complete FHE program. The main contribution is an execution-time judge. It runs the client and server as separate OS users, holds the keys and inputs, checks a security floor and a depth budget, rerandomizes the ciphertexts that cross between the stages, and decrypts the result itself. Evaluating GLM-5.3, GPT-5.6 and Claude Opus 5.5, the authors report:

  • 147/150 passes on the ten tasks that have a verified reference;
  • up to 88× spread in runtime and 164× in key size among passing submissions;
  • a 3.3× reduction in key material once a depth budget is enforced.
Critical strengths
  1. An execution-based judge built on a sound principle. The judge follows the rule that "a boundary holds only when it constrains something the agent does not author". It applies this rule through OS-level role separation, rerandomization and independent decryption. Nine control types are validated by marker. It is a reusable improvement over source-level checks, and its gaps are disclosed candidly.
  2. Shows, and explains, that a pass verdict hides deployment cost. The matrix-inverse case study traces the 88× gap to a single packing decision, d² vs. d³ slots. That decision changes the depth, then log₂QP, then the ring dimension. The resulting recommendation to report all four cost axes is well supported.
Critical weaknesses
  1. The central differentiator is that a pass certifies homomorphic execution rather than cleartext computation. By the paper's own account, this holds for at most one of the ten graded tasks.

    • Confidentiality against the server is enforced. However, the judge cannot detect a client that computes in cleartext and then encrypts the result, unless one side holds a private asset (Sec. 3.1, l. 218–221; App. F.4, l. 1310–1313). Only ml-inference-mnist has such an asset (Table 4, l. 878). The other nine tasks account for 135 of the 150 attempts (derived: 9 tasks × 3 models × 5 attempts).
    • Table 1 nonetheless gives AutoFHE-Bench a full ● for rejecting "Cleartext compute" (l. 177–178, p. 4).
    • The BERT claim (Sec. 4.4, l. 412–414) is that "Every mechanism of the judge accepts them". This contradicts Table 4 footnote h: "No judge programs exist for this backend" (l. 889).
    • Why consequential: the novelty over prior evaluators rests on this guarantee. The authors themselves call these tasks "screening tasks for the pipeline rather than as tests of homomorphic execution" (App. F.4, l. 1318–1319), but the abstract omits this.
  2. The headline cost numbers compare submissions under unreported or unconstrained security levels and different grading regimes.

    • The paper says the judge records the "achieved security level" (Sec. 1, l. 099–101), but no table or figure reports it (not reported in Fig. 2, Table 4, or App. C.5–E.2).
    • The 164× spread comes from fetch-by-similarity, whose security floor is 0 (Table 4, footnote d, l. 886–887).
    • The 88× endpoint (650.1 s) comes from the unguarded sweep (App. C.6, l. 1068) and declared depth 12 against a budget of 10 (App. E.1, l. 1227; Table 4, l. 875).
    • The CIFAR "quarter" compares a 128-bit reference with unconstrained parameters (App. A.4, l. 719–721; Sec. 4.4, l. 398–399).
    • Why consequential: the paper itself argues that an insecure ring "does not provide weaker confidentiality, it provides none" (Sec. 6, l. 472–473). Its own headline efficiency numbers make exactly that kind of unscored comparison.
Questions for authors
  1. Are the grading-time MNIST server weights also visible during development? If so, what prevents a client from embedding a classifier (Sec. 3.2, l. 257; App. A.3, l. 702–703)? A mechanism ruling this out would weaken W1. If there is none, I would consider the guarantee established on no task.
  2. Please report the achieved security of every passing submission (Fig. 2a,b) and of the CIFAR-10 context (App. E.2, l. 1253–1255). Please also restate the 164×, 88×, 26% and "quarter" figures using ≥128-bit submissions graded under one rule. If the conclusions survive, I would raise Soundness and Rating.
  3. How is the causal claim "the cause is the grading rule rather than the models" (Sec. 6, l. 470–471) justified? The evidence is two sequential sweeps with 5 attempts per cell, only 4 of 8 tasks are comparable, and three budgets were tightened after models found cheaper designs (App. C.6, l. 1052–1061; App. F.4, l. 1370–1371). Concurrent arms or confidence intervals would address this.
  4. Turn the kernels into real tests, for example with an FHERMA-style judge-authored client or a server-private operand. This would directly address W1.
2 comments

Question 4. Turn the kernels into real tests, for example with an FHERMA-style judge-authored client or a server-private operand.

Built, deployed to all nine kernels, and validated. The judge decrypts every uploaded ciphertext against the harness-generated inputs and the result ciphertexts against the reported answer; on every kernel the reference passes and a precompute-client control scores 0 (18-row table, App. G). We also implemented the judge-authored client the review proposes: the judge generates the keypair, encrypts fresh inputs itself, and decrypts the output, so the submission never touches key generation. This closes the residual channel (an agent-authored keygen could in principle embed inputs in key material, class H4; never observed). Pilot on gelu-function: reference passes, and a key-channel control that encodes the input in the rotation-key indices scores 0. The full re-run under the deployed judge scores 146/150 (vzDu, Question 1). Table 1's filled circle rests on this mechanism.

Question 1. Are the grading-time MNIST server weights also visible during development? If so, what prevents a client from embedding a classifier?

Yes, they are visible: development is single-user by design (App. F.4). The batch rule is what prevents it from paying: grading uses 100 fresh images with accuracy required within one point of the plaintext model, and both crossings are rerandomized. As a measurement, we wrote an embedded-classifier control (a logistic head distilled from the development images): it scores 91/100 against the plaintext model's 97/100 and fails the one-point margin. We will state this analysis in App. A.3. We do not claim the guarantee for the single-image mnist task.

Question 3. How is the causal claim "the cause is the grading rule rather than the models" justified?

As written, it was not justified: two sequential sweeps with five attempts per cell cannot support the sentence (Sec. 6, l. 470–471). We will reword it, and we ran the confirmatory experiment the review implies: a concurrent two-arm sweep, budget on versus off, n = 20 attempts per arm per task, same model snapshots, interleaved scheduling.

Results: median evaluation-key material 2410 MB → 980 MB (2.5×, permutation p = 0.0002, 95% CI 1.8–3.4×); median server runtime 41 s → 33 s (p = 0.04); pass rate unchanged (39/40 vs 38/40). The stored-data analyses agree: on the one retrospectively comparable task, key material fell 2715 MB → 1006 MB (p = 0.0043); across ten budget values, 154 of 166 passes sit exactly at budget and none above; zero of 156 budgeted trajectories triggered a runtime depth violation, so the mechanism is prompt-mediated.

We will revise the claim (Sec. 6) to: "Enforcing a depth budget reduces key material 2.5× (concurrent two-arm sweep, n = 20/arm, p = 0.0002) without reducing pass rate. Runtime effects are smaller (1.2×, p = 0.04)."

Presentation revisions we will make: the Table 1 footnote (Question 4); the abstract will state the scope of the guarantee; Sec. 4.4 will become "every judge mechanism that exists for this backend accepts them" (Table 4, footnote h); Sec. 6 will carry the revised causal claim; Tier 1 will be reported as grader and reviewed.

Reviewer_hUcqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AutoFHE-Bench, a benchmark for evaluating the security and efficiency of LLM agent-generated FHE programs beyond functional correctness. It employs an independent judge with client-server isolation, cryptographic validation, and ciphertext rerandomization. Experiments show that three frontier models pass 147/150 attempts, yet their programs exhibit substantial differences in runtime and evaluation-key size. Enforcing depth constraints improves efficiency without reducing pass rates. The paper also highlights that verifying encrypted execution does not guarantee that the intended computation was performed.

Critical strengths
  1. The paper introduces a runtime-based evaluation framework that incorporates client-server isolation, cryptographic parameter validation, and ciphertext rerandomization to assess the security of agent-generated FHE programs.

  2. This work reveals substantial efficiency differences among passing FHE programs and demonstrates that enforcing multiplicative-depth constraints can reduce resource consumption without compromising pass rates.

Critical weaknesses
  1. In Section 3 and Section 5, most benchmark tasks cannot fully establish genuine homomorphic execution. For eight kernels and fetch-by-similarity, client-side plaintext computation remains possible, weakening the benchmark's central claim of providing security-aware evaluation.

  2. In Section 3, security enforcement mechanisms are applied only where workloads support them, resulting in inconsistent verification guarantees across tasks. This heterogeneity raises concerns about whether benchmark results can be meaningfully compared under a unified security evaluation standard.

  3. In Section 4, near-saturated pass rates across the ten reference-validated tasks raise concerns about benchmark difficulty and discriminative power. Even after removing methodological hints, completion remains high, limiting the benchmark's ability to distinguish frontier models by functional capability.

  4. In Section 4 and Section 5, surrogate models can pass the judge without performing the intended computation, as demonstrated by the BERT-MRPC experiments. This exposes a fundamental gap between verifying encrypted execution and establishing computational fidelity, leaving an important evaluation objective unresolved.

Questions for authors
  1. In Section 3, how can the benchmark reliably verify genuine homomorphic execution when client-side plaintext computation remains possible for most tasks?

  2. In Section 3, how do the authors justify cross-task comparisons when security enforcement mechanisms and verification guarantees differ across workloads?

  3. In Section 4, how can AutoFHE-Bench effectively distinguish frontier models when pass rates are nearly saturated, even after removing methodological hints?

  4. In Section 4, how could the evaluation framework detect surrogate model substitution and ensure that passing submissions perform the intended computation rather than merely produce correct outputs?

1 comment

Response to Reviewer hUcq

We thank Reviewer hUcq for the careful review.

Question 1. How can the benchmark reliably verify genuine homomorphic execution when client-side plaintext computation remains possible for most tasks?

It now verifies it by mechanism on every kernel, and the recorded results are verified by audit. The judge decrypts every uploaded ciphertext against the harness-generated inputs and the output against the reported answer, so the computation can occur only under encryption; on all nine kernels the reference passes and a precompute-client control scores 0, and two post-decryption controls score 0 at the result check. The audit of the recorded sweep used a judge-independent error bound (depth d bounds polynomial degree at 2^d) and found two violations; Tier 1 is reported as grader 147/150, reviewed 145/150, and the re-run under the new judge scores 146/150. Detail in our response to Reviewer vzDu, Question 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.