- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In my reading, the paper presents FHE-Agent, an LLM-guided multi-agent framework that automates the configuration of CKKS homomorphic encryption for private CNN inference on top of the Orion compiler and the Lattigo backend. A configuration couples global CKKS parameters (ring dimension with in all reported results, encoding scale , the RNS modulus chain , the key-switching auxiliary chain , the bootstrapping auxiliary primes, the secret Hamming weight ) with Orion packing options and layer-local overrides (embedding method, baby-step/giant-step ratio):
where contains configurations that pass a 128-bit parameter-security check, complete encrypted execution, and satisfy and on the evaluated input.
Five LLM roles share one model and a common history: InitAgent and RegimeAgent for initial proposals, GlobalTradeoffAgent and LayerwiseAgent for global and layer-local revisions, and PatchGateAgent for selection and outcome assessment. They run in a three-phase loop (propose and analyse, establish a verified best, refine with feedback). A deterministic tool suite (StaticAnalyzer, LayerProfiler, BootstrapScheduleAnalyzer, a calibrated CostModel , and EncryptedEvaluator) supplies evidence. Crucially, only measured, gate-passing results may replace the verified best.
The reported best configurations take 14.56 s for LeNet, 267.66 s for AlexNet and 604.79 s for ResNet20 on CPU. Refinement improves the first feasible configuration by 15.2% (LeNet) and 12.0% (ResNet20) and not at all for AlexNet. Operation-level profiles attribute most of the gain to bootstrapping (AlexNet, ResNet20) or linear transforms (LeNet). In a same-LLM LeNet pilot with three LLMs (GPT-6 Astra, GPT-6 Luna, Gemma-4-31B), FHE-Agent is the only method whose nine attempts all pass the gates. The authors explicitly scope their claims to "configuration quality in recorded runs, not repeated-deployment speedups or search cost".
Critical strengths
Strength 1. In my opinion, the division of authority between the LLM and the deterministic toolchain is sound and well motivated. I appreciate that the LLM is never trusted with feasibility. Parameter-security checks over the effective contexts (including bootstrap contexts and the secret distribution), encrypted execution and the MAE/precision gates are deterministic, and the incumbent is replaced only when a measured result passes the gates and improves (Algorithm 1, line 13). I think this is exactly the right design for a security-critical domain, where a plausible-sounding but wrong LLM suggestion (for instance, an auxiliary-prime width that silently breaks bootstrapping, as in the 61→55-bit failure of Table 6) must never reach deployment. The paper also reports LLM failures candidly, which I value:
- GlobalTradeoffAgent twice proposed reducing the bootstrap from four to three primes in ResNet20, and both proposals slowed inference (785.01 s and 806.87 s versus incumbents of 687.59 s and 682.15 s);
- the second proposal repeated the hypothesis despite having the refuting measurement in its history;
- the CostModel's forecasts were 42–59% below the measured times and ranked the fastest candidate last. Showing that shared feedback does not guarantee that agents correct a mistaken hypothesis is, from my perspective, a useful negative finding for the LLM-agent community.
Strength 2. I find the operation-level attribution and the paired single-setting comparisons genuinely informative for FHE practitioners. The decomposition into bootstrap, linear-transform and polynomial time (Figure 3, Table 9) makes clear where encrypted time goes and how configuration trade-offs move it:
- adding one main prime in AlexNet saves 20.89 s of bootstrap time but adds 12.63 s of linear-transform time;
- square embedding in ResNet20 removes standalone rotations but adds 44.34 s of linear-transform time;
- changing only the bootstrap auxiliary chain from four to five 61-bit primes, with an identical schedule of 38 bootstrap calls, moves ResNet20 from 857.65 s to 604.79 s, with 80% of that gap in bootstrap time (Table 3(b), Figure 6(b)). To my knowledge, such controlled, operation-resolved evidence about Orion/Lattigo configuration effects is not commonly reported, and the paper is careful to call these descriptive differences rather than causal contributions.
Critical weaknesses
Weakness 1. In my view, the paper does not establish that the LLM agents add value over simpler search procedures, the compiler's own defaults, or expert configurations, which I consider the central question for an ICLR paper on LLM-guided optimisation.
The configuration space is a small, structured, mixed discrete space: a handful of global fields with fixed at 16, plus BSGS ratios and embedding choices for a few layers. For such a space, I would expect the following baselines under the same encrypted-evaluation budget:
- Orion's default or reference configuration for each network;
- random search and grid search over the supported fields;
- classical black-box optimisers such as Bayesian optimisation (e.g., SMAC or TPE), or LLM-enhanced Bayesian optimisation (LLAMBO, ICLR 2024);
- a published expert configuration, for example the ResNet-20 settings of Lee et al. (2022) or the Orion paper's own reported setup, measured on the same machine.
None of these is reported. The only external comparisons are two:
- Table 2, which uses LeNet alone (no bootstrapping, so the hardest part of the problem is absent), three attempts per arm and a single input;
- Table 13, a historical prototype with a different backend version and unmatched budgets, which the authors themselves say "establish[es] neither an advantage over expert configurations nor the search-efficiency benefit of feedback".
Even the internal evidence for the refinement loop is weaker than the abstract suggests:
- AlexNet: the best configuration is the first feasible one; refinement improved nothing (267.66→267.66 s), and the five-evaluation diagnostic search passed 0/5 gates (Table 5).
- ResNet20: the 604.79 s result "comes from the validated initial candidates", not from Phase C. The online refinement minimum was 682.15 s (Appendix E).
- LeNet: the 15.2% gain combines a global change (scale 40→26, from twelve to eight primes) with BSGS overrides. At least the first is a standard expert move (reduce scale and chain length when no bootstrapping is needed).
What, then, did the multi-agent refinement contribute that a simple sweep over scale, chain length and bootstrap would not have found?
The same-LLM comparison in Table 2 is, in my opinion, not statistically meaningful. FHE-Agent's mean inference time is within about 1% of the alternatives (15.6 vs. 15.7 s, 16.9 vs. 16.7 s, 16.6 vs. 16.7 s), it is not the fastest under Luna, and "9/9 versus 8/9 attempts passing" with three attempts per arm cannot support a claim of superiority. Note also that the Gemma FHE-Agent configuration achieves its time by accepting 14.2 bits of precision (MAE ) where the others retain 23.2 bits (Table 12). In addition, no ablation isolates the multi-agent decomposition itself: a single agent with the same tools, history and gates is the natural control for the claimed contribution (C1), and it is missing.
Weakness 2. I believe the measurement methodology is too weak to support the reported improvements, and the evaluation scope is narrow for the claim of "practical encrypted inference".
(a) Run-to-run variance is comparable to the claimed gains. All times are single diagnostic measurements on shared CPUs. Yet the authors' own repeat diagnostics show that identical configurations span 300.21–342.15 s for AlexNet (about 14%) and 687.59–736.48 s for ResNet20 (about 7%) across runs (Appendix E.1). The headline refinement gains are 12.0–15.2%, and the bootstrap- effects are 14–18%. With one measurement per configuration, I cannot tell which of these differences are real. I would expect at least five repeated, pinned, isolated runs per reported configuration, with confidence intervals.
(b) Numerical feasibility is validated on a single input. The MAE gate is evaluated on one fixed input tensor, and the authors acknowledge that "verified feasibility applies to the fixed evaluated inputs, not a held-out input distribution". The LeNet best has only a margin to the threshold. For a deployed private-inference service, the relevant guarantee is agreement of encrypted and plaintext predictions (top-1 agreement or accuracy) over a test set, e.g., the 10,000 MNIST or CIFAR-10 test images, together with worst-case error, because CKKS error depends on input-dependent activation ranges. A configuration that passes on one image may fail on others, which I see as a serious issue for the polynomial ReLU approximation used in ResNet20.
(c) Uncontrolled code revisions and selective reporting. The reported results come from different source revisions ("Searches differ in code revision and conditions and are not controlled comparisons"), and the activation metadata and candidate-evidence projection described in Section 3 "were refined after these searches, so the recorded trajectories do not evaluate those refinements end to end". The table of best results takes the best over several searches, one AlexNet search found no feasible result, and the extended ResNet20 row includes three extra validations of initial candidates. I find it difficult to know what system Table 1 actually evaluates.
(d) Search cost is not weighed against the benefit. Appendix I reports compilation alone at 1,382.8 s, 6,835.2 s and 3,552.7 s, with whole-search peak memory up to 325.4 GB, plus diagnostic forward passes and unaccounted LLM/API time. Saving roughly 83 s per ResNet20 inference therefore needs dozens of inferences to break even on compilation alone. That is reasonable for a deployed service, but it should be stated against a baseline search with the same budget.
(e) Scope. The workloads are LeNet (MNIST), an AlexNet-style CNN and ResNet20 (CIFAR-10), with fixed, CPU only, and one backend. The ML community's pressing need for CKKS configuration is privacy-preserving Transformer inference (THE-X, Powerformer, BumbleBee), where softmax, LayerNorm and GELU approximations, long modulus chains and frequent bootstrapping make the space much harder. I would be considerably more convinced if the method were shown on at least one Transformer encoder (e.g., BERT-base or a ViT) or on a GPU backend such as Cheddar.
Finally, as an ICLR submission, I think the methodological contribution is thin. The agent architecture is a fairly standard propose/select/assess loop in the style of AutoGen and Reflexion, the closest prior work (FHECrafter, FHE-Coder, AlphaEvolve for FHE kernels) already combines LLM agents with FHE execution feedback, and there is no learning component, no new optimisation algorithm and no analysis of why the LLM's priors help. The text is also heavily hedged throughout, so the reader must work hard to extract what is actually claimed.
Questions for authors
-
Non-LLM baselines. Could you report, on the same hardware, under the same gates and with the same number of encrypted evaluations, (i) Orion's default/reference configuration, (ii) random search, (iii) a grid sweep over scale, and bootstrap , and (iv) a Bayesian-optimisation baseline (e.g., SMAC or TPE) over the identical configuration space? Without these I cannot tell whether the LLM agents improve on what a simple optimiser would find.
-
Expert configurations. How do your ResNet20 and AlexNet results compare with the configurations reported in the Orion paper (Ebel et al., 2025) and in Lee et al. (2022), re-measured on your dual EPYC 9454 machine? Is 604.79 s faster, comparable or slower than the expert setting?
-
Attribution of the ResNet20 result. Since the 604.79 s configuration comes from the validated initial candidates rather than from Phase C refinement (online minimum 682.15 s), what is the measured contribution of the feedback loop for ResNet20? Similarly, for AlexNet the first feasible configuration was the best. Could you quantify, across all searches, how often Phase C produced the final best?
-
Repeated measurements. Given that identical configurations vary by about 14% (AlexNet) and 7% (ResNet20) across runs, could you re-time every reported best and first-feasible configuration at least five times with pinned cores on an otherwise idle machine, and report means with confidence intervals? Do the 12–15% refinement gains and the 14–18% bootstrap- effects survive?
-
Test-set numerical validation. Could you report encrypted-versus-plaintext top-1 agreement and the distribution of MAE (median and worst case) for each selected configuration over the full MNIST/CIFAR-10 test set, or at least a few hundred held-out inputs? How often does a configuration that passes the single-input gate fail on other inputs, especially given the polynomial ReLU's dependence on the activation-range margin?
-
Multi-agent ablation. What happens with a single LLM agent that receives the same tool suite, history and deterministic gates, but without the role decomposition (InitAgent, RegimeAgent, GlobalTradeoffAgent, LayerwiseAgent, PatchGateAgent)? This seems to me the minimal control for contribution (C1). Please also ablate each tool (e.g., remove the CostModel, whose forecasts were 42–59% low and misordered candidates) to show which evidence actually changes decisions.
-
Security parameters. For the AlexNet configuration with (a dense ternary secret) and the bootstrap encapsulation secret of weight 32, which estimator does the backend checker use (e.g., the lattice-estimator against primal and dual attacks), and with which error distribution? Was 128-bit security verified for every context, including the bootstrapping key-switching moduli ?
-
Search cost versus gain. Could you report end-to-end search cost (compilation, encrypted evaluation, analysis and LLM/API time, in wall-clock and dollar terms) next to the per-inference saving, and the number of inferences needed to amortise the search? How does this compare with the same budget spent on a classical optimiser?
-
Scaling to Transformers. Do you expect FHE-Agent to extend to privacy-preserving Transformer inference (e.g., BERT-base or ViT, as in THE-X, Powerformer and BumbleBee), where softmax, LayerNorm and GELU require high-degree polynomial approximations and deep modulus chains? Which parts of the tool suite and action space would need to change, and does the LLM's prior knowledge still help once the configuration space grows by orders of magnitude?
-
Reproducibility. The same-LLM comparison uses GPT-6 Astra and GPT-6 Luna, and the earlier searches used several source revisions. Will you release the exact prompts, LLM transcripts, source revisions and per-trial artefacts so that the results can be reproduced, including with an open-weight model such as Gemma under fixed decoding?