- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents a method to accelerate federated FHE-based LLM training, especially communciation and server-side optimization algorithms. The contributions are two-fold. First, the authors present two methods to work together: a compression-based method which involves two RTT to first reach a consensus (via secure importance weighting) on the top-k elements, and then aggregate the top-k elements; and a radix based method to pack a pair of elements into one slot to accelerate aggregation. Second, the authors propose a method to implement NormBounding, a simple method in plaintext but requires comparison and branching---hard to achieve by ciphertexts.
The authors experiment with two models: GPT-2, and LLaMA-7B to showcase the effectiveness of the proposed FUSE. The efficiency improvement is about 4x and the perplexity loss lies within in 1% (evaluated on GPT-2 only).
Critical strengths
-
The paper is generally well-written in its technical part. I am not an expert in FHE but the paper presents its methodology in a way that does not require deep understanding into the implementation of FHE, which makes it easy to understand.
-
The experiments (especially the ones in Table 1) are extensive. It covers two models with different sizes (it may determine whether a model is compute-bound or communication-bound), two network models, and various parameters --- providing a good and all-around comparison of FUSE's effectiveness.
Critical weaknesses
-
Questionable relationship with LLMs. Throughout the paper, I find little relationship between the paper's technique/methodology and LLMs. There may be two --- but both are weak. First, LLMs have a large number of parameters, making communication especially slow. However, if we only care about the effectiveness of FUSE, all models will do to showcase the effectivess as comparison is done on the same model. Second, the optimization involves LoRA, but that is also weak -- the methodology does not exploit the low-rank property of LoRA, and also works for full-rank optimization. Therefore, I would say that the relationship between this paper and LLMs is doubtful and not sufficient. This is a positioning issue that should be fixed.
-
The claim seems stronger than what is actually achieved in terms of optimization algorithms. The authors claim in the introduction that the aggregation scheme is versatile. However, the authors only implement FedProx and NormBound in this paper. In practice, there are lots of federated optimization algorithms that go beyond FedProx and norm-bound. For example, FedAdam, FedYogi, etc., algorithms that require server-side states and first & second order moments, are at least non-trivial to support in this framework. Given this in mind, it is important that the authors should clarify the scope of supported optimization algorithms in the framework of FUSE, to support the claim of "versatile".
-
The design choice of Normbound, especially the hidden from clients, is not well justified either. Given the semi-honest setting, there is a simple go-around --- just to disclose the to clients, and everything can be done at client-side. As clients are semi-honest, they are supposed to follow the protocol, and from my understanding the whole process still works. Therefore, the authors should at least justify, why the simple go-around of disclosing to clients would not work, otherwise the designs become doubtful and unnecessary.
-
Evaluations, especially in terms of utility, are not done extensively. The authors only do utility experiments on GPT-2, which is a 2019-year model and by current standards not sufficiently strong model. Therefore, at least utility experiments on a more modern billion-scale LLM should be done to justify the utility claim. Moreover, only relying on perplexity for utility may not be sufficient, especially for LLMs. At least some benchmark scores can be reported as they align better with the Q&A nature of LLMs.
-
Some presentation details can be improved. Notably, I list two specific points:
- In algorithm 1, the authors use and , which I suppose are the largest and smallest absolute values of the top-k updates. However, this is very confusing as this looks very like the in top-k.
- Figure 3 is also very confusing. At a double-log scale, the advantage of NB over FedProx and FedAvg becomes indistinguishable in terms of perplexity. Instead, NB shows significantly larger communication. The figure thus becomes very confusing as to whether the authors want to show the advantages of NB, or the disadvantages.
Questions for authors
Please address the weaknesses 1-5 above.
2 comments
Not fully convinced by the rebuttal.
I am not fully convinced by the rebuttal.
- My questions about the relationship with LLMs are not fully answered.
- If the technique works on LLMs, it should also work on other models as well, and achieve a similar level of compression ratio --- after all, all parameters, LLMs or CNNs, are equal in the scope of this paper. There is no such thing that makes the technique applicable only to LLMs, but not on CNNs.
- The similar argument holds for LoRA. In the scope of this paper, LoRA parameters are not treated differently, so if it works on LoRA, it should also work on full-parameter training, and achieve a similar level of compression ratio --- again, all parameters, LoRA or not LoRA, are equal in the scope of this paper. There is no such thing that makes the technique applicable only to LoRA parameters, but not on others.
- The authors should have limited the claim to "all federated optimizers without server-side non-linear states" then, instead of "Versatile Advanced Aggregation" in the title.
I have no further comments in terms of the rebuttal for Q3 and Q4.
I will accordingly raise my rating to 2.
Thank you for the constructive feedback. We provide the following clarification to address the raised concerns:
Q1 – Relationship between the proposed method and LLMs
FUSE is model-agnostic as an FHE aggregation primitive, the central claim is that FUSE makes non-linear FHE-based aggregation affordable for LLM-scale federated PEFT. First, model scale alters the nature of the problem rather than merely its size. Unlike prior CNN-scale literature with marginal overhead, LLM fine-tuning creates multi-gigabyte ciphertexts (reaching 11.87 GB under Norm Bounding) that make encrypted non-linear aggregation at LLM scale difficult. FUSE is proposed to make this computation feasible, reducing it to 1.80 GB. Second, parameter-efficient fine-tuning (PEFT) is standard for federated LLMs (e.g., OpenFedLLM (Ye et al., 2024), SHE-LoRA (Liu et al., 2026)), but LoRA alone may be inefficient once updates are encrypted due to ciphertext expansion. Thus, FUSE and LoRA are complementary by design. Finally, the need for affordable FHE protection is a rising need for language model, DAGER (Petrov et al., 2024) shows that gradient inversion attacks can recover training text from language model updates. FUSE is one essential layer that makes this cryptographic protection affordable at LLM scale.
Q2 – Scope of the claim of versatile The intended meaning was related to the current state of the encrypted FL literature. Existing FL-FHE compression methods for LLMs are designed around linear FedAvg or relies on non-colluding server architecture. Thus, this study proposed encrypted Norm Bounding, realizing an non-linear aggregator in FL within a single-server FHE pipeline at LLM scale. We will clarify the scope explicitly.
Server-side adaptive optimizers such as FedAdam and FedYogi handle non-linear operations on the final global model after aggregation, not on any encrypted client update. In contrast, Norm Bounding represents a significantly more demanding case, requiring expensive conditional selections on each client's encrypted update before aggregation. This rigorous evaluation makes Norm Bounding the ideal benchmark for research.
Q3 – Why threshold is kept on server?
The current solution is a baseline that isolates the cost of this sign-gating primitive. It becomes necessary for non-linear aggregation that depends on cross-client information and cannot be pushed to clients without revealing cross-client norms. Modern privacy-preserving federated learning increasingly relies on adaptive, data-dependent clipping thresholds, as static thresholds may become mis-calibrated as update norms shrink during training (Allouah et al., 2025). Recent clipping methods therefore set the threshold τ adaptively as a function of all clients' update norms (Andrew et al., 2021; Allouah et al., 2025). Since these thresholds depend on cross-client statistics that cannot be disclosed, clipping must be evaluated server-side under encryption. The current solution serves as a baseline that realizes this core sign-gating primitive, while providing a pathway to adaptive clipping as a future extension.
Q4 – Extent of the utility evaluation
We acknowledge that PPL alone is not sufficient for LLM utility. It is sensitive to the noise and precision loss introduced by FHE-compatible compression, making it a suitable stress test for convergence stability as baseline. Thus, we provide preliminary Llama-7B accuracy results on GSM8K. As shown below, all variants improve from Round 1 to Round 3. A full downstream evaluation across diverse tasks will be included in the final version.
Q5 – Presentation issue: Algorithm 1 and Figure 3 The symbols k denote the max and min magnitudes within a client's local Top-k set. While Figure 3 condenses our full convergence study (Appendix J) to highlight two key findings: (1) FUSE-Fusion convergence stably with lower communication. (2) Norm Bounding achieves the best final utility despite higher communication overhead. We will revise them accordingly for clarification.
References