acceptodds
Under review as a conference paper at ICLR 2027

Federated Fine-Tuning of Code Language Models: A Comparison of FedAvg, FedProx, and FedAdam UnderLanguage-Heterogeneous Clients

Abstract

Large language models specialized for source code excel at program synthesis, but further domain gains require fine-tuning on codebases that are often proprietary and cannot be pooled centrally. Federated Learning (FL) offers a path to collaboratively fine-tune a shared model without exchanging raw data, yet the comparative behavior of FL aggregation algorithms on code generation—particularly under realistic, language-based client heterogeneity—has not been measured systematically. We present a controlled empirical study that fine-tunes Qwen2.5-Coder-1.5B with Quantized Low-Rank Adaptation (QLoRA) inside the Flower framework, comparing three aggregation algorithms (FedAvg, FedProx, FedAdam) across three data-partitioning strategies (random IID, Dirichlet α=0.5, and a realistic non-IID split where each client specializes in one programming language). The resulting global models are benchmarked against a centralized upper bound on HumanEval+, MBPP+, and MultiPL-E using greedy pass@1, with three seeds per condition and paired significance testing. Two findings emerge. First, on algorithm choice, FedAvg consistently matches or exceeds FedProx and substantially outperforms FedAdam in every partition (HumanEval+ pass@1 of 40.0, 36.4, 29.1 for FedAvg across IID, Dirichlet, and by-language), whereas FedAdam is unstable and collapses to 11.6±5.8 under quantity skew; contrary to the expectation that server-adaptive optimization best handles heterogeneity, the simplest aggregator is the most accurate and robust. Second, on the cost of federation, the best federated model is statistically indistinguishable from the centralized reference (38.2) under IID and Dirichlet but lags by ≈9 points under the by-language split. A per-language MultiPL-E analysis reveals a stable competence gradient from Python and JavaScript down to Go, without collapse toward a dominant language. We release adapters and configurations for reproducibility.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Code Generation, Large Language Models, QLoRA, Parameter-Efficient Fine-Tuning, Non-IID Data, HumanEval, Qwen2.5-Coder.

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Gt6EOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the federated fine-tuning of code language models under programming-language heterogeneity. Using Qwen2.5-Coder-1.5B, QLoRA, and Flower, the authors compare FedAvg, FedProx, and FedAdam in a five-client setting with full participation. They evaluate three data-partitioning conditions: IID, Dirichlet, and partitioning by programming language. Across three random seeds, the experiments report greedy pass@1 on HumanEval+, MBPP+, and multilingual code-generation tasks. Among the tested configurations, FedAvg achieves the best average performance, FedProx performs similarly, and FedAdam performs substantially worse. The authors further argue that federated training incurs little performance loss under mild heterogeneity, whereas partitioning by language produces a gap of approximately nine percentage points between centralized and federated training. The paper’s main contribution is an empirical comparison and a reproducible workflow, rather than a new algorithm.

Critical strengths

The research question has practical significance, and the comparative setup is easy to interpret. Differences in programming languages across organizations are a reasonable source of heterogeneity to study in federated fine-tuning of code models. By holding the base model, adapter configuration, and client participation scheme constant while comparing three established algorithms, the paper provides useful preliminary evidence for settings with limited computational resources.

Critical weaknesses

1、The paper is explicitly positioned as a controlled empirical study rather than a novel algorithmic contribution. Its main contribution relies on a programming-language-specific experimental setup and systematic comparison; However, it has not yet sufficiently demonstrated that this setup yields scientific findings that are substantially novel relative to prior work.

2、The claim that language heterogeneity incurs an approximately nine-percentage-point federated performance penalty lacks an appropriate control. When the training data source and format change, the centralized baseline cannot simply be reused on the grounds that it is “independent of the partition.” The observed gap may also reflect differences in data source, instruction format, and data distribution, so it cannot be attributed specifically to federated training or language heterogeneity.

Questions for authors

1、Is there a separate centralized model trained on the union of the client data in the language-partitioned setting? Please specify the training samples, total training tokens, optimization steps, and learning-rate schedule for each condition, and report the centralized–federated gap using the same data source. This directly affects the interpretation of RQ1.

2、Please provide a sweep over a reasonable range of server learning rates, the full server optimizer configuration, and training-loss or validation-performance curves. Does the ranking of the algorithms remain the same when each method receives the same validation and hyperparameter-tuning budget?

3、The main text describes this condition as quantity skew, but Figure 2 shows changes in language proportions; Section 3.4 also states that every client uses 200 samples per round. Please clarify which variable the Dirichlet distribution governs, whether samples are drawn again each round, and whether aggregation weights use full partition sizes or the number of samples actually used for training. These details determine whether the claim that “FedAdam collapses under quantity skew” is supported.

4、Could you provide per-language results for the base model and a centralized model trained on matched data? Please also explain whether Python’s HumanEval+ and the test suites for other languages have comparable test rigor. Nonzero scores across languages alone do not establish how much of the model’s original capability fine-tuning preserves or support the authors’ interpretation of the capability gradient.

Reviewer_3X8dOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduce an empirical comparison of three different approaches ( FedAvg, FedProx and FedAdam ) for federated fine-tuning of Qwen2.5 coder using QLoRA.

Critical strengths

S1: The paper introduces an empirical evaluation of three different federated learning algorithms for fine-tuning LLMs.

S2: The authors study both IID and Non-IID data.

S3: The authors also report the computational cost of each approach and per-round communication requirements.

Critical weaknesses

W1: What is the novelty of this paper? I don't see any contribution for the scope of ICLR. There is no new algorithm contribution or theoretical contribution, and the paper is only about an empirical comparison of three traditional federated learning approaches.

W2: There are also multiple issues in the experimental setup. For example, the main conclusion of the experiment section is that FedAvg consistently outperforms FedProx and FedAdam under a non-IID scenario. That makes sense because the authors fix the learning rate for all experiments at 10−210^-2. FedAdam is basically very sensitive to learning rate and momentum, and the authors didn't do a hyperparameter search to find the best set of parameters for each approach. I would also suggesnt to include local training only so one can see the effect of federated collaboration compared to fine-tune each model locally on its own dataset.

W3: It's not clear to me how the aggregation steps are done for each approach. The paper states that each client sends their LoRA adapters and the server aggregates these adapters using FedAvg , FedProx and FedAdam but it's not clear if the aggregation is done on the LoRA factor matrices or on the reconstructed low rank weight updates. Explaining this is very important as those approaches are not developed for fine-tuning LLMs and need to be adapted.

Side Note: All text or labels for the figures in the paper are a bit small and there is a lot of space, so I suggest increasing the font of the text in all figures to be more readable.

Questions for authors

Check Weaknesses

Reviewer_1XbuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper compares FedAvg, FedProx, and FedAdam for federated fine-tuning of Qwen2.5-Coder-1.5B using QLoRA and the Flower framework. The experiments involve five clients under IID, Dirichlet, and programming-language-based data assignments, with evaluation on HumanEval+, MBPP+, and MultiPL-E. The authors report that FedAvg achieves the highest average performance under their chosen configurations and that the by-language setting exhibits a larger performance gap from the centralized reference. The paper presents its implementation and empirical comparison as its main contributions.

Critical strengths

I did not identify major scientific strengths sufficient to support acceptance. The principal merit is the effort to implement and evaluate established algorithms under a common framework, including repeated runs and standard benchmarks, but this effort does not by itself establish a substantive research contribution.

Critical weaknesses
  1. The submission does not establish a sufficiently substantive contribution beyond existing work: The main contribution consists of integrating existing tools and comparing three established algorithms on one base model with five clients. OpenFedLLM (Ye et al., KDD 2024) already directly compares seven federated learning algorithms on code-generation benchmarks, including FedAvg, FedProx, and FedAdam. Although the present submission examines an additional combination of model and partitioning settings, it does not convincingly establish what substantial new understanding this comparison provides. Its scientific value therefore depends heavily on the validity of its two headline findings, both of which are undermined by the experimental issues below.
  2. The algorithm comparison does not support the claimed conclusions about robustness: Section 3.4 fixes the FedAdam server learning rate at (0.01) and the FedProx coefficient at (0.01), explicitly dismissing algorithm-specific tuning. However, a comparison with comparable tuning budgets is needed to distinguish algorithmic limitations from an unsuitable hyperparameter choice. No learning-rate sensitivity analysis or training curves are provided to explain FedAdam's poor performance or establish the claimed instability. Therefore, the experimental analysis does not adequately justify the broader conclusions in Section 5 that FedAvg is the most robust choice and that server adaptivity produces instability.
Questions for authors

Please address the concerns detailed in the Critical Weaknesses section, particularly regarding the substantive advance over prior work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.