- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the federated fine-tuning of code language models under programming-language heterogeneity. Using Qwen2.5-Coder-1.5B, QLoRA, and Flower, the authors compare FedAvg, FedProx, and FedAdam in a five-client setting with full participation. They evaluate three data-partitioning conditions: IID, Dirichlet, and partitioning by programming language. Across three random seeds, the experiments report greedy pass@1 on HumanEval+, MBPP+, and multilingual code-generation tasks. Among the tested configurations, FedAvg achieves the best average performance, FedProx performs similarly, and FedAdam performs substantially worse. The authors further argue that federated training incurs little performance loss under mild heterogeneity, whereas partitioning by language produces a gap of approximately nine percentage points between centralized and federated training. The paper’s main contribution is an empirical comparison and a reproducible workflow, rather than a new algorithm.
Critical strengths
The research question has practical significance, and the comparative setup is easy to interpret. Differences in programming languages across organizations are a reasonable source of heterogeneity to study in federated fine-tuning of code models. By holding the base model, adapter configuration, and client participation scheme constant while comparing three established algorithms, the paper provides useful preliminary evidence for settings with limited computational resources.
Critical weaknesses
1、The paper is explicitly positioned as a controlled empirical study rather than a novel algorithmic contribution. Its main contribution relies on a programming-language-specific experimental setup and systematic comparison; However, it has not yet sufficiently demonstrated that this setup yields scientific findings that are substantially novel relative to prior work.
2、The claim that language heterogeneity incurs an approximately nine-percentage-point federated performance penalty lacks an appropriate control. When the training data source and format change, the centralized baseline cannot simply be reused on the grounds that it is “independent of the partition.” The observed gap may also reflect differences in data source, instruction format, and data distribution, so it cannot be attributed specifically to federated training or language heterogeneity.
Questions for authors
1、Is there a separate centralized model trained on the union of the client data in the language-partitioned setting? Please specify the training samples, total training tokens, optimization steps, and learning-rate schedule for each condition, and report the centralized–federated gap using the same data source. This directly affects the interpretation of RQ1.
2、Please provide a sweep over a reasonable range of server learning rates, the full server optimizer configuration, and training-loss or validation-performance curves. Does the ranking of the algorithms remain the same when each method receives the same validation and hyperparameter-tuning budget?
3、The main text describes this condition as quantity skew, but Figure 2 shows changes in language proportions; Section 3.4 also states that every client uses 200 samples per round. Please clarify which variable the Dirichlet distribution governs, whether samples are drawn again each round, and whether aggregation weights use full partition sizes or the number of samples actually used for training. These details determine whether the claim that “FedAdam collapses under quantity skew” is supported.
4、Could you provide per-language results for the base model and a centralized model trained on matched data? Please also explain whether Python’s HumanEval+ and the test suites for other languages have comparable test rigor. Nonzero scores across languages alone do not establish how much of the model’s original capability fine-tuning preserves or support the authors’ interpretation of the capability gradient.