- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FedEDS studies federated LoRA fine-tuning with task heterogeneity and within-task label skew. It uses cross-client centered query-A factors and affinity propagation to discover client groups, constructs group-specific experts by aggregating effective LoRA updates followed by truncated SVD, and applies an input-subspace overlap regularizer to encourage expert specialization. Experiments on four GLUE classification tasks report improved average accuracy over federated LoRA baselines, accompanied by analyses of client clustering and expert subspace overlap.
Critical strengths
-
Relevant problem setting and coherent design. The paper considers task differences alongside statistical heterogeneity within each task, where indiscriminate parameter sharing can be detrimental. Its discovery, initialization, and specialization stages form a clear procedure for organizing collaboration without requiring known task labels during grouping.
-
Informative empirical diagnostics. The centered query-A representation achieves strong clustering results under the evaluated label-skew settings, and the overlap analysis shows that even task-grouped experts can share highly similar input subspaces. The main GLUE comparison also reports an average accuracy of 88.13%, compared with 87.12% for the strongest listed baseline. These observations provide a useful empirical motivation for studying client grouping and expert differentiation together.
Critical weaknesses
1. The independent contribution of centered query-A to client discovery and final accuracy remains unclear.
FedEDS extends earlier work on client grouping and federated LoRA experts, including FedLEASE and FedTreeLoRA. The use of a centered query-A representation is a potentially useful distinction, but the evidence currently focuses on clustering quality rather than its effect on the complete training procedure. Section 5.2.1 reports strong clustering results after the warmup phase, while the diagnostics use a single random seed and the end-to-end ablations in Table 3 keep centered query-A clustering unchanged.
A controlled end-to-end comparison with raw query-A, the displacement from the shared initialization (), and oracle task grouping would help establish the benefit of cross-client centering. In particular, provides a relevant alternative reference for removing a shared component. Holding expert initialization and subsequent training fixed would make it possible to determine whether better grouping accounts for a meaningful portion of the final performance gain. Such evidence is important for distinguishing the proposed client representation from existing grouping-based approaches.
2. Reduced geometric overlap has not yet been linked convincingly to functional expert specialization.
Table 2 shows that A-only regularization reduces input-subspace overlap from 0.8643 to 0.0061 in the reported experiment. This directly supports the geometric effect of the penalty, but smaller overlap alone does not establish that experts develop complementary task-specific capabilities. Task-conditioned routing statistics, expert utilization, or the effect of removing individual experts would help assess whether the separated subspaces correspond to meaningful differences in model behavior.
The choice of A-only regularization also merits a more explicit trade-off analysis. Table 2 already compares it with joint A+B regularization; the latter reports 89.13% accuracy versus 88.69% for A-only regularization in a single-seed IID setting. This result does not establish a consistent advantage across seeds, but it makes the rationale for the input-only design worth explaining. Measured runtime or memory overhead for the two alternatives would help determine whether the simpler regularizer offers a practical benefit.
Additional experimental and presentation observations (minor). The primary evaluation is restricted to RoBERTa-Large, four fixed classification tasks, 16 clients, and full client participation. Unequal task-group sizes or partial participation would be useful tests of expert discovery, particularly because the clustering and main ablations rely on one seed even though the principal results use three. Reporting paired per-seed differences, along with empirical runtime and expert-bank storage relative to the closest multi-expert baselines, would improve the assessment of robustness and cost. Figure 5 has small labels and legends, and the closely placed tables and explanatory text on page 9 would benefit from more space.
Questions for authors
-
Value of client centering. How does final model performance change when centered query-A clustering is replaced with raw query-A, , or oracle task grouping, keeping expert initialization and later training fixed?
-
Functional specialization. Does the substantial reduction in input-subspace overlap correspond to task-dependent expert utilization or distinct contributions to task accuracy? If available, routing statistics or expert-removal experiments would help answer this question.
-
Choice of regularizer. What accounts for the choice of A-only over joint A+B regularization, given the accuracy difference reported in Table 2? Are there measurable computational savings, and how stable is the comparison across seeds?
Dear Reviewer,
We sincerely thank you again for your recognition of our work. As we noted in our previous response, we have addressed your suggestions in our point-by-point response. We hope that these responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript. If you have any additional comments, we will do our best to address them.
Best regards,
The Authors