acceptodds
Under review as a conference paper at ICLR 2027

FedConsult: Turning Client Heterogeneity into On-Demand Expertise for Personalized Federated Learning

Abstract

Federated learning enables decentralized clients to collaboratively train models without sharing private data. Existing methods combine collaborative knowledge sharing with local adaptation to improve predictions under statistical heterogeneity, yet personalization gains remain client-bound, leaving substantial performance gaps across clients on specific classes. To address these limitations, we propose FedConsult, a federated expertise-transfer framework that distills locally acquired strengths into reusable visual specialty adapters. These adapters let each client complement its predictions with others' expertise on a sample-by-sample basis, improving weak-class recognition while preserving local personalized strengths. Specifically, FedConsult identifies each client's regions of specialty competence from discrepancies between its global and personalized prediction paths, and distills visual knowledge with similar prediction-improvement patterns into specialty adapters in a shared feature space. Recipient clients subsequently use response-aware sparse routing to selectively invoke complementary expertise within their personalized pathways. Experiments on both natural and medical image classification tasks show that FedConsult outperforms the strongest baselines by up to 3.96 and 9.13 percentage points in overall and client-specific weak-class accuracy, respectively. Beyond viewing heterogeneity solely as an optimization challenge, FedConsult turns the resulting capability diversity into transferable expertise, closing local capability gaps and supporting more reliable personalized federated systems.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federal learning, Cross-Client Knowledge Transfer, Mixture of Experts, Medical Image Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KtxYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

I understand FedConsult as a framework for sharing the strengths of personalized federated models through compact visual adapters. it identifies samples where personalization improves over a global predictor, groups their prediction-correction directions, and distills each group into an adapter. Each recipient then learns a response-aware router that combines adapter responses with its original prediction. the experiments report improvements in overall accuracy and accuracy on clients’ weakest classes.

Critical strengths

I find the shared feature interface a sensible way to transfer personalized knowledge. each recipient interprets an adapter’s correction through its own prediction pathway, addressing the compatibility problem of directly transferring another client’s classifier. each adapter contains only 512 parameters, which also keeps the additional communication small.

I also appreciate the experiments examining where the gains come from. The component ablations and comparisons with local-only, random, and averaged adapters provide useful evidence that learned external expertise contributes beyond additional local adaptation. The prediction-transition analysis further supports the intended benefit for weaker client–class pairs.

Critical weaknesses

Actually I have a concrete concern about the specialty decomposition on the binary medical tasks. For 2 classes, centering and normalizing the logit correction in Equation (3) leaves only two possible directions. however he paper uses four directional clusters per client on Retina and BreaKHis without explaining how duplicate centers or empty clusters are handled. This does not rule out useful adapters, but it leaves their construction unclear and weakens the interpretation that these results demonstrate four distinct correction specialties. I am also concerned about the independence of the medical evaluation. Appendix A.3 describes image-level partitioning followed by local train–test splitting, while BreaKHis contains multiple images from each of only 82 patients. the stated protocol does not establish patient-disjoint evaluation, so correlated images from the same patient could appear in training and testing. i cannot determine from the paper whether this occurred, but without clarification or results on grouped splits, the evidence for generalization to unseen patients is insufficient for a positive recommendation.

Questions for authors

How is four-way directional clustering implemented on the binary tasks? I would like to see the resulting cluster occupancies and comparisons with two adapters and random within-class partitions on Retina and BreaKHis to understand what the decomposition contributes.

Are all medical training and test sets disjoint at the patient or lesion level, including across clients? If this was not enforced, could you repeat the main comparisons with grouped splits and report variability across multiple runs? These results would materially affect my assessment.

4 comments
AuthorsOpenReview ↗

Concern 1: Directional Clustering on Binary Tasks

How is four-way directional clustering implemented on the binary tasks? I would like to see the resulting cluster occupancies and comparisons with two adapters and random within-class partitions on Retina and BreaKHis to understand what the decomposition contributes.

Response:

We thank the reviewer for pointing this out. We set M=4M = 4 as an upper bound and handle insufficient samples or distinguishable directions by limiting the actual cluster count and training adapters only for nonempty groups. Centering and normalizing binary logit corrections yields at most two nonzero directions, ±(1,−1)/2\pm (1, -1)/\sqrt{2}, so four directional clusters are not enforced. We have added the effective cluster counts and comparisons with M=2, M=1, and random within-class partitions to explain the actual construction and evaluate the contribution of decomposition.

Specifically, after obtaining a nonempty specialty sample set using Eq. (2), we first limit the actual cluster count according to the available samples and distinguishable directions:

Ki=max⁡1,min⁡(M,Ni,Ui),K_i = \max\\{1, \min(M, N_i, U_i)\\},

where NiN_i is the number of selected samples and UiU_i is the number of distinct centered and normalized direction vectors after rounding to six decimal places. If Ki=1K_i = 1, all samples are assigned directly to one group. Otherwise, we use k-means++ initialization and apply the utility-weighted K-means in Eq. (4).

After clustering, we remove unoccupied cluster labels and train adapters only for nonempty groups, without duplicating centers or adding groups to reach four. Thus, when samples or distinguishable directions are insufficient, fewer adapters are constructed rather than forcing the training of experts without sufficient sample support.

On Retina and BreaKHis with the revised grouped split, setting the upper bound to four or two produces the same groups. The effective cluster counts are reported below:

DatasetEffective clusters per clientTotal adapters
Retina2, 2, 26
BreaKHis2, 2, 2, 2, 210

The final expert banks contain neither empty groups nor duplicate centers among occupied clusters. These binary-task results correspond to the two directional clusters actually formed, rather than four distinct correction directions.

Following your suggestion, we have added comparisons on Retina and BreaKHis between directional grouping with M=2, a single adapter distilled from all selected samples (M=1), and two adapters distilled from random within-class partitions. For the random control, we randomly divide each class's selected samples into two subsets, then pool subsets with the same index across classes. All other training settings are held constant.

ConfigurationRetina Acc / WeakBreaKHis Acc / Weak
M=2, directional grouping and distillation95.13 / 93.2888.11 / 52.51
M=1, pooled samples and distillation94.95 / 92.7187.44 / 47.60
Random within-class partitions, two distilled adapters94.81 / 92.6587.36 / 48.91

Directional grouping outperforms pooled distillation with M=1 on both datasets, supporting the value of distilling different correction patterns separately. It also outperforms random within-class partitioning at the same adapter count, showing that organizing distillation samples by correction direction provides practical benefits. These results support the value of both decomposition and the grouping strategy, beyond simply increasing the number of adapters.

We have clarified the clustering procedure in the revised manuscript.

AuthorsOpenReview ↗

Concern 2: Medical Data Separation and Variability

Are all medical training and test sets disjoint at the patient or lesion level, including across clients? If this was not enforced, could you repeat the main comparisons with grouped splits and report variability across multiple runs? These results would materially affect my assessment.

Response:

We thank the reviewer for pointing this out. We have reviewed the data further, strengthened the separation of identified groups of related images across training/test sets and across clients in Skin14 and BreaKHis, and repeated the benchmark comparisons and three-seed experiments on the revised splits. The results support FedConsult's effectiveness under stricter separation, and the repeated runs also support its performance advantage.

Our original splits followed the official protocols of FedDRM, FedKWAZ, and dFLMoE. Following your suggestion, we reviewed the data further and adopted stricter preprocessing and grouped partitioning to ensure that each identified patient, lesion, or case group belongs to exactly one client and appears exclusively in either training or testing. We then repeated the experiments for FedConsult and the comparison methods under this protocol.

  • Skin14: Our review identified pre-generated variants, conflicting labels, and records that could not be linked to a patient identity. After removing these records, we retained 23,367 images and applied grouped separation using the available patient or lesion identifiers.
  • BreaKHis: We conservatively grouped images by their year/numeric case prefixes, keeping section and magnification variants potentially belonging to the same patient together. The resulting dataset contains 7,909 images.

Both revised datasets use five clients to maintain sufficient class-specific samples per client after grouped separation and avoid missing rare classes due to excessive fragmentation. We retain the Dirichlet heterogeneity setting of α=0.3\alpha = 0.3, allocate data at the group level, and maintain an approximately 70/30 train/test ratio, ensuring that every client has training and test samples from each class. Our additional review of Retina identified no new issues, so we retain FedDRM's official preprocessing and three-center split.

The benchmark results on the revised splits are reported below (%).

MethodSkin14 AccSkin14 WeakBreaKHis AccBreaKHis Weak
FedAvg62.819.1880.1243.62
FedProx61.2114.0781.3741.41
FedSAM65.0714.1081.0840.72
FedALA68.657.1485.1831.40
FedPAC68.220.0085.6035.35
FedAS68.426.2085.1033.70
DualFed67.387.9586.2638.60
ConFree68.294.6586.3529.53
cwFedAvg67.622.8883.8030.93
Fed-MoE65.950.0083.0543.12
PM-MoE67.237.9587.1844.89
FedFew69.050.0085.8131.86
FedDRM69.084.6282.6332.60
FedConsult (Ours)69.6212.8688.1152.51

FedConsult retains the highest accuracy (Acc) among the evaluated methods on both datasets under the revised splits, which enforce stricter data separation.

Following your suggestion, we also evaluated stability across three random seeds. We selected ConFree, FedFew, and FedDRM as three representative methods with strong performance on the medical datasets and repeated their experiments alongside FedConsult using the same grouped splits. Results are reported as mean ± standard deviation (%).

MethodSkin14 AccSkin14 WeakBreaKHis AccBreaKHis Weak
ConFree68.23 ± 0.283.63 ± 1.8186.54 ± 0.2829.76 ± 2.22
FedFew68.22 ± 0.721.31 ± 1.5386.02 ± 0.1936.28 ± 4.11
FedDRM68.39 ± 0.854.84 ± 1.3284.02 ± 2.1732.49 ± 6.86
FedConsult (Ours)69.55 ± 0.1813.11 ± 1.5489.23 ± 0.9754.40 ± 8.07

These three-seed results further support FedConsult's stability on the revised splits.

We have updated the data partitioning procedures and results in the revised manuscript.

Thank you very much for your comments. As the discussion period is drawing to a close, we would like to ask whether our responses have addressed your concerns. If you have any further questions, we would be happy to address them in the remaining time.

Thank you for the detailed rebuttal and additional experiments. The new results have addressed some of my main concerns. Therefore, I am raising my rating from 2 to 3.

Reviewer_a4rXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

FedConsult first trains global and personalized predictors, identifies source samples benefiting from personalization, clusters their logit-correction directions, and distills these groups into FiLM visual specialty adapters. Clients download a federated expert bank and train response-aware sparse routers with a no-op option. Experiments evaluate overall accuracy and a client-specific weak-class metric on CIFAR and three medical imaging datasets.

Critical strengths
  1. The paper addresses a practical problem in personalized federated learning by exploiting complementary capabilities across heterogeneous clients, rather than treating heterogeneity only as an optimization challenge.
  2. The proposed specialty decomposition and response-aware routing provide a coherent way to transfer fine-grained knowledge across clients while preserving local personalized models.
  3. The experimental evaluation is relatively comprehensive. The authors compare against recent methods and provide component ablations, expert-granularity studies, and cross-client transfer analyses to support the effectiveness of the proposed framework.
Critical weaknesses
  1. The claimed on-demand expertise mechanism is not sufficiently supported. In Section 3.4, the response-aware router requires the prediction responses of all candidate experts before computing the sparse routing weights. Therefore, the method performs sparse selection over already evaluated experts, rather than selecting experts before execution. This distinction should be clarified, and the actual inference overhead should be evaluated.
  2. The generality of the proposed expertise-transfer mechanism remains insufficiently evaluated. The method relies on a shared feature representation and a synchronized expert bank. However, its effectiveness under larger client populations, partial participation, and more heterogeneous client capabilities remains unclear. Additional experiments would help establish whether the proposed approach scales beyond the relatively small federations considered in the paper.
Questions for authors
  1. The proposed response-aware router evaluates all candidate expert responses before applying sparse selection. In what sense is the expertise utilization "on-demand"? Does the sparse routing mechanism reduce inference computation, or does it only select among already computed predictions?
  2. How does the inference latency and computational cost scale with the number of clients and specialty adapters? Can the authors provide a comparison with FedDRM and other expert-based personalized federated learning methods?
  3. What are the main advantages of the proposed fine-grained specialty decomposition over existing client-level expert selection approaches, such as FedDRM and dFLMoE, beyond the reported accuracy improvements?
5 comments
AuthorsOpenReview ↗

W2: Effectiveness with More Clients and Different Degrees of Heterogeneity

The generality of the proposed expertise-transfer mechanism remains insufficiently evaluated. The method relies on a shared feature representation and a synchronized expert bank. However, its effectiveness under larger client populations, partial participation, and more heterogeneous client capabilities remains unclear. Additional experiments would help establish whether the proposed approach scales beyond the relatively small federations considered in the paper.

Response:

We thank the reviewer for pointing this out. We have added experiments with more clients and different degrees of label heterogeneity. FedConsult achieves the best performance among the expert-based personalized federated learning methods compared in all the additional settings, showing that the benefits of expertise transfer extend to larger federations and different data distributions.

All experiments use CIFAR-100. In the additional experiments, FedConsult and PM-MoE start from the same final DualFed checkpoint. Both tables report Acc and Weak separately, in percentages.

First, following FedDRM's client-scaling evaluation, we increase the number of clients from 8 to 16 and 24 under Dir-0.3. We use the same types of image transformations for covariate shift and expand the combinations of gamma correction, hue adjustment, and saturation scaling to construct distinct input distributions for the additional clients.

MethodAcc (%)Weak (%)
8 clients16 clients24 clients8 clients16 clients24 clients
FedDRM56.6954.8656.1517.127.225.70
FedAS53.2155.0856.005.048.853.89
Fed-MoE45.3940.0239.1715.6711.298.96
PM-MoE55.2056.6056.4618.9513.917.62
FedConsult60.6560.6559.3128.7822.3412.98

We then keep the number of clients at eight and vary the Dirichlet concentration parameter to examine different degrees of label heterogeneity, using the same covariate-shift construction as in the original eight-client experiment.

MethodAcc (%)Weak (%)
Dir-0.1Dir-0.3Dir-0.5Dir-0.1Dir-0.3Dir-0.5
FedDRM59.4356.6950.3311.7717.127.27
FedAS59.3453.2150.4712.505.047.90
Fed-MoE43.1245.3940.0917.5815.6715.88
PM-MoE61.1955.2052.7021.4718.9518.41
FedConsult66.7160.6556.3533.2028.7824.70

FedConsult continues to achieve the best performance across the expanded client populations and different heterogeneity settings, supporting its effectiveness under these conditions.

We have added these statements to our manuscript.

AuthorsOpenReview ↗

Q1: Does On-Demand Expertise Utilization Reduce Expert Execution?

The proposed response-aware router evaluates all candidate expert responses before applying sparse selection. In what sense is the expertise utilization "on-demand"? Does the sparse routing mechanism reduce inference computation, or does it only select among already computed predictions?

Response:

We thank the reviewer for pointing this out. FedConsult computes all candidate responses before applying Sparsemax for sparse fusion, so the cost of evaluating every candidate is still incurred. Here, "on-demand" refers to selecting cross-client expertise responses to address the recipient's local capability gaps for the current input. Sparsemax determines which already computed responses contribute to the final prediction; it does not eliminate candidate forward passes. Our additional experiments demonstrate the effectiveness of this mechanism.

Specifically, expert matching based solely on input-side information may not reliably distinguish suitable from unsuitable candidates: an expert with a higher matching score is not necessarily more useful for correcting the current prediction. Our analysis shows that, even when the candidate pool contains experts that predict correctly, input gating can assign greater weight to experts that predict incorrectly. Moreover, an external adapter produces a prediction through the recipient's own personalized pathway, so its effect depends on its interaction with local capabilities. We therefore let the router observe candidate responses from the complete prediction pathway, together with their changes relative to the original prediction, to learn which external adjustments effectively complement local capabilities. With the same Stage 1 model, frozen expert bank, and training budget, the results on CIFAR-100 Dir-0.3 are as follows (%):

Gating strategyAccWeak
Input gating58.8123.60
Response-aware gating60.6528.78

Response-aware gating improves Acc and Weak by 1.84 and 5.18 percentage points, respectively. These results show that observing actual candidate responses within the recipient helps make more effective use of external expertise, providing the main rationale for the additional response computation.

We have added these statements to our manuscript.

AuthorsOpenReview ↗

Q2: How Does Inference Cost Scale with Clients and Adapters?

How does the inference latency and computational cost scale with the number of clients and specialty adapters? Can the authors provide a comparison with FedDRM and other expert-based personalized federated learning methods?

Response:

We thank the reviewer for pointing this out. We have added comparisons of total parameters and average inference time with 8, 16, and 24 clients against FedDRM, PM-MoE, and Fed-MoE. By representing expertise as lightweight adapters in a shared feature space and combining shared computation with batched candidate execution, FedConsult maintains competitive parameter counts and inference latency. Its inference time also remains approximately stable over the tested range of client and adapter counts.

All methods were measured on the same RTX 4090, with timings averaged over repeated measurements after warm-up. FedConsult's timing includes all candidate response computation, gating, and fusion.

MethodPara (M)Time (ms / image)
8 clients16 clients24 clients8 clients16 clients24 clients
FedDRM11.46811.47011.4721.7831.6261.720
PM-MoE13.88616.41318.9402.7953.0513.845
Fed-MoE89.72589.72589.7255.8485.7915.825
FedConsult11.65011.66611.6833.1983.3143.175

As the client population increases from 8 to 16 and 24, FedConsult's bank grows from 32 to 64 and 96 adapters. Its total parameter counts are 11.650, 11.666, and 11.683 M, respectively, close to FedDRM's and lower than those of PM-MoE and Fed-MoE. Mean latency is 3.198, 3.314, and 3.175 ms per image, remaining approximately stable over this range, close to or lower than PM-MoE's, and substantially lower than Fed-MoE's.

This efficiency is related to how we represent and evaluate specialties. Although FedConsult evaluates every candidate response, encoding expertise as lightweight visual adapters in a shared feature space allows all candidates to reuse the recipient's prediction network. Each adapter has only 512 parameters, so adding candidates does not require storing additional complete models. During inference, the shared feature-extraction prefix is computed once, and adapted features are batched and processed through the same downstream recipient pathway. At the tested candidate-bank sizes, this batched execution can exploit GPU parallelism, consistent with the stable millisecond-scale latency observed. Together with the prediction improvements in Q1, these results show that FedConsult can use external expertise effectively while maintaining a compact parameter count and stable deployment performance.

We have added these statements to our manuscript.

Reviewer_s42BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedConsult, a cross-client expertise transfer framework for personalized federated learning, aiming to address local predictive weaknesses by leveraging complementary strengths across clients. The method identifies and distills client-specific expertise into a shared visual expert bank and employs response-aware sparse routing to dynamically combine expert predictions, thereby improving overall accuracy and weak-class recognition performance.

Critical strengths
  1. The paper recognizes that client heterogeneity not only poses challenges to model training but can also lead to complementary predictive capabilities. In particular, clients with similar overall accuracy may exhibit substantial performance differences on specific classes. This provides a reasonable motivation for fine-grained cross-client knowledge transfer.

  2. The method represents different correction patterns using lightweight visual adapters in a shared feature space and routes them based on their actual responses within the recipient's predictor. This design is well motivated and also retains the original personalized prediction as an alternative.

  3. The paper compares the proposed method against 11 federated learning baselines on natural and medical image classification tasks. Some of the ablation results provide encouraging evidence supporting the effectiveness of the proposed method.

Critical weaknesses
  1. The advantages over existing multi-expert methods have not been sufficiently demonstrated. Although the authors emphasize fine-grained expertise extraction and recipient-aware expert fusion, the unique contributions of the core mechanism remain insufficiently distinguished from existing cross-client expert-sharing methods such as FedDRM and dFLMoE. Moreover, the paper does not include directly relevant MoE-based methods. In the main experimental comparisons, making it difficult to sufficiently establish its advantages over existing expert collaboration frameworks.

  2. A reduction in prediction loss does not necessarily indicate that a client possesses genuinely reliable expertise. Equation (2) identifies specialty samples solely based on the reduction in cross-entropy loss achieved by the personalized model relative to the global model. However, a lower loss does not necessarily imply a correct prediction. For example, the personalized model may merely increase the ground-truth class probability from 0.05 to 0.10. Even if the final prediction remains incorrect, the sample would still be included in the specialty region.

  3. The paper clusters samples based on the logit differences between the personalized and global models. However, these differences may primarily reflect class biases or differences in class distributions rather than distinct visual features or semantic patterns. Although Figure 2 demonstrates functional differences among the adapters, it does not provide sufficient direct evidence that these clusters correspond to independent, fine-grained visual expertise.

  4. After the base federated training, FedConsult requires an additional 20 epochs of adapter training and 50 epochs of router training. Moreover, the router must first compute the prediction responses of all candidate adapters before determining their fusion weights through Sparsemax. Therefore, even if only a few experts are ultimately selected, the forward computations associated with the other experts have already been performed. This raises concerns about the trade-off between the additional training and inference costs and the resulting performance improvements.

  5. The medical datasets contain multiple images from the same patient or lesion. Such correlated images may be distributed across the training set, test set, and different clients, potentially leading to overestimated generalization performance and cross-client knowledge transfer effectiveness.

  6. Table 5 shows that using only four local adapters improves Acc from 55.40% to 58.38%, while incorporating external adapters further increases it to 60.65%. However, the number of experts also increases from 4 to 32 when external adapters are introduced. Therefore, the independent contributions of complementary knowledge transfer and the increased number of experts have not yet been fully disentangled.

Questions for authors
  1. Table 4 compares the performance with different numbers of adapters, where M=2 achieves higher Weak accuracy than M=4. Why do the authors ultimately choose to fix the number of groups at four? In addition, how is the weighted K-means in Equation (4) initialized? How are cases handled when a client has insufficient specialty samples or when empty clusters occur?

  2. In Section 3.2, what is the specific joint training objective for the global and personalized prediction branches? Does it completely follow the training objective of DualFed? If any modifications are made, please clarify the individual loss terms and their corresponding weights.

  3. In Equation (6), the source client computes the adapter output using only the global prediction head during adapter training, whereas in Equation (7), the recipient client evaluates adapter responses using both the global prediction head and the personalized module. Could the authors explain the rationale behind using these different prediction pathways and whether this discrepancy affects the transfer effectiveness of adapters across clients?

  4. In Figure 1(a), the specialty region selection criterion appears to indicate that the personalized cross-entropy loss is greater than the global cross-entropy loss, which contradicts the definition in Equation (2). Please check and correct the direction of the inequality in the figure.

12 comments
AuthorsOpenReview ↗

W1: Distinction from Existing Multi-Expert Methods and Direct Comparisons

The advantages over existing multi-expert methods have not been sufficiently demonstrated. Although the authors emphasize fine-grained expertise extraction and recipient-aware expert fusion, the unique contributions of the core mechanism remain insufficiently distinguished from existing cross-client expert-sharing methods such as FedDRM and dFLMoE. Moreover, the paper does not include directly relevant MoE-based methods. In the main experimental comparisons, making it difficult to sufficiently establish its advantages over existing expert collaboration frameworks.

Response:

We thank the reviewer for raising this point. To clarify how FedConsult differs from FedDRM, Fed-MoE, and PM-MoE, we compare the methods directly and conduct a controlled experiment to isolate the effect of the choice of transfer carrier.

Shared-expert MoE methods, such as Fed-MoE, collaboratively learn shared experts and combine them to accommodate data heterogeneity across clients. Personalized expert-sharing methods, such as FedDRM and dFLMoE, instead reuse personalized models or prediction components trained by individual clients. These methods exploit client-specific capabilities, but the transferred components remain closely coupled to the source clients' data distributions and prediction structures, which may limit their transferability across clients. Recipients primarily use this external knowledge by invoking source prediction components and selecting or combining the predictions they produce.

In contrast, our proposed FedConsult changes both what is transferred and how external expertise interacts with local capabilities. We separately distill the distinct prediction-correction patterns acquired through personalization at each source client into visual adapters in a shared feature space. This makes local strengths individually reusable while reducing dependence on source-specific prediction structures. The adapters modify the recipient's feature representations, which are then processed further by its own personalized prediction pathway to produce predictions. External expertise thus participates directly in the recipient's prediction computation and interacts with its local personalized capabilities, helping address capability gaps while preserving the benefits of local personalization.

The original submission included FedDRM. Following the reviewer’s suggestion, we added two baselines: Fed-MoE for collaborative learning of shared experts and PM-MoE for cross-client reuse of personalized components. We could not locate an official public implementation of dFLMoE, so we used PM-MoE as a publicly available baseline that also supports cross-client reuse of personalized components. Both PM-MoE and FedConsult use DualFed in Stage 1. The table below reports Acc and Weak (both in %) across seven settings, using the revised data splits for Skin14 and BreaKHis.

MethodAcc / Weak
CIFAR-10 Dir-0.3CIFAR-10 5-SPCCIFAR-100 Dir-0.3CIFAR-100 25-SPCRetinaSkin14BreaKHis
Fed-MoE71.97 / 58.1568.24 / 54.0545.39 / 15.6742.19 / 19.2690.67 / 80.9765.95 / 0.0083.05 / 43.12
PM-MoE89.77 / 64.5277.86 / 70.1455.20 / 18.9562.85 / 39.4194.15 / 92.6267.23 / 7.9587.18 / 44.89
FedDRM86.74 / 59.0583.88 / 73.8556.69 / 17.1260.83 / 36.2391.78 / 86.5869.08 / 4.6282.63 / 32.60
FedConsult (Ours)90.47 / 65.4988.17 / 84.0760.65 / 28.7867.61 / 51.4795.13 / 93.2869.62 / 12.8688.11 / 52.51

FedConsult achieves higher Acc and Weak scores than the compared baselines in all seven settings. The controlled comparison in Table 4 of the main paper further shows that, for the same number of transferred units, transferring shared-feature adapters yields better performance than transferring classifier heads or entire personalized pathways. These results support FedConsult’s design of distilling source clients’ local strengths into transferable adapters that complement the receiving client’s personalized predictions.

We have incorporated these findings and their implications into the revised manuscript.

AuthorsOpenReview ↗

W2: Does Reduced Loss Imply Reliable Expertise?

A reduction in prediction loss does not necessarily indicate that a client possesses genuinely reliable expertise. Equation (2) identifies specialty samples solely based on the reduction in cross-entropy loss achieved by the personalized model relative to the global model. However, a lower loss does not necessarily imply a correct prediction. For example, the personalized model may merely increase the ground-truth class probability from 0.05 to 0.10. Even if the final prediction remains incorrect, the sample would still be included in the specialty region.

Response:

We thank the reviewer for pointing this out. FedConsult uses the reduction in prediction loss from the global model to the personalized model as a continuous signal for distillation. This signal captures improvements even when the predicted class remains unchanged. In response to the reviewer’s suggestion, we evaluated two controlled variants that restrict distillation to samples for which (i) the personalized model predicts correctly or (ii) the global model predicts incorrectly and the personalized model predicts correctly. The results support retaining the continuous loss-reduction signal for distillation.

An increase in true-class probability does not necessarily change the top-1 prediction. We use loss improvement to retain continuous information about how personalization improves upon the global model, rather than only samples that have already crossed the decision boundary. The usefulness of this information should be assessed through the resulting transfer performance, rather than inferred from sample selection alone.

We therefore compare three selection criteria using the same Stage 1 model and downstream training settings (CIFAR-100 Dir-0.3, %). Let u=LCE(lglo,y)−LCE(lper,y)u = \mathcal{L}_{CE}(\mathcal{l}^{glo},y) - \mathcal{L}_{CE}(\mathcal{l}^{per},y); thus, u>0u > 0 means that the personalized model has lower cross-entropy on the true label than the global model.

Selection criterionAccWeak
u>0u > 060.6528.78
u>0u > 0, personalized prediction correct60.3628.61
u>0u > 0, global prediction incorrect and personalized prediction correct59.8728.11

The first criterion retains all samples with positive improvement. The second additionally requires the personalized prediction to be correct. The third further requires the global prediction to be incorrect, retaining only explicit incorrect-to-correct transitions. Both constrained variants perform worse than positive-improvement selection on both metrics. These results support retaining the broader relative-improvement signal rather than requiring correct classification before distillation. Thus, expertise here denotes relative predictive advantages available for distillation, whose value is assessed through downstream transfer, rather than a guarantee that every selected sample is correctly classified.

We have incorporated this clarification and the supporting results into the revised manuscript.

AuthorsOpenReview ↗

W3: Do Directional Groups Merely Reflect Class Bias?

The paper clusters samples based on the logit differences between the personalized and global models. However, these differences may primarily reflect class biases or differences in class distributions rather than distinct visual features or semantic patterns. Although Figure 2 demonstrates functional differences among the adapters, it does not provide sufficient direct evidence that these clusters correspond to independent, fine-grained visual expertise.

Response:

We thank the reviewer for pointing this out. Directional grouping aims to preserve different correction patterns separately, avoiding the loss of some capabilities when these patterns are learned together; it does not require each expert to correspond to an independent visual-semantic factor. Our additional random-grouping controls and within-class cross-correction analyses show that decomposition retains complementary corrective capabilities that a single adapter does not cover. Here, fine granularity refers to different corrective effects on samples within a class, rather than independent visual-semantic concepts.

We organize distillation samples by their logit-correction directions so that different adapters learn different prediction-correction patterns. This aims to reduce interference that may arise when these patterns are learned together and to retain corrective capabilities that can be used separately. The manuscript's terms "prediction-correction direction" and "distinct correction patterns" describe this functional specialization; they do not require each cluster to correspond to an independent visual concept or clinical subtype.

First, we examine whether decomposition based on logit differences improves the final predictive performance. To distinguish the contribution of the decomposition strategy from that of increasing the adapter count, we hold the number of adapters and all other training settings fixed and introduce a random within-class partitioning control. Selected samples from each class are randomly divided into four subsets, and subsets with the same index are pooled across classes. Both strategies train four adapters; they differ in whether distillation samples are organized by their prediction-correction directions. Results on CIFAR-100 Dir-0.3 are as follows (%):

Grouping strategyAccWeak
Directional grouping, M=460.6528.78
Random within-class partitions, M=459.3525.17

With the same number of adapters, directional grouping achieves higher Acc and Weak. This indicates that the benefit is not solely due to having more adapters: organizing specialty decomposition using logit differences also contributes.

We then examine whether the decomposition actually separates functionally different correction patterns, allowing adapters to correct errors that other adapters cannot. To address the concern about class differences, we evaluate adapters within the same recipient, source client, and ground-truth class, testing whether each adapter turns an incorrect prediction of the original personalized model into a correct one. An error is counted once if at least one adapter corrects it. The results are:

Adapters per source MOriginal errors corrected by at least one adapter
18.09%
213.54%
420.82%
828.75%

As M increases from 1 to 4, the proportion of original errors corrected by at least one adapter rises from 8.09% to 20.82%. At M=4, even after selecting the adapter that corrects the most errors in each group of four, another 4.33% of the original errors can only be corrected by other adapters. This accounts for approximately one fifth of all correctable errors. Moreover, 52.85% of the class groups exhibit bidirectional cross-corrections: A corrects some errors that B cannot, while B corrects some that A cannot. *These findings show that decomposition preserves complementary corrective capabilities beyond those covered by a single adapter, including complementarity within the same ground-truth class.

Together, the final prediction results and correction analysis support our design: decomposition based on logit differences provides benefits beyond an increased adapter count and preserves complementary correction patterns that can be invoked independently, broadening the range of errors that can be corrected.

We have incorporated these analyses and their implications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.