acceptodds
Under review as a conference paper at ICLR 2027

LATENT DOMAIN DISCOVERY FOR DOMAIN-GENERALISED PROTOTYPICAL FEDERATED LEARNING

Abstract

Federated Learning (FL) enables collaborative model training without sharing raw data, but heterogeneous client distributions can substantially hinder the learning of transferable representations. This challenge is particularly difficult under domain shift, where clients may contain diverse or mixed domain characteristics, while explicit domain labels are often unavailable. SOTA domain-aware FL methods require predefined domain identities, limiting their applicability to such settings. We propose FedLDD, a Latent Domain Discovery framework that jointly discovers latent domain structure from client prototypes and exploits it for domain-aware federated representation learning. Our key insight is that variations in class-wise client prototypes can be factorised into a compact set of latent domain prototypes, while each client can be represented as a soft composition of these latent domains. Specifically, FedLDD factorises the client prototype matrix into a client-to-latent-domain assignment matrix and a latent-domain prototype matrix. The learnt assignments are not treated as domain identities; instead, they provide soft domain supervision for local representation learning. Each client constructs class-specific soft prototypes by combining latent-domain prototypes according to its learnt assignment and uses them to perform domain-consistent prototype alignment and cross-domain prototype contrastive learning. This creates a closed-loop process in which client representations reveal latent domain structure, the discovered structure guides local learning, and the resulting prototypes refine subsequent domain discovery. Extensive experiments on diverse domain-shift benchmarks demonstrate that FedLDD improves federated generalisation while discovering compact latent domain structures. Further analyses show that soft latent-domain supervision effectively captures mixed client characteristics and provides a more flexible representation of cross-client distribution shifts than hard domain assignments.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Prototypical Federated Learning, Domain Shift, Latent Domain Discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G6E3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a new method for federated domain generalization, a setting in which each client holds a different domain. In this setting, clients hold different data that follow different probability distributions, resulting in a degradation of performance when performing federated learning. The authors tackle this problem a slightly different and harder problem, where the domain labels (i.e., to which domain each client's data belong) is not known by the server. The authors' method combines (i) learning a set of prototypes from each client (e.g., class mean vectors per domain) and (ii) a matrix factorization method that discovers/models the latent structure of the different domains. The paper validates on 3 image benchmarks: Office-10, PACS and DomainNet demonstrating superior performance to previous prototypical FL methods w/o domain labels, and competitive performance to prototypical FL with domain labels.

Critical strengths
  • The paper is overall well written and easy to follow
  • If demonstrated (see weaknesses) the competitiveness against FedDAP is a relevant finding
Critical weaknesses

There are several major problems with this paper.

Empirical Comparison Issues

Issue 1: The reported DomainNet results isn't really DomainNet

The original DomainNet dataset proposed by (Peng et al., 2019) contains 345 classes. The authors remark, in the appendix, that they use a subset of 10 classes, following (Zhou, 2024). There are a couple of serious problems here,

  • Using a subset of 10 classes severely downscales the DomainNet dataset (the authors don't even report which classes were selected, nor how many samples that represent) which is the precise setting it is supposed to test (in the DA/DG/FedDG community, this benchmark serves to demonstrate scalability).

  • By looking at (Zhou, 2024), these authors don't seem to define this 10 classes subset (they don't even experiment with DomainNet), so there's an attribution error here.

  • Even if we would overlook the previous 2 issues, reporting the results as "DomainNet" is misleading in the main text, as the only mention of subsampling comes in the appendix.

Issue 2: Missing Statistical Analysis of the reported results

The authors claim in Appendix A.2, lines 585-586,

Each experiment is independently repeated 3 times, and we report the mean accuracy averaged over the final 5 communication rounds.

The authors should,

  • Report which seeds were used in these experiments,

  • Report mean ±\pm std-dev so a clearer comparison can be established against the baselines.

Methodological Issues

Looking at Table 3, the over-complete row (e.g., R=8R=8) shows a relevant performance degradation for Office-10 (70.22 best vs. 64.47 R=8, worst) and PACS (85.25 best vs. 83.46 R=8 worse). Therefore the claim,

This prune-and-merge refinement enables the factorisation to begin with an over-complete set of latent components while progressively eliminating weak or redundant ones. Consequently, the effective number of latent domains is determined adaptively by the structure of the learnt client representations, rather than being explicitly prescribed or constrained by the number of observed domains.

is too strong. If the prune-and-merge mechanism was effective, we would not see a degradation at high RR. The reported results in Table 1, by the way, seem to come from the best rows in Table 3 which indicate a "select-on-test", which is apparently unfair to other methods.

Missing Analysis

  • The authors do not include a complexity analysis of Algorithm 2 nor a running time evaluation of how it affects Algorithm 1

  • It would be nice to have the prune-and-merge strategies included somewhere in Algorithms 1 and 2.

Questions for authors

I am going to use this section for requesting changes from the authors. If they manage to answer these in their rebuttal, I would be happy to increase my score,

  1. The authors did their experiments on an fairly heterodox setting. They should: (i) Run their method on the full DomainNet dataset (see [1]) and report performance against the competitors. (ii) Experiment with Office Home (see [2]) besides the other presented benchmarks, and desirably (iii; somewhat optional) use a more standard setting for the backbone, such as ResNet-50 for Office Home, ResNet-101 for DomainNet and Office 10 (see [1]) and ResNet-18 for PACS.

  2. The authors should include the mean +- std-dev in all results.

  3. The authors should include a practical way of selecting RR in their algorithm, desirably without the use of labeled information from the target domain

  4. The authors should report running times for Algorithm 2 and their algorithmic complexity

  5. The prune-and-merge strategies deserve more analysis and a dedicated algorithm.

References

[1] Peng, Xingchao, et al. "Moment matching for multi-source domain adaptation." 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2019.

[2] Venkateswara, Hemanth, et al. "Deep hashing network for unsupervised domain adaptation." 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, 2017.

3 comments

Response to Reviewer G6E3 (Part 1/3)

We sincerely thank the reviewer for the detailed and actionable feedback, and for explicitly listing the changes that would address their concerns. We have uploaded a revised manuscript and respond to each request below.


Issue 1 / Q1. DomainNet subset, attribution, benchmarks and backbones.

Clarification of the subset. We thank the reviewer for pointing out the insufficient description of our DomainNet setting. Our experiments use a 10-class subset of DomainNet, not the full 345-class benchmark. This subset follows the experimental setting of FedDAP (Le et al., 2026), and we adopted it to keep our comparison consistent with the baseline results reported there. The 10 classes are: bird, feather, headphones, ice cream, teapot, tiger, whale, windmill, wine glass and zebra, used across all six domains (Clipart, Infograph, Painting, Quickdraw, Real, Sketch). We agree that the original presentation was insufficiently clear and that the attribution to Zhou et al. (2024) is incorrect.

Backbones. Following the protocol of FedDAP, we use ResNet-10 for DomainNet and Office-10, and ResNet-18 for PACS (Appendix A.2). PACS therefore already uses the backbone suggested by the reviewer. All baselines in Table 1 use the same backbones, so the comparison is like-for-like.

Additional benchmarks. We agree that Office-Home and the full DomainNet would strengthen the evaluation. The full 345-class DomainNet with 20 clients exceeds our compute budget within the rebuttal period; we will state this explicitly as a limitation.

Revision:

  • Dataset specification: We will list the 10 selected classes above in Appendix A.2 and report the number of training/test samples per domain and in total.
  • Attribution: We will correct the attribution to FedDAP as the source of this 10-class setting.

Issue 2 / Q2. Statistical analysis: seeds and mean ± std.

We agree that reporting seeds and standard deviations improves transparency and reproducibility. Each FedLDD experiment is repeated with 3 seeds; for each seed, we average test accuracy over the final 5 communication rounds, and we will report mean ± std across seeds. Baseline results are taken from FedDAP, which follows an identical protocol. To make the comparison with the closest competitors statistically grounded, we are additionally re-running FedDAP with the same seeds and data partitions, and will report their mean ± std as well.

Revision: All FedLDD results will be reported as mean ± std, the seeds will be stated, and results we re-ran will be clearly distinguished from results taken from prior work.

(continued in Part 2/3)

Response to Reviewer G6E3 (Part 2/3)

Methodological issue / Q3. Sensitivity to RR, the prune-and-merge claim and selection of RR.

The reviewer's statement is correct. We agree that the quoted sentence overstates the effect of prune-and-merge. Prune-and-merge is a refinement mechanism that reduces redundant latent components; it does not guarantee invariance to the initial rank. To characterise the behaviour fully, we have extended Table 3 of the revised manuscript to initial ranks up to R=14R=14. Accuracy (%) for each initial RR (Table 3 of the revised manuscript; best per dataset in bold):

RR234568101214
DomainNet64.4964.7264.4964.1664.9864.6864.2764.2765.24
Office-1068.8167.7270.1066.6670.2264.4767.7268.6866.99
PACS85.0284.2085.2584.2685.0183.4684.5584.9685.37

The extended results show two distinct behaviours:

  • DomainNet and PACS are robust to RR. For all R∈{2,…,14}R \in \{2, \dots, 14\}, accuracy stays within 64.16–65.24% and 83.46–85.37%, respectively, and every tested RR outperforms the best label-free prototypical baseline (FedPLVM: 62.22% / 82.06%). On these benchmarks, performance is insensitive to the initial capacity.
  • Office-10 shows larger fluctuations across RR (64.47–70.22%), and several values of RR fall below FedPLVM (68.77%). These fluctuations are not monotonic in RR: R=8R=8 is the lowest, while R=10R=10 and R=12R=12 recover to 67.72% and 68.68%. This pattern points to variance rather than a systematic over-capacity effect. Office-10 is the smallest benchmark, so client prototypes, and hence the factorisation input XX, are noisier than on the other datasets. We acknowledge that prune-and-merge does not remove this sensitivity on small benchmarks.

Revision: We will rewrite the quoted sentence to state that (i) the initial rank R remains a hyperparameter; (ii) prune-and-merge refines the latent components but does not fully remove the need to specify R; (iii) performance is robust to R on DomainNet and PACS, while the small Office-10 benchmark shows larger variability. The extended Table 3 is included in the revised manuscript, and mean ± std for every R will follow.


Select-on-test and choice of RR.

We acknowledge that the submitted Table 1 reported the best R per dataset. The revised Table 1 now uses a single fixed R=4R=4 for all datasets, which yields 64.49 / 70.10 / 85.25 (DomainNet / Office-10 / PACS). This still outperforms FedPLVM (62.22 / 68.77 / 82.06) on all three datasets and exceeds FedDAP on PACS (84.63), so our conclusions do not rely on per-dataset tuning. In practice, we recommend a small shared RR (we use R=4R=4 throughout), since a larger initial capacity brings no consistent benefit on any benchmark (Table 3). No labelled information from the test domains is used to set RR in the revised results.

(continued in Part 3/3)

Response to Reviewer G6E3 (Part 3/3)

Missing analysis / Q4. Complexity and running time of Algorithm 2.

Let mm be the number of participating clients, CC the number of classes, dd the feature dimension, b=C⋅db = C \cdot d, RR the initial rank, and Ts=50T_s = 50 the number of optimisation steps.

  • Initialisation (FINCH): O(mlog⁡m⋅b)\mathcal{O}(m \log m \cdot b) on mm client vectors.
  • Joint optimisation: each Adam step computes AGAG and its gradients in O(mRb)\mathcal{O}(mRb), and the simplex projection costs O(mRlog⁡R)\mathcal{O}(mR \log R), giving O(Ts mRb)\mathcal{O}(T_s\, mRb) in total.
  • Prune-and-merge (Algorithm 3): O(R2b+mR)\mathcal{O}(R^2 b + mR).

For DomainNet (m=20m=20, R=6R=6, b=5120b=5120), TsmRb≈3×107T_s mRb \approx 3 \times 10^7, i.e. roughly 10810^8 FLOPs per round including the backward pass, independent of the number of local samples. The additional download per client is AkA_k and GG (Reff⋅C⋅d+ReffR_{\text{eff}} \cdot C \cdot d + R_{\text{eff}} floats), at most 120 KB for R=6R=6; after prune-and-merge, the measured download is 28–40 KB per round, below 0.05% of the model payload. Overall, FedLDD's per-client communication is on par with FedDAP's (78.97 vs. 79.05 MB on DomainNet).

Table 9 of the revised manuscript reports measured wall-clock costs on a single NVIDIA TITAN RTX (clients run sequentially; mean over steady-state rounds):

DatasetAlg. 2 / round (ms)Local training / round (s)Alg. 2 / totalFedDAP total / round (s)FedLDD total / round (s)
DomainNet59.0144.860.04%179.46144.99 (−19.2%)
Office-1057.919.720.29%28.6219.81 (−30.8%)
PACS54.4120.930.04%145.70121.05 (−16.9%)

Algorithm 2 takes 54–59 ms per round, below 0.3% of the round time when clients run sequentially. Moreover, because FedLDD replaces FedDAP's heavier per-batch client computation (1.35 vs. 22.32 ms per batch on DomainNet), a full FedLDD round is 17–31% faster than a FedDAP round.


Q5. A dedicated algorithm for prune-and-merge.

We agree. We have added Algorithm 3 (Appendix A.1), which details the prune-and-merge steps. It is invoked at the end of Algorithm 2 (line 9).


We thank the reviewer again for the concrete requests, which have improved the rigour and reproducibility of the paper. In summary, the revised manuscript now includes a fixed-R main result (Table 1), results up to R=14R=14 (Table 3), measured running time (Table 9) and a dedicated prune-and-merge algorithm (Algorithm 3). We will further add a corrected and transparent DomainNet description, mean ± std with seeds for FedLDD. We hope these address the reviewer's concerns and would be grateful if the reviewer would consider increasing their score.

Reviewer_PFRHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FedLDD, a prototype-based FL framework for domain shift that does not require domain labels. The server factorizes client prototypes into a few latent-domain prototypes plus soft client assignments, and clients use these assignments for soft prototype alignment and contrastive learning. Experiments on DomainNet, Office-10, and PACS show gains over prototype-based FL baselines and results comparable to FedDAP, which requires domain labels.

Critical strengths
  • The motivation is clear and easy to follow. The limitation of methods that need domain labels are well argued.

  • Discovering latent domain structure through a soft client-to-latent-domain assignment is an interesting idea. It does not require the need for domain labels.

Critical weaknesses
  • Limited novelty on client side. The client-side domain-aware training looks like a soft extension of FedDAP: SLCA and SLCC mirror FedDAP's prototype alignment and cross-domain contrastive loss, with hard domain assignments replaced by learned soft weights. The main novelty is therefore the server-side latent domain discovery, and the paper should state its contribution relative to FedDAP more explicitly.

  • The presentation of key mechanism needs improvement. Latent The presentation of the key mechanism needs improvement. Latent domain discovery is the core of the method, but Figure 2 shows it only as a small schematic of X ≈ AG. It does not show what a latent domain represents, how the soft assignments are produced or how they shape the personalized prototypes.

  • The mixed-domain claim is not well supported by experiments. L251-252 states taht the formulation naturally accomodates mixed-domain clients. However, section 3.1 (L162) assumes each client holds data from a single domain and main experiments follow that setting also. If possible, please evaluate the paritally mixed setting with two domains per client and provide the comparison against baselines to support the claim.

  • The paper is not well orgainized in the experiment section. Section 4.3 discusses Tables 3 to Table 7 in detail, however all tables are placed in the appendix so it is hard for the readers to follow.

  • Section 3.2 (L281–293) claims the number of latent domains is determined adaptively through prune-and-merge and doesn’t need to be specified. It is unclear whether R denotes the initial over-complete size or the final number of active latent domains, and whether it was tuned on test accuracy. The best values also match the true number of domains on DomainNet (6) and PACS (4), which raises the question of whether good performance still implicitly depends on knowing the domain count.

  • FedLDD still underperforms FedDAP on two of three datasets. It trails FedDAP on DomainNet (64.98% vs. 65.20%) and Office-10 (70.22% vs. 72.53%). This is understandable since FedDAP uses domain labels, but the paper also claims the latent representation is more compact and flexible than domain-specific prototypes. Please discuss this gap, and when the label-free approach should be preferred.

Questions for authors
  • How do the communication and computation costs compare with other prototype-based FL methods (e.g., FedProto, FPL, FedDAP)? Specifically, what is the server-side cost of the factorization and prune-and-merge at each round, and the extra download cost of sending A and G to clients?

  • Do the learned latent domains correspond to meaningful structure? Figure 3 shows t-SNE of the representations but not the assignment matrix A itself. Could the authors visualize A against the ground-truth domain labels (e.g., as a heatmap) and report how many latent domains remain active after prune-and-merge?

4 comments

Response to Reviewer PFRH (Part 1/3)

We sincerely thank the reviewer for the careful and constructive review, and for recognising the clear motivation of our work and the interest of label-free latent domain discovery. We have uploaded a revised manuscript and addressed each point below.


W1. Client-side novelty relative to FedDAP.

We agree that SLCA and SCCL are inspired by FedDAP's prototype alignment and cross-domain contrastive loss, and we will state this explicitly. We do not claim new loss families. Our claim is that these objectives are re-designed to operate on a learnt, soft and evolving latent-domain structure, which FedDAP cannot provide. Concretely:

  1. Alignment target. FedDAP aligns each sample with the prototype of its annotated domain. In SLCA, the target is a client-specific personalised prototype {p}{k,c} = \sum{r} A_{k,r}, g_{r,c}. This is a convex combination of latent-domain prototypes that need not correspond to any annotated domain. SLCA reduces to FedDAP-style single-domain alignment only in the special case where A_k is one-hot.
  2. Adaptive contrastive weighting. In SCCL (Eq. 12), same-class prototypes from all latent domains act as positives, each weighted by A_{k,r}. The pull strength toward each latent domain therefore adapts per client (Fig. 1b), instead of following a fixed same-domain / cross-domain split.
  3. Closed-loop coupling. A and G are re-estimated every round from prototypes produced by the updated local models, so the supervision targets evolve with representation learning. With hard labels, FedDAP's grouping is fixed throughout training.

The server-side discovery and client-side objectives are therefore coupled rather than independent. Table 4 supports this: with identical client objectives, replacing our factoriser with SSMF reduces Office-10 accuracy from 70.10% to 58.10% at the default R=4, and from 70.22% to 54.14% at R=6. The client-side gains directly depend on the quality of the discovered structure.

Revision: We will rewrite the contribution list in Sec. 1 to state that the primary novelty is server-side latent domain discovery, and that the client-side contribution is enabling domain-aware prototype supervision under soft, label-free and evolving assignments.


W2. Presentation of the latent domain discovery mechanism.

We thank the reviewer for this helpful suggestion. We will redesign Figure 2 to make the core mechanism explicit: how X is formed from client prototypes, how A and G are obtained, and how they define each client's personalised prototypes. We will also add a short paragraph clarifying what a latent domain represents: a shared set of class-wise prototype factors that explains variation across client prototypes. A latent domain may align with one observed domain or capture characteristics shared by several.


W3. Mixed-domain claim.

We thank the reviewer for pointing out this inconsistency. Our main experiments follow the FedDAP protocol (one domain per client) so that FedLDD can be compared directly with the label-dependent state of the art, since FedDAP requires a single domain label per client.

The statement at L251–252 refers to the representational capacity of the simplex formulation (Eqs. 7–8). It was not intended as an empirically validated claim, and we agree that the current wording overstates it.

The closest evidence in the paper is the boundary case in Table 7 ("All-D"), where every client holds all domains. Here cross-client prototype variation largely vanishes, leaving little structure to factorise. FedLDD drops in this setting, but FedAvg drops as well under the same partition, and FedLDD remains at or above FedAvg (56.07 vs. 53.55 / 61.50 vs. 61.01 / 74.40 vs. 72.84 on DomainNet / Office-10 / PACS; Table 8 of the revised manuscript). This indicates that FedLDD benefits from heterogeneity in client-domain composition and degrades to roughly FedAvg level, not below it, when that heterogeneity disappears.

Revision: We will (i) rephrase L251–252, the abstract and the conclusion to state that the formulation can represent mixed-domain clients, while our empirical evaluation focuses on the single-domain-per-client protocol; (ii) discuss the All-D result explicitly as a limitation; (iii) list systematic evaluation under partially mixed clients as future work.

(continued in Part 2/3)

Response to Reviewer PFRH (Part 2/3)

W4. Organisation of the experiment section.

We agree. In the revision, we will move Table 3 (sensitivity to RR) and Table 4 (factorisation ablation) into the main text and condense Sec. 4.3 accordingly. Tables 5–9 will remain in the appendix with concise summaries and explicit forward references.


W5. Role of RR and dependence on the true number of domains.

Definition of RR. RR is the initial (maximum) number of latent factors, as defined in Algorithm 2. It is not the final number of active latent domains. Prune-and-merge (Algorithm 3 in the revised manuscript) removes factors whose column mass falls below τp\tau_p times the largest column mass and merges factors with cosine similarity above τm\tau_m, so the number of active latent domains is ≤R\le R and is determined by the learnt representations.

Selection of RR. The submitted Table 1 reported the best RR per dataset from Table 3. To remove any concern about test-set selection, the revised Table 1 now uses a single fixed R=4R=4 for all datasets, which yields 64.49 / 70.10 / 85.25 (DomainNet / Office-10 / PACS). This still outperforms the best label-free prototypical baseline, FedPLVM (62.22 / 68.77 / 82.06), on all three datasets.

Dependence on the true number of domains. Good performance does not require knowing the domain count:

  • With R=2R=2, below the true count, FedLDD obtains 64.49% on DomainNet (6 domains) and 85.02% on PACS (4 domains). This is identical to the fixed-R=4R=4 result on DomainNet and 0.23 points below it on PACS.
  • On DomainNet and PACS, every tested R∈{2,…,14}R \in \{2, \dots, 14\} outperforms FedPLVM (Table 3 of the revised manuscript, now extended to R=14R=14).
  • On Office-10 (4 domains), accuracy is more sensitive to RR, and the best value (R=6R=6) does not match the true count. We discuss this sensitivity in our response to Reviewer G6E3.

The best RR coinciding with the true domain count on DomainNet and PACS therefore does not reflect a dependency, because a smaller RR performs comparably.

Revision: The revised Table 1 reports the fixed-R=4R=4 configuration, and Table 3 reports all values of RR. We will further clarify the definition of RR in Sec. 3.2 and report the number of active latent domains after prune-and-merge for each initial RR.


W6. Gap to FedDAP and when the label-free approach is preferable.

With the fixed R=4R=4 configuration, FedLDD trails FedDAP on DomainNet (64.49 vs. 65.20, 0.71 points) and Office-10 (70.10 vs. 72.53, 2.43 points), and outperforms it on PACS (85.25 vs. 84.63). When reliable domain labels exist, they provide direct supervision, and we do not claim that FedLDD universally outperforms domain-supervised methods.

Regarding compactness, FedLDD with R=2R=2 already achieves 64.49% on DomainNet (FedDAP: 65.20% using 6 domain-specific prototype sets) and 85.02% on PACS (FedDAP: 84.63% using 4). This shows that a small number of shared latent factors can capture most of the domain structure. We will rephrase the compactness claim in these concrete terms.

FedLDD should be preferred when (i) domain labels are unavailable or costly to annotate; (ii) domain definitions are inconsistent across institutions, as is common in practice (e.g., different scanners or acquisition protocols); (iii) domain identity is itself sensitive information that clients prefer not to disclose. We will add this discussion to Sec. 4.2.

(continued in Part 3/3)

Response to Reviewer PFRH (Part 3/3)

Q1. Communication and computation cost.

Let mm be the number of participating clients, CC the number of classes, dd the feature dimension, b=C⋅db = C \cdot d, and RR the initial latent capacity. In our setting, C=10C=10, d=512d=512, m≤20m \le 20, and R=4R=4 by default (R≤14R \le 14 in Table 3).

Server-side computation. Each round performs FINCH initialisation on mm vectors, followed by 50 Adam steps on ∥X−AG∥F2\|X - AG\|_F^2. Each step costs O(mRb)\mathcal{O}(mRb), and prune-and-merge costs O(R2b+mR)\mathcal{O}(R^2 b + mR). For DomainNet (m=20m=20, R=6R=6, b=5120b=5120), this amounts to roughly 10810^8 FLOPs per round. The measured time of Algorithm 2 is 54–59 ms per round, below 0.3% of the round time (Table 9 of the revised manuscript).

Communication. The upload is identical to FedProto/FPL/FedDAP (model + CC class prototypes). The additional download consists of AkA_k (RR scalars) and GG (R⋅C⋅dR \cdot C \cdot d floats), i.e. at most about 120 KB for R=6R=6 (fp32), comparable to FedDAP's domain prototypes (D⋅C⋅dD \cdot C \cdot d with D=6D=6 on DomainNet). After prune-and-merge, the measured download is 28–40 KB per round, below 0.05% of the model payload. Overall, per-client communication is on par with FedDAP (78.97 vs. 79.05 MB on DomainNet).


Q2. Meaning of the learnt latent domains and visualisation of AA.

We agree that Fig. 3 shows feature representations rather than AA itself. In Fig. 3, colours denote classes, small points denote local sample representations, and larger markers denote the learnt latent-domain prototypes gr,cg_{r,c}. The latent-domain prototypes of each class lie within the corresponding class cluster rather than collapsing to a single point, so each class is described by several latent-domain prototypes that capture intra-class domain variation while class discrimination is preserved. On DomainNet and PACS, additional prototypes at larger RR remain inside the class structure instead of creating spurious clusters, consistent with the robustness to RR in Table 3.

To answer the reviewer's question directly, we will add (i) heatmaps of AA against the ground-truth client domains, (ii) the number of active latent domains after prune-and-merge for each initial RR.


We thank the reviewer again for the constructive feedback, which has helped us clarify our contributions, tighten our claims, and improve the presentation. We hope these responses address the reviewer's concerns, and we would be grateful if the reviewer would consider raising their score. We are happy to provide any further clarification.

Reviewer_PFRHOpenReview ↗

Response to Authors' Rebuttal

I would like to thank the authors for their detailed rebuttal, which has addressed some of my initial concerns. The clarification regarding the novelty of FedLDD is helpful, but I believe it needs to be better reflected in the manuscript. Overall, I find the proposed idea interesting and promising. However, the paper still requires substantial revisions, particularly in clarifying the technical contributions and improving the overall presentation. I believe these revisions would be difficult to fully address within the current submission cycle. Therefore, I will maintain my original score.

Reviewer_JZfdOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets prototype-based federated learning under domain shift when clients do not provide domain labels. Each round, the server stacks the class-wise prototypes of the mm participating clients into X∈Rm×bX \in \mathbb{R}^{m \times b}. It then factorises X≈AGX \approx AG, where each row of AA lies on the simplex, GG holds RR latent-domain prototypes, and a group-sparsity term on the columns of AA is added. Latent domains are pruned and merged starting from an over-complete RR. Clients then train with two soft, AA-weighted versions of FedDAP's losses: alignment to a personalised prototype p~k,c=∑rAk,rgr,c\tilde p_{k,c}=\sum_r A_{k,r} g_{r,c} and a cross-latent-domain contrastive term. The method is evaluated on DomainNet (20 clients), Office-10 and PACS (10 clients each). It beats label-free prototypical baselines and comes close to label-using FedDAP. Ablations over losses, RR, factoriser, initialisation and client composition are in Tables 2~7.

Critical strengths

A practical removal of the domain-label requirement with small cost (Table 1).

FedLDD is +2.8/+1.5/+3.2 points over the best label-free prototypical baseline and within 0.2/2.3 points of FedDAP on DomainNet/Office-10. It is 0.6 points above FedDAP on PACS. The server-side change is cheap (m×bm \times b factorisation, 50 Adam steps). Whatever the mechanism, this matters for deployments where domain metadata is missing.

Critical weaknesses
  1. The central hypothesis is never tested in the setting that motivates it.

The claim is that client prototypes are explained by a few latent factors and that soft assignments capture mixed clients. But the paper assumes every client holds a single domain, and every main experiment uses that partition. Under this assumption, the client ID is already a valid, label-free hard domain partition: it is just finer than the true one. So "no domain labels" is weaker than the paper suggests. The only mixed setting drops FedLDD by 9~11 points, to 55.6/61.0/74.4. On DomainNet and PACS that is below FedAvg in the default setting (59.59/77.07, Table 1). It is not clear how much of the drop comes from the setting itself, because no baseline is run in All-D. The paper reads this drop as showing that heterogeneity is needed, but it is the regime the method is motivated by.

  1. The discovered structure is never checked, and RR is chosen on test accuracy.

    • No check on AA/GG. The paper reports no agreement between arg⁡max⁡rAk,r\arg\max_r A_{k,r} and the known client domains (e.g., ARI/NMI), no view of AA, no reconstruction error, and no effective number of latent domains after prune-and-merge. Fig. 3 shows feature representations, not AA or GG.
    • Table 3 does not show robustness. RR is only an upper bound, which pruning then reduces. So a flat curve may mean that every run collapses to the same effective RR, not that the method is robust. The design itself (group sparsity, τp=0.05\tau_p=0.05, τm=0.95\tau_m=0.95) pushes toward few components. Results saturate at R=2R=2, below the 4-6 true domains, and become unstable beyond it (Office-10: 64.47-70.22). R=1R=1, one shared prototype per class, is the key control for whether the factorisation helps at all, and it is missing.
    • Test-set selection. Table 1 uses the best RR per dataset from Table 3. The ablation in Table 2 is run at R=8R=8, which is not the reported configuration.
  2. Table 5 appears to report the -SR results, not a revitalisation run (Table 4 vs. Table 5).

Table 5 "Rev." matches Table 4 "-SR" in 10 of 12 cells: every cell of DomainNet and Office-10 (R=2,3,4,6R=2,3,4,6), plus PACS at R=4,6R=4,6. The −SR column itself looks genuine, because its R=8R=8 row (64.24/65.28/84.34) matches the SLCA+SCCL-without-SR row of Table 2. The two PACS cells that differ (R=2,3R=2,3: 85.18/83.19 vs. 84.60/84.43) suggest the table was partly copied, not regenerated. If so, the claim that revitalisation "does not provide a consistent benefit" rests on no experiment, and the default pruning-only design is not justified against it. It also reduces my trust in the other ablation tables, which I cannot cross-check in the same way.

Questions for authors

Please see the weaknesses

4 comments

Response to Reviewer JZfd (Part 1/3)

We sincerely thank the reviewer for the exceptionally careful review, including the cross-checking of our tables, and for recognising the practical value of removing the domain-label requirement at small cost. We have uploaded a revised manuscript. We take every point seriously and respond below.


W1. The central hypothesis and the single-domain-per-client setting.

(a) Client ID as an implicit partition. We agree with the reviewer. When each client holds a single domain, client identity implicitly provides a hard partition that is finer than the true domain partition, even though no domain annotation is given to the algorithm. Our description of "no domain labels" was therefore stronger than warranted, and we will revise it.

We would, however, like to clarify what FedLDD adds on top of this implicit partition. A client-level partition only states that each client is its own group; it does not reveal which clients share a domain. FedLDD discovers this cross-client structure, compressing KK clients (20 on DomainNet) into a small number of shared latent domains via the factorisation, without any domain metadata. That is the information FedDAP obtains from domain labels. We will rephrase the claim accordingly: FedLDD does not use explicit domain labels or a predefined taxonomy, and it discovers shared domain structure across clients. We will also explicitly acknowledge that the single-domain-per-client protocol provides implicit structure that facilitates this discovery.

(b) The All-D setting. We agree that the original evaluation was incomplete without a baseline under the same partition. We have now run FedAvg under All-D with identical client allocation and training configuration. The comparison is reported in Table 8 of the revised manuscript; FedLDD values are taken from Table 7 (All-D) and Table 3 (default), both at R=4.

SettingMethodDomainNetOffice-10PACS
Default (1D)FedAvg59.5957.4777.07
Default (1D)FedLDD64.4970.1085.25
All-DFedAvg53.5561.0172.84
All-DFedLDD56.0761.5074.40

Three observations follow:

  1. FedLDD remains above FedAvg under the same All-D partition (+2.52 / +1.56 points on DomainNet / PACS) and is on par on Office-10 (+0.49).
  2. The partition itself causes part of the degradation. FedAvg also drops on DomainNet (−6.04) and PACS (−4.23) when moving to all domains per client, so the absolute decrease of FedLDD should not be attributed to the method alone.
  3. FedLDD's advantage shrinks markedly in All-D: it falls from +4.90 / +12.63 / +8.18 points in the default setting to the small margins above. On Office-10, FedAvg even improves under All-D (+3.54), since identical domain mixtures make clients closer to IID. This agrees with how FedLDD works: its gains come from exploiting cross-client differences in domain composition. When every client holds the same mixture of all domains, the rows of XX become near-duplicates and there is little latent structure to discover. FedLDD then degrades gracefully to roughly FedAvg-level performance rather than below it.

We therefore agree with the reviewer that All-D does not validate our motivating hypothesis. It represents the limiting case of homogeneous mixtures, whereas our motivation concerns clients with heterogeneous mixtures of domains, for example two domains per client in varying proportions. We acknowledge that this intermediate regime is not evaluated in the current paper.

Revision: We have (i) added the FedAvg All-D comparison as Table 8. We will (ii) correct the interpretation of Table 7 in Sec. 4.3; (iii) revise the abstract and Sec. 1 so that we no longer overstate the generality of latent-domain discovery; and (iv) explicitly state the evaluation of partially mixed clients as a limitation.

(continued in Part 2/3)

Response to Reviewer JZfd (Part 2/3)

W2. Verification of the discovered structure and selection of RR.

(a) Checks on AA and GG. We agree that Fig. 3 shows feature representations rather than AA or GG. We will add (i) heatmaps of AA against the ground-truth client domains; (ii) the number of active latent domains after prune-and-merge for each initial R.

(b) Interpretation of Table 3. We agree that RR is an upper bound that prune-and-merge reduces. However, we respectfully note that this is the intended design of FedLDD rather than a confound: the method starts from an over-complete capacity and lets the data determine the effective number of latent domains (Sec. 3.2, Algorithm 3 in the revised manuscript). If runs from different initial RR converge to a similar effective number, this indicates that the effective capacity is determined adaptively rather than prescribed, which is the property we claim.

The effective number may be smaller than the number of observed domains. This is consistent with our view that latent factors need not map one-to-one onto observed domains: several domains can share underlying factors. We acknowledge that Office-10 is less stable across RR, and we will state this explicitly.

(c) Selection of RR and the Table 2 configuration. We acknowledge that the submitted Table 1 reported the best RR per dataset from Table 3. To remove any concern about test-set selection, the revised Table 1 now uses a single fixed R=4R=4 for all datasets, which yields 64.49 / 70.10 / 85.25 (DomainNet / Office-10 / PACS). This still outperforms the best label-free prototypical baseline (FedPLVM: 62.22 / 68.77 / 82.06) on all three datasets, by +2.27 / +1.33 / +3.19 points. Per-dataset results for all R∈{2,…,14}R \in \{2, \dots, 14\} are reported in the extended Table 3.

Regarding Table 2, we stated in the text that the ablation was run at R=8R=8, which is not the reported configuration, and that its conclusions on SR should be read with this in mind.

(continued in Part 3/3)

Response to Reviewer JZfd (Part 3/3)

W3. Table 5 duplicates Table 4.

We thank the reviewer for carefully identifying this inconsistency, and we sincerely apologise. This was an error during manuscript preparation: the values in the Rev. column of Table 5 were incorrectly copied from the −SR column of Table 4.

We have revisited the original experimental logs, recovered the actual revitalisation results and corrected Table 5 in the revised manuscript:

RRDomainNet OursDomainNet Rev.Office-10 OursOffice-10 Rev.PACS OursPACS Rev.
264.4964.8068.8169.3885.0285.18
364.7263.5667.7267.3784.2083.19
464.4963.9170.1067.0385.2583.88
664.9864.8470.2264.1185.0183.40

Revitalisation is slightly better only at R=2R=2 (+0.31 / +0.57 / +0.16 on DomainNet / Office-10 / PACS) and worse in all nine cells for R≥3R \ge 3 (e.g., Office-10 at R=6R=6: 64.11 vs. 70.22). This supports our pruning-only default design, and the conclusion in Sec. 4.3 now rests on these verified results; we will update the associated discussion accordingly.

Given that this error reasonably reduces confidence in the other tables, we have re-checked every entry of the remaining tables against the experimental logs.


We believe these revisions address the reviewer's concerns about soundness, and that the core finding remains supported by verified results: domain-aware prototype supervision can be made effective without domain labels, at negligible cost. We are grateful for the reviewer's rigorous assessment, which has substantially improved the paper's rigour. We hope the clarifications and new results address the concerns, and we would kindly ask the reviewer to reconsider their score.

The corrections make the tables more reliable. However, the claim that soft assignments capture clients with different domain mixtures is still untested, and the main mechanism has not been separated from simple clustering. I am keeping my rating at 2.

Comments on the paper

1 comment

General Response to All Reviewers and the AC

We thank all three reviewers for their careful and constructive reviews. We appreciate that the reviewers found the motivation clear (PFRH, G6E3), the removal of the domain-label requirement practically valuable at small cost (JZfd), and the competitiveness with label-dependent FedDAP a relevant finding (G6E3). We have uploaded a revised manuscript. Below we summarise our contribution, the shared concerns, and the changes made.


Contribution, stated precisely. The main novelty of FedLDD is server-side latent domain discovery: client prototypes are factorised as X≈AGX \approx AG with simplex-constrained soft assignments AA and a compact set of latent-domain prototypes GG, estimated every round without domain metadata. The client-side losses (SLCA, SCCL) are soft, assignment-weighted versions of FedDAP's objectives, and we will state this explicitly in Sec. 1. Following the reviewers' feedback, we have also toned down claims that were stronger than our evidence (see point 2).


Shared concerns and how we addressed them

1. Choice of RR and test-set selection (JZfd, G6E3, PFRH). The revised Table 1 now uses a single fixed R=4R=4 for all datasets (64.49 / 70.10 / 85.25 on DomainNet / Office-10 / PACS), which still outperforms the best label-free baseline FedPLVM (62.22 / 68.77 / 82.06) on all three datasets. Table 3 is extended to R∈{2,…,14}R \in \{2, \dots, 14\}: DomainNet and PACS are robust across all RR, while Office-10, the smallest benchmark, fluctuates more. We acknowledge this openly and will rewrite the prune-and-merge claim in Sec. 3.2: RR remains a hyperparameter, and prune-and-merge refines but does not remove it.

2. Mixed-domain clients and the "no domain labels" claim (PFRH, JZfd). We agree that the single-domain-per-client protocol provides implicit structure and that the mixed-domain statement was not empirically validated. We will rephrase the abstract, Sec. 1, L251–252 and the conclusion accordingly, and list partially mixed clients as a limitation. For the All-D setting, we added FedAvg under the same partition (new Table 8): FedAvg also degrades, and FedLDD remains at or above it, so part of the drop stems from the partition itself.

3. Verification of the discovered structure (PFRH, JZfd, G6E3). We will add heatmaps of AA against the true client domains, the number of active latent domains after prune-and-merge for each RR.

4. Prune-and-merge and computational cost (G6E3, PFRH). We added a dedicated Algorithm 3, a complexity analysis, and measured wall-clock costs (new Table 9): Algorithm 2 takes 54–59 ms per round (<0.3% of round time), the extra download is 28–40 KB per round, and a full FedLDD round is 17–31% faster than FedDAP.

5. Table 5 error (JZfd). The Rev. column of Table 5 had been copied from Table 4 by mistake. We recovered the actual results from our logs and corrected Table 5. Revitalisation is better only at R=2R=2 and worse in all nine cells for R≥3R \ge 3, which supports our prune-and-merge design.

6. DomainNet subset and reporting (G6E3). The 10-class subset follows FedDAP, not Zhou et al. (2024); the classes are bird, feather, headphones, ice cream, teapot, tiger, whale, windmill, wine glass and zebra, which we will list in Appendix A.2, and we will refer to the benchmark as DomainNet throughout. PACS already uses ResNet-18, as suggested.


Summary of changes

Already in the revised manuscript: fixed-R=4R=4 main results (Table 1); RR up to 14 (Table 3); corrected Table 5; FedAvg under All-D (Table 8); wall-clock costs (Table 9); prune-and-merge algorithm (Algorithm 3).

In progress, to be included in the final version: analysis of AA (heatmaps, active latent domains, R=1R=1 control); mean ± std with seeds for FedLDD and re-run FedDAP; Table 2 rerun at R=4R=4; revised wording of the claims listed above.

We are grateful for the reviewers' rigorous feedback, which has substantially improved the paper, and we welcome any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.