acceptodds
Under review as a conference paper at ICLR 2027

When Multi-Domain Federated Learning Does Not Need Domain-Aware Capacity

Abstract

Domain heterogeneity in federated learning does not by itself require full domain-aware capacity. When domains differ only in class proportions, adjusting class biases can suffice. For a fixed representation and an observed domain context, we compare shared predictors that ignore context, prior-aware predictors that adjust only class biases, and full domain-aware predictors that may change the feature-to-label mapping across contexts. Under log-loss, we characterize the shared-to-domain Bayes gap through conditional mutual information and derive the exact residual after canonical prior correction. Combined with family- and protocol-dependent training-quality bounds, these gaps yield a sequential one-sided rule: certify a learned shared predictor when possible, otherwise test the prior-aware predictor, and abstain when neither certificate passes, without declaring richer capacity necessary. For full-participation local SGD on frozen affine heads, we derive a recoverability bound separating optimization transient, aggregated stochastic noise, local stochastic accumulation, and client-objective drift. Controlled studies audit the reference quantities and calibrate the stopping rule under known ground truth. Real-data frozen-feature experiments show that full domain-aware heads can lower centralized log-loss, while ordinary shared aggregation often fails to recover this gain. Structure-aligned routing recovers substantially more but does not close the centralized gap. The framework separates the statistical value of domain-conditioned capacity from its federated recoverability.

Primary area:
learning theory
Keywords:
Federated learning, Domain heterogeneity, Capacity sufficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aaErOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when multi-domain FL needs domain-aware capacity. It derives information-theoretic analysis under fixed representation scenarios, showing that shared or prior-aware heads may be sufficient when domain differences mainly reflect class-prior shifts.

Critical strengths
  1. Clear separation between statistical value of domain-aware capacity and federated recoverability.

  2. Elegant theoretical analysis, and the experiment results well aligned to the theory.

Critical weaknesses
  1. The capacity hierarchy is too coarse. For example, the evolving lines may contain: Shared → Prior-aware → Clustered / grouped heads → Routed / mixture heads → Full domain-aware

  2. The key limitation is the fixed-representation assumption: the choice of head family can change the learned representation, and in FL, we train the whole model in most cases.

  3. The experiments are mostly frozen-feature affine-head studies, which are quite simplified for practical scenarios.

Questions for authors
  1. Can the framework handle grouped or clustered domain heads?
5 comments
AuthorsOpenReview ↗

Response to Reviewer aaEr (1/4): Capacity hierarchy

Thank you for highlighting the distinction between statistical value and federated recoverability. We agree that a broader range of head-sharing structures is relevant. Below we explain the purpose of the three reference levels, give an explicit fixed-group extension with experiments, and clarify the scope of the additional nonlinear experiments and the fixed-representation assumption, including a first experiment in which part of the encoder is trained jointly with each head family.

The capacity hierarchy is too coarse. For example, the evolving lines may contain: Shared → Prior-aware → Clustered / grouped heads → Routed / mixture heads → Full domain-aware

We agree that the three reference levels do not exhaust the useful sharing structures. They were chosen to distinguish three specific uses of the observed context: no context dependence, context-dependent class offsets with a shared feature-to-score map, and unrestricted context-dependent prediction. The PA level is important because it separates the improvement captured by canonical prior correction from the residual relative to unrestricted domain-aware prediction. Without this intermediate reference, those quantities would be combined in the Shared-to-Dom comparison. When a Shared or PA predictor is certified as ϵ\epsilon-sufficient, the comparison is with unrestricted domain-aware prediction, so the same tolerance also bounds the gain of every intermediate family that uses the same representation and observed context.

The hierarchy can be refined exactly. Let G1,…,Gk=DG_1, \dots, G_k = D be nested groupings of the observed domains, each a deterministic function of the next, and let G0G_0 be constant. The chain rule gives

I(Y;D∣Z)=∑j=1kI(Y;Gj∣Z,Gj−1),I(Y;D\mid Z)=\sum_{j=1}^{k} I(Y;G_j\mid Z,G_{j-1}),

and the level that uses GjG_j has the exact reference gap I(Y;D∣Z,Gj)I(Y;D\mid Z,G_j) to full domain-aware prediction. Every intermediate grouping level therefore has its own reference gap and its own one-sided certificate. The revision will state this as a new proposition, and our answer on grouped heads below gives the case k=2k = 2 with experiments.

Grouped heads fit into the hierarchy once it is specified what the domains in a group share. A pure grouped head uses G=g(D)G=g(D) and is identical across domains assigned to the same group. It need not contain PA, since PA permits different class offsets for domains within a group. Conversely, a grouped head can use a different feature-to-score map in each group, which PA cannot. So in general neither family contains the other. Adding per-domain class offsets gives a grouped family that contains PA and is contained in Dom under compatible base score classes. Our answer on grouped heads below evaluates both specifications.

Routed or mixture heads combine several expert heads through a gate, and the gate determines how they are handled. A fixed hard gate that depends only on the domain gives a grouped head, so the grouped analysis applies. For a fixed soft gate G=g(Z,D)G = g(Z, D) with experts acting on ZZ, all measurable predictors of (Z,G)(Z, G) provide an unrestricted reference family. The mixture is contained in this family, and its approximation and training errors enter the training margin. For a learned gate, the margin would also need to account for gate learning and selection. Routed WA in our submitted experiments is different: it changes how domain blocks are aggregated while retaining the domain-specific prediction architecture. It is therefore a training protocol for that architecture, rather than an additional intermediate capacity level.

The revision will describe the three levels as the reference structures studied here and explain how additional explicitly defined families are assessed.

AuthorsOpenReview ↗

Response to Reviewer aaEr (2/4): Grouped and clustered heads

Can the framework handle grouped or clustered domain heads?

Yes, for a fixed grouping of the observed domains there is a direct population extension. Let G=g(D)G=g(D) be a prespecified group label, and let the grouped family contain all measurable predictors of (Z,G)(Z,G). Its Bayes risk is H(Y∣Z,G)H(Y\mid Z,G). Because GG is a deterministic function of DD, the remaining gap to full domain-aware prediction is

RGrp⋆−RDom⋆=H(Y∣Z,G)−H(Y∣Z,D)=I(Y;D∣Z,G).R_{\mathrm{Grp}}^\star-R_{\mathrm{Dom}}^\star =H(Y\mid Z,G)-H(Y\mid Z,D) =I(Y;D\mid Z,G).

The shared-to-domain gap correspondingly decomposes as

I(Y;D∣Z)=I(Y;G∣Z)+I(Y;D∣Z,G).I(Y;D\mid Z)=I(Y;G\mid Z)+I(Y;D\mid Z,G).

This separates the value captured by knowing the group from the domain information that remains useful within groups. If a learned grouped predictor satisfies a valid bound R(hGrp)−RGrp⋆≤ηGrpR(h_{\mathrm{Grp}}) - R_{\mathrm{Grp}}^\star \leq \eta_{\mathrm{Grp}}, it has the sufficient condition

I(Y;D∣Z,G)+ηGrp≤ϵ.I(Y;D\mid Z,G)+\eta_{\mathrm{Grp}}\leq\epsilon.

As with the original rule, both terms require valid upper controls for finite-sample use. A finite affine or neural grouped head contributes approximation and training error to its margin, and Theorem 6 is stated for Shared and PA heads, so the margin of a grouped head must be supplied separately. If the grouping is learned or selected from data, the guarantee must additionally account for that selection or use an appropriate independent assessment. The revision will include this fixed-group extension as a new proposition.

We have also completed a fixed-group empirical study on OfficeHome and VLCS, using all three balanced two-group partitions of the four domains under the original domain ordering: AB/CD (g1), AC/BD (g2), and AD/BC (g3). The groups were fixed before evaluating test performance, and all three are reported. We fitted two grouped families: a pure grouped head, identical across the domains of a group, and a grouped head with per-domain class offsets, the specification that can contain PA. The table gives the held-out NLL of the fitted affine heads under centralized training with the protocol of the submission, as the mean over three data splits. These are results for fitted heads under one training protocol. The grouped reference identities above concern the pure grouped family.

Dataset and regimeSharedPAGrouped, g1 / g2 / g3Grouped with per-domain offsets, g1 / g2 / g3Dom
OfficeHome, natural0.46510.45890.4556 / 0.4627 / 0.45640.4524 / 0.4595 / 0.45430.4554
OfficeHome, prior 0.30.53750.52650.5505 / 0.5554 / 0.55230.5457 / 0.5508 / 0.54810.5823
VLCS, natural0.36870.36060.3459 / 0.3630 / 0.35630.3456 / 0.3603 / 0.35370.3466
VLCS, prior 0.30.31610.30340.2977 / 0.3034 / 0.31180.2960 / 0.3027 / 0.30450.3005

Adding the per-domain offsets lowers the NLL of the matched pure grouped head in 34 of the 36 split-level comparisons. How the grouped head with offsets compares with PA and Dom depends on the setting. On OfficeHome with prior 0.3, it is worse than PA and better than Dom in all nine comparisons, so along the nested PA, grouped-with-offsets, and Dom families, more capacity gives a higher fitted test NLL. In the other three settings, it is better than PA in 21 of 27 comparisons and better than Dom in 11 of 27. The revision will report these results, with the split-level values, in the new appendix on grouped heads.

AuthorsOpenReview ↗

Response to Reviewer aaEr (3/4): Fixed-representation assumption

The key limitation is the fixed-representation assumption: the choice of head family can change the learned representation, and in FL, we train the whole model in most cases.

We agree that the head family can affect representation learning during end-to-end training. The current analysis intentionally compares the use of domain context on a common representation. This makes it possible to isolate the statistical value of domain-dependent prediction from changes in the information encoded by the features. It applies directly when a pretrained or otherwise fixed feature map is used, but does not characterize all benefits that domain-aware capacity may produce through joint representation learning.

For a specified representation, the identity RSh⋆−RDom⋆=I(Y;D∣Z)R_{\mathrm{Sh}}^\star - R_{\mathrm{Dom}}^\star = I(Y;D\mid Z) remains valid. If different head choices learn different representations, the comparison involves different random variables ZZ and additional representation-dependent errors. A single conditional mutual information term on one common representation no longer describes that complete comparison. The explicit affine local-SGD bound also relies on a fixed head optimization problem and does not cover the moving representation in joint training.

The framework can be applied after representation learning by freezing the learned representation and comparing heads on it, and we ran a first experiment of this kind. On VLCS we fine-tuned the last transformer block, the final layer norm, and the projection of the image encoder jointly with a Shared, PA, or Dom head under centralized training, on three data splits. The jointly trained Shared, PA, and Dom systems have test NLL 0.3052, 0.3077, and 0.2956 nats, and the Dom system is below the PA system in all three splits. We then froze each learned encoder and refitted the three heads on it. The table reports the test NLL of the refitted PA head minus that of the refitted Dom head, as mean ± sample standard deviation over the three splits, in nats per example. Positive values indicate a gain from Dom.

Encoder trained jointly withPA minus Dom test NLL
Shared head0.0017 ± 0.0023
PA head0.0075 ± 0.0047
Dom head0.0233 ± 0.0029

In this experiment the head family used during training changes the head-level gain measured on the resulting representation, consistent with the effect you describe, and on each fixed representation the comparison is the one the paper analyzes. A guarantee for fully joint training would need a training-quality bound that controls the representation drift and how closely the heads track it under federated updates. Comparing systems that learn different representations would additionally require control of representation-dependent approximation, optimization, and generalization errors. The revision will report this experiment in a new appendix and will strengthen the scope statement in the introduction and conclusions accordingly.

Reviewer_xStJOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates domain heterogeneity in federated learning (FL) and questions the standard practice of introducing local or domain-aware model capacity (e.g., domain-specific heads or normalization). The authors analyze whether shared predictors or prior-aware predictors (which adjust class biases) are sufficient when domains differ primarily in class proportions. Under log-loss, they derive exact population reference gaps using conditional mutual information: the shared-to-domain Bayes gap and the canonical prior-aware residual. Combining these with family- and protocol-dependent training quality bounds, they present sequential one-sided certificates. Finally, they provide optimization-drift recoverability bounds for full-participation local SGD on frozen affine heads and perform extensive controlled and real-data audits (e.g., PACS, OfficeHome, VLCS, DomainNet).

Critical strengths

Rigorous Information-Theoretic Framing: The derivation of exact population reference gaps (such as RSh⋆−RDom⋆=I(Y;D∣Z)R^\star_{\text{Sh}} - R^\star_{\text{Dom}} = I(Y; D \mid Z)) provides a highly rigorous mathematical foundation for evaluating when simpler model architectures are theoretically sufficient.

Practical Sequential Gating Rule: The introduction of sequential one-sided certificates offers a systematic and computationally inexpensive way to determine whether a shared or prior-aware predictor suffices, helping avoid over-parameterization and costly local-capacity configurations.

Critical weaknesses

Focus on Frozen/Affine Bottlenecks: The theoretical analysis and recoverability bounds are largely limited to frozen representation features and affine classification heads. It remains unclear how these guarantees extend to end-to-end representation learning or highly non-linear classification setups.

Conservative Bounds: The optimization and recoverability envelopes are quite conservative. As acknowledged by the authors, no numerical tightness is claimed, which limits the immediate numerical applicability of these certificates to real-world edge devices.

Questions for authors

How do you envision relaxing the assumption of a fixed representation Z = f(X)? Would an end-to-end training setup fundamentally break the conditional mutual information guarantees?

Can you discuss how the sequential certificates perform when client participation is highly partial and stochastic? How does this affect the convergence of the training-quality envelopes?

5 comments
AuthorsOpenReview ↗

Response to Reviewer xStJ (1/4): Nonlinear classification heads

Thank you for the positive assessment of the information-theoretic formulation and for the questions about its scope. We clarify below which conclusions concern population prediction, which require an explicit training-quality bound, and which are specific to full-participation affine local SGD. We also report the available nonlinear-head and partial-participation evidence and a first experiment with a partially fine-tuned encoder.

Focus on Frozen/Affine Bottlenecks: The theoretical analysis and recoverability bounds are largely limited to frozen representation features and affine classification heads. It remains unclear how these guarantees extend to end-to-end representation learning or highly non-linear classification setups.

As you note, the analysis is carried out on a fixed representation, and the explicit recoverability guarantees in Theorem 6 and Corollary 7 concern full-participation local SGD on affine heads. Fixing the representation isolates how the predictor uses the domain context from how the representation is learned (Section 3.1), so the three predictor families are compared on the same features. For affine softmax heads, bounded features and positive ridge regularization on all trainable coordinates yield smooth, strongly convex objectives (Proposition 9 in Appendix B.3). These properties let us bound the local-SGD optimization error with separate terms for the optimization transient, aggregated stochastic noise, local stochastic accumulation, and client-objective drift.

On how these guarantees extend to nonlinear classification heads, the population-level results already cover them. A Shared predictor may be any measurable function of ZZ, the PA score map s(z)s(z) may be nonlinear, and the full domain-aware predictor may be any measurable function of (Z,D)(Z,D), so Proposition 1 and the canonical PA decomposition in Proposition 4 do not require an affine head. Proposition 5 is conditional on a valid learned-to-reference risk bound, not on a specific architecture.

Theorem 6 does not supply such a bound for nonlinear heads, so we examined them empirically. The additional experiments use one-hidden-layer ReLU heads of widths 64 and 128 on the same frozen features. Shared uses one common head, PA adds domain-specific output biases, and Dom uses an independent head for each domain. The following centralized width-64 results report test NLL in nats per example as mean ± sample standard deviation over three original data splits. The final column uses paired split differences, with positive values indicating a gain from Dom. These are held-out differences between fitted heads, not estimates of the population gaps.

Dataset and regimeShared NLLPA NLLDom NLLPA minus Dom
OfficeHome, natural0.4557 ± 0.03420.4505 ± 0.03530.4547 ± 0.0337−0.0042 ± 0.0130
OfficeHome, prior 0.30.4995 ± 0.03790.4895 ± 0.03700.5124 ± 0.0636−0.0229 ± 0.0293
VLCS, natural0.3650 ± 0.01390.3621 ± 0.01270.3558 ± 0.0084+0.0063 ± 0.0054
VLCS, prior 0.30.2902 ± 0.05400.2847 ± 0.05300.2913 ± 0.0565−0.0066 ± 0.0104

Additional settings include width 128, stronger prior skew, and DomainNet-common100, and the revision will report all of them in a new appendix. DomainNet has a positive centralized PA-to-Dom gain of 0.0746 ± 0.0053 nats with width 64. The additional results include both gains and losses from domain-specific heads, showing that their achieved benefit varies across datasets and regimes beyond affine prediction. Over all 11 centralized settings, the mean difference favors Dom by more than one standard deviation in 3, favors PA by more than one in 2, and lies within one standard deviation of zero in 6. In all six prior-skewed settings the mean difference is negative, and in four of the five natural settings it is positive.

AuthorsOpenReview ↗

Response to Reviewer xStJ (2/4): End-to-end representation learning

How do you envision relaxing the assumption of a fixed representation Z = f(X)? Would an end-to-end training setup fundamentally break the conditional mutual information guarantees?

End-to-end training does not invalidate the log-loss identity for a specified representation. For any fixed representation ff, setting Z=f(X)Z=f(X) gives the shared and domain-aware Bayes risks H(Y∣Z)H(Y\mid Z) and H(Y∣Z,D)H(Y\mid Z,D), so their difference remains I(Y;D∣Z)I(Y;D\mid Z). This also applies to a realized learned representation when the risk is evaluated on a fresh population draw and the representation is held fixed for the comparison.

One way to apply the framework after a representation-learning stage is to fix the learned representation and assess head sufficiency on it, using independent assessment data with valid bounds conditional on that representation. This would assess whether a learned Shared or PA predictor can be certified as ϵ\epsilon-sufficient for the resulting features.

We ran a first experiment in this direction, comparing fitted heads on learned representations. On VLCS, we fine-tuned the last transformer block, the final layer norm, and the projection of the image encoder jointly with a Shared, PA, or Dom head, under centralized training and on three data splits. This gives three learned encoders per split, one for each head family. We then froze each encoder and refitted Shared, PA, and Dom heads on its features. On each encoder, the three heads are thus compared on the same representation, which is the setting the paper analyzes. The table reports the test NLL of the refitted PA head minus that of the refitted Dom head, as mean ± sample standard deviation over the three splits, in nats per example. Positive values indicate a gain from Dom.

Encoder trained jointly withPA minus Dom test NLL
Shared head0.0017 ± 0.0023
PA head0.0075 ± 0.0047
Dom head0.0233 ± 0.0029

The gain is smallest for the encoder trained with the Shared head and largest for the encoder trained with the Dom head. This illustrates how the achieved benefit of a domain-specific head can vary with the learned representation.

Relaxing the fixed representation involves two further steps. First, the representation may be updated during training while the predictors are still compared on the same realized representation. The identity above then applies at any point of training, but the head objective moves with the representation, so the fixed-feature, strongly convex local-SGD bound does not cover the joint dynamics. A training-quality bound would need to control the representation drift and how closely the heads track it. Second, the competing systems may learn different representations, as the three encoders in the table do. Their total risk difference then combines representation learning and the use of domain context, and it is not characterized by the single conditional mutual information term defined on a common ZZ. Comparing such systems requires additional control of representation-dependent approximation, optimization, and generalization errors. The revision will report the experiment above in a new appendix, will make this scope explicit in Sections 4 and 6, and will retain end-to-end training as an extension beyond the current guarantee.

AuthorsOpenReview ↗

Response to Reviewer xStJ (3/4): Conservativeness of the envelopes

Conservative Bounds: The optimization and recoverability envelopes are quite conservative. As acknowledged by the authors, no numerical tightness is claimed, which limits the immediate numerical applicability of these certificates to real-world edge devices.

We agree, and Section 4 states that no numerical tightness is claimed for these envelopes. Their role is structural. Theorem 6 separates four protocol-dependent error sources, and their coefficients give the conditional guidance of Section 4 (Eq. (27)): changing the number of local steps trades off three of the terms, more clients reduce only the aggregated stochastic noise term, and more rounds shrink only the optimization transient. The controlled audits in Appendix E.3 support the local-step and client-count implications at the component level, in all 20 matched trajectories of each full-participation arm. Conservatism costs stop rate and not safety: when a bound is loose the rule abstains, and abstention is an unresolved sufficiency decision, not evidence that richer capacity is required.

We see two directions for sharpening the envelopes. One is noise and drift controls that are tighter than the uniform bounds used in Theorem 6. The other is to replace the generic Markov conversion in Corollary 7, with its conservative 1/δA1/\delta_A factor, by a sharper high-probability bound under suitable tail assumptions. The revision will state these directions in Section 4.

Reviewer_nyjYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies when simpler models are sufficient for multi-domain federated learning. It uses information-theoretic quantities to characterize the benefit of domain-aware prediction and combines them with federated optimization bounds to determine when shared or prior-aware classifiers are sufficient. The paper also evaluates how well federated training can recover the benefits of domain-aware models.

Note: didn't check the proofs and extra lemma, propositions in the appendix.

Critical strengths
  1. The experiments design is fairly thorough which covers several datasets and federated training methods.
Critical weaknesses

[Problem setup] My primary concern is the necessarity of a separate capacity-certification framework. In the paper's frozen-feature linear-head setting, the three predictor families can be naturally represented within a single domain-conditioned linear model with shared parameters, domain-specific bias offsets, and domain-specific feature interactions. Appropriate structured regularization could continuously control these components. In reality, also, if consider a pretrained frozen feature extractor, a simple domain-aware nonlinear classifier h(z,d)h(z,d) could handle the scenario. So it is unclear to me if this investigation is necessary given the solution is fairly simple and the analysis of it does not provide significant new understanding to the community.

[Theory] Propositions 1, 4, and 5 largely follow from standard bayes properties, KL decompositions, and straightforward risk bound arguments, while Theorem 6 is a fairly standard convergence analysis of local SGD under strong convexity. Overall, the theory mainly serves to formalize the paper's intuition rather than providing substantial and standalone theoretical contribution.

[Empricial Evaluation] Lastly, in reality, is the certificate good enough? I feel like it might be too restrictive under most of cases. Despite this, it is still unclear to me why the certificate proposed in this paper necessary compared to just train a linear classifier and choose the hyperparameter on a validation set? Without further demonstration here that the certificates are practically usable OR meaningfully advantageous over standard model selection, the empirical justification is not unconvincing.

Questions for authors

N/A.

7 comments
AuthorsOpenReview ↗

Response to Reviewer nyjY (1/6): Problem setup

Thank you for the detailed assessment and for recognizing the breadth of the experimental design. Our framework asks when a learned Shared or PA predictor can be certified to leave at most ϵ\epsilon improvement available to any domain-aware predictor on the same representation. Validation among trained candidates does not by itself establish this sufficiency conclusion. Below we explain the theoretical conclusions and present the two estimators suggested in your review, fitted on the same features, together with a direct comparison of the certificate with validation.

Response to [Problem setup]

On "the three predictor families can be naturally represented within a single domain-conditioned linear model". The affine instances of our three predictor families can be expressed within the unified model you describe, with logits Wz+b+bd+VdzWz + b + b_d + V_d z. Eq. (21) already gives the Shared and PA cases of this parameterization. The question we study is what can be concluded about the value of its domain-dependent components: when domain context adds predictive value on a common representation, and when a learned Shared or PA predictor is already sufficient. The three levels of the hierarchy distinguish prediction without domain context, prediction with domain-specific class offsets, and unrestricted domain-dependent prediction. Their reference gaps, together with valid training margins, give sufficient conditions under which further domain-aware capacity can improve risk by at most a specified tolerance.

On "the analysis of it does not provide significant new understanding". Domain heterogeneity alone does not determine whether domain-specific feature interactions can improve prediction, whether prior offsets already capture their statistical value, or whether a given federated training protocol can realize an available gain. Our population analysis makes the distinctions explicit. Under log-loss, the value of observing the domain beyond the representation is I(Y;D∣Z)I(Y;D\mid Z). The residual after canonical prior correction is JPAJ_{\mathrm{PA}}. Under pure prior shift, ZZ is conditionally independent of DD given YY, so the canonical PA predictor attains the domain-aware Bayes risk even though the shared-to-domain gap can remain positive. Thus a heterogeneous federation can benefit from domain information while requiring no domain-dependent feature map. Conversely, when JPA>0J_{\mathrm{PA}} > 0, the domain-aware Bayes predictor improves on the canonical PA predictor by exactly JPAJ_{\mathrm{PA}} in population log-loss. The controlled grid of Appendix D.1 gives two exact cases (the revision will add a table of all 29 populations). In a prior-only population, I(Y;D∣Z)I(Y;D\mid Z) is 0.170 nats and JPA=0J_{\mathrm{PA}} = 0. In a nuisance-shift population, the class-conditional dependence GcondG_{\mathrm{cond}} is 0.368 nats while both reference gaps are zero, so a large feature shift across domains does not by itself imply a benefit from domain-aware prediction.

AuthorsOpenReview ↗

Response to Reviewer nyjY (2/6): Problem setup, continued

On "Appropriate structured regularization could continuously control these components". Structured regularization provides a natural way to control the domain-dependent components of the unified model, and we have implemented this approach. A penalty on VdV_d changes the estimator, not the model family. We trained the unified affine model with a penalty on VdV_d centrally on the same frozen features, with the penalty strength and the learning rate selected on the validation data of one split and then fixed. The table reports test NLL in nats per example as the mean over three data splits, next to the affine PA and Dom heads trained with the protocol of the submission.

Dataset and regimePADomRegularized unified model
OfficeHome, natural0.45890.45540.4148
OfficeHome, prior 0.30.52650.58230.4515
VLCS, natural0.36060.34660.3390
VLCS, prior 0.30.30340.30050.2739
DomainNet-common100, natural0.72420.62720.6272

The regularized model has the lowest NLL in the first four settings. On DomainNet the selected penalty is zero, so it coincides with the Dom head. The PA and Dom heads in this table use the fixed learning rate 0.001 of the submission, while the regularized model selects 0.003 on OfficeHome and VLCS, so the differences reflect the learning-rate selection as well as the penalty. The model is a practical implementation of the hierarchy. Fitting it and choosing the penalty on held-out data compares the fitted candidates with each other. Even when validation favors a Shared or PA fit, this comparison alone does not establish that further domain-aware prediction can improve its population risk by at most the target tolerance. The fitted domain-aware candidates may themselves fall short of the unrestricted reference. The OfficeHome prior 0.3 row shows how much a fitted comparison depends on the estimator: the Dom head is 0.056 nats above PA, while the regularized model, which belongs to the same domain-aware family, is 0.075 nats below PA. Our response to [Empirical Evaluation] gives a controlled comparison that includes cases where the predictor selected by confidence-aware validation has risk within the tolerance of the best candidate's risk, yet more than the tolerance above the domain-aware Bayes risk. The identities quantify the value of unrestricted domain-aware prediction relative to the shared Bayes and canonical PA references, and the training margin links these references to a fitted estimator.

AuthorsOpenReview ↗

Response to Reviewer nyjY (3/6): Problem setup, continued

On "a simple domain-aware nonlinear classifier h(z,d)h(z,d) could handle the scenario". Our population-level domain-aware reference already includes nonlinear classifiers h(z,d)h(z,d), and the Shared score map in our PA definition is also allowed to be nonlinear. The affine restriction enters the explicit federated optimization bound. We evaluated two explicit nonlinear instantiations of h(z,d)h(z,d) on the same frozen features, with one-hidden-layer ReLU networks of width 64: an independent network for each observed domain (Dom), and the form you describe, a shared network that receives the domain one-hot together with zz. Both are compared with a shared network that has domain-specific output biases (PA). The table reports centralized test NLL in nats per example as the mean over three data splits, and the paired difference PA minus Dom as mean ± sample standard deviation, with positive values favoring Dom. These are held-out differences between fitted heads, not estimates of the population gaps.

Dataset and regimePA NLLDom NLLPA minus DomDomain-input NLL
OfficeHome, natural0.45050.4547−0.0042 ± 0.01300.4515
OfficeHome, prior 0.30.48950.5124−0.0229 ± 0.02930.4867
VLCS, natural0.36210.3558+0.0063 ± 0.00540.3566
VLCS, prior 0.30.28470.2913−0.0066 ± 0.01040.2810
DomainNet-common100, natural0.76370.6891+0.0746 ± 0.00530.7619

The mean NLL of the domain-input network (last column) is within 0.006 nats of PA in all five settings. Relative to the independent per-domain networks it is better in 7 and worse in 8 of the 15 split-level comparisons, and on DomainNet it is worse in all three splits, by 0.073 nats on average. The two models differ in parameter count. For the per-domain networks, over all 11 centralized settings, which add width 128 and a stronger prior skew, the mean difference favors Dom by more than one standard deviation in 3 settings, favors PA by more than one in 2, and lies within one standard deviation of zero in 6. The per-domain head has higher mean NLL than PA in all six prior-skewed settings. The fitted result of a domain-aware nonlinear classifier thus depends on how the domain enters the model and on the setting. Which case applies to a given federation is the question the framework addresses. The revision will report all 11 settings and both additional models in a new appendix, and its Section 3.1 will present the unified parameterization and regularization-based selection as an implementation of the hierarchy.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised Paper Uploaded: Summary of Changes and Request for Reviewer Feedback

Dear Area Chair and Reviewers,

Thank you again for the time and effort dedicated to reviewing our work. Following the individual responses we posted earlier, we have now uploaded the revised manuscript, which incorporates all the promised clarifications and additional experiments.

With the discussion period concluding on November 18 (AoE), we would be very grateful to hear your thoughts on our responses and the revised manuscript. Please let us know if our revisions have adequately addressed your concerns, or if any questions remain. We are happy to provide any further clarification promptly. If you find that your concerns have been resolved, we kindly invite you to re-evaluate our work accordingly.

Summary of the revision. Changes in the main text and the appendix are marked in blue. The submitted results and their numbering are unchanged, so the references in the reviews and in our responses still apply.

Main text

  • Section 1. The contribution list states the four conclusions given in our response to Reviewer nyjY on the theory, and the introduction states the scope of the analysis on a fixed representation (Reviewer aaEr).
  • Section 3.1. The unified parameterization of the three affine classes and regularization-based selection are presented as an implementation of the hierarchy (Reviewer nyjY).
  • Section 3.3. The three levels are described as the reference structures studied here, with an explanation of how other explicitly defined families, such as grouped heads, are assessed (Reviewer aaEr).
  • Section 4. The analytic route and the calibration-based route are stated separately, together with the additional terms that the analytic route needs under partial participation, two directions for sharpening the envelopes, and the scope of each result (Reviewer xStJ).
  • Section 5. Pointers to the new experiments.
  • Section 6. The conclusion states which results hold for any head on a fixed representation and which are specific to affine heads under full-participation local SGD (Reviewers xStJ and aaEr).

Appendix

  • Appendix D.1, Table 2a. The exact population quantities of all 29 populations of the controlled grid (Reviewer nyjY).
  • Appendix D.2, new material at the end, Tables 3a to 3c. Table 3a gives the certificate results without the output floor of Eq. (221), which are the ones used in our response, next to the totals of the submitted Table 3. Table 3b compares the certificate with ordinary and confidence-aware validation on the same training and calibration samples, and Table 3c reports the variant in which only the domain-aware candidate is degraded (Reviewer nyjY).
  • Appendix G, Tables 14 to 17. One-hidden-layer ReLU heads in all 11 settings, with absolute NLLs and split-level values, the regularized unified affine model, and the shared network that receives the domain as an additional input (Reviewers nyjY, xStJ, and aaEr).
  • Appendix H, Proposition 30 and Tables 18 and 19. The extension of the reference gaps and certificates to fixed and nested groupings of the observed domains, and grouped affine heads on OfficeHome and VLCS, with and without per-domain offsets, with split-level values (Reviewer aaEr).
  • Appendix I, Table 20. The experiment in which part of the image encoder is trained jointly with each head family on VLCS, followed by a comparison of the three heads on each learned representation (Reviewers xStJ and aaEr).

Thank you for your time and consideration.

AuthorsOpenReview ↗

Concise summary of our responses

Dear Area Chair and Reviewers,

Our individual responses are long, so here is a short version of the main answers. The details and evidence are in the individual responses and the revised paper.

1. Framework and theory (Reviewer nyjY). We implemented the suggested regularized affine model and domain-input network (Tables 15 and 16). Our framework can certify that a learned Shared or PA predictor leaves at most ϵ\epsilon improvement available to any domain-aware predictor on the same representation. Validation compares trained candidates and does not by itself control their gap to the domain-aware Bayes risk. Under the client-refined factorization assumption, Proposition 8 links the distributional diagnostics to the local-SGD client-drift bound.

2. Certificates (Reviewer nyjY). In the finite-support setting of Table 3c, the certificate stops in 1,400 of the 1,500 decisions where a trained Shared or PA predictor is ϵ\epsilon-sufficient, with no observed false-safe decision. When only the Dom candidate is degraded, confidence-aware validation still meets its own target but selects a predictor more than ϵ\epsilon above the domain-aware Bayes risk in up to 1,500 decisions, while the certificate's decisions do not change. Once the required bounds are available, certification can justify stopping without fitting richer candidates.

3. Intermediate heads (Reviewer aaEr). Proposition 30 extends reference gaps and one-sided certificates to fixed and nested domain groupings. A Shared or PA certificate also bounds the gain of every intermediate family. Adding per-domain offsets to a grouped head lowers its test NLL in 34 of 36 matched split-level comparisons (Tables 18 and 19).

4. Nonlinear heads and learned representations (Reviewers xStJ and aaEr). The population reference gaps do not require affine heads. We added 11 ReLU-head settings on frozen features and a VLCS partial fine-tuning experiment showing that the achieved domain-aware gain varies across learned representations (Tables 14, 17, and 20). The log-loss identity remains valid on each fixed learned representation. Section 4 states the additional controls needed for a fully joint training guarantee.

5. Bounds and participation (Reviewer xStJ). Conservatism costs stop rate, not safety: when a bound is loose, the rule abstains, and abstention is not evidence that richer capacity is required. In the finite-support, bounded-loss setting of Appendix D.2, the validity of the calibration-based certificate does not depend on the training participation pattern, including partial and stochastic participation, provided the predictor is fixed independently of the calibration data drawn from the target distribution.

The discussion period ends on November 18 (AoE). We would be grateful to hear whether these answers address your main concerns, and we will reply promptly to any remaining question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.