- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when multi-domain FL needs domain-aware capacity. It derives information-theoretic analysis under fixed representation scenarios, showing that shared or prior-aware heads may be sufficient when domain differences mainly reflect class-prior shifts.
Critical strengths
-
Clear separation between statistical value of domain-aware capacity and federated recoverability.
-
Elegant theoretical analysis, and the experiment results well aligned to the theory.
Critical weaknesses
-
The capacity hierarchy is too coarse. For example, the evolving lines may contain: Shared → Prior-aware → Clustered / grouped heads → Routed / mixture heads → Full domain-aware
-
The key limitation is the fixed-representation assumption: the choice of head family can change the learned representation, and in FL, we train the whole model in most cases.
-
The experiments are mostly frozen-feature affine-head studies, which are quite simplified for practical scenarios.
Questions for authors
- Can the framework handle grouped or clustered domain heads?
5 comments
Response to Reviewer aaEr (2/4): Grouped and clustered heads
Can the framework handle grouped or clustered domain heads?
Yes, for a fixed grouping of the observed domains there is a direct population extension. Let be a prespecified group label, and let the grouped family contain all measurable predictors of . Its Bayes risk is . Because is a deterministic function of , the remaining gap to full domain-aware prediction is
The shared-to-domain gap correspondingly decomposes as
This separates the value captured by knowing the group from the domain information that remains useful within groups. If a learned grouped predictor satisfies a valid bound , it has the sufficient condition
As with the original rule, both terms require valid upper controls for finite-sample use. A finite affine or neural grouped head contributes approximation and training error to its margin, and Theorem 6 is stated for Shared and PA heads, so the margin of a grouped head must be supplied separately. If the grouping is learned or selected from data, the guarantee must additionally account for that selection or use an appropriate independent assessment. The revision will include this fixed-group extension as a new proposition.
We have also completed a fixed-group empirical study on OfficeHome and VLCS, using all three balanced two-group partitions of the four domains under the original domain ordering: AB/CD (g1), AC/BD (g2), and AD/BC (g3). The groups were fixed before evaluating test performance, and all three are reported. We fitted two grouped families: a pure grouped head, identical across the domains of a group, and a grouped head with per-domain class offsets, the specification that can contain PA. The table gives the held-out NLL of the fitted affine heads under centralized training with the protocol of the submission, as the mean over three data splits. These are results for fitted heads under one training protocol. The grouped reference identities above concern the pure grouped family.
| Dataset and regime | Shared | PA | Grouped, g1 / g2 / g3 | Grouped with per-domain offsets, g1 / g2 / g3 | Dom |
|---|---|---|---|---|---|
| OfficeHome, natural | 0.4651 | 0.4589 | 0.4556 / 0.4627 / 0.4564 | 0.4524 / 0.4595 / 0.4543 | 0.4554 |
| OfficeHome, prior 0.3 | 0.5375 | 0.5265 | 0.5505 / 0.5554 / 0.5523 | 0.5457 / 0.5508 / 0.5481 | 0.5823 |
| VLCS, natural | 0.3687 | 0.3606 | 0.3459 / 0.3630 / 0.3563 | 0.3456 / 0.3603 / 0.3537 | 0.3466 |
| VLCS, prior 0.3 | 0.3161 | 0.3034 | 0.2977 / 0.3034 / 0.3118 | 0.2960 / 0.3027 / 0.3045 | 0.3005 |
Adding the per-domain offsets lowers the NLL of the matched pure grouped head in 34 of the 36 split-level comparisons. How the grouped head with offsets compares with PA and Dom depends on the setting. On OfficeHome with prior 0.3, it is worse than PA and better than Dom in all nine comparisons, so along the nested PA, grouped-with-offsets, and Dom families, more capacity gives a higher fitted test NLL. In the other three settings, it is better than PA in 21 of 27 comparisons and better than Dom in 11 of 27. The revision will report these results, with the split-level values, in the new appendix on grouped heads.
Response to Reviewer aaEr (3/4): Fixed-representation assumption
The key limitation is the fixed-representation assumption: the choice of head family can change the learned representation, and in FL, we train the whole model in most cases.
We agree that the head family can affect representation learning during end-to-end training. The current analysis intentionally compares the use of domain context on a common representation. This makes it possible to isolate the statistical value of domain-dependent prediction from changes in the information encoded by the features. It applies directly when a pretrained or otherwise fixed feature map is used, but does not characterize all benefits that domain-aware capacity may produce through joint representation learning.
For a specified representation, the identity remains valid. If different head choices learn different representations, the comparison involves different random variables and additional representation-dependent errors. A single conditional mutual information term on one common representation no longer describes that complete comparison. The explicit affine local-SGD bound also relies on a fixed head optimization problem and does not cover the moving representation in joint training.
The framework can be applied after representation learning by freezing the learned representation and comparing heads on it, and we ran a first experiment of this kind. On VLCS we fine-tuned the last transformer block, the final layer norm, and the projection of the image encoder jointly with a Shared, PA, or Dom head under centralized training, on three data splits. The jointly trained Shared, PA, and Dom systems have test NLL 0.3052, 0.3077, and 0.2956 nats, and the Dom system is below the PA system in all three splits. We then froze each learned encoder and refitted the three heads on it. The table reports the test NLL of the refitted PA head minus that of the refitted Dom head, as mean ± sample standard deviation over the three splits, in nats per example. Positive values indicate a gain from Dom.
| Encoder trained jointly with | PA minus Dom test NLL |
|---|---|
| Shared head | 0.0017 ± 0.0023 |
| PA head | 0.0075 ± 0.0047 |
| Dom head | 0.0233 ± 0.0029 |
In this experiment the head family used during training changes the head-level gain measured on the resulting representation, consistent with the effect you describe, and on each fixed representation the comparison is the one the paper analyzes. A guarantee for fully joint training would need a training-quality bound that controls the representation drift and how closely the heads track it under federated updates. Comparing systems that learn different representations would additionally require control of representation-dependent approximation, optimization, and generalization errors. The revision will report this experiment in a new appendix and will strengthen the scope statement in the introduction and conclusions accordingly.
Response to Reviewer aaEr (1/4): Capacity hierarchy
Thank you for highlighting the distinction between statistical value and federated recoverability. We agree that a broader range of head-sharing structures is relevant. Below we explain the purpose of the three reference levels, give an explicit fixed-group extension with experiments, and clarify the scope of the additional nonlinear experiments and the fixed-representation assumption, including a first experiment in which part of the encoder is trained jointly with each head family.
We agree that the three reference levels do not exhaust the useful sharing structures. They were chosen to distinguish three specific uses of the observed context: no context dependence, context-dependent class offsets with a shared feature-to-score map, and unrestricted context-dependent prediction. The PA level is important because it separates the improvement captured by canonical prior correction from the residual relative to unrestricted domain-aware prediction. Without this intermediate reference, those quantities would be combined in the Shared-to-Dom comparison. When a Shared or PA predictor is certified as ϵ-sufficient, the comparison is with unrestricted domain-aware prediction, so the same tolerance also bounds the gain of every intermediate family that uses the same representation and observed context.
The hierarchy can be refined exactly. Let G1,…,Gk=D be nested groupings of the observed domains, each a deterministic function of the next, and let G0 be constant. The chain rule gives
I(Y;D∣Z)=∑j=1kI(Y;Gj∣Z,Gj−1),
and the level that uses Gj has the exact reference gap I(Y;D∣Z,Gj) to full domain-aware prediction. Every intermediate grouping level therefore has its own reference gap and its own one-sided certificate. The revision will state this as a new proposition, and our answer on grouped heads below gives the case k=2 with experiments.
Grouped heads fit into the hierarchy once it is specified what the domains in a group share. A pure grouped head uses G=g(D) and is identical across domains assigned to the same group. It need not contain PA, since PA permits different class offsets for domains within a group. Conversely, a grouped head can use a different feature-to-score map in each group, which PA cannot. So in general neither family contains the other. Adding per-domain class offsets gives a grouped family that contains PA and is contained in Dom under compatible base score classes. Our answer on grouped heads below evaluates both specifications.
Routed or mixture heads combine several expert heads through a gate, and the gate determines how they are handled. A fixed hard gate that depends only on the domain gives a grouped head, so the grouped analysis applies. For a fixed soft gate G=g(Z,D) with experts acting on Z, all measurable predictors of (Z,G) provide an unrestricted reference family. The mixture is contained in this family, and its approximation and training errors enter the training margin. For a learned gate, the margin would also need to account for gate learning and selection. Routed WA in our submitted experiments is different: it changes how domain blocks are aggregated while retaining the domain-specific prediction architecture. It is therefore a training protocol for that architecture, rather than an additional intermediate capacity level.
The revision will describe the three levels as the reference structures studied here and explain how additional explicitly defined families are assessed.