- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a novel method MESSI, a data-driven Domain Generalization (DG) Mixture-of-Experts (MoE) that replaces global invariance with subset-shared invariance.
Compared with the existing state-of-the-art method GMoE, which tries to learn specialized feature representations, such as texture and shape, this paper performs data-driven selective source-domain alignment at the subset level. This method also surpasses GMoE in performance.
The paper also provides a theoretical analysis of the information loss caused by enforcing globally invariant representations across all domains, and discusses the necessity of subset-shared invariance.
However, I am concerned that whether such a data-driven mechanism may fail when the source and target gap is large; the paper exhibits the same limitation, as its performance degrades when the number of source domains is limited (e.g., ks=3 in table 14).
Critical strengths
Strength1:The paper provides a theoretical analysis of the trade-off between domain invariance and information loss, showing that enforcing global conditional invariance can shrink the feasible representation class. It then proposes a novel principle, subset-shared invariance, to preserve such partially shared predictive structure.
Strength2: The authors realize this principle with a routing-based Mixture-of-Experts (MoE) framework that performs data-driven subset alignment without explicit subset labels, and empirically surpasses GMoE and other baselines under increasing domain heterogeneity.
These are the primary reasons for my score.
Critical weaknesses
Weaknesses1: This data-driven MoE alignment may not constitute true domain generalization; it appears closer to training multiple domain-specific experts and then matching inputs to the appropriate expert via data statistics. Compared with GMoE, which learns potentially transferable representations, the latter may generalize better when the data gap is large.
Weaknesses2:The performance gain over GMoE is marginal. As shown in Table 1, MESSI-S-MMD (64.3) is only 0.1% higher than GMoE (64.2). A more noticeable improvement (66.2) is achieved only when replacing Maximum Mean Discrepancy (MMD) with Optimal Transport (OT), which is about 6× more expensive than GMoE, while MMD is still about 2× more expensive (Table 5). This suggests that the method may be unstable and that its gains depend on a precise alignment metric, raising concerns about its practical value and robustness.
Questions for authors
Could the authors design an experiment where the target domain has a large discrepancy from all source domains, and report how MESSI behaves in that setting?
3 comments
Q2. Marginal performance gains, alignment-metric dependence, and stability
A2: We agree that MESSI-S-MMD is only slightly ahead of GMoE on the DomainBed average (64.3 vs. 64.2). However, our claimed contribution is the way routing decides which domain pairs to align. A fair test therefore keeps the distance measure fixed and changes only how the pairs are weighted. Global-OT is the Global-MoE control from Table 4, and Global-MMD is a new run.
Table F: Controlled alignment ablation on PACS and OfficeHome (accuracy, %).
| Method | PACS ↑ | OfficeHome ↑ |
|---|---|---|
| GMoE | 87.3 0.1 | 73.5 0.1 |
| Global-MMD | 88.4 0.5 | 73.2 0.4 |
| MESSI-S-MMD | 89.3 0.6 | 74.0 0.3 |
| Global-OT | 87.1 0.2 | 73.8 0.6 |
| MESSI-S-OT | 90.9 0.7 | 76.1 0.2 |
With the distance measure held fixed, routing-based alignment beats uniform alignment on both PACS and OfficeHome. The margin is 3.8 and 2.3 points with OT, and 0.9 and 0.8 points with MMD. The gain therefore does not come only from replacing MMD with OT, although OT gives the larger improvement in our setting.
The results are also stable. MESSI-S-OT beats GMoE on all four DomainBed datasets, with seed standard deviations between 0.2 and 0.7. In the sensitivity study of App. E.2, every setting that keeps the routing sparsity term stays above GMoE's PACS accuracy of 87.3, with the lowest at 88.8. This holds from 2 to 16 experts and across the full range of routing weights we tried. Halving the number of Sinkhorn iterations also lowers the average accuracy only from 66.2 to 66.0.
Our response to Reviewer 4cfG (W3) gives the full timing. Compared with GMoE, training takes about 2.1 times as long with MMD and 6.1 times as long with OT. Using 25 Sinkhorn iterations brings OT down to 3.3 times while keeping 66.0 average accuracy. The extra cost applies only during training, and at test time MESSI is faster than GMoE (61.4 vs. 74.2 in Table 5).
Finally, each DomainBed dataset has only four to six domains, which limits how much partial sharing between domains can matter. On iWildCam, with 243 source cameras, MESSI reaches 33.2 OOD macro-F1 against 26.7 for GMoE. The gap widens as more source cameras are added (Table A in our response to Reviewer b71J, W3). We will add Table F, the results with fewer Sinkhorn iterations, and a timing row for ERM with the same backbone to the revised paper.
We sincerely thank you for the time and effort you have devoted to reviewing our work. We would greatly appreciate your thoughts on our response and the additional results, particularly whether any concerns still need clarification. We would be happy to discuss any remaining questions and would value your further feedback as we refine the manuscript.
Response to the Concern Raised in the Final Paragraph of the Paper Summary: Robustness to Limited Source Diversity and Large Source–Target Gaps.
We thank the reviewer for raising this concern. We would like to clarify that the low accuracy at Ks=3 in Table 14 is not specific to MESSI. As reported in Appendix D.3, all evaluated methods, including CORAL across all penalty strengths, perform near chance level (approximately 50.8%). With only three source environments, the available diversity may be insufficient to distinguish digit-shape information from spurious color correlations. Notably, when Ks increases to 5, MESSI is the only method that substantially exceeds chance-level accuracy (57.1% versus at most 50.9% for the baselines), and it maintains the highest accuracy across all larger source-domain settings. These results suggest that while limited source diversity remains challenging, MESSI benefits effectively from additional source environments.
Q1. Domain-specific expert specialization versus transferable invariance
A1: MESSI’s experts are trained to be shared across domains. The router never sees domain labels (Sec. 4), and the alignment loss (Eq. 20) pulls together each expert’s features for the same class across the source domains that use it. An expert that behaves differently across domains is therefore penalized, whereas GMoE is trained with classification alone and has no such constraint. Fig. 4 confirms this sharing, since each source domain spreads its inputs over several experts (at most 48% on any one) and different domains rely on the same experts. Although each input goes mostly to one expert (Table 17), that expert is trained to be consistent across all the domains it serves. Our response to Reviewer b71J (W2) shows that this alignment, rather than the MoE head, drives the gain.
Behavior under large shifts. If MESSI only matched inputs to domain-specific experts, its advantage would shrink on distant targets. It grows instead. On the hardest target of each DomainBed dataset (lowest GMoE accuracy, App. D.2), MESSI-S-OT beats GMoE by 7.4 points on PACS-Sketch, 3.2 on OfficeHome-Clipart, 2.1 on TerraInc-L38 and 0.9 on DomainNet-Quickdraw. On iWildCam, the gain is larger on unseen cameras (+6.5 F1) than on seen ones (+3.9).
To answer your question directly, we evaluate targets increasingly far from all sources. All methods share the DeiT-S/16 backbone and are trained once on the same Rotated-Colored MNIST sources (App. C.2), with the five targets held out.
Table E. Rotated-Colored MNIST, OOD accuracy (%), mean ± std over 3 seeds. Min. PAD is the smallest proxy A-distance between the target and any source environment, so a larger value means a target further from all sources.
MESSI scores higher than the other methods on all five targets, and the gap to GMoE grows as the target moves further from the sources. When the link between color and label is reversed, the gap goes from 3.6 to 15.7 points, and under rotations not seen in training it is 4.6 and 6.7 points. We will add Table E to App. D and the expert-sharing statistics to App. E.4. In Sec. 2, we will also explain that GMoE gets its transferability from its architecture, while MESSI asks for it directly through alignment.