- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a framework combining mixture of experts and shared experts, along with a theoretical analysis to try to prove this architecture has collaborative mechanism that allows the two to correct each other. Experiments are extensive, on 7 datasets and multiple baselines.
Critical strengths
- The experiment amount is big, on multiple baselines and datasets across different tasks.
- The theoretical analysis makes effort to demonstrate the generalization ability of combining shared experts and MoE, though there are some flaws in the analysis.
- Figures are clear to understand.
Critical weaknesses
-
The theoretical analysis is right but has great limitations, section 2.1 is the same as [1], proving that ERM has the ability to learn generaliziable neural networks if the Theorem 1 (a) branches holds. However, in the following analysis, you do not discuss when the model aligns with invariant correlations. Noting that in [1], only one of Theorem 1 (a) and (b) holds, not both, but in Assumption 4, you didn't care about which correlations the S or R aligns. Furthuremore, Theorem 3 shows the alignment errors of the sum(+) in epoch (t) is smaller than epoch 0, but this does not support the architecture has the ability to avoid over-reduction and under-reduction. For example, S or R aligns with spurious correlations, or whether the model relies heavily on shared experts—a situation that does indeed occur. Thus, the conclusion of line 145-149 is too strong and can not directly been drawn from the analysis.
-
I appreciate the effort of Table 4, but it can sitll not answer the flaws in the theoretical analysis. I only say one: since you use in the shared branch, which can be regarded as a kind of regularization loss, the Gap of shared branch in Table 4 is samller than the gap of routed branch can only prove the effectiveness of the regularization.
-
The qualitative experiments are not enough. And some validation experiments are not conducted like in [1], for example, in [1], the author do some validation experiments on toy datasets to show that sometimes neural networks align with spurious correlations, sometimes with invariant correlations.
[1] Li B, Shen Y, Yang J, et al. Sparse mixture-of-experts are domain generalizable learners, ICLR 2023.
Questions for authors
See above.