- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a parameter-efficient Mixture-of-Experts framework that uses Hurwitz–Radon (HR) transformations to provide experts with structured, orthogonal views of the input. the authors name method as STODA,. Each expert applies a fixed HR transform, a learnable Tensor-Train core, and an inverse transformation. The paper develops theoretical results on orthogonality, non-degenerate mixing, and resistance to expert collapse, and evaluates STODA on diffusion-model personalization, style adaptation, preference alignment, and additional language-model experiments. Empirically, STODA is competitive with or improves over several PEFT baselines while using fewer active parameters.
Critical strengths
The use of HR structure to induce expert diversity is well motivated. In particular, the pointwise orthogonality and mixing properties provide a principled alternative to encouraging diversity indirectly through load-balancing objectives.
The paper includes useful ablations of the proposed mechanism. Comparisons with random orthogonal transformations, sign flips, and identity suggest that the HR structure contributes to reduced expert similarity and improved performance.
Critical weaknesses
My main concern is the gap between the theory and the actual architecture. The anti-collapse theorem is derived for a shared linear core, while the implemented method uses nonlinear Tensor-Train cores. For the nonlinear setting, the connection is made through local Jacobian linearization rather than a global anti-collapse result. Similarly, the clean cross-term result assumes identical core perturbations, while independently learned experts need not satisfy this assumption. Consequently, the strongest theoretical claims do not directly characterize the model evaluated experimentally. This is a major drawback in my opinion.
The empirical evidence is not sufficiently robust to fully compensate for this gap. Most experiments are single runs, with multi-seed evaluation reported only for one setting. Some broader claims about general applicability are also supported by relatively limited experiments, e.g., the small-scale pretraining experiment and a single LLM instruction-tuning setup.
Questions for authors
-
Can the authors provide stronger theoretical or empirical evidence that the anti-collapse result meaningfully extends from the shared linear-core setting to the nonlinear Tensor-Train experts used in practice?
-
Can the authors directly measure the proposed cross-term suppression for independently learned experts and compare it against random orthogonal views?
-
Are the main improvements over LoRA and the strongest MoE-PEFT baselines consistent across multiple random seeds?
6 comments
Part 2/4
W2. The empirical evidence is not sufficiently robust to fully compensate for this gap. Most experiments are single runs, with multi-seed evaluation reported only for one setting.
We agree. Earlier, we had multiple seeds for only one setting. We have now run the view ablation on two backbones and two datasets with three seeds each. The FID results (mean std) are below, and the full tables with CLIP and expert similarity are in our response to Rev. ATFW's W2.
| Setting | HR (STODA) | Random orthogonal | Random sign flips | Identity |
|---|---|---|---|---|
| Barbie, PixArt- | ||||
| Barbie, Sana | ||||
| Cyberpunk, PixArt- | ||||
| Cyberpunk, Sana |
HR views have a clear advantage over identity map and random sign flips in all four settings (2.2 to 8.8 FID), which is beyond the seed variation. The advantage over random orthogonal views is smaller (ranging 0.95 to 2.70 FID), although it is in the same direction in all four settings. For comparison with LoRA and the MoE-PEFT baselines, please see the Q3 answer. Thank you.
W3. Some broader claims about general applicability are also supported by relatively limited experiments, e.g., the small-scale pretraining experiment and a single LLM instruction-tuning setup.
Yes, the pre-training and language-model experiments are small in scale. While the concepts of STODA can be generally applied to MoE architectures, we want to keep the focus on diffusion model fine-tuning in the current paper. Hence, most of our evaluations are focused on image generation tasks. To show that the construction is not tied to fine-tuning image generation models, we included a small-scale DiT pre-training experiment on MNIST (App. F) and LLM instruction tuning of Qwen-3 4B on Alpaca. Due to compute budget limits, we limit the experiments in these settings, and we have revised the text to say that these are preliminary evidence.
Q1 Can the authors provide stronger theoretical or empirical evidence that the anti-collapse result meaningfully extends from the shared linear-core setting to the nonlinear Tensor-Train experts used in practice?
Yes, we can do both! On the theory side, the tensor-train experts used in practice are linear (please see the code in W1 above), so Thm. 4.3 applies to them directly. For independently learned cores, we have for every core that expert could learn, where is the relative distance from the rotated core to the set of bond-2 tensor trains. Thm. 4.3 is the special case of equal cores.
On the experimental side, we computed for all pairs of experts in our trained PixArt- model. The median is 0.979 between routed experts and 0.933 between a routed expert and the shared expert, and training does not reduce it (0.977 at initialization). The measured distance between trained experts has a median of 1.414, which is the value for two orthogonal maps of equal norm. In addition, with three seeds on two backbones and two datasets, HR views are ahead of the identity map by 2.2 to 8.8 FID (W2 above).
Part 3/4
Q2. Can the authors directly measure the proposed cross-term suppression for independently learned experts and compare it against random orthogonal views?
Yes. Because the experts are linear, the cross-term can be computed exactly from the trained weights. For expert , let , where is the trace-free part of its core, and let . For independently learned experts, this quantity is expected to be close to zero for any choice of views. If and point in independent directions in the -dimensional space of trace-free matrices, then and , so , with median at . Our measurements match this chance level. The suppression in Prop. A.1 concerns correlated perturbations. For identical perturbations, the normalized cross-term equals ; for our learned HR cores the median is at most , compared with exactly 1 for the identity map. The Frobenius cross-term weights all input directions equally, whereas real tokens are strongly anisotropic. The interaction that matters for the model is therefore the one on real tokens, which weights the experts' maps by the token covariance : . Here the views do separate. The table reports the mean cosine similarity between the outputs of two experts on real evaluation tokens, across two backbones, two datasets, and three seeds (s.d. is below 0.0005 in every cell). For reference, independently initialized experts give 0.00094.
| Setting | HR | Identity | Random sign flips | Random orthogonal |
|---|---|---|---|---|
| Barbie, PixArt- | 0.0033 | 0.0059 | 0.0057 | 0.0091 |
| Barbie, Sana | 0.0031 | 0.0060 | 0.0056 | 0.0087 |
| Cyberpunk, PixArt- | 0.0033 | 0.0059 | 0.0056 | 0.0088 |
| Cyberpunk, Sana | 0.0032 | 0.0056 | 0.0057 | 0.0090 |
On real tokens, the interaction is smallest with HR views in all four settings. Moreover, HR views achieve this with higher throughput than random orthogonal views. We do not have a proof that learned cores are balanced between the two eigenspaces; The revised App. A states this as an assumption supported by these measurements.
Official comment. Part 1/4
Yes, Thm. 4.3 is for a shared linear core. Your comment has three parts: (a) linear vs. nonlinear cores, (b) a shared core vs. independently learned cores, and (c) the cross-term result. We clarify each below.
For (a), the model evaluated in our fine-tuning experiments is linear. We described the framework in Sec. 3 with a general core ϕi simply because the view-core-unview construction does not depend on the form of the core. In all fine-tuning experiments in the paper, the core is the two-factor tensor train, which is a linear map in x (only exception is the small-scale DiT pre-training in Appendix F, where the experts are MLPs). We can check this quickly in the code where the expert forward pass is
So each routed expert is exactly Ei(x)=Hi⊤AiHix, where Ai=∑r=12Br(i)⊗Cr(i) is the tensor-train core. So the linear-core assumption of Thm. 4.3 holds exactly for the evaluated model, and the local Jacobian linearization in lines 255-259 is not needed. Since this created confusion, we have revised that paragraph to make this clear.
For (b), yes. Thm. 4.3 is written for a shared core because this is the extreme case for collapse. In a standard MoE, experts with identical parameters are identical functions, while Thm. 4.3 shows that with HR views they cannot all coincide unless the core lies in a subspace of relative dimension at most 1/64 (for s=7). For independently learned cores, the same argument gives us a statement that holds for every pair of cores. Let Rij=HjHi⊤. Since {Hj} is orthogonal, ∥Ei−Ej∥F=∥RijAiRij⊤−Aj∥F. So expert j can reproduce expert i only if its core equals the rotated core RijAiRij⊤. And this rotated core is in general far from a tensor train of bond dimension 2. Let ρij be the relative distance from the rotated core to the set of bond-2 tensor trains, which is given exactly by one SVD. Then, for any core that expert j can learn,
∥Ei−Ej∥F≥ρij∥Ei∥F.
This bound is the independent-core counterpart of Thm. 4.3, which characterizes the equal-core case exactly. On our trained PixArt-Σ model, the median ρij over pairs of routed experts is 0.979 (mean 0.941), and training does not reduce it (0.977 at initialization). So the bound pertains to the trained, independently parameterized experts directly. Full table and the cases in which the bound is zero are in our response to Rev. ATFW's W1(a). We would be glad to add this as an extension of Thm. 4.3 in the appendix.
For (c), we agree that independently learned experts need not satisfy the identical-perturbation assumption of Prop. A.1(iii). For this reason, we measure the interaction between independently learned experts directly. Please see Q2 below.