acceptodds
Under review as a conference paper at ICLR 2027

STODA: Structured Tiling via Orthogonal Design Adapters

Abstract

Modern foundation models increasingly rely on functional decomposition, representing complex mappings as mixtures of simpler primitives. Sparse Mixture-of-Experts (MoE) architectures provide an efficient realization of this principle, but often suffer from expert collapse. Existing load-balancing heuristics encourage statistical uniformity, yet do not enforce functional diversity by construction. We propose **STODA**: **S**tructured **T**iling via **O**rthogonal **D**esign **A**dapters, a framework that partitions the global mapping space using algebraic structures derived from Orthogonal Designs. Each expert follows a lightweight view–core–unview architecture: a fixed Hurwitz–Radon (HR) view transform, a learnable core module, and an inverse mapping. The anti-commutativity and skew-symmetry of the HR family provide a tractable algebraic mechanism for expert diversification: at the generator level they yield orthogonal token views and non-degenerate mixing, while for linearized cores they identify collapse conditions and balanced cross-term structure under routed mixtures. To validate the framework under strict resource constraints, we study STODA in the Parameter-Efficient Fine-Tuning (PEFT) setting for diffusion models. Empirically, STODA matches or surpasses state-of-the-art methods while requiring dynamic computation and fewer active parameters.

Primary area:
generative models
Keywords:
Orthogonal designs, hurwitz radon, Diffusion Models, fine-tuning, MoE

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gFaGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a parameter-efficient Mixture-of-Experts framework that uses Hurwitz–Radon (HR) transformations to provide experts with structured, orthogonal views of the input. the authors name method as STODA,. Each expert applies a fixed HR transform, a learnable Tensor-Train core, and an inverse transformation. The paper develops theoretical results on orthogonality, non-degenerate mixing, and resistance to expert collapse, and evaluates STODA on diffusion-model personalization, style adaptation, preference alignment, and additional language-model experiments. Empirically, STODA is competitive with or improves over several PEFT baselines while using fewer active parameters.

Critical strengths

The use of HR structure to induce expert diversity is well motivated. In particular, the pointwise orthogonality and mixing properties provide a principled alternative to encouraging diversity indirectly through load-balancing objectives.

The paper includes useful ablations of the proposed mechanism. Comparisons with random orthogonal transformations, sign flips, and identity suggest that the HR structure contributes to reduced expert similarity and improved performance.

Critical weaknesses

My main concern is the gap between the theory and the actual architecture. The anti-collapse theorem is derived for a shared linear core, while the implemented method uses nonlinear Tensor-Train cores. For the nonlinear setting, the connection is made through local Jacobian linearization rather than a global anti-collapse result. Similarly, the clean cross-term result assumes identical core perturbations, while independently learned experts need not satisfy this assumption. Consequently, the strongest theoretical claims do not directly characterize the model evaluated experimentally. This is a major drawback in my opinion.

The empirical evidence is not sufficiently robust to fully compensate for this gap. Most experiments are single runs, with multi-seed evaluation reported only for one setting. Some broader claims about general applicability are also supported by relatively limited experiments, e.g., the small-scale pretraining experiment and a single LLM instruction-tuning setup.

Questions for authors
  1. Can the authors provide stronger theoretical or empirical evidence that the anti-collapse result meaningfully extends from the shared linear-core setting to the nonlinear Tensor-Train experts used in practice?

  2. Can the authors directly measure the proposed cross-term suppression for independently learned experts and compare it against random orthogonal views?

  3. Are the main improvements over LoRA and the strongest MoE-PEFT baselines consistent across multiple random seeds?

6 comments
AuthorsOpenReview ↗

Official comment. Part 1/4

W1. The strongest theoretical claims fail to characterize the evaluated model, as the global anti-collapse and cross-term theorems assume shared linear cores and identical perturbations, mismatching the implemented method's nonlinear Tensor-Train cores, local Jacobians, and independent experts.

Yes, Thm. 4.3 is for a shared linear core. Your comment has three parts: (a) linear vs. nonlinear cores, (b) a shared core vs. independently learned cores, and (c) the cross-term result. We clarify each below.

For (a), the model evaluated in our fine-tuning experiments is linear. We described the framework in Sec. 3 with a general core ϕi\phi_i simply because the view-core-unview construction does not depend on the form of the core. In all fine-tuning experiments in the paper, the core is the two-factor tensor train, which is a linear map in xx (only exception is the small-scale DiT pre-training in Appendix F, where the experts are MLPs). We can check this quickly in the code where the expert forward pass is

# tensor-train core of expert e_: the product of its two factors
W_eff = torch.matmul(expert_cores[e_][0], expert_cores[e_][1]).view(
    self.in_dims[0], self.out_dims[0], self.in_dims[1], self.out_dims[1])
W_eff = W_eff.permute(0, 2, 1, 3).reshape(d, -1)
# HR view and its inverse: a signed permutation of the rows and of the columns
W_eff = W_eff[perm1] * sign1
W_eff = W_eff[:, perm2] * sign2
res = torch.matmul(x, W_eff)

So each routed expert is exactly Ei(x)=Hi⊤AiHixE_i(x) = H_i^\top A_i H_i x, where Ai=∑r=12Br(i)⊗Cr(i)A_i = \sum_{r=1}^{2} B^{(i)}_r \otimes C^{(i)}_r is the tensor-train core. So the linear-core assumption of Thm. 4.3 holds exactly for the evaluated model, and the local Jacobian linearization in lines 255-259 is not needed. Since this created confusion, we have revised that paragraph to make this clear.

For (b), yes. Thm. 4.3 is written for a shared core because this is the extreme case for collapse. In a standard MoE, experts with identical parameters are identical functions, while Thm. 4.3 shows that with HR views they cannot all coincide unless the core lies in a subspace of relative dimension at most 1/641/64 (for s=7s = 7). For independently learned cores, the same argument gives us a statement that holds for every pair of cores. Let Rij=HjHi⊤R_{ij} = H_j H_i^\top. Since {HjH_j} is orthogonal, ∥Ei−Ej∥F=∥RijAiRij⊤−Aj∥F\lVert E_i - E_j \rVert_F = \lVert R_{ij} A_i R_{ij}^\top - A_j \rVert_F. So expert jj can reproduce expert ii only if its core equals the rotated core RijAiRij⊤R_{ij} A_i R_{ij}^\top. And this rotated core is in general far from a tensor train of bond dimension 2. Let ρij\rho_{ij} be the relative distance from the rotated core to the set of bond-2 tensor trains, which is given exactly by one SVD. Then, for any core that expert jj can learn,

∥Ei−Ej∥F≥ρij∥Ei∥F.\lVert E_i - E_j \rVert_F \ge \rho_{ij} \lVert E_i \rVert_F .

This bound is the independent-core counterpart of Thm. 4.3, which characterizes the equal-core case exactly. On our trained PixArt-Σ\Sigma model, the median ρij\rho_{ij} over pairs of routed experts is 0.979 (mean 0.941), and training does not reduce it (0.977 at initialization). So the bound pertains to the trained, independently parameterized experts directly. Full table and the cases in which the bound is zero are in our response to Rev. ATFW's W1(a). We would be glad to add this as an extension of Thm. 4.3 in the appendix.

For (c), we agree that independently learned experts need not satisfy the identical-perturbation assumption of Prop. A.1(iii). For this reason, we measure the interaction between independently learned experts directly. Please see Q2 below.

W2. The empirical evidence is not sufficiently robust to fully compensate for this gap. Most experiments are single runs, with multi-seed evaluation reported only for one setting.

We agree. Earlier, we had multiple seeds for only one setting. We have now run the view ablation on two backbones and two datasets with three seeds each. The FID results (mean ±\pm std) are below, and the full tables with CLIP and expert similarity are in our response to Rev. ATFW's W2.

SettingHR (STODA)Random orthogonalRandom sign flipsIdentity
Barbie, PixArt-Σ\Sigma169.15±1.14\textbf{169.15} \pm \textbf{1.14}171.46±1.18171.46 \pm 1.18173.33±1.23173.33 \pm 1.23173.50±1.08173.50 \pm 1.08
Barbie, Sana180.98±1.25\textbf{180.98} \pm \textbf{1.25}183.56±0.99183.56 \pm 0.99188.21±1.20188.21 \pm 1.20188.03±1.31188.03 \pm 1.31
Cyberpunk, PixArt-Σ\Sigma112.87±1.23\textbf{112.87} \pm \textbf{1.23}115.57±0.53115.57 \pm 0.53121.54±0.92121.54 \pm 0.92121.67±0.94121.67 \pm 0.94
Cyberpunk, Sana124.20±0.50\textbf{124.20} \pm \textbf{0.50}125.15±0.44125.15 \pm 0.44126.48±0.74126.48 \pm 0.74126.35±0.80126.35 \pm 0.80

HR views have a clear advantage over identity map and random sign flips in all four settings (2.2 to 8.8 FID), which is beyond the seed variation. The advantage over random orthogonal views is smaller (ranging 0.95 to 2.70 FID), although it is in the same direction in all four settings. For comparison with LoRA and the MoE-PEFT baselines, please see the Q3 answer. Thank you.

W3. Some broader claims about general applicability are also supported by relatively limited experiments, e.g., the small-scale pretraining experiment and a single LLM instruction-tuning setup.

Yes, the pre-training and language-model experiments are small in scale. While the concepts of STODA can be generally applied to MoE architectures, we want to keep the focus on diffusion model fine-tuning in the current paper. Hence, most of our evaluations are focused on image generation tasks. To show that the construction is not tied to fine-tuning image generation models, we included a small-scale DiT pre-training experiment on MNIST (App. F) and LLM instruction tuning of Qwen-3 4B on Alpaca. Due to compute budget limits, we limit the experiments in these settings, and we have revised the text to say that these are preliminary evidence.

Q1 Can the authors provide stronger theoretical or empirical evidence that the anti-collapse result meaningfully extends from the shared linear-core setting to the nonlinear Tensor-Train experts used in practice?

Yes, we can do both! On the theory side, the tensor-train experts used in practice are linear (please see the code in W1 above), so Thm. 4.3 applies to them directly. For independently learned cores, we have ∥Ei−Ej∥F≥ρij ∥Ei∥F\lVert E_i - E_j \rVert_F \ge \rho_{ij}\, \lVert E_i \rVert_F for every core that expert jj could learn, where ρij\rho_{ij} is the relative distance from the rotated core RijAiRij⊤R_{ij} A_i R_{ij}^\top to the set of bond-2 tensor trains. Thm. 4.3 is the special case of equal cores.

On the experimental side, we computed ρij\rho_{ij} for all pairs of experts in our trained PixArt-Σ\Sigma model. The median is 0.979 between routed experts and 0.933 between a routed expert and the shared expert, and training does not reduce it (0.977 at initialization). The measured distance between trained experts has a median of 1.414, which is the value for two orthogonal maps of equal norm. In addition, with three seeds on two backbones and two datasets, HR views are ahead of the identity map by 2.2 to 8.8 FID (W2 above).

Q2. Can the authors directly measure the proposed cross-term suppression for independently learned experts and compare it against random orthogonal views?

Yes. Because the experts are linear, the cross-term can be computed exactly from the trained weights. For expert ii, let Mi=Hi⊤ΔiHiM_i = H_i^\top \Delta_i H_i, where Δi=Ai−1dtr⁡(Ai) I\Delta_i = A_i - \tfrac{1}{d}\operatorname{tr}(A_i)\, I is the trace-free part of its core, and let ρij=⟨Mi,Mj⟩F/(∥Mi∥F ∥Mj∥F)\rho_{ij} = \langle M_i, M_j\rangle_F / (\lVert M_i \rVert_F\, \lVert M_j \rVert_F). For independently learned experts, this quantity is expected to be close to zero for any choice of views. If MiM_i and MjM_j point in independent directions in the (d2−1)(d^2-1)-dimensional space of trace-free matrices, then E[ρij]=0\mathbb{E}[\rho_{ij}] = 0 and E[ρij2]=1/(d2−1)\mathbb{E}[\rho_{ij}^2] = 1/(d^2-1), so ∣ρij∣=O(1/d)|\rho_{ij}| = O(1/d), with median ≈0.674/d≈5.9×10−4\approx 0.674/d \approx 5.9\times10^{-4} at d=1152d = 1152. Our measurements match this chance level. The suppression in Prop. A.1 concerns correlated perturbations. For identical perturbations, the normalized cross-term equals ηij=⟨Δi,RijΔiRij⊤⟩F/∥Δi∥F2\eta_{ij} = \langle \Delta_i, R_{ij}\Delta_i R_{ij}^\top\rangle_F / \lVert \Delta_i \rVert_F^2; for our learned HR cores the median ∣ηij∣|\eta_{ij}| is at most 1.5×10−31.5\times10^{-3}, compared with exactly 1 for the identity map. The Frobenius cross-term weights all dd input directions equally, whereas real tokens are strongly anisotropic. The interaction that matters for the model is therefore the one on real tokens, which weights the experts' maps by the token covariance Σx=E[xx⊤]\Sigma_x = \mathbb{E}[x x^\top]: Ex⟨Eix,Ejx⟩=tr⁡(Ei⊤EjΣx)\mathbb{E}_x\langle E_i x, E_j x\rangle = \operatorname{tr}(E_i^\top E_j \Sigma_x). Here the views do separate. The table reports the mean cosine similarity between the outputs of two experts on real evaluation tokens, across two backbones, two datasets, and three seeds (s.d. is below 0.0005 in every cell). For reference, independently initialized experts give 0.00094.

SettingHRIdentityRandom sign flipsRandom orthogonal
Barbie, PixArt-Σ\Sigma0.00330.00590.00570.0091
Barbie, Sana0.00310.00600.00560.0087
Cyberpunk, PixArt-Σ\Sigma0.00330.00590.00560.0088
Cyberpunk, Sana0.00320.00560.00570.0090

On real tokens, the interaction is smallest with HR views in all four settings. Moreover, HR views achieve this with 3.5×\textbf{3.5}\times higher throughput than random orthogonal views. We do not have a proof that learned cores are balanced between the two eigenspaces; The revised App. A states this as an assumption supported by these measurements.

Reviewer_ATFWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces STODA, a parameter-efficient fine-tuning (PEFT) framework that combines sparse Mixture-of-Experts adapters with Hurwitz–Radon (HR) transforms. The framework encourages expert diversity by transforming each routed expert's input with a fixed HR matrix, running it through a small learnable tensor-train core, and then mapping the output back to the original coordinates. The experiments show competitive results across several diffusion model adaptation tasks.

Critical strengths
  1. The paper provides an interesting and inexpensive idea for encouraging expert diversity. The HR transform gives exactly orthogonal input views through structured sign changes and permutations. This provides a principled geometric bias for diversification without introducing additional load-balancing losses.

  2. The experimental results show competitive performance with fewer active parameters. The evaluation covers different tasks such as subject-driven generation, style adaptation, and preference alignment across multiple diffusion backbones. STODA shows improved quality relative to most of the established PEFT methods.

Critical weaknesses
  1. The theoretical analysis does not establish the claimed anti-collapse and stable-mixing properties for STODA.

    (a) There is a mismatch between the theoretical assumptions and the actual learned experts. Theorem 4.3 assumes a shared linear core AA, whereas STODA uses separately learned tensor-train cores. Different independently learned cores might compensate for their respective view transforms and give out redundant functions. Despite the authors acknowledging the limitations of the shared linear core, local linearization does not necessarily resolve this mismatch.

    (b) The cross-term suppression requires additional assumptions. Proposition A.1 established equal-dimensional eigenspaces, but this does not lead to the fact that learned perturbations distribute their energy equally between them. That means the cross-terms do not need to be small. Although the authors acknowledge that cross-term suppression relies on a genericity argument, they do not establish why this assumption should hold for the learned core Jacobians. Therefore, the stronger claim in Appendix A that the HR structure maintains a well-conditioned mixed Jacobian and ensures stable gradient flow requires additional justification.

  2. The shown experimental results do not convincingly establish that the expert diversity resulting from the HR structure leads to the performance gain.

    (a) In Table 4, HR only improves FID over random orthogonal by 1.0 while getting a lower CLIP score. This difference is comparable to the seed variation of ±1.32 reported in Appendix J. The reliability of this advantage is unclear.

    (b) Although the overall evaluation covers multiple backbones and tasks, the controlled ablation in Table 4 is restricted to only one backbone and one dataset. Showing comparisons with additional backbones and datasets would be beneficial.

Questions for authors

Please refer to the weaknesses discussed above. There are also a few formatting and syntax issues, including those on line 186 and in the captions of Figure 1.

8 comments
AuthorsOpenReview ↗

Official comment. Part 1/3

We thank you for your valuable time and effort in reviewing our paper and providing feedback. Please find our responses to the questions and concerns you have raised below.

W1(a). The theoretical analysis does not establish the claimed anti-collapse and stable-mixing properties for STODA. There is a mismatch between the theoretical assumptions and the actual learned experts. Theorem 4.3 assumes a shared linear core, whereas STODA uses separately learned tensor-train cores. Different independently learned cores might compensate for their respective view transforms and give out redundant functions. Despite the authors acknowledging the limitations of the shared linear core, local linearization does not necessarily resolve this mismatch.

Yes, Thm. 4.3 is for a shared linear core. Your comment has two points, (a) linear vs. nonlinear cores and (b) a shared core vs. separately learned cores. We will answer this one by one and then clarify if separately learned cores can compensate for their view transforms.

For (a), the model evaluated in our fine-tuning experiments is linear. We described the framework in Sec. 3 with a general core ϕi\phi_i because the view-core-unview construction does not depend on the form of the core. In all fine-tuning experiments in the paper, the core is the two-factor tensor train, which is a linear map in xx. The expert forward pass in our implementation is

# tensor-train core of expert e_: the product of its two factors
W_eff = torch.matmul(expert_cores[e_][0], expert_cores[e_][1]).view(
    self.in_dims[0], self.out_dims[0], self.in_dims[1], self.out_dims[1])
W_eff = W_eff.permute(0, 2, 1, 3).reshape(d, -1)
# HR view and its inverse: a signed permutation of the rows and of the columns
W_eff = W_eff[perm1] * sign1
W_eff = W_eff[:, perm2] * sign2
res = torch.matmul(x, W_eff)

So each routed expert is exactly Ei(x)=Hi⊤AiHixE_i(x) = H_i^\top A_i H_i x, where Ai=∑r=12Br(i)⊗Cr(i)A_i = \sum_{r=1}^{2} B^{(i)}_r \otimes C^{(i)}_r is the tensor-train core. Thm. 4.3 therefore applies to the evaluated model directly. We agree completely that local linearization should not be relied on (to close this gap). It is not needed for the evaluated model. We have revised lines 255-259 to make this clear.

For (b), yes, Thm. 4.3 is stated for a shared core because this is the extreme case for collapse. In a standard MoE, experts with identical parameters are identical functions, while Thm. 4.3 shows that with HR views they cannot all coincide unless the core lies in a subspace of relative dimension at most 1/641/64 (for s=7s = 7).

Now, can separately learned cores compensate for their views? We can answer this quite cleanly. Let Rij=HjHi⊤R_{ij} = H_j H_i^\top. Since HjH_j is orthogonal,

∥Ei−Ej∥F=∥RijAiRij⊤−Aj∥F.\lVert E_i - E_j \rVert_F = \lVert R_{ij} A_i R_{ij}^\top - A_j \rVert_F .

So expert jj can reproduce expert ii only by learning Aj=RijAiRij⊤A_j = R_{ij} A_i R_{ij}^\top. For this type of compensation to be possible, the rotated core RijAiRij⊤R_{ij} A_i R_{ij}^\top must itself be a tensor train of bond dimension 2, and in general it is far from one. Let ρij\rho_{ij} be the relative distance from the rotated core to the set of bond-2 tensor trains, which is given exactly by one SVD. Then, for every core that expert jj could learn,

∥Ei−Ej∥F≥ρij∥Ei∥F.\lVert E_i - E_j \rVert_F \ge \rho_{ij} \lVert E_i \rVert_F .

Thm. 4.3 is the special case Ai=AjA_i = A_j. We computed ρij\rho_{ij} for all pairs of experts in the adapted layers of our trained PixArt-Σ\Sigma model, together with the actual distance between the trained experts.

QuantityMeanMedian
ρij\rho_{ij}, routed vs. routed0.9410.979
ρij\rho_{ij}, routed vs. shared0.9020.933
Measured ∥Ei−Ej∥F/∥Ei∥F\lVert E_i - E_j \rVert_F / \lVert E_i \rVert_F1.5561.414

For the median pair, at least 97.9% of the norm of expert ii lies outside what expert jj can represent, whatever core expert jj learns. We found that training does not reduce this since for random cores at initialization the median is 0.977. The median measured distance, 1.414, is the value for two orthogonal maps of equal norm.

The bound also tells us when compensation is possible, and we can write these cases explicitly. (i) Without views, Rij=IR_{ij} = I and ρij=0\rho_{ij} = 0, so two experts can be made identical. (ii) If the core were an unconstrained d×dd \times d matrix, we would have ρij=0\rho_{ij} = 0 and the views would be a change of coordinates. So your concern is valid for unconstrained cores, and it is the low bond dimension of the tensor-train core that prevents compensation.

We will add this also as an extension of Thm. 4.3. Thanks for the suggestion.

W1(b). The cross-term suppression requires additional assumptions. Proposition A.1 established equal-dimensional eigenspaces, but this does not lead to the fact that learned perturbations distribute their energy equally between them. That means the cross-terms do not need to be small. Although the authors acknowledge that cross-term suppression relies on a genericity argument, they do not establish why this assumption should hold for the learned core Jacobians. Therefore, the stronger claim in Appendix A that the HR structure maintains a well-conditioned mixed Jacobian and ensures stable gradient flow requires additional justification.

Yes, we agree with your point. Prop. A.1 shows that the two eigenspaces of the conjugation have equal dimension, and equal dimension does not imply that a particular learned perturbation will put equal energy in each of them. So Prop. A.1 on its own is insufficient to say that the cross-terms of the learned cores are small. We clarify below what the analysis shows and what we have checked empirically.

What is shown in the analysis is the following. For identical perturbations, Prop. A.1(iii) gives the cross-term exactly as ∥Δ+∥F2−∥Δ−∥F2\lVert \Delta_+\rVert_F^2 - \lVert \Delta_-\rVert_F^2. The condition number bound in Appendix A, κ(βˉI+D)≤(∣βˉ∣+∥D∥2)/(∣βˉ∣−∥D∥2)\kappa(\bar{\beta} I + D) \le (|\bar\beta| + \lVert D \rVert_2)/(|\bar\beta| - \lVert D \rVert_2), will hold whenever the isotropic part dominates the residual, i.e., ∣βˉ∣>∥D∥2|\bar\beta| > \lVert D \rVert_2. What we have to assume is that learned perturbations are roughly balanced between the two eigenspaces. We do not have a proof of this for the learned cores yet (in the revised Appendix A we now state it clearly as an assumption).

Since the experts are linear and independently learned, we can empirically measure their interaction directly on the trained models. First, the normalized Frobenius inner product between the Jacobians of two experts is 0.0007±0.00030.0007 \pm 0.0003 (lines 434-436). Second, we check mean cosine similarity between outputs of two experts on real evaluation tokens. And we can compare HR views with the identity map, random sign flips, and random orthogonal views across two backbones, two datasets, and multiple seeds (s.d. is below 0.0005 in every cell).

SettingHRIdentityRandom sign flipsRandom orthogonal
Barbie, PixArt-Σ\Sigma0.00330.00590.00570.0091
Barbie, Sana0.00310.00600.00560.0087
Cyberpunk, PixArt-Σ\Sigma0.00330.00590.00560.0088
Cyberpunk, Sana0.00320.00560.00570.0090

Both quantities are full interaction between two independently learned experts (recall the cross-term in Prop. A.1 is the anisotropic part). So, for the learned experts, the interaction is small, and it is smallest with HR views in all four settings.

To avoid confusion and based on your suggestion, in the revised Appendix A, we state that the cross-term decomposition is exact for identical perturbations, that the small cross-terms of the learned cores are an empirical observation supported by the measurements above.

W2. HR's link between expert diversity and performance is unconvincing, as its marginal 1.0 FID gain falls within seed variation alongside a lower CLIP score; additionally, the ablation is limited to a single setting, requiring further testing across more datasets and backbones.

We agree with your critique that Table 4 is limited to one backbone, one dataset/seed etc. We have now extended it to two backbones and two datasets with three seeds each. We report results below.

First, we would like to show that some form of an orthogonal transform is required to achieve good performance in our Tensor Train MoE architecture. We show this by incrementally designing STODA starting with a monolithic Tensor Train, adding MoE and finally integrating the HR-views. The numbers in the two tables below are from single runs, as in the paper, and the seeded comparison is after that. We can see that finetuning with a single Tensor Train already performs better than LoRA, as the Tensor Train can capture high rank parameter updates and is not restricted by the rank like LoRA. But directly using them in an MoE architecture actually degrades performance, as all experts then operate on the same view of the token. Adding in the HR views recovers this loss: the MoE now matches the single full-budget Tensor Train at about half the active parameters (5.7M vs. 10.3M). We believe this is interesting. Scaling capacity while keeping only a small subset of experts active per token is the principle behind many current large language models, and it is now coming up in diffusion models as well (Sec. 1). Our tables show that this trade-off does not come for free. Simply placing Tensor Trains in an MoE architecture loses quality, and specializing the experts with different HR views is what allows the sparse model to keep the quality of the dense one.

Pixart-Σ, BarbieCore

MethodActive Params/ Total Params (M)FID
LoRA10.3/10.3180.4
Single TT expert, no MoE10.3/10.3171.3
TT-MoE, identity views5.7/10.3173.8
TT-MoE, HR views5.7/10.3170.3

SaNa, Expedition

MethodActive Params/ Total Params (M)FID
LoRA10.3/10.3145.5
Single TT expert, no MoE10.3/10.3140.3
TT-MoE, identity views5.7/10.3141.2
TT-MoE, HR views5.7/10.3139.1

Second, we run STODA with HR views, random orthogonal views, random sign flips, and identity map (no transformation) on two backbone models and two different datasets (three seeds each). HR views in STODA have a clear advantage over identity and random sign flips in all four settings (2.2 to 8.8 FID), which is beyond seed variation. Advantage over random orthogonal transforms is smaller, although in the same direction in all four settings. This is expected as random transforms can also rotate the input differently for different experts. On CLIP score, with three seeds the HR views are higher than random orthogonal views in all four settings, so the lower CLIP score in Table 4 does not persist. In addition, STODA can achieve this in O(dlog⁡d)\mathcal{O}(d \log d) steps without any multiplications. We measured the tokens/ms of an adapter with HR views vs random orthogonal views on an A100 GPU. STODA achieves 7.7±0.3\textbf{7.7} \pm \textbf{0.3}K tokens/ms vs 2.2±0.12.2 \pm 0.1K tokens/ms for random orthogonal views. Hence, STODA gets the best FID scores with 3.5×\textbf{3.5} \times higher throughput than random orthogonal transforms.

BarbieCore, Pixart-Σ

TransformationFID↓CLIP↑Avg. Sim↓
Random orthogonal171.46±1.18171.46 \pm 1.1829.67±0.2329.67 \pm 0.230.0091±0.000160.0091 \pm 0.00016
Random sign flips173.33±1.23173.33 \pm 1.2330.04±0.1330.04 \pm 0.130.0057±0.000100.0057 \pm 0.00010
Identity173.50±1.08173.50 \pm 1.0830.15±0.11\textbf{30.15} \pm \textbf{0.11}0.0059±0.000210.0059 \pm 0.00021
HR (STODA)169.15±1.14\textbf{169.15} \pm \textbf{1.14}30.01±0.1630.01 \pm 0.160.0033±0.00042\textbf{0.0033} \pm \textbf{0.00042}

BarbieCore, SaNa

TransformationFID↓CLIP↑Avg. Sim↓
Random orthogonal183.56±0.99183.56 \pm 0.9931.81±0.0931.81 \pm 0.090.0087±0.000190.0087 \pm 0.00019
Random sign flips188.21±1.20188.21 \pm 1.2032.03±0.12\textbf{32.03} \pm \textbf{0.12}0.0056±0.000100.0056 \pm 0.00010
Identity188.03±1.31188.03 \pm 1.3131.97±0.1431.97 \pm 0.140.0060±0.000220.0060 \pm 0.00022
HR (STODA)180.98±1.25\textbf{180.98} \pm \textbf{1.25}31.91±0.0631.91 \pm 0.060.0031±0.00030\textbf{0.0031} \pm \textbf{0.00030}

Cyberpunk, Pixart-Σ

TransformationFID↓CLIP↑Avg. Sim↓
Random orthogonal115.57±0.53115.57 \pm 0.5328.71±0.1528.71 \pm 0.150.0088±0.000200.0088 \pm 0.00020
Random sign flips121.54±0.92121.54 \pm 0.9228.83±0.1128.83 \pm 0.110.0056±0.000150.0056 \pm 0.00015
Identity121.67±0.94121.67 \pm 0.9428.85±0.1228.85 \pm 0.120.0059±0.000200.0059 \pm 0.00020
HR (STODA)112.87±1.23\textbf{112.87} \pm \textbf{1.23}29.06±0.09\textbf{29.06} \pm \textbf{0.09}0.0033±0.00039\textbf{0.0033} \pm \textbf{0.00039}

Cyberpunk, SaNa

TransformationFID↓CLIP↑Avg. Sim↓
Random orthogonal125.15±0.44125.15 \pm 0.4429.55±0.2629.55 \pm 0.260.0090±0.000180.0090 \pm 0.00018
Random sign flips126.48±0.74126.48 \pm 0.7429.54±0.1629.54 \pm 0.160.0057±0.000110.0057 \pm 0.00011
Identity126.35±0.80126.35 \pm 0.8029.47±0.1429.47 \pm 0.140.0056±0.000140.0056 \pm 0.00014
HR (STODA)124.20±0.50\textbf{124.20} \pm \textbf{0.50}29.62±0.21\textbf{29.62} \pm \textbf{0.21}0.0032±0.00035\textbf{0.0032} \pm \textbf{0.00035}

Q1. Formatting issues.

Thank you for pointing these out. At line 186, the paragraph heading "Compute benefits." lost the formatting and ran into the previous sentence. It is now a new paragraph. In the caption of Figure 1, the panel labels were inconsistent and have been fixed. All changes are marked in red in the revised PDF, and we much appreciate the help.


Thank you for the questions and feedback. We hope we addressed all your questions and concerns, and we remain available if you have any additional questions.

Reviewer_JXQMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents STODA, a PEFT method that integrates structured orthogonal transformations into a MoE adapter for diffusion models. Each routed expert adopts a "view–core–unview" architecture comprising a fixed Hurwitz–Radon transform, a learnable Tensor Train core, and the inverse transform. The authors argue that the algebraic properties of HR matrices promote expert diversity and stable expert mixing. Experiments cover subject-driven generation, style tuning, preference alignment, language-model instruction tuning, and small-scale DiT pre-training.

Critical strengths
  • Using HR matrices, and their link to Clifford algebra, to construct expert views is a new idea, and applying them costs only O(dlog⁡d)O(d \log d) (Appendix L).

  • STODA obtains the best average FID on PixArt-Σ, SANA, and Nitro-T (Table 2) with about half the active parameters of LoRA and OFT.

Critical weaknesses
  1. The paper does not discuss prior work that uses orthogonality to diversify MoE-LoRA experts such as OMoE (Feng et al., ECAI 2025), MoORE (Yuan et al., NeurIPS 2025) and D²MoRA (Zuo et al., AAAI 2026). The paper does not explain how HR-based input-view orthogonality relates to these output-level, parameter-level, and regularization-based approaches. OMoE targets the same expert-collapse problem, so an empirical comparison with it would be especially useful.

  2. The theory does not cover the implemented model. Prop. 4.1 shows orthogonality of the transformed inputs, not of the expert functions. Thm. 4.3 assumes a shared linear core, while the implementation trains independent nonlinear TT cores. Prop. A.1(iii) assumes identical perturbations, and the general case rests on a genericity argument. These results motivate the design but do not guarantee that the trained experts avoid collapse, so "provable guarantees" in the Conclusion overstates them.

  3. The contribution of HR specifically, as opposed to generic orthogonal views, is not clearly established. In Table 4, dense random orthogonal views improve over the identity map (FID 173.8 → 171.3) and HR adds a further 1.0 (170.3), on a single dataset. This is smaller than the run-to-run standard deviation of STODA reported in Appendix J (169.29 ± 1.32). The identity-view row already improves substantially over LoRA (173.8 vs. 180.4 on the same dataset), so much of the gain appears to come from the TT + MoE design. HR does give the lowest expert similarity, but all values in Table 4 are below 0.01. Results on more datasets or backbones would show whether the HR-specific advantage holds.

Questions for authors

-How does HR-based input-view orthogonality differ from the output-level orthogonality in OMoE, conceptually and empirically?

-Under what assumptions does Thm. 4.3 say anything about independently parameterized nonlinear TT cores?

-Does the advantage of HR over random orthogonal views in Table 4 hold on other style datasets or backbones?

7 comments
AuthorsOpenReview ↗

Official comment

We thank you for your time and effort in reviewing our paper and providing feedback. Please find our responses to the concerns you have raised below.

W1. The paper does not discuss prior work that uses orthogonality to diversify MoE-LoRA experts such as OMoE (Feng et al., ECAI 2025), MoORE (Yuan et al., NeurIPS 2025), and D$^2$MoRA (Zuo et al., AAAI 2026). The paper does not explain how HR-based input-view orthogonality relates to these output-level, parameter-level, and regularization-based approaches. OMoE targets the same expert-collapse problem, so an empirical comparison with it would be especially useful.

Our related work section discussed MoE-PEFT methods that address expert collapse through routing and load-balancing objectives, and we compared against MixLoRA and FlyLoRA from that set of results. We totally agree that OMoE, MoORE and D$^2$MoRA form a second set of results, which targets expert diversity through orthogonality, and that STODA should also be positioned relative to them. The revised section now does this. Briefly, these methods achieve expert diversity via explicit post-processing of the outputs (OMoE), SVD-based parameter initialization (MoORE), and regularization (D$^2$MoRA). STODA uses orthogonality in the coordinates in which each expert's core operates.

(i) OMoE orthogonalizes experts' outputs using the Gram-Schmidt. This makes sure that the MoE layer produces orthogonal outputs, but it has two major drawbacks. First is computational: one needs to run the Gram-Schmidt on the layer outputs for each token. This process is sequential in the number of selected experts, and it degrades the throughput of the model as that number increases. In the efficiency table in their paper, top-2 OMoE is about 15% slower than MixLoRA (4863 ms vs. 4217 ms). The second issue is scalability. Even though the experts' outputs are orthogonalized, it does not mean the router can take advantage of it. This is shown in their ablation study (see page 6 in OMoE). The paper says that with the increasing number of experts, accuracy degrades during soft routing, and top-2 routing benefits only slightly. On the other hand, we show that the expert diversity in STODA is achieved via HR matrices, which are computationally efficient and scale very well with the number of experts (see Figure 7 and Table 8 in the paper). In our comparison in Q1 below, STODA has lower FID than OMoE in all nine settings.

(ii) MoORE decomposes pretrained weights of the model into rank one experts and uses a hybrid router that weights the experts based on the task and the sample. Given a task, the sample dependent term is given by:

U((Q⊤Γx)⊙(V⊤Hx))U\big((Q^\top \Gamma x) \odot (V^\top H x)\big)

where V⊤HV^\top H is the rotated input basis and Q⊤ΓQ^\top \Gamma is the low rank projection. The orthogonal rotation matrix HH is shared across all the experts and does not contribute to their diversity. Because of this parameterization, the adapter reweights the singular components of the pretrained weight through a low rank projection of the input, and its output stays in the range of the pretrained weight (Eq. 8 in the paper).

(iii) D$^2$MoRA parameterizes experts using low rank matrices AiA_i, BiB_i, and two routers that select different AiA_i, BiB_i independently. They encourage expert diversity using a regularization loss LorthoL_{\text{ortho}} that enforces orthogonality on the flattened matrices AiA_i, BiB_i. The regularizer term adds more hyperparameters to tune. In the ablation study (figure 4 in D$^2$MoRA), they show that using a large value of λ\lambda can negatively affect the accuracy of the model. On the other hand, STODA has no diversity-loss weight to tune, since its diversity comes from the fixed views.

W2. The theory does not cover the implemented model. Prop. 4.1 shows orthogonality of the transformed inputs, not of the expert functions. Thm. 4.3 assumes a shared linear core, while the implementation trains independent nonlinear TT cores. Prop. A.1(iii) assumes identical perturbations, and the general case rests on a genericity argument. These results motivate the design but do not guarantee that the trained experts avoid collapse, so "provable guarantees" in the Conclusion overstates them.

Response:

Yes, Prop. 4.1 concerns the transformed inputs. It says that each expert receives an exactly orthogonal copy of the token. Here we do not make any statement about the expert functions. The function-level result is actually Thm. 4.3, so we focus on it below. Your comment on Thm. 4.3 contains two parts: (a) linear vs. nonlinear cores, and (b) a shared core vs. independently trained cores. We clarify each below.

For (a), the model evaluated in our fine-tuning experiments is linear. We presented the framework in Sec. 3 with a general core ϕi\phi_i because the view-core-unview construction does not really depend on the form of the core. In all fine-tuning experiments in the paper, the core is the two-factor tensor train, which is a linear map in xx. We can check this quickly in the code where the expert forward pass is

# tensor-train core of expert e_: the product of its two factors
W_eff = torch.matmul(expert_cores[e_][0], expert_cores[e_][1]).view(
    self.in_dims[0], self.out_dims[0], self.in_dims[1], self.out_dims[1])
W_eff = W_eff.permute(0, 2, 1, 3).reshape(d, -1)
# HR view and its inverse: a signed permutation of the rows and of the columns
W_eff = W_eff[perm1] * sign1
W_eff = W_eff[:, perm2] * sign2
res = torch.matmul(x, W_eff)

So each routed expert is exactly Ei(x)=Hi⊤AiHixE_i(x) = H_i^\top A_i H_i x, where Ai=∑r=12Br(i)⊗Cr(i)A_i = \sum_{r=1}^{2} B^{(i)}_r \otimes C^{(i)}_r is the tensor-train core. Thm. 4.3 therefore applies to the evaluated model directly, and the local linearization in lines 255-259 is not needed for it. Since this created confusion, we have revised that paragraph to make this clear.

For (b), yes, Thm. 4.3 is for a shared core, because this is the extreme case for collapse. In a standard MoE, experts with identical parameters are identical functions, while Thm. 4.3 shows that with HR views they cannot all coincide unless the core lies in a subspace of relative dimension at most 1/641/64 (for s=7s = 7). For independently trained cores, the same argument gives us a statement that holds for every pair of cores. Let Rij=HjHi⊤R_{ij} = H_j H_i^\top. Since HjH_j is orthogonal, ∥Ei−Ej∥F=∥RijAiRij⊤−Aj∥F\lVert E_i - E_j \rVert_F = \lVert R_{ij} A_i R_{ij}^\top - A_j \rVert_F. So expert jj can reproduce expert ii only if its core equals the rotated core RijAiRij⊤R_{ij} A_i R_{ij}^\top, and this rotated core is in general far from a tensor train of bond dimension 2. Let ρij\rho_{ij} be the relative distance from the rotated core to the set of bond-2 tensor trains, which is given exactly by one SVD. Then, for any core that expert jj could learn,

∥Ei−Ej∥F≥ρij∥Ei∥F.\lVert E_i - E_j \rVert_F \ge \rho_{ij} \lVert E_i \rVert_F .

Thm. 4.3 is the special case Ai=AjA_i = A_j. On our trained PixArt-Σ\Sigma model, the median ρij\rho_{ij} over pairs of routed experts is 0.979 (mean 0.941), and training does not reduce it much at all (0.977 at initialization). So, for the median pair, at least 97.9% of the norm of expert ii lies outside what expert jj can represent. This addresses your point about the results "do not guarantee that the trained experts avoid collapse", since the bound holds for the trained, independently parameterized experts. The cases in which the bound is zero are explained in our response to Rev. ATFW's W1(a). We will include this as a result extending Thm. 4.3.

On Prop. A.1(iii), we agree that the clean statement assumes identical perturbations, and that equal-dimensional eigenspaces do not mean small cross-terms for learned cores. For this reason, we measure the interaction directly on the trained, independently learned experts. The normalized Frobenius inner product between the Jacobians of two experts is 0.0007±0.00030.0007 \pm 0.0003 (lines 434-436). Since the experts are linear, each Jacobian is the expert map itself, so this is the full interaction between two independently learned experts, of which the cross-term in Prop. A.1 is actually the anisotropic part. The same interaction measured on real tokens, with a comparison against the identity map, random sign flips, and random orthogonal views, is in the Avg. Sim columns of tables in W3 below, where HR value is the lowest in all four settings.

Finally, we agree that "provable guarantees" may be claiming more than necessary. In the revision, the Conclusion reads "exact algebraic properties of the HR views and a characterization of collapse for linear cores".

W3. HR's slight FID improvement falls within the standard error, with most gains actually coming from the TT+MoE design; additionally, its reduction in expert similarity is practically insignificant, requiring further testing across more datasets and backbones.

First, we would like to address the point about much of the gain coming from TT + MoE design. The tables below show the incremental performance of STODA's design. We start with a monolithic Tensor Train, add MoE and finally include the HR-views on two different backbone models with different datasets. The numbers in these two tables are from single runs (as in our paper). Seeded comparison is after that. We can see that finetuning with a single Tensor Train already performs better than LoRA, as the Tensor Train can capture high rank parameter updates but is not restricted by the rank like LoRA. So yes, part of the gain over LoRA comes from the Tensor Train. We now state this clearly in the paper. However, directly using them in an MoE architecture actually degrades performance, as all experts then operate on the same view of the token. Adding in the HR views recovers this loss: the MoE now matches the single full-budget Tensor Train at about half the active parameters (5.7M vs. 10.3M). We believe this is the significant part of our result. Scaling capacity while keeping only a small subset of experts active per token is the principle behind many current large language models, and it is now emerging in diffusion models as well (Sec. 1). The tables describe this trade-off. Simply placing Tensor Trains in an MoE architecture loses quality. And specializing the experts with different HR views is what allows the sparse model to keep the quality of the dense one!

Pixart Σ\Sigma on BarbieCore dataset

MethodActive Params/ Total Params (M)FID
LoRA10.3/10.3180.4
Single TT expert, no MoE10.3/10.3171.3
TT-MoE, identity views5.7/10.3173.8
TT-MoE, HR views5.7/10.3170.3

SaNa on Expedition dataset

MethodActive Params/ Total Params (M)FID
LoRA10.3/10.3145.5
Single TT expert, no MoE10.3/10.3140.3
TT-MoE, identity views5.7/10.3141.2
TT-MoE, HR views5.7/10.3139.1

Second, to check if the performance and the expert similarity metrics hold across different backbone models, datasets and seeds, we evaluated STODA with HR views, random orthogonal views, random sign flips and identity map (no transformation) on two different backbone models with two different datasets across three seeds. The results (mean ±\pm std over seeds) are presented in the tables below. HR views have a clear advantage over the identity map and random sign flips in all four settings, by 2.2 to 8.8 FID, which is well beyond the seed variation. The advantage over random orthogonal transforms is smaller, 0.95 to 2.70 FID, although it is in the same direction in all four settings. This is expected as random transforms also rotate the input differently for different experts, so we do not claim a large FID advantage of HR views over dense orthogonal views. However, STODA achieves this at a much lower cost. An HR view is a signed permutation, applied without any multiplications in O(dlog⁡d)\mathcal{O}(d \log d) (Appendix L), and it stores one index and one sign per coordinate (about 2.3K values at d=1152d = 1152). A dense orthogonal view costs O(d2)\mathcal{O}(d^2) per token and stores a 1152×11521152 \times 1152 matrix (1.33M values). To check this advantage, we measured the tokens/ms of an adapter with HR views vs random orthogonal views on an A100 GPU. STODA achieves 7.7±0.3\textbf{7.7} \pm \textbf{0.3}K tokens/ms vs 2.2±0.12.2 \pm 0.1K tokens/ms for random orthogonal views. Hence, STODA achieves the best FID scores on the downstream finetuning with 3.5×\textbf{3.5} \times higher throughput than random orthogonal transforms.

On the expert similarity, yes, the values are small for all variants. However, the ordering is stable across the four settings and across seeds: about 0.003 for HR views, about 0.006 for the identity map and random sign flips, and about 0.009 for random orthogonal views. These differences are many times the seed standard deviation, so the HR value is reliably the lowest, at about half of the identity value.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.