- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Mixture of Space (MoS), which unifies hyperbolic, spherical, and Euclidean constant-curvature spaces by recovering the time-like coordinate from the space-like component under the corresponding curvature constraint (Eq. 5), and replaces exp/log maps with inverse/forward stereographic projections (Eq. 6–7). Building on this, MoSLoRA places 8 LoRA experts with learnable curvatures (default H3-E2-S3) on the FFN projections, with top-4 token routing, a grouped (within-space) load-balancing auxiliary loss, and a separate optimizer for curvature parameters. Experiments on Qwen2-1.5B, Qwen2.5-3B/7B, and Gemma2-2B across NLU, math reasoning, and commonsense benchmarks report higher average accuracy than LoRA, DoRA, HydraLoRA, HypLoRA, and other baselines.
Critical strengths
- Extending PEFT from a single geometric assumption (e.g., HypLoRA, nGPT) to a routed mixture of heterogeneous geometries is a natural idea and connects well with recent work on the representation geometry of LLMs.
- The unified projection uses only norm-based and element-wise operations, avoiding exp/log maps, and is easy to plug into existing MoE-LoRA frameworks. The proof of Prop. 4.1 (scale equivariance) is correct.
- Four backbones and around ten benchmarks, together with single-space variants (Table 3), expert-allocation ablations (Table 4), an optimizer ablation (Table 11), and curvature-trajectory visualizations (Fig. 3).
- An anonymous code repository is provided, and the appendix reports hyperparameters and data splits.
Critical weaknesses
- The average gains are driven largely by a single dataset (MAWPS). Across all tables, MoSLoRA outperforms every baseline on MAWPS by 10–19 points, while differences on the other datasets are small or negative. Recomputing the averages from the reported numbers with MAWPS excluded:
- Gemma2-2B (Table 9): MoSLoRA 61.38 < HydraLoRA 61.64
- Qwen2.5-7B (Table 10): MoSLoRA 53.20 < HydraLoRA 54.21 On Qwen2.5-7B, MoSLoRA also trails HypLoRA on GSM8K by 20 points (52.46 vs. 72.48), which is not discussed. Moreover, the Euclidean-only variant MoSLoRA-E already reaches 70.77 on MAWPS (Table 3), far above all baselines (~62–64). This suggests the MAWPS jump stems from pipeline or evaluation differences rather than from the geometric mixture.
- The geometric component may be numerically negligible.Composing Eq. 6–8, each expert reduces to a linear map sandwiched between two curvature-dependent radial rescalings, and it degenerates to a plain LoRA as κ‖x‖² → 0. By Prop. 4.1, input scaling by γ is equivalent to replacing the curvature with κγ². With γ = 1e-3 (Table 5) and learned |κ| ≲ 1 (Fig. 3), the effective curvature is on the order of 1e-6, so the radial correction on hidden states may be only ~0.1%. This is consistent with the mixture outperforming the best single-space variant by only 0.35 points (Table 3, single run). In addition, the curvatures in Fig. 3 cross κ = 0, where φ(κ) = 1/|κ| is discontinuous; how this is handled (and how such experts are regrouped in the grouped auxiliary loss) is not explained.
- The experimental protocol is insufficient to support state-of-the-art claims. All results come from a single run without variance, while most per-dataset differences are within 1 point. Baselines reuse MoSLoRA's learning rate without tuning. HMoRA obtains 12.28 on GSM8K and 2.60 on MATH500 with Qwen2-1.5B, indicating a failed configuration. Activated parameters are not matched (MoSLoRA 1.31% vs. LoRA ~0.45%).Some base-model scores look anomalous: Qwen2.5-3B scores 0.40 on MATH500, and CSQA and OBQA are both exactly 32.40. The pair 87.15/87.19 (CoLA/MRPC) appears identically in three different rows: DoRA (Table 1), MoSLoRA-E (Table 3), and MoSLoRA-UNI (Table 11).
- Insufficient positioning of novelty.Unifying positive, zero, and negative curvature via stereographic projection was already done by the κ-stereographic model (Bachmann et al., ICML 2020), which is smooth at κ = 0; it is not cited. Mixture-of-curvature experts in LLMs was proposed by HELM (He et al., 2026), which the paper dismisses in one sentence as orthogonal. In addition, the abstract's "5.6% on MATH500 / 15.9% on MAWPS" cannot be matched to any table.
Questions for authors
- What explains the MAWPS jump, including for MoSLoRA-E? Please provide the MAWPS split, deduplication procedure, and answer-parsing method.
- What is the actual magnitude of κ_eff‖x‖² per layer after training? Can you report a control in which all κ are fixed to 0 with everything else unchanged?
- Can you report mean and standard deviation over at least 3 seeds, and baseline results with individually tuned learning rates?
- What is the essential difference from the κ-stereographic model and HELM?