acceptodds
Under review as a conference paper at ICLR 2027

Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts

Abstract

Large Language Models (LLMs) have achieved remarkable progress, with Parameter-Efficient Fine-Tuning (PEFT) emerging as a key technique for downstream task adaptation. However, existing PEFT methods mainly operate in Euclidean space, fundamentally limiting their capacity to capture complex geometric structures inherent in language data. Although hyperbolic and spherical geometries provide useful inductive biases for hierarchical and circular structures, respectively, imposing a single manifold type still limits representational expressiveness, even with learnable curvature. To address this, we propose Mixture of Space (MoS), a unified framework that leverages multiple geometric spaces simultaneously to learn richer, curvature-aware representations. Building on this scheme, we develop MoSLoRA, which extends Low-Rank Adaptation (LoRA) with heterogeneous geometric experts, enabling models to dynamically select or combine appropriate geometric spaces based on input context. Furthermore, MoSLoRA integrates lightweight token routing to select suitable geometric experts for each input token and aggregates their outputs through a unified projection, avoiding frequent exponential and logarithmic manifold mappings while preserving multi-space specialization. Our experiments across diverse benchmarks demonstrate that MoSLoRA outperforms strong baselines, achieving up to 5.6% improvement on MATH500 and 15.9% on MAWPS. Moreover, we provide empirical analyses of curvature dynamics, revealing how adaptive space selection and curvature optimization affect training stability and model performance. The code and datasets are available at https://anonymous.4open.science/r/MoSLoRA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Non-Euclidean Space, Parameter-Efficient Fine-tuning, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w8UPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Mixture of Space (MoS), which unifies hyperbolic, spherical, and Euclidean constant-curvature spaces by recovering the time-like coordinate from the space-like component under the corresponding curvature constraint (Eq. 5), and replaces exp/log maps with inverse/forward stereographic projections (Eq. 6–7). Building on this, MoSLoRA places 8 LoRA experts with learnable curvatures (default H3-E2-S3) on the FFN projections, with top-4 token routing, a grouped (within-space) load-balancing auxiliary loss, and a separate optimizer for curvature parameters. Experiments on Qwen2-1.5B, Qwen2.5-3B/7B, and Gemma2-2B across NLU, math reasoning, and commonsense benchmarks report higher average accuracy than LoRA, DoRA, HydraLoRA, HypLoRA, and other baselines.

Critical strengths
  1. Extending PEFT from a single geometric assumption (e.g., HypLoRA, nGPT) to a routed mixture of heterogeneous geometries is a natural idea and connects well with recent work on the representation geometry of LLMs.
  2. The unified projection uses only norm-based and element-wise operations, avoiding exp/log maps, and is easy to plug into existing MoE-LoRA frameworks. The proof of Prop. 4.1 (scale equivariance) is correct.
  3. Four backbones and around ten benchmarks, together with single-space variants (Table 3), expert-allocation ablations (Table 4), an optimizer ablation (Table 11), and curvature-trajectory visualizations (Fig. 3).
  4. An anonymous code repository is provided, and the appendix reports hyperparameters and data splits.
Critical weaknesses
  1. The average gains are driven largely by a single dataset (MAWPS). Across all tables, MoSLoRA outperforms every baseline on MAWPS by 10–19 points, while differences on the other datasets are small or negative. Recomputing the averages from the reported numbers with MAWPS excluded:
  • Gemma2-2B (Table 9): MoSLoRA 61.38 < HydraLoRA 61.64
  • Qwen2.5-7B (Table 10): MoSLoRA 53.20 < HydraLoRA 54.21 On Qwen2.5-7B, MoSLoRA also trails HypLoRA on GSM8K by 20 points (52.46 vs. 72.48), which is not discussed. Moreover, the Euclidean-only variant MoSLoRA-E already reaches 70.77 on MAWPS (Table 3), far above all baselines (~62–64). This suggests the MAWPS jump stems from pipeline or evaluation differences rather than from the geometric mixture.
  1. The geometric component may be numerically negligible.Composing Eq. 6–8, each expert reduces to a linear map sandwiched between two curvature-dependent radial rescalings, and it degenerates to a plain LoRA as κ‖x‖² → 0. By Prop. 4.1, input scaling by γ is equivalent to replacing the curvature with κγ². With γ = 1e-3 (Table 5) and learned |κ| ≲ 1 (Fig. 3), the effective curvature is on the order of 1e-6, so the radial correction on hidden states may be only ~0.1%. This is consistent with the mixture outperforming the best single-space variant by only 0.35 points (Table 3, single run). In addition, the curvatures in Fig. 3 cross κ = 0, where φ(κ) = 1/|κ| is discontinuous; how this is handled (and how such experts are regrouped in the grouped auxiliary loss) is not explained.
  2. The experimental protocol is insufficient to support state-of-the-art claims. All results come from a single run without variance, while most per-dataset differences are within 1 point. Baselines reuse MoSLoRA's learning rate without tuning. HMoRA obtains 12.28 on GSM8K and 2.60 on MATH500 with Qwen2-1.5B, indicating a failed configuration. Activated parameters are not matched (MoSLoRA 1.31% vs. LoRA ~0.45%).Some base-model scores look anomalous: Qwen2.5-3B scores 0.40 on MATH500, and CSQA and OBQA are both exactly 32.40. The pair 87.15/87.19 (CoLA/MRPC) appears identically in three different rows: DoRA (Table 1), MoSLoRA-E (Table 3), and MoSLoRA-UNI (Table 11).
  3. Insufficient positioning of novelty.Unifying positive, zero, and negative curvature via stereographic projection was already done by the κ-stereographic model (Bachmann et al., ICML 2020), which is smooth at κ = 0; it is not cited. Mixture-of-curvature experts in LLMs was proposed by HELM (He et al., 2026), which the paper dismisses in one sentence as orthogonal. In addition, the abstract's "5.6% on MATH500 / 15.9% on MAWPS" cannot be matched to any table.
Questions for authors
  1. What explains the MAWPS jump, including for MoSLoRA-E? Please provide the MAWPS split, deduplication procedure, and answer-parsing method.
  2. What is the actual magnitude of κ_eff‖x‖² per layer after training? Can you report a control in which all κ are fixed to 0 with everything else unchanged?
  3. Can you report mean and standard deviation over at least 3 seeds, and baseline results with individually tuned learning rates?
  4. What is the essential difference from the κ-stereographic model and HELM?
Reviewer_1n15OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoSLoRA, a parameter-efficient fine-tuning framework that integrates LoRA with heterogeneous geometric experts operating in hyperbolic, Euclidean, and spherical spaces. The method introduces token-level routing, unified stereographic projection, and a grouped auxiliary loss to enable adaptive geometric representations while reducing the computational overhead of non-Euclidean transformations. Experiments on multiple language models and downstream tasks demonstrate competitive performance, particularly on mathematical reasoning benchmarks.

Critical strengths
  1. Novel geometric expert design. The paper introduces a mixture of heterogeneous geometric experts into LoRA-based fine-tuning, enabling adaptive integration of different geometric representations. The proposed unified projection and grouped auxiliary loss provide a coherent framework for multi-space adaptation.
  2. Promising performance on reasoning tasks. MoSLoRA achieves improvements over several PEFT baselines, particularly on mathematical reasoning benchmarks such as MATH500 and MAWPS, demonstrating the potential of incorporating heterogeneous geometric representations into LLM fine-tuning.
Critical weaknesses
  1. Limited understanding of geometry-specific specialization. Although the mixed-space model outperforms single-space variants on most benchmarks, the complementary roles of different geometric experts remain insufficiently understood. The paper does not clearly establish how expert routing and geometric specialization contribute to performance gains across different tasks.
  2. Insufficient experimental fairness and efficiency validation. The comparisons mainly control activated parameters rather than total trainable parameters. Moreover, the efficiency evaluation focuses on geometric mapping operations, lacking end-to-end training and inference comparisons with standard LoRA.
Questions for authors
  1. Complementarity of geometric spaces: Could the authors further analyze how different geometric experts specialize across tasks and how their routing patterns contribute to the observed performance gains?
  2. Parameter and computational efficiency: Could the authors compare MoSLoRA with major baselines under matched total trainable parameter budgets and report end-to-end training time, inference latency, and peak GPU memory usage?
  3. Applicability to different modules: The experiments primarily apply MoSLoRA to FFN layers. Could the authors provide additional experiments on QKV projections or different combinations of target modules to demonstrate the applicability of the method?
Reviewer_caHsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes MoSLoRA, a mixture-of-LoRA-experts PEFT method whose experts live in hyperbolic, Euclidean, and spherical spaces with learnable curvature. Tokens are routed to experts, mapped with a unified stereogrphic projection instead of exp/log maps, and balanced with a grouped auxiliary loss within each geometry. Experiments validate most of its claims.

Critical strengths

The papers explore quite interesting idea. The idea of treating different constant-curvature geometries as routable LoRA experts is interesting, and the stereographic projection is a practical replacement for exp/log maps (up to 4× faster and lower forward memory). Emperical analysis holds most of its claims, overall analysis is informative.

Critical weaknesses
  1. There are no Llama or Mistral families and nothing at 13B–70B. The tasks are narrow: three GLUE tasks, five math sets, and two commonsense sets. Standard suites such as the 8-task commonsense benchmark, MMLU, code (HumanEval/MBPP), and instruction following (MT-Bench/AlpacaEval) would show whether the geometric benefit generalizes beyond small math word problems.

  2. All results are single runs, with no seeds or variance. Report atleast some runs on multiple seeds to show it generalize.

  3. In some case, tasks gains are small or negative: on Qwen2.5-7B GSM8K, MoSLoRA scores 52.46 vs. 72.48 for HypLoRA (about −20 points), and it also trails on GSM8K for 3B and Gemma., Provide an explaination for this.

  4. It is unclear whether the gain comes from geometry or from the added nonlinearity. Needed controls: fixed κ = 0 for all experts using the same projection pipeline, fixed vs. learnable curvature, and random curvature.

  5. Also Adding ablation to the paper will strengthen it. Like, grouped vs. standard auxiliary loss, number of experts and top-K, the accuracy effect of stereographic vs. exp/log maps.

  6. Efficiency is compared only to exp/log maps, not to LoRA or HydraLoRA in end-to-end training time and inference latency.

Novality is quite limited, The unified stereographic projection closely resembles the κ-stereographic model, and the propositions are straightforward.

Questions for authors

See Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.