Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts

Large Language Models (LLMs) have achieved remarkable progress, with Parameter-Efficient Fine-Tuning (PEFT) emerging as a key technique for downstream task adaptation. However, existing PEFT methods mainly operate in Euclidean space, fundamentally limiting their capacity to capture complex geometric structures inherent in language data. Although hyperbolic and spherical geometries provide useful inductive biases for hierarchical and circular structures, respectively, imposing a single manifold type still limits representational expressiveness, even with learnable curvature. To address this, we propose Mixture of Space (MoS), a unified framework that leverages multiple geometric spaces simultaneously to learn richer, curvature-aware representations. Building on this scheme, we develop MoSLoRA, which extends Low-Rank Adaptation (LoRA) with heterogeneous geometric experts, enabling models to dynamically select or combine appropriate geometric spaces based on input context. Furthermore, MoSLoRA integrates lightweight token routing to select suitable geometric experts for each input token and aggregates their outputs through a unified projection, avoiding frequent exponential and logarithmic manifold mappings while preserving multi-space specialization. Our experiments across diverse benchmarks demonstrate that MoSLoRA outperforms strong baselines, achieving up to 5.6% improvement on MATH500 and 15.9% on MAWPS. Moreover, we provide empirical analyses of curvature dynamics, revealing how adaptive space selection and curvature optimization affect training stability and model performance. The code and datasets are available at https://anonymous.4open.science/r/MoSLoRA.