PRISM: Spectral Analysis Aided Nested Manifold Learning for Efficient Multimodal Large Language Model Adaptation on Edge

Parameter-efficient fine-tuning (PEFT) has established itself as a dominant paradigm for adapting lightweight multimodal large language models, aiming to balance adaptation efficacy with computational efficiency. However, prevalent approaches predominately adhere to static parameterization, imposing a monolithic computational regimen that is insensitive to the intrinsic variance in visual complexity. This rigidity induces significant computational redundancy, thereby hindering optimal inference efficiency, particularly for adaptation on the edge. To surmount this bottleneck, we introduce a dynamic PEFT framework, dubbed probabilistic representation via information-theoretic spectral modeling (PRISM). Synthesizing spectral analysis with nested manifold learning, PRISM exploits frequency-domain features to identify the intrinsic low-dimensional manifold of task-specific parameters. This mechanism aids the precise, on-demand alignment between computational resources and instance-level complexity by dynamically traversing a nested parameter subspace. Extensive empirical validation demonstrates that, even utilizing a mere 2B-parameter model, PRISM maintains high accuracy, surpassing the Gemini 3 Pro in specialized domains. Furthermore, compared to mainstream PEFT methodologies, PRISM exhibits superior performance in terms of both accuracy and inference latency.