UTPS: Universal Taylor Proxy Scoring for Sparse Multimodal Low-Rank Adaptation

Adapting Multimodal Large Language Models (MLLMs) to downstream tasks is computationally expensive. While Low-Rank Adaptation (LoRA) reduces trainable parameters, applying it uniformly across all layers is inefficient and can disrupt pretrained representations. We propose Universal Taylor Proxy Scoring (UTPS), a lightweight, training-free method that identifies the most task-relevant layers for sparse adaptation using only a single forward-backward pass. UTPS integrates gradient-weight sensitivity, activation-aware scaling, and parameter-density normalization to prioritize layers with the highest performance-per-parameter. Guided by UTPS scores, we design a sparse dynamic multi-expert LoRA architecture with multimodal-aware budget allocation. Experiments on InternVL2-2B, LLaVA-1.5-7B, InternVL2-8B, Qwen2.5-VL-3B, and LLaMA-2-7B demonstrate that UTPS, using only 50% of LoRA parameters, closely approaches or matches full LoRA performance across diverse multimodal and text-only benchmarks. Our work thus enables efficient, targeted adaptation of large models without compromising accuracy.